7. Embeded Face and Facial Expression Recognition

We are currently studying a Bayesian recognition framework in which a model of the whole face cosists on models of facial feature position and appearances. A hidden variable is used to model facial expressions. Face recognition and facial expression recognition are carried out using maximum likelihood decisions.

Face recognition is carried out by selecting the person's model and the facial expression that maximizes the likelihood of the test images. Then, in a person dependent context, expression recognition is achieved by selecting the facial expression with maximum likelihood.

In this framework, face models jointly capture information about facial appearance and expression patterns so that recognition of faces and facial expressions are carried at the same time. Face and facial expression recognition cooperate so that the similarity measure used for face recognition benefits from facial expression modeling. Conversely, expression recognition is improved by facial appearance modeling.

In our current implementation, the face is divided into 9 facial features grouped in 4 regions which are detected and tracked automatically in video segments. The following figure illustrates schematically the probability network used to model faces.

The following MPEG videos show the results of our algorithm for maximum likelihood expression recognition

We are also extending this framework to account for temporal correlation of facial expression. The hidden variable used to index the expression in is replaced by a hidden state in a HMM. Face and facial expression recognition from video segments is then carried out using either the Viterbi algorithm or the forward-backward algorithm in a maximum likelihood setup.