Summary and Conclusions

 

In this paper we have presented an extensive survey of machine recognition of human faces and a brief review of related psychological studies. We have considered two types of face recognition tasks: one from still images and the other from video. We have categorized the methods used for each type, and discussed their characteristics and their pros and cons. In addition to a detailed review of representative work, we have provided summaries of current developments and of challenging issues. We have also identified two important issues in practical face recognition systems: the illumination problem and the pose problem. We have categorized proposed methods of solving these problems and discussed the pros and cons of these methods. To emphasize the importance of system evaluation, three sets of evaluations were described: FERET, FRVT, and XM2VTS.

Getting started in performing experiments in face recognition is very easy. The Colorado State University's Evaluation of Face Recognition Algorithms website, http://www.cs.colostate.edu/evalfacerec/ has an archive of baseline face recognition algorithms. Baseline algorithms available are PCA, LDA, elastic bunch graph matching, and Bayesian Intrapersonal / Extrapersoanl Image Diffference Classifier. Source code and scripts for running the algorithms can be downloaded. The website includes scripts for running the FERET Sep96 evaluation protocol (the FERET data set needs to obtained from the FERET website). The baseline algorithms and FERET Sep96 protocol provide a framework for benchmarking new algorithms. The scripts can be modified to run different sets of images against the baseline. For on-line resources related to face recognition, such as research papers and databases, see Table .

We give below a concise summary, followed by conclusions, in the same order as the topics appear in the paper.

• Machine recognition of faces has emerged as an active research area spanning disciplines such as image processing, pattern recognition, computer vision, and neural networks. There are numerous applications of FRT to commercial systems such as face verification based ATM and access control, as well as law enforcement applications to video surveillance, etc. Due to its user-friendly nature, face recognition will remain a powerful tool in spite of the existence of very reliable methods of biometric personal identification such as fingerprint analysis and iris scans.
• Extensive research in psychophysics and the neurosciences on human recognition of faces is documented in the literature. We do not feel that machine recognition of faces should strictly follow what is known about human recognition of faces, but it is beneficial for engineers who design face recognition systems to be aware of the relevant findings. On the other hand, machine systems provide tools for conducting studies in psychology and neuroscience.
• Numerous methods have been proposed for face recognition based on image intensities [37]. Many of these methods have been successfully applied to the task of face recognition, but they have advantages and disadvantages. The choice of a method should be based on the specific requirements of a given task. For example, the EBGM-based method [115] has very good performance, but it requires an image size, e.g., , which severely restricts its possible application to video-based surveillance where the image size of the face area is very small. On the other hand, the subspace LDA method [169] works well for both large and small images, e.g., or .
• Recognition of faces from a video sequence (especially, a surveillance video) is still one of the most challenging problems in face recognition because video is of low quality and the images are small. Often, the subjects of interest are not cooperative, e.g., not looking into the camera. One particular difficulty in these applications is how to obtain good-quality gallery images. Nevertheless, video-based face recognition systems using multiple cues have demonstrated good results in relatively controlled environments.
• A crucial step in face recognition is the evaluation and benchmarking of algorithms. Two of the most important face databases and their associated evaluation methods are reviewed: the FERET, FRVT, and XM2VTS protocols. The availability of these evaluations has had a significant impact on progress in the development of face recognition algorithms.
• Although many face recognition techniques have been proposed and have shown significant promise, robust face recognition is still difficult. There are at least three major challenges: illumination, pose and recognition in outdoor imagery. A detailed review of methods proposed to solve these problems has been presented. Some basic problems remain to be solved; for example, pose discrimination is not difficult but accurate pose estimation is hard. In addition to these two problems, there are other even more difficult ones, such as recognition of a person from images acquired years apart.
• The impressive face recognition capability of the human perception system has one limitation: the number and types of faces that can be easily distinguished. Machine systems, on the other hand, can store and potentially recognize as many people as necessary. Is it really possible that a machine can be built that mimics the human perception system without its limitations on number and types?

To conclude our paper, we present a conjecture about face recognition based on psychological studies and lessons learned from designing algorithms. We conjecture that different mechanisms are involved in human recognition of familiar and unfamiliar faces. For example, it is possible that 3D head models are constructed, by extensive training for familiar faces, but for unfamiliar faces, multi-view 2D images are stored. This implies that we have full probability density functions for familiar faces; while for unfamiliar faces we only have discriminant functions.

  
Table: Internet resources for research and databases.

%