Introduction

 

%As one of the most successful applications of image analysis and understanding, face recognition has recently received significant attention, especially during the past few years. This is evidenced by the emergence of face recognition conferences such as AVBPA [1] and AFGR [2], systematic empirical evaluations of face recognition techniques (FRT), including the FERET  [126, 129, 125] FRVT 2000 [25], FRVT 2002 [122], and XM2VTS [108] protocols, and many commercially available systems (Table ). There are at least two reasons for this trend; the first is the wide range of commercial and law enforcement applications and the second is the availability of feasible technologies after 30 years of research. In addition, the problem of machine recognition of human faces continues to attract researchers from disciplines such as image processing, pattern recognition, neural networks, computer vision, computer graphics, and psychology.

The strong need for user-friendly systems that can secure our assets and protect our privacy without losing our identity in a sea of numbers is obvious. At present, one needs a PIN to get cash from an ATM, a password for a computer, a dozen others to access the internet, and so on. Although very reliable methods of biometric personal identification exist, e.g., fingerprint analysis and retinal or iris scans, these methods rely on the cooperation of the participants, whereas a personal identification system based on analysis of frontal or profile images of the face is often effective without the participant's cooperation or knowledge. Some of the advantages/disadvantages of different biometrics are described in [124]. Table  lists some of the applications of face recognition.

  
Table: Typical applications of face recognition.

Commercial and law enforcement applications of FRT range from static, controlled-format photographs to uncontrolled video images, posing a wide range of technical challenges and requiring an equally wide range of techniques from image processing, analysis, understanding and pattern recognition. One can broadly classify FRT systems into two groups depending on whether they make use of static images or of video. Within these groups, significant differences exist, depending on the specific application. The differences are in terms of image quality, amount of background clutter (posing challenges to segmentation algorithms), variability of the images of a particular individual that must be recognized, availability of a well-defined recognition or matching criterion, and the nature, type and amount of input from a user. A list of some commercial systems is given in Table .

  
Table: Available commercial face recognition systems. (Some of these websites may have changed or been removed.) [The identification of any company, commercial product, or trade name does not imply endorsement or recommendation by the National Institute of Standards and Technology or any of the authors or their institutions.]

A general statement of the problem of machine recognition of faces can be formulated as follows: Given still or video images of a scene, identify or verify one or more persons in the scene using a stored database of faces. Available collateral information such as race, age, gender, facial expression, or speech may be used in narrowing the search (enhancing recognition). The solution to the problem involves segmentation of faces (face detection) from cluttered scenes, feature extraction from the face regions, recognition, or verification (Fig. ). In identification problems, the input to the system is an unknown face, and the system reports back the determined identity from a database of known individuals, whereas in verification problems, the system needs to confirm or reject the claimed identity of the input face.

  
Figure: Configuration of a generic face recognition system.

Face perception is an important part of the capability of human perception system and is a routine task for humans, while building a similar computer system is still an on-going research area. The earliest work on face recognition can be traced back at least to the 1950's in psychology [35] and to the 1960's in the engineering literature [28]. (Some of the earliest studies include work on facial expression of emotions by Darwin [44] (see also Ekman [48]) and on facial profile based biometrics by Galton [54]). But research on automatic machine recognition of faces really started in the 1970's after the seminal work of Kanade [80] and Kelly [81]. Over the past thirty years extensive research has been conducted by psychophysicists, neuroscientists, and engineers on various aspects of face recognition by humans and machines. Psychophysicists and neuroscientists have been concerned with issues such as whether face perception is a dedicated process (this issue is still being debated in the psychology community [49, 21, 56, 55]) and whether it is done holistically or by local feature analysis.

Many of the hypotheses and theories put forward by researchers in these disciplines have been based on rather small sets of images. Nevertheless, many of the findings have important consequences for engineers who design algorithms and systems for machine recognition of human faces. Section 2 will present a concise review of these findings.

Barring a few exceptions that use range data [62], the face recognition problem has been formulated as recognizing 3D objects from 2D images . Earlier approaches treated it as a 2D pattern recognition problem. As a result, during the early and mid-1970's, typical pattern classification techniques, which use measured attributes of features (e.g. the distances between important points) in faces or face profiles, were used [28, 81, 80]. During the 1980's, work on face recognition remained largely dormant. Since the early 1990's, research interest in FRT has grown significantly. One can attribute this to several reasons: An increase in interest in commercial opportunities; the availability of real-time hardware; and the increasing importance of surveillance-related applications.

Over the past 15 years, research has focused on how to make face recognition systems fully automatic by tackling problems such as localization of a face in a given image or video clip and extraction of features such as eyes, mouth, etc. Meanwhile, significant advances have been made in the design of classifiers for successful face recognition. Among appearance-based holistic approaches, eigenfaces [82, 153] and Fisherfaces [12, 50, 168] have proved to be effective in experiments with large databases. Feature-based graph matching approaches [160] have also been quite successful. Compared to holistic approaches, feature-based methods are less sensitive to variations in illumination and viewpoint and to inaccuracy in face localization. However, the feature extraction techniques needed for this type of approach are still not reliable or accurate enough [42]. For example, most eye localization techniques assume some geometric and textural models and do not work if the eye is closed. Section 3 will present a review of still-image based face recognition.

During the past five to eight years, much research has been concentrated on video-based face recognition. The still image problem has several inherent advantages and disadvantages. For applications such as drivers' licenses, due to the controlled nature of the image acquisition process, the segmentation problem is rather easy. However, if only a static picture of an airport scene is available, automatic location and segmentation of a face could pose serious challenges to any segmentation algorithm. On the other hand, if a video sequence is available, segmentation of a moving person can be more easily accomplished using motion as a cue. But the small size and low image quality of faces captured from video can significantly increase the difficulty in recognition. Video-based face recognition is reviewed in Section 4.

As we propose new algorithms and build more systems, measuring the performance of new systems and of existing systems becomes very important. Systematic data collection and evaulation of face recognition systems is reviewed in Section 5.

Recognizing a 3D object from its 2D images poses many challenges. The illumination and pose problems are two prominent issues for appearance- or image-based approaches. Many approaches have been proposed to handle these issues, with the majority of them exploring domain knowledge. Details of these approaches are discussed in Section 6.

In 1995, a review paper [37] gave a thorough survey of FRT at that time. (An earlier survey [134] appeared in 1992.) At that time, video-based face recognition was still in a nascent stage. During the past eight years, face recognition has received increased attention and has advanced technically. Many commercial systems for still face recognition are now available. Recently, significant research efforts have been focused on video-based face modeling/tracking, recognition, and system integration. New datasets have been created and evaluations of recognition techniques using these databases have been carried out. It is not an overstatement to say that face recognition has become one of the most active applications of pattern recognition, image analysis and understanding.

In this paper we provide a critical review of current developments in face recognition. This paper is organized as follows: In Section 2 we briefly review issues that are relevant from a psychophysical point of view. Section 3 provides a detailed review of recent developments in face recognition techniques using still images. In Section 4 face recognition techniques based on video are reviewed. Data collection and performance evaluation of face recognition algorithms are addressed in Section 5 with descriptions of representative protocols. In Section 6 we discuss two important problems in face recognition that can be mathematically studied, lack of robustness to illumination and pose variations, and we review proposed methods of overcoming these limitations. Finally, a summary and conclusions are presented in Section 7.