c Face Recognition Using Eigenfaces Matthew A. Turk and Alex P. Pentland Vision and Modeling Group, The Media Laboratory Massachusetts Institute of Technology Abstract another. The approach transforms face images into We present an approach to the detection and a small set of characteristic feature images, called “ ’ identification of human faces and describe a work- eigenfaces” , which are the principal components of ing, near-real-time face recognition system which the initial training set of face images. Recognition is tracks a subject’s head and then recognizes the per- performed by projecting a new image into the snb- son by comparing characteristics of the face to those space spanned by the eigenfaces (“face space”) and of known individuals. Our approach treats face then classifying the face by comparing its position in recognition as a two-dimensional recognition prob- face space with the positions of known individuals. lem, taking advantage of the fact that faces are are Automatically learning and later recognizing new normally upright and thus may be described by a faces is practical within this framework. Recogni- small set of 2-D characteristic views. Face images tion under reasonably varying conditions is achieved are projected onto a feature space (“face space”) by training on a limited number of characteristic that best encodes the variation among known face views (e.g., a “straight on” view, a 45’ view, and images. The face space is defined by the “eigen- a profile view). The approach has advantages over faces”, which are the eigenvectors of the set of faces; other face recognition schemes in its speed and sim- they do not necessarily correspond to isolated fea- plicity, learning capacity, and relative insensitivity tures such as eyes, ears, and noses. The framework to small or gradual changes in the face image. provides the ability to learn to recognize new faces in an unsupervised manner. 1.1 Background and related work Much of the work in computer recognition of faces 1 Introduction has focused on detecting individual features such as the eyes, nose, mouth, and head outline, and defin- Developing a computational model of face recogni- ing a face model by the position, size, and relation- tion is quite difficult, because faces are complex, ships among these features. Beginning with Bled- multidimensional, and meaningful visual stimuli. soe’s [2] and Kanade’s [3] early systems, a number They are a natural class of objects, and stand in of automated or semi-automated face recognition stark contrast to sine wave gratings, the “blocks strategies have modeled and classified faces based world”, and other artificial stimuli used in human on normalized distances and ratios among feature and computer vision research[l]. Thus unlike most points. Recently this general approach has been early visual functions, for which we may construct continued and improved by the recent work of Yuille detailed models of retinal or striate activity, face et al. [4]. recognition is a very high level task for which com- Such approaches have proven difficult to extend putational approaches can currently only suggest to multiple views, and have often been quite frag- broad constraints on the corresponding neural ac- ile. Research in human strategies of face recogni- tivity. tion, moreover, has shown that individual features We therefore focused our research towards devel- and their immediate relationships comprise an insuf- oping a sort of early, preattentive pattern recogni- ficient representation to account for the performance tion capability that does not depend upon having of adult human face identification [5]. Nonetheless, full three-dimensional models or detailed geometry. this approach to face recognition remains the most Our aim was to develop a computational model of popular one in the computer vision literature. face recognition which is fast, reasonably simple, Connectionist approaches to face identification and accurate in constrained environments such as sepk to capture the configurational, or gestalt-like an office or a household. nature of the task. Fleming and Cottrell [6], build- Although face recognition is a high level visual ing on earlier work by Kohonen and Lahtio [7], use problem, there is quite a bit of structure imposed on nonlinear units to train a network via back propa- the task. We take advantage of some of this struc- gation to classify face images. Stonham’s WISARD ture by proposing a scheme for recognition which is system [8] has been applied with some success to bi- based on an information theory approach, seeking nary face images, recognizing both identity and ex- to encode the most relevant information in a group pression. Most connectionist systems dealing with of faces which will best distinguish them from one faces trrat thr input image as a general 2-D pattern, 586 CH2983-5/91/0000/0586/$01.OO (0 1991 IEEE and can make no explicit use of the configurational The idea of using eigenfaces was motivated by a properties of a face. Only very simple systems have technique developed by Sirovich and Kirby [lo] for been explored to date, and it is unclear how they efficiently representing pictures of faces using prin- will scale to larger problems. cipal component analysis. They argued that a col- Recent work by Burt et al. uses a “smart sensing” lection of face images can be approximately recon- approach based on multiresolution template match- structed by storing a small collection of weights for ing [9]. This coarse-to-fine strategy uses a special- each face and a small set of standard pictures. purpose computer built to calculate multiresolution It occurred to us that if a multitude of face im- pyramid images quickly, and has been demonstrated ages can be reconstructed by weighted sums of a identifying people in near-real-time. The face mod- small collection of characteristic images, then an ef- els are built by hand from face images. ficient way to learn and recognize faces might be to build the characteristic features from known face 2 Eigenfaces for Recognition images and to recognize particular faces by compar- ing the feature weights needed to (approximately) Much of the previous work on automated face recog- reconstruct them with the weights associated with nition has ignored the issue of just what aspects of the known individuals. the face stimulus are important for identification, The following steps summarize the recognition assuming that predefined measurements were rele- process: vant and sufficient. This suggested to us that an 1. Initialization: Acquire the training set of face information theory approach of coding and decod- images and calculate the eigenfaces, which de- ing face images may give insight into the information fine the face space. content of face images, emphasizing the significant local and global “features”. Such features may or 2. When a new face image is encountered, calcu- may not be directly related to our intuitive notion late a set of weights based on the input image of face features such as the eyes, nose, lips, and hair. and the M eigenfaces by projecting the input In the language of information theory, we want image onto each of the eigenfaces. to extract the relevant information in a face image, 3. Determine if the image is a face at all (whether encode it as efficiently as possible, and compare one known or unknown) by checking to see if the face encoding with a database of models encoded image is sufficiently close to “face space.” similarly. A simple approach to extracting the infor- 4. If it is a face, classify the weight pattern as mation contained in an image of a face is to somehow either a known person or as unknown. capture the variation in a collection of face images, independent of any judgement of features, and use 5. (Optional) If the same unknown face is seen this information to encode and compare individual several times, calculate its characteristic weight face images. pattern and incorporate into the known faces In mathematical terms, we wish to find the prin- (i.e., learn to recognize it). cipal components of the distribution of faces, or the eigenvectors of the covariance matrix of the set of 2.1 Calculating Eigenfaces face images. These eigenvectors can be thought of Let a face image 1(z,y) be a two-dimensional N by as a set of features which together characterize the N array of intensity values, or a vector of dimension variation between face images. Each image location N2. A typical image of size 256 by 256 describes a contributes more or less to each eigenvector, so that vector of dimension 65,536, or, equivalently, a point we can display the eigenvector as a sort of ghostly in 65,536-dimensional space. An ensemble of im- face which we call an eigenface. Some of these faces ages, then, maps to a collection of points in this are shown in Figure 2. huge space. Each face image in the training set can be repre- Images of faces, being similar in overall configura- sented exactly in terms of a linear combination of tion, will not be randomly distributed in this huge the eigenfaces. The number of possible eigenfaces is image space and thus can be described by a rel- equal to the number of face images in the training atively low dimensional subspace. The main idea set. However the faces can also be approximated us- of the principal component analysis (or Karhunen- ing only the “best” eigenfaces - those that have the Loeve expansion) is to find the vectors which best largest eigenvalues, and which therefore account for account for the distribution of face images within the most variance within the set of face images. The the entire image space. These vectors define the primary reason for using fewer eigenfaces is compu- subspace of face images, which we call “face space”.