1934 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 27, NO. 12, DECEMBER 2005 Where Are Linear Feature Extraction Methods Applicable?
Aleix M. Martı´nez, Member, IEEE, and Manli Zhu
Abstract—A fundamental problem in computer vision and pattern recognition is to determine where and, most importantly, why a given technique is applicable. This is not only necessary because it helps us decide which techniques to apply at each given time. Knowing why current algorithms cannot be applied facilitates the design of new algorithms robust to such problems. In this paper, we report on a theoretical study that demonstrates where and why generalized eigen-based linear equations do not work. In particular, we show that when the smallest angle between the ith eigenvector given by the metric to be maximized and the first i eigenvectors given by the metric to be minimized is close to zero, our results are not guaranteed to be correct. Several properties of such models are also presented. For illustration, we concentrate on the classical applications of classification and feature extraction. We also show how we can use our findings to design more robust algorithms. We conclude with a discussion on the broader impacts of our results.
Index Terms—Feature extraction, generalized eigenvalue decomposition, performance evaluation, classifiers, pattern recognition. æ
1INTRODUCTION
N recent years, linear methods have played a major role in respectively, [28], [17], [10]. Finding the eigenvectors, v,of Ithe study of large quantities of data. Primarily, to find (1) is therefore equivalent to selecting those vectors which which features best describe a class (e.g., features that best maximize represent cars), and to build classifiers able to assign class T labels to new feature vectors (e.g., cars versus houses) [28], jv MW vj T : ð2Þ [13], [10], [16]. Linear methods have predominated because, jv MU vj when the underlying distributions of the classes are linearly Here, we assume that M and M are always symmetric separable, a small number of samples usually suffices to W U perform the above mentioned tasks, and because most and positive-defined which is necessary for these matrices problems can be formulated as an eigenvalue decomposition to define a metric. equation for which many efficient algorithms exist [11]. In this Equation (1) has been used to define many feature paper, we show that even when the classes are linearly extraction algorithms, both for classification and regression. separable, the results of such linear methods are not A well-known example is linear discriminant analysis (LDA) guaranteed to be correct. We present a simple mechanism [28], [24]. In this case, MW is defined as the sample between- to test the validity of the results of such linear methods and class scatter matrix (i.e., the covariance of the class means), define a simple robust algorithm to compute the results of Xc T eigen-based equations where our assumptions hold. In this MW ¼ SB ¼ ðÞ i ðÞ i ; ð3Þ study, a geometric interpretation of eigen-based linear i¼1 methods is presented and used as a tool for the design of where c is the number of classes, the sample mean of class i the robust algorithm. Several generalization properties of the i and the global mean (including the samples of all classes). A linear methods studied are also outlined. first option for M is the within-class scatter matrix (i.e., the The aforementioned tasks, namely, feature extraction (i.e., U sample mean covariance matrix of every class), defined as where a set of features or linear combination of them are chosen to represent each class) and classification (i.e., where 1 Xc S ¼ ; ð4Þ the feature space is linearly divided into a set of regions W n i belonging to distinct classes), can be readily computed from i¼1 thefollowinggeneralizedeigenvaluedecompositionproblem where i is the covariance matrix of the samples in class i and n and c are the number of samples and classes, respectively. M V ¼ M V ; ð1Þ W U Because minimizing the sample mean covariance matrix of all where MW and MU are the metric matrices that define the classes is equivalent to the minimization of the covariance of measure to be maximized and that to be minimized, the data, a well-known alternative for MU is the sample covariance matrix, X [10]. The sample covariance matrix is defined as . The authors are with the Department of Electrical and Computer Engineering, 205 Dreese Lab, 2015 Neil Ave., The Ohio State University, Xn 1 T Columbus, OH 43210. E-mail: {aleix, zhum}@ece.osu.edu. ¼ ðÞx ðÞx ; ð5Þ X n i i Manuscript received 20 Oct. 2004; revised 4 May 2005; accepted 5 May 2005; i¼1 published online 13 Oct. 2005. x ith Recommended for acceptance by J. Buhmann. where i represent the sample vectors. For information on obtaining reprints of this article, please send e-mail to: Figs. 1a and 1d show classical examples where LDA is [email protected], and reference IEEECS Log Number TPAMI-0569-1004. known to successfully extract the most discriminant feature,
0162-8828/05/$20.00 ß 2005 IEEE Published by the IEEE Computer Society MART IINEZ AND ZHU: WHERE ARE LINEAR FEATURE EXTRACTION METHODS APPLICABLE? 1935
Fig. 1. This figure shows three two-class and three three-class examples. (a) Shows the first and optimal (in the sense of minimizing the Bayes error) eigenvector, v1, of LDA. (b) Shows the first and optimal eigenvector of NDA. (c) The two independent components that best describe the classes (in the sense of maximizing independence between the samples of different classes), I1 and I2, are orthogonal to those of (1) when MW ¼ and MU ¼ X. (d), (e), and (f) Do the same, but for the three-class cases. v1, from a simple two-dimensional space. These results with (1) when MW estimates the variance between the were obtained by first generating a large number of samples covariance matrix of every class, i.e., from each of the two distribution and using MW ¼ SB and Xc 2 MU ¼ SW as metrics in (1). MW ¼ ¼ ðÞE½ i i ; ð7Þ Another successful application of (1), which is also well- i¼1 known within the computer vision community, is Non- where E½ i is the expected (mean) sample covariance parametric Discriminant Analysis (NDA) [9], [10]. In this matrix. Since discrimination usually requires to minimize second case, MW is the sample nonparametric between- the distance among the samples of the same class, the class scatter matrix, given by covariance matrix is generally assign as metric in MU . Figs. 1c and 1f show the independent components of the Xn 1 T I S ¼ ðÞx M ðÞx M ; ð6Þ data, i, recovered using the approach defined in the B n i i i i i i¼1 preceding paragraph. Note that the extraction of these feature vectors is necessary if one wants to project the data of the two where Mi is the mean of the k nearest samples to xi classes in a reduced space where the samples of each class belonging to a different class than that of xi and i is any collapse in a small area and those of different classes fall far scale factor which prevents the results to be affected by apart. The discriminant vectors where this happens are either isolated samples located far from the rest. orthogonal or equal to each Ii. We should also point out that As in LDA, NDA has two options for MU , the within-class neither LDA or NDA would be able to successfully obtain such results [1]. scatter matrix, SW , or the sample covariance matrix, X.In Of course, (1) is not limited to the three examples defined Figs. 1b and 1e, we show two examples where NDA is known above. For example, in metric-based Principal Component to successfully obtain the most discriminant feature vector for Analysis (PCA) [31], MW is the sample covariance matrix the given distributions. Again, the result of NDA is and MU is the covariance matrix of errors or uncertainties. represented by the vector v1. In spatiotemporal applications, the goal is to maximize the We have recently shown that (1) can also be used to signal to noise ratio which can be modeled as MW being the obtain the independent components of the data [36]. These vectors of known signals at time t and MU the spatiotem- are orthogonal or identical to the basis vectors obtained poral covariance matrix of the noise term. Linear regression 1936 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 27, NO. 12, DECEMBER 2005
Fig. 2. (a) A classical example where LDA performs as expected. Note that the angle between the first eigenvectors of MW and MU is large. (b) An example where LDA fails. In this second case, the angle between the first eigenvector of each of the matrices defining the metrics to be maximized and minimized is zero. In (a) and (b), MW ¼ SB and MU ¼ SW . Xr Xi Xr Xi 2 approaches have also been successfully formulated within 2 T K ¼ cos i;j ¼ uj wi ; ð8Þ this framework [20], [5]. And among many others, (1) has i¼1 j¼1 i¼1 j¼1 also been used in redundancy analysis, canonical correlate analysis, and principal components of instrumental vari- where r q, and i;j is the angle between the eigenvectors wi and ables [25], [22], [17], [28]. uj. Then, if K>0 the basis vectors given by (1) are not In computer vision, linear methods that are either based or guaranteed to minimize the Bayes error for the given distribu- inspired on equations similar to (1) have also flourished. tions of the data. These may be slightly different to those defined above, but our results should be extendable to several of these methods 2AGEOMETRIC INTERPRETATION OF (1) as well. Before we give a formal proof of the relationship between To date, techniques based on (1) are assumed to generally (1) and (8), it is easier to show the geometric implications of work for homoscedastic data, where all classes share a Theorem 1; which may also serve as an intuitive proof. common covariance matrix (i.e., i ¼ j, 8i; j) but have As discussed above, Fig. 2 shows the eigenvectors of MW distinct means (i.e., i 6¼ j, 8i 6¼ j). Experimental results, and MU for the LDA algorithm in two different examples. however, show otherwise. Equation (1) has been successfully In Fig. 2a, the angle between the first eigenvector of MW applied to heteroscedastic data and, most importantly, has (i.e., the first eigenvector of SB, w1) and the first of MU , u1, sometimes failed when applied to homoscedastic data. It is, is relatively large. Hence, the value of K will be small, however, unknown when and why such models do not work. which is an indication of good generalization and, if the In the following, we show that the success of the linear Bayes error is small, of good predictability. methods defined in (1) does not depend on whether the data For comparison, we also showed results on a problem is homoscedastic or not. We will demonstrate that the results where LDA does not perform as expected. This is in Fig. 2b. of (1) are not guaranteed to be correct when the smallest angle In this example, there are three classes, each represented by between the ith eigenvector of MW and the first i eigenvectors of a single Gaussian with identical covariance matrix (i.e., MU is close to zero. This problem is illustrated in Fig. 2 where homoscedastic data). We see that the basis vector generated we compare the results of LDA on two types of data. The first by LDA, v1, does not minimize the Bayes error. In fact, the example (shown in Fig. 2a) is that previously illustrated in optimal result would be the one orthogonal to v1. Fig. 1d, for which LDA is known to perform well. In contrast, To see why this happens, it is necessary to go back to the definition given in (2). According to that result, the goal is to our second example (Fig. 2b) shows an unsuccessful case for maximizethemeasuregivenbyM ,whileminimizingthatof LDA. Note from Fig. 2 that while in Fig. 2a the angle between W MU . This works well when the first eigenvector of MW and the the eigenvectors of MW and MU (w1 and u1 in the figure) is firsteigenvectorofMU areorthogonaltoeachother.However, relatively large, in the example of Fig. 2b the angle between whentheitheigenvector ofMW andthe jtheigenvector ofMU the first eigenvector of M and the first of M is zero and the W U (for any j i) are the same, the results obtained using (1) will results are incorrect. This can be nicely summarized as depend on the ratio between the eigenvalues of MW and MU . follows: And, unfortunately, in these cases, there is no way of knowing
Theorem 1. Let W ¼fw1; ...; wqg and W ¼f W1 ; ...; Wq g which option is best for classification that given by MW or that be the eigenvectors and eigenvalues of MW W ¼ W W , and of MU . Fig. 2b illustrates this. We note that in this example,
U ¼fu1; ...; upg and U ¼f U1 ; ...; Up g the eigenvectors the first eigenvector of MW and MU are the same (i.e., and eigenvalues of MU U ¼ U U , where q and p are the e1 ¼ w1 ¼ u1) and, therefore, when we use (1) to reduce the
ranks of MW and MU , respectively, W1 W2 Wq , dimensionality of our feature space to one, we cannot select a
U1 U2 Up , and p q. Define vector that has large variance according to MW and small MART IINEZ AND ZHU: WHERE ARE LINEAR FEATURE EXTRACTION METHODS APPLICABLE? 1937
Fig. 3. This figure shows that when u1 ¼ w1 the algorithms defined by (1) become unstable and may not minimize the Bayes error for the given data distributions. In these two examples, we used the metrics of LDA defined above. (a) Equation (1) does not minimize the Bayes error of the given distributions. (b) Although in this second example u1 is also equal to w1, the result given by (1) is correct. accordingtoMU .Aswecanseeinthefigure,MW wouldliketo Equations (1) and (11) have the same eigenvalues , select e1 as a solution, whereas MU would discourage the use but different eigenvectors, which are related by Y ¼ðU 1=2ÞT V. e M e U of 1. In our example, U has a larger variance along 1 than 0 1 T MW doesand,therefore,(1)willselectthevectororthogonalto u1 ! B C Xq 1 e1 (i.e. u2) as the solution. This is obviously an undesirable 1=2B . C T 1=2 U @ . A Wi wiwi U U Y ¼ Y result; see Fig. 3a. i¼1 uT We refer to this problem as the existence of a conflict 0 p 1 qP 1 between the eigenvectors of MW and MU . More formally, a B hu ; w iwT C B Wi 1 i i C measure of this conflict is given by (8). Low values of K B i¼1 C B C indicate there is no or little conflict. High values mean the 1=2B C 1=2 U B C u1; ; up U Y ¼ Y B C results of (1) need not be correct. @ qP 1 A T To formally prove this result, we first note that the Wi hup; wiiwi discriminant power of (1), 0 i¼1 1 q 1 q 1 P P ffiffiffiffiffiffiffiffiffiffiWi T ffiffiffiffiffiffiffiffiffiffiWi T p hu1;wiiw u1 p hu1;wiiw up 1 B i i C tr M MW ; ð9Þ B i¼1 U1 U1 i¼1 U1 Up C U B C B C B CY ¼ Y ; can also be expressed in the form of the geometric B C @ Pq 1 Pq 1 A interpretation given above in (8). ffiffiffiffiffiffiffiffiffiffiWi T ffiffiffiffiffiffiffiffiffiffiWi T p hup;wiiv u1 p hup;wiiw up Bi i 1 i¼1 Up U1 i¼1 Up Up Theorem 2. The tr MU MW is equal to q p where ha; bi is the inner product between a and b. X X 2 Wi T M p p uj wi : ð10Þ For anyP matrix (of rows and columns) the U p i¼1 j¼1 j trðÞ¼M i¼1 i and, therefore, Xp Proof. Since 1 trðM MW Þ¼ j U T j¼1 T 1=2 1=2 T 1=2 1=2 MU ¼ U U U ¼ U U U U ¼ U U U U ; Xp Xq pffiffiffiffiffiffiffiffiffiffiffiffiffiffiWi T ¼ huj; wiiwi uj we can rework (1) as follows: j¼1 i¼1 Uj Uj T q p 1=2 1=2 X X 2 Wi T MW V ¼ U U U U V ¼ u ; w : ut j i i¼1 j¼1 Uj 1 T 1 T T 1=2 1=2 1=2 1=2 U U MW U U U U V ¼ U U V : Equation (10) has a simple geometric interpretation too. 1=2 T Based on (10), (1) weights each pair of vectors ðu ; w Þ Now, let Y ¼ðU U Þ V. And, since U is a matrix of j i orthonormalvectors,wecansimplifytheaboveequationto according to their agreement. Those pairs with similar vectors T 2 1=2 T 1=2 will have a higher weight, wj;i ¼ðuj wiÞ , than those that U MW U Y ¼ Y : ð11Þ U U differ (i.e., those that are orthonormal to each other). Then, for 1938 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 27, NO. 12, DECEMBER 2005 those pairs that agree (i.e., those vectors of MU and MW that the basis vectors given by (1) may not minimize the Bayes favor a common direction), the goal is to maximize the ratio error of the given data distributions. Wi ; Equation (12) only searches for those vectors that result Uj in a one to one conflict between MW and MU , which in most circumstances is the trigging factor of the problem which is equivalent to (2). described above. The results presented above have the following interpreta- Moreover, it is important to note that the results of (1) tion: When the agreeing pair of vectors ðuj; wiÞ correspond to may still be correct when (12) holds or when K>0. This is a uj with associated small eigenvalue and a wi with associated illustrated in Fig. 3. In this figure, we apply LDA to two large variance, the result of (1) is optimal according to Bayes. three-class examples. The first is the same as that shown in However, when the ðuj; wiÞ pair agreeing correspond to a wi Fig. 2b, which did not produce the correct results. For with smaller eigenvalue than that of uj, the results of (1) are comparison, we now show a second example in Fig. 3b not guaranteed to be optimal because we cannot maximize where we also have u1 ¼ w1 (same as above). Yet, the result T T jv MW vj and minimize jv MU vj simultaneously. obtained by (1) in this second example is correct. This is the For example, the goal of NDA is to maximize the reason why we refer to the cases with K>0 as examples distance between the bordering samples of different classes where (1) is not stable, because, in such cases, it is not known while minimizing the intraclass distance. However, if a a priori whether the results given by (1) will be correct or vector e1 maximizes the first measure and a different vector not. Ideally, we would only want to use (1) where we know e2 (orthogonal to e1) minimizes the second, NDA will it to be stable. But, in practice, we may decide to proceed become unstable and may not be able to successfully with caution where the algorithm is know to be unstable. compute the optimal result. This problem was quantitatively measured by (10). 3ROBUST ALGORITHMS u ; w u When the agreeing vectors, ð j iÞ, are such that j has a A common problem with (1) is that of computing the small associated eigenvalue and wi has a large one, the inverse of MU . In many applications in computer vision value of (10) is large. And, when uj has a large associated (e.g., feature extraction), the number of samples is usually eigenvalue and wi a small one, the value of (10) is small. much smaller than the number of dimensions and, there- Unfortunately, we do not know how large or how small (10) fore, MU is singular. In this section, we show derivations to needs to be to help us determine when the results may be solve this problem. Our solution is based on the geometrical correct or wrong. interpretation of (1) given above. A more useful measure is, however, accounted for within We first note that the subspace spanned by the basis (10). Note that the results of (1) may be incorrect when an vectors of (1) is, in general, the same as or a subspace of that eigenvector of MW , wi, is equal or highly correlated to one spanned by the eigenvectors of MW W ¼ W W and of the first i eigenvectors of MU , because the discriminant MU U ¼ U U . Note that both matrices, MW and MU , are power will (in such cases) be very small. Therefore, the goal computed using the same set of samples and, therefore, is to minimize their basis vectors will (almost always) span a subspace of the PCA space. In those rare cases where this is not so, we Xr Xi 2 T can easily solve the problem by adding synthetic samples of uj wi ; i¼1 j¼1 such statistics to our matrices. It is therefore possible to compute the basis vectors of (1) which is the result we gave in Theorem 1, i.e., (8). Our directly from W and U as follows: equation can then be used to determine where the results of Theorem 4. The basis vectors of (1) are equivalent to those of (1) may not be correct. This is, when the value of K is large, ! the result of (1) needs not minimize the Bayes error for the Xq b T given distributions. Even when the data is homoscedastic wjwj V ¼ V ; and linearly separable. j¼1 p P In general, r needs to be smaller than or equal to b c. b p Wj T 2 where wj ¼ i¼1 ui wjui is the reconstruction of wj with M M Ui Otherwise, an eigenvector of W that is equal to one of U regard to all nonzero eigenvectors of MU U ¼ U U and can (almost always) be found and then K>0 although, in Wj properly scaled by . this case, the results may very well be correct. An obvious Ui Proof. Equation (1) can be reworked as follows: alternative is to select r so that Wr = Ur is small compared q to W = U . X 1 1 T T Another, arguably more convenient way, to calculate Wj wjwj V ¼ U U U V where (1) is applicable, is as follows: 0j0¼1 1 0 1 1 uT uT Theorem 3. If for some (i,j), with i r and j i, BB 1 C B 1 C C . . BB . C w wT þ þB . C w wT CV¼ UT V : 2 @@ . A W1 1 1 @ . A Wq q q A U T max uj wi 1; ð12Þ T T i;j up up MART IINEZ AND ZHU: WHERE ARE LINEAR FEATURE EXTRACTION METHODS APPLICABLE? 1939 ! Xq If MW is full ranked, p ¼ m (where m m is the size e T 1 wjwj V ¼ V ; of MU ), we can multiply both sides by U , U j¼1 0 1 P Pq e W T e j T u ; w w where now wj ¼ i¼1 ui wjui is the reconstruction B Wj h 1 ji j C Ui B j¼1 C obtained with the e eigenvectors associated to the e largest 1 1 B C u1 up B CV ¼ V eigenvalues of MU U ¼ U U and e Fig. 4. (a) In this two-dimensional example, the result obtained by the robust method defined in Theorem 4 is correct, whereas those computed with other approaches are not. Note that, in this example, MU , which was made equal to the covariance matrix of the data, is not singular. In (b) and (c), we have two Gaussian distributions embedded in a high-dimensional space, where now MU is singular. (b) Shows the results of our robust algorithm defined in Theorem 4. (c) Shows the result obtained using (16), (17), (18), and (19). The result shown in (b) minimizes the Bayes error, but that of (c) does not. MW W ¼ W W ð16Þ would obtained using the algorithm described in Theorem 4. and then recover the eigenvectors V of (1) by projecting MU Of course, our result is the same we would get with (1), which onto the space of MW , (in this case) is the correct one. Figs. 4b and 4c show the projection of two Gaussians 1=2 embedded in a high-dimensional space (where now X is Z ¼ W W ; ð17Þ singular) onto the one-dimensional space computed by the Y ZT M Z; ¼ U algorithm of Theorem 4 and that defined in (16), (17), (18), calculate the basis vectors in the reduced space, and (19). As we can see in the figure, our algorithm finds the optimal (according to Bayes) solution, while the other YVY ¼ VY Y; ð18Þ method does not. and, finally, map these results back to our original feature space, 4CLASSIFICATION 1=2 T T In general terms, an algorithm generalizes when the V ¼ V Z : ð19Þ Y Y empirical error (which is obtained using the training This algorithm is adequate in some cases and, generally, samples) is a close match of the test error. In what follows, yields reasonable results. However, cases exist where the we assume generalization to mean that the empirical error algorithm defined in (16), (17), (18), and (19) does not produce is a close match of the Bayes error of the underlying data the correct solution. In Fig. 4, we show two examples for the distributions. This is a useful definition within the context LDA algorithm with MW ¼ SB and MU ¼ X. In Fig. 4a, we of feature extraction as used in this paper. have a simple two-dimensional example where X is not In this section, we briefly outline some of the general- singular. In this case, the result obtained with the algorithm ization problems of (1). We also argue that variations on the defined in (16), (17), (18), and (19) does not correspond to the value of K when the ith sample is left out can be used to desirable one. For comparison, we have shown the result we determine where the linear methods discussed above may MART IINEZ AND ZHU: WHERE ARE LINEAR FEATURE EXTRACTION METHODS APPLICABLE? 1941 not generalize well. By understanding when (1) does not In the example above, we could have also used the result generalize well, we are in a position to define better of Theorem 3. Similarly to what we did above, we can algorithms in the future and, at present, help determine proposed a normalized version of that result, where current algorithms cannot be successfully applied. Xr 2 Examples demonstrating how and where we can apply e 1 T K ¼ max uj wi : r 8j i distinct linear algorithms were presented in the sections i¼1 e above. These corresponded to examples on synthetic data Using this equation, we see that K 0:06, a very low where we knew the classes were linearly separable. While our value, when we apply LDA to our data set of face images. This results are best and most easily applicable to such cases, we means that LDA should work well on faces—result con- can also use our criteria to understand why some techniques firmed by our leave-one-out test with a recognition rate of e work on some nonlinearly separable problems while others about 98 percent. However, K 0:32 when LDA is used on do not. For illustration, we will work with an example using the categorization problem, which means the classification two real data sets of objects for which the Ho-Kashyap test [6] results will now be considerably lower. This is once more ratified by our test, where the recognition rate is only about shows the classes are not linearly separable. The first of these e sets is the ETH-80 database [19], which contains several 70 percent. Similarly, K 0:6 when NDA is applied to the Ke 0:3 images of the following visual categories: apples, pears, cars, object categorization problem, and when applied to faces. This indicates that the probability of misclassification cows, horses, dogs, tomatoes, and cups. Each category (or, in general terms, of poor generalization) on the category includes the images of 10 objects (e.g., 10 different pears) task is much higher than that of faces. These results are photographed at a total of 41 orientations, which gives a total consistent with the leave-one-sample-out test, where faces are of 410 sample images per category. The second set is the AR- successfully classified about 92.4 percent of the time and face database [23]. The AR-database consists of frontal-view objects only about 33 percent. faces of a large number of subjects. Each person appears One may still wonder how much of the low performance of photographed under different lighting conditions and dis- LDA and NDA on the ETH-80 data set is due to the fact that tinct facial expressions and some images have partial the data is not linearly separable and how much to the fact occlusions. The first 13 images for a total of 100 individuals that our criteria predicted that these would not perform will be used in our test. adequately on this set. To further study this, we have In this example, we use the appearance-based frame- computed the classification accuracy of a linear Support work, where each image pixel represents a dimension Vector Machine (SVM) with soft margin and that of a K-SVM (feature). This means that for the images in the ETH-80 (Kernel-SVM) using the same leave-one-object-out test. In this experiment, we see that while the successful classification database, we will crop the smallest window that circum- rate for the K-SVM (with a Gaussian kernel) is above scribes the whole object and then resize all the resulting 86percentthatofthesoft-marginSVMisonlyabout subimages to a standard size of 25 by 30 pixels. This gives a 75 percent (which may be interpreted as an approximation feature space of 750 dimensions. The same procedure is to an upper bound of the classification accuracy of a linear applied to the face images, except that these are reduced to method). These results confirm our two main findings. First a common size of 21 by 29 pixels. Hence, our faces will be that the data is not linearly separable (since K-SVM is able to represented as vectors in a feature space of 609 dimensions. outperform SVM [34]). And, second, that the values of K=r e The two subspaces defined by MU ¼ X and MW equal to and K are good predictors of where a linear feature extraction SB (namely, LDA) and SB (NDA) are computed using all but will work, even when the data is not linearly separable. one sample image. The sample left out is then used for testing From the results discussed in the preceding paragraphs, using the Euclidean-based nearest neighbor algorithm. This we first note that the value of K should, at most, be is repeated for all possible ways of leaving one sample out. proportional to the number of eigenvectors of MW and MU . The results obtained with this leave-one-sample-out test can To see this, we need to go back to the definition of K given in then be analyzed with the use of our criteria. For example, our Theorem 1, (8). There, the ith eigenvector of MW can only be implementation of LDA (as defined in Theorem 4) performs in conflict with the first i eigenvectors of MU . Therefore, an well for faces (with a successful classification rate of upper bound of K is given by the total number of possible 98 percent), but worse on objects ( 70 percent). While combinations. And, although the result of (1) is not the number of K depend on r and, thus, cannot be used guaranteed to be correct if K 6 0, in practice K can be directly, we can (nonetheless) compare the value of K once allowed to grow when the number of eigenvectors of MW , q, this has been properly normalized by r. In our example, r can increases. Of course, this does not guarantee success, but may be made equal to the number of classes minus 1, r ¼ c 1, be used as a measure of generalization. Unfortunately, when since this is smaller than q. We see that while for faces each eigenvector of MW is in conflict with one of MU , the K=r 0:2, for objects K=r 0:34. The probability of mis- results will be incorrect most of the time. This last observation classification (as given by Theorem 1) in the category problem can be summarized as follows: Define is higher and, therefore, our implementation should only be 2 T used with care. We also note that our implementation of NDA ai ¼ max uj wi : ð20Þ j i gives K=r 0:68 when applied to objects. This indicates that NDA will most probably perform poorly on the ETH-80 data If ai 1 for all i r, the results of (1) will generally not set. This result is confirmed by our leave-one-sample-out test, minimize the Bayes error of the actual data distributions. where the successful recognition rate is about 33 percent. That is to say, the results will not necessarily generalize. 1942 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 27, NO. 12, DECEMBER 2005 This means that, in practice, if K=rremains low (as r either Finally, we would like to bring a point of caution to the decreases or increases), the results given by (1) can be results discussed in this section. While, above, some criteria expected to generalize well. When K=r is high, the results do were cited as sufficient conditions for generalization, these not usually generalize. In such cases, one may want to turn to usually assume that a sufficiently large number of training nonlinear methods or, if the goal is classification only, to other samples representing the shape of the data distributions is types of linear algorithms such as SVM [32], [6], although this available. Caution needs to be taken when the number of comes to a considerably extra computational cost. samples is small, especially when the number of samples is A somehow different way of estimating where a learning smaller than the number of dimensions. When the value of algorithm generalizes is to look at its stability. One K remains low and approximately constant but the number possibility, which has recently proven success, is to study of samples is small, (1) usually results in overfitting. This what happens when one of the training samples is left out means that the training data is very well learned (and [4], [26]. Typically, if the results (or the empirical error) do usually optimally separated), but that the results do not not change when different samples are left out, the generalize to new samples. The main reason for such result is algorithm is said to be stable. Under some conditions, such given by the known fact that when the number of samples is stabilities imply generalization. much smaller than the number of dimensions, the classes In our case, we would like to know whether (8) or (20) can be generally separated with many distinct hyperplanes. remain close to zero (or, alternatively, proportional to r) While this problem has been largely studied within some when different training samples are left out. If this happens contexts [32], it remains largely unsolved in multivariate and the sample metrics are close matches of the real ones analysis and, in particular, in feature extraction. (even if one of the samples is left out), we can assume the The problem described in this paper is not only algorithm generalizes well. In practice, stability and gen- encountered in computer vision problems [23], [3], [15]. In eralization may be given when Ki Kj for all i 6¼ j and other areas, such as bioinformatics, this is also predomi- Ki Aleix M. Martı´nez is an assistant professor in the Manli Zhu received the MS degree in electrical Department of Electrical and Computer Engi- engineering in 2002 from Zhejiang University, neering at The Ohio State University (OSU). He China. She is currently a PhD candidate in the is also affiliated with the Department of Biome- Department of Electrical and Computer Engi- dical Engineering and to the Center for Cognitive neering at The Ohio State University. Her Science, both at OSU. Prior to joining OSU, he research interests are in the field of statistical was affiliated with the Electrical and Computer pattern recognition, especially in feature extrac- Engineering Department at Purdue University tion. She is also working on object classification and with the Sony Computer Science Lab. He and face recognition. coorganized the First IEEE Workshop on Com- puter Vision and Pattern Recognition for Human-Computer Interaction (2003) and served as a cochair for the Second IEEE Workshop on Face Processing in Video (2005). In 2003, he served as a coguest editor of the special issue on face recognition in the journal Computer Vision and Image Understanding. He is a coauthor of the AR-face database and the . For more information on this or any other computing topic, Purdue ASL database. His areas of interest are learning, vision, please visit our Digital Library at www.computer.org/publications/dlib. linguistics, and their interaction. He is a member of the IEEE.