COS 429: COMPUTER VISON Face Recognition • Intro to pattern recognition • Intro to visual recognition • PCA and Eigenfaces • LDA and Fisherfaces • generic object models for faces: the Constellation Model length length weight weight Face Recognition • Digital photography Face Recognition • Digital photography • Surveillance Face Recognition • Digital photography • Surveillance • Album organization Face Recognition • Digital photography • Surveillance • Album organization • Person tracking/id. Face Recognition • Digital photography • Surveillance • Album organization • Person tracking/id. • Emotions and expressions Face Recognition • Digital photography • Surveillance • Album organization • Person tracking/id. • Emotions and expressions • Security/warfare • Tele-conferencing • Etc. What’s ‘recognition’? vs. Identification or Discrimination What’s ‘recognition’? vs. Identification or Categorization or Discrimination Classification What’s ‘recognition’? Yes, there are faces No localization Identification or Categorization or Discrimination Classification What’s ‘recognition’? Yes, there is John Lennon No localization Identification or Categorization or Discrimination Classification What’s ‘recognition’? John Lennon Detection or Localizatoin No localization Identification or Categorization or Discrimination Classification What’s ‘recognition’? Detection or Localizatoin No localization Identification or Categorization or Discrimination Classification What’s ‘recognition’? Detection or Localizatoin No localization Identification or Categorization or Discrimination Classification What’s ‘recognition’? Segment. Detection or Localizatoin No localization Identification or Categorization or Discrimination Classification Today’s agenda 4. AdaBoost 3. Constellation model Detection or Localizatoin 1. PCA & Eigenfaces 2. LDA & Fisherfaces No localization Identification or Categorization or Discrimination Classification Eigenfaces and Fishfaces • Introduction • Techniques – Principle Component Analysis (PCA) – Linear Discriminant Analysis (LDA) • Experiments The Space of Faces = + • An image is a point in a high dimensional space – An N x M image is a point in RNM – We can define vectors in this space as we did in the 2D case [Thanks to Chuck Dyer, Steve Seitz, Nishino] Key Idea P • Images in the possible set χ = { xˆ RL } are highly correlated. • So, compress them to a low-dimensional subspace that captures key appearance characteristics of the visual DOFs. • EIGENFACES: [Turk and Pentland] USE PCA! PrincipalPrincipal ComponentComponent AnalysisAnalysis (PCA)(PCA) • PCA is used to determine the most representing features among data points. – It computes the p-dimensional subspace such that the projection of the data points onto the subspace has the largest variance among all p-dimensional subspaces. IllustrationIllustration ofof PCAPCA x2 4 x2 4 6 6 1 2 1 2 3 5 3 5 X1’ x1 x1 One projection PCA projection IllustrationIllustration ofof PCAPCA 2rd principal component x2 x1 1st principal component GoGo toto thethe handouthandout…… Mathematical Formulation Find a transformation, W, m-dimensional Orthonormal n-dimensional Total scatter matrix: Wopt corresponds to m eigen- vectors of ST Eigenfaces • PCA extracts the eigenvectors of A – Gives a set of vectors v1, v2 , v3, ... – Each one of these vectors is a direction in face space • what do these look like? Projecting onto the Eigenfaces • The eigenfaces v1, ..., vK span the space of faces – A face is converted to eigenface coordinates by AlgorithmAlgorithm Training 1. Align training images x1 , x2 , …, xN Note that each image is formulated into a long vector! 2. Compute average face u = 1/N Σ xi 3. Compute the difference image φi = xi –u AlgorithmAlgorithm 4. Compute the covariance matrix (total scatter matrix) T T ST = 1/NΣφi φi = BB , B=[φ1 , φ2 … φN ] 5. Compute the eigenvectors of the covariance matrix , W Testing 1. Projection in Eigenface Projection ωi = W (X – u), W = {eigenfaces} 2. Compare projections IllustrationIllustration ofof EigenfacesEigenfaces The visualization of eigenvectors: These are the first 4 eigenvectors from a training set of 400 images (ORL Face Database). They look like faces, hence called Eigenface. Eigenfaces look somewhat like generic faces. EigenvaluesEigenvalues ReconstructionReconstruction andand ErrorsErrors P = 4 P = 200 P = 400 Only selecting the top P eigenfaces Æ reduces the dimensionality. Fewer eigenfaces result in more information loss, and hence less discrimination between faces. SummarySummary forfor PCAPCA andand EigenfaceEigenface • Non-iterative, globally optimal solution • PCA projection is optimal for reconstruction from a low dimensional basis, but may NOT be optimal for discrimination… LinearLinear DiscriminantDiscriminant AnalysisAnalysis (LDA)(LDA) • Using Linear Discriminant Analysis (LDA) or Fisher’s Linear Discriminant (FLD) • Eigenfaces attempt to maximise the scatter of the training images in face space, while Fisherfaces attempt to maximise the between class scatter, while minimising the within class scatter. IllustrationIllustration ofof thethe ProjectionProjection Using two classes as example: x2 x2 x1 x1 Poor Projection Good Projection ComparingComparing withwith PCAPCA Variables • N Sample images: {x1,,Λ xN } • c classes: {χ1,,Λ χ c} 1 • Average of each class: μi = ∑ xk N i xk∈χ i • Total average: 1 N μ = ∑ xk N k=1 Scatters S= xμ − x− μ T • Scatter of class i: i ∑ ( k i)( k i) xk∈χ i c • Within class scatter: SSW = ∑ i i=1 c T • Between class scatter: SBχ=()∑ μi− i μi μ − () μ i=1 • Total scatter: TWBS = S + S Illustration x2 SW = S1+ S 2 S1 S2 x1 SB Mathematical Formulation (1) T After projection: yk = Wk x ~ T Between class scatter (of y’s): SWSWB = B ~ T Within class scatter (of y’s): SWSWW = W Mathematical Formulation (2) • The desired projection: ~ T SB WSWB Wargopt = max~arg= max W W T SW WSWW • How is it found ? Æ Generalized Eigenvectors SBw i= λ S iW w i i = Κ m1 , , Data dimension is much larger than the number of samples n>> N Rank(S ) ≤ N − c The matrix S W is singular: W (PCA+FLD)Fisherface T • Project with PCA to N − c space zk = Wpca k x T Wargpca = WSWmax T W T • Project with FLD to c − 1 spaceyk= W fld k z T T WWSWWpcaB pca Wargfld = max W T T WWSWWpcaW pca Illustration of FisherFace • Fisherface Results: Eigenface vs. Fisherface (1) • Input: 160 images of 16 people • Train: 159 images • Test: 1 image • Variation in Facial Expression, Eyewear, and Lighting With Without 3 Lighting 5 expressions glasses glasses conditions Eigenface vs. Fisherface (2) discussion • Removing the first three principal components results in better performance under variable lighting conditions • The Firsherface methods had error rates lower than the Eigenface method for the small datasets tested. Today’s agenda 4. AdaBoost Detection or Localizatoin 3. Constellation model 1. PCA & Eigenfaces 2. LDA & Fisherfaces No localization Identification or Categorization or Discrimination Classification Parts and Structure Literature • Fischler & Elschlager 1973 • Yuille ‘91 • Brunelli & Poggio ‘93 • Lades, v.d. Malsburg et al. ‘93 • Cootes, Lanitis, Taylor et al. ‘95 • Amit & Geman ‘95, ‘99 • et al. Perona ‘95, ‘96, ’98, ’00, ‘03 • Huttenlocher et al. ’00 • Agarwal & Roth ’02 etc… Deformations A B C D Background clutter Frontal faces Face images 3D Object recognition – Multiple mixture components 3D Orientation Tuning Orientation Tuning 100 95 90 85 80 75 % Correct % Correct 70 65 60 55 50 0 20 40 60 80 100 angle in degrees Frontal Profile Thursday this week 4. AdaBoost Detection or Localizatoin 3. Constellation model 1. PCA & Eigenfaces 2. LDA & Fisherfaces No localization Identification or Categorization or Discrimination Classification.