COS 429: COMPUTER VISON Face Recognition

• Intro to pattern recognition • Intro to visual recognition • PCA and Eigenfaces • LDA and Fisherfaces • generic object models for faces: the Constellation Model

length length

weight

weight

Face Recognition

• Digital photography Face Recognition

• Digital photography • Surveillance Face Recognition

• Digital photography • Surveillance • Album organization Face Recognition

• Digital photography • Surveillance • Album organization • Person tracking/id. Face Recognition

• Digital photography • Surveillance • Album organization • Person tracking/id. • Emotions and expressions Face Recognition

• Digital photography • Surveillance • Album organization • Person tracking/id. • Emotions and expressions • Security/warfare • Tele-conferencing • Etc. What’s ‘recognition’?

vs.

Identification or Discrimination What’s ‘recognition’?

vs.

Identification or Categorization or Discrimination Classification What’s ‘recognition’?

Yes, there are faces No localization Identification or Categorization or Discrimination Classification What’s ‘recognition’?

Yes, there is John Lennon No localization Identification or Categorization or Discrimination Classification What’s ‘recognition’?

John Lennon Detection or Detection Localizatoin No localization Identification or Categorization or Discrimination Classification What’s ‘recognition’? Detection or Detection Localizatoin No localization Identification or Categorization or Discrimination Classification What’s ‘recognition’? Detection or Detection Localizatoin No localization Identification or Categorization or Discrimination Classification What’s ‘recognition’? Segment. Detection or Detection Localizatoin No localization Identification or Categorization or Discrimination Classification Today’s agenda

4. AdaBoost

3. Constellation model Detection or Detection Localizatoin

1. PCA & Eigenfaces 2. LDA & Fisherfaces No localization Identification or Categorization or Discrimination Classification Eigenfaces and Fishfaces

• Introduction • Techniques – Principle Component Analysis (PCA) – Linear Discriminant Analysis (LDA) • Experiments The Space of Faces

= +

• An image is a point in a high dimensional space – An N x M image is a point in RNM – We can define vectors in this space as we did in the 2D case

[Thanks to Chuck Dyer, Steve Seitz, Nishino] Key Idea

P • Images in the possible set χ = { xˆ RL } are highly correlated.

• So, compress them to a low-dimensional subspace that captures key appearance characteristics of the visual DOFs.

• EIGENFACES: [Turk and Pentland]

USE PCA! PrincipalPrincipal ComponentComponent AnalysisAnalysis (PCA)(PCA)

• PCA is used to determine the most representing features among data points. – It computes the p-dimensional subspace such that the projection of the data points onto the subspace has the largest variance among all p-dimensional subspaces. IllustrationIllustration ofof PCAPCA

x2 4 x2 4 6 6 1 2 1 2 3 5 3 5

X1’

x1 x1

One projection PCA projection IllustrationIllustration ofof PCAPCA

2rd principal component

x2

x1

1st principal component GoGo toto thethe handouthandout…… Mathematical Formulation

Find a transformation, W,

m-dimensional Orthonormal n-dimensional

Total scatter matrix:

Wopt corresponds to m eigen- vectors of ST Eigenfaces

• PCA extracts the eigenvectors of A

– Gives a set of vectors v1, v2 , v3, ... – Each one of these vectors is a direction in face space • what do these look like? Projecting onto the Eigenfaces

• The eigenfaces v1, ..., vK span the space of faces

– A face is converted to eigenface coordinates by AlgorithmAlgorithm Training

1. Align training images x1 , x2 , …, xN

Note that each image is formulated into a long vector!

2. Compute average face u = 1/N Σ xi

3. Compute the difference image φi = xi –u AlgorithmAlgorithm 4. Compute the covariance matrix (total scatter matrix) T T ST = 1/NΣφi φi = BB , B=[φ1 , φ2 … φN ] 5. Compute the eigenvectors of the covariance matrix , W

Testing 1. Projection in Eigenface

Projection ωi = W (X – u), W = {eigenfaces} 2. Compare projections IllustrationIllustration ofof EigenfacesEigenfaces The visualization of eigenvectors:

These are the first 4 eigenvectors from a training set of 400 images (ORL Face Database). They look like faces, hence called Eigenface. Eigenfaces look somewhat like generic faces. EigenvaluesEigenvalues ReconstructionReconstruction andand ErrorsErrors

P = 4

P = 200

P = 400

Only selecting the top P eigenfaces Æ reduces the dimensionality. Fewer eigenfaces result in more information loss, and hence less discrimination between faces. SummarySummary forfor PCAPCA andand EigenfaceEigenface

• Non-iterative, globally optimal solution • PCA projection is optimal for reconstruction from a low dimensional basis, but may NOT be optimal for discrimination… LinearLinear DiscriminantDiscriminant AnalysisAnalysis (LDA)(LDA)

• Using Linear Discriminant Analysis (LDA) or Fisher’s Linear Discriminant (FLD) • Eigenfaces attempt to maximise the scatter of the training images in face space, while Fisherfaces attempt to maximise the between class scatter, while minimising the within class scatter. IllustrationIllustration ofof thethe ProjectionProjection

Š Using two classes as example:

x2 x2

x1 x1

Poor Projection Good Projection ComparingComparing withwith PCAPCA Variables

• N Sample images: {x1 Λ ,, xN } • c classes: {χ1 Λ ,, χ c} 1 • Average of each class: μi = ∑ xk N i x ∈χ ik

• Total average: 1 N μ = ∑ xk N k=1 Scatters

( −= )(xxS − μμ)T • Scatter of class i: i ∑ ik ik x ∈χ ik

c • Within class scatter: W = ∑ SS i i=1

c T • Between class scatter: SB = ∑ ()()ii i −− μμμμχ i=1

• Total scatter: S = S + SBWT Illustration

x2 SW = S + S21

S1

S2 x1 SB Mathematical Formulation (1)

T After projection: k = xWy k

~ T Between class scatter (of y’s): B = BWSWS ~ T Within class scatter (of y’s): W = WWSWS Mathematical Formulation (2) • The desired projection: ~ T SB BWSW Wopt = arg max ~ = arg max W W T SW WWSW • How is it found ? Æ Generalized Eigenvectors

S w = λiiB SW wi i = Κ ,,1 m Data dimension is much larger than the number of samples >> Nn Rank(S )≤ N − c The matrix S W is singular: W Fisherface (PCA+FLD)

T • Project with PCA to N − c space k = pca xWz k

T Wpca = arg max TWSW W

T • Project with FLD to c − 1 space = fldk zWy k

T T pca pcaB WWSWW W fld = arg max W T T pca pcaW WWSWW Illustration of FisherFace • Fisherface Results: Eigenface vs. Fisherface (1)

• Input: 160 images of 16 people • Train: 159 images • Test: 1 image • Variation in Facial Expression, Eyewear, and Lighting

With Without 3 Lighting 5 expressions glasses glasses conditions Eigenface vs. Fisherface (2) discussion • Removing the first three principal components results in better performance under variable lighting conditions • The Firsherface methods had error rates lower than the Eigenface method for the small datasets tested. Today’s agenda

4. AdaBoost Detection or Detection Localizatoin 3. Constellation model

1. PCA & Eigenfaces 2. LDA & Fisherfaces No localization Identification or Categorization or Discrimination Classification Parts and Structure Literature

• Fischler & Elschlager 1973

• Yuille ‘91 • Brunelli & Poggio ‘93 • Lades, v.d. Malsburg et al. ‘93 • Cootes, Lanitis, Taylor et al. ‘95 • Amit & Geman ‘95, ‘99 • et al. Perona ‘95, ‘96, ’98, ’00, ‘03 • Huttenlocher et al. ’00 • Agarwal & Roth ’02 etc… Deformations

A B

C D Background clutter Frontal faces Face images 3D Object recognition – Multiple mixture components 3D Orientation Tuning

Orientation Tuning 100

95

90

85

80

75 % Correct

% Correct 70

65

60

55

50 0 20 40 60 80 100 angle in degrees

Frontal Profile Thursday this week

4. AdaBoost Detection or Detection Localizatoin 3. Constellation model

1. PCA & Eigenfaces 2. LDA & Fisherfaces No localization Identification or Categorization or Discrimination Classification