Harris Corners 16-385 (Kris Kitani) Carnegie Mellon University How do you find a corner? [Moravec 1980]

Easily recognized by looking through a small window

Shifting the window should give large change in intensity Easily recognized by looking through a small window

Shifting the window should give large change in intensity

“flat” region: “edge”: “corner”: no change in all no change along the edge significant change in all directions direction directions

[Moravec 1980] Design a program to detect corners (hint: use image ) Finding corners (a.k.a. PCA)

@I @I I = Iy = x @x @y 1.Compute image gradients over small region

2.Subtract mean from each image

3.Compute the covariance IxIx IxIy IxIt p P p P u p P 2 2 = 2 4.Compute eigenvectors and 2 P IyIx P IyIy 3 v 2 P IyIt 3 p P p P  p P eigenvalues 4 P2 P2 5 4 P2 5

5.Use threshold on eigenvalues to detect corners 1. Compute image gradients over a small region (not just a single pixel) 1. Compute image gradients over a small region (not just a single pixel)

array of x gradients

@I I = x @x

array of y gradients

@I I = y @y visualization of gradients

image

X derivative

Y derivative @I @I @I I = I = I = y @y y @y y @y

@I @I @I I = Ix = Ix = x @x @x @x

What does the distribution tell you about the region? @I @I @I I = I = I = y @y y @y y @y

@I @I @I I = Ix = Ix = x @x @x @x

distribution reveals edge orientation and magnitude @I @I @I I = I = I = y @y y @y y @y

@I @I @I I = Ix = Ix = x @x @x @x

How do you quantify orientation and magnitude? 2. Subtract the mean from each image gradient 2. Subtract the mean from each image gradient

plot intensities

constant intensity gradient

intensities along the line 2. Subtract the mean from each image gradient

plot intensities

constant intensity gradient

intensities along the line

@I I = y @y subtract mean

@I I = x @x

plot of image gradients 2. Subtract the mean from each image gradient

plot intensities

constant intensity gradient

intensities along the line

@I @I I = I = y @y y @y subtract mean

@I @I I = Ix = x @x @x

data is centered plot of image gradients (‘DC’ offset is removed) 3. Compute the covariance matrix 3. Compute the covariance matrix

IxIx IxIy IxIt p P p P u p P 2 2 = 2 2 P IyIx P IyIy 3 v 2 P IyIt 3 p P p P  p P 4 P2 P2 5 4 P2 5 @I @I I = I = x @x y @y

IxIx IxIy =sum( *.IxIt ) p P p P u p P 2 2 2 =array of x gradients array of y gradients 2 P IyIx P IyIy 3 v 2 P IyIt 3 p P p P  p P 2 2 2 4 P P Where5 does this covariance4 P matrix come5 from? Some mathematical background… Error function

Change of intensity for the shift [u,v]:

Error Window Shifted Intensity function function intensity

Window function w(x,y) = or

1 in window, 0 outside Gaussian Error function approximation

Change of intensity for the shift [u,v]:

Second-order Taylor expansion of E(u,v) about (0,0) (bilinear approximation for small shifts):

first derivative second derivative Bilinear approximation

For small shifts [u,v] we have a ‘bilinear approximation’:

Change in appearance for a shift [u,v] where M is a 2×2 matrix computed from image derivatives:

‘second moment’ matrix ‘structure ’

M By computing the gradient covariance matrix…

IxIx IxIy IxIt p P p P u p P 2 2 = 2 2 P IyIx P IyIy 3 v 2 P IyIt 3 p P p P  p P 4 P2 P2 5 4 P2 5 we are fitting a quadratic to the gradients over a small image region Visualization of a quadratic

The surface E(u,v) is locally approximated by a quadratic form Which error surface indicates a good image feature?

What kind of image patch do these surfaces represent? flat edge corner ‘line’ ‘dot’ 4. Compute eigenvalues and eigenvectors eig(M) 4. Compute eigenvalues and eigenvectors

eigenvalue

Me = e (M I)e =0 eigenvector 4. Compute eigenvalues and eigenvectors

eigenvalue

Me = e (M I)e =0 eigenvector

1. Compute the determinant of M I (returns a polynomial) 4. Compute eigenvalues and eigenvectors

eigenvalue

Me = e (M I)e =0 eigenvector

1. Compute the determinant of M I (returns a polynomial) 2. Find the roots of polynomial det(M I)=0 (returns eigenvalues) 4. Compute eigenvalues and eigenvectors

eigenvalue

Me = e (M I)e =0 eigenvector

1. Compute the determinant of M I (returns a polynomial) 2. Find the roots of polynomial det(M I)=0 (returns eigenvalues)

3. For each eigenvalue, solve (M I)e =0 (returns eigenvectors) Visualization as an ellipse

Since M is symmetric, we have

We can visualize M as an ellipse with axis lengths determined by the eigenvalues and orientation determined by R

direction of the fastest change Ellipse equation: direction of the slowest change -1/2 (λmax) -1/2 (λmin) interpreting eigenvalues

λ2

λ2 >> λ1

What kind of image patch does each region represent?

1 0 ⇠ λ >> λ 0 1 2 2 ⇠ λ1 interpreting eigenvalues

λ2 ‘horizontal’ corner edge

λ2 >> λ1

λ1 ~ λ2

>> flat λ1 λ2 ‘vertical’ edge

λ1 interpreting eigenvalues

λ2 ‘horizontal’ corner edge

λ2 >> λ1

λ1 ~ λ2

>> flat λ1 λ2 ‘vertical’ edge

λ1 5. Use threshold on eigenvalues to detect corners 5. Use threshold on eigenvalues to detect corners

λ2

Think of a function to score ‘cornerness’

flat

λ1 5. Use threshold on eigenvalues to detect corners

λ2

strong corner Think of a function to score ‘cornerness’

flat

λ1 5. Use threshold on eigenvalues to detect corners ^ CHWPEVKQPQH λ2 corner

Use the smallest eigenvalue as the response function

R =min(1, 2)

flat

λ1 5. Use threshold on eigenvalues to detect corners ^ CHWPEVKQPQH λ2 corner

Eigenvalues need to be bigger than one. R = ( + )2 1 2 1 2 Can compute this more efficiently…

flat

λ1 5. Use threshold on eigenvalues to detect corners ^ CHWPEVKQPQH λ2 corner

R < 0 R > 0

R =det(M) trace2(M)

R 0 R < 0 ⌧ flat

λ1 Harris & Stephens (1988) R =det(M) trace2(M)

Kanade & Tomasi (1994)

R =min(1, 2)

Nobel (1998) det(M) R = trace(M)+ Harris Detector C.Harris and M.Stephens. “A Combined Corner and Edge Detector.”1988.

1. Compute x and y derivatives of image x y I x = Gσ ∗ I I y = Gσ ∗ I

2. Compute products of derivatives at every pixel

I 2 I I I 2 I I I I I x = x ⋅ x y = y ⋅ y xy = x ⋅ y 3. Compute the sums of the products of derivatives at each pixel

S 2 G I 2 S 2 = G ∗ I 2 x = σ ' ∗ x y σ ' y Sxy = Gσ ' ∗ I xy Harris Detector C.Harris and M.Stephens. “A Combined Corner and Edge Detector.”1988.

4. Define the matrix at each pixel

S 2 (x, y) S (x, y) & x xy # M (x, y) = $ ! S (x, y) S 2 (x, y) %$ xy y "! 5. Compute the response of the detector at each pixel R = det M − k(traceM )2

6. Threshold on value of R; compute non-max suppression.

Corner response

Thresholded corner response Non-maximal suppression

Harris corner response is rotation invariant

Ellipse rotates but its shape (eigenvalues) remains the same

Corner response R is invariant to image rotation intensity changes

Partial invariance to affine intensity change ü Only derivatives are used => invariance to intensity shift I → I + b

ü Intensity scale: I → a I

R R threshold

x (image coordinate) x (image coordinate) The Harris detector not invariant to changes in …