Last Week

GV12/3072 1 Image Processing. Last Week: Performance

• How can we evaluate edge-detector performance? • Probability of false edges • Probability of missing edges • Error in edge angle • Mean squared distance from true edge

GV12/3072 2 Image Processing. Last Week: Summary

• Resize by re-sampling (must pre-filter!!) • Image pyramids applications • • Simple • Canny •

GV12/3072 3 Image Processing. GV12/3072 4 Image Processing. Corner Detection

GV12/3072 5 Image Processing. Outline • Corners and point features • Moravec operator • Image • Harris corner detector • SUSAN • FAST • SIFT

GV12/3072 6 Image Processing. Point Features (a.k.a. Corners or Interest Points) • Many applications need to match corresponding points in images. • Stereo matching and 3D reconstruction • Tracking • Localization • Recognition

• Points along lines are poorly defined – See Aperture Problem phenomenon (next)

GV12/3072 7 Image Processing.

Tracking & Reconstruction

• 2D3 Commercial application: Boujou – Match-moving for special effects – Computes a point-cloud + camera poses

• UNC city-scanning (video) (Pollefeys et al.)

GV12/3072 12 Image Processing. Feature matching vs. tracking

Image-to-image correspondences are key to passive triangulation-based 3D reconstruction

Extract features independently and Extract features in first images and then match by comparing then try to find same feature back descriptors in next view

What is a good feature?

From Marc Pollefeys Stereo Example

GV12/3072 14 Image Processing. Stereo Example

GV12/3072 15 Image Processing. “Corner” Interest Points • We want corner points that are: • Distinctive • Stable from image to image • (Sparse) ? • And are invariant to: • View point (scale, orientation, translation) • Lighting conditions • Object deformations • Partial occlusion • And are, if possible, • Geometrically meaningful, though not necessarily scene-object corners. 16 Intersections

• Edge and line intersections provide recognizable features

Intersections can be stable or unstable 17 Moravec “Interest” Operator

• Use a window surrounding each pixel as its own matching template • Tests local autocorrelation of the image: – SSD = Sum of Squared Differences • Good matches in any direction • Flat image region • Good matches in only one direction • Linear feature or edge • No good matches in any direction • Distinctive point feature • Corner point GV12/3072 18 Image Processing. Neighborhood stands out?

GV12/3072 20 Image Processing. Check SSDs

21 sum(sum( (patch - Ipad(rowStart:rowStart+2, colStart:colStart+2) ).^2 )); Check SSDs

1.34 2.6 1.79

1.04 2.46 2.34

2.39 0 3.24

1.14 2.12 0.60

22 Implementation

• Match quality for displacements (∆x, ∆y) uses L2 norm:  (x,y)  w(x, y)I(x  x, y  y)  I(x, y)2 (x,y)N (3,3)of I, centered at p0 • w is a window function – e.g. constant or Gaussian • Cornerness is min  over the eight- neighbourhood N, excluding (0, 0)

GV12/3072 23 Image Processing. Moravec “Interest” Operator

• Use a window surrounding each pixel as its own matching template T. • Tests local autocorrelation of the image: SSD • Good matches in any direction • Flat image region • Good matches in only one direction • Linear feature or edge • No good matches in any direction • Distinctive point feature • Corner point

GV12/3072 24 Image Processing. Cornerness

• Using 7x7 matching window.

25 Cornerness distribution

# Pixels

26 Cornerness Non-maximal suppression

• Set pixels that have an 8 neighbour with higher cornerness to zero.

27 Threshold T = 1

GV12/3072 28 Image Processing. Threshold T = 2

GV12/3072 29 Image Processing. Threshold T = 3

GV12/3072 30 Image Processing. Non-max Suppression Efficiently?

Input, I (this time, pretend these are cornerness values)

31 Non-max Suppression by Dilation?

Input, I (this time, pretend these are cornerness values)

imdilate(I, ones(3,3))

33 Moravec Algorithm Summary

• Enhance corner features • Non-maximal suppression • Threshold

GV12/3072 34 Image Processing. Problems

• Multiple responses at high interest points • Extend non-maximal suppression to windows

• Weak response to “blurred” corners

• Slow

GV12/3072 35 Image Processing. General Form

Can we do better? Derive general theory of “cornerness” (x, y)   w(x, y)I(x  x, y  y)  I(x, y)2 (x,y)N (3,3)of I, centered at p0

I(x, y) I(x, y)x I(x  x, y  y)  I(x, y)   ,    x y y

(See online Taylor series expansion/approximation) 38 2 (x, y)   w(x, y)I(x  x, y  y)  I(x, y) (assume constant w) (x,y)N (3,3)of I, centered at p0 I(x, y) I(x, y)x I(x  x, y  y)  I(x, y)   ,    x y y  (x, y)  2  I(x, y) I(x, y)x  I(x , y )  I(x , y )  ,     i i i i  x y y xiN yiN    2 I(x, y) I(x, y)x  ,  (Note : u2  uT u)    x y y xiN yiN   I     xI I  x [x, y]  ,     I x y y xiN yiN        39 y   General Form  (x,y)  w(x, y)I(x  x, y  y)  I(x, y)2 (x,y)N (3,3)of I, centered at p0 I(x, y) I(x, y)x I(x  x, y  y)  I(x, y)   ,    x y y

 I 2 I I x (x, y)  x, y x x y    xT Sx  I I I 2 y  x y y  

40 General Form  (x,y)  w(x, y)I(x  x, y  y)  I(x, y)2 (x,y)N (3,3)of I, centered at p0 I(x, y) I(x, y)x I(x  x, y  y)  I(x, y)   ,   -<…> denotes average overx the windowy  N. y - S is the image structure tensor - a.k.a „Harris‟ Matrix

 I 2 I I x (x, y)  x, y x x y    xT Sx  I I I 2 y  x y y  

41 Structure tensor

• S captures the of the local autocorrelation surface • The eigenvalues are the principal

• They are the solutions to 2 2 2 2 2 2    I x  I y  I x I y  I x I y  0

GV12/3072 44 Image Processing. Principal Axes (i.e. Eigenvectors) (Note: see Simon Prince’s SVD slides online)

homogeneous

edge

corner

i.e. maximize smallest eigenvalue of S 45 From Marc Pollefeys Harris Corner Detector (Example Video) • Defines cornerness as size of smallest eigenvalue, or C  det(S) /Tr(S) 2 C   I 2 I 2  I I  / I 2  I 2   x y x y  x y

C  12 /(1  2 ) • Non-maximal suppression and thresholding as before

GV12/3072 47 Image Processing. Cornerness and NMS

48 Sigma = 1, Threshold T=0.1

GV12/3072 49 Image Processing. Sigma = 1, Threshold T=0.15

GV12/3072 50 Image Processing. Sigma = 1, Threshold T=0.2

GV12/3072 51 Image Processing. Variants

• Can include Gaussian smoothing when computing Ix and Iy. • Less important than for edge detection. • Helps eliminate multiple responses to the same corner • Similar effect using larger regions in non-maximal suppression • Harris and Stephens combined edge and corner detector 2 • R  det(S)  kTr (S) • Various other corner measures, thresholding schemes, non-max suppression techniques • 0

GV12/3072 53 Image Processing. Sigma = 5, Threshold T=0.004

GV12/3072 54 Image Processing. Sigma = 5, Threshold T=0.007

GV12/3072 55 Image Processing. Harris and Stephens

k = 0.02

Harris and Stephens, Proc. 4th Alvey Vision

Conference, 147-151, GV12/3072 56 1988. Image Processing. SUSAN

• Method for edge and corner detection • No image derivatives • Insensitive to noise

• Smith & Brady, IJCV, 23(1):45-78, 1997

GV12/3072 62 Image Processing. USAN

“Univalue Segment Assimilating Nucleus”

It is the portion of the template with intensity within a threshold of the “nucleus”.

GV12/3072 63 Image Processing. Edges and Corners

• In flat regions the USAN has similar area to the template • At edges the USAN area is about half the template area • At corners the USAN area is smaller than half the template area. • “SUSAN” = Smallest USAN.

GV12/3072 64 Image Processing. Example

GV12/3072 65 Image Processing. Implementation 0011100 0111110 • Circular mask M with radius 3.4 1111111 (|M| = 37 pixels). M=1111111 1111111 • The nucleus is the centre pixel r0 0111110 • U is the USAN area. 0011100 • C is the corner/edge strength 1 I(r)  I(r )  t  0 n  u(r,r0 ) u(r,r0 )    0 otherwise rC(r0 ) t=3|M|/4 for edge detection  M  n n  T t=|M|/2 for corner detection C(r0 )   0 otherwise Select t by considering image noise level. Refinements

• „Band‟ edge orientation from USAN moments: 1 tan 20 / 02  • Step edge orientation from centre of gravity.

• Eliminate false positive corners using

• Distance of USAN centre of gravity from r0. • Connectivity of USAN. • Non-maximal suppression

GV12/3072 67 Image Processing. FAST Machine learning for high-speed corner detection Rosten & Drummond, ECCV 2006 • Reuse nucleus concept – From Trajkovic & Hedley‟98

• Machine learning to train it to be fast

• Retain performance

GV12/3072 68 Image Processing. Trajkovic & Hedley • P and P’ are opposite points, diameter D apart

GV12/3072 69 Image Processing. From Rosten & Drummond FAST • Set of n contiguous pixels in the circle which are all brighter / darker by some T – n = 12..?

70 • For each location (1-16) on the circle x, the pixel at that position relative to p (denoted by p  x) can have one of three states:

Train decision tree to maximize information gain:

GV12/3072 71 Image Processing. FAST Performance

• On average, 2.26 (for n = 9) and 2.39 (for n = 12) questions are asked per pixel to determine whether or not it is a feature. By contrast, the handwritten detector asks on average 2.8 questions. 72 SIFT

• Scale Invariant Feature Transform. • Detects “scale-space extrema”. • Highly stable features • Now widely used in .

• D.G. Lowe, IJCV Vol. 60(2) 91-110 2004.

GV12/3072 73 Image Processing. SIFT Application: Autostitch

GV12/3072 74 Image Processing. DoG

…

Subtract

Efficiently … computed Laplacian scale space.

GV12/3072 75 Image Processing. LoG vs DoG

k=LoG(9,3); k= GausKern(9,3)-GausKern(9,4.8); surf(x,y,-k); surf(x,y,-k);

This shows that when the difference-of-Gaussian function has scales differing by a constant factor it already incorporates the σ2 scale normalization required for the scale-invariant Laplacian. - From Lowe, 2004 76 GV12/3072 77 Image Processing. Scale space extrema

• Find local maxima and minima in the scale space stack. • These are SIFT keypoints.

GV12/3072 78 Image Processing. SIFT Keypoints

GV12/3072 79 Image Processing. Threshold local

• Keep the top 50 %. • Notice the response along edges.

GV12/3072 80 Image Processing. Threshold Harris cornerness

Lowe thresholds a cornerness measure in the scale space. GV12/3072 81 Image Processing. SIFT Features

• Extrema in Laplacian are distinctive (after removing edges) • Extrema in scale space give scale independence.

• Lowe creates features at each keypoint from the histogram of local edge orientations • Very stable features for affine matching

GV12/3072 82 Image Processing. Evaluation of Interest- point Detectors? • Ideas?

• Innovations?

GV12/3072 83 Image Processing. Summary • Corners and point features • Various algorithms • Moravec • Harris (image structure tensor) • SUSAN • FAST

• Coming up: Description!

GV12/3072 84 Image Processing.