Outline •Features • Harris corner detector • SIFT Features •Extensions • Applications Digital Visual Effects Yung-Yu Chuang with slides by Trevor Darrell Cordelia Schmid, David Lowe, Darya Frolova, Denis Simakov, Robert Collins and Jiwon Kim Features • Also known as interesting points, salient points or keypoints. Points that you can easily point out their correspondences in multiple images using only local information. Features ? Desired properties for features Applications • Distinctive: a single feature can be correctly • Object or scene recognition matched with high probability. • Structure from motion • Invariant: invariant to scale, rotation, affine, •Stereo illumination and noise for robust matching • Motion tracking across a substantial range of affine distortion, viewpoint change and so on. That is, it is •… repeatable. Components • Feature detection locates where they are • Feature description describes what they are • Feature matching decides whether two are the same one Harris corner detector Moravec corner detector (1980) Moravec corner detector • We should easily recognize the point by looking through a small window • Shifting a window in any direction should give a large change in intensity flat Moravec corner detector Moravec corner detector flat flat edge Moravec corner detector Moravec corner detector Change of intensity for the shift [u,v]: E(u,v) w(x, y)I(x u, y v) I(x, y) 2 x, y window shifted intensity function intensity corner flat edge Four shifts: (u,v) = (1,0), (1,1), (0,1), (-1, 1) isolated point Look for local maxima in min{E} Problems of Moravec detector Harris corner detector • Noisy response due to a binary window function Noisy response due to a binary window function • Only a set of shifts at every 45 degree is Use a Gaussian function considered • Only minimum of E is taken into account Harris corner detector (1988) solves these problems. Harris corner detector Harris corner detector Only a set of shifts at every 45 degree is considered Only a set of shifts at every 45 degree is considered Consider all small shifts by Taylor’s expansion Consider all small shifts by Taylor’s expansion E(u,v) w(x, y)I(x u, y v) I(x, y) 2 x, y 2 2 2 w(x, y)I xu I yv O(u ,v ) x, y E(u,v) Au2 2Cuv Bv 2 2 A w(x, y)I x (x, y) x, y 2 B w(x, y)I y (x, y) x, y C w(x, y)I x (x, y)I y (x, y) x, y Harris corner detector Harris corner detector (matrix form) Equivalently, for small shifts [u,v] we have a bilinear 2 E(u) w(x0 ) | I(x0 u) I(x0 ) | approximation: x0W (p) 2 | I(x0 u) I(x0 ) | u 2 E(u,v) u v M I T I u I v 0 0 x 2 , where M is a 22 matrix computed from image derivatives: I T u 2 x I x I x I y M w(x, y) 2 T T I I x, y I x I y I y u u x x uT Mu Harris corner detector Harris corner detector Only minimum of E is taken into account High-level idea: what shape of the error function A new corner measurement by investigating the will we prefer for features? shape of the error function 100 100 100 80 80 80 T 60 60 60 u Mu represents a quadratic function; Thus, we 40 40 40 can analyze E’s shape by looking at the property 20 20 20 of M 0 0 0 10 10 10 12 12 12 10 10 10 8 5 8 8 6 5 6 5 6 4 4 4 2 2 2 0 0 0 0 0 0 flat edge corner Quadratic forms Symmetric matrices • Quadratic form (homogeneous polynomial of • Quadratic forms can be represented by a real degree two) of n variables xi symmetric matrix A where • Examples = Eigenvalues of symmetric matrices Eigenvectors of symmetric matrices Brad Osgood Eigenvectors of symmetric matrices Harris corner detector Intensity change in shifting window: eigenvalue analysis u , – eigenvalues of M E(u,v) u,v M 1 2 T v x Ax T T T x Q Λ Q x z z 1 Ellipse E(u,v) = const direction of the 2 q2 fastest change T 1 q1 direction of the QTx Λ QTx slowest change T -1/2 y Λy (max) T ( )-1/2 1 T 1 x x 1 min Λ 2 y Λ 2 y zTz Visualize quadratic functions Visualize quadratic functions T T 1 0 1 01 01 0 4 0 1 04 01 0 A A 0 1 0 10 10 1 0 1 0 10 10 1 Visualize quadratic functions Visualize quadratic functions T T 3.25 1.30 0.50 0.871 0 0.50 0.87 7.75 3.90 0.50 0.871 0 0.50 0.87 A A 1.30 1.75 0.87 0.500 40.87 0.50 3.90 3.25 0.87 0.500 100.87 0.50 Harris corner detector Harris corner detector 2 Only for reference, Classification of a00 a11 (a00 a11) 4a10a01 2 edge you do not need image points 2 them to compute R using eigenvalues 2 >> 1 Corner Measure of corner response: of M: 1 and 2 are large, 1 ~ 2; E increases in all 2 directions R detM ktraceM det M 12 and are small; 1 2 edge E is almost constant flat trace M 1 2 in all directions 1 >> 2 1 (k – empirical constant, k = 0.04-0.06) Harris corner detector Another view Another view Another view Summary of Harris detector Summary of Harris detector 1. Compute x and y derivatives of image 4. Define the matrix at each pixel x y S 2 (x, y) S (x, y) I G I I G I x xy x y M (x, y) S (x, y) S 2 (x, y) xy y 2. Compute products of derivatives at every pixel 5. Compute the response of the detector at each I 2 I I I 2 I I I I I x x x y y y xy x y pixel R det M ktraceM 2 3. Compute the sums of the products of derivatives at each pixel 6. Threshold on value of R; compute nonmax suppression. S 2 G I 2 Sx2 G ' I x2 y ' y Sxy G ' I xy Harris corner detector (input) Corner response R Threshold on R Local maximum of R Harris corner detector Harris detector: summary • Average intensity change in direction [u,v] can be expressed as a bilinear form: u E(u,v) u,v M v • Describe a point in terms of eigenvalues of M: measure of corner response 2 R 12 k1 2 • A good (corner) point should have a large intensity change in all directions, i.e. R should be large positive Now we know where features are Harris detector: some properties • But, how to match them? • Partial invariance to affine intensity change • What is the descriptor for a feature? The Only derivatives are used => simplest solution is the intensities of its spatial invariance to intensity shift I I + b neighbors. This might not be robust to brightness change or small shift/rotation. Intensity scale: I a I 12 3 45 6 R R threshold 78 9 x (image coordinate) x (image coordinate) ( 1 2 3 4 5 6 7 8 9 ) Harris Detector: Some Properties Harris Detector is rotation invariant • Rotation invariance Repeatability rate: # correspondences # possible correspondences Ellipse rotates but its shape (i.e. eigenvalues) remains the same Corner response R is invariant to image rotation Harris Detector: Some Properties Harris detector: some properties • Quality of Harris detector for different scale • But: not invariant to image scale! changes Repeatability rate: # correspondences # possible correspondences All points will be Corner ! classified as edges Scale invariant detection Scale invariant detection • Consider regions (e.g. circles) of different sizes • The problem: how do we choose corresponding around a point circles independently in each image? • Regions of corresponding sizes will look the • Aperture problem same in both images SIFT • SIFT is an carefully designed procedure with empirically determined parameters for the invariant and distinctive features. SIFT (Scale Invariant Feature Transform) SIFT stages: 1. Detection of scale-space extrema • Scale-space extrema detection • For scale invariance, search for stable features detector • Keypoint localization across all possible scales using a continuous • Orientation assignment function of scale, scale space. descriptor • Keypoint descriptor • SIFT uses DoG filter for scale space because it is efficient and as stable as scale-normalized Laplacian of Gaussian. ( ) local descriptor A 500x500 image gives about 2000 features DoG filtering Scale space Convolution with a variable-scale Gaussian doubles for the next octave Difference-of-Gaussian (DoG) filter K=2(1/s) Convolution with the DoG filter Dividing into octave is for efficiency only. Detection of scale-space extrema Keypoint localization X is selected if it is larger or smaller than all 26 neighbors Decide scale sampling frequency Decide scale sampling frequency • It is impossible to sample the whole space, tradeoff efficiency with completeness. • Decide the best sampling frequency by experimenting on 32 real image subject to synthetic transformations. (rotation, scaling, affine stretch, brightness and contrast change, adding noise…) Decide scale sampling frequency Pre-smoothing for detector, repeatability for descriptor, distinctiveness s=3 is the best, for larger s, too many unstable features =1.6, plus a double expansion Scale invariance 2. Accurate keypoint localization • Reject points with low contrast (flat) and poorly localized along an edge (edge) • Fit a 3D quadratic function for sub-pixel maxima 6 5 1 -10 +1 2.