Edges and Features - 2 CAP 5415, FALL 2019 Credits and Sources

▪This lecture contains open source materials authored by ▪ Dr. Mubarak Shah ▪ Dr. Ajit Rajwade ▪ Dr. David Lowe (inventor of SIFT) ▪ Dr. George Bebis ▪ Dr. Cornelia Fermüller Aspects of Classical

Edges & Features

Tracking Classification

Shape From 3D

Edges and Features are precursors to other functions such as tracking, recovering 3D shape from 2D images, and object classification

Uses of Features -

•Automate object tracking •Point matching for computing disparity •Stereo calibration •Estimation of fundamental matrix •Motion based segmentation •Recognition •3D object reconstruction •Robot navigation •Image retrieval and indexing

푤 푥, 푦 window

SIFT: Motivation

The Harris operator is not invariant to scale and correlation is not invariant to rotation1.

For better image matching, Lowe’s goal was to develop an interest operator that is invariant to scale and rotation.

Also, Lowe aimed to create a descriptor that was robust to the variations corresponding to typical viewing conditions. The descriptor is the most-used part of SIFT.

1But Schmid and Mohr developed a rotation invariant descriptor for it in 1997.

4/28/2020 30 Idea of SIFT

Image content is transformed into local feature coordinates that are invariant to translation, rotation, scale, and other imaging parameters

SIFT Features

4/28/2020 31

Overall Procedure at a High Level 1. Scale-space extrema detection Search over multiple scales and image locations. 2. Keypoint localization Fit a model to detrmine location and scale. Select keypoints based on a measure of stability. 3. Orientation assignment Compute best orientation(s) for each keypoint region.

4. Keypoint description Use local image gradients at selected scale and rotation to describe each keypoint region.

4/28/2020 35 A blob detector for different sizes

Lowe’s Scale-space Interest Points

Laplacian of Gaussian kernel ◦ Scale normalised (x by scale2) ◦ Proposed by Lindeberg Scale-space detection ◦ Find local maxima across scale/space ◦ A good “blob” detector

[ T. Lindeberg IJCV 1998 ] 4/28/2020 40 Lowe’s Scale-space Interest Points:

Gaussian is an ad hoc solution of heat diffusion equation

Hence

k is not necessarily very small in practice

4/28/2020 41 Lowe’s Pyramid Scheme

• is separated into octaves: • Octave 1 uses scale  • Octave 2 uses scale 2 • etc.

• In each octave, the initial image is repeatedly convolved with Gaussians to produce a set of scale space images.

• Adjacent Gaussians are subtracted to produce the DOG

• After each octave, the Gaussian image is down-sampled by a factor of 2 to produce an image ¼ the size to start the next level.

4/28/2020 42

Keypoint localization with orientation

832 233x189 initial keypoints

536 729 keypoints after keypoints after gradient threshold ratio threshold

4/28/2020 54 4. Scale Invariant Feature Transform (SIFT)

1. Take a 16 x16 window around interest point (i.e., at the scale detected). 2. Divide into a 4x4 grid of cells. 3. Compute histogram of image gradient directions in each cell (8 bins each). 16 histograms x 8 orientations = 128 features

5. Feature matching

For each feature in A, find nearest neighbor in B

A B Example: Planar recognition

Training images Example: Planar recognition

• Reliably recognized at a rotation of 60° away from the camera • Affine fit is an approximation of perspective projection • Only 3 points are needed for recognition Training images 3D object recognition

Only 3 keys are needed for recognition, so extra keys provide robustness Affine model is no longer as accurate Recognition under occlusion Illumination invariance Robot Localization Credits and Sources

▪This lecture contains open source materials authored by ▪ Dr. Mubarak Shah ▪ Dr. Ajit Rajwade ▪ Dr. David Lowe (inventor of SIFT) ▪ Dr. George Bebis ▪ Dr. Cornelia Fermüller