CS 1674: Intro to Midterm Review

Prof. Adriana Kovashka University of Pittsburgh October 10, 2016 Reminders

• The midterm exam is in class on this coming Wednesday • There will be no make-up exams unless you or a close relative is seriously ill! Review requests I received

• Textures and texture representations, image responses to size and orientation of Gaussian filter banks, comparisons – 4 • Corner detection alg, Harris – 4 • Invariance vs covariance, affine intensity change, and applications to know – 3 • Scale-invariant detection, blob detection, Harris automatic scale selection – 3 • Sift and feature description – 3 • Keypoint matching alg, feature matching – 2 • Examples of how to compute and apply homography, – 2 • Why it makes sense to use the ratio: distance to best match / distance to second best match when matching features across images • Summary of equations students need to know • Pyramids • Convolution practical use • Filters for transforming the image Transformations, Homographies, Epipolar Geometry 2D Linear Transformations

x' a bx       y' c dy

Only linear 2D transformations can be represented with a 2x2 . Linear transformations are combinations of … • Scale, • Rotation, • Shear, and • Mirror

Alyosha Efros 2D Affine Transformations

 x' a b c  x       y'  d e f  y w' 0 0 1 w

Affine transformations are combinations of … • Linear transformations, and • Translations

Maps lines to lines, parallel lines remain parallel

Adapted from Alyosha Efros Projective Transformations

 x' a b c  x  y'  d e f  y      w' g h i w Projective transformations: • Affine transformations, and • Projective warps

Parallel lines do not necessarily remain parallel

Kristen Grauman How to stitch together a panorama (a.k.a. mosaic)? • Basic Procedure – Take a sequence of images from the same position • Rotate the camera about its optical center – Compute the homography (transformation) between second image and first – Transform the second image to overlap with the first – Blend the two together to create a mosaic – (If there are more images, repeat)

Modified from Steve Seitz Computing the homography x, y x1, y1  1 1

x2 , y2 

x2 , y2 

… …

xn , yn  xn , yn 

To compute the homography given pairs of corresponding points in the images, we need to set up an equation where the parameters of H are the unknowns…

Kristen Grauman Computing the homography p’ = Hp wx' a b c  x wy'  d e f  y        w  g h i  1 Can set scale factor i=1. So, there are 8 unknowns. Set up a system of linear equations: Ah = b where vector of unknowns h = [a,b,c,d,e,f,g,h]T Need at least 8 eqs, but the more the better… Solve for h. If overconstrained, solve using least-squares: min Ah  b 2

Kristen Grauman Computing the homography

• Assume we have four matched points: How do we compute homography H?

h1  w' x' h h h    1 2 3  h       2  p’=Hp p' w' y' H  h4 h5 h6     h3       w'  h7 h8 h9  h  4  A h  h5    h  x  y 1 0 0 0 xx' yx' x'  6     h  0 h7  0 0 0  x  y 1 xy' yy' y'   h8    h9  • Apply SVD: UDVT = A  [U, S, V] = svd(A);

• h = Vsmallest (column of V corr. to smallest singular value)

Derek Hoiem Transforming the second image Image 2 Image 1 canvas

Test point: wy wx  x, y  w, w  x, y

To apply a given homography H wx' * * * x • Compute p’ = Hp (regular matrix multiply)     • Convert p’ from homogeneous to image wy'  * * * y coordinates      w  * * * 1 Modified from Kristen Grauman p’ H p Transforming the second image

Image 2 Image 1 canvas

H(x,y) y y’

x f(x,y) x’ g(x’,y’)

Forward warping: Send each pixel f(x,y) to its corresponding location (x’,y’) = H(x,y) in the right image

Modified from Alyosha Efros Depth from disparity We have two images taken from cameras with different intrinsic and extrinsic parameters. • How do we match a point in the first image to a point in the second?

image I(x,y) Disparity map D(x,y) image I´(x´,y´)

So if we could find the corresponding points in two images, we could estimate relative depth…

Kristen Grauman Epipolar geometry: notation

X

x x’

• Baseline – line connecting the two camera centers • Epipoles = intersections of baseline with image planes = projections of the other camera center • Epipolar Plane – plane containing baseline • Epipolar Lines - intersections of epipolar plane with image planes (always come in corresponding pairs) • Note: All epipolar lines intersect at the epipole. Derek Hoiem Epipolar constraint

The epipolar constraint is useful because it reduces the correspondence problem to a 1D search along an epipolar line.

Kristen Grauman, image from Andrew Zisserman Essential matrix

XTRX  0

X[Tx ]RX  0

Let E [T x]R XEX  XT EX  0 E is called the essential matrix, and it relates corresponding image points between both cameras, given the rotation and translation. Before we said: If we observe a point in one image, its position in other image is constrained to lie on line defined by above. • Turns out Ex’ is the epipolar line through x in the first image, corresp. to x’. Note: these points are in camera coordinate systems.

Kristen Grauman Basic stereo matching algorithm

• For each pixel in the first image – Find corresponding epipolar scanline in the right image – Search along epipolar line and pick the best match x’ – Compute disparity x-x’ and set depth(x) = f*T/(x-x’)

Derek Hoiem Correspondence search

Left Right

scanline

Matching cost disparity

• Slide a window along the right scanline and compare contents of that window with the reference window in the left image • Matching cost: e.g. Euclidean distance

Derek Hoiem Geometry for a simple stereo system

• Assume parallel optical axes, known camera parameters (i.e., calibrated cameras). What is expression for Z?

Similar triangles (pl, P, pr) and (Ol, P, Or): T  x  x T l r  Z  f Z

depth T Z  f xr  xl disparity

Kristen Grauman Results with window search Data

Left image Right image

Window-based matching Ground truth

Window-based matching Ground truth

Derek Hoiem How can we improve? • Uniqueness – For any point in one image, there should be at most one matching point in the other image • Ordering – Corresponding points should be in the same order in both views • Smoothness – We expect disparity values to change slowly (for the most part)

Derek Hoiem Many of these constraints can be encoded in an energy function and solved using graph cuts

Before

Graph cuts Ground truth Y. Boykov, O. Veksler, and R. Zabih, Fast Approximate Energy Minimization via Graph Cuts, PAMI 2001 For the latest and greatest: http://vision.middlebury.edu/stereo/ Derek Hoiem Projective structure from motion • Given: m images of n fixed 3D points

xij = Pi Xj , i = 1,… , m, j = 1, … , n

• Problem: estimate m projection matrices Pi and n 3D points Xj from the mn corresponding 2D points xij

Xj

x1j

x3j

x2j P1

P3

P2 Svetlana Lazebnik Photo synth

Noah Snavely, Steven M. Seitz, Richard Szeliski, "Photo tourism: Exploring photo collections in 3D," SIGGRAPH 2006

http://photosynth.net/ 3D from multiple images

Building Rome in a Day: Agarwal et al. 2009 Recap: Epipoles • Point x in left image corresponds to epipolar line l’ in right image • Epipolar line passes through the epipole (the intersection of the cameras’ baseline with the image plane

C

C

Derek Hoiem Recap: Essential, Fundamental Matrices • Fundamental matrix maps from a point in one image to a line in the other

• If x and x’ correspond to the same 3d point X:

• Essential matrix is like fundamental matrix but more constrained

Adapted from Derek Hoiem Recap: stereo with calibrated cameras • Given image pair, R, T • Detect some features • Compute essential matrix E • Match features using the epipolar and other constraints • Triangulate for 3d structure and get depth

Kristen Grauman Texture representations Correlation filtering

Say the averaging window size is 2k+1 x 2k+1:

Attribute uniform weight Loop over all pixels in neighborhood around to each pixel image pixel F[i,j]

Now generalize to allow different weights depending on neighboring pixel’s relative position:

Non-uniform weights

Kristen Grauman Convolution vs. correlation

Cross-correlation F 5 2 5 4 4

5 200 3 200 4

1 5 5 4 4 (i, j) u = -1, v = -1 5 5 1 1 2

200 1 3 5 200

1 200 200 200 1

Convolution H

.06 .12 .06

.12 .25 .12 (0, 0) .06 .12 .06 Filters for computing gradients

1 0 -1 2 0 -2 1 0 -1 * =

Slide credit: Derek Hoiem Texture representation: example

mean mean d/dx d/dy value value Win. #1 4 10

Win.#2 18 7 …

Win.#9 20 20

original image …

statistics to summarize derivative filter patterns in small Kristen Grauman responses, squared windows Filter banks orientations scales “Edges” “Bars”

“Spots”

• What filters to put in the bank? – Typically we want a combination of scales and orientations, different types of patterns.

Matlab code available for these examples: http://www.robots.ox.ac.uk/~vgg/research/texclass/filters.html

Kristen Grauman Matching with filters • Goal: find in image • Method 0: filter the image with eye patch g[m,n]   h[k,l] f [m  k,n  l] k,l f = image g = filter

What went wrong?

Input Filtered Image Derek Hoiem Likes bright pixels where filters are Matching with filters above average, dark pixels where filters are below average. • Goal: find in image • Method 1: filter the image with zero-mean eye g[m,n]  (h[k,l]mean(h)) ( f [m  k,n  l]) k,l

True detections

False detections

Input Filtered Image (scaled) Thresholded Image Derek Hoiem Showing magnitude of responses

Kristen Grauman Kristen Grauman Kristen Grauman Representing texture by mean abs response Filters

Mean abs responses Derek Hoiem Computing distances using texture

2 2 a D(a,b)  (a1  b1)  (a2  b2) #dim 2 D(a,b)  (ai  bi )

b i1 Dimension 2 Dimension

Dimension 1

Kristen Grauman Feature detection: Harris Corners as distinctive interest points • We should easily recognize the keypoint by looking through a small window • Shifting a window in any direction should give a large change in intensity

“flat” region: “edge”: “corner”: no change in no change along significant change all directions the edge direction in all directions

A. Efros, D. Frolova, D. Simakov Harris Detector: Mathematics Window-averaged squared change of intensity induced by shifting the image data by [u,v]:

Window Shifted Intensity function intensity

Window function w(x,y) = or

1 in window, 0 outside Gaussian

D. Frolova, D. Simakov Harris Detector: Mathematics

Expanding I(x,y) in a Taylor series expansion, we have, for small shifts [u,v], a quadratic approximation to the error surface between a patch and itself, shifted by [u,v]:

where M is a 2×2 matrix computed from image derivatives:

D. Frolova, D. Simakov Harris Detector: Mathematics

I x I x I x I y  M  w(x, y)   I x I y I y I y 

I I I I Notation: I  I  I I  x x y y x y x y

K. Grauman What does the matrix M reveal?

1 0  T Since M is symmetric, we have M  X  X  0 2 

Mxi  i xi

The eigenvalues of M reveal the amount of intensity change in the two principal orthogonal gradient directions in the window.

K. Grauman Corner response function

“edge”: “corner”: “flat” region:

1 >> 2 1 and 2 are large, 1 and 2 are small  ~  2 >> 1 1 2

Adapted from A. Efros, D. Frolova, D. Simakov, K. Grauman Harris Detector: Algorithm

• Compute image gradients Ix and Iy for all pixels • For each pixel – Compute

by looping over neighbors x, y – compute (k :empirical constant, k = 0.04-0.06) • Find points with large corner response function R (R > threshold) • Take the points of locally maximum R as the detected feature points (i.e., pixels where R is bigger than for all the 4 or 8 neighbors)

55 D. Frolova, D. Simakov Example of Harris application

K. Grauman Feature detection: Scale-invariance Invariance vs covariance “A function is invariant under a certain family of transformations if its value does not change when a transformation from this family is applied to its argument. A function is covariant when it commutes with the transformation, i.e., applying the transformation to the argument of the function has the same effect as applying the transformation to the output of the function. […] [For example,] the area of a 2D surface is invariant under 2D rotations, since rotating a 2D surface does not make it any smaller or bigger. But the orientation of the major axis of inertia of the surface is covariant under the same family of transformations, since rotating a 2D surface will affect the orientation of its major axis in exactly the same way.”

“Local Invariant Feature Detectors: A Survey” by Tinne Tuytelaars and Krystian Mikolajczyk, in Foundations and Trends in Computer Graphics and Vision Vol. 3, No. 3 (2007) 177–280 Chapter 1, 3.2, 7 http://homes.esat.kuleuven.be/%7Etuytelaa/FT_survey_interestpoints08.pdf What happens if: Affine intensity change

I  a I + b

• Only derivatives are used => invariance to intensity shift I  I + b • Intensity scaling: I  a I

R R threshold

x (image coordinate) x (image coordinate)

Partially invariant to affine intensity change

L. Lazebnik What happens if: Image translation

• Derivatives and window function are shift-invariant

Corner location is covariant w.r.t. translation

L. Lazebnik What happens if: Image rotation

Second moment ellipse rotates but its shape (i.e. eigenvalues) remains the same

Corner location is covariant w.r.t. rotation

L. Lazebnik What happens if: Scaling

Corner

All points will be classified as edges

Corner location is not covariant to scaling!

L. Lazebnik Scale Invariant Detection

• Problem: – How do we choose corresponding circles independently in each image? – Do objects in the image have a characteristic scale that we can identify?

D. Frolova, D. Simakov Scale Invariant Detection

• Solution: – Design a function on the region which is “scale invariant” (has the same shape even if the image is resized) – Take a local maximum of this function

f Image 1 f Image 2 scale = 1/2

s1 region size s2 region size Adapted from A. Torralba Automatic Scale Selection

• Function responses for increasing scale (scale signature)

f (I (x, )) f (I (x, )) i1im i1im K. Grauman, B. Leibe Automatic Scale Selection

• Function responses for increasing scale (scale signature)

f (I (x, )) f (I (x, )) i1im i1im K. Grauman, B. Leibe Automatic Scale Selection

• Function responses for increasing scale (scale signature)

f (I (x, )) f (I (x,)) i1im i1im K. Grauman, B. Leibe What Is A Useful Signature Function? • Laplacian of Gaussian = “blob” detector

K. Grauman, B. Leibe Difference of Gaussian ≈ Laplacian • We can approximate the Laplacian with a difference of Gaussians; more efficient to implement.

2 L Gxx(,,)(,,) x y  G yy x y   (Laplacian)

DoG G(,,)(,,) x y k G x y (Difference of Gaussians) Difference of Gaussian: Efficient computation • Computation in Gaussian scale pyramid

Sampling with step 4 =2

1    2 4 Original image 

K. Grauman, B. Leibe Find local maxima in position-scale space of Difference-of-Gaussian

5 Position-scale space:

4

3

2 Find places where X greater than all of its neighbors (in green)

  List of

Adapted from K. Grauman, B. Leibe (x, y, s) Laplacian pyramid example • Allows detection of increasingly coarse detail Results: Difference-of-Gaussian

K. Grauman, B. Leibe Feature description Gradients

m(x, y) = sqrt(1 + 0) = 1 Θ(x, y) = atan(0/1) = 0 Scale Invariant Feature Transform Full version • Divide the 16x16 window into a 4x4 grid of cells (2x2 case shown below) • Quantize the gradient orientations i.e. snap each gradient to one of 8 angles • Each gradient contributes not just 1, but magnitude(gradient) to the histogram, i.e. stronger gradients contribute more • 16 cells * 8 orientations = 128 dimensional descriptor for each detected feature

Adapted from L. Zitnick, D. Lowe Scale Invariant Feature Transform Full version • Divide the 16x16 window into a 4x4 grid of cells (2x2 case shown below) • Quantize the gradient orientations i.e. snap each gradient to one of 8 angles • Each gradient contributes not just 1, but magnitude(gradient) to the histogram, i.e. stronger gradients contribute more • 16 cells * 8 orientations = 128 dimensional descriptor for each detected feature • Normalize + clip (threshold normalize to 0.2) + normalize the descriptor • After normalizing, we have:

such that:

0.2

Adapted from L. Zitnick, D. Lowe Making descriptor rotation invariant

CSE 576: Computer Vision

• Rotate patch according to its dominant gradient orientation • This puts the patches into a canonical orientation

K. Grauman Image from Matthew Brown Keypoint matching Matching local features

?

Image 1 Image 2 • To generate candidate matches, find patches that have the most similar appearance (e.g., lowest feature Euclidean distance) • Simplest approach: compare them all, take the closest (or closest k, or within a thresholded distance)

K. Grauman Robust matching

? ? ? ?

Image 1 Image 2 • At what Euclidean distance value do we have a good match? • To add robustness to matching, can consider ratio : distance to best match / distance to second best match • If low, first match looks good. • If high, could be ambiguous match. K. Grauman Ratio: example

• Let q be the query from the first image, d1 be the closest match in the second image, and d2 be the second closest match • Let dist(q, d1) and dist(q, d2) be the distances • Let r = dist(q, d1) / dist(q, d2) • What is the largest that r can be? • What is the lowest that r can be? • If r is 1, what do we know about the two distances? • What about when r is 0.1? Indexing local features: Setup

• When we see close points in feature space, we have similar descriptors, which indicates similar local content.

Query Descriptor’s image feature space Database

K. Grauman images Image matching Describing images w/ visual words

• Summarize entire image

based on its distribution times appearing times (histogram) of word occurrences. • Analogous to bag of words

representation commonly appearing times used for documents.

Feature patches: times appearing appearing times

Visual words K. Grauman Bag of visual words: Two uses

1. Represent the image 2. Using that representation, look for similar images 3. Can also use BOW to compute an inverted index, to simplify application #2 Visual words: main idea

• Extract some local features from a number of images …

e.g., SIFT descriptor space: each point is 128-dimensional

D. Nister, CVPR 2006 Visual words: main idea

D. Nister, CVPR 2006 “Quantize” the space by grouping (clustering) the features. Note: For now, we’ll treat clustering as a black box. D. Nister, CVPR 2006 Inverted file index and bags of words similarity

w91

1. (offline) Extract features in database images, cluster them to find words, make index 2. Extract words in query (extract features and map each to closest cluster center) 3. Use inverted file index to find frames relevant to query 4. For each relevant frame, rank them by comparing word counts (BOW) of query and frame Adapted from K. Grauman Scoring retrieval quality

Database size: 10 images Results (ordered): Relevant (total): 5 images Query (e.g. images of Golden Gate)

precision = # returned relevant / # returned recall = # returned relevant / # total relevant

1

0.8

0.6

precision 0.4

0.2

0 0 0.2 0.4 0.6 0.8 1 recall

Ondrej Chum