Fundamental

Lecture‐13 Transformations Between Two Images • Translation • Rotation • Rigid • Similarity (scaled rotation) • Affine • Projective • Pseudo Perspective • Bi‐linear Applications

• Stereo • Structure from Motion • View Invariant Action Recognition • .. Stereo Pairs and Depth Maps (from Szeliski’s book) For Stereo Photosynth Fundamental Matrix

• Longuet Higgins (1981) • Hartley (1992) • Faugeras (1992) • Zhang (1995) Fundamental Matrix Song

http://www.youtube.com/watch?v=DgGV3l82NTk Preliminaries

• Linear Independence • Rank of a Matrix • Matrix Norm • Singular Value Decomposition • Vector to Matrix Multiplication • RANSAC Linearly Independence

A finite subset of n vectors, v1, v2, ..., vn, from the vector space V, is linearly dependent if and only if there exists a set of n scalars, a1, a2, ..., an, not all zero, such that Rank of a Matrix

• The column rank of a matrix A is the maximum number of linearly independent column vectors of A. • The row rank of a matrix A is the maximum number of linearly independent row vectors of A. • The column rank of A is the dimension of the column space of A • The row rank of A is the dimension of the row space of A.

Copyright Mubarak Shah 2003 Example (Row Echelon)

Rank is 2 Singular Value Decomposition (SVD)

Theorem: Any m by n matrix A, for which ,can be written as is diagonal

are orthogonal mxn mxn nxn nxn

Copyright Mubarak Shah 2003 Matrix Norm

L1 matrix norm is maximum of absolute column sum.

L infinity norm is maximum of sum of absolute of row sum. Vector Cross Product to Matrix‐ vector multiplication

 i j k    A B  Ax Ay Az    Az By  Ay Bz Az Bx  Ax Bz  Bx Ay  By Ax   Bx By Bz 

 0  Az Ay Bx     A B  S.B   Az 0  Ax By    Az By  Ay Bz Az Bx  Ax Bz  Bx Ay  By Ax  A A 0    y x Bz  How to Fit A Line?

• Least squares Fit (over constraint) • RANSAC (constraint) • Hough Transform (under constraint)

Alper Yilmaz, Mubarak Shah, Fall 2011 UCF RANSAC Song

http://www.youtube.com/watch?v=1YNjMxxXO-E&feature=relmfu How to Fit A Line?

y  mx  c

Alper Yilmaz, Mubarak Shah, Fall 2011 UCF Least Squares Fit

• Standard linear solution to estimating unknowns. – If we know which points belong to which line – Or if there is only one line

y  mx  c  f x,m,c

2 Minimize E  yi  f xi ,m,c i Take derivative wrt m and c set to 0

Alper Yilmaz, Mubarak Shah, Fall 2011 UCF Line Fitting

 y   x 1 y  mx  c 1 1  y  x 1 m  2   2       B  AD      1 c     D  yn  xn 1  B A y1  mx1  c y2  mx2  c

 AT B  AT AD y  mx  c n n 1 1 AT A AT B  AT A A T A D 1 D  AT A AT B

Alper Yilmaz, Mubarak Shah, Fall 2011 UCF RANSAC: Random Sampling and Consensus

Alper Yilmaz, Mubarak Shah, Fall 2011 UCF RANSAC Song

http://www.youtube.com/watch?v=1YNjMxxXO- E&feature=relmfu RANSAC: Random Sampling and Consensus

1. Randomly select two points to fit a line

2. Find the error between the estimated solution an all other points. If the error is less than tolerance, then quit, else go to step (1).

Alper Yilmaz, Mubarak Shah, Fall 2011 UCF Derivation of Fundamental Matrix

 Defined for two static cameras

3D World Left Right camera camera

Left camera Right camera image plane image plane Epipolar Geometry

P Epipolar line: set of world xl xr points that project to the same point in left image, Cl C e r when projected to right el r image forms a line. T

Epipolar plane: plane defined by the camera centers and world point. Epipole: intersection of image plane with line connecting camera centers. Image of a left camera center in the right, and vice versa. Essential Matrix

P P Pl r xl xr Cl C e r el r T

Coplanarity constraint between vectors (Pl-T), T, Pl. P  T  T  P  0 l l  Pr RT  Pl  0 Pr  RPl  T

T R Pr  Pl  T T T Pr R  Pl  T Vector Cross Product to Matrix‐ vector multiplication

 i j k    A B  Ax Ay Az    Az By  Ay Bz Az Bx  Ax Bz  Bx Ay  By Ax   Bx By Bz 

 0  Az Ay Bx     A B  S.B   Az 0  Ax By    Az By  Ay Bz Az Bx  Ax Bz  Bx Ay  By Ax  A A 0    y x Bz  Essential Matrix

P P Pl r xl xr Cl C e r el r T

   Pr R S Pl  0 Pr ΕPl  0 Pr RT  Pl  0 essential matrix

 0 T T  E  R S  z y   Tz 0 Tx    Ty Tx 0  Fundamental Matrix

P P Pl r xl xr Cl C e r el r T

  1 xr M r EM l xl  0 xl  M l Pl   1 Apply Camera model xr M r EM l xl  0 xr  M r Pr 1  M l xl  Pl xr Fxl  0 P ΕP  0 1 r l fundamental matrix M r xr  Pr T T T xr M r  Pr Fundamental Matrix

 a b c      x' Fx  x' d e f x  0    g h m

• Given a point x in left camera, epipolar line in

right camera is: ur=Fx Fundamental Matrix

 a b c      xl Fxr  xl d e f xr  0    g h m • 3x3 matrix with 9 components • Rank 2 matrix (due to S) • 7 degrees of freedom • Given a point in left camera x, epipolar line in right camera is: ur=Fx Fundamental Matrix

• Longuet Higgins (1981) • Hartley (1992) • Faugeras (1992) • Zhang (1995) Fundamental Matrix

• Fundamental matrix captures the relationship between the corresponding points in two views. Fundamental Matrix

One equation for one point correspondence

To solve the equation, the rank(M) must be 8. Computation of Fundamental Matrix Normalized 8-point algorithm (Hartley) Objective: Compute fundamental matrix F such that

Algorithm Normalize the image

Find centroid of points in each image, determine the range, and normalize all points between 0 and 1 Linear solution determining the eigen vector corresponding to the smallest eigen value of A, Normalized 8-point algorithm (Hartley) Construct

L1 matrix norm is maximum of absolute column sum. Normalize

Constraint enforcement SVD decomposition L infinity norm is maximum of sum of absolute of row sum.

Rank enforcement

De-normalization: Robust Fundamental Matrix Estimation (by Zhang)

• Uniformly divide the image into 8×8 grid. • Randomly select 8 grid cells and pick one pair of corresponding points from each grid cell, then use Hartley’s 8-point algorithm to

compute Fundamental Matrix Fi.

• For each Fi, compute the median of the squared residuals Ri.

– Ri = mediank[d(p1k,Fip2k) + d(p2k,F’ip1k)]

• Select the best Fi according to Ri.

• Determine outliers if Rk>Th. • Using the remaining points compute the fundamental Matrix F by weighted least square method. Epi-polar Lines Epi-polar lines

Stereo Stereo Pairs and Depth Maps (from Szeliski’s book) Image Rectification For Stereo Correlation Based Stereo Methods • Once disparity is available compute depth using fB Z  d

Alper Yilmaz, Mubarak Shah, Fall 20011 UCF Correspondence using Search Correlation Based Stereo Methods • Depth is computed only at tokens and interpolated/extrapolated to remaining pixel • Disparity map is constructed based on a correlation measure

2 SSD  I  I Ileft .Iright   left right NC   Ileft .Iright AD  | I  I |   left right 1 MC  Ileft  right Ileft  right  64 left right CC  It1It Alper Yilmaz, Mubarak shah, Fall 2011 UCF Barnard’s Stereo Method

 Similar intensity – Similar to brightness constraint  Smoothness of disparity

1 1 E   Ileft (x  i, y  j)  I right (x  i  Dx (x, y), y  j)   D(x, y) ij1  1

1 1 D(x, y)  D(x  i, y  j)  D(x, y) ij1  1

Alper Yilmaz, Mubarak Shah, Fall 2011 UCF Barnard’s Stereo Method

 Energy can be minimized using brute force search – Let max allowed disparity is 10 pixels – For 128x128 image for 10 possible levels of disparity  There 1016384 possible disparity values – We can select any minimization technique – Barnard choose simulated annealing

Alper Yilmaz, Mubarak Shah, Fall 2011 UCF Simulated Annealing

 Select a random state S (disparities)  Select a high temperature – Select random S’ – Compute E=E(S’)-E(S) – If (E<0) SS’ – Else  Pexp(-E/T)  Xrandom(0,1) – If X

Alper Yilmaz, Mubarak Shah, Fall 2011 UCF Examples

bread toy apple

Alper Yilmaz, Mubarak Shah, Fall 2011 UCF Examples

Left Image Right Image Depth Map

Alper Yilmaz, Fall 2004 UCF Stereo results

• Data from University of Tsukuba • Similar results on other images without ground truth

Scene Ground truth Results with window correlation

Window-based matching Ground truth (best window size) Results with better method

State of the art method Ground truth Boykov et al., Fast Approximate Energy Minimization via Graph Cuts, International Conference on , September 1999. Applications of Stereo (from Szeliski’s book) Reading Material

• Fundamental of Computer Vision – 6.2.1, 6.2.4 and 6.2.5 • Computer Vision: Algorithms and Applications, Richard Szeliski – Chapter 11