Fundamental Matrix Lecture‐13 Transformations Between Two Images • Translation • Rotation • Rigid • Similarity (scaled rotation) • Affine • Projective • Pseudo Perspective • Bi‐linear Applications • Stereo • Structure from Motion • View Invariant Action Recognition • .. Stereo Pairs and Depth Maps (from Szeliski’s book) Image Rectification For Stereo Photosynth Fundamental Matrix • Longuet Higgins (1981) • Hartley (1992) • Faugeras (1992) • Zhang (1995) Fundamental Matrix Song http://www.youtube.com/watch?v=DgGV3l82NTk Preliminaries • Linear Independence • Rank of a Matrix • Matrix Norm • Singular Value Decomposition • Vector Cross product to Matrix Multiplication • RANSAC Linearly Independence A finite subset of n vectors, v1, v2, ..., vn, from the vector space V, is linearly dependent if and only if there exists a set of n scalars, a1, a2, ..., an, not all zero, such that Rank of a Matrix • The column rank of a matrix A is the maximum number of linearly independent column vectors of A. • The row rank of a matrix A is the maximum number of linearly independent row vectors of A. • The column rank of A is the dimension of the column space of A • The row rank of A is the dimension of the row space of A. Copyright Mubarak Shah 2003 Example (Row Echelon) Rank is 2 Singular Value Decomposition (SVD) Theorem: Any m by n matrix A, for which ,can be written as is diagonal are orthogonal mxn mxn nxn nxn Copyright Mubarak Shah 2003 Matrix Norm L1 matrix norm is maximum of absolute column sum. L infinity norm is maximum of sum of absolute of row sum. Vector Cross Product to Matrix‐ vector multiplication i j k A B Ax Ay Az Az By Ay Bz Az Bx Ax Bz Bx Ay By Ax Bx By Bz 0 Az Ay Bx A B S.B Az 0 Ax By Az By Ay Bz Az Bx Ax Bz Bx Ay By Ax A A 0 y x Bz How to Fit A Line? • Least squares Fit (over constraint) • RANSAC (constraint) • Hough Transform (under constraint) Alper Yilmaz, Mubarak Shah, Fall 2011 UCF RANSAC Song http://www.youtube.com/watch?v=1YNjMxxXO-E&feature=relmfu How to Fit A Line? y mx c Alper Yilmaz, Mubarak Shah, Fall 2011 UCF Least Squares Fit • Standard linear solution to estimating unknowns. – If we know which points belong to which line – Or if there is only one line y mx c f x,m,c 2 Minimize E yi f xi ,m,c i Take derivative wrt m and c set to 0 Alper Yilmaz, Mubarak Shah, Fall 2011 UCF Line Fitting y x 1 y mx c 1 1 y x 1 m 2 2 B AD 1c D yn xn 1 B A y1 mx1 c y2 mx2 c AT B AT AD y mx c n n 1 1 AT A AT B AT A AT A D 1 D AT A AT B Alper Yilmaz, Mubarak Shah, Fall 2011 UCF RANSAC: Random Sampling and Consensus Alper Yilmaz, Mubarak Shah, Fall 2011 UCF RANSAC Song http://www.youtube.com/watch?v=1YNjMxxXO- E&feature=relmfu RANSAC: Random Sampling and Consensus 1. Randomly select two points to fit a line 2. Find the error between the estimated solution an all other points. If the error is less than tolerance, then quit, else go to step (1). Alper Yilmaz, Mubarak Shah, Fall 2011 UCF Derivation of Fundamental Matrix Epipolar Geometry Defined for two static cameras 3D World Left Right camera camera Left camera Right camera image plane image plane Epipolar Geometry P Epipolar line: set of world xl xr points that project to the same point in left image, Cl C e r when projected to right el r image forms a line. T Epipolar plane: plane defined by the camera centers and world point. Epipole: intersection of image plane with line connecting camera centers. Image of a left camera center in the right, and vice versa. Essential Matrix P P Pl r xl xr Cl C e r el r T Coplanarity constraint between vectors (Pl-T), T, Pl. P T T P 0 l l Pr RT Pl 0 Pr RPl T T R Pr Pl T T T Pr R Pl T Vector Cross Product to Matrix‐ vector multiplication i j k A B Ax Ay Az Az By Ay Bz Az Bx Ax Bz Bx Ay By Ax Bx By Bz 0 Az Ay Bx A B S.B Az 0 Ax By Az By Ay Bz Az Bx Ax Bz Bx Ay By Ax A A 0 y x Bz Essential Matrix P P Pl r xl xr Cl C e r el r T Pr R S Pl 0 Pr ΕPl 0 Pr RT Pl 0 essential matrix 0 T T E R S z y Tz 0 Tx Ty Tx 0 Fundamental Matrix P P Pl r xl xr Cl C e r el r T 1 xr M r EM l xl 0 xl M l Pl 1 Apply Camera model xr M r EM l xl 0 xr M r Pr 1 M l xl Pl xr Fxl 0 P ΕP 0 1 r l fundamental matrix M r xr Pr T T T xr M r Pr Fundamental Matrix a b c x' Fx x' d e f x 0 g h m • Given a point x in left camera, epipolar line in right camera is: ur=Fx Fundamental Matrix a b c xl Fxr xl d e f xr 0 g h m • 3x3 matrix with 9 components • Rank 2 matrix (due to S) • 7 degrees of freedom • Given a point in left camera x, epipolar line in right camera is: ur=Fx Fundamental Matrix • Longuet Higgins (1981) • Hartley (1992) • Faugeras (1992) • Zhang (1995) Fundamental Matrix • Fundamental matrix captures the relationship between the corresponding points in two views. Fundamental Matrix One equation for one point correspondence To solve the equation, the rank(M) must be 8. Computation of Fundamental Matrix Normalized 8-point algorithm (Hartley) Objective: Compute fundamental matrix F such that Algorithm Normalize the image Find centroid of points in each image, determine the range, and normalize all points between 0 and 1 Linear solution determining the eigen vector corresponding to the smallest eigen value of A, Normalized 8-point algorithm (Hartley) Construct L1 matrix norm is maximum of absolute column sum. Normalize Constraint enforcement SVD decomposition L infinity norm is maximum of sum of absolute of row sum. Rank enforcement De-normalization: Robust Fundamental Matrix Estimation (by Zhang) • Uniformly divide the image into 8×8 grid. • Randomly select 8 grid cells and pick one pair of corresponding points from each grid cell, then use Hartley’s 8-point algorithm to compute Fundamental Matrix Fi. • For each Fi, compute the median of the squared residuals Ri. – Ri = mediank[d(p1k,Fip2k) + d(p2k,F’ip1k)] • Select the best Fi according to Ri. • Determine outliers if Rk>Th. • Using the remaining points compute the fundamental Matrix F by weighted least square method. Epi-polar Lines Epi-polar lines Stereo Stereo Pairs and Depth Maps (from Szeliski’s book) Image Rectification For Stereo Correlation Based Stereo Methods • Once disparity is available compute depth using fB Z d Alper Yilmaz, Mubarak Shah, Fall 20011 UCF Correspondence using Search Correlation Based Stereo Methods • Depth is computed only at tokens and interpolated/extrapolated to remaining pixel • Disparity map is constructed based on a correlation measure 2 Ileft .Iright SSD Ileft I right NC Ileft .Iright AD | I I | left right 1 MC Ileft right Ileft right 64 left right CC It1It Alper Yilmaz, Mubarak shah, Fall 2011 UCF Barnard’s Stereo Method Similar intensity – Similar to brightness constraint Smoothness of disparity 1 1 E Ileft (x i, y j) I right (x i Dx (x, y), y j) D(x, y) ij1 1 1 1 D(x, y) D(x i, y j) D(x, y) ij1 1 Alper Yilmaz, Mubarak Shah, Fall 2011 UCF Barnard’s Stereo Method Energy can be minimized using brute force search – Let max allowed disparity is 10 pixels – For 128x128 image for 10 possible levels of disparity There 1016384 possible disparity values – We can select any minimization technique – Barnard choose simulated annealing Alper Yilmaz, Mubarak Shah, Fall 2011 UCF Simulated Annealing Select a random state S (disparities) Select a high temperature – Select random S’ – Compute E=E(S’)-E(S) – If (E<0) SS’ – Else Pexp(-E/T) Xrandom(0,1) – If X<P then SS’ If no decrease in several iterations lower T Alper Yilmaz, Mubarak Shah, Fall 2011 UCF Examples bread toy apple Alper Yilmaz, Mubarak Shah, Fall 2011 UCF Examples Left Image Right Image Depth Map Alper Yilmaz, Fall 2004 UCF Stereo results • Data from University of Tsukuba • Similar results on other images without ground truth Scene Ground truth Results with window correlation Window-based matching Ground truth (best window size) Results with better method State of the art method Ground truth Boykov et al., Fast Approximate Energy Minimization via Graph Cuts, International Conference on Computer Vision, September 1999. Applications of Stereo (from Szeliski’s book) Reading Material • Fundamental of Computer Vision – 6.2.1, 6.2.4 and 6.2.5 • Computer Vision: Algorithms and Applications, Richard Szeliski – Chapter 11.