Fundamental Matrix / Image Rectification COMPSCI 773 S1 T VISION GUIDED CONTROL A/P Georgy Gimel’farb Epipolar Geometry • Ol,Or - projection centres – Origins of the reference frames – fl, fr - focal lengths of cameras • πl, πr - image planes – 3D reference frame for each camera: Z-axis = the optical axis T T Pl=[Xl,Yl,Zl] , Pr=[Xr,Yr,Zr] - the same 3D point P in the reference frames T T pl=[xl ,yl, zl =fl] , pr=[xr,yr, zr =fr] - projections of P onto the image planes Epipoles COMPSCI 773 1 Basics of Epipolar Geometry • Reference frames of the left and right cameras - related via the extrinsic parameters – Translation vector T = (Or - Ol) and a rotation matrix R defining a rigid transformation in 3-D space, given a 3-D point P, between Pl and Pr: Pr = R(Pl - T) • Epipoles el and er - the points at which the line through the centres of projection intersects the image planes – Left epipole - the image of the right projection centre – Right epipole - the image of the left projection centre – Canonical geometry: the epipole is at infinity of the baseline COMPSCI 773 2 Basics of Epipolar Geometry T • 3-D point P = [X,Y,Z] ⇔ its projections pl and pr: flPl f rPr pl = ; pr = Zl Zr • Epipolar plane: the plane through P, Ol, and Or – Epipolar line: its intersection with each image plane – Conjugated€ lines: both the lines for an epipolar plane – Given pl, the 3-D point P can lie anywhere on the ray pl Ol depicted by the epipolar line through the corresponding pr – Epipolar constraint: the true match lies on the epipolar line COMPSCI 773 3 Basics of Epipolar Geometry • All epipolar lines go through the epipole – With the exception of the epipole, only one epipolar line goes through any image point – Mapping between points on one image and corresponding epipolar lines on the other image ⇒ the 1-D search region – Rejection of false matches due to occlusions – Corresponding points must lie on conjugated epipolar lines • The obvious question: how to estimate the epipolar geometry, i.e. determine the ‘point-to-line’ mapping for images COMPSCI 773 4 Vector product a×b = n |a||b|sinθ where the unit vector n is perpendicular to and θ is the angle between the vectors a and b The Essential Matrix, E • Determining the mapping between points in one image and epipolar lines in the other image: – The equation of the epipolar plane through a 3-D point P as the co-planarity of the vectors Pl, T, and Pl-T: T (Pl − T) (T× Pl ) = 0 Pl P T T T Ol ⇒ (R Pr ) (T× Pl ) = 0 ⇒ Pr R(T× Pl ) = 0 Full rank matrix 0 −Tz Ty T T ⇒ T× Pl ≡ Tz 0 −Tx Pl ⇒ Pr (RS)Pl P -T RTP ) { l =( r Matrix E − Ty Tx 0 of rank 2 14 4 2 4 4 3 O Matrix S of rank 2 r Essential matrix COMPSCI 773 5 € The Essential Matrix, E – By construction, the matrix S (and thus E) are of rank 2 – The essential matrix gives a natural link between the epipolar constraint and the extrinsic parameters of the stereo system: Zlpl Zrpr Zl Zr T T Pl = ; Pr = ⇒ pr Epl = 0 ⇒ pr Epl = 0 f l f r f l fr Matrix E: the mapping between the points and epipolar lines T – Vector ar= Epl → parameters of the epipolar line pr ar= 0 in the right image corresponding to the point p in the left image € l T T T – Vector al = pr E → parameters of the epipolar line al pl = 0 in the left image corresponding to the point pr in the right image COMPSCI 773 6 The Fundamental Matrix, F • The mapping “points ↔ epipolar lines” can be obtained from corresponding points only – No prior information on the stereo system! • Points pl, pr in pixel and pl, pr in camera coordinates: x l x r 1 1 p y M p ; p y M p p M − p ; p M − p l ≡ l = l l r ≡ r = r r ⇔ l = l l r = r r 1 1 T −T −1 T M and M - matrices of the ⇒ p M EM p ⇒ p Fp l r r 14r 2 4 3l l r l intrinsic camera parameters Fundamental matrix F COMPSCI 773 7 € The Fundamental Matrix, F • Matrix F - the “pixels - epipolar lines” mapping: T – Vector ar= Fpl → parameters of the epipolar line pr ar= 0 in the right image related to the pixel pl in the left image T T T – Vector al = pr F → parameters of the epipolar line al pl = 0 in the left image related to the pixel pr in the right image – Just as the matrix E, the fundamental matrix F has rank 2 – F accounts for both the intrinsic and extrinsic parameters • The epipolar constraint can be established with no prior knowledge of the stereo parameters! COMPSCI 773 8 The Eight-point Algorithm • n ≥ 8 corresponding points in the images are known – Each correspondence i - a homogeneous linear equation: F11 F12 F13 x l,i T p Fp 0 x y 1 F F F y 0 r,i l,i = ⇒ [ r,i r,i ] 21 22 23 l,i = F 31 F32 F33 1 ⇒ x r,i x l,iF11 + x r,i y l,iF12 + x r,iF13 + y r,i x l,iF21 + y r,i y l,iF22 + y r,iF23 + x l,iF31 + y l,iF32 + F33 = 0 – If the n points do not form a degenerate configuration, the 9 entries of F are given by the non-trivial solution of this homogeneous linear system € COMPSCI 773 9 The Eight-point Algorithm – Since the system is homogeneous, the solution is unique up to a signed scaling factor – Typically, n > 8, so that the system is over-determined, and its solution is obtained by singular value decomposition (SVD) related techniques • A - the system’s matrix n × 9: x r,1x l,1 x r,1y l,1 x r,1 y r,1x l,1 y r,1y l,1 y r,1 x l,1 y l,1 1 A = M M M M M M M M M x r,n x l,n x r,n y l,n x r,n y r,n x l,n y r,n y l,n y r,n x l,n y l,n 1 COMPSCI 773 10 € n α γ β δ α β γ δ Xr Yr Xl Yl ≡ xr,i yr,i xl,i xl,i ∑i=1 The Eight-point Algorithm € T • SVD A=UDV ⇒ the solution is the column of V corresponding to the only null singular value of A T • V = [v1 … v9]; vi - the eigenvectors of the 9×9 matrix A A 2 2 2 2 2 2 Xr Xl Xr Xl Yl Xr Xl XrYrXl XrYrXl Yl XrYrXl XrXl XrXl Yl XrXl 2 2 2 2 2 2 Xr Xl Yl Xr Yl Xr Yl XrYrXl Yl XrYr Yl XrYrYl XrXl Yl XrYl XrYl 2 2 2 Xr Xl Xr Yl Xr XrYrXl XrYrYl XrYr XrXl XrYl Xr 2 2 2 2 2 2 XrXl Yr XrYrXl Yl XrYrXl Yr Xl Yr Xl Yl Yr Xl YrXl YrXl Yl YrXl T 2 2 2 2 2 2 A A = X rYrXl Yl XrYr Yl XrYrYl Yr Xl Yl Yr Yl Yr Yl YrXl Yl YrYl YrYl 2 2 2 XrYrXl XrYrYl XrYr Yr Xl Yr Yl Yr YrXl YrYl Yr 2 2 2 XrXl XrXl Yl XrXl Yr Yl YrXl Y YrXl Xl Xl Yl Xl X X Y X Y2 X Y Y X Y Y Y2 Y Y X Y Y2 Y r l l r l r l r l l r l r l l l l l XrXl XrYl Xr YrXl YrYl Yr Xl Yl n COMPSCI 773 11 € The Eight-point Algorithm • Due to noise, the solution is the column of V associated with the least singular value • The estimated fundamental matrix Fest is almost always non-singular, i.e. is full rank (3) rather than the expected rank 2 – The singularity is enforced by adjusting the entries of Fest: T • The SVD Fest = UDV • Set the smallest singular value in the diagonal matrix D to zero to obtain the corrected matrix D′ T • The corrected estimate: F ′ = UD′V COMPSCI 773 12 To Avoid Numerical Instabilities: • Coordinates of the corresponding points have to be normalised to make entries of A of comparable size – Translate the two coordinates of each point to the centroid of n n 1 1 each data set: mx = xi; my = yi n ∑i=1 n ∑i=1 – Scale the norm of each point so that the average norm over 2 2 the data set is 1: d = 1 x − m + y − m n 2 ∑i ( i x ) ( i y ) € xi (xi − mx ) d 1 d 0 −mx d xi p = y ⇒ p′ = y − m d ⇔ p′ = Hp ≡ 0 1 d −m d y i i €i ( i y ) i i y i 1 1 0 0 1 1 COMPSCI 773 13 € Stable Eight-Point Algorithm • Input: n pixel-to-pixel correspondences T T p = x y 1 ; p = x y 1 : i =1, ,n {( l,i [ l,i l,i ] r,i [ r,i r,i ] ) K } • Data normalisation: {(p′ l, i = Hlpl,i; p′ r ,i = Hrpr,i ) : i =1,K,n} € 1 m 1 m 0 − l,x 0 − r,x dl dl dr dr d l 0 ml,x d r 0 mr,x 1 ml,y 1 1 mr,y 1 H = 0 − ; H− = 0 d m ; H = 0 − ; H− = 0 d m l € l l l,y r r r r,y dl dl dr dr 0 0 1 0 0 1 0 0 1 0 0 1 COMPSCI 773 14 € Stable Eight-Point Algorithm T • SVD A = UDV of the n×9 matrix A for the system of n linear equations; n ≥ 8 (over-determined for n > 8): F1 F2 F3 x l′, i T T p′ F ′p ′ 0 x ′ ,y ′ ,1 F F F y ′ 0 a f 0 : i 1,2,...,n r,i l,i = ⇒ [ r,i r,i ] 4 5 6 l,i = ⇒ { i = = } F7 F8 F9 1 T a1 F1 T F a 2 T 2 Af = 0 where A = ; a i = [x l′, i x r′ ,i,y l′, i x r′, i,x r′, i,x l′, i y r′ ,i,y l′, i y r′, i,y r′, i,x l′, i,y l′, i,1]; f = M M T a n F 9 COMPSCI 773 15 € Stable Eight-Point Algorithm – The entries of F ′ (up to an unknown, signed scale factor) are the components of the column of V corresponding to the least singular value of A T • SVD F ′=UDV of F ′to enforce the singularity constraint – Set the smallest singular value in the diagonal of D equal to 0 to obtain the corrected matrix D′ T – Compute the corrected estimate F ″=UD′V of the fundamental matrix -1 -1 • Renormalisation: the output estimate F =Hr F″Hl COMPSCI 773 16 Locating the Epipoles • Accurate localisation of the epipoles: – To refine the locations of the conjugate epipolar lines – To simplify the stereo geometry – To recover 3D structure in the case of uncalibrated stereo • The left epipole el lies on all the epipolar lines in the left T image ⇒ the relationship pr Fel = 0 holds for every pr – F is not identically zero, so it follows that Fel = 0 – F has rank 2 - the epipole el is the null space of F • The null space is the set of all solutions s to the equation Fs = 0 T – Similarly, er is the null space of F COMPSCI 773 17 Algorithm to Locate Epipoles • Input: the fundamental matrix F T • SVD F = UDV – The epipole el : the column of V corresponding to the null singular value – The epipole er : the column of U corresponding to the null singular value 0 0 0 0 1 0 1 0 0 0 1 1 1 2 2 F = 0 0 1 = 1 0 1 0 0 0 1 0 0 ⇒ e = e = 0 2 2 l r 0 1 0 1 0 −1 0 0 −1 0 1 −1 0 2 2 1 4