Epipolar Geometry and the Fundamental Matrix Review About Camera Matrix P (From Lecture on Calibration) • Between the World Coordinates

Epipolar Geometry and the Fundamental Matrix Review about Camera Matrix P (from Lecture on Calibration) • Between the world coordinates

X=(Xs, Xs, Xs, 1) of a scene point and the coordinates x=(u’,v’,w’) of its projection,

we have the following linear xpix = u'/ w' transformation: with y pix = v'/ w' x = P X

• P is a 3x4 matrix that completely represents the mapping from the scene to the image and is therefore called a “camera”. Central Projection

If world and image points are represented by homogeneous vectors, central projection is a linear transformation: x s éx ù xi = f éu ù é f 0 0 0ù s z êy ú s ê ú ê úê s ú x = u / w, y = v / w êv ú = ê 0 f 0 0ú i i êzs ú y êwú ê 0 0 1 0úê ú s ë û ë û Image plane Scene point yi = f ë 1 û z (xs , ys , zs ) s y Image point (x , y , f) center of x i i projection z C f Pixel Components

Transformation uses: y Image point ypix (xi , yi , f) • principal point (x , y ) 0 0 y 0 Image center x • scaling factors kx and ky Principal point x x0 pix

xs xs + zs x0 xi = f x pix = kx xi + x0 = f k x zs zs y + z y ys s s 0 y pix = k y yi + y0 = f k y yi = f zs zs

éxs ù éu'ù éa x 0 x0 0ùê ú y a x = f kx xpix = u'/ w' êv'ú = ê 0 a y 0úê s ú ê ú ê y 0 ú with then êzs ú a y = f k y y pix = v'/ w' ëêw'ûú ëê 0 0 1 0ûúê ú ë 1 û Internal Camera Parameters

éxs ù éu'ù éa x s x0 0ùê ú y a x = f kx xpix = u'/ w' êv'ú = ê 0 a y 0úê s ú ê ú ê y 0 ú with êzs ú a y = - f k y y pix = v'/ w' ëêw'ûú ëê 0 0 1 0ûúê ú ë 1 û

éa x s x0 0ù éa x s x0 ùé1 0 0 0ù ê 0 a y 0ú = ê 0 a y úê0 1 0 0ú = K I | 0 ê y 0 ú ê y 0 úê ú [ 3 3 ] ëê 0 0 1 0ûú ëê 0 0 1 ûúëê0 0 1 0ûú

• ax and ay “focal lengths” in pixels

• x0 and y0 coordinates of image center in pixels

•Added parameter s is skew parameter • K is called calibration matrix. Five degrees of freedom. •K is a 3x3 upper triangular matrix From Camera Coordinates to World Coordinates

y Principal plane (Cx, Cy) x Camera center C Image plane

Principal point (x0, y0 ) Image point Z z (x , y , f) i i Principal axis Cz Y

(xs , ys , zs ) éx ù éX ù S S M O ê ú ê ú (X , Y , Z ) X y R T Y s s s ê S ú é ùê S ú = ê T ú êzS ú ë03 1ûêZ S ú ê ú ê ú ë 1 û ë 1 û Using Camera Center Position in World Coordinates ~ • We can use - R C instead of T

éxS ù éX S ù ê ú ~ ê ú yS é R - R Cù YS ê ú = ê úê ú ê ú T ê ú zS ë03 1 û Z S ê ú ê ú ë 1 û ë 1 û Linear Transformation from World Coordinates to Pixels • Combine camera projection and coordinate transformation matrices into a single matrix P éx ù éu'ù s ê y ú êv'ú = K I | 0 ê s ú ê ú [ 3 3 ] ê zs ú é X S ù ëêw'ûú ê ú éu'ù ~ 1 é ùê Y ú ë û ê ú R - R C ê S ú Þ v' = K [I3 | 03 ]ê ú ê ú T ê Z ú ë03 1 û S éxS ù éX S ù êw'ú ê ú ~ ë û 1 êy ú é ùêY ú ë û ê S ú R - R C ê S ú X = ê ú é S ù ê ú T ê ú éu'ù ê ú zS ë03 1 û Z S Y ê ú ê ú ê ú ê S ú 1 1 Þ ê v'ú = P x = P X ë û ë û êZ S ú ëêw'ûú ê ú ë 1 û Properties of Matrix P

• P has 11 degrees of freedom: ~ • 5 from triangular calibration matrix K, 3 from R and 3 from C • P is a fairly general 3 x 4 matrix •left 3x3 submatrix KR is non-singular Cross-Product in Matrix Form T • If a = (a1, a2, a3) is a 3-vector, then one can define a corresponding skew-symmetric

matrix é 0 - a3 a2 ù [a] = ê a 0 - a ú ´ ê 3 1 ú ëê- a2 a1 0 ûú • The cross-product of 2 vectors a and b can

be written a´b = [a]´ b

• Matrix [a]x is singular. Its null vector (right or left) is a T • b . (a´b) = 0 Û b [a]´ b = 0 "a, b T T T • c [a]´ b = - b [a]´c Þ F [a]´ F = [s]´ "F,a Definition of Epipolar Geometry

• Projective geometry between two views • Independent of scene structure xi • Depends only on the x’i cameras’ internal parameters and relative pose of cameras x'T F x = 0 • Fundamental matrix F i i for any pair of encapsulates this geometry corresponding points

xi and x’i in the 2 images Relation between

Image Points xi and xi’ • Camera centers C and C’,

scene point Xi, image points x and x’ i i X belong to a common i epipolar plane p p • Epipoles e and e’ xi x’i l’ § On baseline CC’ l • Epipolar lines l and l’ C e e' C’ Pencils of Epipolar Lines

C e e' C’ Computation of F

• F can be computed from correspondences between image points alone • No knowledge of camera xi internal parameters x’ required i • No knowledge of relative x'T F x = 0 pose required i i for any pair of corresponding points

xi and x’i in the 2 images Finding the Fundamental Matrix from Known Cameras P and P’ (Outline) • Pick up an image point x in camera P • Find one scene point X on ray of x in X camera P p • Find the image x’ of X in camera P’ x x’ • Find epipole e’ as image of C in camera P’ l l’ is epipole = P’C C e e' • Find epipolar line l’ from e’ to x’ in P’ as C’ function of x • The fundamental matrix F is defined by l’=F x • x’ belongs to l’, so x’T l’= 0, so x’T F x = 0 • The fundamental matrix F is alternately defined by x’T F x = 0 Finding the Fundamental Matrix from Known Cameras P and P’ (Details) • Pick up an image point x in camera P X • Find one scene point on ray from C to x § Point X = P+x satisfies x = PX p x x’ + T T -1 • P = P (P P ) , so l l’ PX = P PT (P PT )-1 x = x • Image of this point in camera P’ is C e e' C’ x’ = P’X = P’ P+x • Image of C in camera P’ is epipole e’ = P’C • Epipolar line of x in P’ is + + l' = (e')´(P'P x) = [e']´ P'P x + • l’=F x defines F fundamental matrix Þ F = [P'C]´ P'P • x’ belongs to l’, so x’T l’= 0, so x’T F x = 0 Properties of Fundamental Matrix F

• Matrix 3X3 (since x’T F x = 0 ) • If F is fundamental matrix of camera pair (P, P’) then the fundamental matrix F’ of camera pair (P’, P) is equal to FT § xT F’ x’ = 0 implies x’T F’T x = 0, so F’ = FT • Epipolar line of x is l’ = F x • Epipolar line of x’ is l = FT x’ More Properties of F

• Epipole e’ is left null space of F, and e is right null space. § All epipolar lines l’ contains epipole e’, so e’T l’= 0, i.e. e’T F x = 0 for all x. Therefore e’T F = 0 Similarly eT FT x’ = 0 implies eT FT = 0, therefore F e = 0 + • F is of rank 2 because F = [e’]xP’P and [e’]x is of rank 2 • F has 7 degrees of freedom § There are 9 elements, but scaling is not significant § Det F = 0 removes one degree of freedom Mapping between Epipolar Lines (a Homography) • Define x as intersection between line l and a line k (k does not pass through epipole e): x l’ l x = k ´ l = [k]x l C k e e' C’ l' = F x = F[k]x l • Line e does not pass through point e

l' = F x = F[e]x l • Similarly T T l = F x'= F [e']x l' Retrieving Camera Matrices P and P’ from Fundamental Matrix F ~ • General form of P is P = KR[I3 | -C] • Select world coordinates as camera coordinates of first camera, select focal length = 1, and count pixels from the

principal point. Then P= [ I3 | 0] • Then P’ = [S F | e’] with S any skew-symmetric matrix is a solution. Proof: § x’ T F x = XT P’T F P X T T § P’ F P = [S F | e’] F [ I3 | 0 ] is skew-symmetric § For any skew-symmetric matrix S’ and any X, XT S’ X = 0

• S = [e’]x is a good choice. Therefore P’=[[e’]x F | e’] T [S F | e’] F [ I3 | 0 ] is skew-symmetric

T T T T T éF S F 03 ù éF S F 03 ù [SF | e'] F [I3 | 03 ]= ê T ú = ê T ú ë e' F 0 û ë 03 0 û • e’T F=0 because e’ is left null space of F • FT ST F is skew-symmetric for any F and any skew-symmetric S Essential Matrix E • Specialization of fundamental matrix for calibrated cameras and normalized coordinates § x = P X = K [R | T] X -1 § Normalize coordinates: x0 = K x = [R | T] X • Consider pair of normalized cameras § P = [I | 0], P’ = [R | T] + • Then we compute F = [P'C]´ P'P T [P'C]´ =[[R | T] [0 0 0 1] ]´ = [T]´

+ éI3 ù + P = ê T ú , P'P = R Þ F = [T]´ R º E ë03 û Essential Matrix and Fundamental Matrix • The defining equation for essential matrix is T x0’ E x0 = 0, with -1 § x0 = K x -1 § x0’ = K’ x’ • Therefore x’T K’-T E K-1 x = 0 • Comparing with x’T F x = 0, we get

E = K’T F K Computing Fundamental Matrix from Point Correspondences • The fundamental matrix is defined by the T for any pair of equation x'i F xi = 0 corresponding points

xi and x’i in the 2 images • The equation for a pair of points

(x, y, 1) and (x’, y’, 1) is: x' x f11 + x' y f12 + x' f13 +

+ y' x f 21 + y' y f 22 + y' f 23 +

• For n point matches: + x f31 + y f32 + f33 = 0

é x'1 x1 x'1 y1 x'1 y'1 x1 y'1 y1 y'1 x1 y1 1ù A f = ê M M M M M M M M Mú f = 0 ê ú ëêx'n xn x'n yn x'n y'n xn y'n yn y'n xn yn 1ûú Computing Fundamental Matrix from Point Correspondences • We have a homogeneous set of equations A f = 0 • f can be determined only up to a scale, so there are 8 unknowns, and at least 8 point matchings are needed § hence the name “8 point algorithm” • The least square solution is the singular vector corresponding the smallest singular value of A, i.e. the last column of V in the SVD A = U D VT Next Class

• 3D Reconstruction from Multiple Views References

• Multiple View Geometry in Computer Vision, R. Hartley and A. Zisserman, Cambridge University Press, 2000.