3D Reconstruction COMPSCI 527 — Computer Vision COMPSCI 527 — Computer Vision 3D Reconstruction 1 / 25 Outline 1 The Epipolar Geometry of a Pair of Cameras 2 The Essential Matrix 3 The Eight-Point Algorithm: t, R 4 Triangulation: Pm 5 Bundle Adjustment COMPSCI 527 — Computer Vision 3D Reconstruction 2 / 25 The Epipolar Geometry of a Pair of Cameras The Epipolar Geometry of a Pair of Cameras P p r y oje ra c ction tion roje p r a y epipolar plane epipolar line of Ib p Ia center of a projection p p b a p a baselin epipolar line of b epipole eb e camera a camera b center of b projection epipole e a COMPSCI 527 — Computer Vision 3D Reconstruction 3 / 25 The Epipolar Geometry of a Pair of Cameras The Epipolar Constraint P p r y oje ra c ction tion roje p r a y epipolar plane I • The point pepipolara in line image of a that corresponds to pointb pb in p Ia imagecenter ofb is ona the epipolar line of pb projection p pa ... and vice versa b p epipole ae baselin epipolar line of b • This is the only generalb constrainte between two images of the samecame scene;ra a 3D reconstruction dependscamera b once itnter of epipole be projection • Epipolar lines come in corresponding pairsa • Two pencils of lines supported by the two epipoles COMPSCI 527 — Computer Vision 3D Reconstruction 4 / 25 The Epipolar Geometry of a Pair of Cameras Another Way to State the Epipolar Constraint P p r y oje ra c ction tion roje p r a y epipolar plane epipolar line of Ib p Ia center of a projection p pa The two projection raysb and the baseline are coplanar for p epipole ae baselin epipolar line of b corresponding pointsb e camera a camera b center of b projection epipole e a COMPSCI 527 — Computer Vision 3D Reconstruction 5 / 25 The Epipolar Geometry of a Pair of Cameras The Epipolar Constraint and 3D Reconstruction • Relative position and orientation of the two cameras are unknown • Given corresponding points pa, pb (found, say, by tracking) a a a we can write one algebraic constraint on P, Rb, tb • With enough pairs of corresponding points, we can write a system in these quantities • Solving the system is 3D reconstruction COMPSCI 527 — Computer Vision 3D Reconstruction 6 / 25 The Essential Matrix The Essential Matrix • How to write the epipolar constraint algebraically? a a a • The constraint is nonlinear in P, Rb, tb • Introduce a new 3 × 3 essential matrix E that combines rotation and translation to a a a • Separate structure ( P) from motion ( Rb, tb) • Make motion estimation a linear problem in E • Computation sequence: • Find E by solving a homogeneous linear system • Find rotation and translation from E • Find structure given rotation and translation COMPSCI 527 — Computer Vision 3D Reconstruction 7 / 25 The Essential Matrix P Coordinates p p b a Zb Za Xb Xa Yb Ya • Image points as world points: 2 a 3 2 b 3 xa xb a a b b pa = 4 ya 5 and pb = 4 yb 5 f f • Each camera measures a point in its own reference system b a a a • Transformation: p = Rb( p − tb) • Inverse: a b b b b a T b a a p = Ra( p − ta) where Ra = Rb and ta = − Rb tb COMPSCI 527 — Computer Vision 3D Reconstruction 8 / 25 The Essential Matrix Writing all Quantities in System a a a • Pose of camera b in a is specified by Rb, tb, both in a a • Image point pa is in a b a • Image point pb is in b, need to transform to pb b a a a a a T b a • Invert pb = Rb( pb − tb) to obtain pb = Rb pb + tb • Too many super/subscripts to keep track of. Define a b a a a a = pa , b = pb , R = Rb , t = tb , e = eb a def a T • b = pb = R b + t b a Zb Za e Xb Xa Yb Ya R, t COMPSCI 527 — Computer Vision 3D Reconstruction 9 / 25 The Essential Matrix Aside: Epipole and Translation • The epipole of b in a is the same as t up to norm a • Define: e = eb • e / t b a Zb Za e Xb Xa Yb Ya R, t COMPSCI 527 — Computer Vision 3D Reconstruction 10 / 25 The Essential Matrix The Epipolar Constraint, Algebraically ab = RT b + t b • The two projectiona rays and the baseline are coplanar Zb Za a X a T • The triple product of eb, t, and a is zero:b b (t × a) = 0 X (RT b + t)T (t × a) = 0,a but tT (t × a) = 0 so that Y T T Y b (R b) (t × a) =a 0 R, t COMPSCI 527 — Computer Vision 3D Reconstruction 11 / 25 The Essential Matrix The Essential Matrix (RT b)T (t × a) = 0 bT R (t × a) = 0 T b R [t]×a = 0 2 3 0 −tz ty T where t = (tx ; ty ; tz ) and [t]× = 4 tz 0 −tx 5 −ty tx 0 T b E a = 0 where E = R [t]× • This equation is the epipolar constraint, written in algebra • Holds for any corresponding a, b in the two images (as world vectors in their reference systems) • E is the essential matrix • The epipolar constraint is linear in E but not in R and t COMPSCI 527 — Computer Vision 3D Reconstruction 12 / 25 The Essential Matrix The Structure of E: Rank and Null Space • E has rank 2 and null(E) = span(t) = span(e) • Geometry: • The epipole e is in the epipolar line bT Ex = 0 for every b • Therefore, bT Ee = 0 for all b • Therefore Ee = 0, so e 2 null(E) • Algebra: [t]×t = t × t = 0 t × v 6= 0 if v is not parallel to t b • Therefore, thea rank of [t]× is 2 for t 6= 0 Z Z b • Since R is full rank,a the solutions of [t]×x = 0 and Ex = 0 e Xb (i.e., R[t] x = 0) are the same × Xa • Therefore, rank(E) = 2 for nonzero t and null(E) = span(t) Yb Ya • Either way, null(E) = spanR, t(e) = span(t) = baseline COMPSCI 527 — Computer Vision 3D Reconstruction 13 / 25 The Essential Matrix The Structure of E: Singular Values • E has two equal singular values and one zero singular value • Proof • Let v be perpendicular to t. Then k[t]×vk = ktk kvk (geometric definition of cross product) • Let kvk = 1. Then k[t]×vk = ktk • v ? t means that v 2 row space([t]×) because null(E) = span(t) • Therefore, all unit-norm vectors v 2 row space([t]×) are mapped to a circle • Therefore [t]× has two equal singular values • Third is zero because t 2 null([t]×) • Ditto for E, since E = R[t]× and R is orthogonal • Therefore v3 ∼ e ∼ t • If we have E, we can find camera translation t by SVD! COMPSCI 527 — Computer Vision 3D Reconstruction 14 / 25 The Essential Matrix A Fundamental Ambiguity • The equation bT E a = 0 is homogeneous in E • Therefore, we cannot tell the magnitude of E, or of t in E = R [t]× • Absolute scale cannot be determined from images alone • This ambiguity is general, has nothing to do with the specifics of the formulation • Cameras fundamentally measure angles, not distances • This ambiguity is often exploited in movie special effects • W.l.o.g., let ktk = 1 • Measure everything in units of inter-camera distance COMPSCI 527 — Computer Vision 3D Reconstruction 15 / 25 The Essential Matrix Next Problem: How to Find E? bT E a = 0 • Given pairs (a1; b1);::: (an; bn) (tracking) • Write one epipolar constraint equation per pair • Linear and homogeneous in E COMPSCI 527 — Computer Vision 3D Reconstruction 16 / 25 The Eight-Point Algorithm: t, R The Eight-Point Algorithm • H. C. Longuet-Higgins, Nature, 293:133–135, 1981 • Needs at least 8 corresponding point pairs • Preferably many more • Overview: • Given pairs (a1; b1);::: (an; bn) (tracking) • Write one epipolar constraint equation per pair • T T Solve linear equations b1 E a1 = 0, ::: , bn E an = 0 for E • Solve E = R [t]× for t, R • Compute the 3D structure (points Pm) from am, bm, t, R • The last step is called triangulation COMPSCI 527 — Computer Vision 3D Reconstruction 17 / 25 The Eight-Point Algorithm: t, R Rewriting the Epipolar Constraint bT E a = 0 2 3 2 3 e11 e12 e13 a1 b1 b2 b3 4 e21 e22 e23 5 4 a2 5 = 0 e31 e32 e33 a3 e11a1b1 + e12a2b1 + e13a3b1+ e21a1b2 + e22a2b2 + e23a3b2+ e31a1b3 + e32a2b3 + e33a3b3 = 0 2 3 e11 6 e12 7 6 7 6 e13 7 6 7 6 e21 7 6 7 a1b1 a2b1 a3b1 a1b2 a2b2 a3b2 a1b3 a2b3 a3b3 6 e22 7 = 0 6 7 6 e23 7 6 e 7 6 31 7 4 e32 5 e33 cT η = 0 T • With n point pairs, cmη = 0 for m = 1;:::; n COMPSCI 527 — Computer Vision 3D Reconstruction 18 / 25 The Eight-Point Algorithm: t, R Solving for E T cmη = 0 for m = 1;:::; n Cη = 0 where C is n × 9 • Because of the scale ambiguity, we cannot tell the norm of η • Set kηk = 1 • Homogeneous, least squares problem on the unit sphere • We know how to solve that! • Repackage η into 3 × 3 matrix E COMPSCI 527 — Computer Vision 3D Reconstruction 19 / 25 The Eight-Point Algorithm: t, R Solving for t • We have E now E = R [t]× • We saw that null(E) = span(t) • So we know how to find t with ktk = 1, up to a sign • ±t (and also ±[t]×) COMPSCI 527 — Computer Vision 3D Reconstruction 20 / 25 The Eight-Point Algorithm: t, R Solving for R • We have both E and T = ±[t]× E = R [t]× • Linear system in R, but with the constraints RT R = I and det(R) = 1 • Linear, constrained LSE optimization problem: The Procrustes problem, arg minRT R=I kE − RT kF • Appendix in the notes gives a solution based on the SVD • Since T has rank 2, it turns out that the there are two solutions, R1 and R2 for each choice of sign in T = ±[t]× COMPSCI 527 — Computer Vision 3D Reconstruction 21 / 25 The Eight-Point Algorithm: t, R The Fourfold Ambiguity (t; R1) ; (−t; R1) ; (−t; R2) ; (t; R2) P ZD ZC ZC ZD XD XD D C XC C XC D P P ZC ZC D’ D’ XC XC XD’ C C XD’ ZD’ ZD’ P • Only one solution places all world points in front of both cameras • Try all four solutions, and reconstruct world points by triangulation • Pick the one solution that makes sense COMPSCI 527 — Computer Vision 3D Reconstruction 22 / 25 Triangulation: Pm Triangulation 2 0 3 2 3 a1 a1 0 0 • For simplicity, divide a = 4 a2 5 by f so that now a = 4 a2 5 f 1 def a • Let α = 1 (coordinates in canonical image reference system) a2 • Ditto for b, β • Projection equations in each camera reference frame: A is P in frame a 1 A1 1 B1 α = A and β = B 3 A2 3 B2 A1 B1 • Rewrite as αA3 = and βB3 = A2 B2 Plug B = R(A − t) into second set of equations • All equations are linear.