Lecture 3 - Two View Geometry & the Essential Matrix

Lecture 3 - Two view geometry & the Essential Matrix

DD2429

September 10, 2018 Review of the general perspective projection camera matrix

Note: Most slides are copied/adapted from Multiple View Geometry Tutorial Part 1, CVPR 1999, by R. Hartley and A. Zisserman Imaging Geometry

Multiple View Geometry Perspective projection Imaging Geometry - Perspective projection

¥ ¥ X §©¨ Richard Hartley and Andrew Zisserman X y ¤

x ¦ ¦ Y x O Z p Z

¤ where Z . image plane CVPR June 1999 This can be written as a•linearWorld andmapping camera axesbetw aligned,een imagehomogeneous plane at (0, 0, f)coordinatesT . (the equation is only up to a scale• Perspectivefactor): projection of point X can be written as

¥ x X ¥ §¨ y =Xλ Y  ¦ f Y Z ¦

where λ = f/Z. Z £ ¡ where a ¢ projection matrix represents a map from 3D to 2D.

Image Coordinate System

Part I: Single and Two View Geometry Internal camera parameters

ycam ! cam

The main points covered in this part are: " # # # ! y p cam 0 x £ ¡

¢ cam y

A perspective (central) projection camera is represented by a matrix. " where the units of $ %'& / 4 ,.- , 3 (*) 2 1 The most general perspective transformation transformation between two planes + + 0 are . x x (a world plane and the image plane, or two image planes induced by a world 0 plane) is a plane projective transformation. This can be computed from the correspondence of four (or more) points. 6 < @ D 6 6 < < C !

The epipolar geometry between two views is represented by the fundamental : cam cam G 798 =9> 798 798 =9> =9> A9B E9F 5 #;: C # # # ! " matrix. This can be computed from the correspondence of seven (or more) ? cam cam ? ? points. : ? ? " C C " where $ . Imaging Geometry

Multiple View Geometry Perspective projection Imaging Geometry - Perspective projection

¥ ¥ X §©¨ Richard Hartley and Andrew Zisserman X y ¤

x ¦ ¦ Y x O Z p Z

¤ where Z . image plane CVPR June 1999

T This can be written as a•linearWorld andmapping camera axesbetw aligned,een imagehomogeneous plane at (0, 0, f)coordinates. (the equation is only up to a scale• Perspectivefactor): projection written as a linear mapping between homogeneous coordinates (λ is the scale factor):

¥   X §¨ x 1 0X 0 0 Y ¦   y = λ 0 1Y 0 0  

    ¦ Z f 0 0 1 0   Z 1 £ ¡ where a ¢ projection matrix represents a map from 3D to 2D.

Image Coordinate System

Part I: Single and Two View Geometry Internal camera parameters

ycam ! cam

The main points covered in this part are: " # # # ! y p cam 0 x £ ¡

¢ cam y

Multiple View Geometry Perspective projection

X ¥ ¥ X §©¨ Richard Hartley and Andrew Zisserman X y ¤

x ¦ ¦ Y x O Z p Z

¤ where Z . image plane CVPR June 1999 This can be written as a linear mapping between homogeneous coordinates (the equation is only up to a scale factor):

¥ ¥

§¨ X ¦ Y ¦

Z £ ¡ where a ¢ projection matrix represents a map from 3D to 2D.

Image Coordinate System Image Coordinate System Part I: Single and Two View Geometry Internal camera parameters

• Internal camera parameters ycam ! cam kx xcam = x − x0

The main points covered in this part are: " # # # ! y p cam 0 x ky ycam = y − y0 £ ¡

¢ cam y

A perspective (central) projection camera is represented by a matrix. " where the units of $ 4 %'& / 3 ,.- , where the units of k , k are (*)

2 x y 1 The most general perspective transformation transformation between two planes + + 0 are . x x (a world plane and the image plane, or two image planes induced by a world 0 pixels/length. plane) is a plane projective transformation. This can be computed from the correspondence of four (or more) points. D @ 6 < 6 6 < < C !

:         The epipolar geometry between two views is represented by the fundamental cam x cam αx 0 x0 xcam xcam G 798 =9> 798 798 =9> =9> A9B E9F 5 C # #;: # #

! 1 " matrix. This can be computed from the correspondence of seven (or more) ? cam x = y cam= 0 αy y0 ycam = K ycam

?   ?       : f points. 1 0 0 1 f f ? ? " C C " where $ . where αx = f kx, α = fky. Camera Calibration

• K is a 3 × 3 upper-triangular matrix called the camera calibration matrix

  αx 0 x0 K =  0 αy y0 0 0 1

• There are four parameters:

1. The scaling in the image x and y directions - αx and αy.

2. The principal point (x0, y0) - point where the optic axis intersects the image plane.

• The aspect ratio is αy/αx. Concatenating the three matrices, 6 Camera Calibration Matrix < D @ 6 < :

X ¥ § ¦ 7 = % 4

: § G G ¦ 7 7 = = E A 5 #©: Y

G © ¡ : ¨ 798 8 > =9> is a upper triangular matrix, called the camera calibration F B : Z : matrix: @ D C ! G A9B E9F C # ! " ¡ : which deﬁnes the projection matrix from Euclidean 3-space to an image as

¢ There are four parameters: # C C (i) The scaling in the image and directions, and " . ¤ £ # ! ! (ii) The principal point $ , which is the point where the optic axis intersects the image plane. / © © £ ¤ § ¦ ¢ C C

" The aspect ratio is . Note, the camera centre is at X$ Y$ Z . In the following it is often only the ¡ form of that is important, rather than its decomposition.

World Coordinate System A Projective Camera World Coordinate System External camera parameters The camera model for perspective projection is a linear map between

Y homogeneous point coordinates $ cam ' $ /

X % % % & Z 6 6 < < * . - + % & ( ) 0 1 , !#" Y Xcam X ¥

§ O

¦ Z 7 7 = = cam cam Z

7 Y = 7 Y =

R, t " cam © X : ¨

798 =9> 798 =9> cam Zcam Z O

: : Y Image Point Scene Point X Euclidean transf•orExternalmation betwcameraeen parameters:world and camera coordinates

¦ ¡ ¢ ¢ is a rotation matrix The camera centre is the null-vector of © % 4 £ ¤

:     2 : § ¡ ¢ Xcam X is a translation vector e.g. if then the centre is $ $ $ .

Ycam R t Y

    ¢   = T   has 11 degrees of freedom (essential parameters). Zcam  0 1 Z  1 1 ¢ has rank 3.

• Euclidean transformation between world and camera coordinates - R is a 3 × 3 rotation matrix - t is 3 × 1 translation vector Putting everything together

X x 1 0 0 0 R t Y x = y = λK 0 1 0 0   = λK R t X     0T 1 Z  1 0 0 1 0   1

• Above deﬁnes the 3 × 4 projection matrix from Euclidean 3-space to an image as

x = λMX where M = K R t = KR IRT t

• Note the camera centre is at (X,Y,Z) = −RT t. Putting everything together

X x 1 0 0 0 R t Y x = y = λK 0 1 0 0   = λK R t X     0T 1 Z  1 0 0 1 0   1

• Above deﬁnes the 3 × 4 projection matrix from Euclidean 3-space to an image as

x = λMX where M = K R t = KR IRT t

• Note the camera centre is at (X,Y,Z) = −RT t. A Projective Camera

The camera model for perspective projection is a linear map between homogeneous point coordinates

X x Y y = λM     Z  1   1 where M has size 3 × 4 and projects a scene point to an image point.

• The camera centre is the null-vector of M

• M has 11 degrees of freedom (essential parameters).

• M has rank 3. What does calibration give?

• K provides the transformation between an image point and a ray in Euclidean 3-space.

• Once K is known the camera is termed calibrated.

• A calibrated camera is a direction sensor, able to measure the direction of rays - like a 2D protractor. Calibrating a camera correspondences between the 3D world and the camera view Camera Calibration

Problem Statement

• Given n correspondences xi ↔ Xi where each Xi is a scene point and xi is its image:

• Compute M = K R t such that xi ≈ λMXi

• Solution: The algorithm for camera calibration has two parts:

1. Compute the M from the set of point correspondences.

2. Decompose M into K,R and t using QR decomposition. Camera Calibration

Problem Statement

• Given n correspondences xi ↔ Xi where each Xi is a scene point and xi is its image:

• Compute M = K R t such that xi ≈ λMXi

• Solution: The algorithm for camera calibration has two parts:

1. Compute the M from the set of point correspondences.

2. Decompose M into K,R and t using QR decomposition. Algorithm step 1: Estimate the matrix M

• The projection equation in homogeneous coordinates: X x m m m m  11 12 13 14 Y y ' m m m m      21 22 23 24 Z  1 m m m m   31 32 33 34 1

• Express the projection in Euclidean coordinates: m X + m Y + m Z + m x = 11 12 13 14 m31X + m32Y + m33Z + m34 m X + m Y + m Z + m y = 21 22 23 24 m31X + m32Y + m33Z + m34

• Multiply with the denominators and rearrange:

m11X + m12Y + m13Z + m14 − m31xX − m32xY − m33xZ − m34x = 0

m21X + m22Y + m23Z + m24 − m31yX − m32yY − m33yZ − m34y = 0

• Two constraints per match =⇒ six matches needed total. Algorithm step 1 continued

• Set up a system of equations with two equations per match: - Let

X1 Y1 Z1 1 0 0 0 0 −x1X1 −x1Y1 −x1Z1 −x1   0 0 0 0 X1 Y1 Z1 1 −y1X1 −y1Y1 −y1Z1 −y1    X2 Y2 Z2 1 0 0 0 0 −x2X2 −x2Y2 −x2Z2 −x2     0 0 0 0 X2 Y2 Z2 1 −y2X2 −y2Y2 −y2Z2 −y2  Q =    .   .    Xn Yn Zn 1 0 0 0 0 −xnXn −xnYn −xnZn −xn 0 0 0 0 Xn Yn Zn 1 −ynXn −ynYn −ynZn −yn - Then

Q m = 0

T where m = (m11, m12, m13, m14, m21, m22, m23, m24, m31, m32, m33, m34)

• Determine m using least square optimization with the additional condition kmk2 = 1 Algorithm step 1 continued

• Set up a system of equations with two equations per match: - Let

Q m = 0

T where m = (m11, m12, m13, m14, m21, m22, m23, m24, m31, m32, m33, m34)

• Determine m using least square optimization with the additional condition kmk2 = 1 Algorithm step 1 continued

• Have this constrained optimization problem

min mT QT Q m subject to mT m = 1 m • Construct the Lagrangian

L(m, γ) = mT QT Q m + γ(1 − mT m)

and ﬁnd its minimum w.r.t. m

• Know the solution must fulﬁll

QT Q m = γm

• =⇒ mT QT Q m = γ

• =⇒ optimal m is eigenvector of QT Q with smallest eigenvalue. Algorithm step 1 continued

• Have this constrained optimization problem

min mT QT Q m subject to mT m = 1 m • Construct the Lagrangian

L(m, γ) = mT QT Q m + γ(1 − mT m)

and ﬁnd its minimum w.r.t. m

• Know the solution must fulﬁll

QT Q m = γm

• =⇒ mT QT Q m = γ

• equivalently =⇒ m is the vector corresponding to the smallest singular value of the SVD of Q. Algorithm step 1 summary

Solving for M • Concatenate equations from n (≥ 6) correspondences to generate 2n simultaneous equations. Can be written as

Qm = 0 where Q has size 2n × 12

• In general this will not have an exact solution.

• Instead minimize kQmk2 subject to kmk = 1.

• Solution is the eigenvector of QT Q with smallest eigenvalue.

• This linear solution can then be used as the staring point for a non-linear optimization such as

X T T 2 min (xi, yi) − λ M(Xi,Yi,Zi, 1) M i Algorithm step 1 summary

Solving for M • Concatenate equations from n (≥ 6) correspondences to generate 2n simultaneous equations. Can be written as

Qm = 0 where Q has size 2n × 12

• In general this will not have an exact solution.

• Instead minimize kQmk2 subject to kmk = 1.

• Solution is the eigenvector of QT Q with smallest eigenvalue.

• This linear solution can then be used as the staring point for a non-linear optimization such as

X T T 2 min (xi, yi) − λ M(Xi,Yi,Zi, 1) M i Algorithm step 1 summary

Solving for M • Concatenate equations from n (≥ 6) correspondences to generate 2n simultaneous equations. Can be written as

Qm = 0 where Q has size 2n × 12

• In general this will not have an exact solution.

• Instead minimize kQmk2 subject to kmk = 1.

• Solution is the eigenvector of QT Q with smallest eigenvalue.

• This linear solution can then be used as the staring point for a non-linear optimization such as

X T T 2 min (xi, yi) − λ M(Xi,Yi,Zi, 1) M i § G ¦

Algorithm step 1 continued Algorithm step 2: Decompose into $ and

(i) Concatenate the equations from ( ) correspondences to ¢ ¨ £ © ¦ G ¦

generate simultaneous equations, which can be written: , (i) Factor into using the matrix decomposition. This de- ¢ ¢ : £ G ¦ ¡ where is a matrix. termines and . (ii) In general this will not have an exact solution, but a (linear) (ii) Then ¤ © £ ¤ : § £ G

¤ § solution which minimises , subject to is obtained from $ $ © the eigenvector with least eigenvalue of £ £ . Or equivalently from

the vector corresponding to the smallest singular value of the SVD Note, this produces a matrix with an extra skew parameter @ D £ C

of . ! G A E C # ! (iii) This linear solution is then used as the starting point for a " B F non-linear minimisation of the difference between the measured : and projected point: £ £ 2 0 § § £ £ £ ¤ ¤ ¤ ¨

: with , and the angle between the image axes. ( ¤ 0 ¦¥ #

$ X$ Y$ Z$

Example - CalibrationExampleObject - Calibration Object

DeterDeterminemine accur accurateate cor cornerner positions positionsb byy (i)1. ExtrExtractact and linklink edgesedges usingusing CannyCann edgey operator.edge operator. (ii)2. FitFitlines lines toto edgesedges usingusing orthogonalorthogonal regression.regression. (iii)3. IntersectIntersect lineslines to obtainto obtain cornerscor toners sub-pixelto sub-pix accuracy.el accuracy. The ﬁnal error between measured and projected points is typically The lessﬁnal thanerror 0.02betw pixels.een measured and projected points is typically less than 0.02 pixels. Algorithm step 2: Decompose M into K, R and t

• Remember M = K (R, t)

• Have estimated M, how do we recover K, R and t? Algorithm step 2: Decompose M into K, R and t

• Remember M is a 3 × 4 matrix with

M = K(R, t)

• Denote M’s ﬁrst 3 columns as M13 and its last column as m4:

M = (M13, m4)

• Thus M13 = KR

=⇒ M13 is the product of an upper triangular matrix and a rotation matrix.

• Factor M13 into KR using QR matrix decomposition. This determines K and R. Algorithm step 2: Decompose M into K, R and t

• Remember M is a 3 × 4 matrix with

M = K(R, t)

• Denote M’s ﬁrst 3 columns as M13 and its last column as m4:

M = (M13, m4)

• Thus M13 = KR

=⇒ M13 is the product of an upper triangular matrix and a rotation matrix.

• Factor M13 into KR using QR matrix decomposition. This determines K and R. Algorithm step 2: Decompose M into K, R and t

• Remember M is a 3 × 4 matrix with

M = K(R, t)

• Denote M’s ﬁrst 3 columns as M13 and its last column as m4:

M = (M13, m4)

• Thus M13 = KR

=⇒ M13 is the product of an upper triangular matrix and a rotation matrix.

• Factor M13 into KR using QR matrix decomposition. This determines K and R. How QR factorization works for this example

Know 1. K is an upper triangular matrix, so let a b c K = 0 d e 0 0 f

2. R is a rotation matrix =⇒ RT R = I.

Recover last row of R from M13

- Partially compute the explicit matrix multiplication deﬁning M13:       a b c r11 r12 r13 ∗ ∗ ∗ M13 = KR = 0 d e r21 r22 r23 =  ∗ ∗ ∗  0 0 f r31 r32 r33 fr31 fr32 fr33

T T - Last row of M13 denoted by m3 = f (r31, r32, r33) = f r3 . T T T 2 - r3 r3 = 1 as R R = I =⇒ m3 m3 = f =⇒ f = km3k

- Last row of R is deﬁned by: r3 = m3/km3k How QR factorization works for this example

Know 1. K is an upper triangular matrix, so let a b c K = 0 d e 0 0 f

2. R is a rotation matrix =⇒ RT R = I.

Recover last row of R from M13

T T - Last row of M13 denoted by m3 = f (r31, r32, r33) = f r3 . T T T 2 - r3 r3 = 1 as R R = I =⇒ m3 m3 = f =⇒ f = km3k

- Last row of R is deﬁned by: r3 = m3/km3k How QR factorization works for this example

Know 1. K is an upper triangular matrix, so let a b c K = 0 d e 0 0 f

2. R is a rotation matrix =⇒ RT R = I.

Recover last row of R from M13

T T - Last row of M13 denoted by m3 = f (r31, r32, r33) = f r3 . T T T 2 - r3 r3 = 1 as R R = I =⇒ m3 m3 = f =⇒ f = km3k

- Last row of R is deﬁned by: r3 = m3/km3k How QR factorization works for this example

Write    T   T  a b c r1 m1 T T K = 0 d e ,R = r2  ,M13 = m2  T T 0 0 f r3 m3

Recover middle row of R from M13

- Know r3 = m3/km3k

- Have m2 = d r2 + e r3 as     a b c r11 r12 r13 M13 = 0 d e r21 r22 r23 0 0 f r31 r32 r33  ∗ ∗ ∗   ∗ ∗ ∗  = d r21 r22 r23 + e r31 r32 r33 ∗ ∗ ∗ ∗ ∗ ∗ How QR factorization works for this example

Write    T   T  a b c r1 m1 T T K = 0 d e ,R = r2  ,M13 = m2  T T 0 0 f r3 m3

Recover middle row of R from M13

- Know r3 = m3/km3k

- Have m2 = d r2 + e r3 as

 ∗ ∗ ∗   ∗ ∗ ∗  M13 = d r21 r22 r23 + e r31 r32 r33 ∗ ∗ ∗ ∗ ∗ ∗

T T T - r3 m2 = d r3 r2 + e r3 r3 = e T - r2 = (m2 − e r3)/d. Know r2 r2 = 1 =⇒ d = km2 − e r3k - Putting everything together

m2 − er3 T r2 = with e = r3 m2 km2 − er3k How QR factorization works for this example

Write    T   T  a b c r1 m1 T T K = 0 d e ,R = r2  ,M13 = m2  T T 0 0 f r3 m3

Recover ﬁrst row of R from M13

T - Know r3 = m3/km3k, r2 = (m2 − er3)/km2 − er3k with e = r3 m2

- Have m1 = a r1 + b r2 + c r3 as     a b c r11 r12 r13 M13 = 0 d e r21 r22 r23 0 0 f r31 r32 r33       r11 r12 r13 r21 r22 r23 r31 r32 r33 = a  ∗ ∗ ∗  + b  ∗ ∗ ∗  + c  ∗ ∗ ∗  ∗ ∗ ∗ ∗ ∗ ∗ ∗ ∗ ∗ How QR factorization works for this example

Write    T   T  a b c r1 m1 T T K = 0 d e ,R = r2  ,M13 = m2  T T 0 0 f r3 m3

Recover ﬁrst row of R from M13

T - Know r3 = m3/km3k, r2 = (m2 − er3)/km2 − er3k with e = r3 m2

- Have m1 = a r1 + b r2 + c r3 as       r11 r12 r13 r21 r22 r23 r31 r32 r33 M13 = a  ∗ ∗ ∗  + b  ∗ ∗ ∗  + c  ∗ ∗ ∗  ∗ ∗ ∗ ∗ ∗ ∗ ∗ ∗ ∗

T T - Orthogonality constraints =⇒ r3 m1 = c and r2 m1 = b T - Constraint r1 r1 = 1 =⇒ a = km1 − br2 − cr3k - Putting everything together

m1 − br2 − cr3 T T r1 = with b = r2 m1 and c = r3 m1 km1 − br2 − cr3k Algorithm step 2: Decompose M into K, R and t

• Remember M is a 3 × 4 matrix with

M = K(R, t)

• Denote M’s ﬁrst 3 columns as M13 and its last column as m4:

M = (M13, m4)

• Thus M13 = KR where K and R are determined using QR matrix decomposition.

• We must focus on last column of M to estimate t

−1 Kt = m4 =⇒ t = K m4 Algorithm step 2: Decompose M into K, R and t

• Remember M is a 3 × 4 matrix with

M = K(R, t)

• Denote M’s ﬁrst 3 columns as M13 and its last column as m4:

M = (M13, m4)

• Thus M13 = KR where K and R are determined using QR matrix decomposition.

• We must focus on last column of M to estimate t

−1 Kt = m4 =⇒ t = K m4 Algorithm step 2: Decompose M into K, R and t

Note about the decomposition: • Have produced a calibration matrix K with an extra skew parameter s   αx s x0 K =  0 αy y0 0 0 1

with s = tan(θ) and θ is the angle between the image axes. Two view geometry Correspondence Geometry

Given the image of a point in one view, what can wBasice say set-upabout its Two View Geometry position in another?

¨ X ? ¢ Cameras and such that X

X ? ¨ ¨

5 5

¢ Baseline between the cameras is non-zero. x l /

e e/ Given an image point in the ﬁrst view, where is the corresponding epipolar line point in the second view? for x

What is the relative position of the cameras? ¢ A point• Havein cameraone image matrices“gener s.t. ates” a line in the other image.

¢ This line is known as an epipolar 0line,0 and0 the geometry which What is the 3D geometry of the scene? x = λMX x = λ M X gives risewhereto it is known as epipolar geometry. M = K I 0 M 0 = K0 R t • Assume t 6= 0 - baseline between the two cameras is non-zero. Images of Planes Epipolar Geometry

Projective transformation between images induced by a plane Epipolar Plane Π

π ¡ ¡ x : ¡ ¢ π

x x/

¡ e e ¢ /

O O x Left epipolar line Right epipolar line x/ :

¡ ¡ ¡ ¨ ¢ ¢ ¢ 5 :

/ The epipolar line is the image of the ray through . C C ¨ £ ¢ The epipole is the point of intersection of the line joining the

¢ can be computed from the correspondence of four points on camera centres—the baseline—with the image plane. the plane. ¢ The epipole is also the image in one camera of the centre of the other camera.

¢ All epipolar lines intersect in the epipole. CorrespondenceQuestionsGeometry given basic set-up

GivHaveen the cameraimage matricesof a point s.t. in one view, what can we say about its Two View Geometry position in another?x = λ M X x0 = λ0 M 0 X

¨ X ? ¢ Cameras and such that X

X ? ¨ ¨

5 5

¢ Baseline between the cameras is non-zero. x l /

e e/ Given an image point in the ﬁrst view, where is the corresponding epipolar line point in the second view? for x

What is the relative position of the cameras? ¢ 1.A Givenpoint anin one imageimage point in“gener the ﬁrstates” view,a line wherein isthe theother correspondingimage.

¢ Thispointline in theis kno secondwn as view?an epipolar line, and the geometry which What is the 3D geometry of the scene? giv2.es Whatrise isto theit is relativeknown positionas epipolar of the cameras?geometry. 3. What is the 3D geometry of the scene?

Images of Planes Epipolar Geometry

Projective transformation between images induced by a plane Epipolar Plane Π

π ¡ ¡ x ¢ : ¡ π

x x/

¡ e e ¢ /

O O x Left epipolar line Right epipolar line x/ :

¡ ¡ ¡ ¨ ¢ ¢ ¢ 5 :

/ The epipolar line is the image of the ray through . C C ¨ £ ¢ The epipole is the point of intersection of the line joining the

¢ All epipolar lines intersect in the epipole. Correspondence Geometry

Given the image of a point in one vieCorrespondencew, what can Geometrywe say about its Two View Geometry position in another?Know the image of a point in one view:

¨ X ? ¢ Cameras and such that X

X ? ¨ ¨

5 5

¢ Baseline between the cameras is non-zero. x l /

e e/ Given an image point in the ﬁrst view, where is the corresponding epipolar line point in the second view? for x

What is the relative position of the cameras? ¢ A point inWhatone canimage we say about“gener its positionates” in anothera line camera?in the other image.

¢ This line is known as an epipolar line, and the geometry which What is the 3D geometry of the scene? gives rise to it is known as epipolar geometry.

Images of Planes Epipolar Geometry

Projective transformation between images induced by a plane Epipolar Plane Π

π ¡ ¡ x : ¡ ¢ π

x x/

¡ e e ¢ /

O O x Left epipolar line Right epipolar line x/ :

¡ ¡ ¡ ¨ ¢ ¢ ¢ 5 :

/ The epipolar line is the image of the ray through . C C ¨ £ ¢ The epipole is the point of intersection of the line joining the

¢ All epipolar lines intersect in the epipole. Correspondence Geometry

Given the image of a point in one vieCorrespondencew, what can we Geometrysay about its position in another? Two View Geometry Know the image of a point in one view:

¨ X ? ¢ Cameras and such that X

X ? ¨ ¨

5 5

¢ Baseline between the cameras is non-zero. x l /

e e/ Given an image point in the ﬁrst view, where is the corresponding epipolar line point in the second view? for x

What is the relative position of the cameras? ¢ A point• Ain pointone inimage one image“gener “generates”ates” a a line inin thethe otherother image.image. • This line is known as an epipolar line. ¢ This line is known as an epipolar line, and the geometry which What is the 3D geometry of the scene? • The geometry which gives rise to it is known as epipolar gives risegeometry.to it is known as epipolar geometry.

Images of Planes Epipolar Geometry

Projective transformation between images induced by a plane Epipolar Plane Π

π ¡ ¡ x ¢ : ¡ π

x x/

¡ e e ¢ /

O O x Left epipolar line Right epipolar line x/ :

¡ ¡ ¡ ¨ ¢ ¢ ¢ 5 :

/ The epipolar line is the image of the ray through . C C ¨ £ ¢ The epipole is the point of intersection of the line joining the

¢ All epipolar lines intersect in the epipole. Correspondence Geometry

Given the image of a point in one view, what can we say about its Two View Geometry position in another?

¨ X ? ¢ Cameras and such that X

X ? ¨ ¨

5 5

¢ Baseline between the cameras is non-zero. x l /

e e/ Given an image point in the ﬁrst view, where is the corresponding epipolar line point in the second view? for x

What is the relative position of the cameras? ¢ A point in one image “generates” a line in the other image.

¢ This line is known as an epipolar line, and the geometry which What is the 3D geometry of the scene? gives rise to it is known as epipolar geometry.

Epipolar Geometry

Images of Planes Epipolar geometry isEpipolar a consequenceGeometry of the co-planarity of the camera centres and scene point.

Projective transformation between images induced by a plane Epipolar Plane Π

π ¡ ¡

x : ¡ ¢ π

x x/

¡ e e ¢ /

O O x Left epipolar line Right epipolar line x/ :

¡ ¡ ¡ ¨ ¢ ¢ ¢ 5 : 0 / The epipolarThe cameraline centres,isxthe, x andimageX lie inof athe singlera planey through known as the. C C epipolar plane¨ . £ ¢ The epipole is the point of intersection of the line joining the

¢ All epipolar lines intersect in the epipole. Correspondence Geometry

Given the image of a point in one view, what can we say about its Two View Geometry position in another?

¨ X ? ¢ Cameras and such that X

X ? ¨ ¨

5 5

¢ Baseline between the cameras is non-zero. x l /

e e/ Given an image point in the ﬁrst view, where is the corresponding epipolar line point in the second view? for x

What is the relative position of the cameras? ¢ A point in one image “generates” a line in the other image.

¢ This line is known as an epipolar line, and the geometry which What is the 3D geometry of the scene? gives rise to it is known as epipolar geometry.

Images of Planes Epipolar GeometryEpipolar Geometry

Projective transformation between images induced by a plane Epipolar Plane Π

π ¡ ¡

x : ¡ ¢ π

x x/

¡ e e ¢ /

O O x Left epipolar line Right epipolar line x/ :

¡ ¡ ¡ ¨ ¢ ¢ 0 ¢ 5 : • The epipolar line l is the image of the ray through x. / The epipolar line is the image of the ray through . C C 0 • The epipole¨ e is the point of intersection of the line joining the camera £ ¢ The epipolecentres - theisbaselinethe point- with theof imageintersection plane. of the line joining the

¢ can be computed from the correspondence of four points on camera centres—the• Epipole is the imagebaseline in one camera—with of the centrethe image of the otherplane camera.. • All epipolar lines intersect in the epipole. the plane. ¢ The epipole is also the image in one camera of the centre of the other camera.

¢ All epipolar lines intersect in the epipole. Epipolar pencil Epipolar pencil Homogeneous Notation Interlude ¢ ¢ A line is represented by the homogeneous 3-vector < 6 ¥¤ ¡ ¥§ ¢ ¢ =9> 798

# ¤ § e e / for the line . Only the ratio of the homogeneous line

baseline coordinates is signiﬁcant. © © ¢ ¢ ¢

5 5 ¢ 5 point on line: ¦ or or l

q As the position• As theof positionthe 3D ofpoint the 3D pointvar varies,ies, the theepipolar epipolar planesplanes “rotate” ¢

¡ p £ ¢ about the baseline“rotate” about. This thef baseline.amily of planes is known as an epipolar two points deﬁne a line: • This family of planes is known as an epipolar pencil. pencil. All epipolar lines intersect at the epipole. l • All epipolar lines intersect at the epipole. x ¢ ¡ ¤ ¢ 5 two lines deﬁne a point: m

Epipolar geometry example Matrix notation for vector product ¡ The vector product ¥ 5 can be represented as a matrix multiplication e at e / at

infinity infinity % 4 ¡ 5 ¥ 5 ¥ ¤

where D @

¦¨§ ¦

" % 4

E A ¥ ¦¨§ ¦

¤ F B

¦ ¦

" % 4

¡ ¢ ¥ Epipolar geometry depends only on the relative pose (position and ¤ is a skew-symmetric matrix of rank 2. % 4 % 4

orientation) and internal parameters of the two cameras, i.e. the posi- ¨ ¡ ¥ ¢ ¥ ¥ ¥ ¥ ¥ tion of the camera centres and image planes. It does not depend on is the null-vector of ¤ , since ¤ . structure (3D points external to the camera). Epipolar pencil Homogeneous Notation Interlude ¢ ¢ A line is represented by the homogeneous 3-vector 6 < ¥¤ ¡ ¥§ ¢ ¢ 798 =9>

# e e / for the line ¤ § . Only the ratio of the homogeneous line

baseline coordinates is signiﬁcant. © © ¢ ¢ ¢

5 ¢ 5 5 point on line: ¦ or or l

q As the position of the 3D point varies, the epipolar planes “rotate” ¢

¡ p £ ¢ about the baseline. This family of planes is known as an epipolar two points deﬁne a line:

pencil. All epipolar lines intersect at the epipole. l

x ¢ ¡ ¤ ¢ 5 two lines deﬁne a point: m Epipolar geometry example: parallel cameras Epipolar geometry example Matrix notation for vector product ¡ The vector product ¥ 5 can be represented as a matrix multiplication e at e / at

infinity infinity % 4 ¡ ¥ 5 ¥ 5 ¤

where @ D

¦¨§ ¦

" % 4

A E ¥ ¦¨§ ¦

¤ B F

¦ ¦

" % 4

¡ ¢ ¥ Epipolar geometry depends only on the relative pose (position and ¤ is a skew-symmetric matrix of rank 2. % 4 Epipolar geometry depends only on % 4

orientation) and internal parameters of the two cameras, i.e. the posi- ¨ ¡ ¥ ¢ ¥ ¥ ¥ ¥ ¥ ¤ is the null-vector of ¤ , since . 1.tion theof relativethe camer posea centres (positionand image and orientation)planes. It does andnot depend on structure (3D points external to the camera). 2. internal parameters of the two cameras. Epipolar pencil Homogeneous Notation Interlude ¢ ¢ A line is represented by the homogeneous 3-vector 6 < ¥¤ ¡ ¥§ ¢ ¢ 798 =9>

# e e / for the line ¤ § . Only the ratio of the homogeneous line

baseline coordinates is signiﬁcant. © © ¢ ¢ ¢

5 ¢ 5 5 point on line: ¦ or or l

q As the position of the 3D point varies, the epipolar planes “rotate” ¢

¡ p £ ¢ about the baseline. This family of planes is known as an epipolar two points deﬁne a line:

pencil. All epipolar lines intersect at the epipole. l

infinity infinity % 4 ¡ ¥ 5 ¥ 5 ¤

where @ D

¦¨§ ¦

" % 4

A E ¥ ¦¨§ ¦

¤ B F

¦ ¦

" % 4

¡ ¢ ¥ Epipolar geometry depends only on the relative pose (position and ¤ is a skew-symmetric matrix of rank 2. % 4 Epipolar geometry depends only on % 4

orientation) and internal parameters of the two cameras, i.e. the posi- ¨ ¡ ¥ ¢ ¥ ¥ ¥ ¥ ¥ ¤ is the null-vector of ¤ , since . 1.tion theof relativethe camer posea centres (positionand image and orientation)planes. It does andnot depend on structure (3D points external to the camera). 2. internal parameters of the two cameras. It does not depend on structure (3D points external to the cameras.) Mathematical interlude Homogeneous notation and lines

• A line l1x + l2y + l3 = 0 is represented by the homogeneous 3-vector

T l = l1 l2 l3

Only ratio of homogeneous line coordinates is signiﬁcant. Homogeneous notation and lines

• A line l1x + l2y + l3 = 0 is represented by the homogeneous 3-vector

T l = l1 l2 l3

• Cool properties resulting from this representation 1. Point x on line l ⇐⇒

lT x = xT l = 0 (x is represented by its homogeneous coordinates.) Epipolar pencil Homogeneous Notation Interlude ¢ ¢ A line is represented by the homogeneous 3-vector Homogeneous notation and lines < 6 ¥¤ ¡ ¥§ ¢

• A line ¢ 798 =9> l1x + l2y + l3 = 0

is represented by the homogeneous 3-vector T l = l1 l2 l3

# e e / for the line ¤ § . Only the ratio of the homogeneous line • Cool properties resulting from this representation baseline coordinates is signiﬁcant. 2. Two points deﬁne a line: (p, q represented homogeneously.) © © ¢ ¢ ¢

5 5 ¢ 5 point on line: ¦ or or l = p × q l

q As the position of the 3D point varies, the epipolar planes “rotate” ¢

¡ p £ ¢ about the baseline. This family of planes is known as an epipolar two points deﬁne a line: pencil. All epipolar lines intersect at the epipole. l

x ¢ ¡ ¤ ¢ 5 two lines deﬁne a point: m

Epipolar geometry example Matrix notation for vector product ¡ The vector product ¥ 5 can be represented as a matrix multiplication e at e / at infinity infinity % 4 ¡ ¥ 5 ¥ 5 ¤ where @ D

¦¨§ ¦

" % 4

A E ¥ ¦¨§ ¦

¤ B F

¦ ¦

" % 4

¡ ¢ ¥ Epipolar geometry depends only on the relative pose (position and ¤ is a skew-symmetric matrix of rank 2. % 4 % 4

orientation) and internal parameters of the two cameras, i.e. the posi- ¨ ¡ ¥ ¥ ¥ ¥ ¢ ¥ ¥ tion of the camera centres and image planes. It does not depend on is the null-vector of ¤ , since ¤ . structure (3D points external to the camera). Epipolar pencil Homogeneous Notation Interlude ¢ ¢ A line is represented by the homogeneous 3-vector 6 <

¥¤ ¡ ¥§ ¢ Homogeneous notation and lines ¢ 798 =9> • A line l1x + l2y + l3 = 0

# / ¤ § is represented by the homogeneous 3-vector e e for the line . Only the ratio of the homogeneous line T baseline coordinates is signiﬁcant. l = l1 l2 l3 © © ¢ ¢ ¢

5 5 ¢ 5 point on line: ¦ or or • Cool properties resulting from thisl representation

q As the position of the 3D point varies, the epipolar planes “rotate” 3. Two lines deﬁne a point: (lines l, m represented homogeneously.) ¢

¡ p £ ¢ about the baseline. This family of planes is known as an epipolar two points deﬁne a line: x = l × m pencil. All epipolar lines intersect at the epipole. l

x ¢ ¡ ¤ ¢ 5 two lines deﬁne a point: m

Epipolar geometry example Matrix notation for vector product ¡ The vector product ¥ 5 can be represented as a matrix multiplication e at e / at infinity infinity % 4 ¡ ¥ 5 ¥ 5 ¤ where @ D

¦¨§ ¦

" % 4

A E ¥ ¦ ¦¨§

¤ B F

¦ ¦

" % 4

¡ ¢ ¥ Epipolar geometry depends only on the relative pose (position and ¤ is a skew-symmetric matrix of rank 2. % 4 % 4

• The cross-product v × x can be written as a matrix multiplication

v × x = [v]× x where   0 −vz vy [v]× =  vz 0 −vx −vy vx 0

• [v]× is a 3 × 3 skew-symmetric matrix of rank 2.

• v is the null vector of [v]× as

v × v = [v]× v = 0 The Essential Matrix the set-up and derivation Normalized coordinates

• Camera matrix decomposed as M = K R t and let

x = λMX

• Assume calibration matrix K is known. - Then if xˆ = K−1x =⇒ xˆ = λ R t X

- xˆ is an image point expressed in normalized coordinates. =⇒ xˆ has been viewed by camera R t with calibration matrix I.

• The normalized camera matrix

K−1M = R t Pair of Normalized Camera Matrices

• For the next set of slides assume we have a pair of normalized camera matrices:

M = I 0 M 0 = R t Correspondence Geometry

Given the image of a point Thein one Essentialview, what can Matrixwe say -about Derivationits Two View Geometry position in another?

¨ X ? ¢ Cameras and such that X

X ? ¨ ¨

5 5

¢ Baseline between the cameras is non-zero. x l /

e e/ Given an image point in the ﬁrst view, where is the corresponding epipolar line point in the second view? for x

What is the relative position of the cameras? ¢ A point in one imageX“gener ates” a line in the other image. • Know x = λ I 0 thus the viewing ray of x is ¢ This line is known as1 an epipolar line, and the geometry which What is the 3D geometry of the scene? gives rise to it is known as epipolargeometrλ−1x y. X(λ) = 1 • The ray X(λ) gets projected into the other image as Images of Planes Epipolar Geometry λ−1x λ0 R t = λ0λ−1Rx + λ0t 1 X Projective transformation between images induced by a plane Epipolar Plane Π This is the right epipolar line.

π ¡ ¡ x : ¡ ¢ π

x x/

¡ e e ¢ /

O O x Left epipolar line Right epipolar line x/ :

¡ ¡ ¡ ¨ ¢ ¢ ¢ 5 :

/ The epipolar line is the image of the ray through . C C ¨ £ ¢ The epipole is the point of intersection of the line joining the

¢ All epipolar lines intersect in the epipole. Correspondence Geometry

Given the image of a point Thein one Essentialview, what can Matrixwe say -about Derivationits Two View Geometry position in another?

¨ X ? ¢ Cameras and such that X

X ? ¨ ¨

5 5

¢ Baseline between the cameras is non-zero. x l /

e e/ Given an image point in the ﬁrst view, where is the corresponding epipolar line point in the second view? for x

What is the relative position of the cameras? • The¢ A pointrightin epipolarone image line“gener: ates” a line in the other image.

¢ This line is known as an 0epipolar −1 line,and the geometry which What is the 3D geometry of the scene? λ λ Rx + t gives rise to it is known as epipolar geometry. • Two points on the right epipolar line are 1. t (when λ = ±∞ and corresponds to right epipole) and Images of Planes 2. R x + t (when λ Epipolar= 1) Geometry • Thus: X Projective transformation between images induced by a plane l0 = t × (R xEpipolar+ Planet) Π = t × R x + t × t = t × R x

π ¡ ¡ x : ¡ ¢ π

x x/

¡ e e ¢ /

O O x Left epipolar line Right epipolar line x/ :

¡ ¡ ¡ ¨ ¢ ¢ ¢ 5 :

/ The epipolar line is the image of the ray through . C C ¨ £ ¢ The epipole is the point of intersection of the line joining the

¢ All epipolar lines intersect in the epipole. The Essential Matrix - Derivation

• Introduce the skew-symmetric matrix   0 −tz ty [ t ]× =  tz 0 −tx −ty tx 0

• Then 0 l = t × R x = [ t ]×R x = E x where

E = [t]× R is called the Essential Matrix.

• E maps a point in left image to a line in right image. Correspondence Geometry The Essential Matrix - Derivation Given the image of a point in one view, what can we say about its Two View Geometry position in another?

¨ X ? ¢ Cameras and such that X

X ? ¨ ¨

5 5

¢ Baseline between the cameras is non-zero. x l /

e e/ Given an image point in the ﬁrst view, where is the corresponding epipolar line point in the second view? for x

What is the relative position of the cameras? ¢ A point in one image “generates” a line in the other image. • Have ¢ This line is known as an epipolar0 line, and the geometry which What is the 3D geometry of the scene? l = E x gives rise to it is known as epipolar geometry. • Epipolar constraint that x0 lies on l’ can be written as

Images of Planes Epipolarx0T l0 =Geometryx0T Ex = 0

Projective transformation between images induced by a plane Epipolar Plane Π

π ¡ ¡ x ¢ : ¡ π

x x/

¡ e e ¢ /

O O x Left epipolar line Right epipolar line x/ :

¡ ¡ ¡ ¨ ¢ ¢ ¢ 5 :

/ The epipolar line is the image of the ray through . C C ¨ £ ¢ The epipole is the point of intersection of the line joining the

¢ All epipolar lines intersect in the epipole. Properties of the Essential Matrix Properties of E-matrix

• E-matrix, E = [t]× R, is a 3 × 3 matrix.

• E-matrix has 5 degrees of freedom 1. +3 for rotation matrix R 2. +3 for translation t 3. -1 for the scale ambiguity.

• E-matrix’s singular values are β, β and 0.

• As scale of E does not matter, assume β = 1 and

E = Udiag(1, 1, 0)V T Properties of E-matrix

• E-matrix, E = [t]× R, is a 3 × 3 matrix.

• E-matrix has 5 degrees of freedom 1. +3 for rotation matrix R 2. +3 for translation t 3. -1 for the scale ambiguity.

• E-matrix’s singular values are β, β and 0.

• As scale of E does not matter, assume β = 1 and

E = Udiag(1, 1, 0)V T Properties of E-matrix

For completeness: • The epipolar line corresponding to x0 is given by

l = ET x0

• The epipole e0 by deﬁnition has

0 = e0T l0 = e0T E x

• This relation is true for all x =⇒ e0T E = 0

• Thus e0 is the left null-space of E.

• Similarly E e = 0 that is e is the right null-space of E. Properties of E-matrix

For completeness: • The epipolar line corresponding to x0 is given by

l = ET x0

• The epipole e0 by deﬁnition has

0 = e0T l0 = e0T E x

• This relation is true for all x =⇒ e0T E = 0

• Thus e0 is the left null-space of E.

• Similarly E e = 0 that is e is the right null-space of E. Properties of E-matrix

For completeness: • The epipolar line corresponding to x0 is given by

l = ET x0

• The epipole e0 by deﬁnition has

0 = e0T l0 = e0T E x

• This relation is true for all x =⇒ e0T E = 0

• Thus e0 is the left null-space of E.

• Similarly E e = 0 that is e is the right null-space of E. Estimating the Essential Matrix E-matrix estimation (8-point method)

• The epipolar constraint can be expressed using the essential matrix with elements eij:     e11 e12 e13 x 0 0 (x y 1) e21 e22 e23 y = 0 e31 e32 e33 1

• We can write this out as αT e = 0, where

α = [x0x, x0y, y0x, y0y, x0, y0, x, y, 1]T ,

T e = [e11, e12, e13, e21, e22, e23, e31, e32, e33]

• Use ≥ 8 point matches and ﬁnd least square solution with constraint eT e = 1 ⇒ an eigenvalue problem. Extracting pose of right camera from the Essential Matrix Extraction of R and t from E

• E = [t]× R is the multiplication of a skew-symmetric matrix and a rotation matrix.

• =⇒ factorize E = SR where S is skew-symmetric and R is a rotation.

• Use these two matrices 0 −1 0  0 1 0 W = 1 0 0 and Z = −1 0 0 0 0 1 0 0 0

• W is a rotation and Z is skew-symmetric.

• And ZW = diag(1, 1, 0),ZW T = −diag(1, 1, 0) Extraction of R and t from E

• E = [t]× R is the multiplication of a skew-symmetric matrix and a rotation matrix.

• =⇒ factorize E = SR where S is skew-symmetric and R is a rotation.

• Use these two matrices 0 −1 0  0 1 0 W = 1 0 0 and Z = −1 0 0 0 0 1 0 0 0

• W is a rotation and Z is skew-symmetric.

• And ZW = diag(1, 1, 0),ZW T = −diag(1, 1, 0) Extraction of R and t from E

• E = [t]× R is the multiplication of a skew-symmetric matrix and a rotation matrix.

• =⇒ factorize E = SR where S is skew-symmetric and R is a rotation.

• Use these two matrices 0 −1 0  0 1 0 W = 1 0 0 and Z = −1 0 0 0 0 1 0 0 0

• W is a rotation and Z is skew-symmetric.

• And ZW = diag(1, 1, 0),ZW T = −diag(1, 1, 0) Extraction of R and t from E

• Remember have deﬁned E s.t. (also assuming det(UV T ) = 1)

E = Udiag(1, 1, 0) V T

• Two factorisation are:

1. E = S1R1 where

T T T S1 = −UZU ,R1 = UW V

2. E = S2R2 where

T T S2 = UZU ,R2 = UWV

• To show these are valid solutions must show for i = 1, 2

1. Si Ri = E T 2. Ri is a rotation that is Ri Ri = I and det(Ri) = 1. T 3. Si is skew-symmetric that is −Si = Si Extraction of R and t from E

• Remember have deﬁned E s.t. (also assuming det(UV T ) = 1)

E = Udiag(1, 1, 0) V T

• Two factorisation are:

1. E = S1R1 where

T T T S1 = −UZU ,R1 = UW V

2. E = S2R2 where

T T S2 = UZU ,R2 = UWV

• To show these are valid solutions must show for i = 1, 2

1. Si Ri = E T 2. Ri is a rotation that is Ri Ri = I and det(Ri) = 1. T 3. Si is skew-symmetric that is −Si = Si Extraction of R and t from E

• Remember have deﬁned E s.t. (also assuming det(UV T ) = 1)

E = Udiag(1, 1, 0) V T

• Two factorisation are:

1. E = S1R1 where

T T T S1 = −UZU ,R1 = UW V

2. E = S2R2 where

T T S2 = UZU ,R2 = UWV

• To show these are valid solutions must show for i = 1, 2

1. Si Ri = E T 2. Ri is a rotation that is Ri Ri = I and det(Ri) = 1. T 3. Si is skew-symmetric that is −Si = Si Extraction of R and t from E

• Have an expression for S and R.

• Need to ﬁnd t s.t.

[t]× = S

• t is in the nullspace of S as

St = [t]× t = t × t = 0

T • Remember S1 = −UZU

=⇒ third column of U is a valid choice for t.(Slly for S2.)

• Why? Scale ambiguity in the reconstruction

• If t is a null-vector of S then σt is also a null-vector of S.

• For any σ 6= 0 have

[σt]× R = σ [t]× R = σ E

• σE is also a valid essential matrix for the set-up.

• A diﬀerent σ corresponds to re-scaling the solution

σX X 0 σY  Y  x ' R σt   ' R t   σZ  Z  1 1

• Projection looks the same regardless of scale. Recover projection matrices

Miniatur wunderland in Hamburg Sign of scaling is important

• If t is a null-vector of S then σt is also a null-vector of S.

• The sign of σ is important!

• Sign determines whether points are in front of the camera or not in the ﬁnal reconstruction.

• If we constrain ktk = 1 then we have to check if σ = ±1.

0 • If u3 is the third column of U we get the four solutions for M :

T T T T T T UWV u3 , UW V u3 , UWV −u3 , UW V −u3 Scale ambiguity when estimating R, t from E

(1)

3D point (2)

z−axis z−axis Image a Image b Image b Image a

Camera a Camera b Camera b Camera a

Correct Solution (!) 3D point 3D point

(3) (4)

Image a Image a

Camera a Camera b Camera b Camera a

Image b Image b

3D point Only one solution has reconstructed points in-front of both cameras. Summary of the 8-point algorithm for calibrated cameras

1. Extract at least 8 point correspondences. 2. Normalize the coordinates using K−1 and K0−1 where K and K0 are the internal camera matrices of the cameras. 3. Form the constraint matrix Q and solve min kQek2 subject to kek2 = 1 e 4. Form the matrix E˜ from e and compute its SVD: E˜ = USV T Then set ( Udiag(1, 1, 0)V T if det(UV T ) = 1 E = Udiag(1, 1, 0)(−V )T if det(UV T ) = −1 This ensures the singular values of E are 1, 1, 0. 5. Compute the right camera matrix M 0 from E. 6. Compute the scene points using triangulation.