Lecture 3 - Two view geometry & the Essential Matrix DD2429 September 10, 2018 Review of the general perspective projection camera matrix Note: Most slides are copied/adapted from Multiple View Geometry Tutorial Part 1, CVPR 1999, by R. Hartley and A. Zisserman Imaging Geometry Multiple View Geometry Perspective projection Imaging Geometry - Perspective projection Y X ¥ ¥ X §©¨ Richard Hartley and Andrew Zisserman X y ¤ x ¦ ¦ Y x O Z p Z ¤ where Z . image plane CVPR June 1999 This can be written as a•linearWorld andmapping camera axesbetw aligned,een imagehomogeneous plane at (0; 0; f)coordinatesT . (the equation is only up to a scale• Perspectivefactor): projection of point X can be written as 0x1 ¥ 0 X1 ¥ §¨ @yA =Xλ @Y A ¦ f Y Z ¦ where λ = f=Z. Z £ ¡ where a ¢ projection matrix represents a map from 3D to 2D. Image Coordinate System Part I: Single and Two View Geometry Internal camera parameters ycam ! cam The main points covered in this part are: " # # # ! y p cam 0 x £ ¡ ¢ cam y A perspective (central) projection camera is represented by a matrix. " where the units of $ %'& / 4 ,.- , 3 (*) 2 1 The most general perspective transformation transformation between two planes + + 0 are . x x (a world plane and the image plane, or two image planes induced by a world 0 plane) is a plane projective transformation. This can be computed from the cor- respondence of four (or more) points. 6 < @ D 6 6 < < C ! The epipolar geometry between two views is represented by the fundamental : cam cam G 798 =9> 798 798 =9> =9> A9B E9F 5 #;: C # # # ! " matrix. This can be computed from the correspondence of seven (or more) ? cam cam ? ? points. : ? ? " C C " where $ . Imaging Geometry Multiple View Geometry Perspective projection Imaging Geometry - Perspective projection Y X ¥ ¥ X §©¨ Richard Hartley and Andrew Zisserman X y ¤ x ¦ ¦ Y x O Z p Z ¤ where Z . image plane CVPR June 1999 • World and camera axes aligned, image plane at (0; 0; f)T . This can be written as a linear mapping between homogeneous coordinates (the equation is only up to a scale• Perspectivefactor): projection written as a linear mapping between homogeneous coordinates (λ is the scale factor): ¥ 0 1 ¥ 0 1 0 1 X §¨ x 1 0X 0 0 BY C ¦ @ yA = λ @0 1Y 0 0A B C ¦ Z f 0 0 1 0 @ A Z 1 £ ¡ where a ¢ projection matrix represents a map from 3D to 2D. Image Coordinate System Part I: Single and Two View Geometry Internal camera parameters ycam ! cam The main points covered in this part are: " # # # ! y p cam 0 x £ ¡ ¢ cam y A perspective (central) projection camera is represented by a matrix. " where the units of $ %'& / 4 ,.- , 3 (*) 2 1 The most general perspective transformation transformation between two planes + + 0 are . x x (a world plane and the image plane, or two image planes induced by a world 0 plane) is a plane projective transformation. This can be computed from the cor- respondence of four (or more) points. 6 < @ D 6 6 < < C ! The epipolar geometry between two views is represented by the fundamental : cam cam G 798 =9> 798 798 =9> =9> A9B E9F 5 #;: C # # # ! " matrix. This can be computed from the correspondence of seven (or more) ? cam cam ? ? points. : ? ? " C C " where $ . Imaging Geometry Multiple View Geometry Perspective projection Y X ¥ ¥ X §©¨ Richard Hartley and Andrew Zisserman X y ¤ x ¦ ¦ Y x O Z p Z ¤ where Z . image plane CVPR June 1999 This can be written as a linear mapping between homogeneous coordinates (the equation is only up to a scale factor): ¥ ¥ §¨ X ¦ Y ¦ Z £ ¡ where a ¢ projection matrix represents a map from 3D to 2D. Image Coordinate System Image Coordinate System Part I: Single and Two View Geometry Internal camera parameters • Internal camera parameters ycam ! cam kx xcam = x − x0 The main points covered in this part are: " # # # ! y p cam 0 x ky ycam = y − y0 £ ¡ ¢ cam y A perspective (central) projection camera is represented by a matrix. " where the units of $ 4 %'& / 3 ,.- , where the units of k ; k are (*) 2 x y 1 The most general perspective transformation transformation between two planes + + 0 are . x x (a world plane and the image plane, or two image planes induced by a world 0 pixels/length. plane) is a plane projective transformation. This can be computed from the cor- respondence of four (or more) points. D 6 @ < 6 < 6 < C ! : 0 1 0 1 0 1 0 1 The epipolar geometry between two views is represented by the fundamental cam x cam αx 0 x0 xcam xcam G 798 =9> 798 =9> 798 =9> E9F A9B 5 C # #;: # # ! 1 " matrix. This can be computed from the correspondence of seven (or more) ? cam x = y cam= 0 αy y0 ycam = K ycam ? @ A ? @ A @ A @ A : f points. 1 0 0 1 f f ? ? " C C " where $ . where αx = f kx; α = fky. Camera Calibration • K is a 3 × 3 upper-triangular matrix called the camera calibration matrix 0 1 αx 0 x0 K = @ 0 αy y0A 0 0 1 • There are four parameters: 1. The scaling in the image x and y directions - αx and αy. 2. The principal point (x0; y0) - point where the optic axis intersects the image plane. • The aspect ratio is αy/αx. Concatenating the three matrices, 6 Camera Calibration Matrix < @ D 6 < : X ¥ § ¦ 7 = % 4 : § G G ¦ 7 7 = = A E 5 #©: Y G © ¡ : ¨ 798 8 > =9> is a upper triangular matrix, called the camera calibration B F : Z : matrix: @ D C ! G A9B E9F C # ! " ¡ : which defines the projection matrix from Euclidean 3-space to an image as ¢ There are four parameters: # C C (i) The scaling in the image and directions, and " . ¤ £ # ! ! (ii) The principal point $ , which is the point where the optic axis intersects the image plane. / © © £ ¤ § ¦ ¢ C C " The aspect ratio is . Note, the camera centre is at X$ Y$ Z . In the following it is often only the ¡ form of that is important, rather than its decomposition. World Coordinate System A Projective Camera World Coordinate System External camera parameters The camera model for perspective projection is a linear map between Y homogeneous point coordinates $ cam $ ' / X %%%& Z 6 6 < < * . - + %& () 01 , !#" Y Xcam X ¥ § O ¦ Z 7 7 = = cam cam Z 7 Y = 7 Y = R, t " cam © X : ¨ 798 =9> 798 =9> cam Zcam Z O : : Y Image Point Scene Point X Euclidean transf•orExternalmation betwcameraeen parameters:world and camera coordinates ¦ ¡ ¢ ¢ is a rotation matrix The camera centre is the null-vector of © % 4 £ ¤ : 0 1 0 1 2 : § ¡ ¢ Xcam X is a translation vector e.g. if then the centre is $ $ $ . Ycam R t Y B C B C ¢ B C = T B C has 11 degrees of freedom (essential parameters). @Zcam A 0 1 @Z A 1 1 ¢ has rank 3. • Euclidean transformation between world and camera coordinates - R is a 3 × 3 rotation matrix - t is 3 × 1 translation vector Putting everything together 0X1 0x1 01 0 0 01 R t Y x = y = λK 0 1 0 0 B C = λK R t X @ A @ A 0T 1 BZ C 1 0 0 1 0 @ A 1 • Above defines the 3 × 4 projection matrix from Euclidean 3-space to an image as x = λMX where M = K R t = KR IRT t • Note the camera centre is at (X; Y; Z) = −RT t. Putting everything together 0X1 0x1 01 0 0 01 R t Y x = y = λK 0 1 0 0 B C = λK R t X @ A @ A 0T 1 BZ C 1 0 0 1 0 @ A 1 • Above defines the 3 × 4 projection matrix from Euclidean 3-space to an image as x = λMX where M = K R t = KR IRT t • Note the camera centre is at (X; Y; Z) = −RT t. A Projective Camera The camera model for perspective projection is a linear map between homogeneous point coordinates 0X1 0x1 Y y = λM B C @ A BZ C 1 @ A 1 where M has size 3 × 4 and projects a scene point to an image point. • The camera centre is the null-vector of M • M has 11 degrees of freedom (essential parameters). • M has rank 3. What does calibration give? • K provides the transformation between an image point and a ray in Euclidean 3-space. • Once K is known the camera is termed calibrated. • A calibrated camera is a direction sensor, able to measure the direction of rays - like a 2D protractor. Calibrating a camera correspondences between the 3D world and the camera view Camera Calibration Problem Statement • Given n correspondences xi $ Xi where each Xi is a scene point and xi is its image: • Compute M = K R t such that xi ≈ λMXi • Solution: The algorithm for camera calibration has two parts: 1. Compute the M from the set of point correspondences. 2. Decompose M into K; R and t using QR decomposition. Camera Calibration Problem Statement • Given n correspondences xi $ Xi where each Xi is a scene point and xi is its image: • Compute M = K R t such that xi ≈ λMXi • Solution: The algorithm for camera calibration has two parts: 1. Compute the M from the set of point correspondences. 2. Decompose M into K; R and t using QR decomposition. Algorithm step 1: Estimate the matrix M • The projection equation in homogeneous coordinates: 0X1 0x1 0m m m m 1 11 12 13 14 Y y ' m m m m B C @ A @ 21 22 23 24A BZ C 1 m m m m @ A 31 32 33 34 1 • Express the projection in Euclidean coordinates: m X + m Y + m Z + m x = 11 12 13 14 m31X + m32Y + m33Z + m34 m X + m Y + m Z + m y = 21 22 23 24 m31X + m32Y + m33Z + m34 • Multiply with the denominators and rearrange: m11X + m12Y + m13Z + m14 − m31xX − m32xY − m33xZ − m34x = 0 m21X + m22Y + m23Z + m24 − m31yX − m32yY − m33yZ − m34y = 0 • Two constraints per match =) six matches needed total.