Photogrammetry & Robotics Lab A Camera Pair
Relative Orientation, Fundamental and Essential Matrix
Cyrill Stachniss
The slides have been created by Cyrill Stachniss. 1 2
Camera Pair § A stereo camera § One camera that moves Orientation Parameters Camera pair = two configurations for the Camera Pair and from which images have been taken Relative Orientation
3 4 Orientation Orientation § The orientation of the camera pair can § The orientation of the camera pair can be described using independent be described using independent orientations for each camera orientations for each camera
How many parameters are needed? How many parameters are needed? § Calibrated cameras: ? parameters § Calibrated cameras: 12 parameters (angle preserving mapping) (angle preserving mapping) § Uncalibrated cameras: ? parameters § Uncalibrated cameras: 22 parameters (straight-line preserving mapping) (straight-line preserving mapping)
5 6
Can We Estimate the Which Parameters Camera Motion without Can We Obtain and Which Not? Knowing the Scene?
7 8 Cameras Measure Directions What We Can Compute § We cannot obtain the (global) § The rotation of the second camera translation and rotation (if the w.r.t. the first one (3 parameters) cameras maintain their relative § The direction of the line connecting transformation) as well as the scale the to centers of projection (2 params) § We do not know their distance (the length of )
Image courtesy: Förstner & Wrobel 9 Image courtesy: Förstner & Wrobel 10
For Calibrated Cameras Photogrammetric Model § We need 2x6=12 parameters for two § Given two cameras images, we can calibrated cameras for the orientation reconstruct the object only up to a § With a calibrated camera, we obtain similarity transform an angle-preserving model of the § Called a photogrammetric model object § The orientation of the § Without additional information, we can photogrammetric model is called only obtain 12-7 = 5 parameters the absolute orientation (not 7=translation, rotation, scale) § For obtaining the absolute orientation, we need at least 3 points in 3D first camera distance between cameras (to estimate the 7 parameters) 11 12 What Is Needed for Computing For Uncalibrated Cameras an 3D Model of a Scene? § Straight-line preserving but not angle preserving § Object can be reconstructed up to a straight-line preserving mapping § Projective transform (15 parameters) § Thus, for uncalibrated cameras, we can only obtain 22-15=7 parameters given two images § We need at least 5 points in 3D (15 coordinates) for the absolute o. 13 14
Relative Orientation Summary
Cameras #params #params #params #params min #P /img /img pair for RO for AO
Coplanarity Constraint calibrated 6 12 5 7 3 for Straight-Line Preserving not calibrated 11 22 7 15 5 (Uncalibrated) Cameras to Obtain the RO = relative orientation Fundamental Matrix AO = absolute orientation min #P = min. number of control points
15 16 Which Parameters Can We Coplanarity Constraint Compute Without Additional § Consider perfect orientation and the Information About the Scene? intersection of two corresponding rays We start with a perfect orientation and § The rays lie within one plane in 3D the intersection of two corresponding rays
17 18
Coplanarity Constraint for Scalar Triple Product Uncalibrated Cameras § The operator is the triple product Coplanarity can be expressed by § Dot product of one of the vectors with the cross product of the other two
and refer to the same point in space § It is the volume of the parallelepiped of three vectors § means that the vectors lie in one plane
19 Image courtesy: Wikipedia (Niabot) 20 Coplanarity Constraint for Coplanarity Constraint for Uncalibrated Cameras Uncalibrated Cameras Coplanarity can be expressed by § The directions of the vectors and can be derived from the image coordinates
§ with the projection matrices
Reminder: 21 22
Directions to a Point Base Vector § The normalized directions of the § The base vector directly results vectors and are from the coordinates of the projection
image coord. centers § as the normalized projection world coord. § provides the direction to from the center of projection to the point in 3D § Analogous:
23 24 Coplanarity Constraint Derivation § Using the previous relations, § Why is this correct? the coplanarity constraint
§ can be rewritten as
skew-symmetric matrix 25 26
Derivation Coplanarity Constraint § Why is this correct? § By combining and § we obtain
§ Results from the cross product as
§ with being a skew-symmetric matrix 27 28 Coplanarity Constraint Fundamental Matrix § By combining § The matrix is the fundamental and matrix (for uncalibrated cameras): § we obtain
§ It allow for expressing the coplanarity constraint by
29 30
Fundamental Matrix Fundamental Matrix From the Camera Projection Matrices § The fundamental matrix is the matrix that fulfills the equation § If the projection matrices are given, we can derive the fundamental matrix? for corresponding points
§ The fundamental matrix contains the
all the available information about the relative orientation of two § Let the projection matrices be images from uncalibrated cameras partitioned into a left 3×3 matrix and a 3-vector as .
31 32 Fundamental Matrix From the Fundamental Matrix From the Camera Projection Matrices Camera Projection Matrices § We have § We have
§ and can recover the projection center § and § and thus can compute the F
§ so that the base line is given by § with
33 34
Alternative Definition
§ In the context of many images, we will call that fundamental matrix which yields the constraint § Thus in our case, we have . Essential Matrix § Our definition of is not the same as in the book by Hartley and Zisserman (for Calibrated Cameras) § The definition in Hartley and Zisserman is based on , i.e. § The transposition needs to be taken into account when comparing expressions
35 36 Using Calibrated Cameras Coplanarity Constraint § Most photogrammetric systems rely § For calibrated cameras the coplanarity on calibrated cameras constraint can be simplified § Calibrated cameras simplify the § Based on the calibration matrices, we orientation problem obtain the directions as § Often, we assume that both cameras have the same calibration matrix direction in coordinates § Assumption here: no distortions or camera frame in the image other imaging errors § This relation results from
37 38
Coplanarity Constraint Coplanarity Constraint § Exploiting the fundamental matrix § Exploiting the fundamental matrix
39 40 Coplanarity Constraint Essential Matrix § Exploiting the fundamental matrix § From to the essential matrix
same form as the fundamental essential matrix matrix but for calibrated cameras (DE: Essentielle Matrix) 41 42
Essential Matrix Essential Matrix § We derived a specialization of the § The essential matrix as five degrees fundamental matrix of freedom § For the calibrated cameras, it is called § There are five parameters that the essential matrix determine the relative orientation of the image pair for calibrated cameras § There are 4=9-5 constraints to its 9 elements (3 by 3 matrix) § We can write the coplanarity constraint for calibrated cameras as § The essential matrix is homogenous and singular
43 44 Five Parameters – How? § Five parameters that determine the relative orientation of the image pair Popular Parameterizations for the Relative Orientation How to parameterize the essential matrix?
45 46
The Popular Parameterizations The Popular Parameterizations § Five parameters that determine the 1. General parameterization of relative orientation of the image pair dependent images § Three popular parameterizations 2. Photogrammetric parameterization of dependent images 3. Parameterization with independent images
Image courtesy: Förstner 47 Image courtesy: Förstner 48 General Parameterization of Photogrammetric parametrizat. Dependent Images of Dependent Images The general parameterization of Photogrammetric parameterization of dependent images uses a dependent images uses
§ normalized direction vector b § two components BY and BZ of the § rotation matrix R base direction (BX=const) § a rotation matrix R DE: Allgemeine Parametrisierung des Folgebildanschluss DE: Klassich-photogrammetrische Parametrisierung des Folgebildanschluss
49 50
Parameterization with Independent Images The parameterization with independent images uses Parameterization of § a rotation matrix Dependent Images (DE: Parametrisierung des Folgebildanschluss ) § a rotation matrix § a fixed basis of constant length
DE: Parametrisierung mit Bilddrehungen
51 52 For the Parameterizations of General Parameterization of Dependent Images Dependent Images § The reference frame is the frame of § The orientation of the second camera the first camera is and we obtain from the § Describe the second camera relative to coplanarity constraint the first one § Rotation mat. of the first cam is
§ The rotation of the R.O. is then § 6 parameters + 1 constraint
53 54
General Parameterization of Summary Dependent Images § Parameters of image pairs § The resulting 5 degree of freedom are § Relative orientation § Fundamental matrix F § Coplanarity constraint § Essential matrix E (F for the calibrated camera pair) § Coplanarity constraint § Parameterization of the relative orientation
55 56 Literature Slide Information
§ The slides have been created by Cyrill Stachniss as part of the § Förstner, Wrobel: Photogrammetric photogrammetry and robotics courses. Computer Vision, Ch. 12.2 § I tried to acknowledge all people from whom I used images or videos. In case I made a mistake or missed someone, please let me know. § The photogrammetry material heavily relies on the very well written lecture notes by Wolfgang Förstner and the Photogrammetric Computer Vision book by Förstner & Wrobel. § Parts of the robotics material stems from the great Probabilistic Robotics book by Thrun, Burgard and Fox. § If you are a university lecturer, feel free to use the course material. If you adapt the course material, please make sure that you keep the acknowledgements to others and please acknowledge me as well. To satisfy my own curiosity, please send me email notice if you use my slides.
57 Cyrill Stachniss, [email protected], 2014 58