Lecture 4 Single View Metrology Professor Silvio Savarese Computational Vision and Geometry Lab Silvio Savarese Lecture 4 - 16-Jan-15 Lecture 4 Single View Metrology • Review calibration and 2D transformations • Vanishing points and lines • Estimating geometry from a single image • Extensions Reading: [HZ] Chapter 2 “Projective Geometry and Transformation in 2D” [HZ] Chapter 3 “Projective Geometry and Transformation in 3D” [HZ] Chapter 8 “More Single View Geometry” [Hoiem & Savarese] Chapter 2 Silvio Savarese Lecture 4 - 16-Jan-15 The calibration problem was discussed in details during lecture 3. Calibration Problem jC M = K[R T] &ui # cot u P → M P → p = &α −α θ o # i i i $ ! $ β ! K = $0 vo ! vi sinθ % " $0 0 1 ! % " World ref. system In pixels Calibration Problem jC M = K[R T] &u # P M P → p = i 11 unknown i → i i $ ! Need at least 6 correspondences %vi " World ref. system In pixels Once the camera is calibrated (intrinsics are known) and the transformation from the world reference system to the camera reference Once the camera is calibrated... system (which accounts for the extrinsics) is also known, can we estimate Pinhole perspective projection the location of a point P in 3D from its observation p? The general answer to this question is no. This is because, given the observation p, even when the camera intrinsics and extrinsic are known, the only thing we can say is that the point P is located somewhere along the line defined by C and p. Line of This line is called the line of sight. sight P p Ow C M = K[R T] -Internal parameters K are known -R, T are known – but these can only relate C to the calibration rig Can I estimate P from the measurement p from a single image? No - in general ☹ (P can be anywhere along the line defined by C and p) The actual location of P along this line is unknown and cannot be Recovering structure from a single view determined from the observation p alone. Pinhole perspective projection Line of sight P p Ow C unknown known Known/ Partially known/ unknown In the remainder of this lecture, we will introduce tools and techniques for Recovering structure from a single view inferring the geometry of the camera and the 3D environment from a just one image of such environment. http://www.robots.ox.ac.uk/~vgg/projects/SingleView/models/hut/hutme.wrl Before we go into the details of that, let me recap some of the concepts you have already explored when we talked about transformations. Transformation in 2D There are four important transformations in 2D. All these transformations will be described in homogenous coordinates. -Isometries -Similarities -Affinity -Projective The first one is called isometric transformation which is in general the concatenation of a rotation and translation transformation and expressed by the matrix H and Eq. 4. This transformation preserves the distance Transformation in 2D e between any pair of points and has 3 degrees of freedom (2 for translation, 1 for rotation). This group captures the motion of a rigid object. &x'# &x# &x# &R t# $y'! $y! H $y! [Eq. 4] Isometries: $ ! = $ !$ ! = e $ ! % 0 1" [Euclidean] %$1 "! %$1"! %$1"! - Preserve distance (areas) - 3 DOF - Regulate motion of rigid object The second group of transformations is called similarity transformation. The transformation is a concatenation of a translation, rotation and scale Transformation in 2D translation transformations. It preserves the ratio of lengths between any two line segments before and after transformation. It also preserves the angle between any intersecting lines. The group has 4 degrees of freedom (2 translation, 1 rotation, 1 scale) ! x' $ ! x $ ! x $ # & ! SR t $# & # & Similarities: # y' & = # &# y & = Hs # y & # & "# 0 1 %&# & # & " 1 % " 1 % " 1 % ! s 0 $ [Eq. 5] S= # & " 0 s % - Preserve - ratio of lengths - angles -4 DOF The third transformation group is the affine transformation. This group can be interpreted as a series of translation, rotation and anisotropic scaling Transformation in 2D transformations. The core of the affine transformation is characterized by the matrix A which includes a rotation by phi, an anisotropic scaling by [sx 0; 0 sy], an inverse rotation by –phi and an arbitrary rotation by theta (see Eq. 7). This transformation has also a mathematical interpretation: Any &x'# &x# &x# positive definite 2x2 matrix can be decomposed into two orthogonal $ ! &A t#$ ! $ ! matrices and a diagonal matrix by Singular Value Decomposition (this is Affinities: y' = y = Ha y $ ! $ !$ ! $ ! [Eq. 6] by assuming positive scaling; for negative scaling, the transformation may % 0 1" T T T %$1 "! %$1"! %$1"! not be unique.) Thus we can express A = UDV = (UV ) (VDV ) = R(theta) R(-phi) D R(phi) T T &a11 a12 # &sx 0 # by replacing UV = R(theta), V = R(phi) , and where D is a diagonal A = = R(θ )⋅R(−φ)⋅D⋅R(φ) D = $a a ! $ ! matrix with the singular values. Note that transpose of a rotation matrix is % 21 22 " 0 sy [Eq. 7] % " the inverse of the rotation matrix. Thus, an arbitrary 2x2 matrix can be decomposed into R(theta), R(phi), D. This transformation group preserves parallel lines (parallel lines are still parallel after transformation). The ratio of lengths on collinear lines is Transformation in 2D preserved after transformation and thus it follows that the ratio of areas within any arbitrary shapes is preserved. &x'# &x# &x# The transformation has 6 degree of freedom ( 4 elements in the matrix A, &A t# $y'! $y! H $y! 2 for translation ). Affinities: $ ! = $ !$ ! = a $ ! % 0 1" [Eq. 6] %$1 "! %$1"! %$1"! &a11 a12 # &sx 0 # A = = R(θ )⋅R(−φ)⋅D⋅R(φ) D = $a a ! $ 0 s ! % 21 22 " % y " -Preserve: [Eq. 7] - Parallel lines - Ratio of areas - Ratio of lengths on collinear lines - others… - 6 DOF Finally, the fourth transformation is the projective transformation which is a generalization of the affine transformation where two additional elements Transformation in 2D (the 1x2 vector v) are non-zero. This transformation has 8 degrees of freedom and preserves the cross ratio of 4 collinear points. &x'# &x# &x# &A t # $y'! $y! H $y! [Eq. 8] Projective: $ ! = $ !$ ! = p $ ! % v b" %$1 "! %$1"! %$1"! - 8 DOF - Preserve: - cross ratio of 4 collinear points - collinearity - and a few others… An example of cross ratio is defined in Eq.9 for the points P ,P ,P ,P . 1 2 3 4 The cross ratio The cross-ratio of 4 collinear points [Eq. 9] X P4 & i # P3 P − P P − P $ Y ! 3 1 4 2 $ i ! Pi = P $Zi ! 2 P3 − P2 P4 − P1 $ ! 1 P1 % " P − P P − P Can permute the point ordering 1 3 4 2 P1 − P2 P4 − P3 In the next slides we introduce a number of important definitions about lines and points in 2D and 3D and introduce the concepts of vanishing Lecture 4 points and lines. Single View Metrology • Review calibration and 2D transformations • Vanishing points and lines • Estimating geometry from a single image • Extensions Reading: [HZ] Chapter 2 “Projective Geometry and Transformation in 2D” [HZ] Chapter 3 “Projective Geometry and Transformation in 3D” [HZ] Chapter 8 “More Single View Geometry” [Hoiem & Savarese] Chapter 2 Silvio Savarese Lecture 4 - 16-Jan-15 T A line in 2D can be represented as the 3D vector l=[a b c] in homogeneous coordinates; The ratio -a/b captures the slope of the line Lines in a 2D plane and the ratio -c/b defines the point of intersection of the line with the y axis. If a point x belongs to a line l, than the dot product between x and l is y equal to zero (Eq 10). This equations also defines a line in 2D. ax + by + c = 0 -a/b l &a# -c/b l $b! x = $ ! %$c"! T &x1 # &a# If x = [ x , x ]T l $x ! $b! 0 1 2 ∈ $ 2 ! $ ! = %$ 1 "! %$c"! [Eq. 10] In general, two lines l and l’ intersect at a point x. This point is defined as the the cross product between l and l’ [Eq. 11]. This is easy to verify as the Lines in a 2D plane slide shows. y Proof: Given two intersecting lines l and l’, the intersection point x should Intersecting lines lie Ton both lines l and lineT l’; thus the point x is the intersection if and only l! if x l = 0 [Eq.11] and x l’ = 0 [Eq.12]. Let x be l x l’. Then, the vector x is x perpendicular to the vector l and the vector l’ and, thus, it satisfies the [Eq. 11] above constraints. Since the intersection is unique (set arbitrary x’ =\= x x = l×l! l and show that x’ is x), l x l’ is the point of intersection of the two lines. x Proof l×l" ⊥ l → (l×l") ⋅l = 0 → x ∈l [Eq. 12] l×l! ⊥ l! → (l×l!) ⋅l! = 0 → x ∈l! [Eq. 13] x → x is the intersecting point Let now us compute the point of intersection of two parallel lines. We start by observing that a point at infinity in Euclidean coordinates 2D Points at infinity (ideal points) corresponds to a point x in homogenous coordinates whose third inf a coordinate is equal to zero. 'x $ & # 1 − a / b = −a'/ b' $ ! Let us now consider two parallel lines l and l’. When two lines are parallel, x = %x " , x ≠ 0 l = b their slope is equal and thus –a/b=-a’/b’. Let’s now compute the point of % 2 " 3 $ ! intersection of these two lines. Using Eq. 11 we obtain Eq. 13 which is %x " %$c"! exactly the expression of a point at infinity (in homogenous coordinates).