Lecture 4 Single View Metrology

Professor Silvio Savarese Computational Vision and Geometry Lab

Silvio Savarese Lecture 4 - 16-Jan-15

Lecture 4 Single View Metrology

• Review calibration and 2D transformations • Vanishing points and lines • Estimating geometry from a single image • Extensions

Reading: [HZ] Chapter 2 “Projective Geometry and Transformation in 2D” [HZ] Chapter 3 “Projective Geometry and Transformation in 3D” [HZ] Chapter 8 “More Single View Geometry” [Hoiem & Savarese] Chapter 2

Silvio Savarese Lecture 4 - 16-Jan-15

The calibration problem was discussed in details during lecture 3. Calibration Problem

jC

M = K[R T] &ui # cot u P → M P → p = &α −α θ o # i i i $ ! $ β ! K = $0 vo ! vi sinθ % " $0 0 1 ! % " World ref. system In pixels Calibration Problem

jC

M = K[R T] &u # P M P → p = i 11 unknown i → i i $ ! Need at least 6 correspondences %vi " World ref. system In pixels

Once the camera is calibrated (intrinsics are known) and the transformation from the world reference system to the camera reference Once the camera is calibrated... system (which accounts for the extrinsics) is also known, can we estimate Pinhole perspective projection the location of a point P in 3D from its observation p? The general answer to this question is no. This is because, given the observation p, even when the camera intrinsics and extrinsic are known, the only thing we can say is that the point P is located somewhere along the line defined by C and p. Line of This line is called the line of sight. sight P p

Ow C

M = K[R T]

-Internal parameters K are known -R, T are known – but these can only relate C to the calibration rig Can I estimate P from the measurement p from a single image?

No - in general ☹ (P can be anywhere along the line defined by C and p)

The actual location of P along this line is unknown and cannot be Recovering structure from a single view determined from the observation p alone. Pinhole perspective projection

Line of sight P p

Ow C

unknown known Known/ Partially known/ unknown In the remainder of this lecture, we will introduce tools and techniques for Recovering structure from a single view inferring the geometry of the camera and the 3D environment from a just one image of such environment.

http://www.robots.ox.ac.uk/~vgg/projects/SingleView/models/hut/hutme.wrl

Before we go into the details of that, let me recap some of the concepts you have already explored when we talked about transformations. Transformation in 2D There are four important transformations in 2D. All these transformations will be described in homogenous coordinates.

-Isometries

-Similarities

-Affinity

-Projective

The first one is called isometric transformation which is in general the concatenation of a rotation and translation transformation and expressed by the matrix H and Eq. 4. This transformation preserves the distance Transformation in 2D e between any pair of points and has 3 degrees of freedom (2 for translation, 1 for rotation). This group captures the motion of a rigid object.

&x'# &x# &x# &R t# $y'! $y! H $y! [Eq. 4] Isometries: $ ! = $ !$ ! = e $ ! % 0 1" [Euclidean] %$1 "! %$1"! %$1"!

- Preserve distance (areas) - 3 DOF - Regulate motion of rigid object The second group of transformations is called similarity transformation. The transformation is a concatenation of a translation, rotation and scale Transformation in 2D translation transformations. It preserves the ratio of lengths between any two line segments before and after transformation. It also preserves the angle between any intersecting lines. The group has 4 degrees of freedom (2 translation, 1 rotation, 1 scale)

! x' $ ! x $ ! x $ # & ! SR t $# & # & Similarities: # y' & = # &# y & = Hs # y & # & "# 0 1 %&# & # & " 1 % " 1 % " 1 % ! s 0 $ [Eq. 5] S= # & " 0 s %

- Preserve - ratio of lengths - angles -4 DOF

The third transformation group is the affine transformation. This group can be interpreted as a series of translation, rotation and anisotropic scaling Transformation in 2D transformations. The core of the affine transformation is characterized by the matrix A which includes a rotation by phi, an anisotropic scaling by [sx 0; 0 sy], an inverse rotation by –phi and an arbitrary rotation by theta (see Eq. 7). This transformation has also a mathematical interpretation: Any &x'# &x# &x# positive definite 2x2 matrix can be decomposed into two orthogonal $ ! &A t#$ ! $ ! matrices and a diagonal matrix by Singular Value Decomposition (this is Affinities: y' = y = Ha y $ ! $ !$ ! $ ! [Eq. 6] by assuming positive scaling; for negative scaling, the transformation may % 0 1" T T T %$1 "! %$1"! %$1"! not be unique.) Thus we can express A = UDV = (UV ) (VDV ) = R(theta) R(-phi) D R(phi) T T &a11 a12 # &sx 0 # by replacing UV = R(theta), V = R(phi) , and where D is a diagonal A = = R(θ )⋅R(−φ)⋅D⋅R(φ) D = $a a ! $ ! matrix with the singular values. Note that transpose of a rotation matrix is % 21 22 " 0 sy [Eq. 7] % " the inverse of the rotation matrix. Thus, an arbitrary 2x2 matrix can be decomposed into R(theta), R(phi), D.

This transformation group preserves parallel lines (parallel lines are still parallel after transformation). The ratio of lengths on collinear lines is Transformation in 2D preserved after transformation and thus it follows that the ratio of areas within any arbitrary shapes is preserved.

&x'# &x# &x# The transformation has 6 degree of freedom ( 4 elements in the matrix A, &A t# $y'! $y! H $y! 2 for translation ). Affinities: $ ! = $ !$ ! = a $ ! % 0 1" [Eq. 6] %$1 "! %$1"! %$1"!

&a11 a12 # &sx 0 # A = = R(θ )⋅R(−φ)⋅D⋅R(φ) D = $a a ! $ 0 s ! % 21 22 " % y "

-Preserve: [Eq. 7] - Parallel lines - Ratio of areas - Ratio of lengths on collinear lines - others… - 6 DOF Finally, the fourth transformation is the projective transformation which is a generalization of the affine transformation where two additional elements Transformation in 2D (the 1x2 vector v) are non-zero. This transformation has 8 degrees of freedom and preserves the cross ratio of 4 collinear points.

&x'# &x# &x# &A t # $y'! $y! H $y! [Eq. 8] Projective: $ ! = $ !$ ! = p $ ! % v b" %$1 "! %$1"! %$1"!

- 8 DOF - Preserve: - cross ratio of 4 collinear points - collinearity - and a few others…

An example of cross ratio is defined in Eq.9 for the points P ,P ,P ,P . 1 2 3 4 The cross ratio

The cross-ratio of 4 collinear points

[Eq. 9] X P4 & i # P3 P − P P − P $ Y ! 3 1 4 2 $ i ! Pi = P $Zi ! 2 P3 − P2 P4 − P1 $ ! 1 P1 % "

P − P P − P Can permute the point ordering 1 3 4 2 P1 − P2 P4 − P3

In the next slides we introduce a number of important definitions about lines and points in 2D and 3D and introduce the concepts of vanishing Lecture 4 points and lines. Single View Metrology

• Review calibration and 2D transformations • Vanishing points and lines • Estimating geometry from a single image • Extensions

Reading: [HZ] Chapter 2 “Projective Geometry and Transformation in 2D” [HZ] Chapter 3 “Projective Geometry and Transformation in 3D” [HZ] Chapter 8 “More Single View Geometry” [Hoiem & Savarese] Chapter 2

Silvio Savarese Lecture 4 - 16-Jan-15 T A line in 2D can be represented as the 3D vector l=[a b c] in homogeneous coordinates; The ratio -a/b captures the slope of the line Lines in a 2D plane and the ratio -c/b defines the point of intersection of the line with the y axis. If a point x belongs to a line l, than the dot product between x and l is y equal to zero (Eq 10). This equations also defines a line in 2D. ax + by + c = 0 -a/b l

&a# -c/b l $b! x = $ ! %$c"! T &x1 # &a# If x = [ x , x ]T l $x ! $b! 0 1 2 ∈ $ 2 ! $ ! = %$ 1 "! %$c"! [Eq. 10]

In general, two lines l and l’ intersect at a point x. This point is defined as the the cross product between l and l’ [Eq. 11]. This is easy to verify as the Lines in a 2D plane slide shows.

y Proof: Given two intersecting lines l and l’, the intersection point x should Intersecting lines lie Ton both lines l and lineT l’; thus the point x is the intersection if and only l! if x l = 0 [Eq.11] and x l’ = 0 [Eq.12]. Let x be l x l’. Then, the vector x is x perpendicular to the vector l and the vector l’ and, thus, it satisfies the [Eq. 11] above constraints. Since the intersection is unique (set arbitrary x’ =\= x x = l×l! l and show that x’ is x), l x l’ is the point of intersection of the two lines. x Proof l×l" ⊥ l → (l×l") ⋅l = 0 → x ∈l [Eq. 12] l×l! ⊥ l! → (l×l!) ⋅l! = 0 → x ∈l! [Eq. 13] x

→ x is the intersecting point

Let now us compute the point of intersection of two parallel lines. We start by observing that a point at infinity in Euclidean coordinates 2D Points at infinity (ideal points) corresponds to a point x in homogenous coordinates whose third inf a coordinate is equal to zero. 'x $ & # 1 − a / b = −a'/ b' $ ! Let us now consider two parallel lines l and l’. When two lines are parallel, x = %x " , x ≠ 0 l = b their slope is equal and thus –a/b=-a’/b’. Let’s now compute the point of % 2 " 3 $ ! intersection of these two lines. Using Eq. 11 we obtain Eq. 13 which is %x " %$c"! exactly the expression of a point at infinity (in homogenous coordinates). & 3 # This confirms the intuition that two parallel lines intersect at infinity. l &a'# &x'1 # $ ! $ ! The point of intersection of two parallel lines returns a point at infinity x = x' l! l' = b' which is also called ideal point. ∞ $ 2 ! $ ! $c'! %$ 0 "! % " & b # $ ! Let’s intersect two parallel lines: → l ×l) ∝ − a x Eq.13] $ ! = ∞ %$ 0 "! • In Euclidian coordinates this point is at infinity • Agree with the general idea of two lines intersecting at infinity One interesting property of a point at infinity is that all the parallel lines 2D Points at infinity (ideal points) with the same slope -a/b passes through the point [b -a 0] [Eq.15]. a 'x $ & # 1 − a / b = −a'/ b' $ ! x %x " , x 0 l = b = % 2 " 3 ≠ $ ! $c! &%x3 #" % " l &a'# l' $b'! l! = $ ! Note: the line l = [a b c]T pass trough the ideal point $c'! x∞ % " & b # lT x = a b c $− a! = 0 ∞ [ ]$ ! [Eq. 15] %$ 0 "!

So does the line l’ since a b’ = a’ b

We can further extend this concept and define the lines at infinity.

Lines infinity l∞ Consider 2 or more pairs of parallel lines (right side of the slide). Each pair Set of ideal points lies on a line called the line at infinity. of parallel lines intersect into a point at infinity. Let us call these set of points x ’, x ’’, …. The line that passes through all these points at infinity How does it look like? inf infT T T must satisfy l x ’ =0, l x ’’ =0, etc… and is simply l = [0 0 c] . Since c in inf inf T &0# an arbitrary value, we can simply write l=[0,0,1] . & b' # $ ! ' x = $− a'! l∞ = 0 ∞ $ ! A line at infinity can be thought of the set of “directions” of lines in the $ ! %$ 0 "! plane %$1"! b'' T & # '' $ ! Indeed: &x1 # &0# x a'' ∞ = $− ! $x ! $0! 0 $ 0 ! $ 2 ! $ ! = % " %$ 0 "! %$1"! l∞

A line at infinity can be thought of the set of “directions” of lines in the plane

Now, let’s see what happens if we apply a generic projective transformation H to a point at infinity p (Eq. 17). Projective transformation of a point at infinity inf

Notice that the last element H p becomes non-zero which suggests that &A t # inf H = $ ! a projective transformation in general maps points at infinity to points that % v b" are no longer at infinity.

p'= H p is it a point at infinity? This is not true for affine transformations [Eq.18]. If we apply an affine transformation H to p will still obtain a point at infinity. A inf &1# & p'x # &A t # $ ! $ ! H p ? = 1 = p'y …no! ∞ = $v b! $ ! $ ! % " $0! $ p' ! [Eq. 17] % " % z " An affine &1# & p'x # transformation &A t # $ ! $ ! of a point = 1 = p' at infinity is still H A p∞ = ? $ ! $ ! $ y ! %0 b" a point at [Eq. 18] %$0"! %$ 0 "! infinity This time, let’s apply a projective transformation H to a line. The projective -T Projective transformation of a line (in 2D) transformation of a line l is l’ = H l (Eq 19).

Let’s derive this equation. All points that pass through a line l must satisfy &A t # T T T -T H the line equation: x l = 0; —> x H H l = 0 = $ ! T T T T -T % v b" Since x’ = H x (and x’ = x H ), then x’ H l = 0; Because, after the transformation, a projected point x’ must still belong to the projected line T -T l’ (x’ l’ = 0), it implies that l’ = H l. T l" = H− l is it a line at infinity? [Eq. 19] Let’s now apply the projective transformation H to a line at infinity l . Is −T &0# &t x # inf &A t # the projected line still at infinity? No. −T = $0! = $t ! H l∞ = ? $ ! $ ! $ y ! …no! % v b" [Eq. 20] %$1"! %$ b "! Let’s now apply the affine transformation H to a line at infinity l . Is the A inf 0 0 0 projected line still at infinity? Yes, as the derivation next to Eq 21 shows. −T & # −T & # & # −T &A t# & A 0# H l = ? = $0! = $0! = $0! A ∞ $ ! $ ! $ T −T !$ ! $ ! %0 1" %− t A 1" [Eq. 21] %$1"! %$1"! %$1"!

So far we have introduced the concepts of lines and points at infinity in 2D. Let’s introduce the equivalent concepts in 3D (and in the Points and planes in 3D corresponding homogenous coordinates). Points in 3D (homogenous coordinates) are denoted as x and their corresponding projective transformations in the 2D image are denoted as p. x y & 1 # &a# $ ! $ ! Following the 2D case, we can represent a plane as a normal vector x 2 b (a,b,c) and a distance from the origin (d) which is [Eq.22]. Thus, a point is x = $ ! $ ! x x Π = on a plane if and only if [Eq.23] holds. $ 3 ! $c! z $ ! $ ! % 1 " d In 3D, it is tricky to represent a line but one can represent it using an % " intersection of 2 planes. See [HZ] Ch.3.2.2 for detail. x ∈ Π ↔ xTΠ = 0 ax + by + cz + d = 0 [Eq. 22] [Eq. 23] Philosophical interlude by C. Choy (course assistant): When you see an How about lines in 3D? object, you are not perceiving 3D space (there is no way to ‘sense’ space directly), you are merely seeing the 2D perspective projection of the object • Lines have 4 degrees of freedom - hard to represent in 3D-space • Can be defined as intersection of 2 planes on your retina. Thus seeing is equivalent to perspective projection. So it is essential to model the world as is in 3D space and perspective projection as we do.

In 3D, similarly to ideals points in 2D, points at infinity are defined as the Points at infinity in 3D point of intersection of parallel lines in 3D. Points where parallel lines intersect in 3D

point at infinity &x1 # world $x ! $ 2 ! x∞ = $x3 ! $ ! % 0 " Parallel lines Similarly to the 2D case, by applying the projective transformation M to a point at infinity x we obtain a point p in the image plane which is no inf inf Vanishing points longer at infinity. The projective projection of a point at infinity into the image Interestingly, similarly to the 2D case, the direction of the (parallel) lines in plane defines a vanishing point. 3D associated to x is given by the coordinates x , x and x of x inf 1 2 3 inf

p point at infinity &x # ∞ 1 M world $x ! $ 2 ! x∞ = $x3 ! $ ! % 0 " & p1 # Parallel lines = direction of p = $ p ! the line in 3D ∞ $ 2 ! %$ p3 "!

Next we derive a useful relationship between parallel lines in 3D, the Vanishing points and directions corresponding vanishing point in the image and the camera parameters.

Let’s define as d=[a,b,c] the direction of a set of parallel lines in 3D in the camera reference system. These lines intersect to a point at infinity and d the projection of such point in the image returns the vanishing point v. v d = direction of the line It’s easy to prove that v is related to d via Eq. 24, where K is the camera = [a, b, c]T matrix. Equivalently, d can-1 be expressed as function of v by Eq. 25, where the division by || K v || guarantees that d has unit norm (d is a C direction). K −1 v d = v = K d −1 K v The proof of Eq 24 is reported in the bottom part of the slide. [Eq. 24] [Eq. 25]

Proof: " a % $ ' " a % $ ' " a % $ b ' M $ ' X∞ = $ b ' v = M X ∞ = K [I 0] = K b $ c ' $ c ' $ ' $ c ' $ ' $ ' # & # 0 & # 0 &

If we consider a plane π as a superset of set of parallel lines, each set of parallel lines intersects at a point at infinity. The line that passes through Vanishing (horizon) line such set of points at infinity is the line at infinity l associated to π. A line inf at infinity is also defined as the line where two parallel planes intersect (in general, the intersection of two planes in 3D is a line). The projective l transformation of l to the image plane is no longer a line at infinity and is ∞ inf called the vanishing line or horizon line l (see Eq. 26). The horizon line horiz is a line that passes through the corresponding vanishing points in the image. π

Projective transformation M horizon

−T lhor = H P l∞ [Eq. 26]

Image The concept of horizon line allows to answer interesting questions about images. One is the following: Are these two lines parallel or not? Are the two dashed lines in the image parallel or not?

In order to answer this question we can assume that an oracle tells us that the horizon line is the orange one. Then, we can verify whether the 2 dashed lines meet at the horizon or not. If yes, these 2 lines are parallel in 3D.

This is an a nice example which shows that recognition can help solve a •Recognition helps reconstruction! reconstruction problem. If we recognize the horizon line, we can infer •Humans have learnt this properties about the world (verify the property that two lines are parallel or not). Humans have learnt many of these properties from our daily experience and we use those for solving important estimation problems.

- Recognize the horizon line - Measure if the 2 lines meet at the horizon - if yes, these 2 lines are // in 3D

It is easy to show that the normal n of π and l are related by Eq. 27 horiz Vanishing points and planes (see sec. 8.6.2 [HZ] for details), where K is a camera matrix.

Again, Eq. 27 can be useful for estimating properties of the world. If we recognize the horizon and our camera is calibrated (K is known), we can n estimate the orientation of the ground plane. π l∞ …

lhoriz C

T n = K lhoriz [Eq. 27]

Before introducing the last property that relates vanishing points and lines, we define the plane at infinity π . Planes at infinity ∞

&0# A set of 2 or more vanishing lines (blue lines in the figure) defines the plane at infinity π (yellow plane in the figure). The plane at infinity is y $ ! Π 0 ∞ T ∞ … Π = $ ! described by [0 0 0 1] in homogenous coordinates ∞ $0! x $ ! 1 … % " z plane at infinity

• Parallel planes intersect at infinity in a common line – the line at infinity

• A set of 2 or more lines at infinity defines the plane at

infinity Π∞ The last property we introduce (which we don’t prove either) is expressed by Eq. 28. Suppose that two pairs of parallel lines have directions d and 1 Angle between 2 vanishing points d , and are associated to the points at infinity x1 and x2 , respectively. Let 2 ∞ ∞

v1 and v2 be the corresponding vanishing points. Eq. 28 relates the angle between d and d and the vanishing points v and v . We define the x2∞ 1 2 1T -1 2 matrix (K K ) as ω.

x1 ∞ v2 One special case which is useful in practice is when the two set of parallel d1 lines are orthogonal to each other. In this case, d and d are orthogonal, θ 1 2 which gives us [Eq. 29]. Notice that this is a scalar equation. v1 d2 C

vTω v cosθ = 1 2 T −1 T T ω = (K K ) [Eq. 28] v1 ω v1 v2ω v2

T If θ = 90 → v1 ω v2 = 0 [Eq. 29] Scalar equation

The matrix ω has a special geometrical meaning in that it is the projective transformation of the absolute conic Ω in the image. This relationship is ∞ Projective transformation of Ω expressed by [Eq. 30]. What’s the absolute conic Ω ? It’s a conic that lies ∞ ∞ T Absolute conic in the plane at infinity [0 0 0 1] . Note that in general, the projective -T -1 transformation of a quadric or conic Ω is expressed as M Ω M (see HZ −T −1 T −1 ! # page 73, eq. 3.16). ω = M Ω∞M = (K K ) M = K R T [Eq. 30] " $ The matrix ω satisfies other interesting properties: 1. It is not function of R, T - It depends only on the internal matrix and not on the extrinsic parameters of the camera R and T. - It is symmetric and known up to scale ω ω ω & 1 2 4 # - If w is 0, the camera has no skew. $ ! 2 2. ω = ω ω ω symmetric and known up scale - If w = w , the camera has square pixels. $ 2 3 5 ! 1 3 %$ω4 ω5 ω6 "! ω = 0 2 square pixel 3. ω2 = 0 zero-skew 4. ω1 = ω3

These properties are useful for two reasons: Why is this useful? • To calibrate the camera: By using Eq 28 or Eq 29, we can set up a system of equations that allows us to calibrate our camera – that is, to estimate internal parameters of the camera • To calibrate the camera • To estimate the geometry of the 3D world: Once K is estimated or K is known, we can use these equations to estimate the orientation of • To estimate the geometry of the 3D world planes in 3D w.r.t. to the camera reference system

Let’s see some examples next. Lecture 4 Single View Metrology

• Review calibration • Vanishing points and line • Estimating geometry from a single image • Extensions

Reading: [HZ] Chapter 2 “Projective Geometry and Transformation in 3D” [HZ] Chapter 3 “Projective Geometry and Transformation in 3D” [HZ] Chapter 8 “More Single View Geometry” [Hoiem & Savarese] Chapter 2

Silvio Savarese Lecture 4 - 16-Jan-15

Suppose we can identify two planes in an image of the 3D world (e.g., the two building facades) and suppose we can identify a pair of parallel lines Single view calibration - example on each of these planes. This allows to estimate two vanishing points in the image v and v . Suppose we know that these planes are 1 2

] perpendicular in 3D (the two building facades are perpendicular in 3D). vTω v cosθ = 1 2 Than we can set up the system of equations above using Eq 29 and the T T v1 ω v1 v2ω v2 definition of ω. [Eq. 28 [Eq.

Is this sufficient to estimate the camera parameters? K has in general 5 degrees of freedom and Eq.29 is a scalar equation; so clearly we don’t have enough constraints. v v2 1 θ = ? = 90o

vTω v = 0 1 2 Do we have enough constraints to T −1 estimate K? ω = (K K ) K has 5 degrees of freedom and Eq.29 is a scalar equation ☹

Let’s now assume we can identify another pair of parallel lines and its corresponding vanishing point v . With a third vanishing point and by Single view calibration - example 3 assuming that the 3 set of pairs of parallel lines are pairwise orthogonal in 3D (which is true in this example), we can use Eq. 29 to set up a system

] of 3 equations (constraints) (Eqs. 31). vTω v cosθ = 1 2 T T v1 ω v1 v2ω v2 [Eq. 28 [Eq.

v v1 2 v3

T v1 ω v2 = 0 ] T v1 ω v3 = 0

[Eqs. 31 [Eqs. T v2ω v3 = 0 Let’s now also make some assumptions about the camera. Let’s assume Single view calibration - example the camera is zero-skew and with square pixels.

Using the properties of ω: - ω is symmetric which means we have 6 unknown - ω is known up a scale which reduces to 5 unknown &ω1 ω2 ω4 # - ω =0 and ω =ω which reduces to 3 unknown $ ! known up to scale 2 1 3 ω = $ω2 ω3 ω5 ! %$ω4 ω5 ω6 "! We have now enough constraints to solve for the unknowns and can calculate ω. The actual parameters of K can be computed from ω using à Compute : the Cholesky factorization. We don’t proof this results; for more details ω2 = 0 ω1 = ω3 ω please refer to HZ pag 582.

T Thus, at the end of this procedure we have managed to calibrate the Once ω is calculated, we get K: v1 ω v2 = 0 camera from just one single image! ] T −1 T ω = (K K ) v ω v = 0 K 1 3 (Cholesky factorization; HZ pag 582)

[Eqs. 31 [Eqs. T v2ω v3 = 0

Once K is known we can “reconstruct” the geometry of the scene; for instance, we can compute the orientation of all the planes in 3D using Eq. Single view reconstruction - example 27. In order to do so, we need to identify the corresponding lines at infinity and select the orientation discontinuities (that is, where planes fold).

lh

T K known n K l = Scene plane orientation in → = horiz the camera reference system

Select orientation discontinuities

Again, the assortment of tools introduced in this lecture allows us to estimate properties of the camera from observations and/or estimate Single view reconstruction - example properties of the world by assuming we have some knowledge about the world (e.g., where the horizon lines are; planes discontinuities; etc…).

Notice that that actual scale of the scene cannot be recovered unless we assume we have access to some measurements in 3D (e.g., a window size); this is similar to what we did when we calibrated the camera.

C

Recover the structure within the camera reference system Notice: the actual scale of the scene is NOT recovered

•Recognition helps reconstruction! •Humans have learnt this Lecture 4 Single View Metrology

• Review calibration • Vanishing points and lines • Estimating geometry from a single image • Extensions

Reading: [HZ] Chapter 2 “Projective Geometry and Transformation in 3D” [HZ] Chapter 3 “Projective Geometry and Transformation in 3D” [HZ] Chapter 8 “More Single View Geometry” [Hoiem & Savarese] Chapter 2

Silvio Savarese Lecture 4 - 16-Jan-15

Here we see some results obtained using the approach proposed by A. Criminisi & Zisserman, 99 Criminisi and Zisserman (1999). This approach uses many of the results introduced in this lecture.

http://www.robots.ox.ac.uk/~vgg/projects/SingleView/models/merton/merton.wrl

Criminisi & Zisserman, 99

http://www.robots.ox.ac.uk/~vgg/projects/SingleView/models/merton/merton.wrl La Trinita' (1426) Firenze, Santa Maria Novella; by Masaccio (1401-1428)

La Trinita' (1426) Firenze, Santa Maria Novella; by Masaccio (1401-1428)

http://www.robots.ox.ac.uk/~vgg/projects/SingleView/models/hut/hutme.wrl Single view reconstruction - drawbacks

Manually select: • Vanishing points and lines; • Planar surfaces; • Occluding boundaries; • Etc..

A few years later, D Hoiem proposed an approach where the process of recovering the geometry from a single image is mostly automatic. This Automatic Photo Pop-up approach leverages recognition and segmentation results; Hoiem et al, 05

Automatic Photo Pop-up Hoiem et al, 05… Automatic Photo Pop-up Hoiem et al, 05…

Software:

http://www.cs.uiuc.edu/homes/dhoiem/projects/software.html

During the same period, A. Ng and his student A. Saxena at Stanford also demonstrated that it is possible to recover the geometry of the scene from Make3D a single image using recognition results and probabilistic inference. Saxena, Sun, Ng, 05…

Training Prediction

Image Depth

Plane Parameter MRF

Planar Surface Segmentation

youtube Connectivity Co-Planarity

Single Image Depth Reconstruction Saxena, Sun, Ng, 05…

A software: Make3D “Convert your image into 3d model”

http://make3d.stanford.edu/ http://make3d.cs.cornell.edu/ In my own group, we have shown it is possible to combine recognition and Coherent object detection and scene reconstruction from a single image in a coherent formulation. layout estimation from a single image Y. Bao, M. Sun, S. Savarese, CVPR 2010, BMVC 2010

M. Sun Y. Bao

Next lecture:

Multi-view geometry (epipolar geometry)

Appendix