on to 3D…
We want real 3D scene walk-throughs: Scene Modeling for a Camera rotation Camera translation Single View Can we do it from a single photograph?
Reading: A. Criminisi, I. Reid and A. Zisserman, “Single View Metrology” (ICCV 99) B. Zisser And Mundy, appendix
Camera rotations with homographies Camera translation
Original image Does it work? synthetic PP
PP1
St.Petersburg photo by A. Tikhonov PP2 Virtual camera rotations
Yes, with planar scene (or far away) So, what can we do here? PP3 Model the scene as a set PP1 of planes!
PP2 Now, just need to find the orientations of these planes.
PP3 is a projection plane of both centers of projection, so we are OK!
1 Silly Euclid: Trix are for kids! Vanishing points
image plane
vanishing point
camera center
ground plane
Vanishing point • projection of a point at infinity • Caused by ideal line
Parallel lines???
Vanishing points (2D) Vanishing lines
image plane
vanishing point v1 v2
camera center
line on ground plane
Multiple Vanishing Points • Any set of parallel lines on the plane define a vanishing point • The union of all of these vanishing points is the horizon line – also called vanishing line • Note that different planes define different vanishing lines
Vanishing lines Fitting the box volume
Q: What’s the significance of the vanishing point location? Multiple Vanishing Points A: It’s at eye level: ray from COP to VP is perpendicular • Any set of parallel lines on the plane define a vanishing point to image plane. • The union of all of these vanishing points is the horizon line – also called vanishing line • Note that different planes define different vanishing lines
2 Comparing heights Measuring height 5.4 Vanishing 5 Point Camera height 4 3.3 3 2.8 2 1
Example of user input: vanishing point and back Example of user input: vanishing point and back face of view volume are defined face of view volume are defined
High High Camera Camera
Example of user input: vanishing point and back Example of user input: vanishing point and back face of view volume are defined face of view volume are defined
Low Low Camera Camera
3 Comparison of how image is subdivided based Another example of user input: vanishing point on two different camera positions. You should and back face of view volume are defined see how moving the vanishing point corresponds to moving the eyepoint in the 3D world. Left Camera
High Camera Low Camera
Another example of user input: vanishing point Another example of user input: vanishing point and back face of view volume are defined and back face of view volume are defined
Left Right Camera Camera
Another example of user input: vanishing point Comparison of two camera placements – left and and back face of view volume are defined right. Corresponding subdivisions match view you would see if you looked down a hallway. Right Camera
Left Camera Right Camera
4 Comparing heights Perspective cues
Perspective cues Fun with vanishing points
“Tour into the Picture” (SIGGRAPH ’97) The idea Many scenes (especially paintings), can be represented Create a 3D “theatre stage” of as an axis-aligned box volume (i.e. a stage) five billboards Key assumptions: • All walls of volume are orthogonal • Camera view plane is parallel to back of volume Specify foreground objects • Camera up is normal to volume bottom through bounding polygons How many vanishing points does the box have? • Three, but two at infinity • Single-point perspective
Use camera transformations to Can use the vanishing point navigate through the scene to fit the box to the particular Scene!
5 2D to 3D conversion 2D to 3D conversion First, we can get ratios
left right • Size of user-defined back plane must equal top size of camera plane (orthogonal sides) • Use top versus side ratio vanishing point to determine relative left right bottom height and width back top plane dimensions of box • Left/right and top/bot ratios determine part of camera bottom 3D camera placement pos
Depth of the box DEMO Now, we know the 3D geometry of the box We can texture-map the box walls with texture from the image
Can compute by similar triangles (CVA vs. CV’A’) Need to know focal length f (or FOV)
Note: can compute position on any object on the ground • Simple unprojection • What about things off the ground?
Foreground Objects Foreground Objects
Use separate billboard for Add vertical each rectangles for each foreground For this to work, three object separate images used: • Original image. Can compute 3D • Mask to isolate desired coordinates P0, foreground images. P1 since they are • Background with objects on known plane. removed P2, P3 can be computed as before (similar triangles)
6 Quiz: which is 1,2,3-point perspective Automatic Photo Pop-up Image B
Original Image Geometric Labels
Image A
Image C Fit Segments Cut and Fold Novel View
Foreground DEMO Results
Input Image Automatic Photo Pop-up
How can we model more complex scene? Finding world coordinates (X,Y,Z)
Find world coordinates (X,Y,Z) for a few points Define the ground plane (Z=0) Connect the points with planes to model geometry Compute points (X,Y,0) on that plane • Texture map the planes Compute the heights Z of all other points
7 Measurements on planes Unwarp ground plane
4 3 x 2 p’ 1 p
Our old friend – the homography 1234 Need 4 reference points with world coordinates p = (x,y) Approach: unwarp, then measure p’ = (X,Y,0) What kind of warp is this?
Finding world coordinates (X,Y,Z) Preliminaries: projective geometry
Define the ground plane (Z=0) Ames Room Compute points (X,Y,0) on that plane Compute the heights Z of all other points
The projective plane Projective lines Why do we need homogeneous coordinates? What does a line in the image correspond to in • represent points at infinity, homographies, perspective projective space? projection, multi-view relationships What is the geometric intuition? • a point in the image is a ray in projective space
• A line is a plane of rays through origin y (sx,sy,s) – all rays (x,y,z) satisfying: ax + by + cz = 0 (x,y,1) (0,0,0) x z x image plane in vector notation : 0 = []a b c y • Each point (x,y) on the plane is represented by a ray (sx,sy,s) z – all points on the ray are equivalent: (x, y, 1) ≅ (sx, sy, s) lp • A line is also represented as a homogeneous 3-vector l
8 Point and line duality Computing vanishing points • A line l is a homogeneous 3-vector • It is ⊥ to every point (ray) p on the line: lp=0 V
p p2 l2 p P0 Pt l 1 P = P0 + tD l1 D
PX + tDX PX / t + DX DX What is the line l spanned by rays p and p ? P + tD P / t + D D 1 2 P = Y Y ≅ Y Y t → ∞ P ≅ Y • l is ⊥ to p and p ⇒ l = p × p t ∞ 1 2 1 2 PZ + tDZ PZ / t + DZ DZ • l is the plane normal 1 1/ t 0
What is the intersection of two lines l1 and l2 ? Properties v = ΠP∞ • p is ⊥ to l1 and l2 ⇒ p = l1 × l2 • P∞ is a point at infinity, v is its projection Points and lines are dual in projective space • They depend only on line direction • given any formula, can switch the meanings of points and lines to • Parallel lines P + tD, P + tD intersect at P get another formula 0 1 ∞
Computing vanishing points Vanishing Points and Projection Matrix
V
p2
p1
What is the line l spanned by rays p1 and p2 ?
• l is ⊥ to p1 and p2 ⇒ l = p1 × p2 • l is the plane normal
What is the intersection of two lines l1 and l2 ?
• v is ⊥ to l1 and l2 ⇒ v = l1 × l2
What is the intersection of a set of lines l1 , li … ln ? T M = ∑lili i Eigenvector of M with smalest eigenvalues is v
Measuring height without a ruler Measuring Heights
C Z
ground plane Compute Z from image measurements • Need more than vanishing points to do this
9 The Cross Ratio Measuring height
∞ T − B ∞ − R H = R − B ∞ − T R scene cross ratio T (top of object)
t t − b vZ − r H = (reference point) r R r − b v Z − t R C H b R image cross ratio
vZ B (bottom of object)
ground plane X x Y scene points represented as P = image points as p = y Z 1 1
Measuring height vz Measuring Height r
vanishing line (horizon)
t0 t vx v vy H R H
b0 b
Measurements Within Reference Plane
10 Assignment 4 Due: Never
Implement Technique in Criminisi et al. • Load in an image • Click on parallel lines defining X, Y, and Z directions • Compute vanishing points • • Specify points on reference plane, ref. height • Compute 3D positions of several points • Create a 3D model from these points • Extract texture maps – using Assignment 2 warping code • Output a VRML model
11