Single View Geometry Camera model & Orientation + Position estimation

What am I? Vanishing point Mapping from 3D to 2D Point & Line

Goal: Homogeneous coordinates Point – represent coordinates in 2 dimensions with a 3-vector &x# &x# homogeneous coords $y! $ ! ''''''→$ ! %y" %$1"! The projective plane • Why do we need homogeneous coordinates? – represent points at infinity, homographies, perspective projection, multi-view relationships • What is the geometric intuition? – a point in the image is a ray in projective space

y (sx,sy,s) (x,y,1) (0,0,0) z x image plane • Each point (x,y) on the plane is represented by a ray (sx,sy,s) – all points on the ray are equivalent: (x, y, 1) ≅ (sx, sy, s) Projective Lines Projective lines • What does a line in the image correspond to in projective space?

• A line is a plane of rays through origin – all rays (x,y,z) satisfying: ax + by + cz = 0

⎡x⎤ in vector notation : 0 a b c ⎢y⎥ = [ ]⎢ ⎥ ⎣⎢z⎦⎥ l p • A line is also represented as a homogeneous 3-vector l Line Representation • a line is

• is the distance from the origin to the line • is the norm direction of the line

• It can also be written as Example of Line Example of Line (2)

0.42*pi Homogeneous representation

Line in

Is represented by a point in :

But correspondence of line to point is not unique

We define set of equivalence class of vectors in R^3 - (0,0,0) As projective space Projective lines from two points Line passing through two points

Two points: x

Define a line

l is the line passing two points

Proof: Line passing through two points

• More specifically, Matlab codes

• function l = get_line_by_two_points(x, y)

• x1 = [x(1), y(1), 1]'; • x2 = [x(2), y(2), 1]'; • l = cross(x1, x2); • l = l / sqrt(l(1)*l(1)+l(2)*l(2)); Example of Line Verify p2 lies on the line P2 >> 218*0.6+13*0.8 ans =

141.2000

(close) P1 Example of Line P2 Test line: >> p1 = [42,142,1]; >> p2=[218,13,1]; >> l = cross(p1,p2) l = 129 176 -30410 P1

>> l = l/sqrt(l(1)^2+l(2)^2) l = 0.5912 0.8066 -139.3591 >> x = [221;23]; >> y = [218;268]; >> l = get_line_by_two_points(x,y); >> l l =

-0.2448 -0.9696 265.4744 Projective lines from two points

When does the line has the form (a, b, 0)? -all lines passing through the optical center When does the line has the form (0, 0, 1)? -line at infinity Points from two lines Intersection of lines

Given two lines: l ,

Define a point

X is the intersection of the two lines Intersection of lines

• More specifically, Matlab codes

• function x0 = get_point_by_two_line(l, l1)

• x0 = cross(l, l1); • x0 = [x0(1)/x0(3); x0(2)/x0(3)]; Example of Lines Intersection Point and line duality – A line l is a homogeneous 3-vector – It is ⊥ to every point (ray) p on the line: l p=0

p p2 l1 p l 1 l2

What is the line l spanned by rays p1 and p2 ?

• l is ⊥ to p1 and p2 ⇒ l = p1 × p2 • l is the plane normal

What is the intersection of two lines l1 and l2 ?

• p is ⊥ to l1 and l2 ⇒ p = l1 × l2 Points and lines are dual in projective space • given any formula, can switch the meanings of points and lines to get another formula Points from two lines

When P has the form (x,y,0)? Point at infinity

Example: Consider two parallel horizontal lines:

Intersection =

Point at infinity in the direction of y Point at infinity, Ideal points

Intersection:

Any point (x1,x2,0) is intersection of lines at infinity Points at infinity

• Under projective transformation, – All parallel lines intersects at the point at infinity

– One point at infinity ó one parallel line direction

• Where are the points at infinity in the image plane? Line at infinity • A line passing all points at infinity:

• Because : Ideal points and lines

(a,b,0) y (sx,sy,0) y

z image plane z image plane x x • Ideal point (“point at infinity”) – p ≅ (x, y, 0) – parallel to image plane – It has infinite image coordinates Ideal line • l ≅ (a, b, 0) – parallel to image plane • Corresponds to a line in the image (finite coordinates) Comparing heights

Vanishing Point Measuring height

5.4 5 Camera height 4 3.3

3 2.8

2

1 Measuring height vz r

vanishing line (horizon)

t0 t vx v vy H R H

b0 b Focal Length • A Lighting Reproduction Approach to Live-Action Compositing, P. Debevec 2002 FOV depends of Focal Length

f f

Smaller FOV = larger Focal Length Field of View (Zoom) Field of View (Zoom) Large Focal Length compresses depth

400 mm 200 mm 100 mm 50 mm 28 mm 17 mm

© 1995-2005 Michael Reichmann http://www.creativepro.com/img/story/102604_fg3b.jpg Nikon’s 1200-1700mm f/5.6-8P lens. Nikon’s 1200-1700mm f/5.6-8P lens.

"

/012'+$3+/4$&'()5$2

35'0:+$3 +>5'?+!6 ;<=6 $%&'()*+,+-+.

!! ;<=66

36+!+3$(10+0'27)8 # 366+!+3$(10+0'27)8 9.+!+:5*)12('+)$+$%&'()+. 9,+!+:5*)12('+)$+$%&'()+, Canon D30 takes image with 2160x1440 pixels. What is the pixel size? What f to make tower appear half the size of image? Fix f=50mm, D2 = ? in order to make the tower 1/3 size of image? D1=? to make human the same height as tower? What f’ to make tower appear twice as big as before (2/3 the height)? Setting f’=100mm, D1=? in order to keep the original size of human? (1/3 the height) Lens distortion with focal length

Lens distortion with focal length

Camera projection model

The overall goal is to compute 3D geometry of the scene from just 2D images. We will first study 3D->2D camera projection using linear algebra. The Pinhole Camera

– Light enters a darkened chamber through a pinhole opening and forms an image on the further surface The Pinhole Camera Model Ideal case: yc

y Projection equation: y’ x’ = f X / Z p C y’ = f Y / Z z zc f

xc

Zx’ = f X Zy’ = f Y Z = Z Step 1: Camera projection matrix

Zx’ = f X Zy’ = f Y y Z = Z c

y y’ p C f 0 0 0 Xc x’ z Z zc f 0 f 0 0 Yc = y’ c Z 0 0 1 0 c 1 x 1 c

P0 X = x Step 2: Intrinsic camera parameters: map camera coordinate to pixel coordinate yc

αx s px x’ x 0 p y’ y αy y = y 0 0 1 z’ z y’ p C z zc f K Optical world Pixel world (3x3 submatrix) xc

αx, αy is pixel scaling factor

px, py is the principle point (where optical axis hits image plane)

s is the slant factor, when the image plane is not normal to the optical axis Combine the Intrinsic camera parameters X αx s px f 0 0 0 c x Y 0 αy py 0 f 0 0 c = y 0 0 1 0 0 1 0 Zc 1 1 x αxf s px 0 c x y 0 αyf py 0 c = y 0 0 1 0 zc z 1 K (Calibration matrix) P0 X = x

P = [K , 0] = K [I, 0] Step 3: External parameters: rotation and translation map the world to camera coordinates

3x3 3x1 (R,t) Camera coordinate

yc World coordinate

y y’ p C z zc f

xc Combining Internal and External parameters (R,t) Camera coordinate

1) yc World coordinate

y y’ p C z zc f

2) xc

1) Translate the world coordinate into the camera coordinate 2) Translate the camera coordinate into the pixel coordinate Combining Internal and External parameters (R,t) Camera coordinate

yc World coordinate

y y’ p C z After simplication: zc f

xc x = K [R, t] X

pixel world Special case, planar world, homograph

(R,t) X= (xw,yw,zw=0,1) yc World coordinate y y’ p C x = K [R, t] X z zc f

Expand: xc xw Camera coordinate yw x = K r1 r2 r3 t zw=0 1 Special case, planar world, homograph (R,t) X= (xw,yw,zw=0,1) yc World coordinate y After simplication: y’ p C z zc f w x xc x = K r1 r2 t yw Camera coordinate 1

We have the homographic mapping: X = K H3x3 X Special case: rotating camera

(R,t) x = K [R, t] X yc World coordinate with t = 0 y y’ p C Expand: z x = K [R] X zc f xc Camera coordinate

xw w This is also a homography with X = y H = R zw Panoramas

x = K [R] X Visual Odometry: Overview

1) The transformation (R,t) tells us (R,t) how to rotate + move the world coordinate to the camera yc World coordinate. coordinate y 2) In the following slides, we will see y’ p C z 1) how to get the world reference’s zc f frame orientation+position from (R,t), xc 2) how to get the pan/tilt/yaw angles from R 3) If we want to know the camera orientation+position, we need to look at its inverse transform, which we should see is , 1) The computation of the camera pan/tilt/yaw angle is the same as in the previous case Recover Camera orientation Case 1: single vanishing point (z- direction) Pan/tilt/yaw angles

Pan : rotation Yaw :rotation Tilt : rotation around y-axis around z-axis around x-axis

z y y x x z Rotation: Pan/Tilt/Yaw

Given the 3x3 rotation matrix R, we can recover, pan/tilt/yaw angle, and vise. vica.

ycam

T zworld, (0,0,1)

xcam, The basic idea is that if we can see the “north” star, we know how to oriented ourselves (minus yaw angle) (why?) And the ``north” star is just the point of infinity in some direction

(R,t) Camera coordinate

yc World coordinate

y y’ p C z zc f

xc Seeing the vanishing point can tell us:

Camera projection equation: x=K·[R,t]·X, or x • • • • w x • • • • y c r1 r2 r3 t w [R,t]= • • • • K • R• • t• = yc zw • • • • zc • • • • tw Columns of R matrix = image of vanishing points of world-space axes!

r1,r2,r3 == image of x,y,z axis vanishing points z-direction • • • • 0 r r r t 0 • 1 •2 •3 • = r3 (need to normalize to norm =1) 1 • • • • 0 vanishing point Recover Pan/Tilt angle from z-vanishing point

Geometric explanation: Pan Tilt The alignment between z-axis of the world and image is defined only by the pan/tilt angle:

ycam T zworld, (0,0,1) r3 =

xcam,

Note: tilt angle with rotation around x-axis is positive when pointing down in this figure zcam Sanity check Vanishing point in z In this example, we are walking down a track, the ground is tilted up(we are going up the hill). Assume we are b meter tall. We see two parallel lines to our left and right, with x = 1,-1. A line on the ground is described by equation y = az - b, where a is the slope the ground.

Image plane y z

x = 1 x = -1 x y = az -b Ground plane

So we know: y y = az -b

Tilt angle z As a line gets far away, z -> infinity. If (1,y,z) is a point on this line, its image is f(1/z,y/z). As z -> infinity, 1/z -> 0. What about y/z? y/z = (az-b)/z = a - b/z -> a. So a point on the line appears at: (0,a).

y Image plane z VanishingPoint (0,a)

x y = az - b Ground plane = T rz = zworld, (0,0,1)

Now, we check

y (0,a) z x y = az - b Recover Pan/Tilt angle from z-vanishing point Geometric explanation: Pan Tilt The alignment between z-axis of the world and image is defined only by the pan/tilt angle:

ycam r3 =

T zworld, (0,0,1)

xcam,

zcam