Lecture 9:

Professor Fei‐Fei Li Stanford Vision Lab

Fei-Fei Li Lecture 9 - 1 21‐Oct‐11 What we will learn today?

•Why is stereo useful? • Epipolar constraints • Essential and fundamental •Estimating F (Problem Set 2 (Q2)) • Rectification

Reading: [HZ] Chapters: 4, 9, 11 [FP] Chapters: 10

Fei-Fei Li Lecture 9 - 2 21‐Oct‐11 Recovering structure from a single view Pinhole perspective projection

P p

Ow Scene C Calibration rig Camera K

Why is it so difficult? Intrinsic ambiguity of the mapping from 3D to image (2D)

Fei-Fei Li Lecture 9 - 3 21‐Oct‐11 Recovering structure from a single view Pinhole perspective projection Lazebnik

S.

slide

Courtesy

Intrinsic ambiguity of the mapping from 3D to image (2D)

Fei-Fei Li Lecture 9 - 4 21‐Oct‐11 Two eyes help!

Fei-Fei Li Lecture 9 - 5 21‐Oct‐11 Two eyes help!

?

x1 x2 K =known K =known

R, T

O2 O1 This is called triangulation

Fei-Fei Li Lecture 9 - 6 21‐Oct‐11 Triangulation

2 2 • Find X that minimizes d ( x1 , P1 X )  d ( x 2 , P2 X )

X

x2 x1

O1 O2

Fei-Fei Li Lecture 9 - 7 21‐Oct‐11 Stereo‐view geometry

• Correspondence: Given a point in one image, how can I find the corresponding point x’ in another one?

• Camera geometry: Given corresponding points in two images, find camera matrices, position and pose.

• Scene geometry: Find coordinates of 3D point from its projection into 2 or multiple images.

This lecture (#9)

Fei-Fei Li Lecture 9 - 8 21‐Oct‐11 Stereo‐view geometry

Next lecture (#10) • Correspondence: Given a point in one image, how can I find the corresponding point x’ in another one?

• Camera geometry: Given corresponding points in two images, find camera matrices, position and pose.

• Scene geometry: Find coordinates of 3D point from its projection into 2 or multiple images.

Fei-Fei Li Lecture 9 - 9 21‐Oct‐11 What we will learn today?

•Why is stereo useful? • Epipolar constraints • Essential and fundamental matrix •Estimating F • Rectification

Reading: [HZ] Chapters: 4, 9, 11 [FP] Chapters: 10

Fei-Fei Li Lecture 9 - 10 21‐Oct‐11 Epipolar geometry P

p1 p2

e1 e2

O1 O2

• Epipolar Plane • Epipoles e1, e2 • Baseline = intersections of baseline with image planes = projections of the other camera center • Epipolar Lines = vanishing points of camera motion direction

Fei-Fei Li Lecture 9 - 11 21‐Oct‐11 Example: Converging image planes

Fei-Fei Li Lecture 9 - 12 21‐Oct‐11 Example: Parallel image planes

P

e2 p 1 p2 e2

O1 O2 • Baseline intersects the image plane at infinity • Epipoles are at infinity • Epipolar lines are parallel to x axis

Fei-Fei Li Lecture 9 - 13 21‐Oct‐11 Example: Parallel image planes

Fei-Fei Li Lecture 9 - 14 21‐Oct‐11 Example: Forward translation

e2

O2

e1

O2

•The epipoles have same position in both images • Epipole called FOE (focus of expansion)

Fei-Fei Li Lecture 9 - 15 21‐Oct‐11 Epipolar Constraint

‐ Two views of the same object ‐ Suppose I know the camera positions and camera matrices ‐ Given a point on left image, how can I find the corresponding point on right image?

Fei-Fei Li Lecture 9 - 16 21‐Oct‐11 Epipolar Constraint

•Potential matches for p have to lie on the corresponding epipolar line l’.

•Potential matches for p’ have to lie on the corresponding epipolar line l.

Fei-Fei Li Lecture 9 - 17 21‐Oct‐11 Epipolar Constraint P

u u   p  M P  v      p  M P  v  1   1  p’ p

O R, T O’

M  KI 0 M' KR T

Fei-Fei Li Lecture 9 - 18 21‐Oct‐11 Epipolar Constraint P

p’ p

O R, T O’

K1 and K2 are known M  KI 0 (calibrated cameras) M' KR T

M  I 0 M ' R T 

Fei-Fei Li Lecture 9 - 19 21‐Oct‐11 Epipolar Constraint P

p’ p

O R, T O’

 T T (R p ) p  T(R p)  0 Perpendicular to epipolar plane

Fei-Fei Li Lecture 9 - 20 21‐Oct‐11 as matrix multiplication

 0  az ay bx    ab  a 0  a b   [a ]b  z x  y       ay ax 0 bz 

“skew symmetric matrix”

Fei-Fei Li Lecture 9 - 21 21‐Oct‐11 Triangulation P

p’ p

O R, T O’

T T p  T(R p)  0  p  T R p  0 (Longuet‐Higgins, 1981) E = essential matrix

Fei-Fei Li Lecture 9 - 22 21‐Oct‐11 Triangulation P

p 1 p2 l1 l2 e e1 2

O1 O2

• E p2 is the epipolar line associated with p2 (l1 = E p2) T T • E p1 is the epipolar line associated with p1 (l2 = E p1) • E is singular (rank two) T • E e2 = 0 and E e1 = 0 • E is 3x3 matrix; 5 DOF

Fei-Fei Li Lecture 9 - 23 21‐Oct‐11 Triangulation P

p’ p

O O’

u P  M P  p  M  KI 0 v   unknown

Fei-Fei Li Lecture 9 - 24 21‐Oct‐11 Triangulation

P The image part with relationship ID rId8 was not found in the file. p  K 1 p

p’ p

O O’ T 1 T 1  p  T R p  0  (K p)  T R K p  0

T T 1  pT F p  0 p K  T R K p  0

Fei-Fei Li Lecture 9 - 25 21‐Oct‐11 Triangulation P

p’ p

O O’ pT F p  0

F = Fundamental Matrix (Faugeras and Luong, 1992)

Fei-Fei Li Lecture 9 - 26 21‐Oct‐11 Triangulation P

p 1 p2

e e1 2

O1 O2

• F p2 is the epipolar line associated with p2 (l1 = F p2) T T • F p1 is the epipolar line associated with p1 (l2 = F p1) • F is singular (rank two) T • F e2 = 0 and F e1 = 0 • F is 3x3 matrix; 7 DOF

Fei-Fei Li Lecture 9 - 27 21‐Oct‐11 Why is F useful?

T p l’ = F p

‐ Suppose F is known ‐ No additional information about the scene and camera is given ‐ Given a point on left image, how can I find the corresponding point on right image?

Fei-Fei Li Lecture 9 - 28 21‐Oct‐11 Why is F useful?

•F captures information about the epipolar geometry of 2 views + camera parameters

•MORE IMPORTANTLY: F gives constraints on how the scene changes under view point transformation (without reconstructing the scene!) •Powerful tool in: •3D reconstruction •Multi‐view object/scene matching

Fei-Fei Li Lecture 9 - 29 21‐Oct‐11 What we will learn today?

•Why is stereo useful? • Epipolar constraints • Essential and fundamental matrix •Estimating F (Problem Set 2 (Q2)) • Rectification

Reading: [HZ] Chapters: 4, 9, 11 [FP] Chapters: 10

Fei-Fei Li Lecture 9 - 30 21‐Oct‐11 Estimating F The Eight‐Point Algorithm (Longuet‐Higgins, 1981) P (Hartley, 1995)

p’ p

O O’ u u      T P  p  v P  p  v  p F p  0 1  1 

Fei-Fei Li Lecture 9 - 31 21‐Oct‐11 Estimating F pT F p  0

Let’s take 8 corresponding points

Fei-Fei Li Lecture 9 - 32 21‐Oct‐11 Estimating F

Fei-Fei Li Lecture 9 - 33 21‐Oct‐11 Estimating F

W f • Homogeneous system W f  0 •Rank 8 A non‐zero solution exists (unique) •If N>8 Lsq. solution by SVD! ˆ F f 1

Fei-Fei Li Lecture 9 - 34 21‐Oct‐11 Estimating F pT Fˆ p  0 The estimated Fmay^^ have full rank (det(F) ≠0) (F should have rank=2 instead)

Find F that minimizes F  Fˆ  0

Frobenius norm (*) Subject to det(F)=0

SVD (again!) can be used to solve this problem (*) Sqrt root of the sum pf squares of all entries

Fei-Fei Li Lecture 9 - 35 21‐Oct‐11 Example Data courtesy of R. Mohr and B. Boufama. of R. Mohr and Data courtesy

Fei-Fei Li Lecture 9 - 36 21‐Oct‐11 Example

Mean errors: 10.0pixel 9.1pixel

Fei-Fei Li Lecture 9 - 37 21‐Oct‐11 Normalization

Is the accuracy in estimating F function of the ref. system in the image plane?

E.g. under similarity transformation (T = scale + translation):

qi  Ti pi qi  Ti pi

Does the accuracy in estimating F change if a transformation T is applied?

Fei-Fei Li Lecture 9 - 38 21‐Oct‐11 Normalization

•The accuracy in estimating F does change if a transformation T is applied

•There exists a T for which accuracy is maximized Why?

Fei-Fei Li Lecture 9 - 39 21‐Oct‐11 Normalization

Lsq solution W f  0, by SVD f 1 F

‐ SVD enforces Rank(W)=8 ‐ Recall the structure of W: Highly un‐balance (not well conditioned) ‐Values of W must have similar magnitude

More details HZ pag 108

Fei-Fei Li Lecture 9 - 40 21‐Oct‐11 Normalization

IDEA: Transform image coordinate system (T = translation + scaling) such that:

•Origin = centroid of image points •Mean square distance of the data points from origin is 2 pixels

qi  Ti pi qi  Ti pi (normalization)

Fei-Fei Li Lecture 9 - 41 21‐Oct‐11 The Normalized Eight‐Point Algorithm

0. Compute Ti and Ti’ 1. Normalize coordinates:

qi  Ti pi qi  Ti pi

2. Use the eight‐point algorithm to compute F’q from the points q and q’ . ii

T q Fq q  0 3. Enforce the rank‐2 constraint.  Fq det( Fq )  0 T 4. De‐normalize Fq: F  T FqT

Fei-Fei Li Lecture 9 - 42 21‐Oct‐11 Example

Mean errors: 10.0pixel

9.1pixel Without transformation

Mean errors: 1.0pixel 0.9pixel

With transformation

Fei-Fei Li Lecture 9 - 43 21‐Oct‐11 What we will learn today?

•Why is stereo useful? • Epipolar constraints • Essential and fundamental matrix •Estimating F • Rectification

Reading: [HZ] Chapters: 4, 9, 11 [FP] Chapters: 10

Fei-Fei Li Lecture 9 - 44 21‐Oct‐11 Rectification P

p p’

OO’ • Make two camera images “parallel” • Correspondence problem becomes easier

Fei-Fei Li Lecture 9 - 45 21‐Oct‐11 Rectification P

v v e 2 p p’ e2 y u u z KK O x O • Parallel epipolar lines • Epipoles at infinity Let’s see why…. • v = v’

Fei-Fei Li Lecture 9 - 46 21‐Oct‐11 Rectification P

v v e 2 p p’ e2 y u u z KK O x O

K1=K2 = known E  [t ]R x parallel to O1O2

Fei-Fei Li Lecture 9 - 47 21‐Oct‐11 Cross product as matrix multiplication

 0  az ay bx    ab  a 0  a b   [a ]b  z x  y       ay ax 0 bz 

Fei-Fei Li Lecture 9 - 48 21‐Oct‐11 Rectification P

v v e 2 p p’ e2 y u u z KK O x O 0 0 0  K =K = known   1 2 E  [t ]R  0 0 T  v = v’?    x parallel to O1O2 0 T 0 

Fei-Fei Li Lecture 9 - 49 21‐Oct‐11 Rectification P

v v e 2 p p’ e2 y u u z KKpTE p  0 O x O 0 0 0 u  0          u v 1 0 0 T  v   0 u v 1  T   0 Tv  Tv     0 T 0  1   Tv   v = v’

Fei-Fei Li Lecture 9 - 50 21‐Oct‐11 Rectification

H p p’

O O’

GOAL of rectification : Estimate a perspective transformation H that makes images parallel Impose v’=v • This leaves degrees of freedom for determining H • If not appropriate H is chosen, severe projective distortions on image take place • We impose a number of restriction while computing H [HZ] Chapters: 11 (sec. 11.12)

Fei-Fei Li Lecture 9 - 51 21‐Oct‐11 Rectification

H

Courtesy figure S. Lazebnik

Fei-Fei Li Lecture 9 - 52 21‐Oct‐11 Application: view morphing S. M. Seitz and C. R. Dyer, Proc. SIGGRAPH 96, 1996, 21‐30

Fei-Fei Li Lecture 9 - 53 21‐Oct‐11 Application: view morphing

If rectification is not applied, the morphing procedure does not generate geometrically correct interpolations

Fei-Fei Li Lecture 9 - 54 21‐Oct‐11 Application: view morphing

Fei-Fei Li Lecture 9 - 55 21‐Oct‐11 Application: view morphing

Fei-Fei Li Lecture 9 - 56 21‐Oct‐11 Application: view morphing

Fei-Fei Li Lecture 9 - 57 21‐Oct‐11 Application: view morphing

Fei-Fei Li Lecture 9 - 58 21‐Oct‐11 The Fundamental Matrix Song http://danielwedge.com/fmatrix/

Fei-Fei Li Lecture 9 - 59 21‐Oct‐11 What we have learned today?

•Why is stereo useful? • Epipolar constraints • Essential and fundamental matrix •Estimating F (Problem Set 2 (Q2)) • Rectification

Reading: [HZ] Chapters: 4, 9, 11 [FP] Chapters: 10

Fei-Fei Li Lecture 9 - 60 21‐Oct‐11 Supplementary materials

Fei-Fei Li Lecture 9 - 61 21‐Oct‐11 Making image planes parallel

P  KI 0 P P P  KR T P

p p’

K ee’ K’

OO’R,T

0. Compute epipoles

T T e  K R T  [e1 e2 1] e  K T Fei-Fei Li Lecture 9 - 62 21‐Oct‐11 Making image planes parallel P

p p’

e e’

OO’

1. Map e to the x‐axis at location [1,0,1]T (normalization)

T e  [e1 e2 1]  H1  RHTH 1 0 1 T Fei-Fei Li Lecture 9 - 63 21‐Oct‐11 Making image planes parallel P

p p’

e

OO’ 2. Send epipole to infinity: e  1 0 1T  1 0 0T  1 0 0   Minimizes the distortion in a H2  0 1 0   neighborhood (approximates id. mapping) 1 0 1 Fei-Fei Li   Lecture 9 - 64 21‐Oct‐11 Making image planes parallel P

H = H2 H1

e’ e p p’

OO’

3. Define: H = H2 H1 4. Align epipolar lines

Fei-Fei Li Lecture 9 - 65 21‐Oct‐11 Projective transformation of a line (in 2D)

A t  H     v b

l HT l

Fei-Fei Li Lecture 9 - 66 21‐Oct‐11 Making image planes parallel P

H = H2 H1 p’ p l l e’ e

OO’ 3. Define: H = H2 H1 H Tl H T l 4. Align epipolar lines These are called matched pair of transformation [HZ] Chapters: 11 (sec. 11.12)

Fei-Fei Li Lecture 9 - 67 21‐Oct‐11