Binocular : Mars Given two images of a scene where relative locations of cameras are known, estimate depth of all common scene points. Stereo Vision II

Introduction to CSE 152 Lecture 14 Two images of Mars

CSE152, Spring 2011 Intro Computer Vision CSE152, Spring 2011 Intro Computer Vision

BINOCULAR STEREO SYSTEM Stereo Vision Outline Estimating Depth\ 2D world with 1-D image plane • Offline: Calibrate cameras & determine (X,Z) Two measurements: XL, XR B “” Z Two unknowns: X,Z • Online Constants: Baseline: d 1. Acquire stereo images Focal length: f C 2. Rectify images to convenient epipolar geometry d XL X =(X - X ) D L R 3. Establish correspondence XL X Z=f R d f A Z = 4. Estimate depth (XL - XR)

(0,0) (d,0) X Disparity: (XL - XR) XL=f(X/Z) XR=f((X-d)/Z)

CSE152, Spring 2011 Intro Computer Vision CSE152, Spring 2011 (Adapted from Hager) Intro Computer Vision

Reconstruction: General 3-D case The search for correspondence: Given two image measurements p and p’, estimate P. Where do you look?

M M’ Truco Fig. 7.5 • Linear Method: find P such that Where M is camera

• Non-Linear Method: find Q minimizing

CSE152, Spring 2011 where q=MQ and q’=M’Q Intro Computer Vision CSE152, Spring 2011 Intro Computer Vision

1 Human Stereopsis: Binocular Fusion Two Approaches How are the correspondences established? 1. Feature-Based Julesz (1971): Is the mechanism for binocular fusion – From each image, process “monocular” image a monocular process or a binocular one?? to obtain image features or cues(e.g., corners, • There is anecdotal evidence for the latter (camouflage). lines). – Establish correspondence between the detected features.

2. Area-Based – Directly compare image regions between the two images. • Random dot stereograms provide an objective answer CSE152, Spring 2011 Intro Computer Vision CSE152, Spring 2011 Intro Computer Vision

Random Dot Stereograms Random Dot Stereograms

CSE152, Spring 2011 Intro Computer Vision CSE152, Spring 2011 Intro Computer Vision

Epipolar Constraint Epipolar Geometry

• Potential matches for p have to lie on the corresponding epipolar line l’. • Baseline • Epipolar Plane • Epipoles • Epipolar Lines • Potential matches for p’ have to lie on the corresponding epipolar line l.

CSE152, Spring 2011 Intro Computer Vision CSE152, Spring 2011 Intro Computer Vision

2 Family of epipolar Planes Skew Symmetric Matrix & • The cross product a x b of two vectors a and b can

be expressed a matrix vector product [ax]b where[ax] is the skew symmetric matrix:

⎡ 0 −a3 a2 ⎤ ⎢ ⎥ a a 0 a [ × ] = ⎢ 3 − 1⎥ ⎣⎢− a2 a1 0 ⎦⎥

O O’ T • A €matrix S is skew symmetric iff S = -S

Family of planes π and lines l and l’ Intersection in e and e’

CSE152, Spring 2011 Intro Computer Vision CSE152, Spring 2011 Intro Computer Vision

Epipolar Constraint: Calibrated Case Two ways to estimate the Essential Matrix 1. Calibration-based 2. Eight-Point Algorithm

Note: Coordinates of p and p’ are “calibrated” so that (u,v,1) is a 3-D vector pointing in the direction of p.

Essential Matrix (Longuet-Higgins, 1981) CSE152, Spring 2011 Intro Computer Vision CSE152, Spring 2011 Intro Computer Vision

Calibration-Based Method The Eight-Point Algorithm (Longuet-Higgins, 1981) " E11 % $ ' E $ 12 ' $ E13 ' $ ' " E11 E12 E13 % " u'% E $ ' $ ' $ 21 ' [u,v,1] E 21 E 22 E 23 v' = 0 [uu' uv' u uu' vv' v u' v' 1]$ E 22 ' = 0 $ ' $ ' $ ' E 23 #$ E 31 E 32 E 33 &' #$ 1 &' $ ' $ E 31 ' • Set E33 to 1 $ ' E 32 • Consider 8 points (ui,vi), (ui’,vi’), i=1..8 $ ' #$ E 33 &'

€ " u1u'1 u1v'1 u1 v1u'1 v1v'1 v1 u'1 v'1 % " E11 % "1 % $ ' $ ' $ ' u u' u v' u v u' v v' v u' v' E 1 $ 2 2 2 2 2 2 2 2 2 2 2 2 ' $ 12 ' $ ' $ u3u'3 u3v'3 u3 v3u'3 v3v'3 v3 €u '3 v'3 ' $ E13 ' $1 ' 1. From image of known calibration fixture, determine intrinsic $ ' $ ' $ ' Solve E to E -- u4 u'4 u4v'4 u4 v4 u'4 v4v'4 v4 u'4 v'4 E 21 1 11 32 parameters and extrinsic relation of two cameras. $ ' $ ' = −$ ' The Essential Matrix $ u5u'5 u5v'5 u5 v5u'5 v5v'5 v5 u'5 v'5 ' $ E 22 ' $1 ' 2. Compute the relative position and orientation of the two $ u u' u v' u v u' v v' v u' v' ' $ E ' $1 ' $ 6 6 6 6 6 6 6 6 6 6 6 6 ' $ 23 ' $ ' cameras from R1, R2, t1, t2 $ u7u'7 u7v'7 u7 v7u'7 v7v'7 v7 u'7 v'7 ' $ E 31 ' $1 ' $ ' $ ' $ ' 3. Compute the Essential Matrix E=[tx]R # u8u'8 u8v'8 u8 v8u'8 v8v'8 v8 u'8 v'8 & # E 32 & #1 & CSE152, Spring 2011 Intro Computer Vision CSE152, Spring 2011 Intro Computer Vision

€

3 Properties of the Essential Matrix The Eight-Point Algorithm (Longuet-Higgins, 1981) Much more on multi-view in CSE252B!!

T • E p’ is the epipolar line associated with p’. T • ETp is the epipolar line associated with p.

• Following, the text, view this as system of homogenous equations in F11 to F33 • E e’=0 and ETe=0. • Solve as Eigenvector corresponding to the smallest Eigenvalue of matrix created from the image data. Minimize: • E is singular. Equivalent to solving n 2 pT Ep • E has two equal non-zero singular values ∑( i i ) (Huang and Faugeras, 1989). i−1 under the constraint 2 CSE152, Spring 2011 Intro Computer Vision CSE152, Spring 2011 |E | =1. Intro Computer Vision €

The Fundamental Matrix Epipolar geometry example The epipolar constraint is given by: where p and p’ are 3-D coordinates of the image coordinates of points in the two images.

Without calibration, we can still identify corresponding points in two images, but we can’t convert to 3-D coordinates. However, the relationship between the calibrated coordinates (p,p’) and uncalibrated image coordinates (q,q’) can be expressed as p=Aq, and p’=A’q’

Therefore, we can express the epipolar constraint as: (Aq)TE(A’q’) = qT(ATEA’)q’ = qTFq’ = 0 where F is called the Fundamental Matrix. Can estimate F using 8 point algorithm WITHOUT CALIBRATION

CSE152, Spring 2011 Intro Computer Vision CSE152, Spring 2011 Intro Computer Vision

Example: converging cameras Example: motion parallel with image plane

(simple for stereo → rectification) CSE152, Spring 2011 courtesy of AndrewIntro Zisserman Computer Vision CSE152, Spring 2011 courtesy of AndrewIntro Zisserman Computer Vision

4 Example: forward motion

e’

e

CSE152, Spring 2011 courtesy of AndrewIntro Zisserman Computer Vision

5