Announcement

Homework 4 has been posted online and in Blackboard. Due: 11:59pm EST, Thursday, April 1st Announcement

Quiz #5 is available in Blackboard. Due date: 11:59pm EST, Wednesday, March. 24th Open book and open notes Reminder: Final Project Presentation Schedule

Presentation days: April 20 and 22 A single presentation is required for a project Each presentation has 12 + 3 minutes Q&A

Send me an email ([email protected]) that includes: • Your name and ranking of the dates starting from the most preferred • Earlier email has higher priority in choosing the date Today

Early vision on multiple images • Passive Stereo

General strategy: • Camera calibration • Correspondence pages.cs.wisc.edu • Detect a set of tokens (points, lines, and conic curves) • Build correspondence for the tokens in multiple (at least two) images • 3D reconstruction from matched 2D points • Image rectification • 3D triangulation from rectified images courtesy of Dr. Qiang Ji A Simple Case: 3D Triangulation – Rectified Geometry

Rectified geometry: • two image planes are coplanar and parallel to the line 푂1푂2

courtesy of Dr. Qiang Ji A Simple Case: 3D Triangulation – Rectified Geometry

Rectified geometry:

Hence, triangle 푋푈1푈2 is similar to triangle 푋푂1푂2

푍 − 푓 푏 − 푢 + 푢 = 1 2 푍 푏

푓푏 푍 = 푢1 − 푢2 courtesy of Dr. Qiang Ji 3D Triangulation – Rectified Geometry

푓푏 푍 = 푢1 − 푢2 b: base distance 푢1 − 푢2: disparity

푓푏 푢 − 푢 = 1 2 푍 Disparity is inversely proportional to the depth. Establish : there are multiple images taken for the same object. We need to determine which point in one image is corresponding to the point in the others.

courtesy of Dr. Qiang Ji Example of Disparity Map

Head and lamp from University of Tsukuba Stereo Dataset Groundtruth disparity map http://cvlab- home.blogspot.com/2012/05/h2fecha- 2581457116665894170-displaynone.html Establish Pointwise Correspondence using Epipolar Constraint

Epipolar constraint: For a 2D image point 푈푙 in the left image, its corresponding point in the right image must lie on the conjugate epipolar line. Epiploar line is the projection of the 3D line, which passes through 푈푙 and the left camera center, on the right image.

Advantages: • Simplify the searching for the corresponding points from 2D to 1D • improve the efficiency significantly

courtesy of Dr. Qiang Ji Epipolar Geometry

All epipolar lines go through an epipole on one image. The epipole on the left (right) image is the projection of the optical center of the right (left) image on the left (right) image plane.

courtesy of Dr. Qiang Ji Examples Epipolar Geometry

In rectified images,

• the epipolar lines are parallel; The epipole is at infinity • conjugate epipolar lines are collinear and parallel to the base line. Why? Epiploar • The conjugate epipolar lines are plane the intersections of the epipolar plane with the two image planes • The two image planes are coplanar and parallel to the base line Examples Epipolar Geometry

P: a 3D point

푷풍: the coordinate of P on the left camera coordinate system

푷풓: the coordinate of P on the right camera coordinate system 퐓: the relative translation between two camera frames 푷풍 푷풓

풍 풓 푶풓 = 푶풓 + 푻

The right camera The right camera center measured in center measured in the left frame the right frame Epipolar Geometry

P: a 3D point

푷풍: the coordinate of P on the left camera coordinate system

푷풓: the coordinate of P on the right camera coordinate system

푷풍 푷풓 R: the relative rotation between two camera frames (from right frame to left frame)

푷푙 = 퐑푷푟 + 푻

푡 푷푟 = 퐑 푷푙 − 푻 Epipolar Geometry

푷풍 , 푷풍 − 푻 퐚퐧퐝 퐓 are coplanar → in the same epipolar plane 풕 푻 × 푷풍 푷풍 − 푻 = ퟎ Normal of the plane 풕 푻 × 푷풍 퐑푷푟 = ퟎ 푷풍 − 푻 Let 0 −푇푧 푇푦 퐒 = 푇푧 0 −푇푥 −푇푦 푇푥 0

푻 × 푷풍 = 퐒푷풍 Epipolar Geometry

풕 풕 푡 퐒푷풍 퐑푷푟 = ퟎ ↔ 푷풍 퐒 퐑푷푟 = 0

Let 퐄 = 퐒푡퐑

풕 푷풍 퐄푷푟 = ퟎ

E is called the Since rank(퐒) = 2, rank(퐑) = 3 rank(퐄) = 2

The essential matrix • establishes a link between the epipolar constraint and the relative position of the two camera systems • has 5 degree of freedom Fundamental Matrix

푼풍: the homogeneous coordinate of the projection of point P on the left image plane

푼풓: the homogeneous coordinate of the projection of point P on the right image plane

흀푙푼풍 = 퐖푙푷풍 흀푟푼푟 = 퐖푟푷푟

Intrinsic parameters for left and right cameras 푈푙 푈푟 풕 Substitute to 푷풍퐄푷푟 = ퟎ 풕 −풕 −ퟏ 푼풍퐖풍 퐄퐖풓 푼풓 = ퟎ Fundamental Matrix

Fundamental matrix −퐭 −ퟏ A pair of corresponding image 퐅 = 퐖퐥 퐄퐖퐫 points are related by F! 풕 푼풍퐅푼풓 = ퟎ A fundamental matrix encodes • the relative orientation and translation between two camera systems • the intrinsic parameters of the cameras 푈 푈 What is the rank of F? 푙 푟 rank(F)=2 because of E Information from Fundamental Matrix

풕 푼풍퐅푼풓 = ퟎ

풕 풕 풕 Epipolar constraint: 푼풍 퐅푼풓 = ퟎ and 퐅 푼풍 푼풓 = ퟎ

Recall line function: 퐥 ∙ 퐩 = ퟎ, where 퐥 represents the line parameters

퐥푒푙=퐅푼풓

푈푙 푈푟 푼풍 is on a line determined by 퐥푒푙 • Epipolar line on the left image 퐥푒푙 퐥푒푟 • Corresponding to 푼풓 on the right image Information from Fundamental Matrix

풕 푼풍퐅푼풓 = ퟎ

풕 풕 풕 Epipolar constraint:푼풍 퐅푼풓 = ퟎ and 퐅 푼풍 푼풓 = ퟎ Epipolar line on the left image 퐥푒푙 = 퐅푼풓 corresponding to 푼풓 on the right image

Similarly 풕 Epipolar line on the right image 퐥푒푟=퐅 푼풍 corresponding to 푼풍 on the left image 푈푙 푈푟 Given F and a point on one image, we can estimate the 퐥 퐥푒푟 corresponding epipolar line 푒푙 Information from Fundamental Matrix

풕 푼풍 퐅푼풓 = ퟎ Since all epipolar line pass through the epipole, 풆풕 퐥 = ퟎ → 풆풕 퐅풕푼 = 0 푡 풓 푒푟 풓 풍 → 퐅풆풓 푼풍 = 0 풕 풕 푡 풆 퐥푒푙 = ퟎ → 풆 퐅푼풓 = 0 풕 풍 풍 → 퐅 풆풍 푼풓 = 0

퐅풆풓 = 0 퐥푒푟 퐅풕풆 = 0 풍 푈푙 푈푟 퐥푒푙

푒푙 Given F, the epipole of right (left) 푒푟 image is the null eigenvector of F (퐅풕)