Epipolar Geometry 16-385 Computer Vision Carnegie Mellon University (Kris Kitani) Tie tiny threads on HERB and pin them to your eyeball What would it look like?
You see points on HERB
What does the second observer see? You see points on HERB
Second person sees lines This is Epipolar Geometry Epipole
Epipolar lines Epipolar geometry
p
o o0
Image plane Epipolar geometry
p
o o0 Baseline
Image plane Epipolar geometry
p
o o0 e Baseline e0
Image plane Epipole (projection of o’ on the image plane) Epipolar geometry
p
Epipolar plane o o0 e Baseline e0
Image plane Epipole (projection of o’ on the image plane) Epipolar geometry
Epipolar line (intersection of Epipolar p plane and image plane)
Epipolar plane l l0 o o0 e Baseline e0
Image plane Epipole (projection of o’ on the image plane) Epipolar constraint
p
l0 x l x0 o o0 e e0
Potential matches for x lie on the epipolar line l0 p
l0 x l x0 o o0 e e0
The point x (left image) maps to a ______in the right image
The baseline connects the ______and ______
An epipolar line (left image) maps to a ______in the right image
An epipole e is a projection of the ______on the image plane All epipolar lines in an image intersect at the ______Converging cameras
Where is the epipole in this image? Converging cameras
Where is the epipole in this image? It’s not always in the image Parallel cameras
Where is the epipole? Parallel cameras
epipole at infinity Forward moving camera Forward moving camera Where is the epipole?
What do the epipolar lines look like? Epipole has same coordinates in both images. Points move along lines radiating from “Focus of expansion”
e0
e The epipolar constraint is an important concept for stereo vision
Task: Match point in left image to point in right image
Left image Right image
How would you do it? The epipolar constraint is an important concept for stereo vision
Task: Match point in left image to point in right image
Left image Right image
Want to avoid search over entire image
(if the images have been rectified) Epipolar constrain reduces search to a single line
E
Essential Matrix 16-385 Computer Vision Carnegie Mellon University (Kris Kitani) Recall:Epipolar constraint
p
l0 x l x0 o o0 e e0
Potential matches for x lie on the epipolar line l0 The epipolar geometry is an important concept for stereo vision
Task: Match point in left image to point in right image
Left image Right image
How would you do it? The epipolar constraint is an important concept for stereo vision
Task: Match point in left image to point in right image
Left image Right image
Epipolar constrain reduces search to a single line
How do you compute the epipolar line? Essential Matrix E The Essential Matrix is a 3 x 3 matrix that encodes epipolar geometry Given a point in one image, multiplying by the essential matrix will tell us the epipolar line in the second view.
Ex = l0 X
l0 x x0 o o0 e e0 Representing the … Epipolar Line
a ax + by + c =0 in vector form l = b 2 c 3 4 5 l If the point x is on the epipolar line l then e
x x>l =? Epipolar Line
a ax + by + c =0 in vector form l = b 2 c 3 4 5 l If the point x is on the epipolar line l then e
x x>l =0 Recall: Dot Product c = a b ⇥
b
a
c a =0 c b =0 · · So if x > l =0 and E x = l 0 then
x0>Ex =?
X
l0 x x0 o o0 e e0 So if x > l =0 and E x = l 0 then
x0>Ex =0
X
l0 x x0 o o0 e e0 Motivation
The Essential Matrix is a 3 x 3 matrix that encodes epipolar geometry
Given a point in one image, multiplying by the essential matrix will tell us the epipolar line in the second view. Essential Matrix vs Homography
What’s the difference between the essential matrix and a homography? Essential Matrix vs Homography
What’s the difference between the essential matrix and a homography?
They are both 3 x 3 matrices but … Essential Matrix vs Homography
What’s the difference between the essential matrix and a homography?
They are both 3 x 3 matrices but …
l0 = Ex x0 = Hx
Essential matrix maps a Homography maps a point to a line point to a point Where does the Essential matrix come from? X
x x0 o o0 t
R, t
x0 = R(x t) X
x x0 o o0 t
R, t
x0 = R(x t) Does this look familiar? X
x x0 o o0 t
R, t
x0 = R(x t) Camera-camera transform just like world-camera transform X
x x0 o o0 t
These three vectors are coplanar x, t, x0 X
x x0 o o0 t
If these three vectors are coplanar x , t , x 0 then x>(t x)= ? ⇥ X
x x0 o o0 t
If these three vectors are coplanar x , t , x 0 then x>(t x)=0 ⇥ Recall: Cross Product Vector (cross) product takes two vectors and returns a vector perpendicular to both
c = a b ⇥
b
a
c a =0 c b =0 · · X
x x0 o o0 t
If these three vectors are coplanar x , t , x 0 then
(x t)>(t x)= ? ⇥ X
x x0 o o0 t
If these three vectors are coplanar x , t , x 0 then
(x t)>(t x)=0 ⇥ putting it together
rigid motion coplanarity x0 = R(x t) (x t)>(t x)=0 ⇥ (x0>R)(t x)=0 ⇥ putting it together
rigid motion coplanarity x0 = R(x t) (x t)>(t x)=0 ⇥ (x0>R)(t x)=0 ⇥ (x0>R)([t ]x)=0 ⇥ Cross product a b a b 2 3 3 2 a b = a3b1 a1b3 ⇥ 2 a b a b 3 1 2 2 1 4 5
Can also be written as a matrix multiplication
0 a a b 3 2 1 a b =[a] b = a3 0 a1 b2 ⇥ ⇥ 2 a a 0 3 2 b 3 2 1 3 4 Skew symmetric 5 4 5 putting it together
rigid motion coplanarity x0 = R(x t) (x t)>(t x)=0 ⇥ (x0>R)(t x)=0 ⇥ (x0>R)([t ]x)=0 ⇥ x0>(R[t ])x =0 ⇥ putting it together
rigid motion coplanarity x0 = R(x t) (x t)>(t x)=0 ⇥ (x0>R)(t x)=0 ⇥ (x0>R)([t ]x)=0 ⇥ x0>(R[t ])x =0 ⇥
x0>Ex =0 putting it together
rigid motion coplanarity x0 = R(x t) (x t)>(t x)=0 ⇥ (x0>R)(t x)=0 ⇥ (x0>R)([t ]x)=0 ⇥ x0>(R[t ])x =0 ⇥
Essential Matrix" x0>Ex =0 [Longuet-Higgins 1981] properties of the E matrix
Longuet-Higgins equation x0>Ex =0
(points in normalized coordinates) properties of the E matrix
Longuet-Higgins equation x0>Ex =0
x>l =0 x0>l0 =0 Epipolar lines T l0 = Ex l = E x0
(points in normalized coordinates) properties of the E matrix
Longuet-Higgins equation x0>Ex =0
x>l =0 x0>l0 =0 Epipolar lines T l0 = Ex l = E x0
Epipoles e0>E = 0 Ee = 0
(points in normalized coordinates) How do you generalize to uncalibrated cameras? F
Fundamental Matrix 16-385 Computer Vision Recall:Epipolar constraint
p
l0 x l x0 o o0 e e0
Potential matches for x lie on the epipolar line l0 Given a point in one image, multiplying by the essential matrix will tell us the epipolar line in the second view.
Ex = l0 X
l0 x x0
o o0 e e0
Assumption: " points aligned to camera coordinate axis (calibrated camera) The Fundamental matrix " is a generalization of the Essential matrix, " where the assumption of calibrated cameras is removed xˆ0>Exˆ =0
The Essential matrix operates on image points expressed in normalized coordinates (points have been aligned (normalized) to camera coordinates) 1 ˆ 1 xˆ = K x x0 = K x0 camera image point point
Writing out the epipolar constraint in terms of image coordinates 1 x0>K0 >EK x =0 1 x0>(K0 >EK )x =0 x0>Fx =0 Same equation works in image coordinates!
x0>Fx =0 it maps pixels to epipolar lines ( properties of the E matrix
Longuet-Higgins equation x0>E(x =0
x>l =0 x0>l0 =0 Epipolar lines T l0 = (Ex l = (E x0
Epipoles e0>E( = 0 (Ee = 0
(points in image coordinates) Breaking down the fundamental matrix 1 F = K0 >EK 1 F = K0 >[t ]RK ⇥ Depends on both intrinsic and extrinsic parameters Breaking down the fundamental matrix 1 F = K0 >EK 1 F = K0 >[t ]RK ⇥ Depends on both intrinsic and extrinsic parameters
How would you solve for F?
xm0>Fxm =0