Back to the Homography: The Why Sanja Fidler CSC420: Intro to Image Understanding 1/1 Homography In Lecture 9 we said that a homography is a transformation that maps a projective plane to another projective plane. We shamelessly dumped the following equation for homography without explanation: x0 abc x w y = def y 2 03 2 3 2 3 1 ghi 1 4 5 4 5 4 5 Sanja Fidler CSC420: Intro to Image Understanding 2/1 Homography Let’s revisit our transformation in the (new) light of perspective projection. Sanja Fidler CSC420: Intro to Image Understanding 3/1 Homography Let’s revisit our transformation in the (new) light of perspective projection. Figure: We have our object in two di↵erent worlds, in two di↵erent poses relative to camera, two di↵erent photographers, and two di↵erent cameras. Sanja Fidler CSC420: Intro to Image Understanding 3/1 Homography Let’s revisit our transformation in the (new) light of perspective projection. Figure: Our object is a plane. Each plane is characterized by one point d on the plane and two independent vectors a and b on the plane. Sanja Fidler CSC420: Intro to Image Understanding 3/1 Homography Let’s revisit our transformation in the (new) light of perspective projection. Figure: Then any other point X on the plane can be written as: X = d + ↵a + βb. Sanja Fidler CSC420: Intro to Image Understanding 3/1 Homography Let’s revisit our transformation in the (new) light of perspective projection. Figure: Any two Chicken Run DVDs on our planet are related by some transformation T . We’ll compute it, don’t worry. Sanja Fidler CSC420: Intro to Image Understanding 3/1 Homography Let’s revisit our transformation in the (new) light of perspective projection. Figure: Each object is seen by a di↵erent camera and thus projects to the corresponding image plane with di↵erent camera intrinsics. Sanja Fidler CSC420: Intro to Image Understanding 3/1 Homography Let’s revisit our transformation in the (new) light of perspective projection. Figure: Given this, the question is what’s the transformation that maps the DVD on the first image to the DVD in the second image? Sanja Fidler CSC420: Intro to Image Understanding 3/1 We can rewrite this using homogeneous coordinates: ↵ ↵ First plane : X1 = a1 b1 d1 β = A1 β 213 213 ⇥ ⇤ 4↵5 4↵5 Second plane : X2 = a2 b2 d2 β = A2 β 213 213 ⇥ ⇤ 4 5 4 5 Careful: A = a b d and A = a b d are 3 3 matrices. 1 1 1 1 2 2 2 2 ⇥ ⇥ ⇤ ⇥ ⇤ Homography Each point on a plane can be written as: X = d + ↵ a + β b,whered is a point, and a and b are two independent directions on· the plane.· Let’s have two di↵erent planes in 3D: First plane : X = d + ↵ a + β b 1 1 · 1 · 1 Second plane : X = d + ↵ a + β b 2 2 · 2 · 2 Via ↵ and β, the two points X1 and X2 are in the same location relative to each plane. Sanja Fidler CSC420: Intro to Image Understanding 4/1 Careful: A = a b d and A = a b d are 3 3 matrices. 1 1 1 1 2 2 2 2 ⇥ ⇥ ⇤ ⇥ ⇤ Homography Each point on a plane can be written as: X = d + ↵ a + β b,whered is a point, and a and b are two independent directions on· the plane.· Let’s have two di↵erent planes in 3D: First plane : X = d + ↵ a + β b 1 1 · 1 · 1 Second plane : X = d + ↵ a + β b 2 2 · 2 · 2 Via ↵ and β, the two points X1 and X2 are in the same location relative to each plane. We can rewrite this using homogeneous coordinates: ↵ ↵ First plane : X1 = a1 b1 d1 β = A1 β 213 213 ⇥ ⇤ 4↵5 4↵5 Second plane : X2 = a2 b2 d2 β = A2 β 213 213 ⇥ ⇤ 4 5 4 5 Sanja Fidler CSC420: Intro to Image Understanding 4/1 Homography Each point on a plane can be written as: X = d + ↵ a + β b,whered is a point, and a and b are two independent directions on· the plane.· Let’s have two di↵erent planes in 3D: First plane : X = d + ↵ a + β b 1 1 · 1 · 1 Second plane : X = d + ↵ a + β b 2 2 · 2 · 2 Via ↵ and β, the two points X1 and X2 are in the same location relative to each plane. We can rewrite this using homogeneous coordinates: ↵ ↵ First plane : X1 = a1 b1 d1 β = A1 β 213 213 ⇥ ⇤ 4↵5 4↵5 Second plane : X2 = a2 b2 d2 β = A2 β 213 213 ⇥ ⇤ 4 5 4 5 Careful: A = a b d and A = a b d are 3 3 matrices. 1 1 1 1 2 2 2 2 ⇥ Sanja Fidler ⇥ CSC420:⇤ Intro to Image Understanding⇥ ⇤ 4/1 Homography Each point on a plane can be written as: X = d + ↵ a + β b,whered is a point, and a and b are two independent directions on· the plane.· Let’s have two di↵erent planes in 3D: First plane : X = d + ↵ a + β b 1 1 · 1 · 1 Second plane : X = d + ↵ a + β b 2 2 · 2 · 2 Via ↵ and β, the two points X1 and X2 are in the same location relative to each plane. We can rewrite this using homogeneous coordinates: ↵ ↵ First plane : X1 = a1 b1 d1 β = A1 β 213 213 ⇥ ⇤ 4↵5 4↵5 Second plane : X2 = a2 b2 d2 β = A2 β 213 213 ⇥ ⇤ 4 5 4 5 Careful: A = a b d and A = a b d are 3 3 matrices. 1 1 1 1 2 2 2 2 ⇥ Sanja Fidler ⇥ CSC420:⇤ Intro to Image Understanding⇥ ⇤ 4/1 Homography In 3D, a transformation between the planes is given by: X2 = TX1 There is one transformation T between every pair of points X1 and X2. Expand it: ↵ ↵ A2 β = TA1 β for every ↵, β 213 213 4 5 1 4 5 Then it follows: T = A A− ,withT a3 3 matrix. 2 1 ⇥ Let’s look at what happens in projective (image) plane. Note that we have each plane in a separate image and the two images may not have the same camera intrinsic parameters. Denote them with K1 and K2. x1 x2 w1 y1 = K1X1 and w2 y2 = K2X2 2 1 3 2 1 3 4 5 4 5 Sanja Fidler CSC420: Intro to Image Understanding 5/1 Homography From previous slide: x1 x2 w1 y1 = K1X1 and w2 y2 = K2X2 2 1 3 2 1 3 4 5 4 5 Insert X2 = T X1 into equality on the right: x2 w2 y2 = K2 T X1 2 1 3 4 5 Sanja Fidler CSC420: Intro to Image Understanding 6/1 Homography From previous slide: x1 x2 w1 y1 = K1X1 and w2 y2 = K2X2 2 1 3 2 1 3 4 5 4 5 Insert X2 = T X1 into equality on the right: x2 1 w2 y2 = K2 T X1 = K2 T (K1− K1)X1 2 1 3 4 5 Sanja Fidler CSC420: Intro to Image Understanding 6/1 Homography From previous slide: x1 x2 w1 y1 = K1X1 and w2 y2 = K2X2 2 1 3 2 1 3 4 5 4 5 Insert X2 = T X1 into equality on the right: x2 1 w2 y2 = K2 T X1 = K2 T (K1− K1)X1 2 1 3 x 4 5 1 w|12{zy13} 6 1 7 6 7 4 5 Sanja Fidler CSC420: Intro to Image Understanding 6/1 Homography From previous slide: x1 x2 w1 y1 = K1X1 and w2 y2 = K2X2 2 1 3 2 1 3 4 5 4 5 Insert X2 = T X1 into equality on the right: x2 x1 1 1 w2 y2 = K2 T X1 = K2 T (K1− K1)X1 = w1K2 TK1− y1 2 1 3 2 1 3 4 5 4 5 Sanja Fidler CSC420: Intro to Image Understanding 6/1 Homography From previous slide: x1 x2 w1 y1 = K1X1 and w2 y2 = K2X2 2 1 3 2 1 3 4 5 4 5 Insert X2 = T X1 into equality on the right: x2 x1 1 1 w2 y2 = K2 T X1 = K2 T (K1− K1)X1 = w1 K2 TK1− y1 2 1 3 2 1 3 3 3matrix 4 5 ⇥ 4 5 | {z } Sanja Fidler CSC420: Intro to Image Understanding 6/1 Homography From previous slide: x1 x2 w1 y1 = K1X1 and w2 y2 = K2X2 2 1 3 2 1 3 4 5 4 5 Insert X2 = T X1 into equality on the right: x2 x1 1 1 w2 y2 = K2 T X1 = K2 T (K1− K1)X1 = w1 K2 TK1− y1 2 1 3 2 1 3 3 3matrix 4 5 ⇥ 4 5 | {z } And finally: x2 abc x1 w2 y2 = def y1 2 1 3 2ghi3 2 1 3 4 5 4 5 4 5 Sanja Fidler CSC420: Intro to Image Understanding 6/1 Homography The nice thing about homography is that once we have it, we can compute where any point from one projective plane maps to on the second projective plane. We do not need to know the 3D location of that point. We don’t even need to know the camera parameters. We still owe one more explanation for Lecture 9. Sanja Fidler CSC420: Intro to Image Understanding 7/1 Homography The nice thing about homography is that once we have it, we can compute where any point from one projective plane maps to on the second projective plane. We do not need to know the 3D location of that point.