Homography, Transforms, Mosaics Credits: S. Seitz, R. Collins, J. Hays, K. Grauman, C. Choi, C. Brunner, R. Szeliski, L. Zitnick Robert Collins CSE486, Penn StateApplying Homographies to Remove Perspective Distortion from Hartley & Zisserman 4 point correspondences suffice for the planar building facade Robert Collins CSE486, Penn State Homographies for Bird’s-eye Views from Hartley & Zisserman Robert Collins CSE486, Penn State Homographies for Mosaicing from Hartley & Zisserman What are 2D geometric transformations? Translation Preserves: Orientaon Translation and rotation Scale Similarity transformations Similarity transform (4 DoF) = translaon + rotaon + scale Preserves: Angles Aspect ratio Shear Affine transformations Affine transform (6 DoF) = translaon + rotaon + scale + aspect rao + shear Preserves: Parallelism What is missing? Canaleo Are there any other planar transformaons? General affine We already used these How do we compute projecve transformaons? Homogeneous coordinates One extra step: Projective transformations a.k.a. Homographies “keystone” distorons Preserves: Straight Lines Finding the transformation Translaon = 2 degrees of freedom Similarity = 4 degrees of freedom Affine = 6 degrees of freedom Homography = 8 degrees of freedom How many corresponding points do we need to solve? Image warping with homographies image plane in front image plane below Richard Szeliski 17 Finding the transformation - How can we find the transformaon between these images? - How many corresponding points do we need to solve? Planar Homography • A projective mapping from one plane to another • e.g. mapping of points on a ground plane to the image of the camera camera xi xw center image plane world (ground) plane xi Hxw 1 ' H− xi xw '44 Homography Projective – mapping between any two PPs with the same center of projection • rectangle should map to a quadrilateral. – straight lines are conserved. – parallel lines are not conserved. PP2 • same as: project, rotate, reproject. called Homography PP1 p’ H p To apply a homography H • Compute p’ = Hp (regular matrix multiplication) • Convert p’ from homogeneous to image coordinates Planar Homography z key idea: Z = 0 xi xw X x f sc 1000 x x Rt Y y fy cy 0100 2 3 2 3 ' 2 3 2 3 0| 1 Z 1 1 0010 6 1 7 4 5 4 5 4 5 6 7 r r 4 r 5 t X f sc 1000 11 21 31 1 x x r r r t Y = f c 0100 12 22 32 2 y y 2r r r t 3 2Z 3 2 1 3 200103 13 23 33 3 6 00017 6 1 7 4 5 4 5 6 7 6 7 4r r t 5 4 5 f sc 1000 11 21 1 X x x r r t = f c 0100 12 22 2 Y y y 2r r t 3 2 1 3 200103 13 23 3 2 1 3 6 0017 4 5 4 5 6 7 4 5 4 5 homography fx scx r11 r21 t1 X = f c r r t Y matrix 2 y y3 2 12 22 23 2 3 1 r13 r23 t3 1 4 5 4 5 4 5 h11 h21 h31 X = h h h Y 2 12 22 323 2 3 h13 h23 h33 1 4 5 4 5 45 Robert Collins CSE486, Penn State Projection of Planar Points Fitting a homography • Recall: homogeneous coordinates Converting to homogeneous Converting from homogeneous image coordinates image coordinates • Equation for homography: Robert Collins CSE486, Penn State Estimating a Homography Matrix Form: Equations: Robert Collins CSE486, Penn State Degrees of Freedom? There are 9 numbers h11,…,h33 , so are there 9 DOF? No. Note that we can multiply all hij by nonzero k without changing the equations: 10/14/2015 Homography x, y x1, y1 1 1 x2 , y2 x2 , y2 … … xn , yn xn , yn To compute the homography given pairs of corresponding points in the images, we need to set up an equation where the parameters of H are the unknowns… Solving for homographies p’ = Hp wx' a b c x wy' d e f y w g h i 1 Can set scale factor i=1. So, there are 8 unknowns. Set up a system of linear equations: Ah = b where vector of unknowns h = [a,b,c,d,e,f,g,h]T Need at least 8 eqs, but the more the better… Solve for h. If overconstrained, solve using least-squares: min Ah b 2 >> help lmdivide BOARD 13 Image Stitching, Alignment, Blending Image from http://graphics.cs.cmu.edu/courses/15-463/2010_fall/ 10/14/2015 Mosaics . imageSeitz from S. Obtain a wider angle view by combining multiple images. Main questions Alignment: Given two T images, what is the transformation between them? Warping: Given a T source image and a transformation, what does the transformed output look like? 7 Why Mosaic? Are you getting the whole picture? • Compact Camera FOV = 50 x 35° Slide from Brown & Lowe Why Mosaic? Are you getting the whole picture? • Compact Camera FOV = 50 x 35° • Human FOV = 200 x 135° Slide from Brown & Lowe Why Mosaic? Are you getting the whole picture? • Compact Camera FOV = 50 x 35° • Human FOV = 200 x 135° • Panoramic Mosaic = 360 x 180° Slide from Brown & Lowe Image Mosaics + + + virtual wide-angle camera Goal: Stitch together several images into a seamless composite A pencil of rays contains all views real synthetic camera camera Can generate any synthetic camera view as long as it has the same center of projection! Image reprojection mosaic PP The mosaic has a natural interpretation in 3D • The images are reprojected onto a common plane • The mosaic is formed on this plane • Mosaic is a synthetic wide-angle camera How to do it? Basic Procedure 1. Take a sequence of images from the same position. – Rotate the camera about its optical center. 2. Compute transformation between second image and first. 3. Transform the second image to overlap with the first. 4. Blend the two together to create a mosaic. 5. If there are more images, repeat. Full-view Panorama .