Panoramic Image Mosaics Image Stitching Computer Vision CSE 576, Spring 2008 Full screen panoramas (cubic): http://www.panoramas.dk/ Richard Szeliski Mars: http://www.panoramas.dk/fullscreen3/f2_mars97.html Microsoft Research 2003 New Years Eve: http://www.panoramas.dk/fullscreen3/f1.html Richard Szeliski Image Stitching 2 Gigapixel panoramas & images Image Mosaics + + … + = Mapping / Tourism / WWT Medical Imaging Goal: Stitch together several images into a seamless composite Richard Szeliski Image Stitching 3 Richard Szeliski Image Stitching 4 Today’s lecture Readings Image alignment and stitching • Szeliski, CVAA: • motion models • Chapter 3.5: Image warping • Chapter 5.1: Feature-based alignment (in preparation) • image warping • Chapter 8.1: Motion models • point-based alignment • Chapter 8.2: Global alignment • Chapter 8.3: Compositing • complete mosaics (global alignment) • compositing and blending • Recognizing Panoramas, Brown & Lowe, ICCV’2003 • ghost and parallax removal • Szeliski & Shum, SIGGRAPH'97 Richard Szeliski Image Stitching 5 Richard Szeliski Image Stitching 6 Motion models What happens when we take two images with a camera and try to align them? Motion models • translation? • rotation? • scale? • affine? • perspective? … see interactive demo (VideoMosaic) Richard Szeliski Image Stitching 8 Image Warping image filtering: change range of image g(x) = h(f(x)) Image Warping f f h x x image warping: change domain of image g(x) = f(h(x)) f f h x x Richard Szeliski Image Stitching 10 Image Warping Parametric (global) warping image filtering: change range of image Examples of parametric warps: g(x) = h(f(x)) f g h image warping: change domain of image translation rotation aspect g(x) = f(h(x)) f g h perspective affine cylindrical Richard Szeliski Image Stitching 11 Richard Szeliski Image Stitching 12 2D coordinate transformations Image Warping translation: x’ = x + t x = (x,y) Given a coordinate transform x’ = h(x) and a rotation: x’ = R x + t source image f(x), how do we compute a similarity: x’ = s R x + t transformed image g(x’)=f(h(x))? affine: x’ = A x + t perspective: x’ ≅ H x x = (x,y,1) (x is a homogeneous coordinate) These all form a nested group (closed w/ inv.) h(x) xx’f(x) g(x’) Richard Szeliski Image Stitching 13 Richard Szeliski Image Stitching 14 Forward Warping Forward Warping Send each pixel f(x) to its corresponding Send each pixel f(x) to its corresponding location x’ = h(x) in g(x’) location x’ = h(x) in g(x’) • What if pixel lands “between” two pixels? • What if pixel lands “between” two pixels? • Answer: add “contribution” to several pixels, normalize later (splatting) h(x) h(x) xx’f(x) g(x’) xx’f(x) g(x’) Richard Szeliski Image Stitching 15 Richard Szeliski Image Stitching 16 Inverse Warping Inverse Warping Get each pixel g(x’) from its corresponding Get each pixel g(x’) from its corresponding location x’ = h(x) in f(x) location x’ = h(x) in f(x) • What if pixel comes from “between” two pixels? • What if pixel comes from “between” two pixels? •Answer: resample color value from interpolated (prefiltered) source image h(x) xx’f(x) g(x’) xx’f(x) g(x’) Richard Szeliski Image Stitching 17 Richard Szeliski Image Stitching 18 Interpolation Prefiltering Possible interpolation filters: Essential for downsampling (decimation) to • nearest neighbor prevent aliasing • bilinear MIP-mapping [Williams’83]: • bicubic (interpolating) 1. build pyramid (but what decimation filter?): • sinc / FIR – block averaging Needed to prevent “jaggies” – Burt & Adelson (5-tap binomial) and “texture crawl” (see demo) – 7-tap wavelet-based filter (better) 2. trilinear interpolation – bilinear within each 2 adjacent levels – linear blend between levels (determined by pixel size) Richard Szeliski Image Stitching 19 Richard Szeliski Image Stitching 20 Prefiltering Essential for downsampling (decimation) to prevent aliasing Other possibilities: Motion models (reprise) • summed area tables • elliptically weighted Gaussians (EWA) [Heckbert’86] Richard Szeliski Image Stitching 21 Motion models Plane perspective mosaics • 8-parameter generalization of affine motion – works for pure rotation or planar surfaces Translation Affine Perspective 3D rotation • Limitations: – local minima – slow convergence – difficult to control interactively 2 unknowns 6 unknowns 8 unknowns 3 unknowns Richard Szeliski Image Stitching 23 Richard Szeliski Image Stitching 24 Image warping with homographies Rotational mosaics • Directly optimize rotation and focal length • Advantages: – ability to build full-view panoramas – easier to control interactively – more stable and accurate estimates image plane in front image plane below Richard Szeliski Image Stitching 25 Richard Szeliski Image Stitching 26 3D → 2D Perspective Projection Rotational mosaic Projection equations (Xc,Yc,Zc) 1. Project from image to 3D ray f uc (x0,y0,z0) = (u0-uc,v0-vc,f) u 2. Rotate the ray by camera motion (x1,y1,z1) = R01 (x0,y0,z0) 3. Project back into new (source) image (u1,v1)= (fx1/z1+uc,fy1/z1+vc) Richard Szeliski Image Stitching 27 Richard Szeliski Image Stitching 28 Image reprojection Rotations and quaternions How do we represent rotation matrices? 1. Axis / angle (n,θ) 2 R = I + sinθ [n]× + (1- cosθ) [n]× (Rodriguez Formula), with [n]× = cross product matrix (see paper) 2. Unit quaternions [Shoemake SIGG’85] mosaic PP q = (n sinθ/2, cosθ/2) = (w,s) The mosaic has a natural interpretation in 3D quaternion multiplication (division is easy) • The images are reprojected onto a common plane q0 q1 = (s1w0+ s0w1, s0 s1-w0·w1) • The mosaic is formed on this plane Richard Szeliski Image Stitching 29 Richard Szeliski Image Stitching 30 Incremental rotation update Perspective & rotational motion 1. Small angle approximation Solve 8x8 or 3x3 system (see papers for 2 ΔR = I + sinθ [n]× + (1- cosθ) [n]× details), and iterate (non-linear) ≈ θ [n]× = [ω]× Patch-based approximation: linear in ω 1. break up image into patches (say 16x16) 2. Update original R matrix 2. accumulate 2x2 linear system in each R ← R ΔR (local translational assumption) 3. compose larger system from smaller 2x2 results [Shum & Szeliski, ICCV’98] Richard Szeliski Image Stitching 31 Richard Szeliski Image Stitching 32 Image Mosaics (stitching) Blend together several overlapping images into one seamless mosaic (composite) Image Mosaics (Stitching) + + … + = [Szeliski & Shum, SIGGRAPH’97] [Szeliski, FnT CVCG, 2006] Richard Szeliski Image Stitching 34 Mosaics for Video Coding Establishing correspondences Convert masked images into a background sprite 1. Direct method: for content-based coding • Use generalization of affine motion model [Szeliski & Shum ’97] +++ 2. Feature-based method • Extract features, match, find consisten inliers [Lowe ICCV’99; Schmid ICCV’98, = Brown&Lowe ICCV’2003] •Compute R from correspondences (absolute orientation) Richard Szeliski Image Stitching 35 Richard Szeliski Image Stitching 36 Absolute orientation Stitching demo [Arun et al., PAMI 1987] [Horn et al., JOSA A 1988] Procrustes Algorithm [Golub & VanLoan] Given two sets of matching points, compute R pi’= R pi 3D rays T T T T T T A = Σi pi pi’ = Σi pi pi R = U S V = (U S U ) R VT = UT RT R = VUT Richard Szeliski Image Stitching 37 Richard Szeliski Image Stitching 38 Panoramas Cylindrical panoramas What if you want a 360° field of view? Steps mosaic Projection Cylinder • Reproject each image onto a cylinder • Blend • Output the resulting mosaic Richard Szeliski Image Stitching 39 Richard Szeliski Image Stitching 40 Cylindrical Panoramas Determining the focal length Map image to cylindrical or spherical coordinates 1. Initialize from homography H • need known focal length (see text or [SzSh’97]) 2. Use camera’s EXIF tags (approx.) 3. Use a tape measure 4m 1m Image 384x300 f = 180 (pixels) f = 280 f = 380 4. Ask your instructor Richard Szeliski Image Stitching 41 Richard Szeliski Image Stitching 42 3D → 2D Perspective Projection Cylindrical projection • Map 3D point (X,Y,Z) onto cylinder (Xc,Yc,Zc) f uc • Convert to cylindrical coordinates Y Z X u • Convert to cylindrical image coordinates unit cylinder – s defines size of the final image unwrapped cylinder cylindrical image Richard Szeliski Image Stitching 43 Richard Szeliski Image Stitching 44 Cylindrical warping Spherical warping Given focal length f and Given focal length f and image center (xc,yc) image center (xc,yc) φ (X,Y,Z) (x,y,z) cos φ (sinθ,h,cosθ) (sinθcosφ,cosθcosφ,sinφ) Y Y sin φ Z Z X X cos θ cos φ Richard Szeliski Image Stitching 45 Richard Szeliski Image Stitching 46 3D rotation Radial distortion Rotate image before Correct for “bending” in wide field of view lenses placing on unrolled sphere (x,y,z) φ cos φ (sinθcosφ,cosθcosφ,sinφ) sin φ Z Y X cos θ cos φ p = R p _ _ _ _ Richard Szeliski Image Stitching 47 Richard Szeliski Image Stitching 48 Fisheye lens Image Stitching Extreme “bending” in ultra-wide fields of view 1. Align the images over each other • camera pan ↔ translation on cylinder 2. Blend the images together (demo) Richard Szeliski Image Stitching 49 Richard Szeliski Image Stitching 50 Project 2 – image stitching Matching features 1. Take pictures on a tripod (or handheld) 2. Warp images to spherical coordinates 3. Extract features 4. Align neighboring pairs using RANSAC 5. Write out list of neighboring translations 6. Correct for drift 7. Read in warped images and blend them 8. Crop the result and import into a viewer What do we do about the “bad” matches? Richard Szeliski Image Stitching 51 Richard Szeliski Image Stitching 52 RAndom SAmple Consensus RAndom