Panoramic Image Mosaics

Image Stitching

Computer Vision CSE 576, Spring 2008 Full screen panoramas (cubic): http://www.panoramas.dk/ Richard Szeliski Mars: http://www.panoramas.dk/fullscreen3/f2_mars97.html Microsoft Research 2003 New Years Eve: http://www.panoramas.dk/fullscreen3/f1.html

Richard Szeliski Image Stitching 2

Gigapixel panoramas & images Image Mosaics

+ + … + =

Mapping / Tourism / WWT

Medical Imaging

Goal: Stitch together several images into a seamless composite Richard Szeliski Image Stitching 3 Richard Szeliski Image Stitching 4 Today’s lecture Readings

Image alignment and stitching • Szeliski, CVAA: • motion models • Chapter 3.5: Image warping • Chapter 5.1: Feature-based alignment (in preparation) • image warping • Chapter 8.1: Motion models • point-based alignment • Chapter 8.2: Global alignment • Chapter 8.3: Compositing • complete mosaics (global alignment) • compositing and blending • Recognizing Panoramas, Brown & Lowe, ICCV’2003 • ghost and parallax removal • Szeliski & Shum, SIGGRAPH'97

Richard Szeliski Image Stitching 5 Richard Szeliski Image Stitching 6

Motion models

What happens when we take two images with a camera and try to align them? Motion models • translation? • rotation? • scale? • affine? • perspective? … see interactive demo (VideoMosaic)

Richard Szeliski Image Stitching 8 Image Warping

image filtering: change range of image g(x) = h(f(x)) Image Warping f f h x x image warping: change domain of image g(x) = f(h(x))

f f h

x x Richard Szeliski Image Stitching 10

Image Warping Parametric (global) warping image filtering: change range of image Examples of parametric warps: g(x) = h(f(x)) f g h image warping: change domain of image translation rotation aspect g(x) = f(h(x))

f g h perspective affine cylindrical Richard Szeliski Image Stitching 11 Richard Szeliski Image Stitching 12 2D coordinate transformations Image Warping translation: x’ = x + t x = (x,y) Given a coordinate transform x’ = h(x) and a rotation: x’ = R x + t source image f(x), how do we compute a similarity: x’ = s R x + t transformed image g(x’)=f(h(x))? affine: x’ = A x + t perspective: x’ ≅ H x x = (x,y,1) (x is a homogeneous coordinate) These all form a nested group (closed w/ inv.) h(x)

xx’f(x) g(x’)

Richard Szeliski Image Stitching 13 Richard Szeliski Image Stitching 14

Forward Warping Forward Warping

Send each pixel f(x) to its corresponding Send each pixel f(x) to its corresponding location x’ = h(x) in g(x’) location x’ = h(x) in g(x’) • What if pixel lands “between” two pixels? • What if pixel lands “between” two pixels? • Answer: add “contribution” to several pixels, normalize later (splatting)

h(x) h(x)

xx’f(x) g(x’) xx’f(x) g(x’)

Richard Szeliski Image Stitching 15 Richard Szeliski Image Stitching 16 Inverse Warping Inverse Warping

Get each pixel g(x’) from its corresponding Get each pixel g(x’) from its corresponding location x’ = h(x) in f(x) location x’ = h(x) in f(x) • What if pixel comes from “between” two pixels? • What if pixel comes from “between” two pixels? •Answer: resample color value from interpolated (prefiltered) source image

h(x)

xx’f(x) g(x’) xx’f(x) g(x’)

Richard Szeliski Image Stitching 17 Richard Szeliski Image Stitching 18

Interpolation Prefiltering

Possible interpolation filters: Essential for downsampling (decimation) to • nearest neighbor prevent aliasing • bilinear MIP-mapping [Williams’83]: • bicubic (interpolating) 1. build pyramid (but what decimation filter?): • sinc / FIR – block averaging Needed to prevent “jaggies” – Burt & Adelson (5-tap binomial) and “texture crawl” (see demo) – 7-tap wavelet-based filter (better) 2. trilinear interpolation – bilinear within each 2 adjacent levels – linear blend between levels (determined by pixel size)

Richard Szeliski Image Stitching 19 Richard Szeliski Image Stitching 20 Prefiltering

Essential for downsampling (decimation) to prevent aliasing Other possibilities: Motion models (reprise) • summed area tables • elliptically weighted Gaussians (EWA) [Heckbert’86]

Richard Szeliski Image Stitching 21

Motion models Plane perspective mosaics

• 8-parameter generalization of affine motion – works for pure rotation or planar surfaces

Translation Affine Perspective 3D rotation • Limitations: – local minima – slow convergence – difficult to control interactively

2 unknowns 6 unknowns 8 unknowns 3 unknowns

Richard Szeliski Image Stitching 23 Richard Szeliski Image Stitching 24 Image warping with homographies Rotational mosaics

• Directly optimize rotation and focal length • Advantages: – ability to build full-view panoramas – easier to control interactively – more stable and accurate estimates

image plane in front image plane below Richard Szeliski Image Stitching 25 Richard Szeliski Image Stitching 26

3D → 2D Perspective Projection Rotational mosaic

Projection equations (Xc,Yc,Zc) 1. Project from image to 3D ray f uc (x0,y0,z0) = (u0-uc,v0-vc,f) u 2. Rotate the ray by camera motion

(x1,y1,z1) = R01 (x0,y0,z0) 3. Project back into new (source) image

(u1,v1)= (fx1/z1+uc,fy1/z1+vc)

Richard Szeliski Image Stitching 27 Richard Szeliski Image Stitching 28 Image reprojection Rotations and quaternions

How do we represent rotation matrices? 1. Axis / angle (n,θ) 2 R = I + sinθ [n]× + (1- cosθ) [n]× (Rodriguez Formula), with

[n]× = cross product matrix (see paper) 2. Unit quaternions [Shoemake SIGG’85] mosaic PP q = (n sinθ/2, cosθ/2) = (w,s) The mosaic has a natural interpretation in 3D quaternion multiplication (division is easy)

• The images are reprojected onto a common plane q0 q1 = (s1w0+ s0w1, s0 s1-w0·w1) • The mosaic is formed on this plane Richard Szeliski Image Stitching 29 Richard Szeliski Image Stitching 30

Incremental rotation update Perspective & rotational motion

1. Small angle approximation Solve 8x8 or 3x3 system (see papers for 2 ΔR = I + sinθ [n]× + (1- cosθ) [n]× details), and iterate (non-linear) ≈ θ [n]× = [ω]× Patch-based approximation: linear in ω 1. break up image into patches (say 16x16) 2. Update original R matrix 2. accumulate 2x2 linear system in each R ← R ΔR (local translational assumption) 3. compose larger system from smaller 2x2 results [Shum & Szeliski, ICCV’98]

Richard Szeliski Image Stitching 31 Richard Szeliski Image Stitching 32 Image Mosaics (stitching)

Blend together several overlapping images into one seamless mosaic (composite) Image Mosaics (Stitching)

+ + … + = [Szeliski & Shum, SIGGRAPH’97] [Szeliski, FnT CVCG, 2006]

Richard Szeliski Image Stitching 34

Mosaics for Video Coding Establishing correspondences Convert masked images into a background sprite 1. Direct method: for content-based coding • Use generalization of affine motion model [Szeliski & Shum ’97] +++ 2. Feature-based method • Extract features, match, find consisten inliers [Lowe ICCV’99; Schmid ICCV’98, = Brown&Lowe ICCV’2003] •Compute R from correspondences (absolute orientation)

Richard Szeliski Image Stitching 35 Richard Szeliski Image Stitching 36 Absolute orientation Stitching demo

[Arun et al., PAMI 1987] [Horn et al., JOSA A 1988] Procrustes Algorithm [Golub & VanLoan]

Given two sets of matching points, compute R

pi’= R pi 3D rays T T T T T T A = Σi pi pi’ = Σi pi pi R = U S V = (U S U ) R VT = UT RT R = VUT

Richard Szeliski Image Stitching 37 Richard Szeliski Image Stitching 38

Panoramas Cylindrical panoramas

What if you want a 360° field of view?

Steps mosaic Projection Cylinder • Reproject each image onto a cylinder • Blend • Output the resulting mosaic

Richard Szeliski Image Stitching 39 Richard Szeliski Image Stitching 40 Cylindrical Panoramas Determining the focal length

Map image to cylindrical or spherical coordinates 1. Initialize from homography H • need known focal length (see text or [SzSh’97]) 2. Use camera’s EXIF tags (approx.) 3. Use a tape measure

4m 1m

Image 384x300 f = 180 (pixels) f = 280 f = 380 4. Ask your instructor

Richard Szeliski Image Stitching 41 Richard Szeliski Image Stitching 42

3D → 2D Perspective Projection Cylindrical projection • Map 3D point (X,Y,Z) onto cylinder (Xc,Yc,Zc)

f uc • Convert to cylindrical coordinates Y Z X u • Convert to cylindrical image coordinates unit cylinder – s defines size of the final image

unwrapped cylinder cylindrical image Richard Szeliski Image Stitching 43 Richard Szeliski Image Stitching 44 Cylindrical warping Spherical warping

Given focal length f and Given focal length f and image center (xc,yc) image center (xc,yc) φ (X,Y,Z) (x,y,z) cos φ

(sinθ,h,cosθ) (sinθcosφ,cosθcosφ,sinφ) Y Y sin φ Z Z X X cos θ cos φ

Richard Szeliski Image Stitching 45 Richard Szeliski Image Stitching 46

3D rotation Radial distortion

Rotate image before Correct for “bending” in wide field of view lenses placing on unrolled sphere

(x,y,z) φ cos φ (sinθcosφ,cosθcosφ,sinφ) sin φ Z Y X cos θ cos φ p = R p _ _

_ _

Richard Szeliski Image Stitching 47 Richard Szeliski Image Stitching 48 Fisheye lens Image Stitching

Extreme “bending” in ultra-wide fields of view 1. Align the images over each other • camera pan ↔ translation on cylinder 2. Blend the images together (demo)

Richard Szeliski Image Stitching 49 Richard Szeliski Image Stitching 50

Project 2 – image stitching Matching features

1. Take pictures on a tripod (or handheld) 2. Warp images to spherical coordinates 3. Extract features 4. Align neighboring pairs using RANSAC 5. Write out list of neighboring translations 6. Correct for drift 7. Read in warped images and blend them 8. Crop the result and import into a viewer What do we do about the “bad” matches?

Richard Szeliski Image Stitching 51 Richard Szeliski Image Stitching 52 RAndom SAmple Consensus RAndom SAmple Consensus

Select one match, count inliers Select one match, count inliers

Richard Szeliski Image Stitching 53 Richard Szeliski Image Stitching 54

Least squares fit Assembling the panorama

Stitch pairs together, blend, then crop

Find “average” translation vector

Richard Szeliski Image Stitching 55 Richard Szeliski Image Stitching 56 Problem: Drift Problem: Drift

(x1,y1)

(xn,yn)

copy of first image Solution • add another copy of first image at the end Error accumulation • this gives a constraint: yn = y1 • small (vertical) errors accumulate over time • there are a bunch of ways to solve this problem

• apply correction so that sum = 0 (for 360° pan.) – add displacement of (y1 –yn)/(n -1) to each image after the first

Richard Szeliski Image Stitching 57 Richard Szeliski– compute a global warp: Image Stitching y’ = y + ax 58 – run a big optimization problem, incorporating this tit

Full-view Panorama

Full-view (360° spherical) + + panoramas + +

Richard Szeliski Image Stitching 60 Texture Mapped Model Global alignment

• Register all pairwise overlapping images • Use a 3D rotation model (one R per image) • Use direct alignment (patch centers) or feature based • Infer overlaps based on previous matches (incremental) • Optionally discover which images overlap other images using feature selection (RANSAC)

Richard Szeliski Image Stitching 61 Richard Szeliski Image Stitching 62

Recognizing Panoramas

Recognizing Panoramas

[Brown & Lowe, Matthew Brown & David Lowe ICCV’03] ICCV’2003

Richard Szeliski Image Stitching 64 Finding the panoramas Finding the panoramas

Richard Szeliski Image Stitching 65 Richard Szeliski Image Stitching 66

Finding the panoramas Finding the panoramas

Richard Szeliski Image Stitching 67 Richard Szeliski Image Stitching 68 Fully automated 2D stitching Get you own free copy

Demo

http://get.live.com/photogallery/overview

Richard Szeliski Image Stitching 69 Richard Szeliski Image Stitching 70

Rec.pano.: system components Multi-Scale Oriented Patches

1. Feature detection and description Interest points • more uniform point density • Multi-scale Harris corners 2. Fast matching (hash table) • Orientation from blurred gradient 3. RANSAC filtering of matches • Geometrically invariant to similarity transforms 4. Intensity-based verification Descriptor vector • Bias/gain normalized sampling of local patch (8x8) 5. Incremental bundle adjustment • Photometrically invariant to affine changes in intensity [M. Brown, R. Szeliski, and S. Winder. Multi-image matching using multi-scale oriented patches, CVPR'2005]

Richard Szeliski Image Stitching 71 Richard Szeliski Image Stitching 72 Feature irregularities Descriptor Vector

Distribute points evenly over the image Orientation = blurred gradient Similarity Invariant Frame • Scale-space position (x, y, s) + orientation (θ)

Richard Szeliski Image Stitching 73 Richard Szeliski Image Stitching 74

Probabilistic Feature Matching RANSAC motion model

Richard Szeliski Image Stitching 75 Richard Szeliski Image Stitching 76 RANSAC motion model RANSAC motion model

Richard Szeliski Image Stitching 77 Richard Szeliski Image Stitching 78

Probabilistic model for verification

How well does this work?

Test on 100s of examples…

Richard Szeliski Image Stitching 79 Matching Mistakes: False Positive

How well does this work?

Test on 100s of examples…

…still too many failures (5-10%) for consumer application

Richard Szeliski Image Stitching 82

Matching Mistakes: False Positive Matching Mistake: False Negative

Moving objects: large areas of disagreement

Richard Szeliski Image Stitching 83 Richard Szeliski Image Stitching 84 Matching Mistakes How can we fix these?

Accidental alignment Tune the feature detector • repeated / similar regions Tune the feature matcher (cost metric) Failed alignments Tune the RANSAC stage (motion model) • moving objects / parallax Tune the verification stage • low overlap Use “higher-level” knowledge • “feature-less” regions (more variety?) • e.g., typical camera motions No 100% reliable algorithm? → Sounds like a big “learning” problem • Need a large training/test data set (panoramas)

Richard Szeliski Image Stitching 85 Richard Szeliski Image Stitching 86

Image feathering

Weight each image proportional to its distance from the edge Image Blending (distance map [Danielsson, CVGIP 1980]

1. Generate weight map for each image 2. Sum up all of the weights and divide by sum:

weights sum up to 1: wi’ = wi / ( ∑i wi)

Richard Szeliski Image Stitching 88 Image Feathering Feathering

+

1 1 0 = 0

Richard Szeliski Image Stitching 89 Richard Szeliski Image Stitching 90

Effect of window size Effect of window size

1 left 1 1 1

right 0 0 0 0

Richard Szeliski Image Stitching 91 Richard Szeliski Image Stitching 92 Good window size Pyramid Blending

1

0

“Optimal” window: smooth but not ghosted • Doesn’t always work... Burt, P. J. and Adelson, E. H., A multiresolution spline with applications to image Richard Szeliski Image Stitching 93 Richard Szeliski Image Stitching 94 mosaics, ACM Transactions on Graphics, 42(4), October 1983, 217-236.

Laplacian level Laplacian image blend 4 1. Compute Laplacian pyramid 2. Compute Gaussian pyramid on weight Laplacian image (can put this in A channel) level 3. Blend Laplacians using Gaussian blurred 2 weights 4. Reconstruct the final image Laplacian Q: How do we compute the original weights? level A: For horizontal panorama, use mid-lines 0 Q: How about for a general “3D” panorama? Richard Szeliski Image Stitching 95 Richard Szeliski Image Stitching 96 left pyramid right pyramid blended pyramid Weight selection (3D panorama) Poisson Image Editing

Idea: use original feather weights to select strongest contributing image

Can be implemented using L-∞ norm: (p = 10) w ’ = [w p / ( ∑ w p)]1/p i i i i Blend the gradients of the two images, then integrate For more info: Perez et al, SIGGRAPH 2003 Richard Szeliski Image Stitching 97 Richard Szeliski Image Stitching 98

Local alignment (deghosting)

Use local optic flow to compensate for small motions [Shum & Szeliski, ICCV’98] De-Ghosting

Richard Szeliski Image Stitching 100 Local alignment (deghosting) Region-based de-ghosting

Use local optic flow to compensate for radial Select only one image in regions-of-difference distortion [Shum & Szeliski, ICCV’98] using weighted vertex cover [Uyttendaele et al., CVPR’01]

Richard Szeliski Image Stitching 101 Richard Szeliski Image Stitching 102

Region-based de-ghosting Cutout-based de-ghosting

Select only one image in •Select only one image regions-of-difference using per output pixel, using weighted vertex cover spatial continuity [Uyttendaele et al., •Blend across seams CVPR’01] using gradient continuity (“Poisson blending”)

[Agarwala et al., SG’2004]

Richard Szeliski Image Stitching 103 Richard Szeliski Image Stitching 104 Cutout-based compositing PhotoMontage

Photomontage [Agarwala et al., SG’2004] Technical details: • Interactively blend different images: • use Graph Cuts to optimize seam placement group portraits Demo: • GroupShot application

Richard Szeliski Image Stitching 105 Richard Szeliski Image Stitching 106

Cutout-based compositing Cutout-based compositing

Photomontage [Agarwala et al., SG’2004] Photomontage [Agarwala et al., SG’2004] • Interactively blend different images: • Interactively blend different images: focus settings people’s faces

Richard Szeliski Image Stitching 107 Richard Szeliski Image Stitching 108 More stitching possibilities Other types of mosaics

• Video stitching • High dynamic range image stitching • see demo… • Flash + Non-Flash • Video-based rendering

Next week’s lecture: Can mosaic onto any surface if you know the geometry Computational Photography • See NASA’s Visible Earth project for some stunning earth mosaics – http://earthobservatory.nasa.gov/Newsroom/BlueMarble/ Richard Szeliski Image Stitching 109 Richard Szeliski Image Stitching 110

Slit images Slit images: cyclographs

y-t slices of the video volume are known as slit images • take a single column of pixels from each input image

Richard Szeliski Image Stitching 111 Richard Szeliski Image Stitching 112 Final thought: Slit images: photofinish What is a “panorama”?

Tracking a subject

Repeated (best) shots

Multiple exposures

“Infer” what photographer wants?

Richard Szeliski Image Stitching 113 Richard Szeliski Image Stitching 114