Robert Collins Robert Collins CSE486, Penn State CSE486, Penn State Recall: Planar Projection Pixel coords Internal u Lecture 17: params

Mosaicing and Stabilization Perspective v projection y Homography x

Point on plane Rotation + Translation

Robert Collins Robert Collins CSE486, Penn State Recall: Projective (un)Warping CSE486, Penn State Recall : Planar Projection

H1 H2 H

from Hartley & Zisserman Source Image Destination image H

Robert Collins Robert Collins CSE486, Penn State Applications: Stabilization CSE486, Penn State Stabilization Example

Given a sequence of video frames, warp them into a common image coordinate system.

k-2 k-1 frame k k+1 k+2

destination frame

This “stabilizes” the video to appear as if the VIVID project camera is not moving.

1 Robert Collins Robert Collins CSE486, Penn State Stabilization by Chaining CSE486, Penn State Applications: Mosaicing

What if the reference image does not overlap with all the Source 1 Destination image Source 2 source images? As long as there are pairwise overlaps, we can chain (compose) pairwise homographies. H 0 0 1 2 3 H1 2 H4 = H1 * H2 * H3 * H4

0 1 2 3 H1 H2 H3 H4 from Hartley & Zisserman

frame k k+1 k+2 k+3 k+4

destination

Not recommended for long sequences, as alignment errors accumulate over time. Mosaic

Robert Collins Robert Collins CSE486, Penn State Note on Planar Mosaicing CSE486, Penn State Ghosting Example

Source image Reference image Assumes scene is roughly planar. H

What if scene isn’t planar? Alignment will not be good if significant 3D relief

“Ghosting”

Robert Collins Robert Collins CSE486, Penn State Ghosting Example (cont) CSE486, Penn StateMosaics from Rotating Cameras Mosaic

However, there is a mitigating factor in regards to ghosting…

Images taken from a rotating camera are related by a 2D homography…

regardless of scene structure!

2 Robert Collins Robert Collins CSE486, Penn StateRotating Camera (top-down view) CSE486, Penn StateRotating Camera (top-down view)

Robert Collins Robert Collins CSE486, Penn StateRotating Camera (top-down view) CSE486, Penn StateRotating Camera (top-down view) Rays in camera coord system are invariant!

Robert Collins Robert Collins CSE486, Penn StateRotating Camera (top-down view) CSE486, Penn StateRotating Camera (top-down view) Rays in camera coord system are invariant! Rays in camera coord system are invariant!

3 Robert Collins Robert Collins CSE486, Penn State Special Case : Rotating Camera CSE486, Penn State Special Case : Rotating Camera

Internal Internal params Projection Relative R,T params Projection Relative R,T

Relative Rotation of camera Translation is 0 This is important!

Robert Collins Robert Collins CSE486, Penn State Relations among Images CSE486, Penn State Mosaicing Example Taken by Rotating Camera Original Images (from a pan/tilt camera) Image 1

Same ray! Panoramic (Mosaic) View Image 2

-1

Robert Collins Robert Collins CSE486, Penn State One more detail: Blending! CSE486, Penn State Approaches to Blending How to combine colors in area of overlap?

1) Straight averaging P = (P1 + P2) / 2

2) Feathering P = (w1*P1 + w2*P2 ) / (w1+w2) With wi being distance from image border

3) Equalize intensity statistics (gain, offset)

4 Robert Collins Robert Collins CSE486, Penn State Before and After Blending CSE486, Penn State 360 Degree Panoramas?

Problem: Can’t just choose a reference image to map all other images to.

Solution: Use cylindrical or spherical mosaic surface rather than a plane.

Robert Collins Robert Collins CSE486, Penn State Panorama Input images CSE486, Penn State Spherical Panorama Result

Sarnoff Sarnoff

Robert Collins Robert Collins CSE486, Penn State Panorama Unwarped CSE486, Penn State Quicktime VR

This example from www.panoguide.com/gallery/

Also big list at http://www.multimedialibrary.com/diana/qtvr_sites.asp

Sarnoff

5 Robert Collins Robert Collins CSE486, Penn State Quicktime VR CSE486, Penn State Quicktime VR

This example from www.ems.psu.edu/~fraser/qtvr/

6