Image-based rendering Output
Michael F. Cohen Image Microsoft Research Model Synthetic Camera
Computer Vision Combined
Output Output
Image Synthetic Model Real Scene Model Camera Real Scene Real Cameras
Real Cameras
1 But, vision technology falls … and so does graphics. short
Output Output
Image Image Model Model Synthetic Real Scene Synthetic Real Scene Camera Camera Real Cameras Real Cameras
Image Based Rendering Ray
Constant radiance Output • time is fixed
Image Synthetic Real Scene Camera Images+Model Real Cameras -or- 5D Expensive Image Synthesis • 3D position • 2D direction
2 All Rays Line
Plenoptic Function Infinite line • all possible images • too much stuff!
4D • 2D direction • 2D position
Ray Image
Discretize What is an image?
Distance between 2 rays All rays through a point • Which is closer together? • Panorama?
3 Image Image
2D • position of rays has been fixed 2D • direction remains • position
Image Object
Image plane Light leaving towards “eye”
2D 2D • position • just dual of image
4 Object Object
All light leaving object
4D • 2D position • 2D direction
Object Lumigraph
All images
How to • organize • capture • render
5 Lumigraph - Organization Lumigraph - Organization
2D position 2D position
2D direction 2D position u s s q
2 plane parameterization
Lumigraph - Organization Lumigraph - Organization
2D position Hold s,t constant 2D position s,t u,v Let u,v vary t s,t v An image
u,v
2 plane parameterization u s s,t u,v
6 Lumigraph - Organization Lumigraph - Capture
Discretization Idea 1 • higher res near object • Move camera carefully • if diffuse over s,t plane • captures texture • Gantry • lower res away • see Lightfield paper • captures directions
s,t u,v s,t u,v
Lumigraph - Capture Lumigraph - Rendering
Idea 2 For each output pixel • Move camera anywhere • determine s,t,u,v • Rebinning • either • see Lumigraph paper • find closest discrete RGB s,t u,v • interpolate near values
s,t u,v
7 Lumigraph - Rendering Lumigraph - Rendering
For each output pixel Nearest • determine s,t,u,v • closest s • closest u • draw it
• either Blend 16 nearest • use closest discrete RGB • quadrilinear interpolation • interpolate near values s u s u
Current practice free viewpoint video High-Quality Video View Interpolation Using a Layered Representation Many cameras Larry Zitnick vs. Sing Bing Kang Matt Uyttendaele Motion Jitter Simon Winder Rick Szeliski
Interactive Visual Media Group Microsoft Research
8 Current practice Video view interpolation free viewpoint video
Fewer cameras Many cameras and Smooth Motion vs. Motion Jitter Automatic
Real-time rendering
Prior work: IBR (static) Prior work: IBR (dynamic)
Plenoptic Modeling McMillan & Bishop, SIGGRAPH ‘95 Stanford Multi-Camera Array Project Light Field Rendering Virtualized RealityTM Dynamic Light Fields Levoy & Hanrahan, SIGGRAPH ‘96 Kanade et al., IEEE Multimedia ‘97 Goldlucke et al., VMV ‘02
The Lumigraph Concentric Mosaics Gortler et al., SIGGRAPH ‘96 Shum & He, SIGGRAPH ‘99 Image-Based Visual Hulls Free-viewpoint Video of Humans 3D TV Matusik et al., SIGGRAPH ‘00 Carranza et al., SIGGRAPH ‘03 Matusik & Pfister, SIGGRAPH ‘04
9 System overview cameras
Video hard Capture disks controlling laptop concentrators OFFLINE
Stereo Representation Compression
File
ONLINE Selective Render Decompression
Calibration Input videos
Zhengyou Zhang, 2000
10 Image correspondence Key to view interpolation: Geometry Image 1 Image 2
Stereo Geometry
Image 1 Image 2 Leg Correct Good Incorrect
Bad Wall Camera 1 Camera 2 Match Score Match Score Virtual Camera
Local matching Global regularization
A Image 1 Image 2 Create MRF (Markov Random Field):
Image 1 Image 2
B E R P Q A Low texture F A C S T D U
colorA ≈ colorB → zA ≈ zB NumberzA ≈ z Pof, zstatesQ, zS = Each segment is a node number of depth levels
11 Iteratively solve MRF Depth through time
Matting Background Rendering with matting Interpolated view withoutSurface matting
Foreground Surface No Matting Matting
Background Background Alpha Strip Foreground Width
Foreground
Bayesian Matting Chuang et al. 2001 Camera
12 Representation Main
Background
Boundary
Strip Foreground “Massive Arabesque” Width
Boundary Layer: Main Layer: videoclip Color Color
Alpha
Depth Depth
13