<<

Image-based rendering Output

Michael F. Cohen Research Model Synthetic

Computer Vision Combined

Output Output

Image Synthetic Model Real Scene Model Camera Real Scene Real

Real Cameras

1 But, vision falls … and so does graphics. short

Output Output

Image Image Model Model Synthetic Real Scene Synthetic Real Scene Camera Camera Real Cameras Real Cameras

Image Based Rendering Ray

 Constant radiance Output • time is fixed

Image Synthetic Real Scene Camera +Model Real Cameras -or-  5D Expensive Image Synthesis • 3D position • 2D direction

2 All Rays

 Plenoptic Function  Infinite line • all possible images • too much stuff!

 4D • 2D direction • 2D position

Ray Image

 Discretize  What is an image?

 Distance between 2 rays  All rays through a point • Which is closer together? • Panorama?

3 Image Image

 2D • position of rays has been fixed  2D • direction remains • position

Image Object

 Image plane  leaving towards “

 2D  2D • position • just dual of image

4 Object Object

 All light leaving object

 4D • 2D position • 2D direction

Object Lumigraph

 All images

 How to • organize • capture • render

5 Lumigraph - Organization Lumigraph - Organization

 2D position  2D position

 2D direction  2D position u s s q

 2 plane parameterization

Lumigraph - Organization Lumigraph - Organization

 2D position  Hold s,t constant  2D position s,t u,v  Let u,v vary t s,t v  An image

u,v

 2 plane parameterization u s s,t u,v

6 Lumigraph - Organization Lumigraph - Capture

 Discretization  Idea 1 • higher res near object • Move camera carefully • if diffuse over s,t plane • captures texture • Gantry • lower res away • see Lightfield • captures directions

s,t u,v s,t u,v

Lumigraph - Capture Lumigraph - Rendering

 Idea 2  For each output • Move camera anywhere • determine s,t,u,v • Rebinning • either • see Lumigraph paper • find closest discrete RGB s,t u,v • interpolate near values

s,t u,v

7 Lumigraph - Rendering Lumigraph - Rendering

 For each output pixel  Nearest • determine s,t,u,v • closest s • closest u • draw it

• either  Blend 16 nearest • use closest discrete RGB • quadrilinear interpolation • interpolate near values s u s u

Current practice free viewpoint video High-Quality Video View Interpolation Using a Layered Representation Many cameras Larry Zitnick vs. Sing Bing Kang Matt Uyttendaele Motion Jitter Simon Winder Rick Szeliski

Interactive Visual Media

8 Current practice Video view interpolation free viewpoint video

Fewer cameras Many cameras and Smooth Motion vs. Motion Jitter Automatic

Real-time rendering

Prior work: IBR (static) Prior work: IBR (dynamic)

Plenoptic Modeling McMillan & Bishop, SIGGRAPH ‘95 Stanford Multi-Camera Array Project Rendering Virtualized RealityTM Dynamic Light Fields Levoy & Hanrahan, SIGGRAPH ‘96 Kanade et al., IEEE ‘97 Goldlucke et al., VMV ‘02

The Lumigraph Concentric Mosaics Gortler et al., SIGGRAPH ‘96 Shum & He, SIGGRAPH ‘99 Image-Based Visual Hulls Free-viewpoint Video of Humans 3D TV Matusik et al., SIGGRAPH ‘00 Carranza et al., SIGGRAPH ‘03 Matusik & Pfister, SIGGRAPH ‘04

9 System overview cameras

Video hard Capture disks controlling laptop concentrators OFFLINE

Stereo Representation Compression

File

ONLINE Selective Render Decompression

Calibration Input videos

Zhengyou Zhang, 2000

10 Image correspondence Key to view interpolation: Image 1 Image 2

Stereo Geometry

Image 1 Image 2 Leg Correct Good Incorrect

Bad Wall Camera 1 Camera 2 Match Score Match Score Virtual Camera

Local matching Global regularization

A Image 1 Image 2 Create MRF (Markov Random Field):

Image 1 Image 2

B E R P Q A Low texture F A C S T D U

colorA ≈ colorB → zA ≈ zB NumberzA ≈ z Pof, zstatesQ, zS = Each segment is a node number of depth levels

11 Iteratively solve MRF Depth through time

Matting Background Rendering with matting Interpolated view withoutSurface matting

Foreground No Matting Matting

Background Background Alpha Strip Foreground Width

Foreground

Bayesian Matting Chuang et al. 2001 Camera

12 Representation Main

Background

Boundary

Strip Foreground “Massive Arabesque” Width

Boundary Layer: Main Layer: videoclip Color Color

Alpha

Depth Depth

13