Acquisition of Point-Sampled Geometry And

The Goal: To Capture Reality

Acquisition of Point-Sampled • Fully-automated 3D model creation of real Geometry and Appearance objects. • Faithful representation of appearance for Hanspeter Pfister, MERL these objects. [email protected]

Wojciech Matusik, MIT Addy Ngan, MIT Paul Beardsley, MERL Remo Ziegler, MERL Leonard McMillan, MIT

Point-Based Computer Graphics Hanspeter Pfister, MERL 1 Point-Based Computer Graphics Hanspeter Pfister, MERL 2

Image-Based 3D Photography Previous Work

• An image-based 3D scanning system. • Active and passive 3D scanners • Handles fuzzy, refractive, transparent objects. • Work best for diffuse materials. • Robust, automatic • Fuzzy, transparent, and refractive objects are difficult. • Point-sampled geometry based on the visual hull. • BRDF estimation, inverse rendering • Objects can be rendered in novel environments. • Image based modeling and rendering • Reflectance fields [Debevec et al. 00] • Light Stage system to capture reflectance fields • Fixed viewpoint, no geometry • Environment matting [Zongker et al. 99, Chuang et al. 00] • Capture reflections and refractions • Fixed viewpoint, no geometry

Point-Based Computer Graphics Hanspeter Pfister, MERL 3 Point-Based Computer Graphics Hanspeter Pfister, MERL 4

Outline The System

•Overview ¾ System •Geometry Light Array • Reflectance •Rendering •Results

Cameras

Multi-Color Monitors Rotating Platform

Point-Based Computer Graphics Hanspeter Pfister, MERL 5 Point-Based Computer Graphics Hanspeter Pfister, MERL 6 Outline Acquisition

•Overview • For each viewpoint ( 6 cameras x 72 •System positions ) ¾ Geometry • Alpha mattes • Use multiple backgrounds [Smith and Blinn 96] • Reflectance • Reflectance images •Rendering • Pictures of the object under different •Results lighting (4 lights x 11 positions) • Environment mattes • Use similar techniques as [Chuang et al. 2000]

Point-Based Computer Graphics Hanspeter Pfister, MERL 7 Point-Based Computer Graphics Hanspeter Pfister, MERL 8

Geometry – Opacity Hull Approximate Geometry

• Visual hull augmented with view-dependent • The approximate visual hull is augmented by opacity. radiance data to render concavities, reflections, and transparency.

Point-Based Computer Graphics Hanspeter Pfister, MERL 9 Point-Based Computer Graphics Hanspeter Pfister, MERL 10

Geometry Example Surface Light Fields

• A surface light field is a function that assigns a color to each ray originating on a surface. [Wood et al., 2000]

Point-Based Computer Graphics Hanspeter Pfister, MERL 11 Point-Based Computer Graphics Hanspeter Pfister, MERL 12 Shading Algorithm Color Blending

• A view-dependent strategy. • Blend colors based on angle between virtual camera and stored colors. • Unstructured Lumigraph Rendering [Buehler et al., SIGGRAPH 2001] • View-Dependent Texture Mapping [Debevec, EGRW 98]

Point-Based Computer Graphics Hanspeter Pfister, MERL 13 Point-Based Computer Graphics Hanspeter Pfister, MERL 14

Point-Based Rendering Geometry – Opacity Hull

• Point-based rendering using LDC tree, • Store the opacity of each observation at visibility splatting, and view-dependent each point on the visual hull [Matusik et al. shading. SIG2002].

Point-Based Computer Graphics Hanspeter Pfister, MERL 15 Point-Based Computer Graphics Hanspeter Pfister, MERL 16

Geometry – Opacity Hull Example

• Assign view-dependent opacity to each ray originating on a point of the visual hull.

Photo Surface B C Light Field

A φ (θ,φ)

Red = invisible Visual Hull Opacity White = opaque Hull A B C Black = transparent θ Point-Based Computer Graphics Hanspeter Pfister, MERL 17 Point-Based Computer Graphics Hanspeter Pfister, MERL 18 Results Opacity Hull – Discussion

• Point-based rendering using EWA splatting, • View dependent opacity vs. geometry A-buffer blending, and edge antialiasing. trade-off. • Similar to radiance vs. geometry trade-off. • Sometimes acquiring the geometry is not possible (e.g. resolution of the acquisition device is not adequate). • Sometimes representing true geometry would be very inefficient (e.g. hair, trees). • Opacity hull stores the “macro” effect.

Point-Based Computer Graphics Hanspeter Pfister, MERL 19 Point-Based Computer Graphics Hanspeter Pfister, MERL 20

Point-Based Models Outline

• No need to establish topology or •Overview connectivity. • Previous Works • No need for a consistent surface •Geometry parameterization for texture mapping. ¾ Reflectance • Represent organic models (feather, tree) •Rendering much more readily than polygon models. •Results • Easy to represent view-dependent opacity and radiance per surface point.

Point-Based Computer Graphics Hanspeter Pfister, MERL 21 Point-Based Computer Graphics Hanspeter Pfister, MERL 22

Light Transport Model Surface Reflectance Fields

ω ω = θ Φ θ Φ • Assume illumination originates from • 6D function:ω W (P, i , r ) W (ur ,vr ; i , i ; r , r ) infinity. i ω • The light arriving at a camera pixel can be i described as: C(x, y) = W (ω)E(ω)dω ∫ P Ω ω r C(x,y) - the pixel value E-the environment W -the reflectance field

Point-Based Computer Graphics Hanspeter Pfister, MERL 23 Point-Based Computer Graphics Hanspeter Pfister, MERL 24 Reflectance Functions Reflectance Field Acquisition

• For each viewpoint, 4D function: • We separate the hemisphere into high resolution Ω and low resolution Ω [Matusik W (ω ) = W (x, y;θ ,Φ ) h l xy i i i et al., EGRW2002].

(θi,φi Ω Ω ) h l T φi

L(ω C(x, y) = W (ξ )T (ξ )dξ + W) (ω )L(ω )dω ∫ h ∫ l i i Ω Ω θi h l

Point-Based Computer Graphics Hanspeter Pfister, MERL 25 Point-Based Computer Graphics Hanspeter Pfister, MERL 26

Acquisition Low-Resolution Reflectance Field

• For each viewpoint ( 6 cameras x 72 C(x, y) = W (ξ )T(ξ )dξ + W (ω )L(ω )dω ∫ h ∫ l i i Ω Ω positions ) h l • Alpha mattes • Wl sampled by taking pictures with each light • Use multiple backgrounds [Smith and Blinn 96] turned on at a time [Debevec et al 00]. • Reflectance images Low resolution • Pictures of the object under different lighting (4 lights x 11 positions) • Environment mattes High resolution n • Use similar techniques as [Chuang et al. 2000] W (ω )L(ω )dω ≈ W L for n lights ∫ l i i ∑ i i Ω i=1 l Point-Based Computer Graphics Hanspeter Pfister, MERL 27 Point-Based Computer Graphics Hanspeter Pfister, MERL 28

Compression High-Resolution Reflectance Field

• Subdivide images into 8 x 8 pixel blocks. C(x, y) = W (ξ )T(ξ )dξ + W (ω )L(ω )dω ∫ h ∫ l i i Ω Ω • Keep blocks containing the object (avg. h l compression 1:7) • Use techniques of environment matting • PCA compression (avg. compression 1:10) [Chuang et al., SIGGRAPH 00].

•Approximate Wh by a sum of up to two Gaussians: PCA N G1 • Reflective G1.

• Refractive G2. a0 a1 a2 a3 a4 a5 G2 ξ = + Wh ( ) a1G1 a2G2

Point-Based Computer Graphics Hanspeter Pfister, MERL 29 Point-Based Computer Graphics Hanspeter Pfister, MERL 30 Surface Reflectance Fields Outline

• Work without accurate geometry. •Overview • Surface normals are not necessary. • Previous Works • Capture more than reflectance: •Geometry • Inter-reflections • Reflectance • Subsurface scattering •Refraction ¾ Rendering • Dispersion •Results • Non-uniform material variations • Simplified version of the BSSRDF [Debevec et al., 00].

Point-Based Computer Graphics Hanspeter Pfister, MERL 31 Point-Based Computer Graphics Hanspeter Pfister, MERL 32

st Ω Rendering 1 Step: Relighting l

• Input: Opacity hull, reflectance data, new •Compute radiance image for each viewpoint. environment New Illumination •Create radiance images from environment and low-resolution reflectance field. Downsample

• Reparameterize environment mattes. x = • Interpolate data to new viewpoint.

The sum is the radiance image of this viewpoint in this environment.

Point-Based Computer Graphics Hanspeter Pfister, MERL 33 Point-Based Computer Graphics Hanspeter Pfister, MERL 34

nd Ω rd 2 Step: Reproject h 3 Step: Interpolation

• Project environment mattes onto the new • From new viewpoint, for each surface point, find environment. four nearest acquired viewpoints. • Store visibility vector per surface point. • Environment mattes acquired was parameterized on plane T (the plasma display). • Interpolate using unstructured lumigraph interpolation [Buehler et al., SIGGRAPH 01] or view- • We need to project the Gaussians to the new dependent texture mapping [Debevec 96]. environment map, producing new Gaussians. •Opacity. • Contribution from low-res reflectance field (in the form of Ωh radiance images). T • Contribution from high-res reflectance field.

Point-Based Computer Graphics Hanspeter Pfister, MERL 35 Point-Based Computer Graphics Hanspeter Pfister, MERL 36 3rd Step: Interpolation Outline

• For low-res reflectance field, we interpolate •Overview the RGB color from the radiance images. • Previous Works For high-resolution •Geometry reflectance field: V1 ~ • Reflectance Interpolate direction of G1r N ~ reflection/refraction. G V2 2r •Rendering Interpolate other ¾ Results parameters of the ~ G2t Gaussians. ~ G Convolve with the 1t environment.

Point-Based Computer Graphics Hanspeter Pfister, MERL 37 Point-Based Computer Graphics Hanspeter Pfister, MERL 38

Results Results

• Performance for 6x72 = 432 viewpoints • 337,824 images taken in total !! • Acquisition (47 hours) • Alpha mattes – 1 hour • Environment mattes – 18 hours • Reflectance images – 28 hours •Processing • Opacity hull ~ 30 minutes • PCA Compression ~ 20 hours (MATLAB, unoptimized) • Rendering ~ 5 minutes per frame •Size • Opacity hull ~ 30 - 50 MB • Environment mattes ~ 0.5 - 2 GB • Reflectance images ~ Raw 370 GB / Compressed 2 - 4 GB

Point-Based Computer Graphics Hanspeter Pfister, MERL 39 Point-Based Computer Graphics Hanspeter Pfister, MERL 40

Results Results Ω Ω High-resolution h Low-resolutionl Combined

Point-Based Computer Graphics Hanspeter Pfister, MERL 41 Point-Based Computer Graphics Hanspeter Pfister, MERL 42 Ω Results Results – h

Point-Based Computer Graphics Hanspeter Pfister, MERL 43 Point-Based Computer Graphics Hanspeter Pfister, MERL 44

Ω Results – l Results – Combined

Point-Based Computer Graphics Hanspeter Pfister, MERL 45 Point-Based Computer Graphics Hanspeter Pfister, MERL 46

Results Results

Point-Based Computer Graphics Hanspeter Pfister, MERL 47 Point-Based Computer Graphics Hanspeter Pfister, MERL 48 Conclusions Future Directions

• A fully automatic system that is able to capture • Use more than 2 Gaussians for the and render any type of object. environment mattes. • Opacity hulls combined with lightfields / surface • Better compression. reflectance fields provide realistic 3D graphics models. • Real-time rendering. • Point-based rendering offers easy surface parameterization of acquired models. • Separation of surface reflectance fields into high- and low-resolution areas is practical. • New rendering algorithm for environment matte interpolation.

Point-Based Computer Graphics Hanspeter Pfister, MERL 49 Point-Based Computer Graphics Hanspeter Pfister, MERL 50

Acknowledgements

• Colleagues: • MIT: Chris Buehler, Tom Buehler. • MERL: Bill Yerazunis, Darren Leigh, Michael Stern. • Thanks to: • David Tames, Jennifer Roderick Pfister. • NSF grants CCR-9975859 and EIA-9802220. • Papers available at: • http://www.merl.com/people/pfister/

Point-Based Computer Graphics Hanspeter Pfister, MERL 51