Synthetic aperture Outline photography and illumination using arrays of cameras and projectors technologies optical effects – large camera arrays synthetic aperture photography Marc Levoy – large projector arrays synthetic aperture illumination – camera–projector arrays synthetic confocal imaging
applications examples Computer Science Department – partially occluding environments foliage Stanford University – weakly scattering media murky water
Marc Levoy Marc Levoy
Multi-camera systems Stanford multi-camera array
• multi-camera vision systems • 640 × 480 pixels × • omni-directional vision Kang’s multi- 30 fps × 128 cameras ÷ baseline stereo • 1D camera arrays 18:1 MPEG = 512 Mbs • 2D camera arrays • snapshot or video • synchronized timing Kanade’s 3D room • continuous streaming Nayar’s Omnicam • cheap sensors & optics Immersive Media’s dodeca camera • flexible arrangement Manex’s bullet time array Marc Levoy Marc Levoy
Page 1 Time = 1 Cameras tightly packed: Applications for the array high-performance imaging
• How are the cameras arranged? • high-resolution – tightly packed high-performance imaging – by abutting the cameras’ fields of view – widely spaced light fields • high speed – intermediate spacing synthetic aperture photography – by staggering their triggering times • high dynamic range – mosaic of shutter speeds, apertures, density filters • high precision – averaging multiple images improves contrast • high depth of field – mosaic of differently focused lenses Marc Levoy Marc Levoy
Cameras tightly packed: Abutting fields of view high-performance imaging
Q. Can we align images this well? • high-resolution – by abutting the cameras’ fields of view • high speed – by staggering their triggering times • high dynamic range – mosaic of shutter speeds, apertures, density filters • high precision – averaging multiple images improves contrast • high depth of field A. Yes. – mosaic of differently focused lenses Marc Levoy Marc Levoy
Page 2 Time = 2 High-speed photography A virtual high-speed video camera [Wilburn, 2004 (submitted) ]
• 52 cameras, each 30 fps • packed as closely as possible • staggered firing, short exposure • result is 1560 fps camera • continuous streaming • no triggering needed
Harold Edgerton, Stopping Time, 1964 Marc Levoy Marc Levoy
Example A virtual high-speed video camera [Wilburn, 2004 (submitted) ]
• 52 cameras, 30 fps, 640 × 480 • packed as closely as possible • short exposure, staggered firing • result is 1536 fps camera • continuous streaming • no triggering needed • scalable to more cameras • limited by available photons • overlap exposure times?
100 cameras 3072 fps Marc Levoy Marc Levoy
Page 3 Time = 3 Cameras tightly packed: High dynamic range (HDR) high-X imaging
• high-resolution • overcomes one of photography’s key limitations – by abutting the cameras’ fields of view – negative film = 250:1 (8 stops) • high speed – paper prints = 50:1 – by staggering their triggering times – [Debevec97] = 250,000:1 (18 stops) • high dynamic range – hot topic at recent SIGGRAPHs – mosaic of shutter speeds, apertures, density filters • high precision – averaging multiple images improves contrast • high depth of field – mosaic of differently focused lenses Marc Levoy Marc Levoy
Cameras tightly packed: Seeing through murky water high-X imaging
• high-resolution • scattering decreases contrast – by abutting the cameras’ fields of view • noise limits perception in low contrast images • high speed • averaging multiple images decreases noise – by staggering their triggering times • high dynamic range – mosaic of shutter speeds, apertures, density filters • high precision – averaging multiple images improves contrast • high depth of field
– mosaic of differently focused lenses Marc Levoy Marc Levoy
Page 4 Time = 4 Seeing through murky water Seeing through murky water
• scattering decreases contrast, but does not blur 16 images 1 image • noise limits perception in low contrast images • averaging multiple images decreases noise
Marc Levoy Marc Levoy
Cameras tightly packed: High depth-of-field high-X imaging
• high-resolution • adjacent views use different focus settings – by abutting the cameras’ fields of view • for each pixel, select sharpest view • high speed – by staggering their triggering times [Haeberli90] • high dynamic range – mosaic of shutter speeds, apertures, density filters • high precision – averaging multiple images improves contrast
• high depth of field close focus distant focus composite – mosaic of differently focused lenses Marc Levoy Marc Levoy
Page 5 Time = 5 Synthetic aperture photography Synthetic aperture photography
Marc Levoy Marc Levoy
Synthetic aperture photography Synthetic aperture photography
Marc Levoy Marc Levoy
Page 6 Time = 6 Synthetic aperture photography Synthetic aperture photography
Marc Levoy Marc Levoy
Long-range synthetic aperture photography Synthetic pull-focus
Marc Levoy Marc Levoy
Page 7 Time = 7 Crowd scene Crowd scene
Marc Levoy Marc Levoy
Synthetic aperture photography using an array of mirrors
?
• 11-megapixel camera • 22 planar mirrors Marc Levoy Marc Levoy
Page 8 Time = 8 Synthetic aperture illumation
Marc Levoy Marc Levoy
Synthetic aperture illumation Confocal scanning microscopy
• technologies – array of projectors – array of microprojectors – single projector + array of mirrors light source pinhole • applications – bright display – autostereoscopic display [Matusik 2004] – confocal imaging [this paper]
Marc Levoy Marc Levoy
Page 9 Time = 9 Confocal scanning microscopy Confocal scanning microscopy
r light source light source pinhole pinhole
pinhole pinhole
photocell Marc Levoy photocell Marc Levoy
Confocal scanning microscopy
[UMIC SUNY/Stonybrook]
light source
pinhole
pinhole
photocell Marc Levoy
Page 10 Time = 10 Synthetic confocal scanning Synthetic confocal scanning
light source light source
→ 5 beams → 5 beams → 0 or 1 beam → 0 or 1 beam
Marc Levoy Marc Levoy
Synthetic coded-aperture Synthetic confocal scanning confocal imaging
→ 5 beams → 0 or 1 beam
d.o.f. • works with any number of projectors ≥ 2 • discrimination degrades if point to left of • different from coded aperture imaging in astronomy • no discrimination for points to left of • [Wilson, Confocal Microscopy by Aperture Correlation, 1996] •slow! • poor light efficiency Marc Levoy Marc Levoy
Page 11 Time = 11 Synthetic coded-aperture Synthetic coded-aperture confocal imaging confocal imaging
Marc Levoy Marc Levoy
Synthetic coded-aperture Synthetic coded-aperture confocal imaging confocal imaging
100 trials → 2 beams × ~50/100 trials ≈ 1 → ~1 beam × ~50/100 trials ≈ 0.5
Marc Levoy Marc Levoy
Page 12 Time = 12 Synthetic coded-aperture Synthetic coded-aperture confocal imaging confocal imaging
100 trials 100 trials → 2 beams × ~50/100 trials ≈ 1 → 2 beams × ~50/100 trials ≈ 1 → ~1 beam × ~50/100 trials ≈ 0.5 → ~1 beam × ~50/100 trials ≈ 0.5
floodlit floodlit → 2 beams → 2 beams → 2 beams → 2 beams
trials – ¼ × floodlit trials – ¼ × floodlit • 50% light efficiency → 1 – ¼ ( 2 ) ≈ 0.5 → 1 – ¼ ( 2 ) ≈ 0.5 → 0.5 – ¼ ( 2 ) ≈ 0 • any number of projectors ≥ 2 → 0.5 – ¼ ( 2 ) ≈ 0 • no discrimination to left of • works with relatively few trials (~16)
Marc Levoy Marc Levoy
Synthetic coded-aperture Example pattern confocal imaging
100 trials → 2 beams × ~50/100 trials ≈ 1 → ~1 beam × ~50/100 trials ≈ 0.5
floodlit → 2 beams → 2 beams
trials – ¼ × floodlit • 50% light efficiency → 1 – ¼ ( 2 ) ≈ 0.5 • any number of projectors ≥ 2 → 0.5 – ¼ ( 2 ) ≈ 0 • no discrimination to left of • works with relatively few trials (~16) • needs patterns in which illumination of tiles are uncorrelated Marc Levoy Marc Levoy
Page 13 Time = 13 Implementation using an Patterns with less aliasing array of mirrors
Marc Levoy Marc Levoy
Synthetic aperture confocal imaging
single viewpoint synthetic aperture image
(video available at http://graphics.stanford.edu/papers/confocal/) confocal image combined
Page 14 Time = 14 Selective illumination using Confocal imaging in scattering media object-aligned mattes
• small tank – too short for attenuation – lit by internal reflections
Marc Levoy Marc Levoy
Experiments in a large water tank Experiments in a large water tank
• 4-foot viewing distance to target • surfaces blackened to kill reflections • titanium dioxide in filtered water 50-foot flume at Wood’s Hole Oceanographic Institution (WHOI) Marc Levoy • transmissometer to measure turbidity Marc Levoy
Page 15 Time = 15 Experiments in a large water tank Seeing through turbid water
• stray light limits performance • one projector suffices if no occluders floodlit scanned tile Marc Levoy Marc Levoy
Other patterns Other patterns
sparse grid staggered grid
swept stripe
Marc Levoy floodlitswept stripe scanned tile Marc Levoy
Page 16 Time = 16 Stripe-based illumination
• if vehicle is moving, no sweeping is needed! • can triangulate from leading (or trailing) edge of stripe, getting range (depth) for free sum of floodlit
[Jaffe90]
Marc Levoy floodlitswept line scanned tile
Strawman conclusions on Application to imaging through a scattering medium underwater exploration
• shaping the illumination lets you see 2-3x further, but requires scanning or sweeping [Ballard/IFE 2004]
• use a pattern that avoids illuminating the media along the line of sight [Ballard/IFE 2004]
• contrast degrades with increasing total illumination, regardless of pattern
Marc Levoy Marc Levoy
Page 17 Time = 17 Relevant publications The team (in reverse chronological order) – Spatiotemporal Sampling and Interpolation for Dense Camera Arrays Bennett Wilburn, Neel Joshi, Katherine Chou, Marc Levoy, Mark Horowitz •staff • collaborators ACM Transactions on Graphics (conditionally accepted) – Interactive Design of Multi-Perspective Images for Visualizing Urban Landscapes – Mark Horowitz –Mark Bolas Augusto Román, Gaurav Garg, Marc Levoy – Marc Levoy – Ian McDowall Proc. IEEE Visualization 2004 – Bennett Wilburn – Guillermo Sapiro – Synthetic aperture confocal imaging Marc Levoy, Billy Chen, Vaibhav Vaish, Mark Horowitz, Ian McDowall, Mark Bolas Proc. SIGGRAPH 2004 • students – High Speed Video Using a Dense Camera Array Bennett Wilburn, Neel Joshi, Vaibhav Vaish, Marc Levoy, Mark Horowitz – Billy Chen • funding Proc. CVPR 2004 – Vaibhav Vaish –Intel – High Speed Video Using a Dense Camera Array – Katherine Chou – Sony –Monica Goyal Bennett Wilburn, Neel Joshi, Vaibhav Vaish, Marc Levoy, Mark Horowitz – Interval Research – Neel Joshi Proc. CVPR 2004 – Hsiao-Heng Kelin Lee –NSF – The Light Field Video Camera – Georg Petschnigg – DARPA Bennett Wilburn, Michael Smulski, Hsiao-Heng Kelin Lee, and Mark Horowitz – Guillaume Poncin Proc. Media Processors 2002, SPIE Electronic Imaging 2002 – Michael Smulski – Augusto Roman Marc Levoy
http://graphics.stanford.edu/projects/array Marc Levoy
Page 18 Time = 18