Mitsubishi Electric Research Laboratories Raskar 2007 Less is More: Coded Computational Photography
Projector Pos=0
Tags
Pos=255
Ramesh Raskar
Mitsubishi Electric Research Labs (MERL) Cambridge, MA
Simplest Visual Organs
Larval Trematode Worm ‘Single Pixel’ Camera
1 Simplest Visual Organs
Larval Trematode Worm ‘Single Pixel’ Camera
Simplest Visual Organs
?
Larval Trematode Worm
2 Special Aperture
?
Larval Trematode Worm
Special Aperture
The aperture of a 100 mm lens is modified
Insert a coded mask with chosen binary pattern Rest of the camera is unmodified
3 LED
In Focus Photo
Out of Focus Photo: Open Aperture
4 Out of Focus Photo: Coded Aperture
Bokeh
5 6 Out of Focus Photo: Coded Aperture
Captured Blurred Photo
7 Refocused on Person
Blurred Photos
Open Aperture Coded Aperture, 7 * 7 Mask
8 After Removing De-Focus Blur
Open Aperture Coded Aperture, 7 * 7 Mask
Motion Blurred Photo
9 Short Exposure Traditional MURA Shutter
Captured Single Photo
Deblurred Result
Banding Artifacts and Dark some spatial frequencies and noisy are lost
10 Blurring == Convolution
Fourier Transform Sharp Photo Blurred Photo PSF == Sinc Function
ω Traditional Camera: Shutter is OPEN: Box Filter
Fourier Transform Sharp Photo Blurred Photo PSF == Broadband Function
Preserves High Spatial Frequencies
Flutter Shutter: Shutter is OPEN and CLOSED
11 Flutter Shutter Camera Raskar, Agrawal, Tumblin [Siggraph2006]
LCD opacity switched in coded sequence
Coded Traditional Exposure
Deblurred Deblurred Image Image
Image of Static Object
12 Application: Aerial Imaging
Sharpness versus Image Pixel Brightness
T=100ms T = 0
Long Exposure: Short Explosure:
Shutter Open
Flutter Shutter Time Shutter Closed
Sharp image with sufficient brightness
13 Coded Exposure Coded Aperture
Temporal 1-D broadband code: Spatial 2-D broadband mask: Motion Deblurring Focus Deblurring
Less is More
Blocking Light == More Information
Coding in Time Coding in Space
14 Larval Trematode Worm
Coded Aperture in Nature ?
Larval Trematode Worm Turbellarian Worm
15 Less is More ..
• Coded Exposure – Motion Deblurring
• Coded Aperture – Focus Deblurring
• Optical Heterodyning – Light Field Capture
Projector Pos=0 • Coded Illumination – Motion Capture Tags – Multi-flash: Cartoons Pos=255
Computational Photography
1. Epsilon Photography – Multi-photos by perturbing camera parameters – HDR, panorama – ‘Ultimate camera’: (Photo-editors)
2. Coded Photography – Single/few snapshot – Reversible encoding of data – Additional sensors/optics/illum – ‘Scene analysis’ : (Consumer software?)
3. Impossible Photos – Beyond single view/illum – Not mimic human eye – ‘New art form’
16 Computational Photography
1. Epsilon Photography – Multiphotos by varying camera parameters – HDR, panorama – ‘Ultimate camera’: (Photo-editor)
2. Coded Photography – Single/few snapshot – Reversible encoding of data – Additional sensors/optics/illum – ‘Scene analysis’ : (Next software?)
3. Impossible Photos – Not mimic human eye – Beyond single view/illum – ‘New artform’
Computational Photography
1. Epsilon Photography – Multiphotos by varying camera parameters – HDR, panorama – ‘Ultimate camera’: (Photo-editor)
2. Coded Photography – Single/few snapshot – Reversible encoding of data – Additional sensors/optics/illum – ‘Scene analysis’ : (Next software?)
3. Impossible Photos – Not mimic human eye – Beyond single view/illum – ‘New artform’
17 Mask? Sensor Mask
Sensor Mask
Full Resolution Digital 4D Light Field from 2D Refocusing: Photo: Coded Aperture Camera Heterodyne Light Field Camera
Capturing Light Field Inside a Camera
18 Capturing Light Field Inside a Camera
Lenslet-based Light Field camera
[Adelson and Wang, 1992, Ng et al. 2005 ]
Stanford Plenoptic Camera [Ng et al 2005]
Contax medium format camera Kodak 16-megapixel sensor
Adaptive Optics microlens array 125μ square-sided microlenses
4000 × 4000 pixels ÷ 292 × 292 lenses = 14 × 14 pixels per lens
19 Digital Refocusing
[Ng et al 2005]
Can we achieve this with a Mask alone?
Heterodyne Light Field Camera
Sensor Mask
Scanner sensor
Mask
20 How to Capture 4D Light Field with 2D Sensor ?
What should be the pattern of the mask ?
Optical Heterodyning Receiver: Demodulation High Freq Carrier 100 MHz
Incoming Signal
Baseband Audio Signal Reference 99 MHz Carrier
Software Demodulation Object Main Lens Mask Sensor
Recovered Light Field
Photographic Carrier Incident Signal Modulated Reference (Light Field) Signal Carrier
21 Captured 2D Photo
Encoding due to Cosine Mask
Traditional Camera vs Heterodyne Camera
2D FFT
Traditional Camera Photo Magnitude of 2D FFT
2D FFT
Heterodyne Camera Photo Magnitude of 2D FFT
22 Computing 4D Light Field
2D Sensor Photo, 1800*1800 2D Fourier Transform, 1800*1800
2D FFT
9*9=81 spectral copies
Rearrange 2D tiles into 4D planes 4D IFFT 200*200*9*9
4D Light Field 200*200*9*9
A Theory of Mask-Enhanced Camera
Object Main Lens Mask Sensor
•Mask == Light Field Modulator •Intensity of ray gets multiplied by Mask •Convolution in Frequency domain
23 Band-limited fθ Light Field fθ0
f x0 fx Sensor Slice – Fourier Slice Theorem
Photo = Slice of Light Field in Fourier Domain [Ren Ng, SIGGRAPH 2005]
How to Capture 2D Light Field with 1D Sensor ?
Band-limited fθ Light Field fθ0
f x0 fx Sensor Slice
Fourier Light Field Space
24 Extra sensor bandwidth cannot capture extra dimension of the light field
fθ Extra sensor fθ0 bandwidth
f x0 fx Sensor Slice
fθ ??? ???
fx ??? ???
25 Solution: Modulation Theorem Make spectral copies of 2D light field
fθ
fθ0
fx0 fx
Modulation Function
Sensor Slice captures entire Light Field
fθ
fθ0
fx fx0
Modulation Function Modulated Light Field
26 Demodulation to recover Light Field
1D Fourier Transform of Sensor Signal
fθ
fx
Reshape 1D Fourier Transform into 2D
Modulation Function == Sum of Impulses Physical Mask = Sum of Cosines
fθ
fθ0
fx fx0
27 Cosine Mask Used Mask Tile
1/f0
Captured 2D Photo
Encoding due to Cosine Mask
28 Computing 4D Light Field
2D Sensor Photo, 1800*1800 2D Fourier Transform
2D FFT
9*9=81 spectral copies
Rearrange 2D tiles into 4D planes 4D IFFT 200*200*9*9
4D Light Field 200*200*9*9
Only cone in focus
Captured Photo
Digital Refocusing
29 Captured 2D Photo Full resolution 2D image of Focused Scene Parts
divide
Image of White Lambertian Plane
Coding and Modulation in Camera Using Masks
Mask? Sensor
Sensor Sensor Mask Mask
Coded Aperture for Full Resolution Heterodyne Light Digital Refocusing Field Camera
30 Coded Imaging
• Coded Exposure – Motion Deblurring
• Coded Aperture – Focus Deblurring
• Optical Heterodyning – Light Field Capture
Projector Pos=0 • Coded Illumination – Motion Capture Tags – Multi-flash: Cartoons Pos=255
Vicon Medical Rehabilitation Athlete Analysis Optical Motion Capture
Body-worn markers
High-speed IR Camera
Performance Capture Biomechanical Analysis
31 Labeling Space
Projector Pos=0 Each location receives a unique temporal code
Tags
Pos=255
But 60Hz video projector is too slow
Coded Illumination Projector
Condensing Optics Light Source Focusing Optics
Gray code Slide
The Gray code pattern
32 Photosensing Tag
Imperceptible tags, Ambient Lighting, Id per marker
Prakash [Raskar, Nii, Summet et al Siggraph 2007]
33 Coded Illumination for Motion Capture
• 500 Hz Tracking • Id for each Marker Tag • Capture in Natural Environment – Visually imperceptible tags – Photosensing Tag can be hidden under clothes – Ambient lighting is ok • Unlimited Number of Tags Allowed • Base station and tags only a few 10’s $
Acknowledgements
• Amit Agrawal, MERL • Jack Tumblin, Northwestern U. • Shree Nayar, Columbia U.
• MERL – Jay Thornton, Keisuke Kojima • Mitsubishi Electric Japan – Kazuhiko Sumi, Haruhisa Okuda
• Coded Aperture and Light Field – Ashok Veerarghavan, Ankit Mohan • Prakash, Motion Capture – Masahiko Inami, Hideaki Nii, Yuki Hashimoto, Jay Summet, Erich Bruns, Paul Dietz, Bert de Decker, Philippe Bekaert
• Prof Yagi, Prof Ikeuchi and ACCV Organizers
34 Future of Coding Light
• How to block light in other ways? – Time, Space, Illumination .. Wavelength? On Sensor?
• What other blockers? – Dynamic masks (LCDs), non-planar or colored masks?
• Applications – Estimate params in presence of low pass convolution – Light Field Applications: lens aberration, microscopy
Coded Photography
• Coded Exposure – Motion Deblurring
• Coded Aperture – Focus Deblurring
• Optical Heterodyning – Light Field Capture
• Coded Illumination – Motion Capture Projector Pos=0 – Multi-flash: Shape Contours
Tags
Pos=255 • Epsilon->Coded->Impossible Photos
35 Blind Camera
Sascha Pohflepp, U of the Art, Berlin, 2006
36 Coded Photography
• Coded Exposure – Motion Deblurring
• Coded Aperture – Focus Deblurring
• Optical Heterodyning – Light Field Capture
• Coded Illumination – Motion Capture Projector Pos=0 – Multi-flash: Shape Contours
Tags
Pos=255 • Epsilon->Coded->Impossible Photos
37