Computational Photography & Cinematography
Total Page:16
File Type:pdf, Size:1020Kb
Computational photography & cinematography (for a lecture specifically devoted to the Stanford Frankencamera, see http://graphics.stanford.edu/talks/camera20-public-may10-150dpi.pdf) Marc Levoy Computer Science Department Stanford University The future of digital photography ! the megapixel wars are over (and it’s about time) ! computational photography is the next battleground in the camera industry (it’s already starting) 2 ! Marc Levoy Premise: available computing power in cameras is rising faster than megapixels 12 70 9 52.5 6 35 megapixels 3 17.5 billions of pixels / second of pixels billions 0 0 2005 2006 2007 2008 2009 2010 Avg megapixels in new cameras, CAGR = 1.2 NVIDIA GTX texture fill rate, CAGR = 1.8 (CAGR for Moore’s law = 1.5) ! this “headroom” permits more computation per pixel, 3 or more frames per second, or less custom hardware ! Marc Levoy The future of digital photography ! the megapixel wars are over (long overdue) ! computational photography is the next battleground in the camera industry (it’s already starting) ! how will these features appear to consumers? • standard and invisible • standard and visible (and disable-able) • aftermarket plugins and apps for your camera 4 ! Marc Levoy The future of digital photography ! the megapixel wars are over (long overdue) ! computational photography is the next battleground in the camera industry (it’s already starting) ! how will these features appear to consumers? • standard and invisible • standard and visible (and disable-able) • aftermarket plugins and apps for your camera ! traditional camera makers won’t get it right • they’ll bury it on page 120 of the manual (like Scene Modes) • the mobile industry will get it right (indie developers will help) 5 ! Marc Levoy 6 7 8 9 Film-like Computational Photography Photography with bits Computational Camera Smart Light Digital Computational Computational Computational Computational Photography Processing Imaging/Optics Sensor Illumination Image processing Processing of a set Capture of optically Detectors that Adapting and applied to captured of captured images coded images and combine sensing Controlling images to produce to create new computational and processing to Illumination to better images. images. decoding to produce create smart Create revealing new images. pixels. image Examples: Examples: Examples: Examples: Examples: Interpolation, Filtering, Mosaicing, Matting, Coded Aperture, Artificial Retina, Flash/no flash, Enhancement, Dynamic Super-Resolution, Optical Tomography, Retinex Sensors, Lighting domes, Range Compression, Multi-Exposure HDR, Diaphanography, Adaptive Dynamic Multi-flash Color Management, Light Field from SA Microscopy, Range Sensors, for depth edges, Morphing, Hole Filling, Mutiple View, Integral Imaging, Edge Detect Chips, Dual Photos, Artistic Image Effects, Structure from Motion, Assorted Pixels, Focus of Expansion Polynomial texture Image Compression, Shape from X. Catadioptric Imaging, Chips, Motion Maps, 4D light Watermarking. Holographic Imaging. Sensors. source [Nayar, Tumblin] Content-aware image resizing [Avidan SIGGRAPH 2007] • to compress: remove pixels along lowest-energy seams, ordered using dynamic programming • to expand: insert pixels along seams that, if removed in order, would yield the original image ! 2010 Marc Levoy Content-aware image resizing [Avidan SIGGRAPH 2007] • to compress: remove pixels along lowest-energy seams, ordered using dynamic programming • to expand: insert pixels along seams that, if removed in order, would yield the original image • application to object removal Now available in Photoshop !! • extendable to video ! 2010 Marc Levoy Film-like Computational Photography Photography with bits Computational Camera Smart Light Digital Computational Computational Computational Computational Photography Processing Imaging/Optics Sensor Illumination Image processing Processing of a set Capture of optically Detectors that Adapting and applied to captured of captured images coded images and combine sensing Controlling images to produce to create new computational and processing to Illumination to better images. images. decoding to produce create smart Create revealing new images. pixels. image Examples: Examples: Examples: Examples: Examples: Interpolation, Filtering, Mosaicing, Matting, Coded Aperture, Artificial Retina, Flash/no flash, Enhancement, Dynamic Super-Resolution, Optical Tomography, Retinex Sensors, Lighting domes, Range Compression, Multi-Exposure HDR, Diaphanography, Adaptive Dynamic Multi-flash Color Management, Light Field from SA Microscopy, Range Sensors, for depth edges, Morphing, Hole Filling, Mutiple View, Integral Imaging, Edge Detect Chips, Dual Photos, Artistic Image Effects, Structure from Motion, Assorted Pixels, Focus of Expansion Polynomial texture Image Compression, Shape from X. Catadioptric Imaging, Chips, Motion Maps, 4D light Watermarking. Holographic Imaging. Sensors. source [Nayar, Tumblin] High dynamic range (HDR) imaging High dynamic range (HDR) imaging Too dark High dynamic range (HDR) imaging Too bright High dynamic range (HDR) imaging this example worked well, but... Tone mapped combination ...this one looks washed out; tone mapping is still hard to do no cameras automatically take HDR pictures (How much to bracket?) Aligning a burst of short-exposure, high-ISO shots using the Casio EX-F1 1/3 sec ! 2010 Marc Levoy Aligning a burst of short-exposure, high-ISO shots using the Casio EX-F1 burst at 60fps ! 2010 Marc Levoy Aligning a burst of short-exposure, high-ISO shots using the Casio EX-F1 burst 1/3 sec at 60fps ! 2010 Marc Levoy iPhone 4, single HD video frame 23 SynthCam, align & average ~30 frames SNR increases as sqrt(# of frames) 24 Removing foreground objects by translating the camera • align the shots • match histograms • apply median filter ! 2010 Marc Levoy Film-like Computational Photography Photography with bits Computational Camera Smart Light Digital Computational Computational Computational Computational Photography Processing Imaging/Optics Sensor Illumination Image processing Processing of a set Capture of optically Detectors that Adapting and applied to captured of captured images coded images and combine sensing Controlling images to produce to create new computational and processing to Illumination to better images. images. decoding to produce create smart Create revealing new images. pixels. image Examples: Examples: Examples: Examples: Examples: Interpolation, Filtering, Mosaicing, Matting, Coded Aperture, Artificial Retina, Flash/no flash, Enhancement, Dynamic Super-Resolution, Optical Tomography, Retinex Sensors, Lighting domes, Range Compression, Multi-Exposure HDR, Diaphanography, Adaptive Dynamic Multi-flash Color Management, Light Field from SA Microscopy, Range Sensors, for depth edges, Morphing, Hole Filling, Mutiple View, Integral Imaging, Edge Detect Chips, Dual Photos, Artistic Image Effects, Structure from Motion, Assorted Pixels, Focus of Expansion Polynomial texture Image Compression, Shape from X. Catadioptric Imaging, Chips, Motion Maps, 4D light Watermarking. Holographic Imaging. Sensors. source [Nayar, Tumblin] Stanford Multi-Camera Array [Wilburn SIGGRAPH 2005] • 640 ! 480 pixels ! • synchronized timing 30 fps ! 128 cameras • continuous streaming • flexible arrangement ! 2008 Marc Levoy Synthetic aperture photography " ! 2008 Marc Levoy Example using 45 cameras [Vaish CVPR 2004] ! 2008 Marc Levoy (movie is available at http://graphics.stanford.edu/projects/array) Light field photography using a handheld plenoptic camera Ren Ng, Marc Levoy, Mathieu Brédif, Gene Duval, Mark Horowitz and Pat Hanrahan (Proc. SIGGRAPH 2005 and TR 2005-02) Light field photography [Ng SIGGRAPH 2005] ! 2010 Marc Levoy Prototype camera Contax medium format camera Kodak 16-megapixel sensor Adaptive Optics microlens array 125µ square-sided microlenses 4000 ! 4000 pixels ÷ 292 ! 292 lenses = 14 ! 14 pixels per lens Typical image captured by camera (show here at low res) Digital refocusing ! ! ! 2010 Marc Levoy Example of digital refocusing ! 2007 Marc Levoy Example of digital refocusing ! 2007 Marc Levoy Example of digital refocusing ! 2007 Marc Levoy Example of digital refocusing ! 2007 Marc Levoy Example of digital refocusing ! 2007 Marc Levoy Refocusing portraits (movie is available at http://refocusimaging.com) ! 2007 Marc Levoy Application to sports photography ! 2010 Marc Levoy Application to sports photography ! 2010 Marc Levoy Application to sports photography ! 2010 Marc Levoy Film-like Computational Photography Photography with bits Computational Camera Smart Light Digital Computational Computational Computational Computational Photography Processing Imaging/Optics Sensor Illumination Image processing Processing of a set Capture of optically Detectors that Adapting and applied to captured of captured images coded images and combine sensing Controlling images to produce to create new computational and processing to Illumination to better images. images. decoding to produce create smart Create revealing new images. pixels. image Examples: Examples: Examples: Examples: Examples: Interpolation, Filtering, Mosaicing, Matting, Coded Aperture, Artificial Retina, Flash/no flash, Enhancement, Dynamic Super-Resolution, Optical Tomography, Retinex Sensors, Lighting domes, Range Compression, Multi-Exposure HDR, Diaphanography, Adaptive Dynamic Multi-flash Color Management, Light Field from SA Microscopy, Range Sensors, for depth edges, Morphing, Hole Filling, Mutiple View, Integral Imaging, Edge Detect Chips,