Computational photography research at Stanford January 2012 Marc Levoy Computer Science Department Stanford University Outline ! Light field photography ! FCam and the Stanford Frankencamera ! User interfaces for computational photography ! Ongoing projects • FrankenSLR • burst-mode photography • languages and architectures for programmable cameras • computational videography and cinematography • multi-bucket pixels 2 ! Marc Levoy Light field photography using a handheld plenoptic camera Ren Ng, Marc Levoy, Mathieu Brédif, Gene Duval, Mark Horowitz and Pat Hanrahan (Proc. SIGGRAPH 2005 and TR 2005-02) Light field photography [Ng SIGGRAPH 2005] ! Marc Levoy Prototype camera Contax medium format camera Kodak 16-megapixel sensor Adaptive Optics microlens array 125µ square-sided microlenses 4000 ! 4000 pixels ÷ 292 ! 292 lenses = 14 ! 14 pixels per lens Digital refocusing ! ! ! 2010 Marc Levoy Example of digital refocusing ! 2010 Marc Levoy Refocusing portraits ! 2010 Marc Levoy Light field photography (Flash demo) ! 2010 Marc Levoy Commercialization • trades off (excess) spatial resolution for ability to refocus and adjust the perspective • sensor pixels should be made even smaller, subject to the diffraction limit, for example: 36mm ! 24mm ÷ 2.5µ pixels = 266 Mpix 20K ! 13K pixels 4000 ! 2666 pixels ! 20 ! 20 rays per pixel or 2000 ! 1500 pixels ! 3 ! 3 rays per pixel = 27 Mpix ! Marc Levoy Camera 2.0 Marc Levoy Computer Science Department Stanford University The Stanford Frankencameras Frankencamera F2 Nokia N900 “F” ! facilitate research in experimental computational photography ! for students in computational photography courses worldwide ! proving ground for plugins and apps for future cameras 14 ! 2010 Marc Levoy CS 448A - Computational Photography 15 ! 2010 Marc Levoy CS 448 projects M. Culbreth, A. Davis!! ! ! A phone-to-phone 4D light field fax machine J. Koberstein, A. Mattos!! ! Motion blur detection as a game controller N. Sakunkoo!! ! ! ! ! ! Camera-assisted self portraits on mobile devices C. DuHadway!! ! ! ! ! ! Driving/parking assistive mosaics using car cameras C.-H. Chen, S.H. Park!!!High dynamic range (HDR) viewfinding V. Chandrasekhar, G. Takacs!High dynamic range imaging on the Nokia N95 L. Kim, S. Que !! ! !User interfaces for adjusting focus and tone A.Y.L. Chan, S. Meador!! !Photography using circular/hemispherical ring flash P. Huang, Z. Zheng !! ! Image deblurring from multiple images M. Smith !! ! ! ! ! ! ! Representative (non-real) thumbnails for small screens A. Medearis !!!!!!Multi-shot noise reduction for sports photography D. Johnson !! ! ! ! ! ! Synthetic aperture photography using a painted aperture D. Chan !! ! ! ! ! ! Synthetic aperture rear view mirror for safer driving 16 ! 2010 Marc Levoy Frankencamera architecture Requests Image Sensor Con!gure Application Devices Processor 1 Lens Actions Expose 2 Flash Readout Metadata + ... 3 Imaging Processor Images and Statistics Image Statistics Processing Collection 4 17 ! 2010 Marc Levoy Frankencamera software: the FCAM API Sensor sensor; Flash flash; vector<Shot> burst(2); burst[0].exposure = 1/200.; burst[1].exposure = 1/30.; Flash::FireAction fire(&flash); fire.time = burst[0].exposure/2; burst[0].actions.insert(fire); sensor.stream(burst); while (1) { Frame flashFrame = sensor.getFrame(); Frame noflashFrame = sensor.getFrame(); } 18 ! 2010 Marc Levoy Demonstration applications • Canon 430EX (smaller flash) strobed continuously • Canon 580EX (larger flash) fired once at end of exposure 19 ! 2010 Marc Levoy Fcam is being actively ported to... ! Android (at least by NVIDIA) ! iOS? ! Nokia/Microsoft? ! Android point-and-shoot camera? 20 ! 2010 Marc Levoy Prospects for computational photography ! the megapixel wars are over (and it’s about time) ! computational photography is the next battleground in the camera industry (it’s already starting) 21 ! Marc Levoy Premise: available computing power in cameras is rising faster than megapixels 12 70 9 52.5 6 35 megapixels 3 17.5 billions of pixels / second of pixels billions 0 0 2005 2006 2007 2008 2009 2010 Avg megapixels in new cameras, CAGR = 1.2 NVIDIA GTX texture fill rate, CAGR = 1.8 (CAGR for Moore’s law = 1.5) ! this “headroom” permits more computation per pixel, 22 or more frames per second, or less custom hardware ! Marc Levoy Interfaces for computational photography ! the megapixel wars are over (long overdue) ! computational photography is the next battleground in the camera industry (it’s already starting) ! how will these features appear to consumers? • standard and invisible • standard and visible (and disable-able) • aftermarket plugins and apps for your camera 23 ! Marc Levoy Interfaces for computational photography ! the megapixel wars are over (long overdue) ! computational photography is the next battleground in the camera industry (it’s already starting) ! how will these features appear to consumers? • standard and invisible • standard and visible (and disable-able) • aftermarket plugins and apps for your camera ! but aren’t cameras too complicated already? • won’t computational photography make this worse? 24 ! Marc Levoy Sony NEX-5 ! manual is 159 pages long • and full of grammatical mistakes ! scene modes are not explained well • Sunset: Shoots the red of the sunset beautifully. (page 43) ! user interface is difficult to navigate 1 8 2 9 3 10 4 11 5 12 6 13 7 25 ! Marc Levoy ProHDR for the iPhone 4S " 1 " " 2 " ! most popular HDR capture method on the market ! app developers are making photography easy again! 26 ! Marc Levoy Free 27 Free 28 Free 29 Free circa 500,000 downloads in 2011 30 iPhone 4, single HD video frame 31 SynthCam, align & average ~30 frames SNR increases as sqrt(# of frames) 32 SynthCam Canon 5DII, 1/400 sec Canon 5DII, average of 15 shots of 1/6 sec (Marc Levoy) (Marc Levoy) Removing foreground objects by translating the camera • align the shots • match histograms • apply median filter ! 2010 Marc Levoy Ongoing projects ! Frankencamera F3 (“FrankenSLR”) • SLR-quality sensor, but big, heavy, and expensive ($10K) Micron MT9001 Cypress LUPA 4000 • cell phone quality • DSLR quality • $150 • $1500 • arbitrary ROIs and non-destructive readout 39 ! 2010 Marc Levoy Ongoing projects ! Frankencamera F3 (“FrankenSLR”) • SLR-quality sensor, but big, heavy, and expensive ($10K) ! high-speed burst-mode photography • all still cameras should capture at (up to) 500 fps • capture while aiming the camera - no shutter half-press • frameless photography - ROIs, MOIs (“M” = Moment) 40 ! 2010 Marc Levoy Burst-mode photography 41 ! 2010 Marc Levoy Ongoing projects ! Frankencamera F3 (“FrankenSLR”) • SLR-quality sensor, but big, heavy, and expensive ($10K) ! high-speed burst-mode photography • all still cameras should capture at (up to) 500 fps • capture while aiming the camera - no shutter half-press • frameless photography - ROIs, MOIs (“M” = Moment) ! languages and architectures for programmable cameras • FCam was an API for camera control • now we need APIs for photographic processing and computer vision 42 ! 2010 Marc Levoy Ongoing projects ! Frankencamera F3 (“FrankenSLR”) • SLR-quality sensor, but big, heavy, and expensive ($10K) ! high-speed burst-mode photography • all still cameras should capture at (up to) 500 fps • capture while aiming the camera - no shutter half-press • frameless photography - ROIs, MOIs (“M” = Moment) ! languages and architectures for programmable cameras • FCam was an API for camera control • now we need APIs for photographic processing and computer vision ! computational videography & cinematography • stabilization, stereo, view interpolation, free-viewpoint video 43 ! 2010 Marc Levoy 3D video stabilization [Agarwala 2011] 44 (http://web.cecs.pdx.edu/~fliu/project/subspace_stabilization/demo.mp4) ! Marc Levoy Camera 3.0: multi-bucket pixels transfer gates TX1 TX2 storage nodes photodetector ! a CMOS pixel architecture with multiple accumulation nodes per pixel ! can flip between TX1 and TX2 very fast • all pixels flip at once 45 ! 2009 Marc Levoy Flash/noflash photography [Agrawal SIGGRAPH 2005] another app: TX1 TX2 HDR capture put flash exposure in middle of (with no differences in object motion ambient exposure between exposures) 46 ! 2009 Marc Levoy !"#$%&'()*$+,-'.+"/ High Dynamic Range Photography Short exposure Long exposure HDR Image 47 !"#$%&'()*$+,-'.+"/ Motion artifacts in HDR 48 !"#$%&'()*$+,-'.+"/ HDR Photography using 4-bucket pixels Images with different exposures tightly interleaved 49 !"#$%&'()*$+,-'.+"/ 4-Bucket Pixel Layout (3D View) 50 Requests Image Sensor Con!gure Application Devices Processor 1 Lens Actions Expose 2 Flash Readout Metadata + ... 3 Imaging Processor Sensor sensor; Images and Statistics Image Statistics Shot low, med, high; Processing Collection 4 low.exposure = 1/80.; med.exposure = 1/20.; high.exposure = 1/5.; sensor.capture(low); sensor.capture(med); sensor.capture(high); Frame frames[3]; frames[0] = sensor.getFrame(); frames[1] = sensor.getFrame(); frames[2] = sensor.getFrame(); fused = mergeHDR(frames); http://graphics.stanford.edu/projects/camera-2.0/.