Computational Photography Research at Stanford

Home , Burst mode (photography), Marc Levoy

Computational photography research at Stanford

January 2012

Marc Levoy Computer Science Department Stanford University Outline

! Light ﬁeld photography

! FCam and the Stanford Frankencamera

! User interfaces for computational photography

! Ongoing projects • FrankenSLR • burst-mode photography • languages and architectures for programmable cameras • computational videography and cinematography • multi-bucket pixels

2 ! Marc Levoy Light field photography using a handheld plenoptic camera

Ren Ng, Marc Levoy, Mathieu Brédif, Gene Duval, Mark Horowitz and Pat Hanrahan

(Proc. SIGGRAPH 2005 and TR 2005-02) Light field photography [Ng SIGGRAPH 2005]

! Marc Levoy Prototype camera

Contax medium format camera Kodak 16-megapixel sensor

Adaptive Optics microlens array 125µ square-sided microlenses

4000 ! 4000 pixels ÷ 292 ! 292 lenses = 14 ! 14 pixels per lens

Digital refocusing

! 2010 Marc Levoy Example of digital refocusing

! 2010 Marc Levoy Refocusing portraits

! 2010 Marc Levoy Light field photography

(Flash demo)

! 2010 Marc Levoy Commercialization

• trades off (excess) spatial resolution for ability to refocus and adjust the perspective

• sensor pixels should be made even smaller, subject to the diffraction limit, for example: 36mm ! 24mm ÷ 2.5µ pixels = 266 Mpix 20K ! 13K pixels 4000 ! 2666 pixels ! 20 ! 20 rays per pixel or 2000 ! 1500 pixels ! 3 ! 3 rays per pixel = 27 Mpix

! Marc Levoy Camera 2.0

Marc Levoy Computer Science Department Stanford University The Stanford Frankencameras

Frankencamera F2 Nokia N900 “F”

! facilitate research in experimental computational photography

! for students in computational photography courses worldwide

! proving ground for plugins and apps for future cameras 14 ! 2010 Marc Levoy CS 448A - Computational Photography

15 ! 2010 Marc Levoy CS 448 projects

M. Culbreth, A. Davis!! ! ! A phone-to-phone 4D light field fax machine J. Koberstein, A. Mattos!! ! Motion blur detection as a game controller N. Sakunkoo!! ! ! ! ! ! Camera-assisted self portraits on mobile devices C. DuHadway!! ! ! ! ! ! Driving/parking assistive mosaics using car cameras C.-H. Chen, S.H. Park!!!High dynamic range (HDR) viewfinding V. Chandrasekhar, G. Takacs!High dynamic range imaging on the Nokia N95 L. Kim, S. Que !! ! !User interfaces for adjusting focus and tone A.Y.L. Chan, S. Meador!! !Photography using circular/hemispherical ring flash P. Huang, Z. Zheng !! ! Image deblurring from multiple images M. Smith !! ! ! ! ! ! ! Representative (non-real) thumbnails for small screens A. Medearis !!!!!!Multi-shot noise reduction for sports photography D. Johnson !! ! ! ! ! ! Synthetic aperture photography using a painted aperture D. Chan !! ! ! ! ! ! Synthetic aperture rear view mirror for safer driving

16 ! 2010 Marc Levoy Frankencamera architecture

Requests Image Sensor Con!gure Application Devices Processor 1 Lens Actions Expose 2 Flash Readout Metadata + ... 3

Imaging Processor Images and Statistics Image Statistics Processing Collection 4

17 ! 2010 Marc Levoy Frankencamera software: the FCAM API

Sensor sensor; Flash flash; vector burst(2);

burst[0].exposure = 1/200.; burst[1].exposure = 1/30.;

Flash::FireAction fire(&flash); fire.time = burst[0].exposure/2; burst[0].actions.insert(fire);

sensor.stream(burst);

while (1) { Frame flashFrame = sensor.getFrame(); Frame noflashFrame = sensor.getFrame(); } 18 ! 2010 Marc Levoy Demonstration applications

• Canon 430EX (smaller ﬂash) strobed continuously

• Canon 580EX (larger ﬂash) ﬁred once at end of exposure

19 ! 2010 Marc Levoy Fcam is being actively ported to...

! Android (at least by NVIDIA)

! iOS?

! Nokia/Microsoft?

! Android point-and-shoot camera?

20 ! 2010 Marc Levoy Prospects for computational photography

! the megapixel wars are over (and it’s about time)

! computational photography is the next battleground in the camera industry (it’s already starting)

21 ! Marc Levoy Premise: available computing power in cameras is rising faster than megapixels

12 70

9 52.5

6 35 megapixels 3 17.5 billions of pixels / second 0 0 2005 2006 2007 2008 2009 2010 Avg megapixels in new cameras, CAGR = 1.2 NVIDIA GTX texture fill rate, CAGR = 1.8 (CAGR for Moore’s law = 1.5)

! this “headroom” permits more computation per pixel,

22 or more frames per second, or less custom hardware ! Marc Levoy Interfaces for computational photography

! the megapixel wars are over (long overdue)

! computational photography is the next battleground in the camera industry (it’s already starting)

! how will these features appear to consumers? • standard and invisible • standard and visible (and disable-able) • aftermarket plugins and apps for your camera

23 ! Marc Levoy Interfaces for computational photography

! the megapixel wars are over (long overdue)

! computational photography is the next battleground in the camera industry (it’s already starting)

! how will these features appear to consumers? • standard and invisible • standard and visible (and disable-able) • aftermarket plugins and apps for your camera

! but aren’t cameras too complicated already? • won’t computational photography make this worse?

24 ! Marc Levoy Sony NEX-5

! manual is 159 pages long • and full of grammatical mistakes

! scene modes are not explained well • Sunset: Shoots the red of the sunset beautifully. (page 43)

! user interface is difﬁcult to navigate 1 8 2 9 3 10 4 11 5 12 6 13 7 25 ! Marc Levoy ProHDR for the iPhone 4S

" 1 " " 2 "

! most popular HDR capture method on the market

! app developers are making photography easy again! 26 ! Marc Levoy Free

27 Free

28 Free

29 Free

circa 500,000 downloads in 2011

30 iPhone 4, single HD video frame

31 SynthCam, align & average ~30 frames

SNR increases as sqrt(# of frames)

32 SynthCam Canon 5DII, 1/400 sec Canon 5DII, average of 15 shots of 1/6 sec

(Marc Levoy)

(Marc Levoy) Removing foreground objects by translating the camera

• align the shots • match histograms • apply median filter

! 2010 Marc Levoy Ongoing projects

! Frankencamera F3 (“FrankenSLR”) • SLR-quality sensor, but big, heavy, and expensive ($10K)

Micron MT9001 Cypress LUPA 4000 • cell phone quality • DSLR quality • $150 • $1500 • arbitrary ROIs and non-destructive readout

39 ! 2010 Marc Levoy Ongoing projects

! Frankencamera F3 (“FrankenSLR”) • SLR-quality sensor, but big, heavy, and expensive ($10K)

! high-speed burst-mode photography • all still cameras should capture at (up to) 500 fps • capture while aiming the camera - no shutter half-press • frameless photography - ROIs, MOIs (“M” = Moment)

40 ! 2010 Marc Levoy Burst-mode photography

41 ! 2010 Marc Levoy Ongoing projects

! Frankencamera F3 (“FrankenSLR”) • SLR-quality sensor, but big, heavy, and expensive ($10K)

! languages and architectures for programmable cameras • FCam was an API for camera control • now we need APIs for photographic processing and computer vision

42 ! 2010 Marc Levoy Ongoing projects

! Frankencamera F3 (“FrankenSLR”) • SLR-quality sensor, but big, heavy, and expensive ($10K)

! languages and architectures for programmable cameras • FCam was an API for camera control • now we need APIs for photographic processing and computer vision

! computational videography & cinematography • stabilization, stereo, view interpolation, free-viewpoint video

43 ! 2010 Marc Levoy 3D video stabilization [Agarwala 2011]

44 (http://web.cecs.pdx.edu/~ﬂiu/project/subspace_stabilization/demo.mp4) ! Marc Levoy Camera 3.0: multi-bucket pixels

transfer gates

TX1 TX2

storage nodes

photodetector

! a CMOS pixel architecture with multiple accumulation nodes per pixel

! can flip between TX1 and TX2 very fast • all pixels flip at once 45 ! 2009 Marc Levoy Flash/noflash photography [Agrawal SIGGRAPH 2005]

another app: TX1 TX2 HDR capture put ﬂash exposure in middle of (with no differences in object motion ambient exposure between exposures)

46 ! 2009 Marc Levoy !"#$%&'()*$+,-'.+"/ High Dynamic Range Photography

Short exposure Long exposure HDR Image

47 !"#$%&'()*$+,-'.+"/ Motion artifacts in HDR

48 !"#$%&'()*$+,-'.+"/ HDR Photography using 4-bucket pixels

Images with different exposures tightly interleaved

49 !"#$%&'()*$+,-'.+"/ 4-Bucket Pixel Layout (3D View)

50 Requests Image Sensor Con!gure Application Devices Processor 1 Lens Actions Expose 2 Flash Readout Metadata + ... 3

Imaging Processor Sensor sensor; Images and Statistics Image Statistics Shot low, med, high; Processing Collection 4 low.exposure = 1/80.; med.exposure = 1/20.; high.exposure = 1/5.;

sensor.capture(low); sensor.capture(med); sensor.capture(high);

Frame frames[3]; frames[0] = sensor.getFrame(); frames[1] = sensor.getFrame(); frames[2] = sensor.getFrame();

fused = mergeHDR(frames); http://graphics.stanford.edu/projects/camera-2.0/