DIGITAL LIGHT FIELD PHOTOGRAPHY a dissertation submitted to the department of computer science and the committee on graduate studies of stanford university in partial fulfillment of the requirements for the degree of doctor of philosophy Ren Ng July © Copyright by Ren Ng All Rights Reserved ii IcertifythatIhavereadthisdissertationandthat,inmyopinion,itisfully adequateinscopeandqualityasadissertationforthedegreeofDoctorof Philosophy. Patrick Hanrahan Principal Adviser IcertifythatIhavereadthisdissertationandthat,inmyopinion,itisfully adequateinscopeandqualityasadissertationforthedegreeofDoctorof Philosophy. Marc Levoy IcertifythatIhavereadthisdissertationandthat,inmyopinion,itisfully adequateinscopeandqualityasadissertationforthedegreeofDoctorof Philosophy. Mark Horowitz Approved for the University Committee on Graduate Studies. iii iv Acknowledgments I feel tremendously lucky to have had the opportunity to work with Pat Hanrahan, Marc Levoy and Mark Horowitz on the ideas in this dissertation, and I would like to thank them for their support. Pat instilled in me a love for simulating the flow of light, agreed to take me on as a graduate student, and encouraged me to immerse myself in something I had a passion for.Icouldnothaveaskedforafinermentor.MarcLevoyistheonewhooriginallydrewme to computer graphics, has worked side by side with me at the optical bench, and is vigorously carrying these ideas to new frontiers in light field microscopy. Mark Horowitz inspired me to assemble my camera by sharing his love for dismantling old things and building new ones. I have never met a professor more generous with his time and experience. I am grateful to Brian Wandell and Dwight Nishimura for serving on my orals commit- tee. Dwight has been an unfailing source of encouragement during my time at Stanford. I would like to acknowledge the fine work of the other individuals who have contributed to this camera research. Mathieu Brédif worked closely with me in developing the simulation system, and he implemented the original lens correction software. Gene Duval generously donated his time and expertise to help design and assemble the prototype, working even throughillnesstohelpmemeetpublicationdeadlines.AndrewAdamsandMengYucon- tributed software to refocus light fields more intelligently. Kayvon Fatahalian contributed the most to explaining how the system works, andmanyoftheraydiagramsinthesepages areduetohisartistry. Assembling the prototype required custom support from several vendors. Special thanks toKeithWetzelatKodakImageSensorSolutionsforoutstandingsupportwiththephoto- sensorchips,ThanksalsotoJohnCoxatMegavision,SethPappasandAllisonRobertsat Adaptive Optics Associates, and Mark Holzbach at Zebra Imaging. v In addition, I would like to thank Heather Gentner and Ada Glucksman at the Stan- ford Graphics Lab for providing mission-critical administrative support, and John Gerth for keeping the computing infrastructure running smoothly. Thanks also to Peter Catrysse, Brian Curless, Joyce Farrell, Keith Fife, Abbas El Gamal, Joe Goodman, Bert Hesselink, Brad Osgood, and Doug Osheroff for helpful discussions relatedtothiswork. AMicrosoftResearchFellowshiphassupportedmyresearchoverthelasttwoyears.This fellowship gave me the freedom to think more broadly about my graduate work, allowing me to refocus my graphics research on digital photography. A Stanford Birdseed Grant provided the resources to assemble the prototype camera. I would also like to express my gratitude to Stanford University and Scotch College for all the opportunities that they have given me over the years. I would like to thank all my wonderful friends and colleagues at the Stanford Graphics Lab. I can think of no finer individual than Kayvon Fatahalian, who has been an exceptional friendtomebothinandoutofthelab.ManuKumarhasbeenoneofmystrongestsup- porters, and I am very grateful for his encouragement and patient advice. Jeff Klingner is a source of inspiration with his infectious enthusiasm and amazing outlook on life. I would especially like to thank my collaborators: Eric Chan, Mike Houston, Greg Humphreys, Bill Mark, Kekoa Proudfoot, Ravi Ramamoorthi, Pradeep Sen and Rui Wang. Special thanks also to John Owens, Matt Pharr and Bennett Wilburn for being so generous with their time and expertise. I would also like to thank my friends outside the lab, the climbing posse, who have helped make my graduate years so enjoyable, including Marshall Burke, Steph Cheng, Alex Cooper, Polly Fordyce, Nami Hayashi, Lisa Hwang, Joe Johnson, Scott Matula, Erika Monahan, Mark Pauly, Jeff Reichbach, Matt Reidenbach, Dave Weaver and Mike Whitfield. Special thanks are due to Nami for tolerating the hair dryer, spotlights, and the click of my shutter in the name of science. Finally,Iwouldliketothankmyfamily,YiFoong,BengLymnandCheeKeongNg,for their love and support. My parents have made countless sacrifices for me, and have provided me with steady guidance and encouragement. This dissertation is dedicated to them. vi ӈ'PS.BNBBOE1BQBӈ vii viii Contents Acknowledgments v 1 Introduction 1 . TheFocusProbleminPhotography....................... . TrendsinDigitalPhotography.......................... . DigitalLightFieldPhotography......................... .DissertationOverview............................... 2 Light Fields and Photographs 11 . PreviousWork................................... . TheLightFieldFlowingintotheCamera.................... . PhotographFormation............................... .ImagingEquations................................. 3 Recording a Photograph’s Light Field 23 . A Plenoptic Camera Records the Light Field . . ComputingPhotographsfromtheLightField.................. . ThreeViewsoftheRecordedLightField..................... .ResolutionLimitsofthePlenopticCamera................... . Generalizing the Plenoptic Camera . . PrototypeLightFieldCamera........................... . RelatedWorkandFurtherReading........................ ix x contents 4 Digital Refocusing 49 . PreviousWork................................... . ImageSynthesisAlgorithms............................ . TheoreticalRefocusingPerformance....................... . TheoreticalNoisePerformance.......................... . ExperimentalPerformance............................ . TechnicalSummary................................ . PhotographicApplications............................ 5 Signal Processing Framework 79 . PreviousWork................................... . Overview...................................... . Photographic Imaging in the Fourier Domain . .. Generalization of the Fourier Slice Theorem . .. Fourier Slice Photograph Theorem . .. Photographic Effect of Filtering the Light Field . .Band-Limited Analysis of Refocusing Performance . . Fourier Slice Digital Refocusing . . LightFieldTomography.............................. 6 Selectable Refocusing Power 113 . Sampling Pattern of the Generalized Light Field Camera . . OptimalFocusingofthePhotographicLens................... ExperimentswithPrototypeCamera....................... .ExperimentswithRay-TraceSimulator..................... 7 Digital Correction of Lens Aberrations 131 . PreviousWork................................... . TerminologyandNotation............................ . Visualizing Aberrations in Recorded Light Fields . .ReviewofOpticalCorrectionTechniques.................... . DigitalCorrectionAlgorithms.......................... contents xi . Correcting Recorded Aberrations in a Plano-Convex Lens . . SimulatedCorrectionPerformance........................ .. Methods and Image Quality Metrics . .. Case Analysis: Cooke Triplet Lens . .. Correction Performance Across a Database of Lenses . 8 Conclusion 167 A Proofs 171 a. Generalized Fourier Slice Theorem . a. FilteredLightFieldImagingTheorem...................... a. Photograph of a Four-Dimensional Sinc Light Field . Bibliography 177 xii List of Figures 1 Introduction 1 . Coupling between aperture size and depth of field . . Demosaickingtocomputecolor......................... . Refocusing after the fact in digital light field photography . .Dissertationroadmap............................... 2 Light Fields and Photographs 11 . Parameterization for the light field flowing into the camera . . Thesetofallraysflowingintothecamera.................... . Photograph formation in terms of the light field . .Photograph formation when focusing at different depths . . Transformingray-spacecoordinates....................... 3 Recording a Photograph’s Light Field 23 . Sampling of a photograph’s light field provided by a plenoptic camera . . Overviewofprocessingtherecordedlightfield................. . Rawlightfieldphotograph............................ .Conventional photograph computed from the light field photograph . . Sub-aperture images in the light field photograph . . Epipolar images in the light field photograph . . Microlens image variation with main lens aperture size . . Generalized light field camera: ray-space sampling . . Generalized light field camera: raw image data . xiii xiv list of figures . Prototype camera body . . Microlens array in prototype camera . . Schematic and photographs of prototype assembly . 4 Digital Refocusing 49 . Examples of refocusing and extended depth of field . . Shift-and-add refocus algorithm . . Aliasing in under-sampled shift-and-add refocus algorithm . . Comparison of sub-aperture image and digitally extended depth of field . . Improvement in effective depth of focus in the light field camera . . Experimental test