Multiple Viewpoint Rendering for Three-Dimensional Displays Michael W. Halle S.B., Massachusetts Institute of Technology (1988) S.M.V.S., Massachusetts Institute of Technology (1991) Submitted to the Program in Media Arts and Sciences, School of Architecture and Planning in partial fulfillment of the requirements for the degree of Doctor of Philosophy at the Massachusetts Institute of Technology June 1997 c Massachusetts Institute of Technology 1997. All rights reserved. Author Program in Media Arts and Sciences, School of Architecture and Planning March 14, 1997 Certified by Stephen A. Benton Professor of Media Arts and Sciences Thesis Supervisor Accepted by Stephen A. Benton Chairman, Departmental Committee on Graduate Students Multiple Viewpoint Rendering for Three-Dimensional Displays by Michael W. Halle Submitted to the Program in Media Arts and Sciences, School of Architecture and Planning on March 14, 1997, in partial fulfillment of the requirements for the degree of Doctor of Philosophy Abstract This thesis describes a computer graphics method for efficiently rendering images of static ge- ometric scene databases from multiple viewpoints. Three-dimensional displays such as parallax panogramagrams, lenticular panoramagrams, and holographic stereograms require samples of im- age data captured from a large number of regularly spaced camera images in order to produce a three-dimensional image. Computer graphics algorithms that render these images sequentially are inefficient because they do not take advantage of the perspective coherence of the scene. A new algorithm, multiple viewpoint rendering (MVR), is described which produces an equivalent set of images one to two orders of magnitude faster than previous approaches by considering the image set as a single spatio-perspective volume. MVR uses a computer graphics camera geometry based on a common model of parallax-based three-dimensional displays. MVR can be implemented us- ing variations of traditional computer graphics algorithms and accelerated using standard computer graphics hardware systems. Details of the algorithm design and implementation are given, includ- ing geometric transformation, shading, texture mapping and reflection mapping. Performance of a hardware-based prototype implementation and comparison of MVR-rendered and conventionally rendered images are included. Applications of MVR to holographic video and other display sys- tems, to three-dimensional image compression, and to other camera geometries are also given. Thesis Supervisor: Stephen A. Benton Title: Professor of Media Arts and Sciences This work has been sponsored in part by the Honda R & D Company, NEC, IBM, and the Design Staff of the General Motors Corporation. Doctoral dissertation committee Thesis Advisor Stephen A. Benton Professor of Media Arts and Sciences Massachusetts Institute of Technology Thesis Reader V. Michael Bove Associate Professor of Media Technology Massachusetts Institute of Technology Thesis Reader Seth Teller Assistant Professor of Computer Science and Engineering Massachusetts Institute of Technology Acknowledgements I begin by thanking my thesis committee: V. Michael Bove of the Media Laboratory, Seth Teller of the Laboratory for Computer Science, and my professor Stephen Benton of the Media Lab. This process has not been a conventional one, and they have shown helpfulness and flexibility throughout it. As my professor, Dr. Benton has been most generous in the support of this work, and my understanding of three-dimensional displays would never have developed without his mentorship over the years. I owe the Spatial Imaging Group at MIT my heartfelt thanks for their support of this research and of me. In particular, Wendy Plesniak, Ravikanth Pappu, and John Underkoffler helped me through my the trying writing and presentation time and looked over drafts of this manuscript. In addition, Wendy designed the beautiful teacup used as a test object in this thesis. All the members of the Spatial Imaging Group and the entire MIT Media Laboratory, both past and present, have been my colleagues and friends since I came to the lab in 1985. I also offer my appreciation for the patience and financial and personal support extended by my group at the Brigham and Women’s Hospital. Ferenc Jolesz and Ron Kikinis have tolerated the lengthy absence this document has caused and have extended only kindness, patience and concern in exchange. The entire gang at the Surgical Planning Lab has just been great to me. Funding from Robert Sproull from Sun Microsystems supported me at the hospital during part of the time this document was written; I am proud to be part of another of his contributions to the field of computer graphics. Finally, I would not be writing this document today if it were not for the education and encour- agement given to me by my parents, the sense of discovery and pride they instilled in me, and the sacrifices they made so that I could choose to have what they could not. They taught me to find wonder in the little things in the world; I could not imagine what my life would be like without that lesson. 4 Contents 1 Introduction 8 1.1Images,models,andcomputation.......................... 9 1.2 Beyond single viewpoint rendering .......................... 11 This work: multiple viewpoint rendering ....................... 12 Chapterpreview.................................... 12 2 The plenoptic function and three-dimensional information 14 2.1 Introduction . ..................................... 14 2.1.1 Asingleobservation............................. 14 2.1.2 Imaging.................................... 15 2.1.3 Animation.................................. 16 2.1.4 Parallax.................................... 17 2.2Depthrepresentations................................. 18 2.2.1 Directional emitters .............................. 21 2.2.2 Occlusion and hidden surfaces ........................ 22 2.2.3 Transparency................................. 22 2.2.4 Fullparallaxandhorizontalparallaxonlymodels.............. 24 2.3Camerasandtheplenopticfunction......................... 25 2.4Parallax-baseddepthsystems............................. 27 2.4.1 Parallaxisthree-dimensionalinformation.................. 28 2.4.2 Displayissues................................ 30 2.5 The photorealistic imaging pipeline ......................... 31 3 Spatial display technology 34 3.1Spatialdisplays.................................... 34 3.1.1 Terminologyofspatialdisplays....................... 34 3.1.2 Photographs:minimalspatialdisplays.................... 35 3.1.3 Stereoscopes................................. 35 3.2 Multiple viewpoint display technologies ....................... 36 3.2.1 Parallaxbarrierdisplays........................... 37 3.2.2 Lenticularsheetdisplays........................... 38 3.2.3 Integral photography . .......................... 39 3.2.4 Holography.................................. 40 3.2.5 Holographicstereograms........................... 41 3.3Acommonmodelforparallaxdisplays........................ 41 3.4Implicationsforimagegeneration.......................... 45 5 4 Limitations of conventional rendering 48 4.1 Bottlenecks in rendering ............................... 48 4.2Improvinggraphicsperformance........................... 49 4.3Testingexistinggraphicssystems........................... 51 4.4Usingperspectivecoherence............................. 53 5 Multiple viewpoint rendering 55 5.1The“regularshearing”geometry........................... 55 5.2Thespatio-perspectivevolume............................ 57 5.2.1 Epipolarplaneimages............................ 58 5.2.2 Polygons and polygon tracks ......................... 60 5.2.3 Polygon slices and PSTs . .......................... 60 5.3 Multiple viewpoint rendering . .......................... 62 5.3.1 BasicgraphicalpropertiesoftheEPI..................... 63 5.4TheMVRalgorithm................................. 65 5.5Overview....................................... 65 5.6SpecificstagesofMVR................................ 68 5.6.1 Geometricscanconversion.......................... 68 5.6.2 Viewindependentshading.......................... 70 5.6.3 Back face culling and two-sided lighting ................... 70 5.6.4 Hiddensurfaceelimination.......................... 73 5.6.5 Anti-aliasing................................. 74 5.6.6 Clipping................................... 74 5.6.7 Texturemapping............................... 75 5.6.8 Viewdependentshading........................... 79 5.6.9 Combiningdifferentshadingalgorithms................... 86 5.6.10 Image data reformatting . .......................... 87 5.6.11Fullparallaxrendering............................ 88 6 Implementation details 90 6.1Basicstructure.................................... 90 6.2BeforetheMVRpipeline............................... 91 6.3 Scene input . ..................................... 91 6.4MVRper-sequencecalculations........................... 92 6.4.1 Per-vertexcalculations............................ 92 6.4.2 Polygon slicing ................................ 94 6.5Device-dependentrendering............................. 96 6.5.1 Conversiontorenderingprimitives...................... 97 6.5.2 Texturemapping............................... 102 6.5.3 Reflectionmapping.............................. 102 6.5.4 Hiddensurfaceremoval........................... 103 6.5.5 Multiple rendering passes .......................... 103 6.6Output......................................... 104 6.7Fullparallaxrenderingissues............................. 105 7 Performance Tests 106 7.1Testparameters...................................