Modeling and Rendering Architecture from Photographs by Paul Ernest Debevec B.S.E. (University of Michigan at Ann Arbor) 1992 B.S. (University of Michigan at Ann Arbor) 1992 A dissertation submitted in partial satisfaction of the requirements for the degree of Doctor of Philosophy in Computer Science in the GRADUATE DIVISION of the UNIVERSITY of CALIFORNIA at BERKELEY Committee in charge: Professor Jitendra Malik, Chair Professor John Canny Professor David Wessel Fall 1996 Modeling and Rendering Architecture from Photographs Copyright Fall 1996 by Paul Ernest Debevec 1 Abstract Modeling and Rendering Architecture from Photographs by Paul Ernest Debevec Doctor of Philosophy in Computer Science University of California at Berkeley Professor Jitendra Malik, Chair Imagine visiting your favorite place, taking a few pictures, and then turning those pictures into a photorealisic three-dimensional computer model. The work presented in this thesis combines techniques from computer vision and computer graphics to make this possible. The applications range from architectural planning and archaeological reconstructions to virtual environments and cinematic special effects. This thesis presents an approach for modeling and rendering existing architectural scenes from sparse sets of still photographs. The modeling approach, which combines both geometry-based and image-based techniques, has two components. The ®rst component is an interactive photogram- metric modeling method which facilitates the recovery of the basic geometry of the photographed scene. The photogrammetric modeling approach is effective, convenient, and robust because it ex- ploits the constraints that are characteristic of architectural scenes. The second component is a model- based stereo algorithm, which recovers how the real scene deviates from the basic model. By mak- 2 ing use of the model, this new technique robustly recovers accurate depth from widely-spaced im- age pairs. Consequently, this approach can model large architectural environments with far fewer photographs than current image-based modeling approaches. For producing renderings, this thesis presents view-dependent texture mapping, a method of compositing multiple views of a scene that better simulates geometric detail on basic models. This approach can be used to recover models for use in either geometry-based or image- based rendering systems. This work presents results that demonstrate the approach's ability to create realistic renderings of architectural scenes from viewpoints far from the original photographs. This thesis concludes with a presentation of how these modeling and rendering techniques were used to create the interactive art installation Rouen Revisited, presented at the SIGGRAPH '96 art show. Professor Jitendra Malik Dissertation Committee Chair iii To Herschel 1983 - 1996 iv Contents List of Figures vii List of Tables ix 1 Introduction 1 2 Background and Related Work 8 2.1Cameracalibration.................................. 8 2.2Structurefrommotion................................. 9 2.3 Shape from silhouette contours ............................ 10 2.4 Stereo correspondence . ............................. 15 2.5Rangescanning.................................... 18 2.6Image-basedmodelingandrendering......................... 18 3 Overview 22 4 Camera Calibration 25 4.1Theperspectivemodel................................ 25 4.2 How real cameras deviate from the pinhole model .................. 28 4.3Ourcalibrationmethod................................ 31 4.4 Determining the radial distortion coef®cients . .................. 31 4.5 Determining the intrinsic parameters . ....................... 38 4.6Workingwithuncalibratedimages.......................... 39 5 Photogrammetric Modeling 42 5.1 Overview of the FacËade photogrammetric modeling system . ............ 45 5.2Themodelrepresentation............................... 47 5.2.1 Parameterreduction............................. 47 5.2.2 Blocks.................................... 48 5.2.3 Relations(themodelhierarchy)....................... 51 5.2.4 Symbolreferences.............................. 52 5.2.5 Computingedgepositionsusingthehierarchicalstructure......... 52 5.2.6 Discussion.................................. 54 v 5.3 FacËade'suserinterface................................ 55 5.3.1 Overview................................... 55 5.3.2 A FacËadeproject............................... 57 5.3.3 The windows and what they do ....................... 57 5.3.4 TheCameraParametersForm........................ 60 5.3.5 TheBlockParametersform......................... 61 5.3.6 Reconstructionoptions............................ 66 5.3.7 Othertoolsandfeatures........................... 67 5.4 The reconstruction algorithm ............................. 67 5.4.1 Theobjectivefunction............................ 67 5.4.2 MinimizingtheObjectiveFunction..................... 69 5.4.3 Obtaining an initial estimate . ....................... 70 5.5Results......................................... 72 5.5.1 TheCampanile................................ 72 5.5.2 University High School ........................... 77 5.5.3 Hoover Tower . ............................. 77 5.5.4 TheTajMahalandtheArcdeTrioumphe.................. 79 6 View-Dependent Texture Mapping 81 6.1Motivation....................................... 81 6.2Overview....................................... 82 6.3Projectingasingleimageontothemodel....................... 83 6.3.1 Computingshadowregions......................... 84 6.4 View-dependent composition of multiple images .................. 84 6.4.1 Determining the ®tness of a particular view ................. 85 6.4.2 Blendingimages............................... 88 6.5 Improving rendering quality ............................. 89 6.5.1 Reducingseamsinrenderings........................ 89 6.5.2 Removalofobstructions........................... 89 6.5.3 Filling in holes . ............................. 91 6.6 Results: the University High School ¯y-around . .................. 92 6.7Possibleperformanceenhancements......................... 92 6.7.1 Approximatingthe®tnessfunctions..................... 95 6.7.2 Visibility preprocessing ........................... 95 7 Model-Based Stereo 96 7.1Motivation....................................... 96 7.2Differencesfromtraditionalstereo.......................... 97 7.3Epipolargeometryinmodel-basedstereo...................... 100 7.4 The matching algorithm . ............................. 102 7.5Results......................................... 103 vi 8 Rouen Revisited 105 8.1Overview....................................... 105 8.2 Artistic description .................................. 105 8.3ThemakingofRouenRevisited........................... 109 8.3.1 Takingthepictures.............................. 109 8.3.2 Mosaicing the Beta photographs ....................... 112 8.3.3 Constructingthebasicmodel......................... 112 8.4Recoveringadditionaldetailwithmodel-basedstereo................ 115 8.4.1 Generatingsurfacemeshes.......................... 115 8.4.2 Rectifying the series of images . ....................... 116 8.5 Recovering a model from the old photographs . .................. 120 8.5.1 Calibrating the old photographs ....................... 120 8.5.2 Generatingthehistoricgeometry....................... 121 8.6RegisteringtheMonetPaintings........................... 122 8.6.1 Catalogingthepaintingsbypointofview.................. 122 8.6.2 Solving for Monet's position and intrinsic parameters ............ 122 8.6.3 Renderingwithview-dependenttexturemapping.............. 123 8.6.4 Signingthework............................... 125 Bibliography 132 A Obtaining color images and animations 139 vii List of Figures 1.1Previousarchitecturalmodelingprojects....................... 2 1.2Schematicofourhybridapproach.......................... 4 1.3TheImmersion'94stereoimagesequencecapturerig................ 5 1.4TheImmersion'94image-basedmodelingandrenderingproject.......... 6 2.1 Tomasi and Kanade 1992 . ............................. 11 2.2 Taylor and Kriegman 1995 . ............................. 12 2.3 The Chevette project 1991 . ............................. 14 2.4 Szeliski's silhouette modeling project 1990 . .................. 15 2.5Modelingfromrangeimages............................. 19 4.1 Convergence of imaged rays in a lens . ....................... 30 4.2Originalcheckerboardpattern............................ 32 4.3Edgesofcheckerboardpattern............................ 33 4.4Scalededgesofcheckerboardpattern......................... 34 4.5 Filtered checkerboard corners ............................. 35 4.6 Distortion error . .................................. 35 4.7 Scaled edges of checkerboard pattern, after undistortion . ............ 37 4.8 The intrinsic calibration object at several orientations ................ 38 4.9TheoriginalBerkeleycampus............................ 40 5.1 Clock tower photograph with marked edges and reconstructed model . ...... 43 5.2Reprojectedmodeledgesandsyntheticrendering.................. 44 5.3Atypicalblock.................................... 49 5.4 A geometric model of a simple building ....................... 50 5.5Themodel'shierarchicalrepresentation....................... 50 5.6Blockparametersassymbolreferences........................ 53 5.7 A typical screen in the FacËademodelingsystem................... 56 5.8Theblockform.................................... 61 5.9Theblockformwithatwirl.............................. 65 5.10 Projection of a line onto the image plane, and the