PROJECTIVE VISUAL HULLS Beckman CVR Technical Report 2002–01 Svetlana Lazebnik, M.S. Department of Computer Science University of Illinois at Urbana-Champaign, 2002 Jean Ponce, Advisor This thesis presents an image-based method for computing the visual hull of an object bounded by a smooth surface and observed by a finite number of perspective cameras. The essential structure of the visual hull is projective: to compute an exact topological (combinatorial) description of its boundary, we do not need to know the Euclidean properties of the input cameras or of the scene. Unlike most existing visual hull computation methods, ours requires only a projective reconstruction of the camera matrices, or equivalently, the epipolar geometry between each pair of cameras in the scene. Starting with a rigorous theoretical framework of oriented projective geometry and projective differential geometry, we develop a suite of algorithms to construct the visual hull and associated data structures. The thesis discusses our implementation of the algorithms, and presents experimental results on synthetic and real data sets. PROJECTIVE VISUAL HULLS BECKMAN CVR TECHNICAL REPORT 2002–01 BY SVETLANA LAZEBNIK B.S., DePaul University, 2000 THESIS Submitted in partial fulfillment of the requirements for the degree of Master of Science in Computer Science in the Graduate College of the University of Illinois at Urbana-Champaign, 2002 Urbana, Illinois c Copyright by Svetlana Lazebnik, 2002 ABSTRACT This thesis presents an image-based method for computing the visual hull of an object bounded by a smooth surface and observed by a finite number of perspective cameras. The essential structure of the visual hull is projective: to compute an exact topological (combinatorial) description of its boundary, we do not need to know the Euclidean properties of the input cameras or of the scene. Unlike most existing visual hull computation methods, ours requires only a projective reconstruction of the camera matrices, or equivalently, the epipolar geometry between each pair of cameras in the scene. Starting with a rigorous theoretical framework of oriented projective geometry and projective differential geometry, we develop a suite of algorithms to construct the visual hull and associated data structures. The thesis discusses our implementation of the algorithms, and presents experimental results on synthetic and real data sets. iii To Max iv ACKNOWLEDGMENTS First and foremost, thanks are due to my advisor, Jean Ponce, for finding my research interesting, holding it to an exacting standard, and constantly telling me to be more positive. I gratefully acknowledge the National Science Foundation for supporting this research under the grant IRI-990709, and the Computer Science Department and the College of Engineering for supporting me with the SURGE fellowship and various awards. I would also like to thank Edmond Boyer for providing the gourd data set, and Steve Sullivan for providing the squash and the Steve data sets. Both Edmond and Steve were responsible for inspiring the research that eventually developed into this thesis. Fred Roth- ganger also deserves a mention for taking time for idle conversations and for carrying on the constant uphill battle to keep the lab machines up and running. Thanks are due to my family: my grandma and parents for constantly asking when the thesis will be done, and my sister Maria for her silent brand of commiseration. Good luck in grad school, Maria! Finally, I must say that this thesis would never have been completed on time without the help and loving care of my husband, Dr. Max Raginsky. I hope you will be there for me when I am writing my Ph.D. thesis! v TABLE OF CONTENTS CHAPTER PAGE 1 Introduction ................................................... 1 1.1 DefiningtheVisualHull........................................ 2 1.2 PreviousWork:ComputingDiscreteVisualHulls ..................... 6 1.2.1 VolumeIntersection...................................... 6 1.2.2 ShapefromDeformingContours............................. 8 1.2.3 ApplicationsofVisualHulls................................ 10 1.3 MathematicalIngredients....................................... 12 1.4 Overview................................................... 13 2 Oriented Projective Geometry .................................... 15 2.1 Basics ..................................................... 16 2.1.1 OrientedProjectiveSpace ................................. 16 2.1.2 Flats................................................. 18 2.1.3 Join,Meet,andRelativeOrientation ......................... 20 2.1.4 OrientedProjectiveTransformations ......................... 23 2.2 ComputingwithFlats.......................................... 25 2.2.1 OrientedProjectiveFrames................................ 25 2.2.2 SimplexOrientation...................................... 27 2.2.3 RepresentingGeneralFlats ................................ 28 2.2.4 RepresentingProjectiveTransformations ...................... 30 2.3 ImagingGeometryofaSingleCamera.............................. 32 2.4 OrientedMulti-ViewGeometry................................... 39 2.4.1 Fundamental Matrix . .................................. 39 2.4.2 OrientedTrifocalTensor .................................. 45 2.4.3 OrientedTransfer ....................................... 48 2.4.3.1 TransferUsingEpipolarGeometry.................... 49 2.4.3.2 TransferUsingtheTrifocalTensor.................... 51 2.5 OrientedProjectiveReconstruction................................ 54 3 Projective Differential Geometry .................................. 58 3.1 Curves..................................................... 59 3.1.1 DifferentialEquationsofCurves............................. 59 3.1.2 OsculatingSpaces ....................................... 62 vi 3.1.3 OrderofContact........................................ 64 3.2 Surfaces.................................................... 65 3.2.1 OrderofContactofSurfaces ............................... 67 3.2.2 DevelopableSurfaces..................................... 67 3.2.3 ConjugateNets......................................... 69 3.2.4 AsymptoticDirections.................................... 72 3.2.5 AlternativeDefinitionsofConjugacy ......................... 75 3.2.6 LocalShape............................................ 79 3.3 OrientingCurvesandSurfaces ................................... 83 3.3.1 OrientingPlaneCurves................................... 84 3.3.2 OrientingSurfaces....................................... 88 4 Visual Hulls .................................................... 93 4.1 PropertiesofRimsandApparentContours.......................... 93 4.2 FrontierPoints...............................................102 4.3 TheRimMesh...............................................109 4.3.1 OrientedStructureoftheRimMesh..........................111 4.3.2 ReconstructingtheRimMesh ..............................112 4.3.3 CombinatorialComplexityoftheRimMesh....................119 4.4 IntersectionCurves............................................120 4.4.1 GeometricPropertiesofIntersectionCurves....................122 4.4.2 TracingIntersectionCurves................................130 4.5 The1-SkeletonoftheVisualHull..................................142 4.5.1 ClippingIntersectionCurves ...............................142 4.5.2 IntersectionPoints.......................................144 4.5.3 AnIncrementalAlgorithm.................................147 4.6 ComputingtheFacesoftheVisualHull ............................151 4.6.1 RayIntervals...........................................152 4.6.2 VerticalDecomposition ...................................158 4.6.3 ConvexObjects:TheVisualHullandtheRimMesh .............162 5 Implementation and Results ......................................166 5.1 ImplementationDetails.........................................166 5.1.1 DiscreteContourRepresentation ............................166 5.1.2 General Position Assumptions . ............................169 5.1.3 3DReconstruction.......................................173 5.1.4 Efficiency .............................................174 5.2 RimMeshResults ............................................175 5.3 VisualHullResults............................................180 6 Conclusion .....................................................194 6.1 Summary...................................................194 6.2 FutureWork.................................................196 vii APPENDIX A Oriented Formulas .................................201 A.1 Formulas for T2 ..............................................201 A.2 Formulas for T3 ..............................................202 A.3 Algebraic and Infinitesimal Properties of Join and Meet . ................205 REFERENCES .................................................207 viii CHAPTER 1 Introduction Suppose that we have taken a few snapshots of an object from a few known camera viewpoints and then extracted the silhouette of the object from each photograph. Clearly, we have lost most of the information about the 3D shape of the object. However, we can still try our best to reverse the imaging process by reconstructing this shape. We can imagine each camera as a slide projector emitting a cone of rays from each pinhole through the silhouette on its image plane. The object is then constrained to lie in the region of space that falls inside the cone due to each camera. This region, called the visual hull, is the least-committed estimate of the shape of the object based on silhouette data alone.