AN IMAGE-BASED APPROACH TO THREE-DIMENSIONAL COMPUTER GRAPHICS by Leonard McMillan Jr. A dissertation submitted to the faculty of the University of North Carolina at Chapel Hill in partial fulfillment of the requirements for the degree of Doctor of Philosophy in the Department of Computer Science. Chapel Hill 1997 Approved by: ______________________________ Advisor: Gary Bishop ______________________________ Reader: Anselmo Lastra ______________________________ Reader: Stephen Pizer © 1997 Leonard McMillan Jr. ALL RIGHTS RESERVED ii ABSTRACT Leonard McMillan Jr. An Image-Based Approach to Three-Dimensional Computer Graphics (Under the direction of Gary Bishop) The conventional approach to three-dimensional computer graphics produces images from geometric scene descriptions by simulating the interaction of light with matter. My research explores an alternative approach that replaces the geometric scene description with perspective images and replaces the simulation process with data interpolation. I derive an image-warping equation that maps the visible points in a reference image to their correct positions in any desired view. This mapping from reference image to desired image is determined by the center-of-projection and pinhole-camera model of the two images and by a generalized disparity value associated with each point in the reference image. This generalized disparity value, which represents the structure of the scene, can be determined from point correspondences between multiple reference images. The image-warping equation alone is insufficient to synthesize desired images because multiple reference-image points may map to a single point. I derive a new visibility algorithm that determines a drawing order for the image warp. This algorithm results in correct visibility for the desired image independent of the reference image’s contents. The utility of the image-based approach can be enhanced with a more general pinhole- camera model. I provide several generalizations of the warping equation’s pinhole-camera model and discuss how to build an image-based representation when information about the reference image’s center-of-projection and camera model is unavailable. iii ACKNOWLEDGMENTS I owe tremendous debts of gratitude to the following: • My advisor and long time friend Gary Bishop who first challenged me to return to graduate school and has subsequently served as both teacher and advocate during the entire process. • My committee members, Fred Brooks, James Coggins, Henry Fuchs, Anselmo Lastra, Steve Pizer, and Turner Whitted who have been endless sources of wisdom, enthusiasm, and inspiration. • The department research faculty, in particular Vern Chi for his advice on limiting cases; and John Poulton, Nick England, and Mary Whitton for their enthusiasm and support. • My department colleagues and fellow students, in particular Bill Mark for his collaborations and willingness to endure my ramblings. • My parents Leonard McMillan Sr. and Joan McMillan for nurturing, encouragement, and their willingness to allow me to take things apart, while knowing that I might not succeed in putting them back together. Also, my brother John McMillan whose belongings I so often dismantled. I wish both to thank and to dedicate this dissertation to my wife Donna for all of the love that she brings to my life, and to my daughter Cassie for all of the joy that she brings. iv v TABLE OF CONTENTS CHAPTER 1 INTRODUCTION.......................................................................................................... 1 1.1 CONVENTIONAL COMPUTER GRAPHICS MODELS ........................................................................... 3 1.2 THESIS STATEMENT AND CONTRIBUTIONS ...................................................................................... 4 1.3 MOTIVATION.................................................................................................................................. 6 1.4 PREVIOUS WORK ............................................................................................................................ 7 1.4.1 Images as approximations........................................................................................................ 8 1.4.2 Images as databases................................................................................................................ 11 1.4.3 Images as models ................................................................................................................... 14 1.5 DISCUSSION.................................................................................................................................. 17 CHAPTER 2 THE PLENOPTIC MODEL......................................................................................... 19 2.1 THE PLENOPTIC FUNCTION........................................................................................................... 20 2.2 GEOMETRIC STRUCTURES IN PLENOPTIC SPACE ............................................................................ 23 2.3 ALTERNATIVE MODELS ................................................................................................................ 26 2.4 SAMPLING AN ENVIRONMENT ...................................................................................................... 26 2.5 SUMMARY .................................................................................................................................... 28 CHAPTER 3 A WARPING EQUATION.......................................................................................... 30 3.1 FROM IMAGES TO RAYS................................................................................................................. 31 3.2 A GENERAL PLANAR-PINHOLE MODEL ....................................................................................... 31 3.3 A WARPING EQUATION FOR SYNTHESIZING PROJECTIONS OF A SCENE......................................... 33 3.4 RELATION TO PREVIOUS RESULTS ................................................................................................. 41 3.5 RESOLVING VISIBILITY................................................................................................................... 44 3.5.1 Visibility Algorithm .............................................................................................................. 45 3.6 RECONSTRUCTION ISSUES ............................................................................................................. 49 3.7 OCCLUSION AND EXPOSURE ERRORS ............................................................................................ 55 3.8 SUMMARY .................................................................................................................................... 59 vi CHAPTER 4 OTHER PINHOLE CAMERAS .................................................................................. 61 4.1 ALTERNATIVE PLANAR PINHOLE-CAMERA MODELS .................................................................... 61 4.1.1 Application-specific planar pinhole-camera models ................................................................ 62 4.1.2 A canonical pinhole model ..................................................................................................... 66 4.1.3 Planar calibration .................................................................................................................. 67 4.2 NONLINEAR PINHOLE-CAMERA MODELS..................................................................................... 68 4.3 PANORAMIC PINHOLE CAMERAS.................................................................................................. 68 4.3.1 Cylindrical pinhole-camera model..........................................................................................70 4.3.2 Spherical model...................................................................................................................... 76 4.4 DISTORTED PINHOLE-CAMERA MODELS....................................................................................... 77 4.4.1 Fisheye pinhole-camera model ................................................................................................ 80 4.4.2 Radial distortion pinhole-camera model ................................................................................. 82 4.5 MODIFYING WARPING EQUATIONS .............................................................................................. 85 4.6 REAL CAMERAS............................................................................................................................. 86 4.6.1 Nonlinear camera calibration................................................................................................. 86 4.7 SUMMARY .................................................................................................................................... 87 CHAPTER 5 WARPING WITHOUT CALIBRATION................................................................... 88 5.1 EPIPOLAR GEOMETRIES AND THE FUNDAMENTAL MATRIX........................................................... 88 5.1.1 The Fundamental Matrix....................................................................................................... 90 5.1.2 Determining a Fundamental Matrix from Point Correspondences ......................................... 93 5.2 RELATING THE FUNDAMENTAL MATRIX TO THE IMAGE-WARPING EQUATION ............................. 95 5.2.1 Image Warps Compatible with a Fundamental Matrix..........................................................