Object Shape and Reflectance Modeling from Color Image Sequence Yoichi Sato CMU-RI-TR-97-06 Submitted in partial fulfillment of the requirements for the degree of Doctor of Philosophy in the field of Robotics The Robotics Institute Carnegie Mellon University Pittsburgh, Pennsylvania 15213 January 1997 © 1997 Yoichi Sato This work was sponsored in part by the Advanced Research Projects Agency under the Department of the Army, Army Research Office under grant number DAAH04-94-G-0006, and partially by NSF under Contract IRI- 9224521. Views and conclusions contained in this document are those of the authors and should not be interpreted as necessarily representing official policies or endorsements, either expressed or implied, of the United States Govern- ment. i Abstract This thesis describes the automatic reconstruction of 3D object models from observa- tion of real objects. As a result of the significant advancement of graphics hardware and image rendering algorithms, 3D computer graphics capability has become available even on low-end computers. However, it is often the case that 3D object models are created manu- ally by users. That input process is normally time-consuming and can be a bottleneck for realistic image synthesis. Therefore, techniques to obtain object models automatically by observing real objects could have great significance in practical applications. For generating realistic images of a 3D object, two aspects of information are neces- sary: the object’s shape and its reflectance properties such as color and specularity. A num- ber of techniques have been developed for modeling object shapes by observing real objects. However, attempts to model reflectance properties of real objects have been rather limited. In most cases, modeled reflectance properties are too simple or too complicated to be used for synthesizing realistic images of the object. One of the main reasons why modeling of reflectance properties has been unsuccessful, compared with modeling of object shapes, is that both diffusely reflected lights and specu- larly reflected lights, i.e., the diffuse and specular reflection components, are treated together, and therefore, estimation of reflectance properties becomes unreliable. To elimi- nate this problem, the two reflection components should be separated prior to estimation of reflectance properties. For this purpose, we developed a new method called goniochromatic space analysis (GSA) which separates two fundamental reflection components from a color image sequence. Based on GSA, we studied two approaches for generating 3D models from observation of real objects. For objects with smooth surfaces, we developed a new method which exam- ines a sequence of color images taken under a moving light source. The diffuse and specular reflection components are first separated from the color image sequence; then, object sur- face shapes and reflectance parameters are simultaneously estimated based on the separation results. For creating object models with more complex shapes and reflectance properties, we proposed another method which uses a sequence of range and color images. In this method, GSA is further extended to handle a color image sequence taken by changing object posture. To extend GSA to a wider range of applications, we also developed a method for shape and reflectance recovery from a sequence of color images taken under solar illumination. The method was designed to handle various problems particular to images taken using solar illuminations, e.g., more complex illumination and shape ambiguity caused by the sun’s coplanar motion. This thesis presents new approaches for modeling object surface reflectance properties, as well as shapes, by observing real objects in both indoor and outdoor environments. The methods are based on a novel method called goniochromatic space analysis for separating the two fundamental reflection components from a color image sequence. ii iii Acknowledgments I would like to express my deepest gratitude to my wife, Imari Sato, and to my parents, Yoshitaka Sato and Kazuko Sato, who always have been supportive throughout my years at Carnegie Mellon University. I would also like to express my gratitude to Katsushi Ikeuchi for being my adviser and mentor. From him, I have learned how to conduct research in the field of computer vision. I have greatly benefited from his support and enthusiasm over the past five years. I am also grateful to my thesis committee members Martial Hebert, Steve Shafer, and Shree Nayar for their careful reading of this thesis and for providing valuable feedback regarding my work. For taking the time to proofread this thesis, I am very grateful to Marie Elm. She always has been kind to spare her time for correcting my writing and improving my writing skills. I was fortunate to have many great people to work with in the VASC group at CMU. In particular I would like to thank members of our Task Oriented Vision Lab group for their insights and ideas which are embedded in my work: Prem Janardhan, Sing Bing Kang, George Paul, Harry Shum, Fred Solomon, and Mark Wheeler; special thanks go to Fred Solomon, who patiently taught me numerous hands-on skills necessary for conducting experiments. I have also benefited from the help of visiting scientists in our group, including Santiago Conant-Pablos, Kazunori Higuchi, Yunde Jiar, Masato Kawade, Hiroshi Kimura, Tetsuo Kiuchi, Jun Miura, Kotaro Ohba, Ken Shakunaga, Yutaka Takeuchi, and Taku Yamazaki. We all had many fun barbecue parties at Katsu’s place during my stay in Pitts- burgh. I will miss very much those parties and Katsu's excellent homemade wine. Finally, I would once again like to thank my family for their love, support, and encour- agement, especially my wife, Imari. Since Imari and I married, my life has always been quite wonderful; she has made the hard times seem as nothing, and the good times an abso- lute delight. iv v Table of Contents Chapter 1 Introduction and Overview . 1 1.1 Goniochromatic Space Analysis of Reflection. .5 1.2 Object Modeling from Color Image Sequence . .7 1.3 Object Modeling from Range and Color Image Sequences . .8 1.4 Reflectance Analysis under Solar Illumination . .11 1.5 Thesis Outline . .12 Chapter 2 Goniochromatic Space Analysis of Reflection . 13 2.1 Background. .13 2.2 The RGB Color Space . .17 2.3 The I-q (Intensity - Illuminating/Viewing Angle) Space . .19 2.4 The Goniochromatic Space. .20 Chapter 3 vi Object Modeling from Color Image Sequence . 23 3.1 Reflection Model . 24 3.1.1 The Lambertian Model . 26 3.1.2 The Torrance-Sparrow Reflection Model . 27 3.1.3 Image Formation Model . 30 3.2 Decomposition of Reflection Components . 31 3.3 Estimation of the Specular Reflection Color . 35 3.3.1 Previously Developed Methods . 35 3.3.1.1 Lee’s Method . 35 3.3.1.2 Tominaga and Wandell’s Method. 36 3.3.1.3 Klinker, Shafer, and Kanade’s Method . 37 3.3.2 Our Method for Estimating an Illuminant Color . 38 3.4 Estimation of the Diffuse Reflection Color . 39 3.5 Experimental Results . 40 3.5.1 Experimental Setup . 41 3.5.2 Estimation of Surface Normal and Reflectance Parameters . 43 3.5.3 Shiny Dielectric Object . 43 3.5.4 Matte Dielectric Object . 48 3.5.5 Metal Object . 51 3.5.6 Shape Recovery . 53 3.5.7 Reflection Component Separation with Non-uniform Reflectance. 55 3.6 Summary . 59 Chapter 4 vii Object Modeling from Range and Color Images: Object Models Without Texture . 61 4.1 Background. .62 4.2 Image Acquisition System . .64 4.3 Shape Reconstruction from Multiple Range Images . .66 4.3.1 Our Method for Merging Multiple Range Images . .68 4.3.2 Measurement. .69 4.3.3 Shape Recovery . .70 4.4 Mapping Color Images onto Recovered Object Shape. .71 4.5 Reflectance Parameter Estimation . .75 4.5.1 Reflection Model . .75 4.5.2 Reflection Component Separation . .76 4.5.3 Reflectance Parameter Estimation for Segmented Regions . .78 4.6 Synthesized Images with Realistic Reflection . .81 4.7 Summary. .83 Chapter 5 Object Modeling from Range and Color Images: Object Models With Texture. 85 5.1 Dense Surface Normal Estimation . .87 5.2 Diffuse Reflection Parameter Estimation . .88 5.3 Specular Reflection Parameter Estimation . .89 5.4 Experimental Results . .90 5.5 Summary. .98 viii Chapter 6 Reflectance Analysis under Solar Illumination . 101 6.1 Background . 101 6.2 Reflection Model Under Solar Illumination . 102 6.3 Removal of the Reflection Component from the Skylight . 106 6.4 Removal of the Specular Component from the Sunlight. 107 6.5 Obtaining Surface Normals . 107 6.5.1 Two Sets of Surface Normals . 107 6.5.2 Unique Surface Normal Solution. 109 6.6 Experimental Results: Laboratory Setup . 110 6.7 Experimental Result: Outdoor Scene (Water Tower) . 114 6.8 Summary ..