Object Shape and Reflectance Modeling from Color Image Sequence
Yoichi Sato
CMU-RI-TR-97-06
Submitted in partial fulfillment of the requirements for the degree of Doctor of Philosophy in the field of Robotics
The Robotics Institute Carnegie Mellon University Pittsburgh, Pennsylvania 15213
January 1997
© 1997 Yoichi Sato
This work was sponsored in part by the Advanced Research Projects Agency under the Department of the Army, Army Research Office under grant number DAAH04-94-G-0006, and partially by NSF under Contract IRI- 9224521. Views and conclusions contained in this document are those of the authors and should not be interpreted as necessarily representing official policies or endorsements, either expressed or implied, of the United States Govern- ment.
i Abstract
This thesis describes the automatic reconstruction of 3D object models from observa- tion of real objects. As a result of the significant advancement of graphics hardware and image rendering algorithms, 3D computer graphics capability has become available even on low-end computers. However, it is often the case that 3D object models are created manu- ally by users. That input process is normally time-consuming and can be a bottleneck for realistic image synthesis. Therefore, techniques to obtain object models automatically by observing real objects could have great significance in practical applications. For generating realistic images of a 3D object, two aspects of information are neces- sary: the object’s shape and its reflectance properties such as color and specularity. A num- ber of techniques have been developed for modeling object shapes by observing real objects. However, attempts to model reflectance properties of real objects have been rather limited. In most cases, modeled reflectance properties are too simple or too complicated to be used for synthesizing realistic images of the object. One of the main reasons why modeling of reflectance properties has been unsuccessful, compared with modeling of object shapes, is that both diffusely reflected lights and specu- larly reflected lights, i.e., the diffuse and specular reflection components, are treated together, and therefore, estimation of reflectance properties becomes unreliable. To elimi- nate this problem, the two reflection components should be separated prior to estimation of reflectance properties. For this purpose, we developed a new method called goniochromatic space analysis (GSA) which separates two fundamental reflection components from a color image sequence. Based on GSA, we studied two approaches for generating 3D models from observation of real objects. For objects with smooth surfaces, we developed a new method which exam- ines a sequence of color images taken under a moving light source. The diffuse and specular reflection components are first separated from the color image sequence; then, object sur- face shapes and reflectance parameters are simultaneously estimated based on the separation results. For creating object models with more complex shapes and reflectance properties, we proposed another method which uses a sequence of range and color images. In this method, GSA is further extended to handle a color image sequence taken by changing object posture. To extend GSA to a wider range of applications, we also developed a method for shape and reflectance recovery from a sequence of color images taken under solar illumination. The method was designed to handle various problems particular to images taken using solar illuminations, e.g., more complex illumination and shape ambiguity caused by the sun’s coplanar motion. This thesis presents new approaches for modeling object surface reflectance properties, as well as shapes, by observing real objects in both indoor and outdoor environments. The methods are based on a novel method called goniochromatic space analysis for separating the two fundamental reflection components from a color image sequence. ii iii Acknowledgments
I would like to express my deepest gratitude to my wife, Imari Sato, and to my parents, Yoshitaka Sato and Kazuko Sato, who always have been supportive throughout my years at Carnegie Mellon University.
I would also like to express my gratitude to Katsushi Ikeuchi for being my adviser and mentor. From him, I have learned how to conduct research in the field of computer vision. I have greatly benefited from his support and enthusiasm over the past five years. I am also grateful to my thesis committee members Martial Hebert, Steve Shafer, and Shree Nayar for their careful reading of this thesis and for providing valuable feedback regarding my work.
For taking the time to proofread this thesis, I am very grateful to Marie Elm. She always has been kind to spare her time for correcting my writing and improving my writing skills.
I was fortunate to have many great people to work with in the VASC group at CMU. In particular I would like to thank members of our Task Oriented Vision Lab group for their insights and ideas which are embedded in my work: Prem Janardhan, Sing Bing Kang, George Paul, Harry Shum, Fred Solomon, and Mark Wheeler; special thanks go to Fred Solomon, who patiently taught me numerous hands-on skills necessary for conducting experiments. I have also benefited from the help of visiting scientists in our group, including Santiago Conant-Pablos, Kazunori Higuchi, Yunde Jiar, Masato Kawade, Hiroshi Kimura, Tetsuo Kiuchi, Jun Miura, Kotaro Ohba, Ken Shakunaga, Yutaka Takeuchi, and Taku Yamazaki. We all had many fun barbecue parties at Katsu’s place during my stay in Pitts- burgh. I will miss very much those parties and Katsu's excellent homemade wine.
Finally, I would once again like to thank my family for their love, support, and encour- agement, especially my wife, Imari. Since Imari and I married, my life has always been quite wonderful; she has made the hard times seem as nothing, and the good times an abso- lute delight. iv v
Table of Contents
Chapter 1
Introduction and Overview ...... 1
1.1 Goniochromatic Space Analysis of Reflection...... 5 1.2 Object Modeling from Color Image Sequence ...... 7
1.3 Object Modeling from Range and Color Image Sequences ...... 8 1.4 Reflectance Analysis under Solar Illumination ...... 11 1.5 Thesis Outline ...... 12
Chapter 2
Goniochromatic Space Analysis of Reflection ...... 13
2.1 Background...... 13 2.2 The RGB Color Space ...... 17 2.3 The I-q (Intensity - Illuminating/Viewing Angle) Space ...... 19 2.4 The Goniochromatic Space...... 20
Chapter 3 vi
Object Modeling from Color Image Sequence ...... 23
3.1 Reflection Model ...... 24 3.1.1 The Lambertian Model ...... 26 3.1.2 The Torrance-Sparrow Reflection Model ...... 27 3.1.3 Image Formation Model ...... 30 3.2 Decomposition of Reflection Components ...... 31 3.3 Estimation of the Specular Reflection Color ...... 35 3.3.1 Previously Developed Methods ...... 35 3.3.1.1 Lee’s Method ...... 35
3.3.1.2 Tominaga and Wandell’s Method...... 36
3.3.1.3 Klinker, Shafer, and Kanade’s Method ...... 37
3.3.2 Our Method for Estimating an Illuminant Color ...... 38 3.4 Estimation of the Diffuse Reflection Color ...... 39 3.5 Experimental Results ...... 40 3.5.1 Experimental Setup ...... 41 3.5.2 Estimation of Surface Normal and Reflectance Parameters ...... 43 3.5.3 Shiny Dielectric Object ...... 43 3.5.4 Matte Dielectric Object ...... 48
3.5.5 Metal Object ...... 51 3.5.6 Shape Recovery ...... 53 3.5.7 Reflection Component Separation with Non-uniform Reflectance...... 55 3.6 Summary ...... 59
Chapter 4 vii
Object Modeling from Range and Color Images: Object Models Without Texture ...... 61
4.1 Background...... 62 4.2 Image Acquisition System ...... 64 4.3 Shape Reconstruction from Multiple Range Images ...... 66 4.3.1 Our Method for Merging Multiple Range Images ...... 68 4.3.2 Measurement...... 69 4.3.3 Shape Recovery ...... 70 4.4 Mapping Color Images onto Recovered Object Shape...... 71 4.5 Reflectance Parameter Estimation ...... 75 4.5.1 Reflection Model ...... 75 4.5.2 Reflection Component Separation ...... 76 4.5.3 Reflectance Parameter Estimation for Segmented Regions ...... 78 4.6 Synthesized Images with Realistic Reflection ...... 81 4.7 Summary...... 83
Chapter 5
Object Modeling from Range and Color Images: Object Models With Texture...... 85
5.1 Dense Surface Normal Estimation ...... 87 5.2 Diffuse Reflection Parameter Estimation ...... 88 5.3 Specular Reflection Parameter Estimation ...... 89 5.4 Experimental Results ...... 90 5.5 Summary...... 98 viii
Chapter 6
Reflectance Analysis under Solar Illumination ...... 101
6.1 Background ...... 101 6.2 Reflection Model Under Solar Illumination ...... 102 6.3 Removal of the Reflection Component from the Skylight ...... 106 6.4 Removal of the Specular Component from the Sunlight...... 107 6.5 Obtaining Surface Normals ...... 107 6.5.1 Two Sets of Surface Normals ...... 107 6.5.2 Unique Surface Normal Solution...... 109 6.6 Experimental Results: Laboratory Setup ...... 110 6.7 Experimental Result: Outdoor Scene (Water Tower) ...... 114 6.8 Summary ...... 118
Chapter 7
Conclusions ...... 119
7.1 Summary ...... 119 7.1.1 Object Modeling from Color Image Sequence ...... 120 7.1.2 Object Modeling from Range and Color Images...... 120 7.1.3 Reflectance Analysis under Solar Illumination ...... 121 7.2 Thesis Contributions ...... 121 7.3 Directions for Future Research ...... 122 7.3.1 More Complex Reflectance Model ...... 122 7.3.2 Planning of Image Sampling ...... 123 7.3.3 Reflectance Analysis for Shape from Motion ...... 123 ix
7.3.4 More Realistic Illumination Model for Outdoor Scene Analysis...... 124 Color Figures...... 127
Bibliography ...... 133 x xi
List of Figures
Figure 1 Object model generation ...... 2 Figure 2 Object model for computer graphics ...... 3 Figure 3 (a) a gonioreflectometer and (b) a typical measurement of BRDF ...... 4 Figure 4 Goniochromatic space ...... 7 Figure 5 Reflection component separation ...... 8 Figure 6 Synthesized image of an object without texture ...... 10 Figure 7 Synthesized images of an object with texture ...... 10 Figure 8 Image taken under solar illumination ...... 11 Figure 9 A sphere and its color histogram as T shape in the RGB color space . . . . .19 Figure 10 Viewer-centered coordinate system ...... 20 Figure 11 The space ...... 20 Figure 12 The goniochromatic space (synthesized data) ...... 22 Figure 13 Polar plot of the three reflection components ...... 24 Figure 14 Reflection model used in our analysis ...... 25 Figure 15 Internal scattering and surface reflection ...... 26 Figure 16 Solid angles of a light source and illuminated surface ...... 26 Figure 17 Geometry for the Torrance-Sparrow reflection model [85] ...... 30 Figure 18 Measurement at one pixel (synthesized data) ...... 32 Figure 19 Diffuse and specular reflection planes (synthesized data) ...... 34 xii
Figure 20 x-y chromaticity diagram showing the ideal loci of chromaticities correspond- ing to colors from five surfaces of different colors ...... 36 Figure 21 Estimation of illuminant color as an intersection of color signal planes . . 37 Figure 22 T-shape color histogram and two color vectors ...... 38 Figure 23 Estimation of the color vector ...... 40 Figure 24 Geometry matrix (synthesized data) ...... 40 Figure 25 Geometry of the experimental setup ...... 42 Figure 26 Geometry of the extended light source ...... 42 Figure 27 Green shiny plastic cylinder ...... 44 Figure 28 Measured intensities in the goniochromatic space ...... 45 Figure 29 Decomposed two reflection components ...... 46 Figure 30 Loci of two reflection components in the goniochromatic space ...... 47 Figure 31 Diffuse and specular reflection planes ...... 47 Figure 32 Result of fitting ...... 48 Figure 33 Green matte plastic cylinder ...... 49 Figure 34 Measured intensities in the goniochromatic space ...... 49 Figure 35 Two decomposed reflection components ...... 50 Figure 36 Result of fitting ...... 51 Figure 37 Aluminum triangular prism ...... 52 Figure 38 Loci of the intensity in the goniochromatic space ...... 52 Figure 39 Two decomposed reflection components ...... 53 Figure 40 Purple plastic cylinder ...... 54 Figure 41 Needle map ...... 54 Figure 42 Recovered object shape ...... 55 Figure 43 Estimation of illuminant color in the x-y chromaticity diagram ...... 56 Figure 44 Multicolored object ...... 57 Figure 45 Diffuse reflection image ...... 58 Figure 46 Specular reflection image ...... 58 Figure 47 Image acquisition system ...... 65 xiii
Figure 48 Input range data ...... 69 Figure 49 Input color images ...... 70 Figure 50 Recovered object shape ...... 71 Figure 51 View mapping result ...... 73 Figure 52 Intensity change with strong specularity ...... 74 Figure 53 Intensity change with little specularity ...... 74 Figure 54 Geometry for simplified Torrance-Sparrow model ...... 75 Figure 55 Separated reflection components with strong specularity ...... 76 Figure 56 Separated reflection component with little specularity ...... 77 Figure 57 Diffuse image and specular image: example 1 ...... 77 Figure 58 Diffuse image and specular image: example 2 ...... 78 Figure 59 Segmentation result (grey levels represent regions) ...... 80 Figure 60 Synthesized image 1 ...... 82 Figure 61 Synthesized image 2 ...... 82 Figure 62 Synthesized image 3 ...... 83 Figure 63 Object modeling with reflectance parameter mapping ...... 86 Figure 64 Surface normal estimation from input 3D points ...... 88 Figure 65 Diffuse saturation shown in the RGB color space ...... 90 Figure 66 Input range data ...... 91 Figure 67 Input color images ...... 91 Figure 68 Recovered object shape ...... 92 Figure 69 Simplified shape model ...... 93 Figure 70 Estimated surface normals and polygonal normals ...... 93 Figure 71 Color image mapping result ...... 95 Figure 72 Estimated diffuse reflection parameters ...... 95 Figure 73 Selected vertices for specular parameter estimation ...... 96 Figure 74 Interpolated and ...... 97 Figure 75 Synthesized object images ...... 99 Figure 76 Comparison of input color images and synthesized images ...... 100 xiv
Figure 77 Comparison of the spectra of sunlight and skylight [48] ...... 104 Figure 78 Change of color of sun with altitude [48] ...... 104 Figure 79 Three reflection components from solar illumination ...... 105 Figure 80 Sun direction, viewing direction and surface normal in 3D case ...... 108 Figure 81 Diffuse reflection component image (frame 8) ...... 111 Figure 82 Two sets of surface normals ...... 111 Figure 83 The boundary region obtained from two surface normal sets ...... 112 Figure 84 The boundary after medial axis transformation ...... 112 Figure 85 Segmented regions (gray levels represent regions) ...... 113 Figure 86 Right surface normal set ...... 113 Figure 87 Recovered object shape ...... 114 Figure 88 Observed color image sequence of a water tank ...... 115 Figure 89 Extracted region of interest ...... 116 Figure 90 Water tank image without sky reflection component ...... 116 Figure 91 Water tank image after highlight removal ...... 117 Figure 92 Surface normals ...... 117 Figure 93 Recovered shape of the water tank ...... 118 1
Chapter 1
Introduction and Overview
As a result of the significant advancement of graphics hardware and image rendering algorithms, 3D computer graphics capability has become available even on low-end com- puters. At the same time, the rapid spread of the internet technology has caused a significant increase in the demand for 3D computer graphics. For instance, a new format for 3D com- puter graphics on the internet, called VRML, is becoming an industrial standard format, and the number of applications using the format is quickly increasing. Therefore, it is important to create suitable 3D object models for synthesizing realistic computer graphics images.
An object model for computer graphics applications should contain two aspects of information: the shape and reflectance properties of the object. Surface reflectance proper- ties of object models are particularly important for synthesizing realistic computer graphics images since appearance of objects greatly depends on reflectance properties, i.e., how inci- dent lights are reflected on the object surfaces. For instance, a polished metal sphere will look completely different after it is coated with diffuse white paint even though the shape remains exactly the same.
Unfortunately, it is often the case that 3D object models are created manually by users. That input process is normally time-consuming and can be a bottleneck for realistic image synthesis. Alternatively, there might be CAD models of 3D objects available. Even in this case, however, reflectance properties are usually not available as a part of CAD models, and therefore need to be determined. Thus, techniques to obtain object model data automatically by observing real objects could have great significance in practical applications. This is the 2 Chapter 1 main motivation of this thesis work. In this thesis, we describe a novel method for automat- ically creating 3D object models with shape and reflectance properties, by observing real objects.
Previously developed techniques for modeling object shapes by observing real objects use various approaches which include: for instance, range image merging, shape-from- motion, shape-from-shadings, shape-from-focus, and photometric stereo. In addition, there exist sensing devices such as range finders, which can measure 3D object shapes directly. In fact, there are many 3D scanning sensors commercially available today. Various kinds of range sensors have been developed and are being marketed. They include: triangulation- based laser range sensors, time-of-flight laser range sensors, and light-pattern-projection type range sensors. The drawback of these sensors is that they are not designed to measure object reflectance properties.
CAD model object model
measurement by hand
automatic generation
real object
Figure 1 Object model generation 3
Shape
real object Reflectance synthesized image illumination & camera
Figure 2 Object model for computer graphics
Attempts to model reflectance properties of real objects have been rather limited. In most cases, modeled reflectance properties are too simple to be used for synthesizing realis- tic images of the object. If only observed color texture or diffuse texture of a real object sur- face is used (e.g., texture mapping), shading effects such as highlights cannot be reproduced correctly in synthesized images. For instance, if highlights on the object surface are observed in original color images, the highlights are treated as diffuse textures on the object surface and, therefore, remain on the object surface permanently regardless of illuminating and viewing conditions. This should be avoided for realistic image synthesis because high- lights on object surfaces are known to play an important role in conveying a great deal of information about object surface finish and material type.
For modeling surface reflectance properties of real objects, there are two approaches. The first approach is to intensively measure a distribution of reflected lights, i.e., a bidirec- tional reflectance distribution function (BRDF), and to record the distribution as reflectance properties. The second approach is to estimate parameters of some sort of parametric reflec- tion model function, based on relatively coarse measurements of reflected lights.
A BRDF is measured by using a device called a gonioreflectometer. The usual design for such a device incorporates a single photometer that is designed to move in relation to a light source, all under the control of a computer. Because a BRDF is, in general, a function of four angles, two incident and two reflected, such a device must have four degrees of mechanical freedom to measure the complete function. This requires substantial complexity in the apparatus design as well as long periods of time to measure a single surface. Also, real object surfaces very often have non-uniform reflectance: therefore, a single measurement of 4 Chapter 1
BRDF per object is not enough. More particularly, accuracy of measured BRDFs is often questionable even if they are carefully measured [88]. For these reasons, BRDFs have rarely been used for synthesizing computer graphics images in the past.
sensor light source
test material
(a) (b)
Figure 3 (a) a gonioreflectometer and (b) a typical measurement of BRDF the BRDF is redrawn from [75].
Alternatively, we can use a parametric reflectance model to reduce the complexity involved in using BRDFs for synthesizing images. When we have relatively coarse mea- surement of reflected light distribution, the measurement has to be somehow interpolated, so that real distribution of reflected lights can be inferred. Here, the best chance would be to assume some underlying reflection model, i.e., the Torrance-Sparrow reflection model, as a starting point. By estimating parameters of the reflection model, we can interpolate the mea- sured distribution of reflected lights.
Depending on object material types and sampling methods of reflected lights, an appro- priate reflection model should be selected from currently available reflection models. Those currently available reflection models were developed either empirically or analytically. For example, reflection models commonly used in computer vision and computer graphics include: the Lambert model, the Phong model [59], the Blinn-Phong model [8], the Tor- rance-Sparrow model [85], the Cook-Torrance model [11], the Beckmann-Spizzino model [5], the He model [20], the Strauss model [80], and the Ward model [87].
The estimation of parameters of a reflection model function has been investigated by other researchers. In some cases, reflectance parameters are estimated only from multiple intensity images. In other cases, both range and intensity images are used to obtain object surface shapes and reflectance parameters. However, all of the previously proposed methods 1.1 Goniochromatic Space Analysis of Reflection 5 for reflectance parameter estimation are limited in one way or another. For instance, some methods can handle only objects with uniform reflectance, and in other methods, estimation of reflectance parameters becomes very sensitive to image noise. To our knowledge, no method is currently being used for estimating reflectance properties of real objects in real applications.
One of the main reasons why modeling of reflectance properties has been unsuccessful, as compared with modeling of object shapes, is that both diffusely reflected lights and spec- ularly reflected lights, i.e., the diffuse and specular reflection components, are examined simultaneously, and therefore, estimation of reflectance properties becomes unreliable. For instance, estimation of the diffuse reflection parameters may be affected by specularly reflected lights observed in input images. Also, estimation of the specular reflection compo- nent’s parameters often becomes unreliable when specularly reflected lights are not observed strongly in the input images.
In this thesis, we tackle the problem of object modeling by using a new approach to analyze a sequence of color images. The new approach allows us to estimate shape and reflectance parameters of real objects in a robust manner even when both the diffuse and specular reflection components are observed.
1.1 Goniochromatic Space Analysis of Reflection
We propose a new framework to analyze object shape and surface properties from a sequence of color images. This framework plays an important role in reflectance analysis described in this thesis. We observe how a color of an object surface varies on change in angular illuminating-viewing conditions using a four dimensional “RGB plus illuminating/ viewing angle” space. We call this space the goniochromatic space based on Standard Ter- minology of Appearance1 from American Society for Testing and Materials [78].
The goniochromatic space is closely related to the two spaces previously used for ana- lyzing color or gray-scale images: the Red-Green-Blue (RGB) color space and the I – θ (image intensity - illumination/viewing direction) space. Typically, the RGB color space is used for analyzing color information from a single color image. One of the epoch- making works using the RGB color space was done by Shafer [72][73]. He demonstrated that, illuminated by a single light source, a cluster of uniformly colored dielectric objects in the RGB color space forms a parallelogram defined by two color vectors, namely the diffuse
1. goniochromatism: change in any or all attributes of color of a specimen on change in angular illuminating-viewing conditions but without change in light source or observer. 6 Chapter 1 reflection vector and the specular reflection vector [72]. Subsequently, Klinker et al. [39] demonstrated that the cluster actually forms a T-shape in the color space instead of a paral- lelogram; they separated the diffuse and specular reflection components by geometrically clustering a scatter plot of the image in the RGB color space. However, their method requires that objects be uniformly colored and that surface shapes are not planar. For exam- ple, if an object has a multiple-colored or highly textured surface, a cluster in the RGB color space becomes cluttered, and therefore, separation of the two reflection components becomes impossible. If the object’s surface is planar, the cluster eventually concentrates to a point in the RGB color space, and again, the separation becomes impossible. As a result, the method can be applied to only a limited class of objects.
On the other hand, the I – θ space was used for analyzing a gray-scale image sequence. This space represents how the pixel intensity changes as illumination or viewing geometry changes. By using the space, Nayar et al. analyzed a gray-scale image sequence given by a moving light source [49]. Their method can separate the diffuse and specular reflection components from an observed intensity change in the I – θ space, and can estimate shapes and reflectance parameters of objects with hybrid surfaces2. The main advantage of their method is that all necessary information is obtained from a single point on the object sur- face, and therefore, the method can be applied locally. This is advantageous compared to the algorithm developed by Klinker et al. [39] because the algorithm by Klinker et al. examines a color cluster formed from an entire image globally. However, the Nayar et al. method is still limited in the sense that only a small group of real objects have hybrid surfaces, and the method requires an imaging apparatus of specific dimension, e.g., the light diffuser’s diam- eter and the distance from a light source to the light diffuser.
2. In the paper [49] by Nayar et al. a hybrid surface is defined as one which exhibits the dif- fuse lobe reflection component and the specular spike reflection components. (Those two reflection components and the specular lobe reflection component will be described in more detail in Chapter 3.) 1.2 Object Modeling from Color Image Sequence 7
image sequence under moving light
Figure 4 Goniochromatic space
Unlike the RGB color space and the I – θ space, the GSA does not require strong assumptions such as uniform reflectance, non-planar surfaces, hybrid surfaces, or an imag- ing apparatus of specific dimension. By using the GSA, we can separate the two reflection components locally from a color image sequence and obtain the shape and reflectance parameters of objects.
1.2 Object Modeling from Color Image Sequence
Based on GSA, we have developed a new method for estimating object shapes and reflectance parameters from a sequence of color images taken by a moving point light source. First, the diffuse and specular reflection components are separated from the color image sequence. The separation process does not have to assume any specific reflection model. Then, using the separated reflection components, object shapes and parameters of a specific reflection model are estimated.
Like the I – θ space analysis, our method requires only local information. In other words, we can recover the object shape and reflectance parameters based on color change at each point on the object surface; the method does not depend on observed color at other por- tions of the object surface. In addition, our method is not restricted to a specific reflection model, i.e., a hybrid surface, or to a specific imaging apparatus. Thus, our method can be applied to a wide range of objects. 8 Chapter 1
Figure 5 Reflection component separation
Currently, our method can handle only the case where object surface normals exist in a 2D plane. This is a rather severe limitation. However, the limitation comes from the fact that only coplanar motion of a light source is used, and it is not an inherent limitation of our method. For instance, if only two light source locations are used for photometric stereo, two surface normals are obtained at each surface point. The ambiguity can be solved by adding one more light source location which is not coplanar to the other two locations. The same thing can be done for our method, but it has not been tested in research conducted for this thesis.
The main limitation of this method is that an object shape cannot be recovered accu- rately if the object has a surface with high curvature. This is generally true for all methods which use only intensity images, e.g., shape-from-shading and photometric stereo. Another limitation of the method is that surface shape and reflectance parameters can be recovered only for a partial portion of the surface. Obviously, we cannot see the back of an object unless we rotate the object or change our eye location.
1.3 Object Modeling from Range and Color Image Sequences
To overcome the limitations noted in the previous section, we investigated another method. The goal was creating object models with complete shapes even if the objects have surfaces with high curvature. To attain this goal, we developed a method for creating com- plete object models from sequences of range and color images. These images are taken by changing object posture.
One advantage of using range image for shape recovery is that object shapes can be 1.3 Object Modeling from Range and Color Image Sequences 9 obtained as triangular mesh models which represent 3D information of the object shapes. In contrast, the method we described in Section 1.2 can produce only surface normals which is 2.5D information of the object surface. Hence, the object surface can be obtained as a trian- gular mesh model only after some sort of integration procedure is applied to the surface nor- mals. In general, this integration process does not work well for object surfaces with high curvature. And, a single range image cannot capture an entire object shape: it measures only the partial shape of the object seen from a range sensor. Therefore, we need to observe the object from various viewing points to see the object surface entirely. Then, we have to somehow merge those multiple range images to create a complete shape model of the object. In this thesis, two different algorithms are used for integrating multiple range images: a surface-based method and a volume-based method.
When we apply the GSA to a sequence of range and color images, we have to face the correspondence problem between color image frames. As we have already mentioned, the GSA examines a sequence of observed colors as illuminating/viewing geometry changes. Therefore, we need to know where the point on the object surface appears in each input color image. This correspondence problem was not an issue in the case of a color image sequence taken with a moving light source. In that case, the object location and the viewing point were fixed. Therefore, the point on the object surface appeared at the same pixel coor- dinate through the color image sequence.
Fortunately, we can solve the correspondence problem by using the reconstructed trian- gular mesh model of the object shape. Having determined object locations and camera parameters from calibration, we project each color image frame back onto the reconstructed object surface. By projecting all of the color image frames, we can determine the observed color change at each point on the object surface as the object is rotated. Then, we apply the GSA to the observed color change to separate the diffuse and specular reflection compo- nents.
After the two reflection components are separated, we estimate reflectance parameters of each reflection component. Here, we consider two different classes of objects. The first class comprises objects which are painted in multiple colors and do not have detailed sur- face textures. In this case, an object surface can be segmented into multiple regions with uniform diffuse color. The second class comprises objects with highly textured surfaces; in this case, object surfaces cannot be clearly segmented.
In this thesis, we investigated two different approaches for these two classes of objects. For the first class of objects without detailed surface texture, we developed a method to esti- mate reflectance parameters based on region segmentation of an object surface. Each seg- mented region on the object surface is assigned the same reflectance parameters for the specular reflection component, by assuming that each region with uniform diffuse color has 10 Chapter 1 more or less the same specular reflectance. Then, each triangle of the triangular mesh object model is assigned its diffuse parameters and specular parameters.
Figure 6 Synthesized image of an object without texture
For the second class of objects with highly textured surfaces, region segmentation can- not be performed reliably. Therefore, we developed another method using a slightly differ- ent approach. Instead of assigning one set of reflectance parameters to each triangle of a triangular mesh object model, each triangle is assigned a texture of reflectance parameters and surface normals. This method is similar to the conventional texture mapping technique. However, unlike that technique, our method can be used to synthesize realistic color images with realistic shading such as highlights. Finally, highly realistic object images are synthe- sized using the created object models with shape and reflectance properties.
Figure 7 Synthesized images of an object with texture 1.4 Reflectance Analysis under Solar Illumination 11 1.4 Reflectance Analysis under Solar Illumination
Most algorithms for analyzing object shape and reflectance properties, including our methods described above, have been applied to images taken in a laboratory.
Images synthesized or taken in a laboratory are well controlled and are less complex than those taken outside under sunlight. For instance, in an outdoor environment, there are multiple light sources of different colors and spatial distributions, namely the sunlight and the skylight. The sunlight can be regarded as a point light source whose movement is restricted to the ecliptic. On the other hand, the skylight acts as a blue extended light source. Those multiple light sources create more than two reflection components from the object surface unlike the case of one known light source in a laboratory setup.
Also, due to the sun’s restricted movement, the problem of surface normal recovery becomes underconstrained under the sunlight. For instance, if the photometric stereo method is applied to two intensity images taken outside at different times, two surface nor- mals which are symmetric with respect to the ecliptic are obtained at each surface point. Those two surface normals cannot be distinguished locally because those two surface nor- mal directions give us exactly the same brightness at the surface point.
In this thesis, we address the issues involved in analyzing real outdoor intensity images taken under solar illumination: the multiple reflection components including highlights, and the ambiguous solution for surface normals. For the difficulties associated with reflectance analysis under solar illumination, we propose a solution and then demonstrate the feasibility of the solution by using test images which are taken in a laboratory setup and outdoors under the sun.
Figure 8 Image taken under solar illumination 12 Chapter 1 1.5 Thesis Outline
This thesis presents new approaches for modeling object surface reflectance properties, as well as shapes, by observing real objects in both indoor and outdoor environments. The methods are based on a novel algorithm called goniochromatic space analysis for separating the diffuse and specular reflection components from a color image sequence.
This thesis is organized as follows. In Chapter 2, we introduce the goniochromatic space and explain the similarities and differences between the goniochromatic space and the two other spaces commonly used for reflectance analysis: the RGB color space and the I – θ space. In Chapter 3, we discuss our method for modeling object shapes and reflectance parameters from a color image sequence. In Chapter 4 and Chapter 5, we describe two dif- ferent methods for modeling object shape and reflectance parameters from a sequence of range and color images. In Chapter 6, we describe our attempt to analyze shape and reflec- tance properties of an object by using a color image sequence taken under solar illumina- tion. Finally, in Chapter 7, we summarize the work presented in this thesis, give conclusions and directions of future research. 13
Chapter 2
Goniochromatic Space Analysis of Reflection
2.1 Background
Color spaces, especially the RGB color space, have been widely used by the computer vision community to analyze color images. One of the first applications of color space anal- ysis was image segmentation by partitioning a color histogram into Gaussian clusters (Haralick and Kelly [18]). A histogram is created by the color values at all image pixels; it tells, for each point in the RGB color space, how many pixels exhibit the color. Typically, the colors tend to form clusters in the histogram, one for each textureless object in the image. By manual or automatic analysis of the histogram, the shape of each cluster is deter- mined. Then each pixel in the color image is assigned to the cluster that is closest to the pixel color in the RGB color space. Following the work by Haralick and Kelly, a number of image segmentation techniques have been developed [1], [9], [10], [61].
Most of the early works in color computer vision used color information as a random variable to be used for image segmentation. Later, many researchers tried using knowledge about how color is created to analyze a color image and compute some important properties of objects in the color image.
Shafer [73] carefully examined the physical properties of reflection when light strikes an inhomogeneous surface which includes materials such as plastics, paints, ceramics, and 14 Chapter 2 paper. An inhomogeneous surface consists of a medium with particles of colorant sus- pended in it. When light hits such a surface, there is a change in the index of refraction at the interface. This reflection occurs in the perfect specular direction where angle of incidence equals angle of reflection, and forms the specular reflection component, i.e., the highlights seen on shiny materials. The light that penetrates through the interface is scattered and selectively absorbed by the colorant. Then, the light is re-emitted to the air to become the diffuse reflection component.
Based on this observation, Shafer proposed the first realistic color reflection model used in computer vision: the dichromatic reflection model. That reflection model states that the reflectance of an object may be divided into two components: the interface or specular reflection component, and the body or diffuse reflection component. In addition, Shafer demonstrated that, illuminated by a single light source, a cluster of uniformly colored dielectric objects in the color space forms a parallelogram defined by two color vectors, namely the specular reflection vector and the diffuse reflection vector.
The dichromatic reflection model proposed by Shafer has inspired a large amount of important related work. Klinker, Shafer and Kanade [39][40] demonstrated that, instead of a parallelogram, the cluster actually forms a T-shape in the color space, and they separated the diffuse reflection component and the specular reflection component by geometrically clus- tering a scatter plot of the image in the RGB color space. They used the separated diffuse reflection component for segmentation of a color image without suffering from disturbances of highlights in the image. Their method is based on the assumption that the directions of surface normals in an image are widely distributed in all directions. This assumption guar- antees that both the diffuse reflection vector and the specular reflection vector will be visi- ble. Therefore, their algorithm cannot handle cases where only a few planar surface patches exist in the image.
Bajcsy and Lee [2] proposed using the hue-saturation-intensity (HSI) color space instead of the RGB color space for analyzing a color image. They studied clusters in the HSI color space formed by scene events such as shading, highlights, shadows, and interreflec- tion. Based on this analysis, their algorithm uses a hue histogram technique to segment indi- vidual surfaces and then follows with a local thresholding to identify highlights and interreflection. This technique was the first to identify interreflection successfully from a single color image. The algorithm is shown to be effective on color images of glossy objects.
Novak and Shafer [56] presented an algorithm for analyzing color histograms. The algorithm yields estimates of surface roughness, the phase angle between the camera and the light source, and the illumination intensity. In their paper, they showed that these properties cannot be computed analytically, and they developed a method for estimating these proper- 2.1 Background 15 ties based on interpolation between histograms that come from images of known scene properties. The method was tested by using both simulated and real images, and success- fully estimated those properties from a single color image.
Lee and Bajcsy [44] presented an interesting algorithm for the detection of specularities from Lambertian reflections using multiple color images from different viewing directions. The algorithm is based on the observation that reflected light intensity from the diffuse reflection component at an object surface does not change depending on viewing directions; however reflected light intensity from the specular reflection component or from a mixture of the diffuse and specular reflection components does change. This algorithm differs from other algorithms described above in that multiple color images taken from different viewing directions are used to differentiate color histograms of the color images. In this aspect, this algorithm is the one most closely related to our color analysis framework proposed in this thesis. However, their method still suffers from the fact that a color histogram of a color image is analyzed. When input color images contain many objects with non-uniform reflec- tance, color histograms of those images become too cluttered to be used for detecting the specular reflection component. Also, Lee and Bajcsy’s method cannot compute reflectance parameters of objects, and it is not clear how the method can be extended for reflectance parameter estimation.
All of the algorithms for color image analysis described in this section examine histo- grams formed either in the RGB color space or in some other color space. This means that the method depends on global information extracted from color images. In other words, those algorithms require color histograms which are not so cluttered and can be segmented clearly. If color images contain many objects with non-uniform reflectance, then color histo- grams become impossible to segment; therefore, the algorithms will fail.
Another limitation of those algorithms is that there is little or no consideration of illu- minating/viewing geometry. In other words, those algorithms, with the exception of the one by Lee and Bajcsy, do not examine how observed color changes as the illuminating/viewing geometry changes. This makes it very difficult to extract any information about object sur- face reflectance properties. (Strictly speaking, this is not true in the work by Novak and Sha- fer [56] where surface roughness is estimated from a color histogram. However, their algorithm does not work well for cluttered color images.)
On the other hand, other techniques have been developed for analyzing gray-scale images. Those techniques include shape-from-shading and photometric stereo. The shape- from-shading technique introduced by Horn [32] recovers object shapes from a single inten- sity image. In this method, surface orientations are calculated starting from a chosen point whose orientation is known a priori, by using the characteristic strip expansion method. Ikeuchi and Horn [33] developed a shape-from-shading technique which uses occluding 16 Chapter 2 boundaries of an object to iteratively calculate surface orientation.
In general, shape-from-shading techniques require rather strong assumptions about object surface shape and reflectance properties, e.g., smoothness constraint and uniform reflectance. The limitation comes from the fact that only one intensity image is used for shape-from-shading techniques, and therefore, it is a fundamentally under-constrained prob- lem.
Photometric stereo was introduced by Woodham [95] as a technique for recovering sur- face orientation from multiple gray-scale images taken with different light source locations. Surface normals are determined from the combination of constraints provided by reflectance maps with respect to different incident directions of a point light source. Unlike shape-from- shading techniques, Woodham’s technique does not rely on assumptions such as the surface smoothness constraint. However, the technique is still based on the assumption of the Lam- bertian surface. Hence, the technique can be applied to object surfaces only with the diffuse reflection component.
While specularities have usually been considered to cause errors in surface normal esti- mation by photometric stereo methods, some researchers have proposed the opposite idea of using the specular reflection component as a primary source of information for shape recov- ery. Ikeuchi was the first to develop a photometric stereo technique that can handle purely specular reflecting surfaces [34].
Nayar, Ikeuchi and Kanade [49] developed a photometric stereo technique for recover- ing object shape with surfaces exhibiting both the diffuse and specular reflection compo- nents, i.e., hybrid surfaces. These reflection components can vary in relative strength, from purely Lambertian to purely specular. The technique determines 2D surface orientation and the relative albedo strength of the diffuse and specular reflection components. The key is to use extended rather than point light sources so that a non-zero specular component is detected from more than just one light source. In fact, the extended nature of the light sources and their spacing are made so that for a hybrid surface, a non-zero specular compo- nent results from two consecutive light sources, with the rest of the observed reflections being only from the diffuse reflection component. Later, this technique was extended further to be able to compute 3D surface orientations by Sato, Nayar, and Ikeuchi [63].
Lu and Little developed a photometric stereo technique to estimate a reflectance func- tion from a sequence of gray-scale images taken by rotating a smooth object, and the object shape was successfully recovered using the estimated reflectance function [47]. Since the reflectance function is measured directly from the input image sequence, the method does not assume a particular reflection model such as the Lambertian model which is commonly used in computer vision. However, their algorithm can be applied to object surfaces with 2.2 The RGB Color Space 17 uniform reflectance properties, and it cannot be easily extended to overcome this limitation.
These photometric stereo techniques determine surface normals and reflectance param- eters by examining how reflected light intensity at a surface point changes as light source direction varies. This intensity change can be represented in the I – θ (image intensity - illu- mination direction) space.
The main difference between the I – θ space and the RGB color space is that the former can represent intensity change caused by illumination/viewing geometry change, while the latter cannot. This ability is a significant advantage when we want to measure various prop- erties of object surfaces such as surface normals and reflectance properties. Also, the I – θ space uses intensity change observed at each surface point. Therefore, necessary informa- tion can be obtained locally, while the RGB color space uses a color histogram which is formed globally from a entire color image.
However, the I – θ space fails to represent important information for reflectance analy- sis, i.e., color. Therefore, it is desirable to have a new framework which can represent both observed color information and change caused by different illumination/viewing geometry.
In this thesis, we propose a new framework to analyze object shape and surface proper- ties from a sequence of color images. We observe how the color of the image varies on change in angular illuminating-viewing conditions using a four dimensional “RGB plus illu- minating/viewing angle” space. We call this space the goniochromatic space based on Stan- dard Terminology of Appearance4 from the American Society for Testing and Materials [78].
This chapter first briefly describes the conventional RGB color space and the I – θ space in Section 2.2 and Section 2.3. Then, in Section 2.4, we introduce the goniochromatic space in comparison to those two other spaces.
2.2 The RGB Color Space
A pixel intensity Iis determined by the spectral distribution of incident light to the camera h()λ and the camera response to the various wavelengths s()λ , i.e.,
4. goniochromatism: change in any or all attributes of color of a specimen on change in angular illuminating-viewing conditions but without change in light source or observer. 18 Chapter 2
Is= ∫ ()λ h()λ dλ (EQ1)
A color camera has color filters attached in front of its sensor device. Each color filter has a transmittance function τλwhich() determines the fraction of light transmitted at each λ , wavelength . Then, pixel intensities IR IG , and IfromB red, green, and blue channels of the color camera are given by the following integrations:
τ ()λ ()λ ()λ λ IR = ∫ R s h d τ ()λ ()λ ()λ λ IG = ∫ G s h d (EQ2) τ ()λ ()λ ()λ λ IB = ∫ B s h d τ ()τλ ,,()τλ ()λ where R G B are the transmittance functions of the red, green, and blue fil- × ters, respectively. The three intensities IR , IG and IformB a 31 color vector Cwhich rep- resents the color of a pixel in the RGB color space.
τ ()λ ()λ ()λ λ ∫ R s h d IR C ==τ ()λ ()λ ()λ λ (EQ3) IG ∫ G s h d I B τ ()λ ()λ ()λ λ ∫ B s h d Klinker, Shafer, and Kanade [39][40] demonstrated that the histogram of dielectric object color in the RGB color space forms a T-shape (Figure 9). They extracted the two components of the T-shape in order to separate the specular reflection component and the diffuse reflection component.
200
BLUE
100 200 GREEN 100
00 0 100 200 RED 2.3 The I-q (Intensity - Illuminating/Viewing Angle) Space 19
Figure 9 A sphere and its color histogram as T shape in the RGB color space (synthesized data)
A significant limitation of the method is that it works only when surface normals in the image are well distributed in all directions. Suppose that the image has only one planar object illuminated by a light source which is located far away from the object. Then, all pix- els on the object are mapped to a single point in the color space because observed color is constant over the object surface. The T-shape converges, in the RGB color space, to a single point which represents the color of the object, because the plane has uniform color. As a result, we cannot separate the reflection components. This indicates that the method cannot be applied locally.
2.3 The I-θ (Intensity - Illuminating/Viewing Angle) Space
Nayar, Ikeuchi, and Kanade [49] analyzed an image sequence given by a moving light source in the I – θ space. They considered how the pixel intensity changes as the light source direction θvaries (Figure 10).
The pixel intensity from a monochrome camera is written as a function of θ :
I()θ = g()θ ∫s()λ h()λ dλ (EQ4) where g()θ represents intensity change with respect to a light source direction θ . Note that the spectral distribution of incident light to the camera h()λ is generally dependent on geo- metric relations such as the viewing direction and the illumination direction. However, as an approximation, we assume that the function his()λ independent of these factors.
The vector p()θ, I()θ shows how pixel intensity changes with respect to light source direction in Ispace– θ (Figure 11).
As opposed to analysis in the RGB color space, the I – θ space analysis is applied locally. All necessary information is extracted from the intensity change at each individual pixel. Nayar et al. [49] used the I – θ space to separate the surface reflection component and the diffuse reflection component, using a priori knowledge of the geometry of the “photo- metric sampler.” 20 Chapter 2
camera
light source n
θ n θ
object
Figure 10 Viewer-centered coordinate system These three vectors are coplanar.
I()θ
p()θ, I()θ
θ
Figure 11 The I – θ space
2.4 The Goniochromatic Space
Without resorting to relatively strong assumptions, neither the RGB color space nor the I – θ space can be used to separate the two reflection components using local pixel informa- tion. To overcome this weakness, we propose a new four dimensional space, which we call 2.4 The Goniochromatic Space 21 the goniochromatic space. This four dimensional space is spanned by the R, G, B, and θ axes. The term “goniochromatic space” implies an augmentation of the RGB color space with an additional dimension that represents varying illumination/viewing geometry. This dimension represents the geometric relationship between the viewing direction, illumination direction, and surface normal. In our method that we describe more fully in the next chapter, we keep the viewing direction and surface normal orientation fixed. Then, we vary the illu- mination direction, taking a new image at each new illumination direction. (The same infor- mation could be obtained if we kept the viewing direction and illumination direction fixed, and varied the surface normal orientation. This case will be described in Chapter 4 and Chapter 5.)
The goniochromatic space can be thought of as a union of the RGB color space and the I – θ space. By omitting the θ axis, the goniochromatic space becomes equivalent to the RGB color space; and by omitting two color axes, the goniochromatic space becomes the I – θ space. Each point in the space is represented by the light source direction θand the color vector C()θ which is a function of θ :
p()θ, C()θ (EQ5)
()τθ ()λ ()λ ()λ λ ()θ g ∫ R s h d IR C()θ ==()θ ()τθ ()λ ()λ ()λ λ (EQ6) IG g ∫ G s h d I ()θ B ()τθ ()λ ()λ ()λ λ g ∫ B s h d The goniochromatic pace represents how the observed color of a pixel C()θ changes as the direction of the light source θ changes (Figure 12). Note that, in Figure 12, the dimen- sion of the goniochromatic space is reduced from four to three for clarity. In this diagram, one axis of the RGB color space is ignored. 22 Chapter 2
200
BLUE
100 200 GREEN 100
00 -50 0 50 100 THETA [deg]
Figure 12 The goniochromatic space (synthesized data) 23
Chapter 3
Object Modeling from Color Image Sequence
In the previous chapter, the goniochromatic space was introduced as a method for ana- lyzing a sequence of color images. In this chapter, we introduce a method for estimating object surface shapes and reflectance properties from a sequence of color images taken by changing the illuminating direction. The method consists of two steps. First, by using the GSA introduced in the previous chapter, the diffuse and specular reflection components are separated from the color image sequence. Then, surface normals and reflectance parameters are estimated based on the separation results. The method was successfully applied to real images of objects made of different materials.
Objects that we consider in this chapter are made of dielectric or metal material. Also, the method can be applied to objects whose surface normals exist in a 2D plane defined by a light source direction and a viewing direction. Note that this is not a limitation inherent to the proposed method; rather, it is due to limited coplanar motion of the light source as we will see later in this chapter.
This chapter is organized as follows. First, a parametric reflectance model used in our analysis is described in Section 3.1. Then, the decomposition of the diffuse and specular reflection components from a color image sequence is explained in Section 3.2. The decom- position method requires the specular reflection color and the diffuse reflection color. A method for estimating these colors is explained in Section 3.3 and Section 3.4, respectively. The results of experiments conducted using objects of different materials are presented in Section 3.5. The summary of this chapter is given in Section 3.6. 24 Chapter 3 3.1 Reflection Model
A mechanism of reflection is described in terms of three reflection components, namely the diffuse lobe, the specular lobe, and the specular spike [50]. Reflected light energy from object surface is a combination of these three components.
The diffuse lobe component may be explained as internal scattering. When an incident light ray penetrates the object surface, it is reflected and refracted repeatedly at a boundary between small particles and medium of the object. The scattered light ray eventually reaches the object surface, and is refracted into the air in various directions. This phenomenon results in the diffuse lobe component. The Lambertian model is based on the assumption that those directions are evenly distributed in all directions.
On the other hand, the specular spike and lobe are explained as light reflected at an interface between the air and the surface medium. The specular lobe component spreads around the specular direction, while the specular spike component is zero in all directions except for a very narrow range around the specular direction. The relative strengths of the two components depends on the microscopic roughness of the surface.
specular spike
camera specular lobe
light source
diffuse lobe
reflecting surface
Figure 13 Polar plot of the three reflection components (redrawn from [50])
Unlike the diffuse lobe and the specular lobe components, the specular spike compo- 3.1 Reflection Model 25 nent is not commonly observed in many actual applications. The component can be observed only from mirror-like smooth surfaces where reflected light rays of the specular spike component are concentrated in a specular direction. That makes it hard to observe the specular spike component from viewing directions at coarse sampling angles. Therefore, in many computer vision and computer graphics applications, a reflection mechanism is mod- eled as a linear combination of two reflection components: the diffuse lobe component and the specular lobe component.
Those two reflection components are normally called the diffuse reflection component and the specular reflection component. The reflection model was formally introduced by Shafer [73] as the dichromatic reflection model. Based on the dichromatic reflection model, the reflection model used in our analysis is represented as a linear combination of the diffuse reflection component and the specular reflection component.
camera specular lobe
light source
diffuse lobe
reflecting surface
Figure 14 Reflection model used in our analysis
The Torrance-Sparrow model is relatively simple and has been shown to conform with experimental data [85]. In our analysis, we use the Torrance-Sparrow model [85] for repre- senting the diffuse reflection component and the specular reflection component. As we will see in Section 3.1.2, this model describes reflection of incident light rays on rough surfaces, i.e., the specular lobe component, and captures important phenomena such as the off-specu- lar effect and spectral change within highlights. 26 Chapter 3 3.1.1 The Lambertian Model
The Torrance-Sparrow model uses the Lambertian model for representing the diffuse reflection component. The Lambertian model has been used extensively for many computer vision techniques such as shape-from-shading and photometric stereo. The Lambertian model is the first model proposed to approximate the diffuse reflection component.
The mechanism of the diffuse reflection is explained as the internal scattering. When an incident light ray penetrates the object surface, it is reflected and refracted repeatedly at a boundary between small particles and medium of the object (Figure 15). The scattered light ray eventually reaches the object surface, and is refracted into the air in various directions. This phenomenon results in body reflection. The Lambertian model is based on the assump- tion that the directions of the refracted lights are evenly distributed in all directions.
incident light specular component diffuse component
pigments
Figure 15 Internal scattering and surface reflection