Mesostructure from Specularity Tongbo Chen Michael Goesele Hans-Peter Seidel MPI Informatik University of Washington MPI Informatik Abstract We describe a simple and robust method for surface mesostructure acquisition. Our method builds on the ob- servation that specular reflection is a reliable visual cue for surface mesostructure perception. In contrast to most photometric stereo methods, which take specularities as outliers and discard them, we propose a progressive ac- (a) (b) (c) (d) quisition system that captures a dense specularity field as the only information for mesostructure reconstruction. Our method can efficiently recover surfaces with fine-scale geo- metric details from complex real-world objects with a wide variety of reflection properties, including translucent, low albedo, and highly specular objects. We show results for a variety of objects including human skin, dried apricot, or- ange, jelly candy, black leather and dark chocolate. (e) (f) (g) (h) 1 Introduction Figure 1. Mesostructure reconstruction of or- ange skin. (a-d) Four cropped input images. The visual appearance of a real-world object is governed (e) Recovered normal field (RGB-encoded). by reflectance properties, illumination condition, and a hi- (f) Filtered normal field [35]. (g) Rendering of erarchy of geometric components. In the geometric hier- the normal field using Ward's isotropic BRDF archy, there are basically three different levels of scales, model [14]. (h) Reconstructed 3D surface namely, the macrostructure level, the mesostructure level, rendered at a novel viewpoint. and the microstructure level. The macrostructure level rep- resents the gross surface geometry, typically expressed as are, however, rarely able to capture the fine-scale details of a polygonal mesh or parametric spline surface. The mi- real-world objects with translucency or highly specular re- crostructure level involves surface microfacets that are vi- flection, such as skin, rough fruit skin, etc. sually indistinguishable. The mesostructure level represents By drawing inspiration from photographs of real-world geometric details that are relatively small but still individ- translucent objects and from the literature on human vision ually visible such as bumps or dents on a surface. Effi- and perception [6, 24, 34, 7], we found that specular high- cient mesostructure reconstruction methods can contribute lights are an important visual cue for surface mesostructure greatly to high-quality graphics models in terms of fine- perception and a reliable visual information for surface de- scale surface geometric details. An accurate and explicit tail representation. In Figure 1, the first row shows four im- mesostructure model can also benefit related mesostructure ages of a piece of orange skin under changing illumination. modeling techniques such as BTFs (Bidirectional Texture The small bumps on the orange skin introduce rich visual ef- Functions) [4, 20]. fects and can be efficiently revealed by specular highlights State of the art high-resolution 3D scanning methods (see Figures 5, 6, 8, 7 for more examples). Based on this include [22, 5, 1, 15]. Photometric stereo methods can observation, we developed a simple and progressive sys- achieve high-resolution surface reconstruction with inex- tem that uses specular highlights in order to solve the dense pensive setup [40, 27, 10, 9, 26, 41]. Existing techniques mesostructure reconstruction problem for a variety of real- world complex objects, which possess a significant specular fine-scale details for very low albedo, translucent, or highly- reflection component. Our method is largely independent specular surfaces [22, 5]. To deal with highly-specular sur- of the underlying reflectance model, and can therefore suc- face, Wang and Dana [37]presented a method that can si- cessfully handle objects with complex reflectance that have multaneously capture fine-scale surface shape and spatially previously been challenging. varying BTFs by using a BTF measurement system. Sim- To summarize our contributions: (1) We simplify the ilar to that work, our method will also depend on specu- problem of mesostructure reconstruction from complex ob- lar reflection. But we extend the idea to include not only jects, e.g., objects with translucency, or high specularity, highly-specular surface, but also very low albedo glossy or which has up to now been expensive or even impossible to translucent glossy materials. Instead of using a complicated solve. (2) We use a dense specularity field as the only re- BTF measurement system, we developed a simple, flexi- liable visual information for mesostructure reconstruction. ble and progressive acquisition system. In [42], Yu and (3) We developed a simple incremental and very flexible ac- Chang introduced shadow graphs for 3D texture reconstruc- quisition system. (4) We acquired high-quality mesostruc- tion. They show that the shadow graph alone is sufficient to ture, for a variety of real-world objects including human solve the shape-from-shadow problem from a dense set of skin, dried apricot, orange skin, jelly candy, black leather, images. They also solved the problem of recovering height and dark chocolate. fields from a sparse set of images by integrating shadow and shading constraints. However, this method cannot work ef- 1.1 Overview fectively for objects where shadow is no longer an accurate information, such as skin or fruit. The remainder of this paper is organized as follows. Photometric stereo methods [40, 27] are known to be Related work on surface mesostructure reconstruction and able to capture fine-scale surface details and to provide an shape-from-specularity is discussed in Section 2. Section efficient alternative to BTF-based methods. In [10], Hertz- 3 describes our acquisition system and the mesostructure mann and Seitz presented an example-based photometric reconstruction method in detail. Extensive experimental stereo for shape reconstruction with general spatially vary- results on mesostructure reconstruction for complex real- ing BRDFs. They assumed that there are no cast shadows, world objects are shown in Section 4. We conclude and no interreflections, and no subsurface scattering. Goldman discuss possible extensions in Section 5. et al. [9] proposed a photometric stereo method for itera- tively recovering shape and BRDFs. They employed a lo- 2 Related Work cal reflectance model, which cannot properly account for shadows, interreflections and subsurface scattering. In [26], Surface mesostructure is one of the key components Paterson et al. developed a simple system for BRDF and ge- of 3D texture [12]. It contributes strongly to the com- ometry capturing. Their system can handle a variety of real- plex surface appearance of real-world objects. One method world objects except highly specular or translucent materi- for modeling and rendering mesostructure is through BTFs als. Wu and Tang [41] presented a simple dense photometric (Bidirectional Texture Functions) [4], which can be re- stereo method, using only a mirror sphere, a spotlight and a garded as a mapping from the 4D space of lighting and DV camera. They achieved surprisingly good results even viewing directions to the space of 2D images. Most previ- with the presence of moderate shadows and specular high- ous work on BTFs aims at capturing appearance data from lights. To our knowledge, photometric stereo methods can natural materials and at efficient representation. Muller¨ et rarely recover dense fine-scale surface details from translu- al. [20] gives a comprehensive report on the state of the cent, highly specular, or low albedo glossy materials. art of BTFs techniques. Liu et al. [16] used a shape-from- In [18], Magda and Zickler take advantage of Helmholtz shading method to recover approximate 3D geometry of reciprocity and light fields to reconstruct surfaces with ar- surface details from a BTF dataset. In [23], Neubeck et bitrary BRDFs. That method makes no assumption of the al. proposed a method for 3D texture reconstruction from surface BRDF and works effectively for a variety of non- extensive BTF data, with only a few and rather weak as- Lambertian surfaces (e.g. glossy surface), but not for highly sumptions about reflectance and geometry. The recon- translucent objects, where subsurface scattering dominates. structed mesostructure can be used for the simplification In our approach, there is no explicit reflectance model as- of the BTF-based texture description and efficient compres- sumed. We only exploit the specular reflection, which is di- sion of a BTF dataset. Even for the most advanced and ex- rectly related to surface geometry. Shape-from-specularity pensive laser scanning systems, mesostructure reconstruc- is a well-investigated field in computer vision. In contrast tion of highly specular or translucent objects is still a dif- to most of the photometric stereo methods, where specu- ficult problem. Most of the scanning technologies based lar highlights are detected and separated as outliers, shape- on structured lighting based will also fail in reconstructing from-specularity methods try to efficiently use the specular reflectance component [25, 28, 29, 30]. Zheng and Mu- rata [43] developed a special system to acquire gross shape from specular motion by using circular-shaped light source. Kutulakos and Steger [13] proposed an effective method for 3D shape reconstruction of refractive and specular ob- jects by light-path triangulation. In [32] Solem et al. intro- duced variational analysis into shape-from-specularity and demonstrated the robustness