International Journal of Computer Vision manuscript No. (will be inserted by the editor) A Practical Approach to 3D Scanning in the Presence of Interreflections, Subsurface Scattering and Defocus Mohit Gupta · Amit Agrawal · Ashok Veeraraghavan · Srinivasa G. Narasimhan Received: date / Accepted: date Abstract Global or indirect illumination effects such of capture time or hardware. We show results for several as interreflections and subsurface scattering severely scenes with complex shape and material properties. degrade the performance of structured light-based 3D scanning. In this paper, we analyze the errors in struc- tured light, caused by both long-range (interreflections) Keywords Structured light 3D scanning, interreflec- and short-range (subsurface scattering) indirect illumi- tions, subsurface scattering, defocus, global illumina- nation. The errors depend on the frequency of the pro- tion, indirect illumination, light transport, projectors. jected patterns, and the nature of indirect illumination. In particular, we show that long-range effects cause de- coding errors for low-frequency patterns, whereas short- range effects affect high-frequency patterns. 1 Introduction Based on this analysis, we present a practical 3D scanning system which works in the presence of a broad Structured light triangulation has become the method range of indirect illumination. First, we design binary of choice for shape measurement in several applica- structured light patterns that are resilient to individual tions including industrial automation, graphics, human- indirect illumination effects using simple logical opera- computer interaction and surgery. Since the early work tions and tools from combinatorial mathematics. Scenes in the field about 40 years ago [37,25,33], research has exhibiting multiple phenomena are handled by combin- been driven by two factors: reducing the acquisition ing results from a small ensemble of such patterns. This time and increasing the depth resolution. Significant combination also allows detecting any residual errors progress has been made on both fronts (see the survey that are corrected by acquiring a few additional images. by Salvi et al. [34]) as demonstrated by systems which Our methods can be readily incorporated into existing can recover shapes at close to 1000 Hz. [41] and at a scanning systems without significant overhead in terms depth resolution better than 30 microns [11]. Despite these advances, the applicability of most A preliminary version of this paper appeared in [16]. structured light techniques remains limited to well be- Mohit Gupta is with the Computer Science Department, haved scenes. It is assumed that scene points receive Columbia University, New York, NY, 10027. illumination only directly from the light source. For E-mail: [email protected] Amit Agrawal is with the Mitsubishi Electrical Research Labs, many real world scenarios, this is not true. Imagine a Cambridge, MA, 02139. robot trying to navigate an underground cave or an E-mail: [email protected] indoor scenario, a surgical instrument inside human Ashok Veeraraghavan is with the Electrical and Computer Engi- body, a robotic arm sorting a heap of metallic machine neering Department, Rice University, Houston, TX, 77005. E-mail: [email protected] parts, or a movie director wanting to image the face Srinivasa G. Narasimhan is with the Robotics Institute, Carnegie of an actor. In all these settings, scene points receive Mellon University, Pittsburgh, PA, 15213. illumination indirectly in the form of interreflections, E-mail: [email protected] subsurface or volumetric scattering. Such effects, col- 2 Translucent Marble Slab Strong Interreflections Blurring due to sub-surface scattering Concave Bowl (a) Concave bowl on a (b) Input image under (c) Input image under translucent marble slab low-frequency illumination high-frequency illumination Errors due to Errors due to interreflections subsurface scattering (d) Shape recovered using (e) Shape recovered using (f) Shape recovered using Conventional Gray codes (11 images) Modulated phase shifting [8] (162 images) our ensemble codes (42 images) Fig. 1 Measuring shape for the ‘bowl on marble-slab’ scene. This scene is challenging because of strong interreflections inside the concave bowl and subsurface scattering on the translucent marble slab. (b) Scene points inside the bowl which are not directly illuminated receive substantial irradiance due to interreflections. (d) This results in systematic errors in the recovered depth. (c) Due to subsurface scattering on the translucent marble slab, high-frequency illumination patterns are severely blurred. Notice the low contrast of the stripes as compared to the bowl. (e) This results in depth-errors on the marble-slab. (f) Our technique uses an ensemble of codes optimized for individual indirect illumination effects, and results in an accurate shape reconstruction. Parentheses contain the number of input images. More results and comparisons with existing techniques are at the project web-page [1]. lectively termed global or indirect illumination1, often range of indirect illumination effects. The focus is on de- dominate the direct illumination and strongly depend signing the projected patterns (coding) and decoding on the shape and material properties of the scene. Not schemes. In particular, we consider binary structured accounting for these effects results in large errors in the light patterns, which are perhaps the simplest to imple- recovered shape (see Figure 1b). Because of the system- ment and widely used in several research and commer- atic nature of these errors 2, it is hard to correct them cial systems. The key observation is that different in- in post-processing. direct illumination effects place contrasting constraints The goal of this paper is to build an end-to-end on the spatial frequencies of projected structured light system for structured light 3D scanning under a broad patterns. In particular, interreflections result in errors for low-frequency structured light patterns 3. On the 1 Global illumination should not be confused with the oft-used “ambient illumination” that is subtracted by capturing image 3 Strictly speaking, since all binary patterns have step edges, with the structured light source turned off. all of them have high spatial frequencies. For the analysis and 2 In photometric stereo, interreflections result in a shallow but discussion in this paper, low-frequency patterns implies patterns smooth reconstruction [29,28]. In structured light 3D scanning, with thick stripes. Similarly, high-frequency patterns mean pat- interreflections result in local errors. terns with only thin stripes. 3 other hand, local effects such as subsurface scattering techniques. Since the first papers [37,25,33], a lot of and defocus blur the high-frequency patterns, making progress has been made in terms of reconstruction speed, it hard to decode them reliably. accuracy and resolution. Broadly, these techniques are We design patterns that modulate indirect illumi- divided into discrete [22] and continuous [40] coding nation and prevent the errors at capture time itself. schemes. For an exhaustive survey on structured light We show that it is possible to construct codes with techniques, reader is referred to the survey by Salvi et only high-frequency binary patterns by introducing the al [34]. In addition, hybrid techniques that combine concept of logical coding and decoding. The key idea is structured light with photometric stereo based tech- to express low-frequency patterns as pixel-wise logical niques have been proposed as well [30,3]. combinations of two high-frequency patterns. Because of high frequencies, these patterns are resilient to long- Shape recovery in the presence of indirect illu- range effects. In order to deal with short-range effects, mination: The seminal work of Nayar et al. [28] pre- we use tools from combinatorial mathematics to design sented an iterative approach for reconstructing shape patterns consisting solely of low frequencies. In compar- of Lambertian objects in the presence of interreflec- ison, most currently used patterns (e.g., Gray codes) tions. Liu et al. [24] proposed a method to estimate contain a combination of both low and high spatial fre- the geometry of a Lambertian scene by using the second quencies, and thus are ill-equipped to deal with indirect bounce light transport matrix. Gupta et al. [19] pre- illumination. sented methods for recovering depths using projector Indirect illumination in most real world scenes is not defocus [39] under indirect illumination effects. Chan- limited to either short or long-range effects. Codes opti- draker et al. [6] use interreflections to resolve the bas- mized for long-range effects make errors in the presence relief ambiguity inherent in shape-from-shading tech- of short-range effects and vice versa. How do we han- niques. Holroyd et al. [21] proposed an active multi- dle scenes that exhibit more than one type of indirect view stereo technique where high-frequency illumina- illumination effect (such as the one in Figure 1(a))? To tion is used as scene texture that is invariant to indirect answer this, we observe that the probability of two dif- illumination. Park et al. [32,31] move the camera or the ferent codes producing the same erroneous decoding is scene to mitigate the errors due to indirect illumination very low. This observation allows us to project a small in a structured light setup. Hermans et al. [20] use a ensemble of codes and use a simple voting scheme to moving