Structured Light In Sunlight Mohit Gupta Qi Yin Shree K. Nayar Columbia University Columbia University Columbia University New York, NY 10027 New York, NY 10027 New York, NY 10027 [email protected] [email protected] [email protected] Abstract Strong ambient illumination severely degrades the per- formance of structured light based techniques. This is espe- cially true in outdoor scenarios, where the structured light sources have to compete with sunlight, whose power is often 2-5 orders of magnitude larger than the projected light. In this paper, we propose the concept of light-concentration to overcome strong ambient illumination. Our key observation is that given a fixed light (power) budget, it is always better to allocate it sequentially in several portions of the scene, (a) An object placed outdoors (b) Imageof the sky as compared to spreading it over the entire scene at once. 6 am 9 am 12 pm For a desired level of accuracy, we show that by distributing light appropriately, the proposed approach requires 1-2 or- ders lower acquisition time than existing approaches. Our approach is illumination-adaptive as the optimal light dis- tribution is determined based on a measurement of the am- bient illumination level. Since current light sources have (c-e) 3D reconstructions at different times of the day a fixed light distribution, we have built a prototype light source that supports flexible light distribution by control- Figure 1. Effect of ambient illumination on structured light 3D ling the scanning speed of a laser scanner. We show several scanning. (a) An object placed outdoors on a clear day receives strong ambient illuminance R from the sun and the sky. (b) Im- high quality 3D scanning results in a wide range of outdoor a age of the sky at 9am. (c-e) 3D reconstructions using conventional scenarios. The proposed approach will benefit 3D vision methods at different times of the day. From left to right, as the day systems that need to operate outdoors under extreme ambi- progresses, Ra increases (2000 lux, 24,000 lux and 90,000 lux, ent illumination levels on a limited time and power budget. respectively) and the reconstruction quality degrades. 1. Introduction to increase the power of the light source. Unfortunately, Structured light 3D scanning, because of its accuracyand this is not always possible. Especially in outdoor scenar- simplicity, is the method of choice for 3D reconstruction ios, vision systems often operate on a limited power budget. in several applications, including factory automation, per- Moreover, low-cost hand-held projectors (e.g., pico projec- formance capture, digitization of cultural heritage and au- tors) are increasingly becoming popular as structured light tonomous vehicles. In many real-world settings, structured sources. For these low-power devices to be useful outdoors, light sources have to compete with strong ambient illumi- it is important to be able to handle strong ambient illumina- nation. In these scenarios, because of the limited dynamic tion on a tight power budget. range of image sensors, the signal (intensity due to struc- In this paper, we introduce the concept of light concen- tured light) in the captured images can be extremely low, tration in order to deal with strong ambient illumination. resulting in poor 3D reconstructions. The key idea is that even with a small light budget, signal This is especially true outdoors, where sunlight is often level can be increased by concentrating the available pro- 2-5 orders of magnitude brighter than the projected struc- jector light on a small portion of the scene. This is illus- tured light. For instance, it is known that Kinect, a popular trated in Figure 2 (center). At first glance, it may appear that structured light device, cannot recover 3D shape in strong concentrating the light will require more measurements, as sunlight [1]. While several optical techniques for ambient only a fraction of the scene is illuminated and encoded at light reduction have been proposed [10], they achieve only a time. However, we show that, it is possible to achieve moderate success. An example using an off-the-shelf laser significantly lower acquisition times by concentrating light 3D scanner is shown in Figure 1. Under strong ambient il- as compared to existing approaches that spread the avail- lumination, the reconstruction quality of an object placed able light over the entire projector image plane, and then outdoors degrades, even when spectral filtering is used. reduce image noise by frame-averaging. We call this the One obvious solution to the ambient light problem is light-concentration advantage. 1 Decreasing Light Spread large spread medium spread narrow spread light light light Projector Projector source source source 1st block th block Projected Images Projected (Left) Stripe Intensity = (Center) Stripe IntensityܥΤ ܭ = (Right) Stripe Intensity = Figure 2. Light redistribution for structured light. We consider different light distributions for designing structured light systems that perform under strong ambient illumination.ͳΤ ܥ Given a fixed light budget, as the lightܭ spreadΤ ܥ decreases (from left to right), the intensityͳ of each projected stripe increases. Existing structured light techniques lie at the two extremes of the power distribution scale. (Left) Systems where light is distributed over the entire projector image plane yield low signal strength and hence poor reconstruction quality. (Right) Systems where all the light is concentrated in to a single column require a large number of measurements. (Center) We show that by concentrating the light appropriately, it is possible to achieve fast and high-quality 3D scanning even in strong ambient illumination. The light-concentration advantage arises from the fact the scanner. The proposed approach achieves fast and high- that frame-averaging increases signal-to-noise-ratio (SNR) quality (pixel-level) 3D reconstruction for even the most ex- bya factorof square-root of the number of averaged frames. treme scenarios (direct sunlight, low-powered light source). However, the same time and power budget, if allocated into These features make our approach especially suitable for smaller scene regions, increases SNR linearly with the num- moving platforms such as autonomous cars which need to ber of measurements. We show that for the same accuracy operate outdoors under varying ambient illuminance levels (SNR) level, while the number of measurements required on a limited power budget. by existing approaches is linear in the ambient illuminance level Ra, i.e., (Ra), the proposed approach requires only O 2. Related Work √Ra measurements. For outdoor ambient illuminance O levels, this translates into 1-2 orders of magnitude (10-100 Structured light 3D scanning: Structured light techniques times) lower acquisition time. are classified based on the coded patterns that they project on the scene. Some typical examples are single line Scope and contributions: This paper introduces light re- stripes [3], sinusoidal patterns [13], binary patterns [11] and distribution as a new dimension in the design of structured deBruijn codes [15]. For a comprehensive survey on exist- light systems. We do not introduce a new structured light ing coding schemes, the reader is referred to [12]. coding scheme. Instead, we show that by managing the light budget appropriately, it is possible to perform fast and Significant work has been done towards designing high accurate 3D scanning outdoors on a limited power budget. SNR structured light coding schemes [14, 4, 6]. It has been After determining the optimal light distribution based on shown that in scenarios with extremely low SNR (such as the ambient illuminance level, any one of the existing high- strong ambient illumination), optimal SNR is achieved by SNR structured light coding scheme [14, 4, 6] can be used. using patterns with the fewest possible intensity levels (bi- The proposed approach can adapt to the ambient light level. nary patterns with two intensity levels) [6]. In Figure 1, For instance, as ambient illuminance decreases, the acquisi- despite binary Gray code patterns being used, result quality tion time required by our approach decreases. The proposed degrades as ambient illumination increases. This is because techniques are not restricted only to ambient illumination using high SNR patterns without considering light redistri- due to sunlight. They are applicable in any scenario that bution is not sufficient to achieve high-quality results under has a wide range of ambient illumination. strong ambient illumination. Hardware prototype and practical implications: Exist- Optical methods for suppressing ambient illumination: ing projectors distribute light over the entire image plane; Examples of such methods include using a narrow spec- they do not have the ability to distribute light in a flexible tral bandwidth laser (sunlight has broad bandwidth) with a manner. We have developed a prototype projector with flex- narrow-band spectral filter [10] and a polarized light source ible light distribution ability by using an off-the-shelf laser (sunlight is unpolarized) with a polarization filter [10]. scanner. Different light distributions over the projector im- This paper proposes a different approach. Given a fixed age plane are achieved by varying the scanning speed of level of ambient illuminance (after optical suppression), we determine the optimal distribution of the light (of the struc- strong ambient illumination, Ra >> Rl, and the dominant tured light source) in order to maximize the SNR. The in- source of noise is the signal-dependent photon noise, i.e., crease in SNR achieved by our method is in addition to, and 2 βRa σr << g . Then, the SNR is approximated as: much higher than, that achieved by the optical methods. In R SNR λ l , (4) order to deal with extreme ambient illumination scenarios, ≈ √R optical suppression techniques can be used in a complemen- a where λ is a constant. In order to achieve a desired depth tary manner to our method.