2017 IEEE International Conference on Robotics and Automation (ICRA) Singapore, May 29 - June 3, 2017 3D tracking of water hazards with polarized stereo cameras Chuong V. Nguyen1, Michael Milford2 and Robert Mahony1 Abstract— Current self-driving car systems operate well in sunny weather but struggle in adverse conditions. One of the most commonly encountered adverse conditions involves water on the road caused by rain, sleet, melting snow or flooding. While some advances have been made in using conventional RGB camera and LIDAR technology for detecting water hazards, other sources of information such as polarization offer a promising and potentially superior approach to this problem in terms of performance and cost. In this paper, we present a novel stereo-polarization system for detecting and tracking water hazards based on polarization and color variation of reflected light, with consideration of the effect of polarized light from sky as function of reflection and azimuth angles. To evaluate this system, we present a new large ‘water on road’ Fig. 1. Model of polarized sky light interaction with water surface for datasets spanning approximately 2 km of driving in various detection of water hazards. on-road and off-road conditions and demonstrate for the first time reliable water detection and tracking over a wide range of realistic car driving water conditions using polarized vision as ground plane depth fitting and then employed a threshold the primary sensing modality. Our system successfully detects technique on depth information to detect puddles. water hazards up to more than 100m. Finally, we discuss several Detection of water hazards from a single camera is also interesting challenges and propose future research directions possible using only color and texture information. Zhao et al for further improving robust autonomous car perception in hazardous wet conditions using polarization sensors. [8] developed SVM-based classification of color and texture information from a single image. Rankin and Matthies [9] I. INTRODUCTION proposed a technique based on variation in color from sky Polarization of light in outdoor environments is very reflections as a camera moves closer to a puddle. Later Rank common. Scattered light from the sky is polarized [1], et al [10] proposed another technique that searches for water especially close to the horizon. Light reflected from water reflection matching with the sky. Rankin’s techniques work surfaces [2] is also polarized, particularly when the angle well for wide open space, well defined water bodies and of reflection is low. Some vegetation and many man-made distance greater than 7m. objects, especially those with glass, polarize light [3], [4]. In this paper we use polarization effect as the primary Water on the road poses a significant hazard when driving cue to identify water in driving conditions. To improve on a vehicle, so the utility of polarized light for detecting it past research, we take a novel approach using the change has obvious appeal, especially in the age of self-driving in the color saturation and polarization as functions of cars where perception systems must match current human reflection and azimuth angles to account for effect from sky capability. polarization. The method is illustrated in Fig. 1. Several authors have considered using polarization as cue to identify water surfaces. Xie et al [5] used three cameras II. LIGHT POLARIZATION AND REFLECTION ON WATER with polarizers to estimate polarizing angle and detect water SURFACES surfaces. Stereo cameras equipped with horizontal and verti- In this section, we provide a treatment of the theoretical cal polarizers have been used to detect still and running water background behind the use of polarization to detect water bodies [6] or wet ground [7]. While Yan [6] used hand-tuned surfaces. We review the Rayleigh sky model of polarization parameters for classification, Kim et al [7] used Gaussian as described in [1] and a reflection model proposed by [9]. Mixture Model (GMM) for hypothesis generation and a To provide the theoretical background for the novel approach Support Vector Machine (SVM) for hypothesis verification. in this paper, we describe how incident light polarization, For puddle detection, Kim et al [7] used RANSAC to perform scattering and refraction can be used to model the color and the polarization of light received from water surfaces. 1Chuong Nguyen and Robert Mahony are with the Australian Centre for Robotic Vision, Research School of Engineering, A. Review of existing work Australian National University, Canberra ACT 2601, Australia [email protected] Ambient light is often treated as non-polarized; however 2Michael Milford is with Australian Centre for Robotic this is not true for light coming from sky. Illguth [1] shows Vision, School of Electrical Engineering and Computer Science, Queensland University of Technology, Brisbane QLD 4000, Australia that sky polarization variations during the day can affect the [email protected] appearance of building glass fac¸ades, which presumably also 978-1-5090-4632-4/17/$31.00 ©2017 Crown 5251 applies to water surfaces. Polarized light is generated by Rayleigh scattering effect. Following the Rayleigh-sky model [1], the degree or intensity of the polarized light component is a function of scattering angle g: h sin2 g h = max (1) 1 + cos2 g cosg = sinqsun sinqview cosy + cosqsun cosqview (2) where qview is viewing angle between a viewed point on sky Fig. 3. Light reflection and refraction in a water puddle with dirt particles and ground bottom. and zenith, qsun angle between the sun and zenith, and y azimuth angle between the sun and the viewed point on the sky. Rre f lect is a sum of two polarization components Rre f lect;? and Rre f lect;k, perpendicular and parallel respectively to the plane formed by the incident and reflected rays: q 2 2 2 2 3 n1 cosq − n2 1 − (n1=n2) sin q Rre f lect;?(n1;n2;q) = 4 q 5 2 2 n1 cosq + n2 1 − (n1=n2) sin q (4) q 2 2 2 2 3 n1 1 − (n1=n2) sin q − n2 cosq Rre f lect;k(n1;n2;q) = 4 q 5 2 2 n1 1 − (n1=n2) sin q + n2 cosq (5) where n1 and n2 are refractive indices of first and second media (here air and water, respectively) and q is reflection angle at the medium interface. Fig. 2. Degree or intensity of sky light polarization as function of viewing The higher the coefficient Rre f lect;?, the stronger the angle qview (between a viewed point on sky and zenith) and angle of the intensity and apparent color of the sky as it appears in the sun qsun (between the sun and zenith). reflection on water surface. The difference between the two coefficients Rre f lect;? and Rre f lect;k gives the polarization in Fig. 2 visualizes equations 1 and 2. The top of Fig. 2 shows the reflected light. However, Equations 4 and 5 alone do not that when observing the horizon (qview = 90), the degree of explain the influence of sky polarization on the polarization, polarization is maximal at azimuth angles of 90 and 270 the color and intensity of reflection on water surfaces in 0 degrees from the sun when it is near the horizon (qsun = 90 ), reality. and maximal across the entire horizon when the sun is at its 0 B. Effect of polarized sky light on polarization of reflection zenith (qsun = 0 ). The bottom of Fig. 2 shows that when the sun is at zenith (qsun = 0), the degree of polarization is To take into account the effect of polarization of sky light maximum at the ground horizon. In reality, the maximum on water reflection, we propose a model of light interaction degree of polarization can reach up to 90% of the total light with water. Fig. 3 illustrates the 4 components of equation 3. energy. The scattering reflection component due to surface roughness Although sky light is generally polarized, existing research at O1 is very small compared to specular reflection for often avoids dealing with this effect [9], [6], [7]. With an water, unless there is a large concentration of particles or assumption of unpolarized incident light, Rankin et al. [9] it is windy. In addition, the light component exiting the modeled total water reflection as a summation of specular water from particles Rparticles and ground bottom Rbottom reflection from the water surface Rre f , scattering reflection are in fact refractions, not reflections. As a result, the term of water surface Rscatter, scattering reflection of particles in ”water reflection” as sensed by eyes or cameras should be water Rparticles, and scattering reflection from the ground understood as a sum of reflection and refraction. bottom of the water Rbottom: Internal reflection coefficients within water at O2 and O3 are also given by equations 4 and 5 with n1 = nwater and Rtotal = Rre f lect + Rscatter + Rparticles + Rbottom (3) n2 = nair. Refraction coefficients (from air to water or from water to air) can be obtained from reflection coefficients: Specular reflection on water is known to polarizes light 0 [2], [5], [9], [7]. Specular reflection from the water surface Rre f ract;?(n1;n2;q ) = 1 − Rre f lect;?(n1;n2;q) (6) 5252 0 Rre f ract;k(n1;n2;q ) = 1 − Rre f lect;k(n1;n2;q) (7) where Snell’s law gives: n sinq 0 = 1 sinq (8) n2 Suppose that polarized light from the sky comprises en- S S ergy components E?(q;y) and Ek (q;y) for perpendicular and parallel components respectively as functions of reflec- tion angle and azimuth angle. Light that enters water surface illuminates particles and the ground bottom. The total energy entering the water is therefore: S S F = E?(q;y)[1 − Rre f lect;?(n1;n2;q)] S + Ek (q;y)[1 − Rre f lect;k(n1;n2;q)] (9) Fig.