Road Weather Condition Estimation Using Fixed and Mobile Based Cameras Koray Ozcan1(&), Anuj Sharma1(&), Skylar Knickerbocker2(&), Jennifer Merickel3(&), Neal Hawkins2(&), and Matthew Rizzo3(&) 1 Institute for Transportation, Iowa State University, Ames, IA, USA {koray6,anujs}@iastate.edu 2 Center for Transportation Research and Education, Ames, IA, USA {sknick,hawkins}@iastate.edu 3 University of Nebraska Medical Center, Omaha, NE, USA {jennifer.merickel,matthew.rizzo}@unmc.edu Abstract. Automated interpretation and understanding of the driving environment using image processing is a challenging task, as most current vision-based systems are not designed to work in dynamically-changing and naturalistic real-world settings. For instance, road weather condition classifica- tion using a camera is a challenge due to high variance in weather, road layout, and illumination conditions. Most transportation agencies, within the U.S., have deployed some cameras for operational awareness. Given that weather related crashes constitute 22% of all vehicle crashes and 16% of crash fatalities, this study proposes using these same cameras as a source for estimating roadway surface condition. The developed model is focused on three road surface con- ditions resulting from weather including: Clear (clear/dry), Rainy-Wet (rainy/slushy/wet), and Snow (snow-covered/partially snow-covered). The camera sources evaluated are both fixed Closed-circuit Television (CCTV) and mobile (snow plow dash-cam). The results are promising; with an achieved 98.57% and 77.32% road weather classification accuracy for CCTV and mobile cameras, respectively. Proposed classification method is suitable for autono- mous selection of snow plow routes and verification of extreme road conditions on roadways. Keywords: Road weather classification Scene classification VGG16 Neural networks CCTV Mobile cameraÁ Á Á Á Á 1 Introduction According to the Federal Highway Administration, there were 1.2 million weather related crashes from 2005 to 2014 [1]. As a result, 445,303 individuals were injured and 5,897 people lost their lives. Weather related crashes constitute 22% of all vehicle crashes and 16% of crash fatalities. It has also been shown that traffic flow rate, and © Springer Nature Switzerland AG 2020 K. Arai and S. Kapoor (Eds.): CVC 2019, AISC 943, pp. 192–204, 2020. https://doi.org/10.1007/978-3-030-17795-9_14 Road Weather Condition Estimation Using Fixed and Mobile Based Cameras 193 vehicle speeds are significantly reduced under inclement weather conditions [2]. Therefore, accurate and comprehensive weather condition monitoring has a key role in road safety, effective winter maintenance, and traveler information and advisories. Imbedded road sensors have been used for the purpose of automated traffic control via variable traffic message signs [3]. However, road sensors are highly dependent on thresholds to estimate the road surface conditions based on sensor measurements such as wetness, temperature, etc. Lasers and other electro-optical technologies have been used to measure road surface grip condition especially for snow and ice [4]. Surveil- lance and mobile cameras can also provide automated monitoring of road weather conditions. Recently, Minnesota Department of Transportation established a snowplow camera network to share frequently captured images with public [5]. This research paper considers providing road weather condition estimations from both snowplow and highway surveillance camera networks. 1.1 Mobile Sourced Images With the increasing popularity of mobile cameras for driver assistance and autonomous driving applications, there has been an increasing demand for using in-vehicle camera systems. Son and Baek [6] proposed a design and implementation taxonomy using real- time in-vehicle camera data for driver assistance and traffic congestion estimation. A camera and other sensors were used to estimate the road’s condition by capturing the vehicle’s response to the roadway. Rajamohan et al. [7] developed road condition and texture estimation algorithms for classes of smooth, rough, and bumpy roads. To do so, they combined in-vehicle camera, GPS, and accelerometer sensors. Kutila et al. [8] proposed an algorithm for road condition monitoring by combining laser scanners and stereo cameras. They reported 95% accuracy with the help of sensor fusion methods for road surface types: dry, wet, snow, and ice. Laser scanners were helpful to estimate depth of the road surface layer for various weather scenarios. Support vector machine (SVM) based classifiers have been used for classifying winter road surfaces for videos recorded with in-vehicle cameras. For three major roadway condition classes (i.e., bare, snow-covered, and snow-covered with tracks), they provided overall accuracy of 81% to 89%. For distinguishing dry and wet road conditions, Yamada et al. performed a multi-variate analysis of images captured by vehicle mounted cameras. Nguyen et al. [9] employed polarized stereo cameras for 3D tracking of water hazards on the road. Moreover, Abdic et al. [10] developed a deep learning based approach to detect road surface wetness from audio and noise. Kuehnle and Burghout [11] developed a neural network with three or four input features such as mean and standard deviation of color levels to classify dry, wet, and snowy conditions from video cameras. They achieved 40% to 50% correct classification accuracy with the network architecture. Pan et al. [12] estimated road conditions based on clear and various snow covered conditions using pre-trained deep convolutional neural network (CNN). They proved CNN based algorithms perform better than traditional, random tree, and random forest based models for estimating if the road is snow covered or not, while omitting rainy/wet 194 K. Ozcan et al. scenario in general. Finally, Qian et al. [13] tested various features and classifiers for road weather condition analysis on a challenging dataset that was collected from an uncalibrated dashboard camera. They achieved 80% accuracy for road images of classes: clear vs. snow/ice covered and 68% accuracy for classes: clear/dry, wet, and snow/ice- covered. Our dataset consists of classes from clear/dry, wet, and snow/ice- covered classes that are annotated manually for this project. 1.2 Fixed Source Images from Road Weather Information Stations (RWIS) Another approach to estimating road conditions involves using a camera mounted above the roadway and looking directly down on the road surface. Jonsson [14] used weather data and camera images to group and classify weather conditions. The study used grayscale images and weather sensor features to estimate road condition group- ings across dry, ice, snow, track (snowy with wheel tracks), and wet. They also provided a classification algorithm for the road condition classes dry, wet, snowy, and icy [15]. Similarly, using the road surface’s color and texture characteristics provided suitable classification with k-nearest neighbor, neural network and SVM for dry, mild snow coverage, and heavy snow coverage [16]. However, these approaches are limited to the locations where cameras are installed and with a confined field of view. In other words, the cameras are only looking down on the road from above while they are attached to poles near road verges. It would be too costly to monitor every segment of the road with such implementations. 1.3 Fixed Source Images from Closed-Circuit Television (CCTV) CCTV cameras provide surveillance of road intersections and highways. They can be utilized as observance sensors for estimating features such as traffic density and road weather conditions. Lee et al. [17] proposed a method to extract weather information from road regions of CCTV videos. They analyzed CCTV video edge patterns and colors to observe how they are correlated with weather conditions. However, rather than estimating road weather conditions, they developed algorithms that estimated overall weather conditions across the scene. They presented 85.7% accuracy with three weather classes: sunny, rainy, and cloudy. Moreover, snowfall detection algorithm, particularly in low visibility scenes, was developed in [18] using CCTV camera images. For modeling various road conditions, we selected VGGNets since it has been proven to be effective for object detection and recognition in stationary images [19]. Recently, Wang et al. implemented VGGNet models in large-scale Places365 datasets [20]. The model was trained with approximately 1.8 million images from 365 scene classes. The model achieved 55.19% for top-1 class accuracy and 85.01% top-5 class accuracy on test dataset. With the model developed by Zhou et al. [21], it has been Road Weather Condition Estimation Using Fixed and Mobile Based Cameras 195 shown that place recognition can be achieved with high accuracy. As it is explained in the next sections, features learned from this network is shown to be useful for differ- entiating various road image features for defined weather classes. This paper adapts place recognition models to fine-tune the last three layers of the network for road condition estimation. 1.4 Objective In this paper, we are proposing to estimate road surface weather conditions. To the authors’ best knowledge, it is the first application of road weather condition estimation using CCTV cameras observing road intersections and highways. Our experimental results show feasibility and utility on CCTV datasets that monitor the road surface. Further algorithm development is also presented to improve road weather