2020 International Conference on Computational Science and Computational Intelligence (CSCI)

Black ice detection using CNN for the Prevention of

Accidents in Automated Vehicle

HoJun Lee MinHee Kang JaeIn Song dept. Urban Design & Planning dept. Smartcity Research Institute of Science and Hongik University Hongik University Technology Seoul, Korea Seoul, Korea Hongik University [email protected] [email protected] Seoul, Korea [email protected] Keeyeon Hwang dept.Urban Design & Planning Hongik University Seoul, Korea [email protected]

Abstract— Black ice is recognized as the main cause of major of AVs, it is expected that technologies that can detect it in accidents in winter because it has characteristics that are advance and prevent accidents will be required. In this study, difficult to identify with the naked eye. This is expected to be a to prevent black ice accidents in AVs in the future, we potential cause of accidents in the era of automated vehicles as suggests method to detect black ice on the road using well. Accordingly, this study presents a CNN-based black ice Convolutional Neural Networks. detection plan to prevent traffic accidents caused by black ice. Due to the characteristic of black ice that is formed only in a This study is conducted in the following order: Chapter 2 certain environment, the data was augmented and the image of discusses the research on the use of CNN in the field of road environment in various environments was learned. Test transportation and derives the differentiation of this research, results show that the proposed CNN model detected black ice while Chapter 3 sets up the CNN model learning environment with 96% accuracy and reproducibility(recall). for the detection of black ice. Chapter 4 identifies and analyzes learning results through models, and Chapter 5 presents Keywords—Automated Vehicle, Blackice, CNN, Traffic implications and future studies with a brief summary. Accident, Prevention II. LIITERATURE REVIEW I. INTRODUCTION In this chapter, we will consider existing detection As discussions on the 4th industrial revolution become methods for black ice and studies using CNN in the more active, there is a movement to utilize big data, artificial transportation sector to derive the differentiation of this study. intelligence, and 5G. Among them, Automated Vehicles(AVs), a collection of various technologies, are A. Blackice Detection Methods attracting attention in the transportation field. AVs are Habib Tabatabai et al. (2017)[9] conducted a study to expected to bring effects such as improving mobility for the detect black ice, ice and water in roads and bridges using vulnerable and reducing traffic congestion costs, and are sensors embedded in concrete. In this study, a sensor that expected to minimize human and material losses in terms of detects the road surface condition was proposed through the preventing traffic accidents caused by driver negligence [1-2]. change of electrical resistance between stainless steel columns Currently, various companies such as Google, NVIDIA, and inside concrete. As a result of conducting experiments under Tesla are developing and experimenting with AV systems, various surface conditions, it was suggested that the proposed and each country is reorganizing its institutional foundation to sensor can effectively detect the road condition, thereby prepare for the commercialization of AVs. Despite these preventing various accidents. efforts, however, traffic accidents continue to occur in Youngis E. Abdalla et al (2017) (2017) [10] proposed a autonomous driving situations, and the social acceptability of system for detecting black ice using Kinect. The types of ice AVs has emerged due to Uber's pedestrian deaths in 2018 [3- (Soft Ice, Wet Snow, Hard Ice, Black Ice) were classified and 5]. In order to solve these problems fundamentally, Germany the thickness and volume of the ice were measured using and the United States have issued an ethics guideline for AVs Kinect. Experiments have shown that the types of ice formed [6-7]. The guidelines specify the need to develop principles to in the range of 0.82m to 1.52m from the camera can be cope with dilemma situations, along with information on the distinguished, and the error rate of measured thickness and preventive design of the AVs to avoid accidents. Preventive volume is very low, suggesting that black ice can be detected design of AVs is an issue about risk management that can by utilizing Kinect. occur in a realistic driving environment, changing from passive safety systems to active safety systems research [8]. Xinxu Ma et al. (2020)[11] studied a black ice detection While various preventive design studies are being carried out, method using a 3-wavelength non-contact optical technology. there is a lack of research on preventing black ice accidents, The study conducted an experiment to distinguish dry, wet, which are the main causes of large-scale traffic accidents in black ice, ice and snowy conditions using 3 wavelengths winter. Black ice is a thin ice film formed on the road by (1310nm, 1430nm, 1550nm). As a result of the experiment, it combining rain and snow with pollutants such as dust, which was confirmed that black ice was detected through the is likely to lead to fatal accidents because it is difficult to reflectance of each wavelength, and it was suggested that it identify the naked eye. As Black Ice is considered to be a can be used as basic data for the development of equipment to potential accident factor even in the era of commercialization detect road conditions.

978-1-7281-7624-6/20/$31.00 ©2020 IEEE 1189 DOI 10.1109/CSCI51800.2020.00222 B. The Utilization of CNN in Transportation C. Summary Lele Xie et al (2018) (2018) [12] proposed an ALMD- In summary, it was confirmed that various studies using YOLO structure to conduct various angles of vehicle license black ice detection research and CNN in the transportation plate detection studies utilizing CNN-based Multi-Directive field are in progress. The black ice detection study proposed a YOLO (MD-YOLO). Comparing performance with various method to detect black ice using a sensor, a detection system, models (ALMD-YOLO, Faster R-CNN, SSD, MD-YOLO, and optical technology. In addition, in the CNN-related study etc.) was found to have the best performance of the proposed in the transportation field, a study was conducted to detect the ALMD-YOLO. This suggested that the simple structure of the most important objects constituting the road environment such model would shorten the computational time, thereby creating as pedestrians, vehicles, traffic signs, and road cracks. a high-performance, multi-way license plate detection model. In spite of such as studies, it is expected that there is a limit Hongbo Gao et al (2018) [13] conducted a study using to preventing black ice accidents in advance due to problems LIDAR and CNN to detect objects in autonomous driving such as the installation of a black ice detection system. situations. The above study collected data with LIDARs Accordingly, this study proposes a method to detect black ice mounted on self-driving cars and used AlexNet to detect five by identifying road conditions based on CNN technique to classes of objects (Pedestrian, Cyclist, Car, Truck and Other). prevent black ice accidents in autonomous vehicles. Experiments have shown near 100% accuracy in most classes except Truck (88.6%). The small number of classes was III. LERANING ENVIRONMENT SETTING pointed out as a limitation, but it was confirmed that CNN- A. Data Collecting & Pre-Process based object detection was possible using LIDAR only. Data used for learning were classified into a total of four Yung Yao Chen et al (2019) [14] conducted a pedestrian (blackice, road, wet road, snow road) and obtained through detection study using the heat detection system and the Faster Google image search. After that, it was processed into a size R-CNN. For data collection, thermal imaging cameras were of 128×128 px so that the characteristics of the road could be used instead of ordinary optical cameras, and analysis was clearly identified. Next, we converted to GRAYSCALE (1 made using the ResNet structure. Experiments have shown channel) for image data learning and proceeded with data better performance than conventional CNN and confirmed padding. Results of processing and learning by data padding that night pedestrian detection is also possible using thermal are shown in Table 1, and image distortion of original data is imaging cameras and Faster R-CNN. confirmed during processing. In addition, experimental Alexander Shustanov, & Yakimov, P. (2017)[15] verification has shown that the loss value (loss) of padding conducted a CNN model design study for real-time traffic sign data is lower than the original data, and a high accuracy has recognition. The above study used modified Generalized been calculated and all data has been padded. Hough Transform (GHT) and CNN, with 99.94% accuracy. It was also confirmed that the proposed algorithm could process TABLE I. RESULT OF AUGMENTATION AND LEARNING high-definition images in real time and accurately recognize Original Data Padding Data traffic signs farther away than similar traffic sign recognition systems. Result of Augmentation Young-Jin Cha and two others (2017) [16] conducted a concrete crack detection study based on CNN. Data was acquired in various situations, and the CNN model was Learning Loss 1.39 0.26 constructed by placing Convolution layer and Max-pooling Result Accuracy 0.253 0.891 layer alternately. 98.22% of the accuracy was obtained as a Through this process, 25,400 images of 150×150 px-sized result of learning, and based on this, cracks in test data were image data were obtained in the form of GRAYSCALE. detected using sliding window technique. The above study has implications for designing models that show high TABLE II. A NUMBER OF DATA performance despite the data acquired in a variety of situations. Class Size Number Zheng Tong et al (2018) [17] conducted a study to classify Blackice 3,900 the length of asphalt cracks using Deep Convolutional Neural Road 4,900 Networks (DCNN). Data collection was carried out in various 150 Ő 150 px places and weather, and was divided into eight classes in 1cm Wet road 4,900 units from 0cm to 8cm. The results of the experiment showed Snow road 3,900 that 94.36% accuracy and 1cm maximum length error were Total 25,400 derived, suggesting that the length of the crack, not just the In the course of this study, there was a limit to collecting presence of cracks, could be classified. black ice image data, so a data set was built through data augmentation to derive high accuracy. Based on the data Shengyuan Li et al (2019) [18] conducted an image-based obtained through the previous process, 1000 sheets for each concrete crack detection study using CNN. We selected class were randomly extracted and constructed as test data. AlexNet to build a CNN model and detected cracks in test data After that, data was augmented using the ImageDataGenerator through the Exhaustive Search method. The experiment function provided by Keras library for the rest data. Through results showed that 99.09% accuracy was derived and it was data augmentation, 10,000 pieces of data per class were confirmed that a high accuracy CNN model could be built by constructed and classified into training data and validation securing variety of crack types and external conditions. data at an 8:2 ratio. The resulting overall final data set is shown in Table 3 and Figure 1 below.

1190 TABLE III. DATA SET COMPOSITION BY CLASS

Validation Test Class Train data Total data data

Blackice Road 8,000 2,000 1,000 11,000 Wet road Snow road

Fig. 3. Result of Learning

To analyze the learning results, performance indicators for each class were checked for train data and test data. As a result of train data learning, it was found that the accuracy of black ice was relatively low as shown in Table 5.

TABLE V. DATA SET COMPOSITION BY CLASS

Class Accuracy

Fig. 1. Building of Dataset Blackice 0.961 Road 0.996 B. Design and Learning Process Wet road 0.981 The structure of the CNN model used in this study is Snow road 0.989 composed of Feature Extraction and Classification, as shown To examine this in detail, Confusion matrix confirmed the in Figure 2. In the feature extraction area, a convolutional classification results by class. The Confusion matrix analysis, layer, a max-pooling layer, and a dropout layer were arranged, consisting of the learning results (x axis) for four classes and and ReLU was used as the activation function. In the class the actual class (y axis), showed that some confusion occurred classification area, the Fully-connected layer and the dropout between the blackice and snow road data and confused the wet layer are alternately arranged, and Softmax is applied to the road image with the road. This is judged to be the result of the output layer (see Figure 2). loss of light characteristics due to the transformation of For model training, 200 epoch, 32 batch size was applied, GRAYSCALE. and SGD (Stochastic Gradient Descent) Optimizer was used, and early stopping and a dropout rate of 0.2 were set to prevent overfitting.



Fig. 2. The CNN Model stucture : In the early stages of learning, two Convolutional layers, two Max-pooling layers, and one Dropout layer are stacked to conduct two iterations. Afterward, the Convolutional layer, max- pooling layer, and dropout layer are placed to conduct two iterations.

IV. RESULT  The analysis result shows that the loss of Train Data and Test Data was identified as 0.008 and 0.097, as shown in Table Fig. 4. Confusion Matrix : For comparing predicted and actual classes to 4 and Figure 3, and the accuracy was identified as 0.998 and measure Prediction performance over Training 0.982, respectively. Next, the test data was compared and analyzed with accuracy, precision, and recall of each class. As shown in TABLE IV. RESULT OF LEARNING Table 6, the accuracy of black ice, wet road, and snowy road Class Loss Accuracy was measured relatively low, and this is considered to be a

Train 0.008 0.998 result of loss of light characteristics in the same way as the Test 0.097 0.982 previously analyzed confusion matrix. However, the average values of accuracy, precision, and recall were 0.982, 0.983, and 0.983, indicating that significant learning results were derived.

1191 TABLE VI. ACCURACY BY CLASS [11] Ma, X., & Ruan, C. (2020). “Method for black ice detection on roads using tri-wavelength backscattering measurements,” Applied Optics, Class Accuracy Precision Recall 59(24), 7242-7246. Blackice 0.961 0.980 0.960 [12] Xie, L., Ahmad, T., Jin, L., Liu, Y., & Zhang, S. (2018). “A new CNN- Road 0.996 0.990 1.000 based method for multi-directional car license plate detection,” IEEE Wet road 0.981 0.970 0.980 Transactions on Intelligent Transportation Systems, 19(2), 507-517. Snow road 0.989 0.990 0.990 [13] Gao, H., Cheng, B., Wang, J., Li, K., Zhao, J., & Li, D. (2018). “Object classification using CNN-based fusion of vision and LIDAR in Average 0.982 0.983 0.983 autonomous vehicle environment,” IEEE Transactions on Industrial Informatics, 14(9), 4224-4231. V. CONCLUSION [14] Chen, Y. Y., Jhong, S. Y., Li, G. Y., & Chen, P. H. (2019, December). This study conducted a study using CNN to detect black “Thermal-based pedestrian detection using faster r-cnn and region decomposition branch,” In 2019 International Symposium on ice that is difficult to judge visually to prevent black ice Intelligent Signal Processing and Communication Systems (ISPACS) accidents in AVs. Data was collected by classifying into four (pp. 1-2). IEEE. classes, and each class Train, Validation, and Test Data were [15] Shustanov, A., & Yakimov, P. (2017). “CNN design for real-time established through pre-processing of crop, padding, and traffic sign recognition,” Procedia engineering, 201, 718-725. enhancement. As a result of learning the model, the accuracy [16] Cha, Y. J., Choi, W., & Büyüköztürk, O. (2017). 專Deep learningʘ of black ice was derived relatively lower than that of other based crack damage detection using convolutional neural networks,尉 classes, but about 97%, showing excellent performance. ComputerʘAided Civil and Infrastructure Engineering, 32(5), 361- 378. In this study, the neural network was designed and studied [17] Li, S., & Zhao, X. (2019). “Image-Based concrete crack detection using through GRAYSCALE with the characteristics of black ice convolutional neural network and exhaustive search technique,” formed mainly at dawn. However, it has been shown that some Advances in Civil Engineering, 2019. specific classes are confused because of the loss of light [18] Tong, Z., Gao, J., Han, Z., & Wang, Z. (2018). “Recognition of asphalt pavement crack length using deep convolutional neural networks,” characteristics. Accordingly, it is deemed necessary to Road Materials and Pavement Design, 19(6), 1334-1349. perform black ice detection by utilizing RGB images in the future. This study is meaningful in that black ice, which is FUNDING considered a potential risk factor even in the era of AVs, was This research was supported by Basic Science Research detected using artificial intelligence, not sensors and Program through the National Research Foundation of wavelengths. It is expected that this will prevent black ice accidents of AVs and will be used as basic data for future Korea(NRF) funded by the Ministry of Education(No. relevant research. 2020R1F1A106988411) and 2019 Hongik University Research Fund. REFERENCE [1] Lee, K.; Jeon, S.; Kim, H.; Kum, D. “Optimal path tracking control of autonomous vehicle: Adaptive full-state linear quadratic gaussian (lqg) control,” IEEE Access, 2019, 7, 109120-109133. [2] Singh, S. “Critical reasons for crashes investigated in the national motor vehicle crash causation survey,” (No. DOT HS 812 115), 2015. [3] “Federal ministry of transport and digital infrastructure,” Ethics commission : automated and connected driving, 2017. [4] “National Highway Traffic Safety Administration. Federal Automated Vehicles Policy : Accelerating the Next Revolution In Roadway Safety,” 2016 [5] Abraham, H.; Lee, C.; Brady, S.; Fitzgerald, C.; Mehler, B.; Reimer, B.; Coughlin, J. F. “Autonomous vehicles and alternatives to driving: trust, preferences, and effects of age,” In Proceedings of the transportation research board 96th annual meeting, 2017. [6] Zhang, T.; Tao, D.; Qu, X.; Zhang, X.; Lin, R.; Zhang, W. “The roles of initial trust and perceived risk in public’s acceptance of automated vehicles,” Transportation research part C: emerging technologies 2019, 98, 207-220. [7] Hartwich, F.; Witzlack, C.; Beggiato, M.; Krems, J. F. “The first impression counts–A combined driving simulator and test track study on the development of trust and acceptance of highly automated driving,” Transportation research part F: traffic psychology and behavior 2019, 65, 522-535. [8] Kim, K., Kim, B., Lee, K., Ko, B., & Yi, K. (2017). “Design of integrated risk management-based dynamic driving control of automated vehicles,” IEEE Intelligent Transportation Systems Magazine, 9(1), 57-73. [9] Tabatabai, H., & Aljuboori, M. (2017). “A novel concrete-based sensor for detection of ice and water on roads and bridges. Sensors,” 17(12), 2912. [10] Abdalla, Y. E., Iqbal, M. T., & Shehata, M. (2017, April). “Black Ice detection system using Kinect,” In 2017 IEEE 30th Canadian Conference on Electrical and Computer Engineering (CCECE) (pp. 1- 4). IEEE.

1192