data Review Deep Learning in Data-Driven Pavement Image Analysis and Automated Distress Detection: A Review Kasthurirangan Gopalakrishnan ID Department of Civil, Construction, and Environmental Engineering, Iowa State University, Ames, IA 50011, USA; [email protected] Received: 15 June 2018; Accepted: 18 July 2018; Published: 24 July 2018 Abstract: Deep learning, more specifically deep convolutional neural networks, is fast becoming a popular choice for computer vision-based automated pavement distress detection. While pavement image analysis has been extensively researched over the past three decades or so, recent ground-breaking achievements of deep learning algorithms in the areas of machine translation, speech recognition, and computer vision has sparked interest in the application of deep learning to automated detection of distresses in pavement images. This paper provides a narrative review of recently published studies in this field, highlighting the current achievements and challenges. A comparison of the deep learning software frameworks, network architecture, hyper-parameters employed by each study, and crack detection performance is provided, which is expected to provide a good foundation for driving further research on this important topic in the context of smart pavement or asset management systems. The review concludes with potential avenues for future research; especially in the application of deep learning to not only detect, but also characterize the type, extent, and severity of distresses from 2D and 3D pavement images. Keywords: pavement cracking; pavement management; pavement imaging; 3D image; deep learning; TensorFlow; deep convolutional neural networks 1. Introduction Transportation infrastructure systems are essential to the minimum operations of the government and commerce, and are considered the backbone of a nation’s economy. Yet, they are literally crumbling across the globe and are considered “to be on life support” even by authorities in charge of maintaining them, especially in the United States. According to the 2017 American Society of Civil Engineers (ASCE) Infrastructure Report Card, the road infrastructure in the United States received a ‘D’ grade [1]. This state is due, in large part, to delayed maintenance and underinvestment in upgrades of transportation infrastructure systems. This, combined with increasing budgetary constraints, necessitates the development of efficient structural and functional health monitoring techniques for early detection of distresses developing in pavements. This can lead to significant cost savings resulting from timely maintenance and repair activities. Highway agencies typically employ dedicated pavement data collection vehicles equipped with high-speed digital cameras or 3D laser scanner for inspecting the pavement surface, and acquire 2D or 3D pavement images [2,3]. In recent years, the 3D automated survey systems have also been introduced to acquire high-resolution 3D images of the pavement surface, which also offers opportunities for detecting other distresses apart from cracking [4,5]. Automated detection of distresses from pavement images (see Figure1) is a challenging problem that has been quite thoroughly studied by the computer vision research community for more than three Data 2018, 3, 28; doi:10.3390/data3030028 www.mdpi.com/journal/data Data 2018, 3, 28 2 of 19 Data 2018, 3, x 2 of 19 decades. However, the challenges associated with 2D pavement images, such as variations in image three decades. However, the challenges associated with 2D pavement images, such as variations in source (digital camera, smartphone, unmanned aerial vehicle (UAV), etc.), non-uniformity of cracks, image source (digital camera, smartphone, unmanned aerial vehicle (UAV), etc.), non-uniformity of surfacecracks, texture surface (e.g., texture tining), (e.g., lack tining), of sufficient lack of suffic backgroundient background illumination, illumination, and presence and presence of other of other features suchfeatures as joints, such among as joints, others, among continue others, to continue keep this to keep area ofthis research area of research active with active researchers with researchers constantly seekingconstantly newer seeking methods newer and methods algorithms and algorithms to address to address these challenges. these challenges. Not Not surprisingly, surprisingly, the the recent achievementsrecent achievements by so-called by deepso-called learning deep (DL)learning algorithms (DL) algorithms have caught have caught the attention the attention of the of pavement the imagepavement analysis communityimage analysis and community have inspired and themhave toin movespired fromthem systems to move that from use systems handcrafted that use features to data-drivenhandcrafted distress features detection to data-driven systems distress (i.e., systems detection that systems automatically (i.e., systems learn that features automatically from the learn images). Becausefeatures of the from availability the images). of inexpensive,Because of the parallelavailability hardware, of inexpensive, and massive parallelamounts hardware, of and unlabeled massive data, deepamounts learning of has unlabeled already data, produced deep learning breakthrough has already results produced in computer breakthrough vision, results speech in computer recognition, vision, speech recognition, and text processing. A popular example of DL success is its deployment and text processing. A popular example of DL success is its deployment in self-driving cars. in self-driving cars. Figure 1. A sample Portland cement concrete (PCC)-surfaced pavement distress image captured using Figurepavement 1. A sample data collection Portland vehicle cement moving concrete at highway (PCC)-surfaced speed and pavement equipped distresswith a downward-looking image captured using pavementhigh-speed data collectiondigital camera vehicle (Source: moving FHWA at LTPP highway database). speed and equipped with a downward-looking high-speed digital camera (Source: FHWA LTPP database). To date, applications of DL to pavement image analysis have mainly employed convolutional Toneural date, networks applications (CNNs of or DL ConvNets), to pavement a specific image DL analysis model with have many mainly convolution employed layers. convolutional Deep CNNs (DCNNs) are characterized by deeper architectures with numerous hidden layers enabling neural networks (CNNs or ConvNets), a specific DL model with many convolution layers. Deep them to learn many levels of abstraction, as opposed to shallow architectures with typically fewer CNNshidden (DCNNs) layers are[6–8]. characterized Although the first by deeper published architectures works on the with application numerous of CNNs hidden or DL layers in general enabling themto to pavement learn many crack levels detection of abstraction, appeared in 2016, as opposed some 12 papers to shallow have architecturesalready been published with typically between fewer hidden2016 layers and beginning [6–8]. Although of 2018. Although the first publishednot as steep works a growth on in the research application productivity of CNNs as in or other DL inareas general to pavementsuch as medical crack detection image analysis, appeared it is in evident 2016, somethat the 12 interest papers in have the alreadyapplication been of publishedDL to address between 2016various and beginning challenges of in 2018. vision-based Although automated not as steep pavement a growth distress in research detection productivity is fast growing. as in other areas such as medicalThis paper image provides analysis, the itfirst is evidentnarrative that review the intereston the application in the application of deep learning of DL to to address pavement various challengesimage inanalysis vision-based and automated automated distress pavement detection. distress Although detection there is is not fast yet growing. a sufficient number of papers on this topic to conduct a comprehensive survey in the traditional sense, this quick review This paper provides the first narrative review on the application of deep learning to pavement nonetheless covers enough ground to assess the state-of-the-art and will hopefully spur future image analysis and automated distress detection. Although there is not yet a sufficient number of research in the application of DL to pavement image analysis. Considering the rather narrow range papersof studies on this published topic to conductso far on this a comprehensive topic, peer-reviewed survey journal in the articles, traditional as well sense,as articles this appearing quick review nonethelessin conference covers proceedings, enough ground were to included assess the in state-of-the-art this narrative andreview. will Additionally, hopefully spur one future preprint research in theappearing application in arXiv of DL online to pavement repository image was also analysis. reviewed. Considering the rather narrow range of studies publishedThe so farrest on of thisthis topic,review peer-reviewed paper is structured journal as follows. articles, In as Section well as 2, articles we summarize appearing thein existing conference proceedings,and emerging were deep included learning in this software narrative frameworks review. for Additionally, computer vision one preprint applications, appearing especially in arXiv online repository was also reviewed. The rest of this review paper is structured as follows. In Section2, we summarize the existing and emerging deep learning software frameworks