PACIFIC EARTHQUAKE ENGINEERING RESEARCH CENTER PEER Hub ImageNet (-Net): A Large-Scale Multi-Attribute Benchmark Dataset of Structural Images Yuqing Gao Khalid M. Mosalam Department of Civil and Environmental Engineering University of California, Berkeley PEER Report No. 2019/07 Pacifi c Earthquake Engineering Research Center Headquarters at the University of California, Berkeley November 2019 PEER 2019/07 November 2019 Disclaimer The opinions, fi ndings, and conclusions or recommendations expressed in this publication are those of the author(s) and do not necessarily refl ect the views of the study sponsor(s), the Pacifi c Earthquake Engineering Research Center, or the Regents of the University of California. PEER Hub ImageNet (-Net): A Large-Scale Multi-Attribute Benchmark Dataset of Structural Images Yuqing Gao Khalid M. Mosalam Department of Civil and Environmental Engineering University of California, Berkeley PEER Report 2019/07 Pacific Earthquake Engineering Research Center Headquarters at the University of California, Berkeley November 2019 ii ABSTRACT In this data explosion epoch, data-driven structural health monitoring (SHM) and rapid damage assessment after natural hazards have become of great interest in civil engineering research. This report introduces deep-learning (DL) approaches and their application to structural engineering, such as post-disaster structural reconnaissance and vision-based SHM. Using DL in vision-based SHM is a relatively new research direction in civil engineering. As researchers begin to apply these concepts to structural engineering concerns, two critical issues remain to be addressed: (1) the lack of a uniform automated detection principle or framework based on domain knowledge; and (2) the lack of benchmark datasets with well-labeled large amounts of data. To address the first issue, an automated and hierarchical framework has been proposed: the PHI-Net or -Net for the PEER Hub Image-Net. This framework consists of eight basic benchmark detection tasks based on current domain knowledge and past reconnaissance experience. The second area of concern is based on the -Net framework; a large number of structural images was collected, preprocessed, and labeled to form an open-source online large- scale multi-attribute image dataset, namely, the -Net dataset. At the time of this writing, this dataset contains 36,413 images with multiple labels. This report introduces herein three deep convolutional neuronal networks (CNN): VGG- 16, VGG-19, and ResNet-50. The architecture design and network properties, etc., are described and discussed. For benchmarking purposes, a series of computer experiments are conducted. Multiple factors are considered in comparison studies under a fair setting of hyper-parameters and training approaches, i.e., using affine data augmentation (ADA) and transfer learning (TL). All experimental results are reported and discussed, which provide benchmark and reference values for future studies by other researchers developing new algorithms. These results reveal the great potential of using DL in vision-based SHM. Finally, the first image-based challenge in structural engineering was held by the Pacific Earthquake Engineering Research (PEER) Center during the Fall of 2018. This challenge, designated as the -Net Challenge, served as a pre-event prior to the open sourcing of the -Net dataset and attracted worldwide attention and participation from researchers from around the globe. iii iv ACKNOWLEDGMENTS This project was sponsored by Tsinghua-Berkeley Shenzhen Institute (TBSI). The authors greatly appreciate the assistance from these student volunteers: Shuyuan Liang, Yujia Xu, Renjie Wu, Yidong Huang, Rongbing Zhou, Boyuan Kong, Jiachen Wang, Shuai Yao, Enze Cheng, Fan Hu, Peijie Li, Hagen Tam, Matthew Yeung, Lillian Lau, Victor Khong, Ke Xu, Jiawei Chen, Wen Tang, Ran Jiang, Jianfei Yang, Pengyuan Zhai, Chrystal Chern, Chenglong Li, and Sachit Shroff. The authors thank the Pacific Earthquake Engineering Research Center (PEER) staff members (Dr. Amaranth Kasalanati, Dr. Selim Günay, Mr. Gabriel Vargas, Ms. Grace Kang, and Ms. Erika Donald) and all the experts who participated in the -Net Challenge. The authors also appreciate the proofreading work by Ms. Fan Hu and Ms. Chrystal Chern, and the excellent editorial effort by Ms. Claire Johnson of this report. Any opinions, findings, and conclusions or recommendations expressed in this report are those of the authors and do not necessarily reflect those of PEER, TBSI, of the Regents of the University of California. v vi CONTENTS ABSTRACT .................................................................................................................................. iii ACKNOWLEDGMENTS .............................................................................................................v TABLE OF CONTENTS ........................................................................................................... vii LIST OF TABLES ....................................................................................................................... ix LIST OF FIGURES ..................................................................................................................... xi 1 INTRODUCTION..............................................................................................................1 1.1 Motivation ...............................................................................................................1 1.2 Related Work .........................................................................................................3 1.3 Report Organization ..............................................................................................5 2 PEER HUB IMAGENET (-NET) ...................................................................................7 2.1 Framework of -Net ..............................................................................................7 2.2 Benchmark Detection Tasks .................................................................................9 2.2.1 Task 1: Scene Level .....................................................................................9 2.2.2 Task 2: Damage State ................................................................................11 2.2.3 Task 3: Spalling Condition (Material Loss) ...............................................11 2.2.4 Task 4: Material Type ................................................................................12 2.2.5 Task 5: Collapse Mode ..............................................................................13 2.2.6 Task 6: Component Type ...........................................................................15 2.2.7 Task 7: Damage Level ...............................................................................17 2.2.8 Task 8: Damage Type ................................................................................18 2.3 Data Collection, Pre-Processing, Labeling, and Splitting ................................21 2.3.1 Data Collection ..........................................................................................21 2.3.2 Data Pre-Processing ...................................................................................21 2.3.3 Data Labeling .............................................................................................22 2.3.4 Data Splitting .............................................................................................24 2.4 Data Summary .....................................................................................................28 2.5 Extensions on Dataset Hierarchy .......................................................................29 vii 3 DEEP CONVOLUTIONAL NEURAL NETWORKS .................................................31 3.1 VGGNet ................................................................................................................31 3.2 ResNet ...................................................................................................................35 4 BENCHMARK EXPERIMENTS AND DISCUSSIONS .............................................37 4.1 Benchmark Experiments .....................................................................................37 4.1.1 Task 1: Scene Level ...................................................................................39 4.1.2 Task 2: Damage State ................................................................................41 4.1.3 Task 3: Spalling Condition ........................................................................43 4.1.4 Task 4: Material Type ................................................................................45 4.1.5 Task 5: Collapse Mode ..............................................................................47 4.1.6 Task 6: Component Type ...........................................................................49 4.1.7 Task 7: Damage Level ...............................................................................51 4.1.8 Task 8: Damage Type ................................................................................53 4.2 Results Summary .................................................................................................55 5 -NET CHALLENGE 2018 ............................................................................................61 5.1 Motivation of the Challenge ................................................................................61 5.2 Participation of the -Net Challenge..................................................................64