Bachelor Thesis Electrical Engineering June 2020 Raspberry Pi Based Vision System for Foreign Object Debris (FOD) Detection Sarfaraz Ahmad Mahammad Sushma Vendrapu Department of Mathematics and Nature Sciences Blekinge Institute of Technology SE–371 79 Karlskrona, Sweden This thesis is submitted to the Department of Mathematics and Nature Science at Blekinge Institute of Technology in partial fulfillment of the requirements for the degree of Bach- elor in Electrical Engineering with Emphasis on Telecommunication. Contact Information: Authors: Sarfaraz Ahmad Mahammad E-mail: [email protected] Sushma Vendrapu E-mail: [email protected] Supervisor: Prof. Wlodek J. Kulesza Industrial Supervisors: Dawid Gradolewski Damian M. Dziak Address: Bioseco Sp. z o. o. Budowlanych 68 Street 80-298 Gdansk´ Poland University Examiner: Irina Gertsovich Department of Mathematics and Nature Sci- Internet : www.bth.se ence Blekinge Institute of Technology Phone : +46 455 38 50 00 SE–371 79 Karlskrona, Sweden Fax : +46 455 38 50 57 Abstract Background: The main purpose of this research is to design and develop a cost-effective system for detection of Foreign Object Debris (FOD), dedicated to airports. FOD detection has been a significant problem at airports as it can cause damage to aircraft. Developing such a device to detect FOD may require complicated hardware and software structures. The proposed solution is based on a computer vision system, which comprises of flexible off the shelf components such as a Raspberry Pi and Camera Module, allowing the simplistic and efficient way to detect FOD. Methods: The solution to this research is achieved through User-centered design, which implies to design a system solution suitably and efficiently. The system solu- tion specifications, objectives and limitations are derived from this User-centered design. The possible technologies are concluded from the required functionalities and constraints to obtain a real-time efficient FOD detection system. Results: The results are obtained using background subtraction for FOD detection and implementation of SSD (single-shot multi-box detector) model for FOD classification. The performance evaluation of the system is analysed by testing the system to detect FOD of different size for different distances. The web design is also implemented to notify the user in real-time when there is an occurrence of FOD. Conclusions: We concluded that the background subtraction and SSD model are the most suitable algorithms for the solution design with Raspberry Pi to detect FOD in a real-time system. The system performs in real-time, giving the efficiency of 84% for detecting medium-sized FOD such as persons at a distance of 75 meters and 72% efficiency for detecting large-sized FOD such as cars at a distance of 125 meters, and the average frame per second (fps) that is the system ’s performance in recording and processing frames of the area required to detect FOD is 0.95. Keywords: Airports, Computer vision, Performance evaluation, Real-time systems, User Centered Design, Web design. Acknowledgements We would like to express our gratitude to the Bioseco Company for assigning both of us in this project. We would like to express our gratitude to Dawid Gradolewski and Damian Dziak. They guided us in the working process and help us with both software and hardware problems. Thank you for giving the opportunity to participate in such unique project. We would also like to thank our supervisor Prof. Wlodek J. Kulesza, for pro- viding us guidance, suggestions, and comments. He gave us inspiration and motivation during the realization of this project. We sincerely thank our examiner Irina Gertsovich for assigning the project. We would also thank our parents and friends, especially to Mr. Mohammad Ali and Mr. Ajay Kumar for helping us in this project. This research was conducted within grant "Carrying out research and development works necessary to develop a new autonomous AIRPORT FAUNA MONITORING SYSTEM (AFMS) reducing the number of collisions between aircraft and birds and mammals" (No. POIR.01.01.01-00-0020/19) from The National Centre for Research and Development of Poland. ii Contents Abstract i Acknowledgements ii List of Figures v List of Tables vi Acronyms vii 1 Introduction 1 2 Survey of Related Works 3 2.1 Problem Overview . 3 2.2 Hardware Solutions . 5 2.3 Software Solutions . 6 2.4 Summary . 7 3 Problem Statement, Objectives and Main Contributions 8 3.1 Problem Statement . 8 3.2 Thesis Objectives . 9 3.3 Main Contributions . 9 4 System Design and Modeling 11 4.1 System Design . 11 4.2 System Modeling . 16 4.2.1 Hardware Model . 16 4.2.2 Software Model . 17 5 System Implementation, Prototyping and Validation 19 5.1 System Implementation and Prototyping . 19 5.1.1 Hardware Implementation and Prototype . 19 5.1.2 Software Implementation and Prototype . 22 5.2 Validation . 30 iii 6 Discussion 36 7 Conclusions and Future Works 40 7.1 Conclusions . 40 7.2 Future Works . 41 References 43 Appendices 48 Appendix A 49 A.1 Program Listing of FOD Detection and Classification . 49 Appendix B 54 B.1 Program Listing of Flask based Web Server . 54 B.2 Program Listing of Web Pages displayed by Web Server . 56 iv List of Figures 4.1 Proposed design process for FOD detection system . 12 4.2 Block diagram of FOD detection system model . 17 4.3 Software modeling of the FOD detection system . 18 5.1 Connection of Raspberry Pi and camera module [35] .......... 20 5.2 Testing Raspberry Pi and camera module . 21 5.3 System prototype for FOD detection (front view) . 21 5.4 System prototype for FOD detection (back view) . 21 5.5 System flowchart of detecting, classifing and notifying the presence ofFOD....................................... 23 5.6 Flowchart for background subtraction on the system to detect FOD . 24 5.7 Flowchart of FOD classification on the system . 27 5.8 Flowchart of web interface on the system . 28 5.9 Web interface display during no FOD occurrence . 29 5.10 Web interface response during FOD occurrence . 29 5.11 Web page displaying the data of occurred FOD . 30 5.12 Detection of FOD (person) at a distance of 25 meters . 32 5.13 Detection and classification of multiple moving objects . 34 6.1 Graphical representation of system efficiency for detecting and clas- sifying medium size FOD - a person . 38 6.2 Graphical representation of system efficiency for detecting and clas- sifying large size FOD-acar......................... 39 v List of Tables 2.1 Sources, types of FOD and causes of occurrence [14] .......... 4 2.2 State-of-the art technologies & research works . 7 4.1 Technologies and algorithms related to itemized functionalities and constraints (selected technologies are bolded) . 15 5.1 Detection and classification of FOD (person) for different distances . 33 5.2 System efficiency for FOD (person) detection . 33 5.3 System efficiency to classify the detected FOD (person) . 34 5.4 Detection and classification of FOD (car) for different distances . 35 5.5 System efficiency for FOD (car) detection . 35 6.1 System efficiency to detect FOD . 37 7.1 Efficiency of the system to detect FOD for different ranges . 41 vi Acronyms CNN Convolutional Neural Network. CSS Cascading Style Sheets. DNN Deep Neural Network. FAA Federal Aviation Administration. FFC Flexible Flat Cables. FOD Foreign Object Debris. FoV Field of View. FPS Frame Per Second. GPS Global Positioning System. HTML Hyper Text Markup Language. mAP mean Average Precision. mmWave Millimeter Wave. MoG Mixture-of-Gaussians. RPN Region Proposal Networks. SA Surface Area. SSD Single Shot Detector. STN Spatial Transformer Network. UDD User-Driven Design. WPT Wireless Power Transmission. vii Chapter 1 Introduction The problem with Foreign Object Debris (FOD) at the airports has increased rapidly in recent years. It is observed that accidents due to FOD occur mainly at airport runways, gateways and taxiways [1]. In unlikely situations, it can cause damage to the aircraft tires or engines excluding them from operating. The resulting situation also gives rise to the substantial delay of multiple aircraft and in extreme cases, it can cause an accident with the possibility of casualties. Based on the research done by the French Study on Automatic Detection Systems, over 60 % of the collection known FOD items were made of metal, followed by 18 % made of rubber [2]. That is to say, FOD arises to a big problem in the aviation industry that im- pacts the security of aircraft. For this reason, in recent years, several research works were performed to develop a suitable solution for FOD detection. The financial loss to the aviation industry is estimated to be 4 billion dollars per year [3]. Besides the money, there are also invaluable losses, like in the year 2000 where Air France flight 4590 was crashed due to a small metal strip resulting in in-flight fire and loss of control. The metal strip was caused by the continental flight, which took off from the same runway moments ago. Unfortunately, this crash resulted in 113 casualties [4]. The detection of birds and other animals at the airport runways is challenging, due to the necessity of monitoring the vast area around the runway. Damage of aircraft is mostly caused by bird ingestion into engines. Along with the birds many kinds of mammals also result in damaging the aircraft due to improper security fencing of airports. Unfortunately during one incident, a deer resulted in crashing of an aircraft at Sitka’s runways [5], and in 2015 a kangaroo caused an aircraft crash [6]. In many airports, wildlife collisions with aircraft are on the rise. According to the Federal Aviation Administration (FAA), the overall number of strikes raised from about 1,800 in 1990 to over 16,000 in 2018 [7].