THÈSETHÈSE En vue de l’obtention du DOCTORAT DE L’UNIVERSITÉ DE TOULOUSE Délivré par : École Nationale Supérieure des Mines d’Albi-Carmaux Présentée et soutenue le 21/11/2016 par : ■❣♦ ❏♦✈❛♥↔❡✈✐➣ Exterior inspection of an aircraft using a Pan-Tilt-Zoom camera and a 3D scanner moved by a mobile robot : 2D image processing and 3D point cloud analysis JURY Sylvie TREUILLET Maître de conférences, Université d’Orléans Rapporteur David FOFI Professeur des Universités, Université de Rapporteur Bourgogne Michel DEVY Directeur de recherche, LAAS-CNRS Président Rafael GARCIA Professor titular, Université de Gérone Examinateur Jean-José ORTEU Professeur de l’Institut Mines Télécom, Directeur École des mines d’Albi-Carmaux Thierry SENTENAC Maître-assistant de l’Institut Mines Télé- Co-directeur com, École des mines d’Albi-Carmaux École doctorale et spécialité : EDSYS : Robotique et Systèmes embarqués 4200046 Unité de Recherche : Institut Clément Ader UMR CNRS 5312, Université de Toulouse - Mines Albi Directeur(s) de Thèse : Jean-José ORTEU et Thierry SENTENAC Rapporteurs : Sylvie TREUILLET et David FOFI Acknowledgments First I would like to thank Thierry CUTARD, the head of the research lab Institut Clément Ader - Albi for welcoming me in the team. Special thanks to my supervisor Jean-José ORTEU for his guidance and friendly attitude from the very first day of my thesis. He always managed to find time for me despite of his busy schedule. I learned a lot from him, about our research field but also about the project management and work organization. I would also like to thank my second supervisor Thierry SENTENAC for useful discussions, advices and ideas about my work, as well as for giving me a wider picture of the Air-Cobot project. I am thankful to my colleague Rémi GILBLAS. It was a great pleasure to work with him on the same project. I am grateful to Sylvie TREUILLET and David FOFI for accepting to be re- viewers for this manuscript and also to Michel DEVY and Rafael GARCIA for accepting to be examiners during the defense. A gratitude is owed to Air-Cobot partners as well as to our colleagues Ludovic BRETHES and Xavier MAURICE, for fruitful collaboration. Utmost respect and gratitude to all the people from VIBOT community. VIBOT master program was my entrance to the field of computer vision and also a personal life changer. It helped me shape my profession choice and I also met extraordinary people from all around the world most of which have remained my close friends. Big thanks to all the people in Ecole des Mines d’Albi for their warm welcome and healthy work environment. I met exceptional people and made couple of close friends during these three years in Albi/Toulouse, which made me a richer person. Thanks to my friends from Montenegro for the great moments we had together and for staying in constant touch as if I was still there. Heartfelt thanks to my family for their constant encouragement. Individually, my parents for teaching me the right values, my sister for her unconditional love and my younger brother for his love, respect but also career advices. i Contents Contents 1 Introduction 1 1.1 Context .................................. 2 1.1.1 Aircraft maintenance ....................... 2 1.1.2 Air-Cobot project ........................ 2 1.1.2.1 Air-Cobot in ’walk-around’ .............. 4 1.1.2.2 Air-Cobot in ’hangar scenario’ ............ 7 1.2 Related work on airplane inspection .................. 8 1.2.1 Robotic airplane skin inspection ................ 8 1.2.2 Inspection of other large structures . 10 1.3 Contribution of the thesis ........................ 10 1.3.1 2D image analysis ........................ 11 1.3.2 3D point cloud analysis ..................... 12 2 Data acquisition and available resources 13 2.1 Sensors .................................. 14 2.1.1 Pan-Tilt-Zoom camera ...................... 14 2.1.1.1 Calibration ....................... 15 2.1.2 3D scanner ............................ 16 2.2 Simplified 3D CAD model of the airplane . 17 2.3 Coordinate frames and transformations . 19 2.4 Navigation and localization of the robot . 22 2.5 PTZ camera pointing and image acquisition . 24 2.6 Projection of an item onto the image . 27 2.7 Acquisition campaigns and dataset ................... 28 2.7.1 PTZ camera acquisition ..................... 32 2.7.2 3D scanner acquisition ...................... 32 3 Inspection by 2D Image processing 35 3.1 Introduction ................................ 37 3.2 Evaluation of developed algorithms ................... 38 3.2.1 Detection ............................. 38 3.2.2 Inspection ............................. 38 3.3 Preprocessing - background subtraction . 40 3.4 Geometrical features extraction ..................... 42 3.4.1 Straight lines extraction ..................... 42 3.4.1.1 Hough transform .................... 42 3.4.1.2 Segments extractors . 43 3.4.2 Elliptical arcs extraction ..................... 47 3.4.2.1 Postprocessing - merging ellipses . 48 3.5 Matching CAD model and image features . 50 iii Contents 3.5.1 Matching of primitives ...................... 51 3.5.1.1 Similarity function ................... 51 3.5.1.2 Search of mutual best match in a bipartite graph . 52 3.5.2 Matching results ......................... 53 3.6 Inspection by image analysis using CAD model . 60 3.6.1 Oxygen bay inspection ...................... 60 3.6.1.1 Watershed segmentation . 62 3.6.1.2 Adapted Hu moments for shape comparison . 62 3.6.1.3 Initial approach - without using CAD model . 63 3.6.1.4 Adopted approach - using CAD model . 66 3.6.1.5 Detection result refinement . 70 3.6.1.6 Pose re-estimation ................... 72 3.6.1.7 CLOSED/OPEN inspection . 75 3.6.1.8 Latched/unlatched inspection . 77 3.6.1.9 Experimental results . 77 3.6.2 Radome latch inspection ..................... 80 3.6.2.1 Detection ........................ 80 3.6.2.2 Pose re-estimation ................... 83 3.6.2.3 Inspection ....................... 86 3.6.2.4 Experimental results . 88 3.6.3 Air inlet vent inspection ..................... 91 3.6.3.1 Detection ........................ 91 3.6.3.2 Inspection ....................... 91 3.6.3.3 Experimental results . 91 3.6.4 Engine inspection ......................... 93 3.6.4.1 Related works ..................... 94 3.6.4.2 Detection ........................ 94 3.6.4.3 Inspection ....................... 96 3.6.4.4 Experimental results . 98 3.6.5 Pitot probe inspection ...................... 99 3.6.5.1 Detection . 100 3.6.5.2 Inspection . 102 3.6.5.3 Experimental results . 104 3.6.6 Pitot cover detection . 113 3.6.6.1 Detection . 113 3.6.6.2 Experimental results . 115 3.7 Inspection by image analysis without using CAD model . 116 3.7.1 Static port inspection . 116 3.7.1.1 Detection . 116 3.7.1.2 Inspection . 120 3.7.1.3 Experimental results . 122 3.7.2 Tire inspection . 125 3.7.2.1 Related work . 126 3.7.2.2 Detection . 127 iv Contents 3.7.2.3 Inspection . 131 3.7.2.4 Experimental results . 133 3.7.3 Wheel chock detection . 135 3.8 Conclusion ................................139 4 Inspection by 3D point cloud analysis 141 4.1 Introduction ................................142 4.1.1 Visual detection and dial gauge characterization . 142 4.1.2 Scanner based detection and characterization . 143 4.2 Related work ...............................144 4.3 Overview of the proposed method . 145 4.4 Acquisition ................................147 4.4.1 3D scanner and acquisition setup . 147 4.4.2 Acquisition parameters . 147 4.4.3 Dataset ..............................149 4.5 Method ..................................156 4.5.1 Defect detection process . 156 4.5.1.1 Step D1 : Preprocessing . 156 4.5.1.2 Step D2 : Normals and Curvature Estimation . 157 4.5.1.3 Step D3 : Segmentation . 159 4.5.1.4 Step D4 : Labeling . 161 4.5.2 Defect characterization process . 161 4.5.2.1 Step C1 : 3D/2D projection . 162 4.5.2.2 Step C2 : Data preparation . 166 4.5.2.3 Step C3 : Reconstruction . 166 4.5.2.4 Step C4 : Extracting information about the defects 168 4.6 Experiments and discussion . 169 4.6.1 Evaluation using dial gauge ground truth . 175 4.6.2 Execution time . 177 4.7 Conclusions ................................177 5 Conclusion and perspectives 179 6 Résumé long en français 183 6.1 Introduction ................................183 6.1.1 Types d’inspection . 184 6.1.2 Méthodes d’inspection développées . 184 6.2 Analyse d’images 2D . 185 6.3 Analyse d’images 2D exploitant le modèle CAO de l’élément à inspecter188 6.4 Analyse de nuages de points 3D (3D data processing) . 189 6.4.1 Conclusion et perspectives . 193 A Representative set of items 197 B Thesis publications 201 v Chapter 1 Introduction 1.1 Context .................................. 2 1.1.1 Aircraft maintenance ....................... 2 1.1.2 Air-Cobot project ........................ 2 1.1.2.1 Air-Cobot in ’walk-around’ .............. 4 1.1.2.2 Air-Cobot in ’hangar scenario’ ............ 7 1.2 Related work on airplane inspection .................. 8 1.2.1 Robotic airplane skin inspection ................ 8 1.2.2 Inspection of other large structures . 10 1.3 Contribution of the thesis ........................ 10 1.3.1 2D image analysis ........................ 11 1.3.2 3D point cloud analysis ..................... 12 1 Chapter 1. Introduction 1.1 Context 1.1.1 Aircraft maintenance Airplane can be in active state in the air and in passive state in the hangar or at the tarmac. While being in any of the two states, airplane is in danger to suffer different kinds of damages, such as cracks, scratches, dents, corrosion, etc. The causes are various: chemical reactions, thermic cycle of the airplane, material properties, hits by airport vehicles or birds, skin aging. Additionally, between two flights, state of many items can be changed, for ex. a door can be left open or a protective cover can be forgotten on the sensor.