http://lib.uliege.ac.be http://matheo.uliege.be Master thesis : Facial recognition using deep neural networks. Auteur : Dubois, Antoine Promoteur(s) : Wehenkel, Louis; Van Droogenbroeck, Marc Faculté : Faculté des Sciences appliquées Diplôme : Master en ingénieur civil en informatique, à finalité spécialisée en "intelligent systems" Année académique : 2017-2018 URI/URL : http://hdl.handle.net/2268.2/4650 Avertissement à l'attention des usagers : Tous les documents placés en accès ouvert sur le site le site MatheO sont protégés par le droit d'auteur. Conformément aux principes énoncés par la "Budapest Open Access Initiative"(BOAI, 2002), l'utilisateur du site peut lire, télécharger, copier, transmettre, imprimer, chercher ou faire un lien vers le texte intégral de ces documents, les disséquer pour les indexer, s'en servir de données pour un logiciel, ou s'en servir à toute autre fin légale (ou prévue par la réglementation relative au droit d'auteur). Toute utilisation du document à des fins commerciales est strictement interdite. Par ailleurs, l'utilisateur s'engage à respecter les droits moraux de l'auteur, principalement le droit à l'intégrité de l'oeuvre et le droit de paternité et ce dans toute utilisation que l'utilisateur entreprend. Ainsi, à titre d'exemple, lorsqu'il reproduira un document par extrait ou dans son intégralité, l'utilisateur citera de manière complète les sources telles que mentionnées ci-dessus. Toute utilisation non explicitement autorisée ci-avant (telle que par exemple, la modification du document ou son résumé) nécessite l'autorisation préalable et expresse des auteurs ou de leurs ayants droit. University of Liege` Faculty of Applied Sciences Departement of Electrical Engineering and Computer Science Automatic Face Detection and Recognition using Neural Networks Graduation Studies conducted for obtaining the Master's degree in Computer Engineering by Antoine Dubois Promoters: Author: Pr. Louis Wehenkel Antoine Dubois Pr. Marc Van Droogenbroeck Academic Year 2017-2018 Abstract Innovation or the introduction of the new. Innovation has always been the motor of civilization by introducing new ideas to solve the countless problems that such a complex system encounters. It has spurred economic growth and brought the comfort we live in today. In my opinion, in Belgium, particularly in the south, we have often rested on our laurels and forgot to innovate. This observation fostered my choice to undertake this thesis in the context of the RAGI project. This innovative project, relying on academic research to produce a commercial product, is developing an \intelligent system for recognition, welcoming, and guidance". Innovation in this project comes partly from the use of a machine learning technique called automatic face recognition to be able to identify people and guide them in a building. The goal of this thesis is to study this concept to help the team working on RAGI take appropriate decisions. To achieve this goal, I search for and analyze state-of-the-algorithms in both face detection and face recognition. The research for algorithms goes through the analysis of recent benchmarks, two of which (i.e. WIDER FACE [111] and MegaFace [47]) are also used for evaluating those algorithms. Simultaneously, this work points out the difficulties to perform such a research and testing process. The results on these benchmarks allows to determine which algorithms perform better, that is to say SSH [65] for detection and both Dlib-R [20] and ArcFace [21] for recognition. For detection, the influence of facial attributes such as pose, size or blur is explored. Finally, to have more relatable results with regards to the RAGI project, we designed a specific dataset on which the same algorithms are tested. Composed of 494 frames with 3561 annotated faces from 13 different identities, it allowed us to study other parameters while confirming the results obtained on the publicly available datasets. All those tests are performed with algorithm efficiency in mind and computation time measurements show that the best techniques tend to work slower but that they can achieve practical execution times. i Acknowledgments First, I would like to thank Professors Louis Wehenkel and Marc Van Droogenbroeck for ac- cepting to be my supervisors for this thesis and Pierre Geurts and Fran¸coisVan Lishout for being part of my thesis jury. I am especially grateful to mister Wehenkel who followed me through this work and gave me useful advice. I am also thankful to the RAGI team members, Fran¸coisVan Lishout, Linda Wang and Tom Ewbank for lending me one of their computer as well as the constructive exchange we had on the project. Then, I want to warmly acknowledge the group of volunteers that accepted to collaborate in the construction of the RAGI dataset. In addition to the RAGI team members, I thank Denis Bourguignon, Michael Di Bartelomeo, Laura Ferire, Marc Fr´ed´eric,Damien G´erard,Samuel Hiard, Beno^ıtMattheus, Olivier Suplis and Nicolas Vecoven. Finally, I would like to give thanks to my family members for supporting me throughout this work. Antoine Dubois ii Contents Abstract i Acknowledgments ii 1 Introduction1 2 Background3 2.1 Face detection.....................................3 2.1.1 Definition...................................3 2.1.2 Evaluation...................................3 2.1.3 Brief literature review............................6 2.2 Face recognition....................................6 2.2.1 Definition...................................6 2.2.2 Evaluation...................................7 2.2.3 Brief literature review............................8 2.3 The in-the-wild concept...............................9 2.4 End-to-end pipeline..................................9 2.5 Using artificial neural networks........................... 10 2.6 Time matters..................................... 11 3 Benchmark Anaysis 12 3.1 Face detection..................................... 12 3.1.1 FDDB..................................... 12 3.1.2 AFLW..................................... 13 3.1.3 AFW...................................... 14 3.1.4 Pascal-Faces.................................. 14 3.1.5 MALF..................................... 15 3.1.6 WIDER FACE................................ 16 3.1.7 IJB-C..................................... 17 3.1.8 Summary and choice of testing benchmark................. 18 3.2 Face recognition.................................... 19 3.2.1 LFW...................................... 19 3.2.2 PubFig..................................... 21 3.2.3 YTF...................................... 21 3.2.4 CFP...................................... 22 3.2.5 CACD..................................... 23 3.2.6 PIPA...................................... 23 3.2.7 MegaFace................................... 24 3.2.8 IJB-C..................................... 25 iii 3.2.9 Summary and choice of testing benchmark................. 26 4 Algorithm Selection 28 4.1 Face detection..................................... 28 4.1.1 Benchmark-based selection.......................... 28 4.1.2 List of potential algorithms to test..................... 31 4.2 Face recognition.................................... 32 4.2.1 Benchmark-based selection.......................... 32 4.2.2 List of potential algorithms to test..................... 34 5 Algorithm Description 35 5.1 Face detection..................................... 35 5.1.1 VJ....................................... 35 5.1.2 HOG...................................... 37 5.1.3 FRCNN.................................... 38 5.1.4 SFD...................................... 42 5.1.5 SSH...................................... 43 5.2 Face recognition.................................... 45 5.2.1 OpenFace................................... 45 5.2.2 Dlib-R..................................... 46 5.2.3 ArcFace.................................... 46 6 Algorithm Evaluation 49 6.1 Face detection..................................... 49 6.1.1 Evaluation protocol.............................. 49 6.1.2 Time evaluation................................ 51 6.1.3 Per-algorithm results............................. 52 6.1.4 Overall results................................. 63 6.1.5 Influence of parameters on overall results.................. 65 6.2 Face recognition.................................... 71 6.2.1 Evaluation protocol.............................. 71 6.2.2 Time evaluation................................ 71 6.2.3 Per-algorithm results............................. 72 6.2.4 Overall results................................. 76 7 Application to RAGI 78 7.1 Dataset construction................................. 78 7.1.1 Data acquisition................................ 78 7.1.2 Face detection annotation.......................... 80 7.1.3 Face recognition annotation......................... 81 7.1.4 Statistics.................................... 81 7.2 Testing......................................... 86 7.2.1 Face detection................................. 86 7.2.2 Face recognition................................ 91 7.2.3 Face detection and recognition........................ 103 8 Conclusion 107 iv A Dataset Samples 110 A.1 WIDER FACE.................................... 110 A.2 MegaFace....................................... 111 A.3 FaceScrub....................................... 111 A.4 RAGI......................................... 112 A.4.1 Face detection................................. 112 A.4.2 Face recognition................................ 113 B Libraries versions 114 v Chapter 1 Introduction This master thesis was