Visual Recognition Systems in a Car Passenger Compartment with the Focus on Facial Driver Identification
Total Page:16
File Type:pdf, Size:1020Kb
Visual recognition systems in a car passenger compartment with the focus on facial driver identification D I S S E R T A T I O N zur Erlangung des akademischen Grades Doktoringenieur (Dr.-Ing.) angenommen durch die Fakultät für Informatik der Otto-von-Guericke-Universität Magdeburg von M. Sc. Andrey Makrushin geb. am 16.02.1980 in Moskau, Russland Gutachterinnen/Gutachter Prof. Dr.-Ing. Jana Dittmann Prof. Dr. Sabah Jassim Prof. Dr. rer. nat. Reiner Creutzburg Magdeburg, den 22.01.2014 Andrey Makrushin: Visual recognition systems in a car passenger compartment with the focus on facial driver identification, Dissertation, Otto-von-Guericke University of Magdeburg, 2014. Acknowledgements First and foremost I would like to express my deep thanks to my supervisor Prof. Jana Dittmann for motivating me to start working towards a PhD degree and for giving me the opportunity to join her research group. Moreover, I thank her for continuously supporting me not only academically but also financially by enabling me to work on diverse projects and generally for my time with the Advanced Multimedia and Security Lab (AMSL). My deep thanks also go to Prof. Claus Vielhauer for evoking my interest in biometrics and encouraging me to start working on face recognition. I thank both Jana and Claus for all that I learned from them, for putting me on the right track regarding my research activities, for fruitful discussions and insightful advice. I am very grateful to all my colleagues from AMSL and Brandenburg University of Applied Science for the time discussing aspects of pattern recognition, working together on projects and papers and generally for the time we spent together. I especially appreciate the help from Tobias Scheidat, Christian Kr¨atzer,Michael Biermann, Mario Hildebrandt, Ronny Merkel, Kun Qian and Maik Schott. I am very grateful to Dr. Mirko Langnickel and Dr. Katharina Seifert from Volkwagen Group Research for the joint work on the VisMes project that evoked my interest in visual recognition systems in the automotive domain. I also appreciate the help of my former students Robert Krauße, Enrico Herrmann, Andy Bertz and Sebastian Kleinau. Our joint work on the project provided me with numerous new ideas. I would like to thank the reviewers Prof. Sabah Jassim and Prof. Reiner Creutzburg, who kindly agreed to read and evaluate my thesis. I thank all whose who agreed to participate in the experiments and acted as donors by gathering data for our visual databases. Special thanks go to my English teachers Marcelo Kauer, Tim Sadler and Hans Harald Huber, who helped me to tremendously improve my language skills making it possible to prepare this manuscript. Last but not least, I would like to express my deep thanks to my family and all my friends for their moral support throughout the long time working on the thesis. They always believed in my potential to finish the PhD study even at moments when I did not. Certainly, I cannot mention everyone. Katharina Franke, Sergei Horbach, Tatjana Bobach-Poltoratski, Dr. Yuriy Tsepkovskiy, Dr. Andriy Telesh, Dr. Dmitry Vlasenko and Valery Makhavikou are those who continuously supported me in struggling with misfortune and frustrations and who were always ready to talk about my ideas and excitements. Frankly, I greatly appreciate this. ——————————– Andrey Makrushin Magdeburg, 2014 iii Abstract (in German) Die Ausr¨ustungder modernen Fahrzeuge mit Kameratechnik ist ein neuer Trend der Automobil- branche. Visuelle Uberwachung¨ der Fahrzeugumgebung sowie des Innenraums bietet eine Reihe zus¨atzlicher Sicherheits- und Komfortfunktionen. Eine Innenraumkamera kann unter anderem eine bedingte Airbagausl¨osungsowie M¨udigkeitskontrolle unterst¨utzenoder zur Verbesserung der Info- tainmentsysteme dienen. Die Personalisierung des Fahrzeugs ist ein weiterer Punkt, wo eine Kamera erfolgreich eingesetzt werden kann. Eine Memoryfunktion f¨urdie Speicherung von Sitz-, Lenkrad-, Spiegelpositionen, sowie Klimaeinstellungen kann nicht nur ¨uber die ¨ubliche tastenbasierte Auswahl des Profils oder den Smart Key realisiert werden, sondern auch durch eine biometrische Identi- fikation des Fahrers. Das Gesicht scheint an dieser Stelle die geeignetste biometrische Modalit¨at zu sein. Dabei wird eine Identifikation und anschließende Anpassung ohne bewusste Interaktion seitens des Fahrers stattfinden, was zu einer Komfortsteigerung f¨uhrt.Außerdem haben kontak- tlose Authentifizierungsverfahren eine hohe Benutzerakzeptanz. Biometrische Modalit¨atensind nicht ¨ubertragbar und sehr schwer verf¨alschbar. In Folge dessen steigt die Entf¨uhrungsresistenz des Autos, indem einen Motorstart nur nach eine erfolgreiche Identifikation erlaubt wird. Diese Arbeit besch¨aftigtsich mit der Analyse der Gesichtserkennungsmethoden und ihre Anwendbarkeit im automotiven Kontext. Es werden ausgew¨ahlte Szenarien f¨urReferenzdatensammlung und Fahreridentifizierung entwickelt sowie ein Konzept des KFZ-Innenraum-Gesichtserkennungssystems entworfen und implementiert. Im Rahmen der Forschung wird eine Menge praktischer Prob- leme analysiert wie z.B. Kamera- und Lichtquellenpositionierung, erforderliche Bildverarbeitung, Gesichtsdetektierung, Merkmalsextraktion, Klassifizierungsverfahren und quantitative Evaluierung der Ergebnisse. Insbesondere werden darstellungsbasierte und merkmalsbasierte Verfahren zur Gesichtserkennung verglichen. Dar¨uber hinaus werden die Anforderungen an die Aufnahmetechnik formuliert. Die experimentelle Evaluierung wird mit zwei Benutzerdatenbanken durchgef¨uhrt.F¨ur den allgemeinen Erkennungsperformanztest stand eine weit verbreitete biometrische Datenbank von 295 Personen (XM2VTS) zur Verf¨ugung.Zur realit¨atsnahenUntersuchung des Systems wird eine eigene Datenbank von 54 Fahrern in einem realen Fahrzeug erstellt. v Abstract Equipping a modern vehicle with optical sensors is a new tendency in the automotive industry. The visual observation of car surroundings as well as a car passenger compartment offers a world of additional mechanisms to increase safety, security and comfort. In current cars, an in-car camera supports smart airbag deployment, drowsiness detection and contributes to an advanced level of interaction with infotainment systems. The car personalization is the next step towards a superior car-driver interaction implying the ability of a car to recognize its driver. Such personalization also guarantees a higher level of anti-theft protection granting the engine start exclusively to registered drivers. Nowadays, the personalization is mainly intended for comfort enhancement and associated with a memory function. The memory function activation for the automatic adjustment of the seat, steering wheel, mirrors and air-conditioning can be reached not only using memory buttons or a personal smart key, but also by applying biometric driver identification. Biometric modalities have the advantage that they are non-transferable and very hard to forge. In this context, the face seems to be the most appropriate biometric modality because of the usage of a contact-less acquisition device, high level of user acceptance and the fact that identification arises unconsciously for a driver, making a car more comfortable. This work is devoted to the feasibility analysis of the automatic face recognition techniques in the automotive context. The concept of an in-car face recognition system is developed and implemented in a real car. Several scenarios for collecting biometric data and for driver identification are proposed. A number of practical problems have been addressed. These are positioning of a camera and light sources, image pre-processing, face detection, feature extraction, classification and experimental evaluation of the system performance. In particular, appearance-based and feature-based approaches for face recognition are compared. On top of that, the requirements to the acquisition and identification subsystems are conceived. The empiric study on the recognition performance comprises experiments on two databases. For the evaluation of the general performance of the developed face recognition system (FaceART) and comparison to state-of-the-art systems, the XM2VTS database of 295 persons is utilized. The realistic recognition performance of the proposed system has been evaluated based on the database of 54 car drivers, collected by the author in the real car. vii Contents List of figures xx List of tables xxii Abbreviations xxvi 1 Introduction 3 1.1 Problem description . .3 1.1.1 Motivation (camera in automotive applications) . .3 1.1.2 Formal view to safety and comfort of a vehicle . .4 1.1.3 Looking inside and outside of a car . .5 1.1.4 Car personalization . .9 1.2 Research challenges . 11 1.3 Research objectives . 12 1.4 Summary of main contributions . 14 1.4.1 Seat occupancy detection (SOD) . 15 1.4.2 Distinguishing between driver and front-seat passenger hands during the interaction with the center console (UDiS) . 16 1.4.3 Facial driver recognition (FDR) . 17 1.4.4 Conclusion . 19 1.5 Thesis outline . 19 2 Thesis fundamentals and summary of related works 21 2.1 Image understanding . 21 2.1.1 Imaging sensors . 21 2.1.2 Semantic gap between an object and its appearance on the image . 23 2.1.3 Pattern recognition . 24 2.1.4 Biometrics as a special case of statistical pattern classification . 28 2.2 State-of-the-art, automotive related part . 30 2.2.1 Requirements to in-vehicle computer vision systems . 30 2.2.2 Drowsiness detection . 32 ix 2.2.3 Support of smart airbag . 34 2.2.4 Distinguishing between driver and front-seat passenger