DESIGN and IMPLEMENTATION of DRIVER DROWSINESS DETECTION SYSTEM by Aleksandar Colic
Total Page:16
File Type:pdf, Size:1020Kb
DESIGN AND IMPLEMENTATION OF DRIVER DROWSINESS DETECTION SYSTEM by Aleksandar Colic A Dissertation Submitted to the Faculty of The College of Engineering & Computer Science in Partial Fulfillment of the Requirements for the Degree of Doctor of Philosophy Florida Atlantic University Boca Raton, FL December 2014 Copyright 2014 by Aleksandar Colic ii ACKNOWLEDGEMENTS I would like to express my sincere gratitude to my advisor Dr. Oge Marques for his support, guidance and encouragement throughout my graduate studies. I also wish to thank my committee: Dr. Borko Furht, Dr. Robert B. Cooper and Dr. Shihong Huang for their invaluable suggestions. I am deeply thankful to Jean Mangiaracina from Graduate Programs for her immeasurable help on this long journey. And last but not least my many thanks go to my family for always believing in me and friends and colleagues for always being there for me. iv ABSTRACT Author: Aleksandar Colic Title: Design and Implementation of Driver Drowsiness Detection System Institution: Florida Atlantic University Dissertation Advisor: Dr. Oge Marques Degree: Doctor of Philosophy Year: 2014 There is a substantial amount of evidence that suggests that driver drowsiness plays a significant role in road accidents. Alarming recent statistics are raising the interest in equipping vehicles with driver drowsiness detection systems. This disserta- tion describes the design and implementation of a driver drowsiness detection system that is based on the analysis of visual input consisting of the driver's face and eyes. The resulting system combines off-the-shelf software components for face detection, human skin color detection and eye state classification in a novel way. It follows a behavioral methodology by performing a non-invasive monitoring of external cues describing a driver's level of drowsiness. We look at this complex problem from a systems engineering point of view in order to go from a proof-of-concept prototype to a stable software framework. Our system utilizes two detection and analysis meth- ods: (i) face detection with eye region extrapolation and (ii) eye state classification. Additionally, we use two confirmation processes { one based on custom skin color detection, the other based on nod detection { to make the system more robust and resilient while not sacrificing speed significantly. The system was designed to be dy- namic and adaptable to conform to the current conditions and hardware capabilities. v DESIGN AND IMPLEMENTATION OF DRIVER DROWSINESS DETECTION SYSTEM List of Tables .............................. x List of Figures ............................. xi 1 Introduction .............................. 1 1.1 Motivation.................................1 1.2 Problem Statement............................3 1.3 Contributions...............................3 1.4 Organization...............................4 2 Background and Context ....................... 6 2.1 Fundamental Concepts and Terminology................6 2.1.1 What is drowsiness?.......................6 2.1.2 What causes drowsiness?.....................7 2.1.3 What can we do about it?....................9 2.2 Drowsiness Detection and Measurement Methods...........9 2.2.1 Subjective Methods........................9 2.2.2 Physiological Methods...................... 14 2.2.3 Vehicle-Based Methods...................... 16 2.2.4 Behavioral Methods....................... 18 2.2.5 Hybrid Methods.......................... 20 2.3 Commercial Solutions........................... 21 2.3.1 Car Manufacturers........................ 21 vi 2.3.2 Independent Products...................... 25 3 Technologies, Algorithms and Research Aspects . 27 3.1 Imaging Sensors.............................. 28 3.1.1 Visible light cameras....................... 28 3.1.2 Near infrared (NIR) cameras................... 29 3.2 Feature detection and extraction..................... 30 3.3 Machine Learning Classifiers....................... 37 3.4 Challenges and practical aspects..................... 39 3.4.1 Data collection.......................... 39 3.4.2 Performance requirements.................... 41 4 Related Work .............................. 43 4.1 Head pose estimation........................... 43 4.2 Yawning.................................. 44 4.3 Eye state estimation........................... 46 5 First Prototype ............................. 50 5.1 System Initialization { Preparation................... 51 5.1.1 Skin Color Feature Analysis................... 53 5.1.2 Eye Model Analysis........................ 54 5.1.3 Head Position Analysis...................... 54 5.2 Regular Stage - Eye Tracking with Eye-State Analysis......... 55 5.3 Warning Stage - Nod Analysis...................... 56 5.3.1 Head Position Monitoring.................... 57 5.4 Alert Stage................................ 58 5.5 Preliminary Experiments......................... 58 5.5.1 Camera rotation test....................... 59 5.5.2 Head rotation test........................ 60 vii 5.5.3 "Real-World" Test........................ 61 5.5.4 Open vs. Closed Eyes test.................... 62 6 Android Implementation ....................... 64 6.1 Initialization Stage............................ 64 6.1.1 Algorithm outlook........................ 64 6.1.2 Record sequence......................... 66 6.1.3 Head position localization.................... 71 6.1.4 Skin color extraction and analysis................ 73 6.1.5 Manual confirmation with eye region extrapolation...... 74 6.1.6 Building eye model using SVM................. 78 6.2 Monitoring stage............................. 80 6.2.1 Algorithm outlook........................ 80 6.2.2 Direction and speed estimation................. 82 6.2.3 Tracking area extrapolation................... 84 6.2.4 Eye tracking............................ 85 6.2.5 Skin color confirmation...................... 90 6.2.6 Eye state analysis......................... 93 6.3 Warning stage............................... 94 6.3.1 Algorithm outlook........................ 94 6.3.2 Closed eyes monitoring...................... 95 6.3.3 Nod detection........................... 96 6.3.4 Distraction analysis........................ 97 6.4 Alert stage................................. 98 6.5 Experiments and results......................... 99 6.5.1 Synchronization Test....................... 99 6.5.2 Face detection speed comparison of original and custom detec- tion area.............................. 101 6.5.3 Eye region extrapolation limitations............... 102 viii 6.5.4 Tracking stage in proactive mode - speed test......... 103 6.5.5 Tracking stage in retroactive mode - speed test........ 104 7 Concluding Remarks . 106 Bibliography .............................. 109 ix LIST OF TABLES 2.1 Typical administration of MSLT test.................. 11 2.2 Stanford Sleepiness Scale......................... 12 2.3 Karolinska Sleepiness Scale verbal cues.................. 13 x LIST OF FIGURES 1.1 Proportion of road traffic deaths by age range..............2 2.1 Latency to sleep at 2-hour intervals across the 24-hour day.......7 2.2 The Lexus driver monitoring system uses six IR sensors (visible imme- diately in front of the instrument panel). Source: Wikimedia Commons. 22 2.3 Lexus driver monitoring system. Source: Wikimedia Commons.... 23 3.1 Typical face detection steps based on Haar-like features........ 32 3.2 Local Binary Pattern example....................... 34 3.3 Examples of eigenfaces........................... 35 3.4 Example of horizontal projection..................... 36 3.5 Concept of a Support Vector Machine. Source: Wikimedia Commons. 39 3.6 National Advanced Driving Simulator.................. 40 5.1 The four stages of our Drowsiness Detection System.......... 50 5.2 Eye detection algorithm: limitations due to horizontal angle change.. 51 5.3 Eye detection algorithm: limitations due to vertical angle change... 52 5.4 Successful initial face and eyes detection................. 52 5.5 Chroma-based skin detection comparison................. 54 5.6 Nod stages: eyes above upper threshold................. 57 5.7 Nodding detection method and its stages................. 58 5.8 Camera rotating: angle change limitations................ 60 5.9 Head rotating: angle change limitations................. 61 5.10 Real-World test results.......................... 62 xi 5.11 Support Vector Machine result examples: (a) & (b) correct; (c), (d) & (e) incorrect................................ 63 6.1 Top: four stages of our Driver Drowsiness Detection System. Bottom: outline of the initialization stage..................... 65 6.2 Bounding box of a detected object is defined with two sets of coordi- nates. En face detection of a driver's head................ 72 6.3 Captured face regions contain artifacts that can skew the results which can be minimized by masking most commonly affected areas...... 74 6.4 Head rotation angle example....................... 78 6.5 Top: four stages of our Driver Drowsiness Detection System. Bottom: outline of the proactive monitoring stage................. 80 6.6 Top: four stages of our Driver Drowsiness Detection System. Bottom: outline of the retroactive monitoring stage................ 81 6.7 Eight potential directions of head movement............... 83 6.8 Extrapolated search area depends on the speed and direction of the driver's head: area within the light blue rectangle is extrapolated... 85 6.9 Eye region extrapolation examples.................... 87 6.10 Extracted information from an image: face and eye region bounding boxes and their relation.........................