VISVESVARAYA TECHNOLOGICAL UNIVERSITY “Jnana Sangama”, Belagavi– 590 018

A PROJECT REPORT ON “MONITORING DRIVER'S ATTENTION LEVEL” Submitted in partial fulfillment for the award of the degree of

BACHELOR OF ENGINEERING IN COMPUTER SCIENCE AND ENGINEERING BY SISIR DAS K (1NH12CS118) SIDDHARTHKUMAR PATEL (1NH12CS117)

Under the guidance of

Ms.Pramilarani K (Senior Assistant Professor, Dept. of CSE, NHCE)

DEPARTMENT OF COMPUTER SCIENCE AND ENGINEERING NEW HORIZON COLLEGE OF ENGINEERING (ISO-9001:2000 certified, Accredited by NAAC ‘A’, Permanently affiliated to VTU) Outer Ring Road, Panathur Post, Near Marathalli, Bangalore – 560103 NEW HORIZON COLLEGE OF ENGINEERING (ISO-9001:2000 certified, Accredited by NAAC ‘A’ Permanently affiliated to VTU) Outer Ring Road, Panathur Post, Near Marathalli, Bangalore-560 103 DEPARTMENT OF COMPUTER SCIENCE AND ENGINEERING

CERTIFICATE

Certified that the project work entitled “MONITORING DRIVER’S ATTENTION LEVEL” carried out by SISIR DAS K (1NH12CS118) and SIDDHARTHKUMAR PATEL (1NH12CS117) bonafide students of NEW HORIZON COLLEGE OF ENGINEERING in partial fulfilment for the award of Bachelor Of Engineering in Computer Science and Engineering of the Visvesvaraya Technological University, Belgavi during the year 2015-2016. It is certified that all corrections/suggestions indicated for Internal Assessment have been incorporated in the report deposited in the department library. The project report has been approved as it satisfies the academic requirements in respect of Project work prescribed for the said Degree.

Name & Signature of Guide Name & Signature of HOD Signature of Principal (Ms. K Pramilarani) (Dr.Prashanth C.S.R.) (Dr.Manjunatha)

External Viva Name of Examiner Signature with date 1. 2. ACKNOWLEDGEMENT

The satisfaction and euphoria that accompany the successful completion of any task would be, but impossible without the mention of the people who made it possible, whose constant guidance and encouragement crowned our efforts with success.

We thank the management, Dr. Mohan Manghnani, Chairman of NEW HORIZON EDUCATIONAL INSTITUTIONS for providing necessary infrastructure and creating good environment.

We also record here the constant encouragement and facilities extended to us by Dr.Manjunatha, Principal, NHCE and Dr.Prashanth.C.S.R, Dean Academics, Head of the Department of Computer Science and Engineering. We extend our sincere gratitude to them.

We express our gratitude to Ms. K Pramilarani, our project guide for constantly monitoring the development of the project and setting up precise deadlines. Her valuable suggestions were the motivating factors in completing the work.

We would also like to express our gratitude to NHCE and to all our external guides at NHCE for their continuous guidance and motivation.

Finally a note of thanks to the teaching and non-teaching staff of Computer Science and Engineering Department for their cooperation extended to us and our friends, who helped us directly or indirectly in the course of the project work.

SISIR DAS K (1NH12CS118)

SIDDHARTHKUMAR PATEL (1NH12CS117)

i

ABSTRACT

There are many instances of road accidents round the world due to a driver’s lack of attention while driving. One of the prime reasons can be drowsiness. For this project, we have created a system that detects if a driver’s eyes are opened or closed using a cascade trained by us. This cascade is trained using a training tool that is present in the openCV library, it’s called ‘traincascade.exe’. An audible alert is triggered in the system to acknowledge if the eyes of the driver are closed for a certain amount of time.

The training is done using a tool from the openCV library. Since it is processing thousands of images it takes a lot of time, mostly hours to days depending on the power and capacity of the system. We have created an interface for the project that is coded in Javafx. The backend of the project is coded using C and C++ languages

ii

CONTENTS

ACKNOWLEDGEMENT I

ABSTRACT II

1. INTRODUCTION 1.1. PROBLEM STATEMENT 2 1.2. PROJECT FEATURES 2 1.3. PROJECT PURPOSE 3

2. LITERATURE SURVEY 2.1. IMAGE PROCESSING 4 2.2. EXISTING SYSTEM 6 2.3. PROPOSED SYSTEM 6 2.4. SOFTWARE DESCRIPTION 7

3. REQUIREMENT ANALYSIS 3.1. FUNCTIONAL REQUIREMENTS 13 3.2. NON FUNCTIONAL REQUIREMENTS 13 3.3. HARDWARE REQUIREMENTS 14 3.4. SOFTWARE REQUIREMENTS 14

4. CASCADE TRAINNING APPROACH 4.1. ENSEMBLE LEARNING 17 4.2. CASCADE CLASSIFIER 18 4.3. OPENCV 18 4.4. HAAR TRAINING 19 4.5. TEMPALTE MATHCING 21 4.6. SMOOTHENING OF THE CAPTURED IMAGE 21 4.7. HAAR LIKE FEATURES 22

iii

5. SYSTEM DESIGN 24

6. IMPLEMENTATION 6.1. VIDEO ACQUISITION 28 6.2. DIVIDING INTO FRAMES 28 6.3. FACE DETECTION 29 6.4. EYE DETECTION 29 6.5. SLEEPINESS DETECTION 30 6.6. CODE IMPLEMENTATION 31

7. TESTING 7.1. TEST CASE 1: WHEN THERE IS AMBIENT LIGHT. 37 7.2. TEST CASE 2: WHEN THE AUTOMOBILE DRIVER IS WEARING SPECTACLES. 38 7.3. TEST CASE 3 : WHEN THE AUTOMOBILE DRIVER’S HEAD IS TILTED. 39

8. SNAPSHOT 41

9. FUTURE WORK AND CONCLUSION 8.1 FUTURE WORK 44 8.2 CONCLUSION 44

BIBLIOGRAPHY 46

iv

LIST OF FIGURES

Fig. 4.1: A nonintrusive eye detection system. 17

Fig. 4.2: 3*3 pixel grid. 21

Fig. 4.3: 14 types of features 22

Fig. 5.1: Proposed System Architecture 24

Fig. 5.2: High Level System Flow. 26

Fig. 7.1: Ambient lighting 37

Fig. 7.2: Ambient lighting 38

Fig. 7.3: Driver with spectacles 38

Fig. 7.4: Driver with spectacles 39

Fig. 7.5: Tilted head 39

Fig. 7.6: Tilted head 40

Fig. 8.1 Snapshot 41

Fig. 8.2 Snapshot 41

Fig. 8.3 Snapshot 42

Fig. 8.4 Snapshot 42

Fig. 8.5 Snapshot 43

v

Monitoring driver’s attention level

CHAPTER 1

INTRODUCTION

A major share of road accidents that happen today are caused by the drowsiness of the driver. Many lives could be saved if there was a mechanism by which the drivers could be given an alert when they start feeling drowsy. Statistics indicate the need of a reliable driver drowsiness detection system which could alert the driver before a mishap happens. This project aims at developing such a system that can observe the mannerisms of the driver, obtain relevant data, process the information and produce the alert according to the situation at hand. Research has shown that there are basically three ways to determine drowsiness in a driver: vehicle-based measures, behavioural patterns, and physiological measures.

This project uses an eye detection system of the driver to determine whether the driver is awake or not. We check if the driver’s eyes are closed for some time to judge if the driver is sleeping or not .When sleepy, every person has a tendency to blink frequently or to shut their eyelids in slow movements. Another behavioural measure displayed by human beings when drowsy is frequent yawning. Also, for most people, the manner in which their head is kept changes abruptly as they fall asleep. That is, our head tends to swing down a few angles in an instant when feeling drowsy.

This project develops the software tools to detect and track only one of these behavioural patterns, that is to see if the eyes are closed for a longer than normal time. Most of the published studies on using behavioural approaches to determine drowsiness focus on blinking. This measurement has been found to be a reliable measure to predict sleepiness and has been used in several commercial products. However, it has been known to produce inaccurate results at times. This project puts forward a system that attempts to check whether the eyes are open or not.

Dept. of CSE, NHCE Page 1 Monitoring driver’s attention level

The system has been implemented using the input from a camera that is to be fixed in the car, in front of the driver. The camera records eyelid movements, after which this data is processed producing the corresponding output. Whenever the user closes his/her eyelids for more than a predefined period of time, the alert goes off. This is detected by comparing and matching the images used in the training phase of the system with the real time images captured by the camera. The comparison and matching is done after each frame, and the user is given an alert upon detection of closed eyelids in a certain number of continuous frames.

1.1 PROBLEM STATEMENT

Road accident is global tragedy with over-rising trend. India suffers from the highest number of deaths –around 1,50,000 in absolute terms annually-due to road accidents in the world owing to the poor infrastructure and dangerous driving habit. The increasing number of traffic accidents due to a diminished driver’s vigilance level has become a serious problem for the society. Statistics show that 20% of all the traffic accidents and up to one-quarter of fatal and serious accidents are due to drivers with diminished vigilance level.

Most motor vehicle crashes are caused by driver error (e.g. speeding) or poor operating practices including lack of seat belt use, distractions, fatigue, rash driving, and alcohol or drug use while driving. Most of the road users are quite well aware of the general rules and safety measures while using roads but it is only the laxity on part of road users, which cause accidents and crashes. Of all these issues, the most important issue to be considered is the driver’s concentration as well as driver’s inattention or distraction.

1.2 PROJECT FEATURES

This project, ‘Monitoring a driver’s attention level’, will be featuring a front end and a back end. The back end is coded using C and some functions and API’s of C++. The front end is coded using javafx and styled with CSS.

Dept. of CSE, NHCE Page 2 Monitoring driver’s attention level

The interface contains a launch button that calls a .exe file created by us using Visual Studio, this file launches the actual eye detection application. The eye detection application is coded such that it takes input from the web camera of a computer system and first detects the face, it then from the face region detects eyes, then from the eye region it uses a cascade trained by us to detect if the drivers eyes are closed or open. In case the eyes are closed for more than the allotted number of frames a notification in the form of a buzzer or an alarm will be activated to alert the driver.

1.3 PROJECT PURPOSE

This project aims at providing safety to the people driving motor vehicles. We here, have built a nonintrusive system that detects if a driver is sleeping or not. This is a simulation system that can be even used in actual vehicles. Our system is designed to alert a driver if he/she is sleeping. A buzzer or alarm is used to alert the driver in case he/she is found to be sleeping.

This application would know the driver is sleeping or not with his or her behavioural pattern. There are many behavioural patterns such as consistently blinking, yawning even the posture of the driver. What this system will be using is the blinking pattern to detect the driver’s drowsiness. If the driver’s eyes are shut for a certain amount of time then we conclude that the driver is sleeping and it’s time to awaken him or her. There is also an interface that is coded using javafx in intelliJ. IntelliJ is a java runtime environment, and javafx is a new interfacing language of java’s. More enhancements can be made for the interface like adding an option to choose the alarm or even an option to save images of the driver sleeping along with the date and time of that event.

Dept. of CSE, NHCE Page 3 Monitoring driver’s attention level

CHAPTER 2

LITERATURE SURVEY

Literature survey is the most important step in software development process. Before developing the tool it is necessary to determine the time factor, economy and company strength. Once these things are satisfied, then next steps is to determine which operating system and language can be used for developing the tool. Once the programmers start building the tool the programmers need lot of external support. This support can be obtained from senior programmers, from book or from websites. Before building the system the above consideration are taken into account for developing the proposed system.

2.1 IMAGE PROCESSING

Image processing is processing of images using mathematical operations by using any form of signal processing for which the input is an image, a series of images, or a video, such as a photograph or video frame; the output of image processing may be either an image or a set of characteristics or parameters related to the image. Most image-processing techniques involve treating the image as a 2D signals and applying standard signal-processing techniques to it. Images are also processed as 3D signals where the third-dimension being time or the z-axis.

Image processing usually refers to digital image processing, but optical and analog image processing also are possible. This article is about general techniques that apply to all of them. The acquisition of images (producing the input image in the first place) is referred to as imaging.

Closely related to image processing are computer graphics and computer vision. In computer graphics, images are manually made from physical models of objects, environments, and lighting, instead of being acquired (via imaging devices such as cameras) from natural scenes, as in most animated movies. Computer vision, on the other hand, is often considered high-level image processing out of

Dept. of CSE, NHCE Page 4 Monitoring driver’s attention level which a machine/computer/software intends to decipher the physical contents of an image or a sequence of images (e.g., videos or 3D full-body magnetic resonance scans).

In modern sciences and technologies, images also gain much broader scopes due to the ever growing importance of scientific visualization (of often large-scale complex scientific/experimental data). Examples include microarray data in genetic research, or real-time multi-asset portfolio trading in finance.

2.1.1 Purpose of Image processing The purpose of image processing is divided into 5 groups. They are:  Visualization - Observe the objects that are not visible.  Image sharpening and restoration - To create a better image.  Image retrieval - Seek for the image of interest.  Measurement of pattern – Measures various objects in an image.  Image Recognition – Distinguish the objects in an image.

2.1.2 Types The two types of methods used for Image Processing are:  Analog Image Processing: Analog or visual techniques of image processing can be used for the hard copies like printouts and photographs. Image analysts use various fundamentals of interpretation while using these visual techniques. The image processing is not just confined to area that has to be studied but on knowledge of analyst. Association is another important tool in image processing through visual techniques. So analysts apply a combination of personal knowledge and collateral data to image processing.  Digital Image Processing: Digital Processing techniques help in manipulation of the digital images by using computers. As raw data from imaging sensors from satellite platform contains deficiencies.

Dept. of CSE, NHCE Page 5 Monitoring driver’s attention level

To get over such flaws and to get originality of information, it has to undergo various phases of processing. The three general phases that all types of data have to undergo while using digital technique are Pre- processing, enhancement and display, information extraction.

2.2 EXISTING SYSTEM

Using technology to detect driver fatigue/drowsiness is an interesting challenge that would help in preventing accidents. In the past various efforts have been reported in the literature on approaches for drowsiness detection of automobile driver. In the last decade alone, many countries have begun to pay great attention to the automobile driver safety problem. Researchers have been working on the detection of automobile driver’s drowsiness using various techniques, such as physiological detection and Road monitoring techniques.

Physiological detection techniques take advantage of the fact that sleep rhythm of a person is strongly correlated with brain and heart activities. Road monitoring is one of the most commonly used technique, systems based on this approach, include Attention assist by Mercedes, Fatigue Detection System by Volkswagen, Driver Alert by Ford and Driver Alert Control by Volvo.

All the mentioned techniques monitor the road and driver behaviour characteristics to detect the drowsiness of the automobile driver. Few parameters used include whether the driver is following the lane rules, appropriate usage of the indicators etc., if there are aberrations in these parameters, above the tolerance level then the system concludes that the driver is drowsy.

2.2 PROPOSED SYSTEM

This project implements an interface that can be used to launch the eye detection system. The eye detection system is created in visual studio andwritten in C and C++.

Dept. of CSE, NHCE Page 6 Monitoring driver’s attention level

It detects the face of the person driving, now we have considered this face the region of interest and from the face we detect the eye region using a cascade present in the openCV library. The face is also detected using a cascade present in the openCV library. The detection of the driver’s open eyes is done by our trained cascade. So whenever driver blinks a red box is formed around the eyes. And if the driver’s eyes are shut for more than five frames an alert will be triggered and the system will notify the driver by raising a buzzer or alarm from the car’s sound system.

2.4 SOFTWARE DESCRIPTION

2.4.1 Visual Studio

Microsoft Visual Studio is an integrated development environment (IDE) from Microsoft. It is used to develop computer programs for Microsoft Windows, as well as web sites, web applications and web services. Visual Studio uses Microsoft software development platforms such as Windows API, Windows Forms, Windows Presentation Foundation, Windows Store and Microsoft Silverlight. It can produce both native code and managed code.

Visual Studio includes a code editor supporting IntelliSense (the code completion component) as well as code refactoring. The integrated debugger works both as a source-level debugger and a machine-level debugger. Other built-in tools include a forms designer for building GUI applications, web designer, class designer, and database schema designer. It accepts plug-ins that enhance the functionality at almost every level—including adding support for source-control systems (like Subversion) and adding new toolsets like editors and visual designers for domain-specific languages or toolsets for other aspects of the software development lifecycle (like the Team Foundation Server client: Team Explorer).

Visual Studio supports different programming languages and allows the code editor and debugger to support (to varying degrees) nearly any programming

Dept. of CSE, NHCE Page 7 Monitoring driver’s attention level language, provided a language-specific service exists. Built-in languages include C, C++ and C++ / CLI (via Visual C++), VB.NET (via Visual Basic .NET), C# (via Visual C#), and F# ( as of Visual Studio 2010). Support for other languages such as Python, Ruby, Node.js, and M among others is available via language services installed separately. It also supports XML/XSLT, HTML/XHTML, JavaScript and CSS. Java (and J#) were supported in the past.

Before Visual Studio 2015, commercial versions of Visual Studio were available for free to students via Microsoft's DreamSpark program, when only commercial versions supported plugins. Starting with Visual Studio 2015, Microsoft provides Community editions, which support plugins, at no cost to all users.

2.4.2 JavaFX

JavaFX is a software platform for creating and delivering desktop applications, as well as rich internet applications (RIAs) that can run across a wide variety of devices. JavaFX is intended to replace Swing as the standard GUI library for Java SE, but both will be included for the foreseeable future. JavaFX has support for desktop computers and web browsers on Microsoft Windows, Linux, and Mac OS X.

Before version 2.0 of JavaFX, developers used a statically typed, declarative language called JavaFX Script to build JavaFX applications. Because JavaFX Script was compiled to Java byte code, programmers could also use Java code instead. JavaFX applications could run on any desktop that could run Java SE, on any browser that could run Java EE, or on any mobile phone that could run Java ME.

JavaFX 2.0 and later is implemented as a native Java library, and applications using JavaFX are written in native Java code. JavaFX Script has been scrapped by Oracle, but development is being continued in the Visage project. JavaFX 2.x does not support the Solaris operating system or mobile phones;

Dept. of CSE, NHCE Page 8 Monitoring driver’s attention level however, Oracle plans to integrate JavaFX to Java SE Embedded 8, and Java FX for ARM processors is in developer preview phase.

On desktops, JavaFX supports Windows XP, Windows Vista, Windows 7, Windows 8, Windows 10, Mac OS X and Linux operating systems. Beginning with JavaFX 1.2, Oracle has released beta versions for OpenSolaris. On mobile, JavaFX Mobile 1.x is capable of running on multiple mobile operating systems, including Symbian OS, Windows Mobile, and proprietary real-time operating systems.

Written as a Java API, JavaFX application code can reference APIs from any Java library. For example, JavaFX applications can use Java API libraries to access native system capabilities and connect to server-based middleware applications. The look and feel of JavaFX applications can be customized. Cascading Style Sheets (CSS) separate appearance and style from implementation so that developers can concentrate on coding.

Graphic designers can easily customize the appearance and style of the application through the CSS. If you have a web design background, or if you would like to separate the user interface (UI) and the back-end logic, then you can develop the presentation aspects of the UI in the FXML scripting language and use Java code for the application logic.

If you prefer to design UIs without writing code, then use JavaFX Scene Builder. As you design the UI, Scene Builder creates FXML markup that can be ported to an Integrated Development Environment (IDE) so that developers can add the business logic.

2.4.3 C language

C is a general-purpose, imperative computer programming language, supporting structured programming, lexical variable scope and recursion, while a static type system prevents many unintended operations. By design, C provides

Dept. of CSE, NHCE Page 9 Monitoring driver’s attention level constructs that map efficiently to typical machine instructions, and therefore it has found lasting use in applications that had formerly been coded in assembly language, including operating systems, as well as various application software for computers ranging from supercomputers to embedded systems.

C is an imperative (procedural) language. It was designed to be compiled using a relatively straightforward compiler, to provide low-level access to memory, to provide language constructs that map efficiently to machine instructions, and to require minimal run-time support.

C was therefore useful for many applications that had formerly been coded in assembly language, as in system programming. Despite its low-level capabilities, the language was designed to encourage cross-platform programming. A standards- compliant and portably written C program can be compiled for a very wide variety of computer platforms and operating systems with few changes to its source code.

The language has become available on a very wide range of platforms, from embedded microcontrollers to supercomputers. Like most imperative languages in the ALGOL tradition, C has facilities for structured programming and allows lexical variable scope and recursion, while a static type system prevents many unintended operations.

In C, all executable code is contained within subroutines, which are called "functions" (although not in the strict sense of functional programming). Function parameters are always passed by value. Pass-by-reference is simulated in C by explicitly passing pointer values. C program source text is free-format, using the semicolon as a statement terminator and curly braces for grouping blocks of statements.

The origin of C is closely tied to the development of the Unix operating system, originally implemented in assembly language on a PDP-7 by Ritchie and Thompson, incorporating several ideas from colleagues. Eventually, they decided to port the operating system to a PDP-11. The original PDP-11 version of Unix was

Dept. of CSE, NHCE Page 10 Monitoring driver’s attention level developed in assembly language. The developers were considering rewriting the system using the B language, Thompson's simplified version of BCPL. However B's inability to take advantage of some of the PDP-11's features, notably byte addressability, led to C.

The development of C started in 1972 on the PDP-11 Unix system and first appeared in Version 2 Unix. The language was not initially designed with portability in mind, but soon ran on different platforms as well: a compiler for the Honeywell 6000 was written within the first year of C's history, while an IBM System/370 port followed soon. The name of C simply continued the alphabetic order started by B.

2.4.4 Cascading Style Sheets

Cascading Style Sheets (CSS) is a style sheet language used for describing the presentation of a document written in a mark-up language. Although most often used to set the visual style of web pages and user interfaces written in HTML and XHTML, the language can be applied to any XML document, including plain XML, SVG and XUL, and is applicable to rendering in speech, or on other media. Along with HTML and JavaScript, CSS is a cornerstone technology used by most websites to create visually engaging webpages, user interfaces for web applications, and user interfaces for many mobile applications.

CSS is designed primarily to enable the separation of document content from document presentation, including aspects such as the layout, colors, and fonts. This separation can improve content accessibility, provide more flexibility and control in the specification of presentation characteristics, enable multiple HTML pages to share formatting by specifying the relevant CSS in a separate .css file, and reduce complexity and repetition in the structural content.

This separation of formatting and content makes it possible to present the same markup page in different styles for different rendering methods, such as on- screen, in print, by voice (when read out by a speech-based browser or screen

Dept. of CSE, NHCE Page 11 Monitoring driver’s attention level reader) and on Braille-based, tactile devices. It can also be used to display the web page differently depending on the screen size or device on which it is being viewed.

Readers can also specify a different style sheet, such as a CSS file stored on their own computer, to override the one the author has specified. Changes to the graphic design of a document (or hundreds of documents) can be applied quickly and easily, by editing a few lines in the CSS file they use, rather than by changing markup in the documents. The CSS specification describes a priority scheme to determine which style rules apply if more than one rule matches against a particular element.

In this so-called cascade, priorities (or weights) are calculated and assigned to rules, so that the results are predictable. The CSS specifications are maintained by the World Wide Web Consortium (W3C). Internet media type (MIME type) is registered for use with CSS by RFC 2318(March 1998). The W3C operates a free CSS validation service for CSS documents.

2.4.5 IntelliJ IDEA

IntelliJ IDEA is a Java integrated development environment (IDE) for developing computer software. It is developed by JetBrains (formerly known as IntelliJ), and is available as an Apache 2 Licensed community edition, and in a proprietary commercial edition. Both can be used for commercial development. The first version of IntelliJ IDEA was released in January 2001, and was one of the first available Java IDEs with advanced code navigation and code refactoring capabilities integrated.

In a 2010 Infoworld report, IntelliJ received the highest test center score out of the four top Java programming tools: Eclipse, IntelliJ IDEA, NetBeans and JDeveloper. In December 2014, Google announced version 1.0 of Android Studio, an open source IDE for Android apps, based on the open source community edition of IntelliJ IDEA. Other development environments based on IntelliJ's framework includes AppCode, CLion, PhpStorm, PyCharm, RubyMine, WebStorm, and MPS.

Dept. of CSE, NHCE Page 12 Monitoring driver’s attention level

CHAPTER 3

REQUIREMENTS ANALYSIS

The requirements for an effective eye detection system are as follows:

 A non-intrusive monitoring system that will not distract the driver.  A real-time monitoring system, to insure accuracy in detecting drowsiness.  A system that will work in both daytime and night conditions.

3.1 FUNCTIONAL REQUIREMENTS

In software engineering, a functional requirement defines a function of a software system or its component. A function is described as a set of inputs, the behaviour, and outputs. Functional requirements may be calculations, technical details, data manipulation and processing and other specific functionality that define what a system is supposed to accomplish. Behavioural requirements describing all the cases where the system uses the functional requirements are captured in use cases.

Here, the system has to perform the following tasks:

 Take input from a web cam  Process one frame at a time  Apply the detector to each frame  Buzz an alarm if eyes are found to be closed in five consecutive frames 3.2 NON-FUNCTIONAL REQUIREMENTS

In systems engineering and requirements engineering, a non-functional requirement is a requirement that specifies criteria that can be used to judge the operation of a system, rather than specific behaviours. This should be contrasted

Dept. of CSE, NHCE Page 13 Monitoring driver’s attention level with functional requirements that define specific behaviour or functions. The plan for implementing functional requirements is detailed in the system design. The plan for implementing non-functional requirements is detailed in the system architecture. Other terms for non-functional requirements are "constraints", "quality attributes", "quality goals", "quality of service requirements" and "non-behavioural requirements".

3.3 HARDWARE USED

 Input device : 1. Web Camera (To capture real-time video) 2. Standard Keyboard 3. Mouse  Output device : 1. Screen (Monitor). 2. Sound system (Speakers)

 A computing system: 1. Processor : Any Processor above 1.8 GHz 2. RAM : 4GB 3. Hard Disk : 100 GB  An infrared light source (For illumination)

3.4 SOFTWARE USED

 Operating System: Windows 10  Back end: Microsoft Visual Studio 2015 Library used: OpenCV (open computer vision) library Language used: C, C++.  Front end: IntelliJ IDEA 2016.1.1 Language used: JavaFX, CSS (for styling).

Dept. of CSE, NHCE Page 14 Monitoring driver’s attention level

 GIMP: GIMP was used to scale all 10,000 images to one size.

Dept. of CSE, NHCE Page 15 Monitoring driver’s attention level

CHAPTER 4

CASCADE TRAINING APPROACH

There are many approaches that can be used to create a system that detects the fatigue of a driver. Some of them are:

 Train a classifier to detect closed eyes.  Train a classifier to detect open eyes.  Use Hough transform to detect the iris.  Blink patterns.  Skin detection in eye area.

What we are using is training a cascade to detect an open eye. Here, training is done using the opencv_traincascade.exe tool available in the OpenCV library.Data for training has to be prepared prior to running the traincascade.exe tool.Training data includes two sets of images – Positive and Negative.Positive images are images which contains the object of interest.Negative images are images that do not contain the required object.

For this project, the positive images are images of open eyes and negative images are images of closed eyes along with a wide variety of various backgrounds.A .vec file is created using a text file that contains the paths of all the positive images along with the co-ordinates and dimensions of the ROI (Region of Interest) in the positive images.Another text file is created which contains paths of all the negative images.These files are then fed to the opencv_traincascade.exe tool to train the classifier.The result is an .xml file that is capable of detecting the object that it is trained to detect.

The accuracy of the detector depends on the quantity, quality and the variety of training data (positive and negative images) used in the training process.The

Dept. of CSE, NHCE Page 16 Monitoring driver’s attention level training process can take from several hours to several days depending on the computer used for the training and the amount of training data.

Fig. 4.1: A nonintrusive eye detection system.

4.1 ENSEMBLE LEARNING

Ensemble learning is a machine learning process by which multiple models, such as classifiers or experts, are strategically generated and combined to solve a particular computational intelligence problem. Ensemble learning is primarily used when improvement in the performance of the system is needed, or to minimize the probability of producing poor output. The whole process involves classification, prediction, function approximation, etc. Other applications of ensemble learning include assigning a confidence to the decision made by the model, selecting optimal (or near optimal) features, data fusion, incremental learning, non-stationary learning and error-correcting. This project focuses on cascade classifier of ensemble

Dept. of CSE, NHCE Page 17 Monitoring driver’s attention level learning, which consists of various stages, each stage of which is a collection of weak learners.

4.2 CASCADE CLASSIFIER

The cascade classifier is a part of ensemble learning, which is implemented in various stages that consists of simpler classifiers. These are applied to a region of interest until input is accepted or rejected. First, a classifier (namely a cascade of boosted classifiers working with haar-like features) is trained using a few hundred sample views of a particular object (i.e., a face or eyes in this case), called positive samples, and negative samples, which are arbitrary images. Every positive and negative sample is scaled to the same size.

After a classifier is trained, it can be applied to a region of interest in an input image. This region should be of the same size as the samples used during the training. The output of the classifier will be “1” if the region is likely to show the object that we are looking for (i.e., face/eyes), and “0” otherwise.

It is also possible to search for the object in the whole image by moving the search window over the image and checking every location using the classifier. The classifier is designed so that it can be easily “resized” in order to be able to find the objects of interest at different sizes. This procedure is found to be more efficient than resizing the image itself. So, to find an object of an unknown size in the image the scan procedure should be done several times at different scales.

4.3 OPENCV

OpenCV (Open Source Computer Vision) is a cross-platform library of programming functions mainly aimed at real time computer vision. It is free to use, and provides an array of functions for machine learning, neural networks, and other applications of artificial intelligence. To support machine learning, OpenCV provides a statistical machine learning library that contains boosting, decision tree

Dept. of CSE, NHCE Page 18 Monitoring driver’s attention level learning, Artificial Neural Networks (ANN), Support Vector Machine (SVM), ensemble learning, etc.

It is written in C++ and the primary interface is in C++ as well. There are bindings in Java, Python, and MATLAB/OCTAVE. To encourage adoption by a wider audience, wrappers in other languages such as C#, C, Perl, and Ruby have been developed. All of the new developments and algorithms in OpenCV are now developed in the C++ interface.

OpenCV is released under a BSD license and hence it’s free for both academic and commercial use. It has C++, C, Python and Java interfaces and supports Windows, Linux, Mac OS, iOS and Android. OpenCV was designed for computational efficiency and with a strong focus on real-time applications. Written in optimized C/C++, the library can take advantage of multi-core processing. Enabled with OpenCL, it can take advantage of the hardware acceleration of the underlying heterogeneous compute platform. Adopted all around the world, OpenCV has more than 47 thousand people of user community and estimated number of downloads exceeding 9 million. Usage ranges from interactive art, to mines inspection, stitching maps on the web or through advanced robotics.

4.4 HAAR TRAINING

The OpenCV library provides a great number of functions for face detection, feature detection such as eyes, mouth, sunglasses, etc. Furthermore, it provides programs (or functions) that are used to train classifiers for their face detection system, called Haar Training. We can create our own object classifiers using these functions. Object Detection using Haar feature-based cascade classifiers is a machine learning based approach where a cascade function is trained from a lot of positive and negative images. It is then used to detect objects in other images.

The algorithm extracts images using a lot of positive (faces, eyes, etc.) and negative images (arbitrary images without faces). Each feature is a single value obtained by subtracting sum of pixels under various regions of the images. Features

Dept. of CSE, NHCE Page 19 Monitoring driver’s attention level include edge features, line features, etc. For each feature, the pixels used for extraction differs.

Now all possible sizes and locations of each image kernel are used to calculate plenty of features. But all the features extracted will not be useful for the purpose at hand. For example, in this project only eyes, mouth, and the face are relevant. Other facial features like cheeks, nose, forehead, etc. are useless as far this project is concerned.

To extract only those features relevant to the project, we use a technique called Adaboost. For this, we apply each and every feature on all the training images. For each feature, it finds the best threshold which will classify the faces to positive and negative. Then we select the features that display the least error rate (best threshold). Initially each feature is given an equal weight. As the process continues, the weights are updated according to the results obtained so that accuracy is improved. The process is repeated until the required level of accuracy or the required number of features is found.

Final classifier is a weighted sum of these weak classifiers. It is called weak because it alone can't classify the image, but together with others forms a strong classifier. With around 6000 images, even 200 features can provide 95% accuracy. In an image, most of the image region is non-face region. So it is a better idea to have a simple method to check if a window is not a face region. If it is not, discard it in a single shot. Don't process it again.

Instead focus on region where there can be a face. This way, we can find more time to check a possible face region. For this they introduced the concept of Cascade of Classifiers. Instead of applying all the 6000 features on a window, group the features into different stages of classifiers and apply one-by-one. (Normally first few stages will contain very less number of features).

If a window fails the first stage, discard it. We don't consider remaining features on it. If it passes, apply the second stage of features and continue the

Dept. of CSE, NHCE Page 20 Monitoring driver’s attention level process. The window which passes all stages is a face region. This is how face detection and other feature detection is done using haar training by means of cascade classifiers.

4.5 TEMPLATE MATCHING

Template Matching is a method used for finding whether a template image is present in a larger image or not. OpenCV comes with a function matchTemplate() for this purpose. To do that, the function simply slides the template image over the larger image and compares the template and patch of input image under the template image.

There are several comparison methods available in OpenCV. The function returns a grey scale image as the output, which will tell you how much of the template image matches with the input image. Once you got the result, you can use minMaxLoc() function to find where the maximum/ minimum value is.

4.6 SMOOTHENING OF THE CAPTURED IMAGE

First, an image of the driver is captured via a webcam installed in front of the driver’s face. Infrared cameras can be used to minimize illumination problems. An average filter is used to minimize the illumination changes in the image. It is a low pass filter.

P1 P2 P3

P4 P5 P6

P7 P8 P9

Fig. 4.2: 3*3 pixel grid.

Dept. of CSE, NHCE Page 21 Monitoring driver’s attention level

It takes the average of all the neighbouring pixels and compute its sum and then dividing it by the number of pixels in the window. One such window can be seen in Fig 4.2. Filtered pixel (p) = (p1 + p2 + p3 + p4 ... + p9) / 9. An average filter is used to smooth out the images from camera. The filter used is of 3*3 size. The use of average filter minimises the illumination changes in images. In RGB image format each channel is considered separately for smoothing.

4.7 HAAR LIKE FEATURES

Haar like features was first developed by Viola and Jones. The need for an efficient algorithm for eye detection is very high. Haar like features are used to detect face and eye of the driver. A Haar like feature is a classifier which is trained with a few hundreds of positive and negative examples that are scaled to the same size. Here positive examples refer to the images of an object which is to be detected and negative examples indicate images of anything other than the desired object. A feature based system works faster than a pixel based system. The Haar features are shown in Fig 4.3.

Fig. 4.3: 14 types of features

Dept. of CSE, NHCE Page 22 Monitoring driver’s attention level

There are three main parts in which Haar like features work:

 Integral Image  Training the classifier and  Cascading.

An integral image can be obtained by computing the sum of two pixels above and to the left of the original pixel. The idea is to make a rectangle which is used as a feature for the comparison. There are 14 types of features. The rectangle shape is divided into two areas like black and white. First take the sum of the black and white individually. Then subtract the sum of white area from the black area. The task of training the classifier includes the features selected by AdaBoost. AdaBoost is a learning algorithm which uses weak features for classification to form a strong classifier. The weak features are reweighted to correct the previous features.

Dept. of CSE, NHCE Page 23 Monitoring driver’s attention level

CHAPTER 5

SYSTEM DESIGN

The system architecture of the proposed system is represented in Fig 5.1.

Haar Samples for Face Dividing

face detection into frames Camera detection

Haar cascade Drowsiness detection samples for Eyes iris detection detection

Alarm

Fig. 5.1: Proposed System Architecture

Fig 3showcases the various important blocks in the proposed system and their high level interaction. It can be seen the system consists of 5 different modules namely:

 Video Acquisition  Dividing into Frames  Face Detection  Eye Detection

Dept. of CSE, NHCE Page 24 Monitoring driver’s attention level

 Sleepiness Detection

In addition to these there are two external typically Hardware components namely, camera for video acquisition and an audio alarm. The functionality of each these modules in the system can be described as follows:

 Video acquisition: Video acquisition mainly involves obtaining the live video feed of the automobile driver. Video acquisition is achieved, by making use of a camera.  Dividing into frames: This module is used to take live video as its input and convert it into a series of frames/ images, which arethen processed.  Face detection: The face detection function takes one frame at a time from the frames provided by the frame grabber, and in each and every frame it tries to detect the face of the automobile driver. This is achieved by making use of a set of pre-defined Haarcascade samples.  Eyes detection: Once the face detection function has detected the face of the automobile driver, the eyes detection function tries to detect the automobile driver's eyes. This is achieved by making use of a set of pre-defined Haarcascade samples.  Sleepiness detection: After detecting the eyes of the automobile driver, the drowsiness detection function detects if the automobile driver is drowsy or not, by taking into consideration the state of the eyes, that is, open or closed and the blink rate.

As the proposed system makes use of OpenCV libraries, there is no necessary minimum resolution requirement on the camera. The schematic representation of the algorithm of the proposed system is depicted in Fig 4. In the proposed algorithm, first video acquisition is achieved by making use of an external camera placed in front of the automobile driver.

Dept. of CSE, NHCE Page 25 Monitoring driver’s attention level

Video Acquisition

Dividing into frames

Face detected for individual frames

Is no. of NO face>0?

YES

Set the image ROI to a fixed area on the face

Detect eyes from ROI

Determine open/closed state of eyes

Eyes YES K=0 open?

NO

K=K+1

NO YES

Blink If K>5 Sleepy

Fig. 5.2: High Level System Flow.

Dept. of CSE, NHCE Page 26 Monitoring driver’s attention level

The acquired video is then converted into a series of frames/images. The next step is to detect the automobile driver's face, in each and every frame extracted from the video.

For detecting the face, since the camera is focused on the automobile driver, we can avoid processing the image at the corners thus reducing a significant amount of processing required. Once the region of interest is defined face has been detected, the region of interest is now the face, as the next step involves detecting eyes. To detect the eyes, instead of processing the entire face region, we mark a region of interest within the face region which further helps in achieving the primary goal of the proposed system.

Next we make use of Haarcascade Xml file constructed for eye detection, and detect the eyes by processing only the region of interest. Once the eyes have been detected, the next step is to determine whether the eyes are in open/closed state, which is achieved by extracting and examining the pixel values from the eye region.

If the eyes are detected to be open, no action is taken. But, if eyes are detected to be closed continuously for two seconds according to, that is a particular number of frames depending on the frame rate, then it means that the automobile driver is feeling drowsy and a sound alarm is triggered. However, if the closed states of the eyes are not continuous, then it is declared as a blink.

Dept. of CSE, NHCE Page 27 Monitoring driver’s attention level

CHAPTER 6

IMPLEMENTATION

The implementation details of each the modules can be explained as follows:

6.1 VIDEO ACQUISITION

OpenCV provides extensive support for acquiring and processing live videos. It is also possible to choose whether the video has to be captured from the in-built webcam or an external camera by setting the right parameters. As mentioned earlier, OpenCV does not specify any minimum requirement on the camera, however OpenCV by default expects a particular resolution of the video that is being recorded, if the resolutions do not match, then an error is thrown. This error can be countered, by over-riding the default value, which can be achieved, by manually specifying the resolution of the video being recorded.

6.2 DIVIDING INTO FRAMES

Once the video has been acquired, the next step is to divide it into a series of frames/images. This was initially done as a 2 step process. The first step is to grab a frame from the camera or a video file, in our case since the video is not stored, the frame is grabbed from the camera and once this is achieved, the next step is to retrieve the grabbed frame. While retrieving, the image/frame is first decompressed and then retrieved. However, the two step process took a lot of processing time as the grabbed frame had to be stored temporarily. To overcome this problem, we came up with a single step process, where a single function grabs a frame and returns it by decompressing.

Dept. of CSE, NHCE Page 28 Monitoring driver’s attention level

6.3 FACE DETECTION

Once the frames are successfully extracted the next step is to detect the face in each of these frames. This is achieved by making use of the Haarcascade file for face detection. The Haarcascade file contains a number of features of the face, such as height, width and thresholds of face colours, it is constructed by using a number of positive and negative samples. For face detection, we first load the cascade file. Then pass the acquired frame to an edge detection function, which detects all the possible objects of different sizes in the frame.

To reduce the amount of processing, instead of detecting objects of all possible sizes, since the face of the automobile driver occupies a large part of the image, we can specify the edge detector to detect only objects of a particular size, this size is decided based on the Haarcascade file, wherein each Haarcascade file will be designed for a particular size. Now, the output the edge detector is stored in an array. Now, the output of the edge detector is then compared with the cascade file to identify the face in the frame.

Since the cascade consists of both positive and negative samples, it is required to specify the number of failures on which an object detected should be classified as a negative sample. In our system, we set this value to 3, which helped in achieving both accuracy as well as less processing time. The output of this module is a frame with face detected in it.

6.4 EYE DETECTION

After detecting the face, the next step is to detect the eyes, this can be achieved by making use of the same technique used for face detection. However, to reduce the amount of processing, we mark the region of interest before trying to detect eyes. The region of interest is set by taking into account the following:

 The eyes are present only in the upper part of the face detected.  The eyes are present a few pixels lower from the top edge of the face.

Dept. of CSE, NHCE Page 29 Monitoring driver’s attention level

Once the region of interest is marked, the edge detection technique is applied only on the region of interest, thus reducing the amount of processing significantly. Now, we make use of the same technique as face detection for detecting the eyes by making use of Haarcascade .xml file for eyes detection. But, the output obtained was not very efficient, there were more than two objects classified as positive samples, indicating more than two eyes. To overcome this problem, the following steps are taken:

 Out of the detected objects, the object which has the highest surface area is obtained. This is considered as the first positive sample.  Out of the remaining objects, the object with the highest surface area is determined. This is considered as the second positive sample.  A check is made to make sure that the two positive samples are not the same.  Now, we check if the two positive samples have a minimum of 30 pixels from either of the edges.  Next, we check if the two positive samples have a minimum of 20 pixels apart from each other.

After passing the above tests, we conclude that the two objects i.e., positive sample 1 and positive sample 2, are the eyes of the automobile driver.

6.5 SLEEPINESS DETECTION

Once the eyes are detected, the next step is to determine if the eyes are in closed or opened state. This is achieved by extracting the pixel values from the eye region. After extracting, we check if these pixel values are white, if they are white then it infers that the eyes are in the open state, if the pixel values are not white then it infers that the eyes are in the closed state. This is done for each and every frame extracted. If the eyes are detected to be closed for two seconds or a certain number of consecutive frames depending on the frame rate, then the automobile driver is detected to be drowsy.

Dept. of CSE, NHCE Page 30 Monitoring driver’s attention level

If the eyes are detected to be closed in non-consecutive frames, then we declare it as a blink. If drowsiness is detected, a text message is displayed along with triggering an audio alarm. But, it was observed that the system was not able to run for an extended period of time, because the conversion of the acquired video from RGB to grayscale was occupying too much memory. To overcome this problem, instead of converting the video to grayscale, the RGB video only was used for processing. This conversion resulted in the following advantages,

 Better differentiation between colours, as it uses multichannel colours.  Consumes very less memory.  Capable of achieving blink detection, even when the automobile driver is wearing spectacles.

Hence there were two versions of the system that was implemented; the version 1.0 involves the conversion of the image to grayscale. Currently version 2.0 makes use of the RGB video for processing.

6.6 CODE IMPLEMENTATION

6.6.1 Back End

The first step is to load the cascades.

//This cascade file comes with the downloaded OpenCV library if (!face_cascade.load("c:\\haar\\haarcascade_frontalface_alt2.xml")) { printf("Error loading cascade file for face"); return 1; } //This cascade file comes with the downloaded OpenCV library if (!eye_cascade.load("c:\\haar\\haarcascade_eye.xml"))

Dept. of CSE, NHCE Page 31 Monitoring driver’s attention level

{ printf("Error loading cascade file for eyes"); return 1; } //This casdcade file is trained by us. Number of samples used : 10000 if (!pupil_cascade.load("c:\\haar\\cascade.xml")) { printf("Error loading cascade file for pupil"); return 1; } The next step is to take input in the form of a video from a web cam.

VideoCapture capture(0); //Settign the resolution of captured frames. capture.set(CV_CAP_PROP_FRAME_WIDTH, 800); capture.set(CV_CAP_PROP_FRAME_HEIGHT, 600); //Check if webcam was successfully opened if (!capture.isOpened()) { printf("Error in initializing the camera\n"); return 1; } Extracting frames, smoothening the image to reduce noise, applying the detectors and then playing the alarm to alert the driver is done using the following code. capture.read(cap_img); //Converting color image to grayscale in order to reduce the processing time.

Dept. of CSE, NHCE Page 32 Monitoring driver’s attention level cvtColor(cap_img, gray_img, CV_BGR2GRAY); //Equalizing the histogram to reduce noise. cv::equalizeHist(gray_img, gray_img); //Running the face detector first to reduce the search are for eyes. face_cascade.detectMultiScale(gray_img, faces, 1.1, 10, CV_HAAR_SCALE_IMAGE | CV_HAAR_DO_CANNY_PRUNING, Size(0, 0), Size(300, 300)); for (int i = 0; i < faces.size(); i++) { Point pt1(faces[i].x, faces[i].y); Point pt2(faces[i].x + faces[i].width, faces[i].y + faces[i].height); //Extracting the captured face or the region of interest. Mat faceROI = gray_img(faces[i]); //Display the detected face. imshow("face", faceROI); //Running the eye detector on the captured face to reduce the search are for open/close eye detector. eye_cascade.detectMultiScale(faceROI, eyes, 1.1, 2, CV_HAAR_SCALE_IMAGE, Size(30, 30)); for (size_t j = 0; j< eyes.size(); j++) { flag = 0; Point pt3(faces[i].x + eyes[j].x + eyes[j].width, faces[i].y + eyes[j].y + eyes[j].height); Point pt4(faces[i].x + eyes[j].x, faces[i].y + eyes[j].y); //Settign the next region of interest Mat eyeROI = faceROI(eyes[0]); imshow("eyes1", eyeROI);

Dept. of CSE, NHCE Page 33 Monitoring driver’s attention level

//Running the open/close eye detector on the region of interest. pupil_cascade.detectMultiScale(eyeROI, pupil, 1.1, 2, CV_HAAR_SCALE_IMAGE, Size(30, 30)); for (size_t k = 0; k < pupil.size(); k++) { Point pt5(faces[i].x + eyes[j].x + pupil[k].x + pupil[k].width / 2, faces[i].y + eyes[j].y + pupil[k].y); Point pt6(faces[i].x + eyes[j].x + pupil[k].x + pupil[k].width / 2, faces[i].y + eyes[j].y + pupil[k].y + pupil[k].height); Point pt7(faces[i].x + eyes[j].x + pupil[k].x, faces[i].y + eyes[j].y + pupil[k].y + pupil[k].height / 2); Point pt8(faces[i].x + eyes[j].x + pupil[k].x + pupil[k].width, faces[i].y + eyes[j].y + pupil[k].y + pupil[k].height / 2); Mat pupilROI = eyeROI(pupil[k]); imshow("pupil", pupilROI); line(cap_img, pt5, pt6, cvScalar(255, 255, 255), 1, 8, 0); line(cap_img, pt7, pt8, cvScalar(255, 255, 255), 1, 8, 0); flag = 1; } if (flag == 1) { cc = 0; rectangle(cap_img, pt3, pt4, cvScalar(255, 255, 255), 1, 8, 0); } else { cc++; rectangle(cap_img, pt3, pt4, cvScalar(0, 0, 255), 2, 8, 0);

Dept. of CSE, NHCE Page 34 Monitoring driver’s attention level if (cc > 3) { PlaySound(TEXT("c:\\haar\\alarm.wav"), NULL, SND_FILENAME); cc = 0; }}} rectangle(cap_img, pt1, pt2, cvScalar(255, 255, 255), 2, 8, 0); } 6.6.2 Front End

Creation of Labels.

Label label0 = new Label(); label0.setText("Monitoring a driver's attention level"); label0.getStyleClass().add("label-top"); GridPane.setConstraints(label0,22,0); GridPane.setHalignment(label0,HPos.CENTER); GridPane.setValignment(label0,VPos.CENTER); Creation of buttons and invoking of an external executable file.

Button button1 = new Button(); button1.setText("Launch"); button1.setOnAction(e -> { File pro = new File("C:\\Users\\Siddharth\\Documents\\Visual Studio 2015\\Projects\\eye\\x64\\Debug\\eye.exe"); try { Desktop.getDesktop().open(pro); } catch (IOException ex)

Dept. of CSE, NHCE Page 35 Monitoring driver’s attention level

{ ex.printStackTrace(); } } ); GridPane.setConstraints(button1,22,9); GridPane.setHalignment(button1, HPos.CENTER); GridPane.setValignment(button1, VPos.CENTER); Setting of the layout and displaying the layout in a window. grid.getChildren().addAll(label0,label1, label2, label3, button1, button2, button3,button4); Scene scene = new Scene(grid,1200,500); scene.getStylesheets().add("eye.css"); window.setScene(scene); window.show(); Creation of alert boxes with modality. Stage window = new Stage(); window.initModality(Modality.APPLICATION_MODAL); window.setTitle("About"); window.setMinWidth(500); window.setMinHeight(300); VBox layout = new VBox(); layout.getChildren().addAll(label, closeButton); layout.setAlignment(Pos.CENTER); Scene scene = new Scene(layout); window.setScene(scene); window.showAndWait();

Dept. of CSE, NHCE Page 36 Monitoring driver’s attention level

CHAPTER 7

TESTING

The tests were conducted in various conditions including:

1. Different lighting conditions.

2. Drivers posture and position of the automobile drivers face.

3. Drivers with spectacles.

7.1 TEST CASE 1: When there is ambient light.

Fig. 7.1: Ambient lighting

Dept. of CSE, NHCE Page 37 Monitoring driver’s attention level

Fig. 7.2: Ambient lighting

RESULT: As shown in Fig 7.1 and 7.2, when there is ambient amount of light, the automobile driver's face and eyes are successfully detected.

7.2 TEST CASE 2: When the automobile driver is wearing spectacles.

Fig. 7.3: Driver with spectacles

Dept. of CSE, NHCE Page 38 Monitoring driver’s attention level

Fig. 7.4: Driver with spectacles

RESULT: As shown in screen snapshot in Fig 7.3 and 7.4, When the automobile driver is wearing spectacles, the face, eyes, eye blinks, and drowsiness was successfully detected.

7.3 TEST CASE 3: When the automobile driver’s head is tilted.

Fig. 7.5: Tilted head

Dept. of CSE, NHCE Page 39 Monitoring driver’s attention level

Fig. 7.6: Tilted head

RESULT: As shown in screen snapshot in Fig 7.5 and 7.6 when the automobile driver's face is tilted for more than 30 degrees from vertical plane, it was observed that the detection of face and eyes failed. The system was extensively tested even in real world scenarios, this was achieved by placing the camera on the visor of the car, focusing on the automobile driver. It was found that the system gave positive output unless there was any direct light falling on the camera.

Dept. of CSE, NHCE Page 40 Monitoring driver’s attention level

CHAPTER 8

SNAPSHOTS

Fig. 8.1

Fig. 8.2

Dept. of CSE, NHCE Page 41 Monitoring driver’s attention level

Fig. 8.3

Fig. 8.4

Dept. of CSE, NHCE Page 42 Monitoring driver’s attention level

Fig. 8.5

Dept. of CSE, NHCE Page 43 Monitoring driver’s attention level

CHAPTER 9

FUTURE WORK AND CONCLUSION

9.1 FUTURE WORK

The system at this stage is a “Proof of Concept” for a much substantial endeavour. This will serve as a first step towards a distinguished technology that can bring about an evolution aimed at ace development. The developed system has special emphasis on real-time monitoring with flexibility, adaptability and enhancements as the foremost requirements. Future enhancements are always meant to be items that require more planning, budget and staffing to have them implemented. There following are couple of recommended areas for future enhancements:

 Standalone product: It can be implemented as a standalone product, which can be installed in an automobile for monitoring the automobile driver.

 Smart phone application: It can be implemented as a smart phone application, which can be installed on smart phones. And the automobile driver can start the application after placing it at a position where the camera is focused on the driver.

9.2 CONCLUSION

This project aims at developing a software tool for timely and accurate detection of sleepiness in a driver while at the wheel by considering multiple facial features as inputs. By capturing normal image of the user and comparing it with the input feed, the system detects eye closures to check whether the user is drowsy or not.

Dept. of CSE, NHCE Page 44 Monitoring driver’s attention level

Input is captured by the cameras, processed by our personal computer system, and the output is in the form of a buzzer that gives the user an alert as and when the drivers are eyes closed for more than 5 frames. The limitation is that it can act as a detriment to the precision of this system would be poor lighting. This can be overcome by using an appropriate infrared lighting system, if required.

Dept. of CSE, NHCE Page 45 Monitoring driver’s attention level

BIBLIOGRAPHY

REFFERENCE PAPERS

1. RuianLiu,et.al, "Design of face detection and tracking system," Image and Signal Processing (CISP), 2010 3rd International Congress, vol.4, no., pp.1840,1844, 16-18 Oct. 2010

2. Xianghua Fan, et.al, "The system of face detection based on OpenCV," Control and Decision Conference (CCDC), 2012, vol., no., pp.648,651, 23- 25 May 2012

3. Goel, P, et.al., "Hybrid Approach of Haar Cascade Classifiers and Geometrical Properties of Facial Features Applied to Illumination Invariant Gender Classification System," Computing Sciences (ICCS), 2012 International Conference, vol., no., pp.132,136, 14-15 Sept. 2012

4. Parris, J., et.al, "Face and eye detection on hard datasets," Biometrics (IJCB), 2011 International Joint Conference, vol., no., pp.1,10, 11-13 Oct. 2011

5. Peng Wang., et.al., "Automatic Eye Detection and Its Validation," Computer Vision and Pattern Recognition - Workshops, 2005. CVPR Workshops. IEEE Computer Society Conference, vol., no., pp.164,164, 25-25 June 2005

6. Picot, A. et.al., "On-Line Detection of Drowsiness Using Brain and Visual Information," Systems, Man and Cybernetics, Part A: Systems and Humans, IEEE Transactions, vol.42, no.3, pp.764,775, May 2012.

Dept. of CSE, NHCE Page 46 Monitoring driver’s attention level

7. Xia Liu; FengliangXu; Fujimura, K., "Real-time eye detection and tracking for driver observation under various light conditions," Intelligent Vehicle Symposium, 2002. IEEE , vol.2, no., pp.344,351 vol.2, 17-21 June 2002.

8. I Isabelle Tang and Toby P.Breckon., “Automatic Road Environment Classification,” IEEE Transactions on Intelligent Transportation Systems, vol. 12, no. 2, pp. 476 - 484, June 2011

9. P. Jansen, W. van der Mark, J. C. van den Heuvel, and F. C. A. Groen , “Colour based off-road environment and terrin type classification,” Proceedings of the 8th International IEEE Conference on Intelligent Transportation Systems, pp. 216 –221, 2005.

Dept. of CSE, NHCE Page 47