Wireless Assistive Head Controlled Mouse with Eye-Blink Detection for Enhanced Actions

Turkish Journal of Physiotherapy and Rehabilitation; 32(2) ISSN 2651-4451 | e-ISSN 2651-446X

WIRELESS ASSISTIVE HEAD CONTROLLED MOUSE WITH EYE-BLINK DETECTION FOR ENHANCED ACTIONS

ALAN FRANCIS CHEERAMVELIL1, ARJUN ALOSHIOUS2, N. DHARANIDEVI3, S. VARUN4, ABEY ABRAHAM5 5Assistant Professor, Department of Information Technology, Rajagiri School of Engineering & Technology, Kerala, India- 682039 1,2,3,4Student, Department of Information Technology, Rajagiri School of Engineering & Technology, Kerala, India-682039 1afc2702@gmail.com, [email protected], [email protected], [email protected], [email protected]

ABSTRACT

The Wireless Assistive Head Controlled Mouse with Eye-Blink Detection for enhanced actions is an assistive Human Interface Device aimed at quadriplegic people. The current pandemic has made the world more reliant on digital means of communications than ever before and people with motor disability have no means to access these resources. A majority of them have good head movement but no torso movement. The proposed device uses a Gyroscope sensor to accurately map the user’s head movements to the corresponding mouse coordinates. The device works with Bluetooth Low Energy technology enabling the user to control the digital devices at a comfortable range without the hassle of wires. The plug-N-play feature allows the use of the device without additional drivers. For more sophisticated usage scenarios, the user can choose between the various traditional mouse operations using a desktop software with the help of the eye-blink detection using image processing.

Keywords— Assistive technologies; Persons with disability; Gyroscope sensor; Human Computer Interaction; Eye-blink Detection;

I. INTRODUCTION According to the Census conducted in 2011, persons with disability (PwD) constitute 2.21% of the total population in India. Assistive technologies make their life easier by creating and redesigning devices which are easily accessible by that community. People who are quadriplegic have limited options for modern communication techniques through computers and mobiles due to lack of input methods, so an alternative mode of giving inputs is required. In the present scenario of COVID-19 pandemic the world is reliant on technology more than ever. We now use technology for work, keeping in touch with our loved ones etc. In this situation, one should not be left behind because of their disability and a method should be devised to help them lead a normal life during this tough time.

People who are quadriplegic have limited options for modern communication techniques through computers and mobiles due to lack of input methods, so an alternative mode of giving inputs is required. The Wireless Assistive Head controlled Mouse with Eye-Blink Detection for Enhanced Actions makes use of a person’s head movements to move the cursor on a computer screen or a mobile device accordingly. It performs a Click operation when the device detects no head movement. This helps in making communication easier for PwDs. It also helps them in implementing various tasks in their everyday lives with less effort and more efficiency.

The Wireless Assistive Head Controlled Mouse with Eye-blink Detection for Enhanced Actions helps in empowering individuals with disabilities by allowing those with motor disabilities to interact with computers. The device should be able to emulate all the basic functionalities of an input device so that a person with disability can get the maximum out of his digital devices. It should enable a disabled user to easily access and switch between the various functionalities of a mouse with the help of a reliable and user-friendly Graphical User www.turkjphysiotherrehabil.org 2378

Turkish Journal of Physiotherapy and Rehabilitation; 32(2) ISSN 2651-4451 | e-ISSN 2651-446X

Interface with enough accessibility options to assist them. Since it is not possible to emulate all the functionalities of a normal mouse with this device, given the lack of inputs, we give the user the option to use the device in various mouse modes including left-click, right-click, drag, scroll, double-click, no click & hover and give the person the option to choose between these modes at any point of time. The device translates the head movements of the user to the pointer coordinates on the screen. It offers the feature of modular attachment to existing devices such as head phones. It can be customized and developed according to the requirements of the individual, as each disabled individual has different need depending on his disability. One of the major reasons for a disabled individual to withdraw from an assistive device is its high cost. So, the product must be cost effective. The device provides an excellent user experience and opens up a plethora of opportunities in the digital world regardless of their physical impairments.

There are a lot of assistive technologies available in the market that allow quadriplegic users to operate computer devices. Some of the most popular devices include Gaze Trackers [3] , Video-Based head tracking [4] , Optical Sensor-Based head tracking [5] and Gyroscope-Based head tracking [6] . Gaze trackers track the user’s eye movement by using an IR light source to illuminate the eye causing highly visible reflections, and a camera to capture an image of the eye showing these reflections. The image captured by the camera is then used to identify the reflection of the light source on the cornea (glint) and in the pupil. We can then calculate a vector formed by the angle between the cornea and pupil reflections—the direction of this vector, combined with other geometrical features of the reflections, is then used to calculate the gaze direction. These are one of the most precise technologies for pointer movement in the market but they are extremely sensitive to external lighting noise and reflections on the eye from other IR sources. They also come in the price range of $300 - $10.000 which is not affordable by most individuals. Video-Based head tracking uses the live camera feed from a generic webcam and applies image processing techniques on the video to track the head movement of the user and convert them to the corresponding mouse coordinates on the screen. This is the most popular solution in the market owing to the lack of external hardware required and the cost effectiveness ranging from free to use product to software costing only up to a few hundred dollars. The major drawback of this technology is its poor precision and its heavy dependence on external lighting conditions and variable lighting can make the video unintelligible rendering image processing almost impossible. The optical sensor-based head tracking has an optical reflector placed on the user’s face and an IR emitter sensor couple detects the motion of the reflector thus identifying the user’s head movement and converting it into the corresponding mouse coordinates. The tracking precision is very high as compared to video-based head tracking and the external lighting conditions do not affect the precision too much as there is no need to identify patterns and it has its own source of illumination. The product is also much more affordable as compared to the gaze trackers and costs only a few hundred dollars. The Gyroscope based head- tracking uses a gyroscope sensor in order to detect the user’s head movement and convert it to the corresponding mouse coordinates. It is very accurate and it is completely independent of external lighting as gyroscope is not a sensor that depends on external illumination. The technology also comes in an affordable cost range of about a few hundred dollars.

II. BACKGROUND The different modules included in the working of the proposed device are:

1. Bluetooth Module

2. Microcontroller Module

3. Gyroscope Module

4. Eye Blink Detection Module

1. Bluetooth Module The Bluetooth module is used to establish a connection between a user system and the device. The Bluetooth module can be provided as an additional hardware on the device itself or a Bluetooth enabled microcontroller module can be used to establish the connection. Once a connection has been established the user can start using the device. This is made possible with the introduction of Bluetooth Low Energy (BLE) Technology which has an in-built Human Interface Device (HID) mode which can run on a host system as a regular HID device without the need for additional drivers. The BLE technology also reduces the power consumption dramatically which is a www.turkjphysiotherrehabil.org 2379

Turkish Journal of Physiotherapy and Rehabilitation; 32(2) ISSN 2651-4451 | e-ISSN 2651-446X requirement for any battery powered device. These allow the user to simply connect the device using Bluetooth and start using it making it a Plug-N-Play device.

2. Microcontroller Module Once the device is connected and ready to use, the user can start moving his head in order to work on his computer. These head movements are converted into mouse pointer movements by the combined efforts of the gyroscope module and microcontroller module.

The proposed device uses an ESP32 series microcontroller module [7][9][10][11]. It is a dual-core system with two Harvard Architecture Xtensa LX6 CPUs. All embedded memory, external memory and peripherals are located on the data bus and/or the instruction bus of these CPUs. With some minor exceptions (see below), the address mapping of two CPUs is symmetric, meaning that they use the same addresses to access the same memory. Multiple peripherals in the system can access embedded memory via DMA.

The two CPUs are named “PRO_CPU” and “APP_CPU” (for “protocol” and “application”), however, for most purposes the two CPUs are interchangeable. Inside the ESP32, it has a 1296 KB embedded memory address space, 19704 KB external memory address space, 512 KB peripheral address space and a 328 KB DMA address space. It also contains a 448 KB Internal ROM, 520 KB Internal SRAM, 8 KB RTC FAST Memory and an 8 KB RTC SLOW Memory as internal memory while supporting up to 16MB off-chip SPI Flash and 8MB off-chip SPI SRAM. Most ESP32 models come with an inbuilt Wi-Fi, Bluetooth and BLE technologies which avoids the need to include an extra Bluetooth module separately. The 32-bit architecture ensures that the processing of the Gyroscope signals occur with minimum latency allowing the user to have a responsive device.

3. Gyroscope Module The MPU-6050 [8] is the world’s first integrated 6-axis MotionTracking device that combines a 3-axis gyroscope, 3-axis accelerometer, and a Digital Motion Processor (DMP) all in a small 4x4x0.9mm package. With its dedicated I2C sensor bus, it directly accepts inputs from an external 3-axis compass to provide a complete 9- axis MotionFusion output. The MPU-60X0 MotionTracking device, with its 6-axis integration, on-board MotionFusion, and run-time calibration firmware, enables manufacturers to eliminate the costly and complex selection, qualification, and system level integration of discrete devices, guaranteeing optimal motion performance for consumers.

The MPU-6050 features three 16-bit analog-to-digital converters (ADCs) for digitizing the gyroscope outputs and three 16-bit ADCs for digitizing the accelerometer outputs. For precision tracking of both fast and slow motions, the parts feature a user-programmable gyroscope full-scale range of ±250, ±500, ±1000, and ±2000°/sec (dps) and a user-programmable accelerometer full-scale range of ±2g, ±4g, ±8g, and ±16g. An on-chip 1024 Byte FIFO buffer helps lower system power consumption by allowing the system processor to read the sensor data in bursts and then enter a low-power mode as the MPU collects more data. With all the necessary on-chip processing and sensor components required to support many motion-based use cases. Communication with all registers of the device is performed using either I2C at 400kHz or SPI at 1MHz (MPU-6000 only).

The triple-axis MEMS gyroscope in the MPU-60X0 includes a wide range of features:

 Digital-output X-, Y-, and Z-Axis angular rate sensors (gyroscopes) with a user-programmable fullscale range of ±250, ±500, ±1000, and ±2000°/sec

  External sync signal connected to the FSYNC pin supports image, video and GPS synchronization

 Integrated 16-bit ADCs enable simultaneous sampling of gyros

 Enhanced bias and sensitivity temperature stability reduces the need for user calibration

 Improved low-frequency noise performance

 Digitally-programmable low-pass filter www.turkjphysiotherrehabil.org 2380

Turkish Journal of Physiotherapy and Rehabilitation; 32(2) ISSN 2651-4451 | e-ISSN 2651-446X

 Gyroscope operating current: 3.6mA

 Standby current: 5µA

 Factory calibrated sensitivity scale factor

 User self-test

The digital inputs from the gyroscope modules are converted to the corresponding mouse coordinates using the following equations:

푋 = −(푔푍 + 100)/200

The X, Y represent the X and Y mouse coordinates푌 = −( and푔푌 gX+ 100, gY)/ and200 gZ represent the gyroscope coordinates of the corresponding axes. This helps to ensure that the mouse movements are accurate in such a way that minor involuntary head movements do not affect the movement of the mouse coordinates. The auto-clicking feature is made available by observing whether the user keeps his head still at a particular mouse coordinate for a threshold amount of time. The default click is made to be the left click as taps and clicks make up the majority of all the operations done using a mouse.

4. Eye Blink Detection The user has only limited options when using the device on its own in its default mode. A quadriplegic person has only limited interactions available to communicate with the device. [1] Since eye-blink is a feature very commonly found in many disabled individuals, it is the most suitable means to choose between the various modes of operation of the mouse. The user, upon blinking the eyes at a certain pattern can bring up the GUI for mode selection thus allowing to easily choose between the various mouse modes.

In the proposed product, eye blink detection carried out using image processing techniques can be availed by the user to select different mouse modes for the seamless operation of the device. The method for performing eye blink detection is as follows:

The eye blink is a fast closing and reopening of a human eye. Each individual has a little difference in the pattern of blinks. The pattern differs in the speed of closing and opening, the degree of squeezing the eye and in the blink duration. The eye blink lasts approximately 100-400 milliseconds (ms). We propose to exploit state-of-the-art facial landmark detectors to localize the eyes and eyelid contours. From the landmarks detected in the image, we derive the eye aspect ratio (EAR) that is used as an estimate of the eye-opening state. Since the per-frame EAR may not necessarily recognize the eye blinks correctly, a classifier that takes a larger temporal window of a frame into account is trained.

For every video frame, the eye landmarks are detected. The eye aspect ratio (EAR) between height and width of the eye is computed

EAR = ||푝2−푝6||+||푝3−푝5|| 2||푝1−푝4|| where p1, p2, p3, p4, p5 and p6 are the 2D landmark locations, depicted in Fig. 1. The EAR is mostly constant when an eye is open and is getting close to zero while closing an eye. It is partially person and head pose insensitive. Aspect ratio of the open eye has a small variance among individuals and it is fully invariant to a uniform scaling of the image and in-plane rotation of the face. Since eye blinking is performed by both eyes synchronously, the EAR of both eyes is averaged.

A low value EAR value does not mean that a person is blinking. It may occur when a subject closes his/her eyes intentionally for a longer time or performs a facial expression, yawning, etc., or the EAR captures a short random fluctuation of the landmarks. For the 30fps videos, we experimentally found that ±6 frames can have a significant www.turkjphysiotherrehabil.org 2381

Turkish Journal of Physiotherapy and Rehabilitation; 32(2) ISSN 2651-4451 | e-ISSN 2651-446X impact on a blink detection for a frame where an eye is the most closed when blinking. Thus, for each frame, a 13-dimensional feature is gathered by concatenating the EARs of its ±6 neighboring frames. This is implemented by a linear SVM classifier (called EAR SVM) trained from manually annotated sequences. Positive examples are collected as ground-truth blinks, while the negatives are those that are sampled from parts of the videos where no blink occurs, with 5 frames spacing and 7 frames margin from the ground-truth blinks. While testing, a classifier is executed in a scanning-window fashion. A 13-dimensional feature is computed and classified by EAR SVM for each frame except the beginning and ending of a video sequence.

5. Graphical User Interface An efficient graphical user interface is required for the uninterrupted working of any application or device regardless of its complexity. Things are no different in case of the device proposed in this paper. If a user wants to experience all the features provided by the Wireless Assistive Head Controlled Mouse, a very efficient GUI is instrumental. The GUI for the proposed device is build using Python’s PyQt5 module.

It contains the following features: Left Click, Right Click, Middle Click, Hover, Scroll and Drag. The user is free to select a mouse mode according to his/her requirement. By default, the system operates on the left click mode. Mode selection is done using eye blink detection. If the user blinks his/her eyes a certain number of times, the corresponding mouse mode is selected. A counter variable can be employed to store the number of eye blinks. The user is made aware about the current mouse mode by using a display on the lower right corner of the computer screen.

The following packages are used to develop the GUI for the product:

• QtWidgets

• QtGui

• QtCore

• QApplication

• QSizeGrip

• QDesktopWidget

• QMainWindow

• QIcon

• QGridLayout

• QDialog

• QGroupBox

• QPushButton

• QVBoxLayout

• QSizePolicy

• QShortcut

• QHBoxLayout

• QMessageBox

www.turkjphysiotherrehabil.org 2382

Turkish Journal of Physiotherapy and Rehabilitation; 32(2) ISSN 2651-4451 | e-ISSN 2651-446X

The interface has been designed with accessibility in mind. The color theme used brings out an efficient contrast between the background and the button text.The popping up of the GUI provides two functionalities, selecting the mouse mode and also to recenter the mouse making re-calibration easier. The status window always indicates the current operational mode of the device. This is created in such a way that it always stays on top of all the windows, so that the user can keep track of the current mode even while using full screen applications. The text is positioned in the most unintrusive position for a regular user away from the main window while still providing access to the task bar. The text which is black in color has a semi-transparent white background, which highlights the current mode always while still allowing the user to see what is happening behind the status window.

III. PERFORMACE MATRICS AND RESULTS The metrics used to calculate the effectiveness of the device are: the index of difficulty, path efficiency and throughput.

Index of Difficulty (ID) is :

퐷 퐼퐷 = Where D is the distance between the original location of the푊 cursor to the target location and W is the width of the target.

Path efficiency is the efficiency in the ability to follow a preset path using the mouse pointer. It is defined as:

퐷 푃퐸 = Where P is the path length travelled by the cursor. 푃

Throughput is the measure of how fast the cursor moved from the source to the destination. It is defined by:

퐼퐷 푇푃 = Where T is the time taken to reach the destination. 푇

With the prototype we created and tested, the user felt the device to be accurate and only had to give minimal inputs to get the desired results. The testing proved a sustained path efficiency of 60%-80% and a throughput of less than 0.2 pixels per second. This being an assistive device, we used a formal method to track the effectiveness of the device to a disabled individual. For this, the Matching Person with Technology (MPT) framework is used. This is a questionnaire which asks the user the most common needs required from the device and comes up with a score according to the answers, in a total of 60. This device along with some of the most commonly available assistive devices were put to test using this framework to a disabled individual and the scores were as follows:

 Head Controlled Mouse – 60

 Camera Mouse – 30

 Speech Recognition – 55

This proves that this device is highly effective as an assistive technology for a quadriplegic user.

www.turkjphysiotherrehabil.org 2383

Turkish Journal of Physiotherapy and Rehabilitation; 32(2) ISSN 2651-4451 | e-ISSN 2651-446X

Figure 1:ESP32 System Architecture

Figure 2:Facial landmarks of the eye Figure 3:Proposed GUI of the project

Figure 3:Working prototype of the product

www.turkjphysiotherrehabil.org 2384

Turkish Journal of Physiotherapy and Rehabilitation; 32(2) ISSN 2651-4451 | e-ISSN 2651-446X

IV. CONCLUSION

In our day-to-day lives, we are heavily dependent on modern technologies since they tend to simplify most of our complex and laborious tasks. People with disabilities find it more difficult to perform tasks that are relatively easier for us. An assistive technology like the Wireless Assistive Head Controlled Mouse with Eyeblink Detection for Enhanced Actions would surely help in empowering these individuals, and make them more self- reliant. This device opens up a new horizon for people with disabilities to expand upon the virtual world as it would help them perform such tasks with relative ease. Common tasks such as communicating with a computer, or even complex tasks such as using a video editing application could be accomplished using this device, without the requirement of another person’s help. This could help them save a lot of their time and effort. The foundation of the project is mapping a user's head movements into corresponding mouse coordinates, which in turn results in the movement of the mouse pointer on the user’s screen. It uses a gyroscope sensor to map the user’s head movements to corresponding mouse pointer movements.

With wireless Bluetooth technology enabled, the user can comfortably control this device from a distance with no additional requirement of any extra drivers or software. The different functionalities of a mouse can be achieved by observing a user's eye blink through image processing. An additional benefit is the low cost involved in manufacturing and maintaining it. The device also offers the feature of modular attachment which makes it compatible with existing devices. The different features of the device like the Plug and Play nature, would help these individuals to build customized versions of the same. The development of the device demands active involvement from the disabled individual. Hence, the device can be customized and developed according to the requirements and disabilities of each individual user. The device provides wireless connectivity which can provide a hassle-free experience for the user. By meeting more users and collecting their feedback, the device and its features could be modified and improved in the future. The precision and control of the device could be improved in the future and assistive switches could be added for performing more mouse functionalities.

REFERENCES 1. Soukupova T, Cech J. Eye blink detection using facial landmarks. In21st computer vision winter workshop, Rimske Toplice, Slovenia 2016 Feb 3. 2. Lee WO, Lee EC, Park KR. Blink detection robust to various facial poses. Journal of neuroscience methods. 2010 Nov 30;193(2):356-72. 3. Chennamma HR, Yuan X. A survey on eye-gaze tracking techniques. arXiv preprint arXiv:1312.6410. 2013 Dec 4. Xie D, Dang L, Tong R. Video based head detection and tracking surveillance system. In2012 9th International Conference on Fuzzy Systems and Knowledge Discovery 2012 May 29 (pp. 2832-2836). IEEE. 5. Zhao Y, Görne L, Yuen IM, Cao D, Sullman M, Auger D, Lv C, Wang H, Matthias R, Skrypchuk L, Mouzakitis A. An orientation sensor-based head tracking system for driver behaviour monitoring. Sensors. 2017 Nov;17(11):2692. 6. Avizzano C, Sorace P, Checcacci D, Bergamasco M, PERCRO SS, zza dei Martiti P. Head tracking based on accelerometer sensors. PERCRO, Scuola Superiore S. Anna. 2004. 7. ESP32 Technical Reference Manual Version 4.4 Espressif Systems 8. MPU-6000 and MPU-6050 Product Specification Revision 3.4 9. Punithavathani, D. Shalini, K. Sujatha, and J. Mark Jain. "Surveillance of anomaly and misuse in critical networks to counter insider threats using computational intelligence." Cluster Computing 18.1 (2015): 435-451. 10. Chang, Jinping, Seifedine Nimer Kadry, and Sujatha Krishnamoorthy. "Review and synthesis of Big Data analytics and computing for smart sustainable cities." IET Intelligent Transport Systems (2020). 11. Song, Hesheng, and Carlos Enrique Montenegro-Marin. "Secure prediction and assessment of sports injuries using deep learning based convolutional neural network." Joural of Ambient Intelligence and Humanized Computing 12.3 (2021): 3399-3410.