Deep-Learning-Based Pupil Center Detection and Tracking Technology for Visible-Light Wearable Gaze Tracking Devices
Total Page:16
File Type:pdf, Size:1020Kb
applied sciences Article Deep-Learning-Based Pupil Center Detection and Tracking Technology for Visible-Light Wearable Gaze Tracking Devices Wei-Liang Ou, Tzu-Ling Kuo, Chin-Chieh Chang and Chih-Peng Fan * Department of Electrical Engineering, National Chung Hsing University, 145 Xingda Rd., South Dist., Taichung City 402, Taiwan; [email protected] (W.-L.O.); [email protected] (T.-L.K.); [email protected] (C.-C.C.) * Correspondence: [email protected] Abstract: In this study, for the application of visible-light wearable eye trackers, a pupil tracking methodology based on deep-learning technology is developed. By applying deep-learning object detection technology based on the You Only Look Once (YOLO) model, the proposed pupil tracking method can effectively estimate and predict the center of the pupil in the visible-light mode. By using the developed YOLOv3-tiny-based model to test the pupil tracking performance, the detection accuracy is as high as 80%, and the recall rate is close to 83%. In addition, the average visible-light pupil tracking errors of the proposed YOLO-based deep-learning design are smaller than 2 pixels for the training mode and 5 pixels for the cross-person test, which are much smaller than those of the previous ellipse fitting design without using deep-learning technology under the same visible-light conditions. After the combination of calibration process, the average gaze tracking errors by the proposed YOLOv3-tiny-based pupil tracking models are smaller than 2.9 and 3.5 degrees at the training and testing modes, respectively, and the proposed visible-light wearable gaze tracking system performs up to 20 frames per second (FPS) on the GPU-based software embedded platform. Citation: Ou, W.-L.; Kuo, T.-L.; Keywords: deep-learning; YOLOv3-tiny; visible-light; pupil tracking; gaze tracker; wearable Chang, C.-C.; Fan, C.-P. eye tracker Deep-Learning-Based Pupil Center Detection and Tracking Technology for Visible-Light Wearable Gaze Tracking Devices. Appl. Sci. 2021, 11, 1. Introduction 851. https://doi.org/10.3390/ Recently, wearable gaze tracking devices (or called eye trackers) have started to app11020851 become more widely used in human-computer interaction (HCI) applications. To evaluate people’s attention deficit, vision, and cognitive information and processes, gaze tracking Received: 29 November 2020 devices have been well-utilized in this field [1–6]. In [4], the eye-tracking technology was a Accepted: 12 January 2021 Published: 18 January 2021 vastly applied methodology to examine hidden cognitive processes. The authors developed a program source code debugging procedure that was inspected by using the eye-tracking Publisher’s Note: MDPI stays neutral method. The eye-tracking-based analysis was elaborated to test the debug procedure and with regard to jurisdictional claims in record the major parameters of eye movement. In [5], the eye-movement tracking device published maps and institutional affil- was used to observe and analyze a complex cognitive process for C# programming. By iations. evaluating the knowledge level and eye movement parameters, the test subjects were involved to analyze the readability and intelligibility of the query and method at the Language-Integrated Query (LINQ) declarative query of the C# programming language. In [6], the forms and efficiency of debugging parts of software development were observed by tracking eye movements with the participation of test subjects. The routes of gaze for Copyright: © 2021 by the authors. Licensee MDPI, Basel, Switzerland. the test subjects were observed continuously during the procedure, and the coherences This article is an open access article were decided by the assessment of the recorded metrics. distributed under the terms and Many up-to-date consumer products have applied embedded gaze tracking technol- conditions of the Creative Commons ogy in their designs. Through the use of gaze tracking devices, the technology improves Attribution (CC BY) license (https:// the functions and applications of intuitive human-computer interaction. Gaze tracking creativecommons.org/licenses/by/ designs use an image sensor based on near-infrared (NIR), which was utilized in [7–16]. 4.0/). Some previous eye-tracking designs used high-contrast eye images, which are recorded by Appl. Sci. 2021, 11, 851. https://doi.org/10.3390/app11020851 https://www.mdpi.com/journal/applsci Appl. Sci. 2021, 11, x FOR PEER REVIEW 2 of 21 Appl. Sci. 2021, 11, 851 2 of 21 designs use an image sensor based on near-infrared (NIR), which was utilized in [7–16]. Some previous eye-tracking designs used high-contrast eye images, which are recorded activeby active NIR NIR image image sensors. sensors. NIR NIR image image sensor-based sensor-based technology technology can achievecan achieve high high accuracy accu- whenracy when used indoors.used indoors. However, However, the NIR-based the NIR-ba designsed design is not is suitablenot suitable for usefor outdoorsuse outdoors or withor with sunlight. sunlight. Moreover, Moreover, long-term long-term operation operatio mayn may have have the the potential potential to to hurt hurt the the user’s user’s eyes.eyes. When When eye eye trackers trackers are are considered considered for for general general and and long-term long-term use, use, visible-light visible-light pupil pupil trackingtracking technology technology becomes becomes a a suitable suitable solution. solution. DueDue to to the the characteristics characteristics of theof the selected selected sensor sensor module, module, the contrastthe contrast of the of visible-light the visible- eyelight image eye mayimage be may insufficient, be insufficient, and the nearfieldand the nearfield eye image eye may image contain may some contain random some types ran- ofdom noise. types In general,of noise. visible-light In general, eyevisible-light images usually eye images include usually insufficient include contrast insufficient and large con- randomtrast and noise. large In random order to noise. overcome In order this to inconvenience, overcome this some inconvenience, other methods some [17 other–19] havemeth- beenods [17–19] developed have for been visible-light-based developed for visible-li gaze trackingght-based design. gaze tracking For the previousdesign. For wearable the pre- eyevious tracking wearable design eye in tracking [15,19] withdesign the in visible-light [15,19] with mode, the visible-light the estimation mode, of pupil the estimation position wasof pupil easily position disturbed was by easily light anddisturbed shadow by occlusion,light and shadow and the predictionocclusion, and error the of theprediction pupil position,error of thus,the pupil becomes position, serious. thus, Figure becomes1 illustrates serious. some Figure of the1 illustrates visible-light some pupil of the tracking visible- resultslight pupil by using tracking the previous results by ellipse using fitting-based the previous methodsellipse fitting-based in [15,19]. Thus, methods a large in [15,19]. pupil trackingThus, a errorlarge willpupil lead tracking to inaccuracy error will of thelead gaze to inaccuracy tracking operation. of the gaze In tracking [20,21], for operation. indoor applications,In [20,21], for the indoor designers applications, developed the eyedesigners tracking developed devices basedeye tracking on visible-light devices based image on sensors.visible-light A previous image studysensors. [20 ]A performed previous experimentsstudy [20] performed indoors, but experiments showed that indoors, the device but couldshowed not that work the normally device could outdoors. not work When normally the wearable outdoors. eye-tracking When the wearable device is correctlyeye-track- useding device in all indoor is correctly and outdoor used in environments,all indoor and theoutdoor method environments, based on the the visible-light method based image on sensorthe visible-light is sufficient. image sensor is sufficient. FigureFigure 1. 1.Visible-light Visible-light pupil pupil tracking tracking results results by by the the previous previous ellipse ellipse fitting-based fitting-based method. method. RelatedRelated Works Works InIn recent recent years, years, the the progress progress of deep-learning of deep-learning technologies technologies has become has become very important, very im- especiallyportant, especially in the field in of the computer field of computer vision and vision pattern and recognition. pattern recognition. Through deep-learningThrough deep- technology,learning technology, inference models inference can models be trained can tobe realize trained real-time to realize object real-time detection object and detection recog- nition.and recognition. For the deep-learning-based For the deep-learning-based designs in de [signs22–37 in], the[22–37], eye trackingthe eye tracking systems systems could detectcould thedetect location the location of eyes orof pupils,eyes or regardlesspupils, regardless of using of the using visible-light the visible-light or near-infrared or near- imageinfrared sensors. image Table sensors.1 describes Table 1 thedescribes overview the ofoverview the deep-learning-based of the deep-learning-based gaze tracking gaze designs.tracking Fromdesigns. the From viewpoints the viewpoints of device of setup, device the setup, deep-learning-based the deep-learning-based gaze tracking gaze designstracking in designs [22–37] in can [22–37] be divided can be into divided two different into two styles, different which styles,