Serious Gaze (Invited Paper) Andrew T. Duchowski School of Computing, Clemson University Clemson, SC 29634 Email: [email protected] Abstract—In this paper I review gaze-based inter- gaze-based interaction action, distinguishing eye movement analysis from synthesis in virtual reality and games for serious ap- expressive plications. My focus is on four forms of gaze-based in- active teraction: diagnostic, active, passive, and expressive. passive diagnostic (off-line) In discussing each, I briefly review seminal results and recent advancements, highlighting outstanding selection/look to shoot gaze synthesis research problems. foveated rendering assessment/training 1. Introduction (gaze-contingent displays) Motivation for this paper was found in the somewhat Figure 1. Gaze interaction taxonomy. sudden inclusion of eye tracking technology in virtual reality. Recent acquisitions of eye tracking companies Eye Tribe, Eyefluence and SMI by Facebook (Oculus), movement is drawn from its synthesis, which can make Google, and Apple, respectively, were notable events. use of a procedural (stochastic) model of eye motion Other eye tracking developments in helmet-mounted dis- driven by goal-oriented tasks such as reading. plays (HMDs) include the FOVE, and SMI or Pupil Labs Before reviewing the four forms of gaze-based inter- add-ons to the HTC Vive. Interestingly, these HMDs action, a short review of eye movement basics offers some are affordable (∼$600) compared to what was available nomenclature and characteristics of gaze. some 15 years ago (∼$60,000) [1]. Most of these systems (including the one I used ca. 2002) feature binocular eye 2. Eye Movement Basics tracking at sampling rates 60 Hz or better. The newer systems sport a larger number of infra-red LEDs, e.g., Detailed human vision is limited to the central 2◦ surrounding each eye, and are more comfortable than my visual angle, about the dimension of one’s thumbnail HMD custom-built by Virtual Research and ISCAN. at arm’s length. Outside of this range, visual acuity Why has eye tracking suddenly become so popular, drops sharply, e.g., about 50% during photopic (day- or, perhaps more importantly, how is tracked gaze being time) conditions. High visual acuity within the central 2◦ exploited in virtual reality and serious games? A useful is due to the tight packing of cone photoreceptors in the taxonomy for reviewing these applications is shown in central foveal region of the retina. Outside foveal vision, Figure 1, which splits gaze-based interaction into four the visual field can be delineated further into parafoveal forms, namely diagnostic (off-line measurement), active vision (out to about 5◦), then perifoveal vision (10◦), and (selection, look to shoot), passive (foveated rendering, then peripheral vision (all the way out to about 80◦ on a.k.a. gaze-contingent displays), and expressive (gaze either the temporal or nasal side of each eye). Sundstedt synthesis). Diagnostic interaction is the mainstay of eye- showed a nice depiction of the human visual field in her tracked serious applications, such as training or assess- course notes [2] and subsequent book [3]. ment of expertise, and is possibly the longest standing Because of the fovea’s limited spatial extent (2◦), in use of gaze due to its mainly offline requirements. Di- order to visually inspect the entire 160◦–180◦ (horizon- agnostic analysis of gaze is still very much in demand, tal) field of view, one needs to reposition the fovea along especially in serious training situations such as flight successive points of fixation. Fixations are characterized or surgery training. Active interaction is rooted in the by tremor, drift, and microsaccades, which are used in desire to use the eyes to point and click, with gaze concert to stabilize gaze on the point of interest on gestures growing in popularity. Passive interaction is the one hand, but keep the eyes in constant motion the manipulation of scene elements in response to gaze on the other, so as to prevent adaptation [4]. This is a direction, e.g., to improve frame rate. Expressive eye consequence of the directional selectivity of retinal and cortical neurons implicated in visual perception [5], [6]. (e.g., with a laser pointer) is involved. Eye movements If the eyes were perfectly still, the visual image would are faster then hand/limb movements, and perhaps for fade from view.1 Most of viewing time (about 90%) is this reason seem more effortless than pointing. The spent in fixations, which is why detection of these eye expert is therefore free to look and make verbal deictic movements is of particular importance. references (e.g., “look at this” [15]) without having to The fovea is repositioned by large jumps of the eyes consciously think about pointing at something. known as saccades. Saccade amplitudes generally range Recorded eye movements of both expert and novice between 1◦–45◦ visual angle (but can be larger; at about can be used to assess the effectiveness of training. A 30◦, the head starts to rotate [8]). particularly good example of a serious game application When tracking an object, smooth pursuits are used is flight simulation, where the study of visual monitoring to match the motion of the moving target. When fixat- is especially important [16]. ing an object, the semi-circular canals of the inner ear Analysis of eye movements depends to a large extent provide signals to counter-rotate the eyes when the head on detection of fixations in the recorded (or real-time) turns—this is known as Vestibular-Ocular Response, or (x, y, t) eye movement signal. Outstanding problems in- VOR. The eyes may also rotate in opposite direction clude better algorithms for scanpath comparison, and during vergence movements; when looking close, the eyes better visualizations. Recent contributions include visu- converge, when looking far, they diverge. Vergence eye alization of dynamic ambient/focal visual attention [17], movements are used for depth perception and are tightly and transition matrix analysis of eye movement [18], but coupled to accommodation, the focusing of the eye’s lens. more advanced developments are sure to come. Further details can be found elsewhere [9]. 4. Active Applications 3. Diagnostic Applications As soon as eye trackers matured sufficiently to pro- duce a real-time signal of the viewer’s (x, y, t) gaze point, Diagnostic analysis of eye movements generally relies they were investigated for their interactive potential. on detection of fixations in an effort to discern what Two seminal contributions from this time are those of elements of the visual scene attracted the viewer’s atten- Jacob [19] and Starker and Bolt [20]. Both contributions tion. The sequential pattern of fixations is referred to as focused on some means of using dwell time to effect the scanpath [10]. What is perhaps most relevant to se- some kind of system response. Jacob used dwell time rious applications is the observation made classically by as a means of disambiguating gaze-based selection, while Yarbus [11]: the pattern of fixations is task-dependent. Starker and Bolt used it as an interest metric, prompting That is, vision is largely top-down, directed by view- the system to provide more detail when something was ing strategy and task demands. However, vision is also fixated for a prolonged period of time. What is especially bottom-up, drawn often involuntarily by eye-catching notable about Jacob’s contribution was his observation elements in the scene [12]. Being able to visualize and of the Midas Touch problem—anything looked at can analyze an expert’s strategy, e.g., during inspection or trigger a response unless some mechanism can be used monitoring, is of prime importance to the understanding to prevent it, e.g., Jacob’s introduction of dwell time. of expertise. Although dwell time is still heavily relied upon for Ericsson et al. [13] surveyed experts’ gaze and noted gaze-based selection, gaze-based gestures have also be- that experts tend to make shorter fixations, make bet- come popular. Along with a review of various gaze-based ter use of extrafoveal/peripheral information, and make gestural interfaces, we compared two leading approaches use of a larger visual span (area around the fixation). and showed that boundary-crossing gestures can be Because experts’ visual search strategies develop with faster than dwell time, depending on the given task [21]. training, it makes sense to not only analyze visual pat- This is obvious as dwell time incurs an artificial delay terns, e.g., to assess expertise, but also as a means of its whereas boundary crossing does not. development via training. In gameplay, a tempting form of interaction is to Following a review of literature related to the use of use the eyes to point at something to aim or shoot scanpaths in training, we showed that Gaze-Augmented at, as in a first-person shooter [22]. This is particularly Think-Aloud can be particularly effective [14]. This pro- effective for arcade-style games (e.g., missile command), tocol records the eye movements of an expert as they as it reduces the amount of mouse movement (although verbalize whatever task they are expert in, and then perhaps spinning that large trackball was part of the fun the video playback is shown to novices as a means of of the old arcade game). Gaze in this context can also training of the same task. This is a fairly straightforward be used to orient the viewpoint, as in Tobii’s (an eye application of eye tracking, yet it holds a number of tracking company) version of Rise of the Tomb Raider. important advantages over alternatives where pointing Sundstedt [3] reviewed various issues of gaze-based gameplay. The Midas Touch is an ever-present consid- 1. An impressive simulation of this phenomenon was demon- strated by Mahowald and Mead [7] in the design of a silicon retina eration.