AmbiGaze: Direct Control of Ambient Devices by Gaze Eduardo Velloso1;2, Markus Wirth2;3, Christian Weichel2, Augusto Esteves4, Hans Gellersen2 1Microsoft Research Centre for Social NUI, The University of Melbourne 2School of Computing and Communications, Lancaster University 3University of Munich (LMU), Munich, Germany 4Institute for Informatics and Digital Innovation, Edinburgh Napier University [email protected], [email protected], [email protected], [email protected], [email protected] ABSTRACT Eye tracking offers many opportunities for direct device con- trol in smart environments, but issues such as the need for calibration and the Midas touch problem make it impractical. In this paper, we propose AmbiGaze, a smart environment that employs the animation of targets to provide users with direct control of devices by gaze only through smooth pursuit tracking. We propose a design space of means of exposing functionality through movement and illustrate the concept through four prototypes. We evaluated the system in a user Figure 1. (A) The user approaches an AmbiGaze object. (B) When the study and found that AmbiGaze enables robust gaze-only in- user faces it, the object presents moving targets. (C) The user then se- lects options by following a target with his eyes. In the example, the light teraction with many devices, from multiple positions in the colour changes according to the windmill paddle the user is looking at. environment, in a spontaneous and comfortable manner. contrast, we propose AmbiGaze, a system that employs anima- Author Keywords tion of targets in the environment to provide users with direct Eye tracking; Ubiquitous Computing; Smart Environments; control of ambient devices by gaze only. Smooth Pursuits Gaze affords implicit identification of interaction targets as we naturally attend to the devices we intend to control [2, ACM Classification Keywords 22]. However, gaze-only control of environments has in the H.5.m. Information Interfaces and Presentation (e.g. HCI): past been hampered by the need to calibrate gaze to different Miscellaneous targets, the inaccuracies introduced by the jittery movements of the eye during fixation, and the accidental activation of INTRODUCTION targets as the eyes scan the environment (the Midas Touch). Our everyday environments are populated with increasing To overcome these problems in AmbiGaze, we employ the numbers of devices that afford digital control. While these Pursuits technique, using animation of targets for activation by devices are diverse in form and functionality, there is a desire gaze-following [23]. Put in motion, a target induces smooth to provide users with uniform control across smart environ- pursuit eye movement when the user follows it. The smooth ments [26,1, 25]. User research in smart homes specifically pursuit movement is distinct from the routine movement of highlighted a need for instant control, for users to adapt their the eye, and supports robust target activation by comparing the environment on impulse. Koskela et al. found users require relative movement of the eyes to the trajectory of the animated control “right now" with minimal action, and “right here" target. across all the devices that affect their situation [14]. Com- mercial and research work has addressed this problem with Figure 1 illustrates the interaction design of AmbiGaze. Users universal remote control strategies that give users environment initiate control of a device implicitly by turning their attention control via mobile intermediary devices [1, 26, 18,5]. In to the device. The device responds to attention by displaying control options in animated form to the user. By following an animated target, the user triggers the associated command. The Permission to make digital or hard copies of all or part of this work for personal or user experience is fluid as gaze is used seamlessly for both classroom use is granted without fee provided that copies are not made or distributed device selection and command input, and users can attend for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM different devices in sequence. must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a In this paper, we investigate three questions raised by the Am- fee. Request permissions from [email protected]. biGaze concept. First, we explore how ambient devices can DIS 2016, June 04–08, 2016, Brisbane, Australia ACM 978-1-4503-4031-1/16/06. $15.00 expose controls in animated form. Previously, the Pursuits DOI: http://dx.doi.org/10.1145/2901790.2901867 technique was used on display devices that easily afford ani- mation of targets [23, 17,8]. In contrast, devices in a smart environment are physically manifest, static in appearance, and not necessarily equipped with a display. We explore device augmentation with animated control through the design of four prototypes, a music player, VoD interface, electric fan, and lamp. Secondly, we consider how pursuit control can scale from single devices to smart environments, and implement Figure 2. The AmbiGaze system integrates four ambient devices, ex- the AmbiGaze system integrating the four device prototypes ploring a design space in which device controls are animated by virtual and managing interaction across them. Thirdly, we evaluate versus mechanical means, by the device itself (internal) or externally. robustness of pursuit control from different vantage points in an environment. Whereas users face targets frontally in provides for interactions that are intuitive to perform, as our conventional display settings, they need to be able to control eyes naturally follow moving stimuli. Other techniques that targets from different viewing angles and distances in a smart involve matching the movement of moving stimuli include environment. Motion Pointing (mouse) and PathSync (hand gestures) [9,4]. In AmbiGaze, we leverage these insights but address new chal- lenges for the technique: first, making it work with physical RELATED WORK and displayless devices; secondly, matching targets correctly By 2010, most households in the UK had 4 or more remote when their trajectories can appear substantially distorted de- controls in their living room [15], making impulsive tasks pending on viewing angle; thirdly, scaling the technique to such as turning the TV on and the radio off, adjusting the disambiguate input in environments with potentially many volume, and dimming the lights complex to perform [14]. The devices. problem has been addressed with universal remotes that can be switched to different target devices [26], personalisable remote controls [12], remotes that adapt control to the device they are pointed at [1, 16, 19], ’magic wands’ for gestural control [25], ANIMATION OF AMBIENT DEVICES and smartphone-based remote control [18,5,3]. In contrast, To enable their control by pursuit, ambient devices need to we provide a solution that enables users to control ambient expose controls in animated form. We propose a design space devices directly by gaze, without need to carry or pick up in which we consider different ways of rendering movement any intermediary device. Though we employ a wearable gaze – virtual by graphics animation on a display versus mechani- tracker in our prototype, the concept could be easily extended cal by physical actuation, and generated internally by device to remote trackers with no user-worn devices. versus projected externally onto the device. We built four pro- totypes as part of the AmbiGaze system to explore the design Previous work on gaze in smart environments has demon- space (see Figure2). strated ambient devices that respond to visual attention [21, 20, 22]. In AmbiGaze we expand upon attention-awareness Virtual + Internal: In this type of implementation, the object by adding pursuit control. Rather than associating attention itself renders moving graphics. These can range from the high directly with a device command, we treat it like a “hover" resolution of TV screens to cruder representations on LED state in which input options are displayed for "clicking" by strips and small displays. Our first prototype is an interface for smooth pursuit. This puts users in full control over the devices a video-on-demand (VoD) service rendered on a TV screen. in their environment; they can look at devices without neces- When a video ends, the system suggests three possible videos sarily invoking any command, and select (where appropriate) for the user to watch next (see Figure3-2). If the user follows from more input options than a single default response. Other the movements of one of the thumbnails floating on the screen, work on direct control of ambient devices has proposed gaze the background changes to match the video and displays its pointing coupled with command gestures [11], and pointing details, including the title, the synopsis and the rating. Below gestures coupled with finger snapping [10] but not tried these the video information, there is a play button with a small dot at environment scale. In more generally related work, gaze moving on an orbit around it. If the user follows the dot, the has been used to switch input among target displays [6] and to currently selected video starts playing. guide visual recognition of objects in the environment [13]. Virtual + External: Our second prototype demonstrates Gaze input based on smooth pursuit was introduced by Vidal et how movement can be added to devices that do not have a al. for interaction with animated content on public displays, display. We implemented a Music Player interface with vir- [23], and employed by Pfeuffer et al. for opportunistic calibra- tual animated widgets that are externally projected onto a tion [17]. Esteves et al. extended the technique with animated pair of speakers in the environment (see Figure3-1).