Feature Article Spatial Analytic Interfaces: Spatial User Interfaces for In Situ Visual Analytics Barrett Ens and Pourang Irani ■ University of Manitoba ersonal computing devices are becoming As wearable device interfaces continue to shrink, smaller, yet more powerful, allowing greater current design solutions are trending further user mobility, increased personal data toward simplicity; new interface paradigms (such Pcollection, and greater ability to manipulate these as Google Glass and Android Wear) are designed to data to benefit our everyday lives. A catalyst in this support micro-interactions, short bursts of activity shift is the increased access to and use of sensors that avoid impinging on the user’s daily activities and interfaces that are being integrated into what by minimizing task duration. we wear. We are experiencing In contrast to these trends, analytic tasks re- a continuing shift to wearable quire unique types of interfaces. Such tasks often What will user interfaces look form factors such as smart require concerted thought, integrating informa- like in a post-smartphone watches and digital eyewear. tion from multiple sources, and applying human world, and will these This new generation of sensemaking abilities. Typical examples of everyday future interfaces support interactive information displays analytic tasks include balancing a checkbook, plan- sophisticated interactions has great potential to enrich ning a vacation, and doing a price search for the in a mobile context? Spatial our lives. Unlike current mobile best available deal on a particular item. Although analytic interfaces can technology, information from we are well accustomed to computer support for leverage the benefits of spatial these devices can be ingested these tasks, they are not well supported by today’s interaction to enable everyday with a glance at the wrist or even mobile interfaces. visual analytic tasks to be a slight eye movement. Such To design interfaces that support analytic tasks, performed in situ, at the most always-available information we can draw from the visual analytics field, which beneficial place and time. access allows in situ computing: is devoted to developing tools that help users gain access to situationally appropriate insights through deep exploration of multiple data at an ideal time and place. By providing interlinked visualizations of diverse datasets. wearable technology with suitable information- Although originally aimed at supporting domain seeking interfaces, we can make computing a experts with intensive analysis, for instance with natural and “invisible” part of our daily activities. biomedical data1 or military intelligence reports,2 The complexity of mobile computing interfaces visual analytic methods have recently been adopted has so far been limited by the available space for for analysis of everyday personal information.3,4 For input and display. For example, some common tasks example, sensors in people’s homes track energy performed on mobile devices include consumption consumption and resource usage patterns, mobile (such as reading and watching videos), mobile computers such as smartphones and embedded communication (such as sending and receiving automobile software continuously track their short messages), and organization (such as owners’ movements, and wearable accessories track keeping a list of contacts and setting reminders). personal health and fitness data. This ubiquitous 66 March/April 2017 Published by the IEEE Computer Society 0272-1716/17/$33.00 © 2017 IEEE Head-Worn Display Technology data-collection trend presents a growing need for tools to comprehend and digest the important he concept of a display worn on the user’s head originated in patterns and to provide actionable results.5 Tthe late 1960s,1 and a wide variety of realizations have under- The benefits to be realized from an increasing gone development since. Many advances in 3D interface design prevalence of mobile and wearable technology are have occurred as a result of VR research since the early 1990s. VR twofold: While these devices allow the routine has seen a recent resurgence in popular culture as advances in collection of useful activity data, they also provide hardware have progressed to the stage where relatively light- an opportunity to facilitate in situ data analysis. weight, low-latency devices such as the Oculus Rift and HTC Vive Homeowners concerned with minimizing their are entering the market. energy consumption, for instance, are better Optical see-through HWDs are most widely known through able to make informed choices if appropriate the introduction of Google Glass, which revealed user concerns information is available at the time when they about privacy and social acceptability. Unlike Google Glass, which are choosing how to consume resources or energy was designed for micro-interactions on a small, peripheral display, (such as when adjusting a thermostat). Similarly, another class of see-through HWDs place binocular displays in the if people are able to consult their banking user’s line of sight. These stereoscopic devices, which superim- history through a mobile app, they can use this pose objects in 3D space, are ideally suited for the development of information directly before making significant spatial analytic interfaces (SAIs). Robust sensing technologies are purchases. The mobile component is essential to also being incorporated into such devices to track the user’s hands in situ computing because the situational context or the external environment. Microsoft’s Hololens, for example, is lost if the user has to wait to view data at home can construct a model of the user’s surroundings in real time and on a personal computer. Nevertheless, viewing use this information to integrate virtual displays on nearby walls. data on the small screen of a personal mobile At the same, hardware is being miniaturized so we can soon device is often still prohibitively cumbersome, expect devices that look similar to typical eyewear in common use and it lacks the potential to provide insight using today. As a result, social acceptance will likely increase to the point multiple, coordinated views of the data.2 where such devices may be commonly worn in a variety of daily One promising approach to developing mobile activities. interfaces for in situ use, with advanced features to support analysis and sensemaking, is the application of spatial user interfaces. Spatial user Reference interfaces leverage benefits such as spatial memory 1. I.E. Sutherland, “A Head-Mounted Three Dimensional Display,” Proc. and proprioception to map information to a Fall Joint Computer Conf., Part I (AFIPS), 1968, pp. 757–764. physical space, and research has shown they can improve performance on some analytic tasks.6 For instance, arranging multiple visualizations side by cameras and inertial sensors that allow us to track side can allow for faster and easier comparison the user’s hand, fingertip, and body motion. These than navigating between multiple components features facilitate intuitive spatial interaction, such on a single abstract interface; the user can easily as the ability to switch between spatially situated switch views and apply spatial memory to recall displays by turning one’s head.7 With robust the location of important items, making for an spatial tracking, these devices essentially provide efficient and intuitive experience. unlimited display space; multiple information We propose the concept of spatial analytic visualizations can be integrated directly into the interfaces (SAIs) for everyday data monitoring and appropriate home, work, or mobile environment. decision making based on in situ analysis. SAIs Furthermore, virtual displays rendered by these leverage the benefits of spatial user interfaces for wearable systems can be situated where they are completing in situ, analytic tasks. Although in most convenient for a given context, such as on this article we focus on head-worn display (HWD) a kitchen counter or backsplash for monitoring technology as the particularly appropriate home energy consumption or in a hemispherical platform for in situ analytic tasks, the concept formation around the body in mobile situations of SAIs is platform-agnostic. We chose to work when a user is shopping or jogging. This spatial with HWDs because the technology is advancing paradigm can also support advanced techniques rapidly and they are available in lightweight form not possible with standard desktop displays; for factors at an affordable cost for general consumers. example, visual links can span a physical space to (See the “Head-Worn Display Technology” sidebar connect data across multiple displays or guide users for more details.) HWDs such as Meta and to information that is not currently in their focus 8 Microsoft Hololens can come equipped with depth of attention. IEEE Computer Graphics and Applications 67 Feature Article (a) (b) (c) Figure 1. Three scenarios depicting beneficial uses of spatial analytic interfaces: (a) comparing heart rate and route records during exercise with visualizations that hover in space, (b) monitoring home water consumption using virtual information panels that appear on a kitchen backsplash, and (a) completing a quick budget before making a purchase using a spatially tracked stylus and virtual documents overlaid on a nearby surface. The goal of this article is to introduce the shows her step count, heart rate, and estimated SAI concept, discuss its benefits over current calories burned.