Synchronous Gestures in Multi-Display Environments Gonzalo Ramos University of Toronto, Live Labs Kenneth Hinckley Microsoft Research Andy Wilson Microsoft Research Raman Sarin Microsoft Research RUNNING HEAD: SYNCH. GESTURES IN DISPLAY ENVIRONMENTS Corresponding Author’s Contact Information: Gonzalo Ramos 136 – 102nd Ave SE, #226 Bellevue, WA, 98004, USA (425)445-3724 [email protected] - 1 - Brief Authors’ Biographies: Gonzalo Ramos received his Honors Bachelors in Computer Science from the University of Buenos Aires where he worked on image compression and wavelets. He later obtained his M.Sc. in Computer Science at the University of Toronto, focusing on numerical analysis and scientific visualization issues. He completed his doctoral studies in Computer Science at the University of Toronto doing research in Human-Computer Interaction. Currently he is a scientist at Microsoft’s Live Labs. Kenneth Hinckley is a research scientist at Microsoft Research. Ken’s research extends the expressiveness and richness of user interfaces by designing novel input technologies and techniques. He attacks his research from a systems perspective, by designing and prototyping advanced functionality, as well as via user studies of novel human-computer interactions and quantitative analysis of user performance in experimental tasks. He holds a Ph.D. in Computer Science from the University of Virginia, where he studied with Randy Pausch. This article is dedicated to Randy’s heroic battle against pancreatic cancer. Andy Wilson is a member of the Adaptive Systems and Interaction group at Microsoft Research. His current areas of interest include applying sensing techniques to enable new styles of human-computer interaction, as well as machine learning, gesture-based interfaces, inertial sensing and display technologies. Before joining Microsoft, Andy obtained his B.A. at Cornell University, and M.S. and Ph.D. at the MIT Media Laboratory. - 2 - Raman Sarin is a Research Software Design Engineer at Microsoft Research. Raman’s current interests include search user interfaces, mobile devices, special-purpose devices, and pen-operated devices, with an emphasis on the realization of research concepts in pragmatic systems that people can use. He holds a B.S. from Rensselaer Polytechnic Institute (RPI). See hci-journal.com/editorial/final-guidelines.html for explanations of the various parts of an article in HCI format. - 3 - ABSTRACT Synchronous gestures are patterns of sensed user or users’ activity, spanning a distributed system that take on a new meaning when they occur together in time. Synchronous gestures draw inspiration from real-world social rituals such as toasting by tapping two drinking glasses together. In this paper, we explore several interactions based on synchronous gestures, including bumping devices together, drawing corresponding pen gestures on touch-sensitive displays, simultaneously pressing a button on multiple smart-phones, or placing one or more devices on the sensing surface of a tabletop computer. These interactions focus on wireless composition of physically co-located devices, where users perceive one another and coordinate their actions through social protocol. We demonstrate how synchronous gestures may be phrased together with surrounding interactions. Such connection-action phrases afford a rich syntax of cross- device commands, operands, and one-to-one or one-to-many associations with a flexible physical arrangement of devices. Synchronous gestures enable co-located users to combine multiple devices into a heterogeneous display environment, where the users may establish a transient network connection with other select co-located users to facilitate the pooling of input capabilities, display resources, and the digital contents of each device. For example, participants at a meeting may bring mobile devices including tablet computers, personal digital assistants (PDA’s), and smart-phones, and the meeting room infrastructure may include fixed interactive displays, such as a tabletop computer. Our techniques facilitate creation of an ad-hoc display environment for tasks such as viewing a large document across multiple devices, presenting information to another user, or offering files to others. The - 4 - interactions necessary to establish such ad-hoc display environments must be rapid and minimally demanding of attention: during face-to-face communication, a pause of even five seconds is socially awkward and disrupts collaboration. Current devices may associate using a direct transport such as Infrared Data Association (IRDA) ports, or the emerging Near-Field Communication (NFC) standard. However, such transports can only support one-to-one associations between devices, and require close physical proximity as well as a specific relative orientation in order to connect the devices (e.g., the devices may be linked when touching head-to-head, but not side-to-side). By contrast, sociology research in proxemics (the study of how people use the “personal space” surrounding their bodies) demonstrates that people carefully select physical distance as well as relative body orientation to suit the task, mood, and social relationship with other persons. Wireless networking can free device-to-device connections from the limitations of direct transports, but results in a potentially large number of candidate devices. Synchronous gestures address these problems by allowing users to express naturally a spontaneous wireless connection between specific proximal (collocated) interactive displays. - 5 - CONTENTS 1. INTRODUCTION 1.1. Proxemics - How People Share Physical Space 1.2. Synchronous Gestures 2. RELATED WORK 2.1. Related Research for Synchronous Gestures 2.2. Physical Transports & Tagging 2.3. Proximity and Near Field Communication 2.4. Pick and Drop 3. BUMPING AS A SYNCHRONOUS GESTURE 3.1. Detection of Bumping 3.2. Arbiter’s Role 3.3. Social Implications of Bumping 4. ARMS-LENGTH STITCHING 4.1 Connection-Action Phrasing with Arms-Length Stitching 4.2 Recognition of Arms-Length Stitching Gestures 4.3 Determining the Geometry of Arms-Length Stitching 4.4 Arms-Length Stitching Usability Study 4.5 Social Implications of Arms-Length Stitching 5. COOPERATIVE STITCHING 5.1. Design Space of Cooperative Stitching Gestures - 6 - 5.2. Graphical Feedback for Cooperative Stitching Gestures 5.3. Cooperative Stitching Usability Testing 5.4. Connection Control for Security and Privacy 6. BLUERENDEZVOUS 6.1. Connection-Action Phrasing in BlueRendezvouz 6.2. Details of the Synchronization / Connection Process 6.3. Social Implications of BlueRendezvouz 7. BLUETABLE 8. DISCUSSION 8.1. Support for Actions at Different Proxemic Distances 8.2. Workflow 8.3. Privacy and Security 8.3. Eight Design Questions for Ad-Hoc Multi Device Environments 9. FUTURE WORK 10. CONCLUSIONS - 7 - 1. INTRODUCTION The world of technology is a rapidly evolving ecology of devices. This ecology of ubiquitous devices is changing how we employ computing (Weiser, 1991). It includes tabletop and wall displays, tablet computers, handheld devices, cell phones, cameras, music players, wristwatches, and ear buds. Wireless networks have the potential to unite these ubiquitous computational pieces into temporary composite devices that suit a particular task, or that serve a brief collaborative encounter between users, and then dissipate when a task is complete. To realize this potential, users need interaction techniques to marshal spontaneous combinations of the desired devices, input capabilities, and display resources. For example, people at a meeting could connect their mobile devices to one another or to existing infrastructure, such as an interactive tabletop computer, to form an ad-hoc heterogeneous multi-display environment. Such an environment facilitates collaborative sharing of files and resources with other devices of the user’s choosing, and enables new functionality emerging from the combined capabilities of multiple devices. The demands of casual gatherings of heterogeneous multi-device environments differ from traditional single-user desktop computer scenarios in that input, display, and device resources are transient and may fluctuate in their roles with respect to one another. Because of their ad- hoc nature, such multi-device environments cannot a priori determine the participating devices’ position, orientation, availability throughout time, or the role each device is to play in a given user task. These are all subject to a user’s changing whims as he or she collaborates with multiple persons, or with a single person on multiple aspects of a collaborative task. - 8 - A dynamic multi-device environment must respect the underlying social fabric of collaborative interaction by providing flexible means for co-located users to work together across a range of physical proximity and relative body orientation. For example, users may wish to (1) view content side-by-side with a co-worker on a shared display, (2) negotiate terms in a face-to-face meeting with a rival seated at the opposite side of a table, or (3) share information with some colleagues on the other side of a crowded meeting room – but not with rivals at the meeting. A technique that requires the devices to rest in close physical proximity at a head-to-head relative orientation might support scenario (2) – provided the table is narrow enough to reach across – but cannot effectively support scenarios (1) or (3). Yet the emerging Near Field Communication (NFC) standard (http://en.wikipedia.org/wiki/Near_Field_Communication),