Sidebyside: Ad-Hoc Multi-User Interaction with Handheld Projectors
Total Page：16
File Type：pdf, Size：1020Kb
SideBySide: Ad-hoc Multi-user Interaction with Handheld Projectors Karl D.D. Willis 1,2 Ivan Poupyrev 1 Scott E. Hudson 3,1 Moshe Mahler 1 1 Disney Research Pittsburgh 2 Computational Design Lab, 3 HCI Institute & 4720 Forbes Avenue Heinz College Center for the Future of Work, Pittsburgh, PA 15213 Carnegie Mellon University {karl, ivan.poupyrev, moshe.mahler} 5000 Forbes Avenue, @disneyresearch.com Pittsburgh, PA 15213 [email protected] ABSTRACT We introduce SideBySide, a system designed for ad-hoc multi-user interaction with handheld projectors. SideBySide uses device-mounted cameras and hybrid visible/infrared light projectors to track multiple independent projected images in relation to one another. This is accomplished by projecting invisible fiducial markers in the near-infrared spectrum. Our system is completely self-contained and can be deployed as a handheld device without instrumentation of the environment. We present the design and implementa- tion of our system including a hybrid handheld projector to project visible and infrared light, and techniques for track- ing projected fiducial markers that move and overlap. We introduce a range of example applications that demonstrate the applicability of our system to real-world scenarios such Figure 1: Interacting with the SideBySide system. as mobile content exchange, gaming, and education. each other in shared interactive spaces. Handheld projectors ACM Classification: H5.2 [Information interfaces and are an enabling technology that could realize this vision. presentation]: User Interfaces. They are sufficiently small to be grasped in a single hand, General terms: Design, Human Factors, Experimentation and light enough to be moved from place to place. Multiple Keywords: Interaction techniques, handheld projector, pico users can project digital content directly into the physical projector, multi-user, ad hoc interaction, games. environment. The relatively large size of projected images allows them to be easily seen by multiple users, making INTRODUCTION them a particularly natural fit for co-located multi-user in- Handheld computing devices have become a ubiquitous teraction (e.g. Figure 1). part of modern life. They allow us to communicate, re- trieve, and record information at any time, view digital con- There has been rapidly growing interest in handheld projec- tent on the go, and play games in any location. Interaction tors from both the industry and research communities with handheld computing devices, however, remains a [17,18], however, most work has focused on single user largely solitary, single user experience. Today’s devices do applications [e.g. 1]. Although there has been some work not typically provide interfaces and supporting technologies exploring the potential of handheld projectors for multi- for co-located multi-user work, learning, and play – crucial user interaction [2,7], these systems require instrumentation elements of human interactions in the real world. of the surrounding environment with often complex sensing infrastructure. This significantly reduces the usefulness of Our research is motivated by the vision of handheld com- multi-user handheld projection systems in any real world puting devices that allow users to dynamically interact with scenario where user mobility is required. We are unaware of any system allowing multiple handheld projectors to Permission to make digital or hard copies of all or part of this work for interact together without instrumenting the environment. personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies We present SideBySide, a system designed for ad-hoc bear this notice and the full citation on the first page. To copy otherwise, multi-user interaction with handheld projectors. SideBySide to republish, to post on servers or to redistribute to lists, requires prior does not require instrumentation of the environment and specific permission and/or a fee. can be used almost anywhere. It uses a device-mounted UIST’11, October 16–19, 2011, Santa Barbara, CA, USA. Copyright © 2011 ACM 978-1-4503-0716-1/11/10... $10.00. ! infrared (IR) camera and a novel hybrid visible/IR light ! handheld projector. The SideBySide software platform use in a stationary position and are not suitable for use tracks multiple independent projected images in relation to when moving between different locations. The recent one another using invisible fiducial markers projected in emergence of handheld projectors offers a lightweight solu- the near-IR spectrum. The resulting system allows a broad tion for unbounded mobile interaction. range of new interaction scenarios where users are not tied Multi-user Handheld Projector Systems to a fixed location. Exchanging digital content, such as per- Early pre-cinema use of handheld projectors stems from the sonal contact information or media files, can be initiated development of the Magic Lantern in 17th century Europe without any infrastructure. Multiple users can play pro- and its later adaption in Japan for the Utsushi-e perform- jected games together by creating ad-hoc gaming spaces. ance [25]. This performance involved small wooden slide Interaction scenarios for education allow teachers and stu- projectors used from behind a large rice paper screen to act dents to study together using a shared information space. out a story for an audience. Multiple performers would Most importantly, these interaction scenarios can happen coordinate their movement to change the size and location anywhere – in an elevator up to a business meeting, in a of projected images. This enabled the creation of relatively child’s bedroom after lights out, or in the hallway outside a complex real-time animation in the pre-cinema era. This classroom. important pre-history of handheld projector-based interac- We present the following contributions: tion underlines the potential for multiple small projectors to 1. We outline the design and implementation of our interact together in a seamless way. hardware system, including a hybrid handheld pro- The relatively large public displays created by modern jector unit that projects both visible and IR light. handheld projectors make them ideal for multi-user interac- 2. We introduce our software system for tracking mul- tion scenarios. Sugimoto et.al. created a mockup system tiple projected images in relation to one another with where two overlapping projection screens were used with a invisible fiducial markers. In particular we present PDA touch screen to initialize file transfer between devices techniques for tracking overlapping fiducial markers [21]. An exhaustive range of multi-user interaction tech- and optical communication between devices. niques were developed by Cao et.al. using a motion capture system for location tracking [2]. These interactions focus 3. We present a range of applications that demonstrate on operations within a virtual workspace, such as content the exciting and engaging multi-user interaction sce- ownership, transfer, viewing, and docking. narios possible with the SideBySide system. Multi-user games have also been developed that allow us- RELATED WORK ers to work together to reach a goal. Hosoi et.al. developed Research related to the SideBySide system can be divided a multi-user handheld projector game for guiding a small primarily into three categories: projector-based augmented robot along a projected path [7]. Users line up pieces of spaces, multi-user handheld projector systems, and invisi- track for the robot to follow and reach its goal. Cao et.al. ble marker tracking. developed a multi-user jigsaw game where users would Projector-Based Augmented Spaces pick up and place pieces of a puzzle together [3]. An important aspect of multi-user interaction is extending To enable interaction between multiple handheld projec- interaction beyond a single user’s screen and into the envi- tors, these systems rely on infrastructure being added to the ronment. Rekimoto’s Pick-and-Drop system defined inter- environment. This ranges from a fixed camera above the action techniques to support data transfer between multiple interaction area [7], to a professional motion capture sys- computers and handheld devices [15]. The Augmented Sur- tem [2]. Relying on fixed infrastructure within the envi- faces project later established a direct spatial relationship ronment severely limits where handheld projectors can be between laptop screen content and content projected onto used, substantially limiting their mobility. Our vision is to nearby surfaces [16]. enable multiple users to interact side-by-side, anywhere, in The use of arbitrary surfaces for content projection was any space. This is one of the main design considerations for explored with the Everywhere Displays Projector [14] to this research, and one that strongly differentiates our tech- augment indoor spaces with projected content at any loca- nical solution from past work. tion using a steerable projector. The Play Anywhere system Invisible Marker Tracking used a portable short-throw projector to create interactive Fiducial markers have been used widely for location track- tabletop experiences with handheld devices and tangible ing due to their lightweight, robust performance. A well- objects [27]. The Bonfire system also offered a portable known issue with structured, 2D barcode-style fiducial form-factor to extend the desktop computing experience markers is their unnatural appearance that users cannot