Scroll, Tilt Or Move It: Using Mobile Phones to Continu- Ously Control
Total Page：16
File Type：pdf, Size：1020Kb
Scroll, Tilt or Move It: Using Mobile Phones to Continu- ously Control Pointers on Large Public Displays Sebastian Boring Marko Jurmu Andreas Butz University of Munich MediaTeam Oulu, Dept. of University of Munich Amalienstr. 17 Electr. and Inf. Eng. Amalienstr. 17 80333 Munich, Germany University of Oulu, Finland 80333 Munich, Germany [email protected] [email protected] [email protected] ABSTRACT On the other hand, small portable devices such as per- Large and public displays mostly provide little interactiv- sonal digital assistants (PDAs) and mobile phones are ity due to technical constraints, making it difficult for available at low prices and are by now usual companions people to capture interesting information or to influence in our everyday lives. Despite their limited visual output the screen’s content. Through the combination of large- capabilities compared to large screens, they come with scale visual output and the mobile phone as an input de- various built-in input options such as a joystick, a stylus, vice, bidirectional interaction with large public displays a keypad or touch-screens. Hence, combining the users’ can be enabled. In this paper, we propose and compare mobile devices and large public displays, for example, three different interaction techniques (Scroll, Tilt and allow the control of a personal pointer on the public dis- Move) for continuous control of a pointer located on a play, turning the phone into a ubiquitous input device for remote display using a mobile phone. Since each of these large screens (Ballagas et al., 2006). The interaction tech- techniques seemed to have arguments for and against niques need to be efficient, enjoyable and easy to learn. them, we conducted a comparative evaluation and dis- Unfortunately, each of them has its limitations in terms of covered their specific strengths and weaknesses. We re- accuracy, operation speed, efficiency and ergonomics. port the implementation of the techniques, their design and results of our user study. The experiment revealed In this paper we discuss three different techniques to con- that while Move and Tilt can be faster, they also introduce trol a pointer on a large public display by using a selec- higher error rates for selection tasks. tion of mobile device input and sensing capabilities. We developed three distinct strategies for continuously con- Author Keywords trolling the user’s pointer: first, the pointer can be moved Optical flow, accelerometers, input techniques/mappings, constantly by pressing the phone’s joystick in the respec- target acquisition, fatigue, cursor control. tive direction (Scrolling). Second, the pointer is acceler- ated by tilting the mobile phone in the desired direction ACM Classification Keywords (Tilting), and third, the pointer’s movement is linearly H.5.2 [Information Interfaces and Presentation]: User mapped to the phone’s movement (Moving). We discuss Interfaces – Evaluation/methodology, Haptic I/O, Input the different input strategies followed by an extensive devices and strategies, Interaction Styles, Prototyping; evaluation in which we analyze the feasibility of each D.2.2 [Software Engineering]: Design Tools and Tech- input mapping in detail. Based on the findings in our niques: User Interfaces evaluation, we provide insights on how to design pointer interaction on large screens using mobile devices. INTRODUCTION Particularly large display technologies are starting to ap- RELATED WORK pear in our everyday lives. Not only can they be found in The usage of mobile devices for interaction with large private environments (e.g., large-scale TVs, projectors or displays – static (paper-based) or dynamic (screen-based) monitors), but more and more in public spaces such as – has been an active research focus in the past years (Bal- airports, subway stations or shopping malls, where they lagas et al., 2006). If users interact with static displays mostly act as large ambient information displays. Instead such as maps, the phone usually acts as a lens, digitally of allowing users to capture interesting information or augmenting the current focus area (Rohs et al., 2007). influencing the display’s content, the usage models of Others investigate the remote control of pointers (Myers these displays tend to be static, offering only unidirec- et al., 1998) and interaction (Want et al., 1995) on large tional broadcast without possibilities for bidirectional screens using mobile stylus-based interfaces. Others in- interaction. In addition, home-installed large screens, e.g., vestigated different issues related to large public display projectors or flat panels, mostly lack sophisticated input interaction. Ballagas et al. define three domains in which capabilities beyond those of a TV remote. this interaction takes place: personal, semi-public and public (Ballagas, 2004). Based on these, they identify constraints that need to be taken into account for design- OZCHI 2009, November 23-27, 2009, Melbourne, Australia. ing interaction techniques. Copyright the author(s) and CHISIG Additional copies are available at the ACM Digital Library An analysis of existing pointing interaction techniques (http://portal.acm.org/dl.cfm) or ordered from the CHISIG secretary shows a trend of using the phone’s camera in combination ([email protected]) with visual markers. Point & Shoot (Ballagas et al., 2005) OZCHI 2009 Proceedings ISBN: x-xxxxx-xxx-x enables users to select objects on a large screen by point- ing at them using the phone’s camera. Users can select Together with commercial applications such as the objects by pressing the phone’s select button. This results iPhone Air Mouse1 and the Nintendo Wii2, research has in a grid of visual markers being temporarily superim- been done utilizing acceleration sensors built into mobile posed on the large display’s canvas. The phone takes a phones (Vajk et al., 2007). One example is Toss-It picture and uses the visual markers in order to detect the (Yatani et al., 2005) which allows users to “throw” im- position the phone has been pointed at. SpotCode (Mad- ages onto a screen in a multi-display environment. How- havapeddy et al., 2004) also uses the principle of tracking ever, this technique only enables users to point at the visual patterns. Another system called Shoot & Copy screen they want to have the image on, but more accurate (Boring et al., 2007) allows users to take a picture of the operations seems to be cumbersome. MobiToss (Scheible desired content on a secondary screen. Using image proc- et al., 2008) utilizes the built-in sensors in two ways: essing techniques, the captured content gets recognized First, users can “throw” a recently captured video clip and is then sent back to the user. However, neither tech- onto a nearby screen. Second, users can then manipulate nique offers permanent visual feedback and hence makes the clip by tilting the phone (e.g., applying different video continuous operations cumbersome. Furthermore, both effects). However, the usage of acceleration sensors for systems keep the controls on the mobile device possibly pointer control has not been examined. leading to macro attention shifts (Holleis et al., 2007). INTERACTION TECHNIQUES Instead of using visual markers, experiments with optical For controlling a pointer on the display, today’s personal flow analysis of the video stream captured by a camera computers offer a standard mouse or equivalent devices, have been conducted. Pears et al. (Pears et al., 2008) in- such as track points or touch pads. While suitable for sin- troduce a mechanism that allows pointing directly on the gle users in desktop environments, the single pointer screen. However, their system requires the phone’s cam- paradigm needs to be extended for use in public spaces. It era to be pointed towards the screen at all times, which nevertheless provides an acceptable mental model as most likely increases fatigue in the user’s arms (also starting point for collaborative or concurrent use. Touch- known as the “gorilla-arm-effect” occurring when inter- based input on large public displays is not always an op- acting with vertically mounted touch screen monitors). tion due to protection reasons (e.g. against vandalism) as Sweep (Ballagas et al., 2005) mimics the behavior of a well as their relatively large dimensions. To interact with standard desktop mouse. When the user moves the phone public screens instead of just passively viewing the in- into a certain direction, the pointer on the large screen formation, we have developed three interaction tech- moves in the same direction (with a linear factor). A fur- niques based on different input mappings, using the tech- ther example is Direct Pointer (Jiang et al., 2006). Their nologies built into a modern mobile phone. system requires the camera to be pointed at the screen at all times. While this reduces distance-related problems of Scrolling Sweep, users might experience fatigue in their arms, lead- The simplest control is to use a mapping from key presses ing to short interaction cycles. As both systems allow a to a constant motion: the cursor is either moving at con- person to use the personal mobile phone as optical mouse, stant speed in a certain direction or remains still. A com- we based one of our interaction techniques on optical mon approach used in past research is to utilize the flow to allow a direct comparison. phone’s joystick or arrow keys (Silfverberg et al., 2001). While optical flow analysis relies on the phone’s camera This technique gives both direction and movement at the (inside-out tracking), other systems use fixed cameras to same time with a control display (CD) ratio of 1. If the observe the phone’s motion (outside-in tracking). Mi- user presses an arrow (or tilts the joystick) in one direc- yaoku et al. introduce C-Blink (Miyaoku et al., 2004) tion, the cursor moves at constant speed in this direction which represents such a system.