MITSUBISHI ELECTRIC RESEARCH LABORATORIES http://www.merl.com Fluid DTMouse: Better Mouse Support for Touch-Based Interactions Esenther, A.; Ryall, K. TR2006-001 May 2006 Abstract Although computer mice have evolved physically (i.e., new form factors, multiple buttons, scroll-wheels), their basic metaphor remains the same: a single-point of interaction, with modifiers used to control the interaction. Many of today’s novel input devices, however, do not directly (or easily) map to mouse interactions. For example, when using one’s finger(s) or hand directly on a touchable display surface, a simple touch movement could be interpreted as either a mouse-over or a drag, depending on whether the left mouse button is intended to be depressed at the time. But how does one convey the state of the left mouse button with a single touch? And how does one fluidly switchbetween states? The problem is confounded by the lack of precision input when using a single finger as the mouse cursor, since a finger has a much larger footprint than a single pixel cursor hotspot. In this paper we introduce our solution, Fluid DTMouse, which has been used to improve the usability of touch tables with legacy (mouse-based) applications. Our technique is applicable to any direct-touch input device that can detect multiple points of contact. Our solution solves problems of smoothly specifying and switching between modes, addressing issues with the stability of the cursor, and facilitating precision input. Advanced Visual Interfaces (AVI) This work may not be copied or reproduced in whole or in part for any commercial purpose. Permission to copy in whole or in part without payment of fee is granted for nonprofit educational and research purposes provided that all such whole or partial copies include the following: a notice that such copying is by permission of Mitsubishi Electric Research Laboratories, Inc.; an acknowledgment of the authors and individual contributions to the work; and all applicable portions of the copyright notice. Copying, reproduction, or republishing for any other purpose shall require a license with payment of fee to Mitsubishi Electric Research Laboratories, Inc. All rights reserved. Copyright c Mitsubishi Electric Research Laboratories, Inc., 2006 201 Broadway, Cambridge, Massachusetts 02139 Fluid DTMouse: Better Mouse Support for Touch­Based Interactions Alan Esenther, Kathy Ryall Mitsubishi Electric Research Laboratories 201 Broadway Cambridge, MA 02139 +1­617­621­7564 {esenther, ryall}@merl.com ABSTRACT 1. INTRODUCTION Although computer mice have evolved physically (i.e., new form Since its introduction almost forty years ago, the mouse concept factors, multiple buttons, scroll­wheels), their basic metaphor has been and continues to be a critical part of computer remains the same: a single­point of interaction, with modifiers interfaces. It is arguably the most ubiquitous part of any GUI. used to control the interaction. Many of today's novel input Mice have evolved, now having multiple buttons, scroll­wheels, devices, however, do not directly (or easily) map to mouse and other sensing devices. However, even as today's computing interactions. For example, when using one's finger(s) or hand moves off the desktop and into the environment, the concept of a directly on a touchable display surface, a simple touch movement mouse remains largely the same: a single­point of interaction, could be interpreted as either a mouse­over or a drag, depending with modifiers used to control the interaction. on whether the left mouse button is intended to be depressed at Newer touch­sensitive surfaces such as horizontal tabletops or the time. But how does one convey the state of the left mouse vertical whiteboards can allow users to interact with displayed button with a single touch? And how does one fluidly switch content more naturally and directly than is possible with a between states? The problem is confounded by the lack of computer mouse. Although some can detect simultaneous points precision input when using a single finger as the mouse cursor, of input, most applications, new and existing, are generally since a finger has a much larger "footprint" than a single pixel written assuming that a mouse is being used. While we note that cursor hotspot. In this paper we introduce our solution, Fluid new interaction paradigms are evolving and that we are actively DTMouse, which has been used to improve the usability of touch researching in this very area, we must recognize the importance tables with legacy (mouse­based) applications. Our technique is of providing support for legacy applications in both today’s and applicable to any direct­touch input device that can detect future systems. Such support requires mouse­based input. multiple points of contact. Our solution solves problems of smoothly specifying and switching between modes, addressing One of the challenges of supporting such touch­sensitive surfaces issues with the stability of the cursor, and facilitating precision is to provide a natural mechanism for emulating mouse buttons input. on a surface that only detects touches. This is particularly problematic with “direct­touch” surfaces where the input and Categories and Subject Descriptors output are the same. For direct­input devices, the user is touching H.5.2 [User Interfaces]: Interaction styles (e.g., commands, the actual image of the content, rather than manipulating an menus, forms, direct manipulation). indirect input device such as a mouse or a touchpad off to the side. In these touch­based systems, there is a difficulty in General Terms interpreting a single­touch when used as a replacement for Design, Human Factors. mouse­input; a simple touch movement could be interpreted as either a mouse­over or a drag, depending on whether the left Keywords mouse button is intended to be depressed at the time. But how visual interaction, tabletop interfaces, mouse emulation, multi­ does one convey the state of the left mouse button with a single touch. touch? And how does one fluidly switch between states? For example, users may have to right­click to open a context menu, drag with the left mouse button to re­position content, double­ click to engage a button, or may simply wish to reposition the mouse without any additional input being generated. Permission to make digital or hard copies of all or part of this work for In addition to the mouse button problem, there is a second, personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies confounding problem – the lack of precision input when using a bear this notice and the full citation on the first page. To copy otherwise, or single finger as the mouse cursor. A finger has a much larger republish, to post on servers or to redistribute to lists, requires prior specific "footprint" than a single pixel cursor hotspot, making it permission and/or a fee. awkward, difficult, and at times impossible to interact with GUI AVI '06, May 23­26, 2006, Venezia, Italy. elements designed for mouse interaction. Copyright 2006 ACM 1­59593­353­0/06/0005...$5.00. In this paper we introduce Fluid DTMouse, our solution to the Tablet PCs [6] are also direct­input devices. They rely on a finger­based, direct­touch mouse problem; it has significantly special pen. These devices have the unusual ability to detect enabled the use of touch tables with legacy (mouse­based) “hovering” when the pen is near the surface but not actually in applications. While we have developed our techniques using the contact with the surface. So if the pen is hovering, then the DiamondTouch platform, they are applicable for any direct­touch mouse is simply moved, and if the pen is in contact with the input device that can detect multiple points of contact (e.g., [2] Tablet PC surface then dragging (left­mouse button down) is [11]) or future technologies with this capability. Our solution engaged. Right­clicking is supported by holding a button on the solves problems of smoothly specifying and switching between Tablet PC pen, or holding the stylus down briefly, or by selecting modes, addressing issues with the stability of the cursor, and a “right­click” menu item to indicate that the next tap should be facilitating precision input. Section 2 introduces the mouse interpreted as a right­click. It is the lack of this third state, emulation problems and discusses related work. Section 3 hovering (the first two states being touching or not touching), describes the Fluid DTMouse work and explains how it solves which makes mouse­over support so difficult on most touch key problems. Section 4 wraps up and discusses future and surfaces. For our research we were interested in solutions that ongoing work. would work with fingers, since requiring the use of a special pen or other hardware may not be as convenient – particularly for a 2. BACKGROUND & RELATED WORK large touch surface. It should be noted, however, that another One of the more fundamental challenges for direct­touch input is advantage of using a fine­tipped pen such as the Tablet PC pen is that users may wish to move a mouse cursor across a touch­ that the fine tip does not obscure content as much as a finger sensitive display without engaging any mouse buttons (e.g., does. At any rate, currently Tablet PCs are only available with simply move the mouse over a tooltip). But when a user touches very small surface areas, compared to tabletop and whiteboard a touch­sensitive surface, how is the system to know whether the surfaces. touch was intended to simply move the mouse, or to “drag” with The DiamondTouch [1] “DiamondTouch Mouse” [7] software the mouse by holding down the left mouse button during the supports a right­click by tapping with a second finger.