Integrating 2D Mouse Emulation with 3D Manipulation for Visualizations on a Multi-Touch Table Luc Vlaming,1 Christopher Collins,2 Mark Hancock,3 Miguel Nacenta,4 Tobias Isenberg,1,5 Sheelagh Carpendale 4 1University of Groningen 2University of Ontario Institute of Technology fl.vlaming@ j [email protected] rug.nl [email protected] 3University of Waterloo 4University of Calgary 5DIGITEO& [email protected] fmanacent j [email protected] CNRS/INRIA ABSTRACT We present the Rizzo, a multi-touch virtual mouse that has been designed to provide the fine grained interaction for information visualization on a multi-touch table. Our so- lution enables touch interaction for existing mouse-based visualizations. Previously, this transition to a multi-touch environment was difficult because the mouse emulation of touch surfaces is often insufficient to provide full informa- tion visualization functionality. We present a unified design, combining many Rizzos that have been designed not only to provide mouse capabilities but also to act as zoomable lenses that make precise information access feasible. The Rizzos and the information visualizations all exist within a touch-enabled 3D window management system. Our ap- proach permits touch interaction with both the 3D windowing environment as well as with the contents of the individual windows contained therein. We describe an implementation Figure 1: With our system, a person can interact of our technique that augments the VisLink 3D visualization with information visualizations using the Rizzo, a tool environment to demonstrate how to enable multi-touch capa- designed to enable mouse interaction. bilities on all visualizations written with the popular prefuse visualization toolkit. search into practical and collaborative information-intensive applications. These applications have their own requirements ACM Classification: H5.2 [Information interfaces and pre- for interaction and usually have fine-grained information sentation]: User Interfaces—Graphical user interfaces, inter- aspects that require a high input resolution. In a traditional action styles. desktop environment these information-rich visualizations would commonly make use of mouse interaction to provide General terms: Design, Human Factors this fine-grained information access. Our challenge is to Keywords: virtual mouse, multi-touch, information visual- move these detail-intensive information visualizations into ization, touch-interaction with 3D environments. a multi-touch tabletop environment, gaining some of the freedoms and advantages of multi-touch interaction without INTRODUCTION losing the fine-grained information access. With the recent surge of touch technology [10, 14, 26, 32, 34] it is increasingly possible to create multi-touch environments To create an environment that functions coherently as a that support collaboration and enable interfaces with rich, multi-touch interaction space and at the same time provides direct manipulation. However, there is still relatively little re- detailed and precise information access, we designed a tool, called the Rizzo (Fig. 1), which is a virtual mouse that trans- Permission to make digital or hard copies of all or part of this work for lates the multi-touch interactions on the surface to mouse personal or classroom use is granted without fee provided that copies are input on the visualizations. To explore this design challenge not made or distributed for profit or commercial advantage and that copies we chose to work with an existing information visualization bear this notice and the full citation on the first page. To copy otherwise, to system, VisLink [8], that provides an interactive, compara- republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. tive visualization environment, which can hold any number ITS 2010, November 7–10, 2010, Saarbrucken,¨ Germany. of visualizations on independent panels in a 3D space. Be- Copyright 2010 ACM 978-1-4503-0399-6/10/11...$10.00. cause we work with VisLink, we enable the use of any infor- not discuss how to interact with the information contained within the artifact (e. g., select a word, copy & paste). In our approach we integrate 3D multi-touch manipulation with the ability to interact directly with the information contained by these artifacts. Hence, we use both the sticky fingers and opposable thumbs technique [16] and precise constrained manipulations, as well as mouse-like control of panels con- taining visualizations. 3D Window Management Systems and Multi-Touch The placing of objects containing information into a 3D envi- ronment as done in VisLink [8] is reminiscent of 3D window management systems including Miramar [24], Metisse [7], BumpTop [2], and Caleydo [23, 37]. Bringing these hy- brid 2D-within-3D window management systems to a multi- Figure 2: Rizzos provide resolution-aware mouse touch interaction setup (e. g., a multi-touch tabletop display) emulation for 2D visualizations embedded on panels promises to enhance the experience over current mouse- in the 3D environment. based interactions. Multi-touch input offers more degrees of freedom, therefore providing greater flexibility in the design mation visualizations written with the widely used prefuse of interaction techniques. Furthermore, tabletop displays toolkit [18]. These visualizations are integrated in our system promote collaboration, an important aspect of many scenar- through panels within VisLink’s 3D space, a setup that places ios, including information visualization analysis [19]. The high demands on the design of the Rizzo, since our tool needs traditional ways of interacting with 3D window management to be general enough to work with any visualization, and si- systems and VisLink [8], however, are mouse- and keyboard- multaneously enable effective multi-touch manipulation. We based, so switching to multi-touch interaction introduces explore this design space of mixed 2D and 3D interaction that issues such as precision and dedicated mouse mappings when allows people to interact in a unified multi-touch interaction interacting with window content. In fact, previous attempts continuum for information-intensive tabletop applications. to add multi-touch interaction to 3D window management Our main contribution is the Rizzo (Fig. 2); a tool that systems (e. g., BumpTop) only offer mappings for very spe- enables precise selection and manipulation of information el- cific information types. We aim to combine the advantages ements within the 2D panels using multi-touch input. We also and generality of multi-touch and mouse interaction. contribute a group of interaction techniques that allow the Precise Tabletop Interaction and Mouse Emulation 3D manipulation of information visualization panels. This Previous studies comparing indirect mouse to direct-touch includes controlling the 3D view and the 3D layout and being input [13] showed that a mouse may be more appropriate for able switch to and from a 2D-only view. unimanual input tasks, and fingers for a bimanual input tasks. We first explain our design decisions and describe the in- However, on digital tabletop settings it is often awkward to tegration of 3D navigation, 3D manipulation of the panels, provide additional room for mice, while combined relative and 2D visualization interaction within the panels. We then and absolute pointing techniques [12] using pen input cannot discuss how our system would be used in practice, provide make use of multi-touch capabilities. Thus, we explore an informal evaluation, and also discuss the benefits and a setting that allows us to make use of both direct-touch limitations of our approach. manipulations of 3D objects and indirect mouse emulation. RELATED WORK One of the advantages of mouse interaction over touch input Our research builds upon three main areas of interest: multi- is its high precision. This issue has been addressed by several touch manipulation on interactive surfaces; integrated 2D/3D techniques such as virtual key tapping, a gain-based cursor window management systems; and precise interaction and [3], stretching methods, menus for parameter setting [5], mouse emulation on multi-touch surfaces. localized views [36], and a rubbing gesture for zoom control [30]. We integrate high-precision control into our mouse Multi-Touch Manipulation emulation via a lens with controllable zoom and a non-linear The introduction of multi-touch technology on digital sur- control-display gain. faces [6, 10, 14, 26, 32, 34] has led to a variety of techniques for manipulating virtual artifacts with one’s fingers. Many Several other techniques exist that enable mouse interaction techniques have since been introduced for moving, rotating, on multi-touch displays. Esenther et al. [11] presented the and scaling 2D artifacts with one [22, 33] and several fingers Fluid DTMouse, an interaction technique in which the num- [17]. These general ideas have been extended to the manip- ber of touch points defines the mouse operation to apply. ulation of 3D objects on touch surfaces with two or more Matejka et al. [25] improved on the Fluid DTMouse and fingers [15, 16, 31, 40]. As an alternative to this, it is possible recommend SDMouse, a set of techniques to enable three- to use simulated physics to control objects [38, 39]. button mouse capabilities through touch gestures. We build on these techniques but realize mouse emulation through a While these techniques allow us to manipulate and transform dedicated tool rather than through gestures.