Using Multiuser Multi-Touch Tables to Visualize GIS Data

MITSUBISHI ELECTRIC RESEARCH LABORATORIES Cambridge, Massachusetts, USA

Using Multi-User Multi-Touch Tables to Visualize GIS Data

Alan Esenther, Kathy Ryall Technology Lab Mitsubishi Electric Research Labs, Cambridge, MA, USA

Masakazu Furuichi Modeling & Simulation, Information Systems Dept Mitsubishi Electric Corp, Kamakura, Japan Petroleum User Group 2007 MITSUBISHI ELECTRIC RESEARCH LABORATORIES

DiamondTouch (DT): Novel Technology Platform Overcoming limitations of the mouse… – Multi-user • Simultaneous multiple users • Supports up to four users at a time –Multi-touch • Each user can touch with multiple fingers at the same time • Enables rich gestures (unlike mouse/stylus) – Toucher identification • It doesn’t just allow multiple touchers – it distinguishes them • Identifies which user is touching each point (unlike tables based on cameras, IR, or pressure) – Direct-input • Input and Display spaces are superimposed • Allows simplicity of directly touching and MITSUBISHI interacting with content ELECTRIC Changes for the better Petroleum User Group 2007 MITSUBISHI ELECTRIC RESEARCH LABORATORIES

DiamondTouch: Hardware Overview • Basic Components – Table (touch surface) is a transmitter array Receiver – Chairs (or a conductive pad) Transmitters are receivers • User capacitively couples a signal between the table and the receiver – Supports seated or standing interaction – Stylus-based interaction also possible – Transmitter signals are interpolated to 2736x2048 input resolution • DT most-often used as horizontal tabletop display – Front-projection from above (any projector will do) – Can be embedded in other forms (e.g. vertical, dashboard, etc) • Powered by 5 volt USB cable MITSUBISHI ELECTRIC Changes for the better Petroleum User Group 2007 MITSUBISHI ELECTRIC RESEARCH LABORATORIES

Multi-User? Multi-Touch? Toucher ID? Direct Input?

MITSUBISHI ELECTRIC Changes for the better Petroleum User Group 2007 MITSUBISHI ELECTRIC RESEARCH LABORATORIES

Multi-Touch: Good for Rich Gestural Interactions • Supports all mouse functionality – including mouse-overs (vs mouse-drag), right/middle-drag, scroll-wheel, precision-input, etc – first/only direct touch surface to achieve this • Can build new functionality on top of mouse – e.g. fist-swipe or fist-drag can be mapped to anything • Flicking with 2 fingers • Simultaneously specify multiple points – e.g. piano chords - can’t do this with a mouse – e.g. Change the size and location of a region simultaneously

MITSUBISHI ELECTRIC Changes for the better Petroleum User Group 2007 MITSUBISHI ELECTRIC RESEARCH LABORATORIES

Mouse Emulation Challenges + Current Solutions • Specifying a particular pixel, finger obscuring content, moving the mouse without dragging: – “Precision-Hover” mode • Timing and spacing (for double-tapping ,etc) – DiamondTouch settings independent of mouse settings (customizable) • Right and middle mouse buttons – Tap with second finger to toggle right mouse button – Tap twice with second finger to toggle middle mouse button • Desktop Incorporation – System tray icon with context menu; special gesture for muting;

audio feedback MITSUBISHI ELECTRIC Changes for the better Petroleum User Group 2007 MITSUBISHI ELECTRIC RESEARCH LABORATORIES

Customer Usage • Modeling & Simulation, Information Systems Dept, Mitsubishi Elec Corp, Kamakura, Japan • In Development: “Decision Support System for Disaster Response”

• Used ArcGIS with maps and satellite images of December 2004 flooding in Indonesia • Collaboratively analyzed on a DiamondTouch table

MITSUBISHI ELECTRIC Changes for the better Petroleum User Group 2007 MITSUBISHI ELECTRIC RESEARCH LABORATORIES

Decision Support System for Disaster Response

• Used before-after image processing to inform damage estimates

Decision Support System for Disaster Response

• Combined with damage reports from on-site personnel

• and inspection of satellite data MITSUBISHI ELECTRIC Changes for the better Petroleum User Group 2007 MITSUBISHI ELECTRIC RESEARCH LABORATORIES

Decision Support System for Disaster Response • To collaboratively determine usable routes and helicopter landing locations

MITSUBISHI ELECTRIC Changes for the better Petroleum User Group 2007 MITSUBISHI ELECTRIC RESEARCH LABORATORIES

Emergency Response and Situational Awareness

• Shared situational awareness: – All participants perceive elements and comprehend meaning/implications – can only be achieved with effective communications. • A shared tabletop environment allows for important face-to-face interactions without which subtleties of communication (gestures, expressions) could be lost. • Direct Input – clearly communicates to other participants who is doing what. Facilitates building shared mental model. • Simultaneous users – In an emergency, you don’t want to take turns. • With customizations, everyone can do things at the same time. • If using the mouse, turn-taking is required, but a first-to-touch-wins policy avoids chaotic mouse cursor movement. • Multi-finger input – Key to rich gestures for advanced functionality

MITSUBISHI ELECTRIC Changes for the better Petroleum User Group 2007 MITSUBISHI ELECTRIC RESEARCH LABORATORIES

Enhancing existing applications

• Customizing function – Drawing with different symbols (emergency unit deployments, routes) – Intelligent Undo – “Privileged” buttons (e.g. emergency Coordinator has final say) • Customizing content – Menus, bookmarks, tooltips and help tailored to the particular toucher’s professional background and to the current situation • Customizing appearance – Change font, color, language, size, thickness, orientation based on who is touching • Customizing group input – Simultaneous touches required to proceed – Modal input sequences: one person can annotate while another is erasing and another is browsing different map layers – Logging and audit trail creation

MITSUBISHI ELECTRIC Changes for the better Petroleum User Group 2007 MITSUBISHI ELECTRIC RESEARCH LABORATORIES

Combining DiamondTouch with ESRI Software

• 3 Directions 1 - Run existing ESRI software as is (convert touches to mouse input)

2 - Write extensions to existing ESRI software to add multi-user/multi- touch capabilities

3 - Use ArcEngine or ArcWeb Services to create new multi-user/multi-touch utilities and applications

MITSUBISHI ELECTRIC Changes for the better Petroleum User Group 2007 MITSUBISHI ELECTRIC RESEARCH LABORATORIES

1. DiamondTouch & Unmodified ESRI applications: DTMouse Utility

• DiamondTouch Mouse Emulator – Allows you to run unmodified ESRI (or any) software on the table as is – Converts touch inputs to mouse inputs – Provides functionality of a 3-button mouse and mousewheel – Allows precision input – Coordinates multiple touchers • First to touch is the mouse – others are ignored – One user at a time, but no need to explicitly hand off control – no physical device (i.e., mouse, keyboard) to pass back and forth MITSUBISHI ELECTRIC Changes for the better Petroleum User Group 2007 MITSUBISHI ELECTRIC RESEARCH LABORATORIES

2 - DiamondTouch & Extending Existing ESRI Applications • Example Prototype – DiamondTouch extension for ArcMap – Works in conjunction with DTMouse mouse emulator

– Provides special buttons for simultaneous operations (line and marker symbol creation)

MITSUBISHI ELECTRIC Changes for the better Petroleum User Group 2007 MITSUBISHI ELECTRIC RESEARCH LABORATORIES

3 - DiamondTouch & ESRI: New Application Development • Use ArcEngine or ArcWeb Services to create new multi-user/multi-touch utilities and applications

• Example: Lensing to reveal different layers. • Example: Simultaneously zooming sub- windows into different regions

• Future work: Exploit DiamondTouch capabilities within ESRI-based apps; e.g.: – Simultaneous input – Audit and logging – Permission/control on a per-user basis – Visualizing parallel geoprocessing

MITSUBISHI ELECTRIC Changes for the better Petroleum User Group 2007 MITSUBISHI ELECTRIC RESEARCH LABORATORIES

Challenges/Issues/Findings

• Mouse concepts are deeply embedded in modern operating systems – There is only one mouse focus – O.S. can’t handle 2 mouse-down events in a row, without a mouse-up (e.g. scrollbars) • Integration with O.S. is critical – Need mouse emulation often for existing buttons, menus, etc – Need to suppress the mouse when custom software is interpreting the touch • Rotating content to face different users – Especially text – COM-based software assumes normal orientation • clipping regions, windows, invalidation regions, etc are all rectangular • ArcGIS exposes a lot of useful APIs • ArcGIS computation and rendering times are too slow to do in-place geo-processing • The ideal tabletop is big enough for multiple people to sit at, but small enough so that everyone can reach anywhere on it MITSUBISHI ELECTRIC Changes for the better Petroleum User Group 2007 MITSUBISHI ELECTRIC RESEARCH LABORATORIES

Multi-surface GIS

• Synchronized content • Synchronized displays

• WebService-based synchronization of different applications (shown here on same surface) MITSUBISHI ELECTRIC Changes for the better Petroleum User Group 2007 MITSUBISHI ELECTRIC RESEARCH LABORATORIES

Summary • DiamondTouch – Novel Direct-Touch development platform – Multi-user, Multi-touch, User-Differentiation • Rich gestures require reliable multi-finger detection • Careful integration of full-featured mouse emulation is critical • Reliable multiple-user support requires ability to distinguish touchers • Several ways to integrate with GIS applications

The computer mouse will continue to be the primary input mechanism, but be aware of it’s inherent limitations

Technology should not get in the way: debris-tolerant, walk-up interaction style, portable/durable

MITSUBISHI ELECTRIC Changes for the better Petroleum User Group 2007 MITSUBISHI ELECTRIC RESEARCH LABORATORIES

Questions? http://www.merl.com/projects/DiamondTouch/

http://diamondtouch.merl.com/stuff/

esenther @ merl.com

MITSUBISHI ELECTRIC Changes for the better Petroleum User Group 2007