Multi-Touch, Consumers and Developers Charles Roberts Media Arts & Technology Program University of California, Santa Barbara email: [email protected] Abstract tantly, developers have been able to em- brace the technology as a new paradigm Multi-Touch capable devices have just in interactivity. These technologies in- begun to hit their stride in the consumer clude consumer devices from the 1990s marketplace. It is a relatively new haptic such as the iGesturePad by Fingerworks paradigm that provides for interface wid- [2], to more recent consumer devices gets that were not previously realizable. such as the Apple iPhone [3], to large Current multi-touch devices vary greatly scale devices found in the work of Jeffrey in their target audience, with some focus- Han [4] and in the new Surface platform ing on consumers and others focusing by Microsoft [5]. on expert users and developers. Devel- opment environments for multi-touch en- The advent of these devices poses ques- vironments continue to be refined along tions about the benefits of multi-touch, with a standardized vocabulary of ges- the challenges of developing new models tures. of interactivity which employ multi-touch and the challenges that developers face Keyword: multi-touch, interface, gesture, in order to include these new capabilities development in their applications and prototypes. Introduction This paper examines previous attempts to answer questions about these issues, Touch sensitive control devices have and discusses the authors recent experi- been developed and explored for dec- ence developing a touch based applica- ades; as one example, frustrated total tion for the Apple iPhone. internal reflection (FTIR), one of the tech- niques used in current multi-touch re- Why touch? Why multi-touch? search, traces its roots back to experi- ments from the 1960s [1]. It is only re- The immediate question facing develop- cently, however, that multi-touch tech- ers creating applications or devices utiliz- nologies have become widely available ing multi-touch is perhaps obvious but to consumers and, possibly more impor- crucial: what advantages does employing multi-touch gestural capabilities bring to screen are all vector, they are also infi- my product? nitely scaleable. By using pinching and expansion multi-touch gestures, users For many applications touch interfaces are able to zoom in on particular inter- are more intuitive than those driven by a face widgets. This is useful, for example, mouse. Painting shapes is a simple ex- if you want to control a fader or knob ample; most of us have finger painted at with greater precision than would be al- one time or another and are well accus- lowed by the widgets normal size. Thus, tomed to the idea of drawing lines with even though in some ways it is designed our fingers. Dragging items from one lo- to emulate a traditional hardware inter- cation to another is slightly less intuitive face, it still provides features (there are as this is not a action that we commonly others besides the aforementioned perform using a single finger; but it does zooming) that would not be possible us- happen (think of sorting coins as an ex- ing hardware that was not augmented ample). But what about scrolling through digitally. documents via finger swipes? What ges- tures should be employed for copy and In contrast to Dexter, the Lemur [7] by paste? Should such gestures be univer- JazzMutant is a highly configurable sal across all applications and devices? multi-touch device. Instead of automati- cally configuring its interface to work with Two multi-touch by the company Jazz- most major DAWs, the Lemur allows the Mutant [6] offer very different approaches user to create their own custom custom to solving this problem. In their product configurations using provided editor Dexter, the interface is a 12x12” multi- software. The Lemur ships with a library touch sensitive screen that is used with of standard interface widgets that users digital audio workstations (DAW) such as can arrange in any fashion they choose. Cubase, Logic and ProTools (Fig. 1). The It also contains its own physics engine, interface is designed to be similar in so that interface items (such as the mul- many ways to the those of the applica- tiball) can be controlled by gravity and tions it works with. This is ironic as much laws of attraction in addition to gestures of the interface of many DAWs is de- by the user. This allows users to “fling” signed to mimic hardware. We are thus items towards a general goal value rather left with hardware providing an interface than precisely placing them, and opens to software that is designed to emulate up novel methods of partially stochastic hardware. control when objects bounce off of con- tainer walls and gravitate towards each Fortunately, this is only the surface of other. Dexter (no pun intended). Since the graphics displayed on Dexter’s touch Fig 1. A vector ren- dering of the Jazz- Mutant Dexter inter- face. Note the simi- larity to traditional multi-channel hard- ware mixers. The configurable nature of the Lemur is Mass Market Multi-Touch Concerns probably less objectionable to the elec- tronic musician / visual artist than it The introduction of multi-touch to de- would be to the average consumer, as vices such as the iPod Touch and the people working in digital media are often iPhone has created the need for a stan- used to customizing their interfaces to dardized set of multi-touch gestures that meet their particular needs. From the can be used across applications, and, authors perspective, this customization is ideally, across devices. Apple took a a a critical part of creating any live per- giant step ahead in this arena when it formance setup. acquired the company Fingerworks [2] in 2005. Fingerworks had already devel- There is a history of electronic music oped a number of products using multi- pushing advances in interface technol- touch gestures, and had created a com- ogy. From early analog synths featuring prehensive gesture vocabulary to go with ribbon controllers to more recent devices them (Fig. 2). such as the Continuum [8] and the Le- mur, musicians and engineers have been It could be argued that the vocabulary exploring novel ways to allow performers suggested by Fingerworks is overly am- to express themselves via touch. When bitious, and the author posits that few we leave the niche of live performance, users would be willing to learn all the however, there is a very different set of gestures outlined in the Fingerworks concerns regarding multi-touch. doucmentation. Many of these gestures would be primarily used by power users. But the many of the gestures that would be used more commonly, such as the one to scroll vertically or horizontally, have been adopted and simplified by Apple. Since Apple products do not cur- rently employ the comprehensive set of gestures that the Fingerworks products possessed, the gestures that Apple products use can be simplified; the sim- plified gestures become more easily rec- ognizable and stratified as there are Fig 2. A small sample of the gestures fewer overall. Thus, scrolling moves from used in the original Fingerworks products, “Touch & slide four fingers up/down” as as taken from their reference manual. taken from the Fingerworks website [9] to the current two finger slide used in Apple It is in the best interest of consumers and products. There are currently ten different developers for a universal standard to be gestures used in the multi-touch track- created. pads shipped with the Apple MacBook Pro and the MacBook Air [10]. The Next Step for Touch Devices? It would be excellent if the gestures used In the author’s opinion, there is a great in Apple products were made into a deal more appeal to interfaces based on standard that could be used across ap- tangible mechanical devices than on vir- plications and devices. Unfortunately, tual widgets. By physically manipulating Apple has filed for patents regarding their real, tangible, physical entities, many multi-touch gestures [11]. It is not certain people feel that they are more connected at this time whether these patents will be to the interfaces they use. Current multi- granted, or, if they are granted, whether touch screens share a lack of haptic or not Apple will pursue violations. Apple feedback with traditional mouse based might be attempting to acquire a so- interfaces; the author would argue that called “defensive” patent; by obtaining overcoming this lack of haptic feedback the relevant patents themselves they en- would be an important next step in in- sure that other companies cannot obtain creasing the usability of touch screen in- them and then sue Apple. It is highly terfaces. likely that if Apple does patent multi- touch gestures that competing sets of There is an experiment using the vibra- gestures will come to the market; Synap- tion on the iPhone to attempt to provide tics, a maker of trackpads, is already such haptic feedback [12]. Everytime us- working on their own set of gestures [11]. ers click keys on the iPhone virtual key- board, vibration is triggered in order to able by April of 2008, potentially putting provide haptic feedback. In a poll on the it just around the corner [15]. internet [13], 28.4% of people respond- ing to the question “Would you download With the recent introduction of the a program to give your iPhone a haptic iPhone SDK, we have the first concrete keyboard?” responded yes, while 26.7% example of how developers might be ex- responded no. Clearly, vibration alone is pected to handle interactivity within a not enough to create the type of haptic multi-touch enabled device.