An Interaction Model for Designing Post-WIMP User Interfaces

Instrumental Interaction: An Interaction Model for Designing Post-WIMP User Interfaces

Michel Beaudouin-Lafon Dept of Computer Science University of Aarhus Aabogade 34 DK-8200 Aarhus N - Denmark [email protected]

ABSTRACT The leap from WIMP to newer "post-WIMP" graphical This article introduces a new interaction model called interfaces, which take advantage of novel interaction Instrumental Interaction that extends and generalizes the techniques, requires both new interaction models and principles of direct manipulation. It covers existing corresponding tools to facilitate development. This paper interaction styles, including traditional WIMP interfaces, as focuses on the first issue by introducing a new interaction well as new interaction styles such as two-handed input and model, called Instrumental Interaction, that extends and augmented reality. It defines a design space for new generalizes the principles of direct manipulation to also interaction techniques and a set of properties for comparing encompass a wide range of graphical interaction techniques. them. Instrumental Interaction describes graphical user The Instrumental Interaction model has the following goals: interfaces in terms of domain objects and interaction cover the state-of-the-art in graphical interaction instruments. Interaction between users and domain objects techniques; is mediated by interaction instruments, similar to the tools and instruments we use in the real world to interact with provide qualitative and quantitative ways to compare physical objects. The article presents the model, applies it interaction techniques, to give designers the basis for to describe and compare a number of interaction techniques, an informed choice when selecting a given technique and shows how it was used to create a new interface for to address a particular interface problem; searching and replacing text. define a design space in which unexplored areas can be identified and lead to new interaction techniques; and Keywords Interaction model, WIMP interfaces, direct manipulation, open the way to a new generation of user interface post-WIMP interfaces, instrumental interaction development tools that make it easy to integrate the latest interaction techniques in interactive applications. INTRODUCTION After a review of related work, this paper analyzes the In the early eighties, the Xerox Star user interface [27] and limits of current WIMP interfaces. The Instrumental the principles of direct manipulation [26] led to a powerful Interaction model is introduced and applied to several graphical user interface model, referred to as WIMP existing interaction techniques as well as to the design of a (Windows, Icons, Menus and Pointing). WIMP interfaces new interface for searching and replacing text. Finally the revolutionized computing, making computers accessible to paper concludes with suggestions for future work. a broad audience for a variety of applications. In the last decade, HCI researchers have introduced RELATED WORK numerous new interaction techniques, such as toolglasses In this paper, an interaction model is defined as follows: [5] and zoomable user interfaces [3]. Although some have An interaction model is a set of principles, rules and been shown to be more efficient than traditional techniques, properties that guide the design of an interface. It e.g., marking menus [19], few have been incorporated into describes how to combine interaction techniques in a commercial systems. A likely reason is that integrating meaningful and consistent way and defines the "look new interaction techniques into an interface is challenging and feel" of the interaction from the user's for both designers and developers. Designers find it faster perspective. Properties of the interaction model can and easier to stick with a small set of well-understood be used to evaluate specific interaction designs. techniques. Similarly, developers find it more efficient to Direct Manipulation [26] is a generic interaction model, take advantage of the extensive support for WIMP while style guides, e.g., Apples guidelines [2], describe interaction provided by current development tools. more precise and specific models. Took introduced a model called Surface Interaction [29] and Holland & Oppenheim a model called Direct Combination [15]. in Proc. ACM Human Factors in Computing Systems An interaction model differs from the architectural model of (CHI 2000), The Hague (Netherlands), April 2000 an interface, which describes the functional elements in the ACM Press, to appear implementation of the interface and their relationships (see

1 [12] for a review). User interface development environments Microsoft Word, graphical layers in Adobe Photoshop or have generated a variety of implementation models for Deneba Canvas, or paint brushes in MetaCreations Painter. developing interfaces (see [24] for a review), e.g. the widget Once the user is familiar with the application, these objects model of the X/Motif toolkit [14] or Garnet's Interactors become part of his or her mental model and may acquire the model [23]. MVC [18] is a well-known model that was status of object of interest. Unfortunately, these are rarely created to support the Xerox Star user interface and has implemented as first-class objects. Thus, for example, influenced many other architectural and implementation Word's styles are editable only via transient dialog boxes models. Whereas architectural models are aimed at interface that must be closed before returning to the text editing task. development, an interaction model is aimed at interface 2. Physical actions on objects vs. complex syntax design. Most computers have only a mouse and keyboard as input The model-based approach and its associated tools [28] help devices limiting the set of user actions to: typing text or bridge the gap between interaction and architectural models "special" keys (e.g. function keys, keyboard shortcuts and by offering a higher-level approach to the design of modifiers), pointing, clicking, and dragging. Given the interactive systems. mismatch between this small vocabulary of actions and the Device-level models such as logical input devices [11] or large vocabulary of commands, WIMP interfaces must rely Card et al.'s taxonomy [7] operate at a lower level of on additional interface elements, usually menus and dialog abstraction than interaction models. Understanding the role boxes, to specify commands. The typical sequence of of the physical devices in interaction tasks is a critical actions to carry out a command is: component of the definition of the Instrumental Interaction select the objet of interest by clicking it; model. select a command from a menu or keyboard shortcut; fill in the fields of a dialog box; and At the theoretical level, Activity Theory [6] provides a click the OK button to see the result. relevant framework for analyzing interaction as a mediation process between users and objects of interest. This is conceptually no different from typing a command in a command-line interface: The user must type a command Finally, Instrumental Interaction is grounded in the large name, file name (the object of interest), arguments (the (and growing) number of graphical interaction techniques fields in the dialog box) and return key (the OK button). In that have been developed in recent years, some of which are both cases the syntax is complex and cannot be considered referenced in the rest of this article. direct manipulation of the objects of interest. In fact, FROM WIMP TO POST-WIMP INTERFACES WIMP interfaces directly violate principle 2 and often use The WIMP interaction model can be outlined as follows: indirect manipulation of the objects of interest, through application objects are displayed in document windows; (direct) manipulation of interface elements such as menus and dialog boxes. objects can be selected and sometimes dragged and dropped between different windows; and 3. Fast, incremental and reversible operations with an immediately-apparent effect on the objects of interest commands are invoked through menus or toolbars, The heavy graphical syntax imposed on the user results in often bringing up a dialog box that must be filled in commands that are neither fast nor incremental. Specifying before the command's effect on the object is visible. a command is not fast because of the amount of time used This section uses Shneiderman's principles of direct for non-semantic actions such as displacing windows and manipulation [26] to analyze WIMP interfaces: flipping through tabs in a tabbed dialog. Inputting 1. Continuous Representation of objects of interest parameter values for a command is often inefficient because Objects of interest are central to direct manipulation. They of the small set of interactors, such as when numeric values are the objects that the user is interested in to achieve a are entered as text. Finally, the specification is not given task, such as the text and drawings of a document or incremental: users must explicitly commit to a command the formulae and values in a spreadsheet. Principle 1 asserts that uses a dialog box before seeing the result. If the result that objects of interest should be present at all times. Since does not match the user's expectations, the whole cycle of objects of interest are often larger than the screen or window command activation must be started over again. This is in which they are displayed, WIMP interfaces makes them especially cumbersome when trial-and-error is an integral accessible at all times through scrolling, panning or part of the task, as when a graphics designer selects a font zooming. This accessibility is hindered by the growing size: specifying the point size numerically is annoying number of interface objects that are not objects of interest when the goal is to see the visual result on the page. such as toolbars, floating palettes and menu bars. These use 4. Layered or spiral approach to learning increasing amounts of screen real-estate, forcing the user to The small number of interaction techniques used by WIMP shrink the windows displaying objects of interest. Dialog interfaces makes it easy to learn the basics of any new boxes also often occlude significant parts of the screen, application. However, interaction shortcuts, such as making the rest of the interface inaccessible to the user. combining keyboard modifiers with mouse buttons to Finally, there are more objects of interest than meet the activate the frequent commands, are concealed and eye: in many applications users must manipulate secondary inconsistent across applications and make the transition objects to achieve their tasks, such as style sheets in from novice to power user more difficult.

2 Towards a new interaction model Domain objects form the set of potential objects of interest Some commercial applications, especially those dedicated to for the user of a given application. Domain objects have creative tasks such as painting, graphic design or music, attributes that describe their characteristics. Attributes can extend the basic WIMP model to address some of the be simple values or more complex objects. For example, in shortcomings identified above. For example, some painting a 3D modeller, the position and size of a sphere are simple programs make brushes first class objects that can be edited values (integer or real numbers), while the material of the and saved into files. Some text editors have inspector sphere is a complex entity (color, texture, transparency, windows that display the state of the current selection and etc.). The user may shift the object of interest, update it when the user enters relevant values. Techniques concentrating on the material as the focus of the interaction. such as the HotBox [21] were designed to access larger Similarly, text styles that describe the formatting attributes numbers of commands. of text also may also obtain the status of objects of These new interaction techniques illustrate the transition interest. Materials and styles are therefore also domain from WIMP to post-WIMP interaction: Windows are not objects in their respective interfaces. used in zoomable interfaces, icons and text are replaced by In summary, domain objects form the basis of the richer representations in interactive visualization, menus are interaction as well as its purpose: Users operate on domain complemented by more powerful interaction widgets such objects by editing their attributes. They also manipulate as toolglasses, pointing and dragging are superseded by them as a whole, e.g. to create, move and delete them. bimanual input and gesture input. Designing Post-WIMP Interaction instruments interfaces that are more faithful to the principles of direct An interaction instrument is a mediator or two-way manipulation and that take advantage of novel interaction transducer between the user and domain objects. The user techniques requires new interaction models. To guide acts on the instrument, which transforms the user's actions interface designers, these models should be: into commands affecting relevant target domain objects. descriptive, incorporating both existing and new Instruments have reactions enabling users to control their applications; actions on the instrument, and provide feedback as the comparative, providing metrics for comparing command is carried out on target objects (Figure 1). alternative designs (as opposed to prescriptive, A scrollbar is a good example of an interaction instrument. deciding a priori what is good and what is bad); and It operates on a whole document by changing the part that generative, facilitating creation of new interaction is currently visible. When the user clicks on one of the techniques. arrows of the scrollbar, the scrollbar sends the document a scrolling command. Note that the transduction here consists INSTRUMENTAL INTERACTION of sending scrolling commands as long as the user presses As shown in the previous analysis, WIMP interfaces do not the arrow. The reaction of the scrollbar consists of follow the principles of direct manipulation. Instead, they highlighting the arrow being pressed. The feedback consists introduce interface elements such as menus, dialog boxes of updating the thumb to reflect the new position of the and scrollbars that act as mediators between users and the document. In addition, the object also responds to the objects of interest. Users have a (limited) sense of instrument by updating its view in the window. engagement, as advocated by direct manipulation, because they manipulate these intermediate objects directly. This Another example is an instrument that creates rectangles in matches our experiences in the physical world: We rarely a drawing editor. As the user clicks and drags the mouse, fingerpaint, but often use pens and pencils to write. We the instrument provides a reaction in the form of a rubber- cook with pots and pans, hang pictures with hammers and band rectangle. When the user releases the button, the power drills, open doors with handles and turn off lights creation operation is actually carried out and a new domain with switches. Our interaction with the physical world is object is created. The feedback of this operation consists in governed by our use of tools. Direct manipulation of displaying the new object. physical objects of interest occurs when we bring them into An instrument decomposes interaction into two layers: the our current context of operation, before we manipulate them interaction between the user and the instrument, defined as with the appropriate tools, usually with two hands [13]. the physical action of the user on the instrument and the The Instrumental Interaction model is based on how we reaction of the instrument and the interaction between the naturally use tools (or instruments) to manipulate objects instrument and the domain object, defined as the command of interest in the physical world. Objects of interest are sent to the object and the response of the object, which the called domain objects, and are manipulated with computer instrument may transform into feedback to the user. The artifacts called interaction instruments. instrument is composed of a physical part, the input device, Domain objects and a logical part, the representation of the instrument in In computer systems, applications operate on data that software and on the screen. represent phenomena or objects. For computer users, this Activating instruments data is the primary focus of their actions. For example, At any one time, an interface provides a potentially large when creating a text document, the focus of the user is on number of instruments. However the user can manipulate the text of the document. Everything else on the screen is only a few of them at the same time, usually only one, there to support the user's task of editing the text document. because of the limited number of input devices. In the most

3 action command materials that complement the "primary" domain objects of the application domain. Instrumental Interaction introduces a second type of reaction response reification: an interaction instrument is the reification of one or more commands. For example, a scrollbar is the feedback reification of the command that scrolls a document. This link between the traditional notion of command and the user instrument domain object notion of instrument makes it easy to analyze existing Figure 1: Interaction instrument mediating the interfaces with the Instrumental Interaction model. It is also interaction between a user and a domain object a useful guideline to identify instruments when designing a common case of a keyboard and mouse, a single input new interface. In the last part of this paper, this rule is used device (the mouse) must be multiplexed between a to reify the traditional search-and-replace command of a text potentially large number of instruments, i.e. a single editor into a search instrument. physical part may be associated with different logical parts. The result of this reification rule is that instruments are An instrument is said to be activated when it is under the themselves potential objects of interest. This is indeed the user's control, i.e. when the physical part has been case in real life, when the focus of attention shifts from the associated with the logical part. In the case of the scrollbar, object being manipulated to the tool used to manipulate it. the user activates the instrument by pointing at it and it For example a pencil is a writing instrument and the remains active as long as the pointer is within the scrollbar. domain object is the text being written. When the lead When creating a rectangle, the user activates the instrument breaks, the focus shifts to a new instrument, a pencil by clicking a button in a tool palette and it remains active sharpener, which operates on the shifted domain object, the until another instrument is activated. pencil lead. The focus may even shift to the pen sharpener, if we need a screwdriver to fix it. Such "meta-instruments" These two types of activation are quite different. The (instruments that operate on instruments) are not only activation of the scrollbar is spatial because it is caused by useful for "fixing" instruments, but can also be used to moving the mouse (and cursor) inside the area of the organize instruments in the workspace, e.g. a toolbox, or to scrollbar. The activation of the rectangle creation tailor instruments to particular tasks, e.g. turning a power- instrument is temporal because it is caused by a former drill into a power-saw. In graphical user interfaces, common action and remains in effect until the activation of another meta-instruments include menus and tool palettes used to instrument. (This is traditionally called a mode). Each type select commands and tools, i.e. to activate instruments. of activation has an associated cost: Spatial activation requires the instrument to be visible on the screen, taking Properties of Instruments up screen real-estate and requiring the user to point at it and An important role of an interaction model is to provide potentially dividing the user's attention. Temporal properties to evaluate and compare alternative designs. This activation requires an explicit action to trigger the can help interface designers who face difficult choices when activation, making it slower and less direct. selecting the interaction techniques for a particular application. The goal of defining properties of instruments Interface designers often face a design trade-off between is not to decide which instruments are good and which are temporal and spatial multiplexing of instruments because bad, but to evaluate them so that designers can make an the activation costs become significant when the user must informed choice and so that researchers can identify and frequently change instruments. Using extra input devices explore areas of the design space that are not mapped by can reduce these costs. For example, the thumbwheel on existing instruments. Microsoft's Intellimouse is a scrolling instrument that is always active. An extreme example is an audio mixing The literature on user interface evaluation techniques is console, which may contain several hundred potentiometers considerable. Here, we use a particular type of evaluation and switches, each corresponding to a single function. This based on properties. This is a common approach in software permits very fast access to all functions, which is crucial engineering and has also proved valid and useful for for sound engineers working in real-time and cannot afford evaluating interactive systems [12]. The rest of this section the cost of activating each function indirectly. A large introduces three properties of interaction instruments. design space lies between a single mouse and hundreds of Degree of indirection potentiometers, posing design challenges to maximally The degree of indirection is a 2D measure of the spatial and exploit physical devices and reduce activation costs. temporal offsets generated by an instrument. The spatial Reification and Meta-instruments offset is the distance on the screen between the logical part Reification is a process for turning concepts into objects. In of the instrument and the object it operates on. Some user interfaces, the resulting objects can be represented instruments, such as the selection handles used in graphical explicitly on the screen and operated upon. For example, a editors, have a very small spatial offset since they are next style in a text editor is the reification of a collection of text to or on top of the object they control. Other instruments, attributes; the notion of material in a 3D modeller is the such as dialog boxes, can be arbitrarily far away from the reification of a set of rendering properties. This type of object they operate on and therefore have a large spatial reification generates new domain objects such as styles and offset. A large spatial offset is not necessarily undesirable.

4 temporal Indirection Integration Compatibility offset dialog Menus -- +/- -- Toolbars boxes - ++ drag and Dialog boxes -- +/- - drop Property boxes + +/- - Handles ++ scrollbars ++ ++ property Window titles ++ + + Scrollbars + - - handles boxes spatial Keyboard shortcuts + + - Drag & drop offset ++ ++ ++ Figure 2: Degree of indirection Table 1: Comparing WIMP interaction techniques For example, placing a light switch far from the light bulb movements of the mouse. Scrolling with a scrollbar has a it controls makes it easier to turn on the light. Similar low degree of compatibility because moving the thumb examples can be found in user interfaces. downwards moves the document upwards. Using text input fields to specify numerical values in a dialog box, e.g. the The temporal offset is the time difference between the point size of a font, has a very low degree of compatibility physical action on the instrument and the response of the because the input data type is different from the output data object. In some cases, the object responds to the user's type. Similarly, specifying the margin in a text document action in real-time. For example, clicking an arrow in a by entering a number in a text field has a lower degree of scrollbar scrolls the document while the mouse button is compatibility than dragging a tab in a ruler. depressed. In other cases, the object responds to the user's action only when the action reaches closure. For example, APPLYING THE MODEL the arguments specified in a dialog box are taken into This section uses the Instrumental Interaction model to account only when the OK or Apply button is activated. In analyze existing interaction techniques, both from WIMP general, short temporal offsets are desirable because they interfaces and from more recent research. It demonstrates the exploit the human perception-action loop and give a sense descriptive power of the model. The generative power of the of causality [22]. model is then illustrated by the design of a new instrument for searching and replacing text. Figure 2 shows the degree of indirection of various WIMP instruments on a 2D chart. Scrollbars occupy a range in the Analyzing WIMP Interfaces diagram. For example, some scrollbars provide immediate The primary components of WIMP interfaces can be easily response when the thumb is moved while others only scroll mapped to instruments and compared (Table 1): the document when the mouse button is released. The figure Menus and toolbars are meta-instruments used to select the shows that the degree of indirection describes a continuum command or tool to activate. This use of meta-instruments between direct manipulation (lower-left corner) and indirect slows down interaction and generates shifts of attention manipulation (upper-right corner). between the object of interest, the meta-instrument and the Degree of integration instrument. Contextual menus have a small spatial offset The degree of integration measures the ratio between the and are therefore more efficient than toolbars and menu bars. number of degrees of freedom (DOF) provided by the logical Toolbars, which can be moved next to their context of use, part of the instrument and the number of DOFs captured by have a better spatial offset than menu bars. the input device. This term comes from the notion of Dialog boxes are used for complex commands. They have a integral tasks [17]: some tasks are performed more high degree of spatial and temporal indirection. They often efficiently when the various DOFs are controlled use a small set of standard interactors such as text fields for simultaneously with a single device. A scrollbar is a 1D numeric values, resulting in a low degree of compatibility. instrument controlled by a 2D mouse, therefore its degree of integration is 1/2. The degree of integration can be larger Inspectors and property boxes are an alternative to dialog than 1: controlling 3 rotation angles with a 2D mouse [16] boxes that have a lower degree of temporal indirection. has a degree of integration of 3/2. This property can be used Since they can stay open, they can be activated with to compare instruments that perform similar operations. pointing (positional activation) rather than selection in a For example, panning over a document can be achieved menu (temporal activation). with two scrollbars or a 2D panner. The latter has a degree Handles are used for graphical editing and provide a very of integration of 1 and is therefore more efficient than two direct interaction: low degree of indirection, high degree of scrollbars, which have a degree of integration of 1/2 and compatibility and good degree of integration. incur additional activation costs. Window titles and borders are instruments activated Degree of compatibility positionally to manipulate the window (move, resize, The degree of compatibility measures the similarity iconify, zoom, close). Scrollbars control the content of the between the physical actions of the users on the instrument window. Because of their low degree of integration, they are and the response of the object. Dragging an object has a not optimal, especially for panning documents in 2D. Also, high degree of compatibility since the object follows the their spatial offset generates a division of attention,

5 especially since they are activated positionally: the user Indirection Integration Compatibility must be sure to point at the right part of the scrollbar. Dynamic Queries + -- - Keyboard shortcuts and accelerator keys are meta- Pad++ navigation ++ ++ instruments, used to quickly switch between instruments Droppable tools + ++ + and save the activation costs of menus and toolbars. Some Toolglasses ++ ++ ++ accelerator keys affect the way the current instrument Graspable interfaces ++ ++ ++ works. For example, on the Macintosh, the Shift key Table 2: Comparing post-WIMP interaction techniques constrains the move tool to horizontal and vertical moves and the resize tool to maintain the current aspect ratio. and provides access to navigation anywhere on the surface, Drag and drop is a generic instrument for transferring or unlike a scrollbar which requires positional activation. It copying information. Compared to traditional cut/copy/ also has high degrees of compatibility and integration. paste commands that use a hidden clipboard, it has a smaller Editing the objects on the surface has led to Dropable Tools degree of indirection. There is no spatial offset because the [4]: tools can be dropped anywhere on the surface and objects are manipulated directly and the temporal offset is grabbed later. Activating these instruments is more direct low because there is feedback about potential drop-zones as than with a traditional toolbar because it does not involve a the user drags the object. meta-instrument and the associated switch of attention. Over the past few years, interaction techniques such as A number of recent interaction techniques rely on new or inspectors, property boxes, drag and drop and contextual additional input devices. This reduces activation costs by menus have become more common in commercial allowing several instruments to be active simultaneously. applications. The above analysis explains why these For example, the thumb wheel of the Intellimouse is techniques are more efficient than their WIMP counterparts, always attached to a scrolling instrument. ToolGlasses [5] demonstrating a useful contribution of the Instrumental are semi-transparent palettes operated with a track-ball in Interaction model. the left (or non-dominant) hand. The right hand is used to click through the palette onto a domain object, therefore Analyzing Post-WIMP Interaction Techniques specifying both the action to perform and the object to Table 2 summarizes the comparison of several post-WIMP operate on. Here the toolglass is a meta-instrument under interaction techniques. Interactive visualization helps users the control of the left hand, while the instruments it explore large quantities of visual data and make sense of it contains are activated by the right hand. In the TTT through filtering and displaying it to exhibit patterns [8]. prototype [20], a combination of three instruments can be These systems use two categories of instruments: active simultaneously: the toolglass itself, an instrument in navigation instruments specify which part of the data to the toolglass and a navigation instrument to pan and zoom visualize and how; and the drawing surface. This makes it possible, for example, to filtering instruments specify queries and display results. pan and zoom while creating an object. The design exploits A key aspect of these systems is a strong coupling between the trackball and mouse input devices to minimize user actions and system response. In other words, these activation costs, to reduce the degree of indirection and to instruments must have a small temporal offset. For increase the degree of integration. example, in the Information Visualizer [9], the instruments Graspable interfaces [10] use physical objects as input used to control Cone Trees and Perspective Walls provide devices to manipulate virtual objects. In effect, they transfer immediate responses and use smooth animations to display most of the characteristics usually found in the logical part changes in visualization parameters. In Dynamic Queries of the instrument into the physical part. This approach was [1], double sliders are used to specify the range of query pioneered by Augmented Reality [30], which explores ways parameters; any change in a slider updates the display of to reconcile the physical and computer world by embedding filtered data. computational facilities into physical objects. Here, the Both navigation and filtering are usually multi-dimensional domain objects, in addition to the instruments, have a tasks: the user wants to control several dimensions strong physical component. This increases the degrees of simultaneously to navigate along arbitrary trajectories. This compatibility and integration since interaction occurs in the calls for the ability to manipulate several instruments real world. simultaneously (which requires additional input devices) Designing a Text Search Instrument and/or for instruments with a high degree of integration. In most text editors, searching and replacing text uses a Current systems do not address this well. For example, dialog box where the user specifies the string to be searched Dynamic Queries permit only one side of a slider to be and the string to replace it with. The operation is controlled manipulated at a time, forcing the user to navigate along with buttons to find the next or previous occurrence and rectangular trajectories in the parameter space. replace it or not. Undoing the command usually means Zoomable user interfaces such as Pad++ [3] are based on the restoring the search string everywhere it has been replaced. display of an infinite flat surface that can be viewed at any This results in a sequential form of interaction where the resolution. Exploring this surface requires navigation system prompts the user and forces him or her to decide instruments to pan and zoom until the desired objects are in what to do with each occurrence, generating a very large sight. Pad++ navigation instruments are activated by mouse temporal offset. buttons or modifier keys. This temporal activation is fast

6 The design of the scrollbar improves the degree of integration since it uses the vertical position of the mouse to control the speed and direction of scrolling (1 output DOF / 2 input DOFs) while traditional scrollbars only use the fact that the user clicked in the arrow (0 output DOF / 2 input DOFs). This design also improves the degree of compatibility since the mouse now works as a joystick. In effect, the scrollbar thumb and arrows are functionally equivalent: the thumb provides positional control while the arrows provide rate control of the visible part of the document. Two variants of the search instrument were also developed. The first variant allows several search instruments to be active simultaneously, each independent of the others and using a different pair of colors to highlight occurrences. In the second variant, multiple strings can be specified in the search string and a regular expression can be specified in the Figure 3: Search and replace instrument replace string. The instrument highlights all the occurrences of all search strings at once. This variant was used to build An instrumental approach to search and replace has led to the index of a book: a list of words to index was entered as the following design, implemented in a Tcl/Tk prototype a set of search strings. The replace string added the proper (Figure 3). The top part of the window is the logical part of markup to include the occurrence in the index. Indexing the the instrument, used to specify the search and replace book became simply a matter of picking which occurrences strings. No buttons are necessary: as the user types a search to include in the index, taking advantage of the display of string, the occurrences highlight in yellow both in the text all occurrences to avoid putting in the index occurrences of window and in the scrollbar. To replace an occurrence, the the same word that are close together in the text. At any user simply clicks on it, immediately replacing it with the time it was possible to change the list of words in the replace string (if any) highlighted in red. Editing the replace index, the content of the text and the occurrences to index. string changes all the replaced occurrences. A replaced occurrence can be undone by clicking on it in the same CONCLUSION AND FUTURE WORK way: the search string is substituted and highlighted in This article has introduced the Instrumental Interaction yellow. Typing text in the document does not cancel the model, which generalizes and operationalizes Direct operation of the search instrument: each time the text is Manipulation. The model has been used to analyze WIMP changed, a new search is performed. interfaces as well as more recent interaction techniques and to design a new interface for searching and replacing text. The scrollbar was modified to facilitate browsing. First, it This demonstrates the descriptive, comparative and includes tick marks representing the occurrences and giving generative power of the Instrumental Interaction model. an overview of the search. Second the arrow buttons were changed so that clicking on an arrow scrolls the document We are currently testing the model as we design a new at a variable speed according to the distance between the graphical editor for Colored Petri Nets. The model is used cursor and the arrow: clicking on the up arrow scrolls the both as a design guide and an evaluation tool to integrate document slowly downwards (as in traditional scrollbars); existing interaction techniques and create new ones. moving the cursor up speeds up scrolling; moving the However further work is needed to develop the model in cursor down slows it down; moving the cursor further more detail and assess its limits. This requires a more down, scrolling stops, then reverts. This reduces the thorough analysis of graphical interfaces and interaction division of attention that occurs when operating the various techniques, the definition and evaluation of new properties, parts of a scrollbar while focusing on the document. It is a taxonomy of interaction instruments, and an exploration similar in effect to the thumbwheel described earlier but of the design space defined by the model. Design principles does not require a separate input device. are also needed to help designers create instruments and The instrument provides feedback about the current state of integrate them effectively into a user interface. the search/replace operation by highlighting all the The other important area for future work is to make occurrences in the text, as in the Document Lens [25], and Instrumental Interaction useful not only to user interface in the scrollbar. This design results in a low degree of designers but also to user interface developers by developing indirection: The spatial offset is reduced by getting rid of a user interface toolkit based on the model. This would the traditional Next, Previous and Replace buttons; The support the adoption of novel interaction techniques in a temporal offset is reduced by displaying the occurrences as wide range of applications and allow a shift from WIMP to the user types the search string, by showing the effect of post-WIMP interfaces. replacing an occurrence immediately and by allowing the user to undo any change, not necessarily in the order they ACKNOWLEDGMENTS were made. Many thanks to Wendy Mackay for numerous discussions about the model and its implications and for her help with

7 the English. Thanks to Peter Bøgh Andersen, Susanne 16. Jacob, I. & Oliver, J. Evaluation of Techniques for Bødker, Yves Guiard and Austin Henderson for discussions Specifying 3D Rotations with a 2D Input Device. and feedback on earlier versions of this paper. Proc. HCI'95 Conference, People and Computers X, p.63-76, 1995. REFERENCES 1. Ahlberg, C., Williamson, C., Shneiderman, B. 17. Jacob, R., Sibert, L., McFarlane, D., Preston Mullen, Dynamic Queries for Information Exploration: An M. Integrability and Separability of Input Devices. Implementation and Evaluation. In Proc. ACM ACM Trans. Human Computer Interaction, 1(1), p.3- Human Factors in Computing Systems, CHI'92, ACM 26, 1994. Press, p.619-626, 1992. 18. Krasner, G.E. & Pope, S.T. A Description of the 2. Apple Computer. Macintosh Human Interface Model-View-Controller User Interface Paradigm in the Guidelines, Addison-Wesley, 1992. Smalltalk80 System. J. Object Oriented Programming 3. Bederson, B. and Hollan, J. Pad++: A Zooming 1(3):26-49, 1988. Graphical Interface for Exploring Alternate Interface 19. Kurtenbach, G. & Buxton, W. User Learning and Physics. In Proc. ACM Symposium on User Interface Performance with Marking Menus. Proc. ACM Human Software and Technology, UIST94, ACM Press, p.17- Factors in Computing Systems, CHI'94, ACM Press, 26, 1994 p.258-264, 1994. 4. Bederson, B., Hollan, J., Druin, A., Stewart, J., 20. Kurtenbach, G., Fitzmaurice, G., Baudel, T., Buxton. Rogers, D., Proft, D. Local Tools : an Alternative to W. The Design of a GUI Paradigm based on Tablets, Tool Palettes. In Proc. ACM Symposium on User Two-hands, and Transparency. In Proc. ACM Human Interface Software and Technology, UIST94, ACM Factors in Computing Systems, CHI'97, ACM Press, Press, p.169-170, 1994. p.35-42, 1997. 5. Bier, E., Stone, M., Pier, K., Buxton, W., De Rose, 21. Kurtenbach, G., Fitzmaurice, G.W., Owen, R.N., T. Toolglass and Magic Lenses : the See-Through Baudel, T. The Hotbox: efficient access to a large Interface. In Proc. ACM SIGGRAPH, p.73-80, 1993. number of menu-items. Proc. ACM Human Factors in 6. Bødker, S. Through the Interface. A Human Activity Computing Systems, CHI'99, ACM Press, p.231-237, Approach to User Interface Design. Lawrence Erlbaum 1999. Associates, 1991. 22. Michotte, A. La perception de la causalité. 7. Card, S.K., Mackinlay, J.D., Robertson, G.G. A Publications Universitaires de Louvain, 1946. Morphological Analysis of the Design Space of Input 23. Myers, B.A. A New Model for Handling Input. ACM Devices. ACM Trans. Information Systems, 9(2):99- Trans. Information Systems, 8(3):289-320, 1990. 122, 1991. 24. Myers, B.A. User Interface Software Tools. ACM 8. Card, S., Mackinlay, J., Shneiderman, B. Readings in Trans. Computer-Human Interaction, 2(1):64-103, Information Visualization: Using Vision to Think. 1995. Morgan Kaufmann Publishers, 1998. 25. Robertson, G.G. & Mackinlay, J.D. The Document 9. Card, S., Robertson, G., Mackinlay, J. The Lens Visualizing Information. Proc. ACM Information Visualizer, an Information Workspace. In Symposium on User Interface Software and Proc. ACM Human Factors in Computing Systems, Technology, p.101-108, 1993. CHI'91, ACM Press, p.181-187, 1991. 26. Shneiderman, B. Direct Manipulation : a Step Beyond 10. Fitzmaurice, G., Ishii, H., Buxton, W. Laying the Programming Languages. IEEE Computer, 16(8), pp Foundations for Graspable User Interfaces. In Proc. 57-69, 1983. ACM Human Factors in Computing Systems, CHI'95, 27. Smith, D., Irby, C., Kimball, R., Verplank, B., ACM Press, p.442-449, 1995. Harslem E. Designing the Star User Interface. Byte, 11. Foley, J., Wallace, V., Chan, P. The Human Factors 7(4), p.242:282, 1982. of Computer Graphics Interaction Techniques. 28. Szekely, P., Luo. P., Neches, R. Beyond Interface Computer Graphics and Applications, 4(11):13-48, Builders: Model-Based Interface Tools Model-Based. 1984. Proc. ACM Human Factors in Computing Systems, 12. Gram, C. & Cockton, G. Design Principles for INTERCHI'93, ACM Press, p.383-390, 1993. Interactive Software, Chapman & Hall, 1996. 29. Took, R. Surface Interaction: A Paradigm and Model 13. Guiard, Y. Asymmetric division of labor in human for Separating Application and Interface. Proc. ACM skilled bimanual action: The kinematic chain as a Human Factors in Computing Systems, CHI'90, ACM model. Journal of Motor Behavior, 19:486-517, 1987. Press, p.35-42, 1990. 14. Heller, D., Ferguson, P.M., Brennan, D. Motif 30. Wellner, P., Mackay, M., Gold R. Computer- Programming Manual, O'Reilly & Associates, 1994. Augmented Environments. Special Issue of 15. Holland, S. & Oppenheim, D. Direct Combination. Communications of the ACM, 23(7). Proc. ACM Human Factors in Computing Systems, CHI'99, ACM Press, p.262-269, 1999.