sign in sign up
<<
Home , Computer graphics

Compatability and Interaction Style In

George W. Fitzmaurice and Bill Buxton sented -- in fact, many prefer not to. However applications such as computer aided , AliaslWavefront Inc. providing access and acquiring such under- modeling, and . standing is often necessary for users to achieve Introduction their goals. Thus, we must find ways of Example I:APL and Teletype Recent trends in human computer interaction exposing the deep structure to the user in I/O Compatibility -- High have focused on representations based on ways that are compatible, intuitive and efficient, UI to Manipulate Deep Structure -- High physical reality [4, 5, 6, 8].The idea is to provide and which enable the user to work at this level UI to Manipulate Structure -- Low richer, more intuitive handles for control and when appropriate. manipulation compared to traditional graphical To explore this issue, we examine three To begin, consider the use of the programming user interfaces (GUIs) using a mouse. This styles of interaction in terms of their ability to languageAPL in computer graphics in the trend underscores the need to examine the support manipulation of both the deep and .The language was cryptic and terse, but concept of manipulation and to further under- surface levels of the graphics. We do so by the handling was wonderful."The stand what we want to manipulate versus what considering the issue of compatibility between power ofAP/comes from its direct manipula- we can easily manipulate. Implicit in this is the input and output devices in addition to the tion of n-dimensional arrays of .The APL notion that the bias of the UI is often incom- ability of the user to manipulate internal repre- primitives express broad ideas of data manipu- patible with user needs. sentations ("deep structure") as well as lation.These rich and powerful primitives can The main goal of UI design is to reduce external representations ("surface structure") be strung together to perform in one complexity while augmenting the ability of of the application data. This is somewhat akin what would require pages in other program- users to get their work done. A fundamental to the model-view-control organizing struc- ming languages"[I] Interaction was via an IBM belief underlying our research is that ture of SmallTalk [7]. Let us look at each one Selectric typewriter-like terminal, and complexity lies not only in what is purchased of these compatibilities. compared to the card-punch readers typical of from the and hardware manufac- • Input with output devices: Does the input the era, it was very interactive. turers, but also in what the user creates with device match the capabilities of the it. It is not just a question of making buttons output display? I/O Compatibility -- High and menus easier to learn and more efficient • with the ability to manipulate The input devices (characters on the to use. It is also a question of"Given that I've internal representation (access to "deep keyboard) and the output (printed characters) created this surface in this way, how can it now structure"): Can the user interface effec- had a strong compatibility. IBM Selectric type- be modified to achieve my current design tively manipulate the internal representa- writers were very popular and familiar to objective?" (The observation is that how the tion of the application data stream? users. Moreover, the symbolic nature of the user created the surface in the first place will • User interface with the ability to manipulate language lent itself to typing. affect the answer to the question.) Our thesis external representation (access to "surface is that appropriate design of the system can structure"): Can the user interface effec- UI to Manipulate Deep Structure -- High minimize both kinds of complexity: that tively manipulate the artifacts generated There was a high compatibility in the manipula- inherent in accessing the functionality provided by the internal representations? tion of the deep structure of the graphics.This by the vendor, and that created by the user. For example, let us consider a form-based was by virtue of the language's facility in manip- The literature focuses on the former. In what UI on a of records that use an ulating n-dimensional arrays, performing matrix follows, we investigate some of the issues in alphanumeric keypad . From the multiplication operations and programming achieving the latter. In so doing~we structure our first , this has good compatibility new functionality. discussion around questions of compatibility. since the primary type of data being input and displayed is alpha-numeric.The deep structure Three Perspectives on might be considered the layout of the forms, UI to Manipulate Surface Structure -- Low and what attributes of the underlying database However, there was a strong incompatibility in Compatibility the manipulation of the surface structure of When we consider manipulation in the are exposed and in what relationship. Since the graphics.APL processed numeric arrays of context of 2D and 3D graphics applications, layout, at least, is a spatial thing, the keyboard data, not graphics per se. While the we often consider properties of the input would likely have low compatibility, compared numeric arrays may represent a graphical device such as degrees of freedom, and how to a mouse, in terms of interacting at this level. , such as a set of curves, a user could not well the device is capable of moving points or Finally, in interacting with the fields exposed, directly adjust the contour of a curve within the shapes in this space.That is, how well can the the surface structure, there is medium to good graphical domain. user move and adjust interactive widgets as compatibility. In entering the alphanumeric data Consequently, the user was forced to func- well as directly transform (e.g., the compatibility is high. However, tabbing from field to field using the keyboard may tion exclusively at the abstraction level of the adjust points, curves and ). What is graphics, manipulating the internal representa- often not considered, however, is the ease of often be less compatible than doing so by tions, or deep structure. The user interface manipulating the underlying structure of the selection with a mouse. was designed and optimized to facilitate the graphics (e.g., the deep structure of the geom- In this paper we explore the progression of manipulation of the graphics as represented by etry, as represented by the scene graph, etc.), manipulation as it relates to input devices, the APL notation, not by way of the pictures even when these are data that were directly or deep structure and surface structure. We do themselves. Nevertheless, APL was used in the indirectly created by that same user. this by giving two historical examples and then discuss a new trend which we call "interactive 1970s to produce a number of innovative Many users, especially artists, do not under- such as by Judson Rosebush's stand the deep structure, or how it is repre- assemblages." We will frame our discussion within the context of sophisticated graphics company, Digital Effects Inc.

64 November1998 Computer Graphics Example 2: GUI Direct Manipulation my goals in a macro sense. Consider or abstraction for the task at hand.The bias is AliaslWavefront's Maya animation package [9], towards the surface structure. Consequently, I/O Compatibility- High as an example. In this application, 3D geometry as the complexity of the (user created) graph- UI to ManipulateDeep Seruccure-- Low is available to the user in a number of repre- ical scene, object or animation increases, the UI to ManipulateSurface Structure -- High sentations. One is a 3D perspective view. effectiveness of the UI breaks down.Without a Another is as a 2D dependency graph which grasp of the deep structure and an effective In this example we leap ahead approximately reflects the deep structure of the graphics. handle on it, users are mired in too much 20 years and consider the shift towards direct Users are able to manipulate the graphics detail, detail which becomes overwhelming man/pulat/0n through the development of the within this view. However, while this exposes regardless of how simple any single step of that (GUl).The GUI signifi- the internal representation to the user in a detail is. cantly improved our ability to manipulate the form that can be manipulated, actually doing so surface structure which was so lacking in the can still be quite cumbersome. Resolving the Dilemma previous example. This is sometimes due co a notational issue: Given these two examples, we are presented the graphical representation or interaction with a dilemma. How can we design interactive I/O Compatibility- High techniques available may not be appropriate to systems which maintain the balance of manipu- The high compatibility of the input and output the task at hand. Other times, however, it is lability and compatibility over deep and surface devices with GUIs is based on the use of due to the data itself being structured in a way structure? Already we have alluded to a hybrid graphical metaphors. Tools and other entities that is incompatible with the manipulation that approach (Houdini and Maya) which offers both are represented graphically, often as icons, and the user wants to perform (even though it was scripting and direct manipulation solutions. But one interacts with them using generic actions likely that same user who structured ir that from the perspective of artists, direct manipu- such as pointing and dragging. Interaction is way.) lation is generally preferred since it is closer to mediated through the use of a graphical input The former notational issue can often be how they work in the physical world. device such as a mouse, tablet, trackball or addressed by offering another form of repre- Consequently, intense scripting is typically touch screen, typically in conjunction with a sentation and manipulation.This is partially delegated to a programmer (technical addressable graphics display.VVhile graph- why packages such as Side Effects Houdini and director) who is fluent in the scripting ical representations and metaphors are used Maya still support procedural scripting in addi- language. to regulate input and output, there is still a tion to the GUI: this enables the user to What we would like to do in the remainder level of indirection employed which serves as manipulate the geometry and perform global, of this paper is discuss an alternative approach an abstraction above that of physical reality large scale manipulations at a variety of granu- to finding a balance between control over the (for example, with a GUI one doesn't grab a larities.This scripting ability is more akin to the deep and surface structure of the graphics.This document to move it up or down. One does previous APL example than direct manipula- approach has more to do with how the user this indirectly, using a widget such as a scroll tion. creates the graphics in the first place, rather bar or scroll arrow, which is itself a metaphoric The latter point, the compatibility of the than the mechanics of the UI per se. Here, one icon). underlying structure itself to the types of assembles the graphics using higher order Moreover, while graphically the GUI's manipulations that the user wants to perform primitives.The assumpl'_ion is that much of the output is explicidy representational, the input is addressed by, and motivates, Example 3, complexity in the deep structure in conven- is limited and generic. For example, while I may below. tional systems stems from the primitives used be presented with a graphical ruler, the typi- being at too low a level. If the granularity of cally one-handed, single device input mecha- UI to Manipulate Surface Structure -- High how one thinks about a model, for example, nisms available only permit me to perform a There is a high compatibility between the GUI matches that of the primitives used to small subset of the overall bimanual actions or and the ability to manipulate the surface struc- construct or modify it, then the assumption is gestures that I might employ with a real ruler. ture of the graphics.This is because it enables that the underlying complexity for the user Just compare how you perform actions like the artist to work directly on an image or 3D will be significantly reduced. Such primitives bending, flipping, squeezing, shaking, tilting or model. Note chat this is not always done. may be procedural or declarative. The key moving objects in the physical world with how Rather, almost more often than not, interme- point is that they be compatible with how the you do so using so-called "direct manipulation" diate dialog boxes are used to manipulate user thinks, and that they afford control at the with a GUI. parameters resulting in what might more prop- appropriate level of detail or granularity appro- In summary, while input and output are erly be described as indirect man/pulat/0n. priate for achieving the user's goals. We will compatible in manipulating the surface struc- Some work has been done to address this refer to this approach as interactive ture, we need to keep in mind that this holds issue, such as building 3D manipulators and assemblages. true only within the range of actions available. attaching them to 3D geometry [3]. Such Between how one interacts with the graphical manipulators provide efficient handles which Example 3: Interactive Assemblages objects in the GUI and their counterparts in afford control over key parameters of the the physical world, there is limited compati- underlying structure. They bring such control I/O Cornl~tibility -- High bility at best, and incompatibility at worst. closer to the geometry itself than is the case Ul to Manipulate Deep Structure -- with dialogue boxes, making control more Moderately High Ul to Manipulate Deep Structure -- Low direct. UI to ManipulateSurface Structure -- High The underlying structure of complex graphical In summary, GUI-based systems generally scenes and objects does not generally lend do a fairly good job of supporting the capability By interactive assemblages we mean the itself to effective graphical representation, that to work directly on the graphics. However, construction of graphical scenes, models and is, in terms of actions that you want to they also have affordances that bias and deter animations out of higher level components perform on itWhile I may "know how" to users from understanding the deep structure. than is traditionally the case. Such components manipulate things in the micro sense (point Thus, users often have trouble manipulating may be declarative (such as a canonical form and click) I still may have problems achieving the graphics at the appropriate level of detail or a generic skeleton with all of the IK handles and dynamics built in), procedural (such as a

Computer Graphics November199S 6S Figure 2: Example of a flock of birds [I 0]. See page 103 for color image.

high level, specifying parameters such as size, occupy. The flock module then populates the seams, fabric properties, texture and how area with a random placement of instances of tightly the material should fit or follow the the representative characters. High-level para- character.The cloth module simulator then can meters can then be defined which move the Figure I : Example of cloth on a computer calculate the behaviour of the cloth as the crowd in a particular direction, make them character [2]. See page 103 for color image. character moves -- factoring in the physical more or less active, and control the dynamics of the cloth material and real-world percentage of the crowd looking in a certain flocking module) or some hybrid of the two. properties such as gravity and object collisions. direction, etc. Again, it would be very tedious Each integrates sophisticated behaviours or With this approach, notice that the specifica- for the artist to have to manipulate each indi- data into a single component having a manage- tion of the character's movement is indepen- vidual character within the crowd consisting of able but rich set of appropriate operating para- dent of the specification of the behaviour of hundreds of characters. meters. These operating parameters, or the cape. Certainly the former affects the handles, may be exposed to the user via virtual behaviour of the other. But unlike traditional Car Design means (such as dialog boxes or 3D manipula- animation, if I change the character's move- The previous two examples were procedural. tors) or through physical means (such as ment, I don't have to reanimate the cape, and if Our third example involves a module that specialized input devices like physical sliders, I change the material of the cape, I don't have consists of declarative data. It is a car module dials or customized graspable objects). to reanimate the whole thing. The in which the designer is presented with a The deep structure now reveals the rela- addresses things at the appropriate level with canonical model of a car. This is in contrast tionship among these components, rather than the ensuing freedom to explore a far broader with current practice where the designer the low-level primitives traditionally used.Thus, range of possibilities with relatively little starts with a blank sheet and works from there is an increased likelihood that any repre- increase in cost. In terms of the UI, one can there. The model contains and encapsulates sentation of the deep structure will be both imagine defining virtual or physical sliders and the necessary engineering specifications and more comprehensible to users, and afford controllers to adjust the parameters of the manufacturing constraints, as well as the manipulation at a level appropriate for the task cloth. Perhaps a set of instrumented pieces of essence of the "style" of that particular at hand. fabric may facilitate specification of cloth product line (see Figure 3).The notion is that let us describe three example modules to behaviours.The idea is that the deep structure by modifying the base model, rather than illustrate this idea: modeling cloth, specifying of the cloth module is abstracted and users building each concept from scratch, one can flocking behaviour and car design. only need to set a few high level parameters to work faster, stay within the style, conform to get sophisticated behaviours. the engineering criteria and end up with a Modeling Cloth much better initial geometry. Imagine animating a super hero who is wearing Specifying Flocking Behaviour Given these three examples, let us now a cape. Specifying the behaviour of the cape A second example is the animation of the consider the manipulation and compatibility through a sequence using keyframe techniques behaviour of a crowd of people, a school of issues of interactive assemblages. is as tedious as it is time consuming, so much fish or a flock of birds. Traditionally, one so that the ability to experiment with different animates each member of the individu- I/O Compatibility- High variations is extremely limited. On the other ally. But if it is the character of the group as a This approach builds upon the GUI and direct hand, if one has a generic parametrized cloth whole which is of concern, one can offer this manipulation paradigm. However, since we are module (see Figure I), the generation of such level of control (see Figure 2). First, the artist exposing more handles of the deep structure variations becomes relatively easy. First, one specifies a few representative characters, then to modify, assemble and control the surface "dresses" the character.The artist does so at a specifies an area where the crowd should structure, we have a stronger opportunity to

66 November1998 Computer Graphics relationships among a number of variable para- There is a potential for strong tension meters. For example, the cloth module may between manipulating the surface and deep have 20 parameters to adjust to generate an structure when there is a UI or representa- enormous variety of cloth behaviour.Therefore tional discontinuity between the two struc- it is not enough to understand the effect of tures. One step towards alleviating this is to changing one parameter but instead one needs display both structures and show how the to know the relationships among a set oF para- deep structure changes as one manipulates the meters. The fundamental point is that having surface structure. Thus, users can build an simultaneous manipulation through dedicated understanding (perhaps limited) of the deep handles allows us to more effectively and structure.Alternatively, we want to strategically rapidly explore these relationships (i.e., the migTate deep structure up to the surface.We parameter space). suggest defining interactive assemblages as a There are at least three characteristics of way of bridging the gap between these two the interaction that result From the interactive structures. The assemblages encapsulate the assemblages approach: deep structure and offer high level control and Figure 3: Example era prefabricatedbase modelorcor. • Active exploration through the provision handles within the same UI representation of Model componentscompliments or'General Motors. See of rich control, including input handles the surface structure. page 103 for color image. (physical and conceptual) with good The key is to focus on manipulation and compatibility. transformation rather than creation from manipulate these parameters using dedicated • A move from a "specification" to "explo- scratch. Further, we want to manipulate rela- input devices. Thus, there are two main differ- ration" style of interaction. The goal is tionships, both temporal and spatial, rather ences compared to the traditional GUI to create a fertile ground for explo- than individual parameters. Graspable systems approach. First, the quantity of input is ration bur also to provide rich points of should expose the deep and surface structure increased when we move towards offering departure rather than starting from through the use of cognitive and physical multiple, dedicated physical input transducers nothing. handles of control. for adjusting high level controls. This is in • The exposure of portions of the deep structure at a high-level of abstraction. Acknowledgments contrast to conventional GUls having one This research was undertaken under the graphical input device, such as the mouse, This allows users to leverage off of the deep structure using traditional GUI auspices of the User Interface Research Group which serves as a "time-multiplexed" physical at AliaslWavefront_ Special thanks to Gordon handle being repeatedly attached and unat- elements without as much of a need for an expert understanding of the micro Kurtenbach for his discussions and for tached to a variety of logical functions of the reviewing early drafts. GUI. Second, we shift to using more specialized level of the internal structure. form factors and capabilities for our input The net result is to place a manageable and References devices.These dedicated physical handles can compatible handle both physical and cognitive I. ACM SIGAPL, http:llwww.acm.org/ serve as iconic input devices (physical icons) as on the task of creation in this space. sigapl. well as graphical output_ 2. Bereft, D. and A. Witkin. "Large Steps in Conclusions Cloth Simulation:' SIGGRAPH 98 Proceedings, UI to Manipulate Deep Structure -- In UI design we want to manage complexity 1998, pp. 43-54. through the use of structure. Today our Moderately High 3. Conner, D. B., S. S. Snibbe, K. P. Herndon, D. There is a moderately high compatibility with elementary building blocks are at a very fine C. Robbins, R. C. Zeleznick, and A. van Dam. manipulating the deep structures.We are able granularity and the onus is on the user (artist) "Three-dimensional widgers:' Proceedings of to deal with the high overhead of creating, to place direct manipulation UI on relevant the ACM Symposium on Interactive 3D parameters of the deep structures.These para- manipulating and managing structure from Graphics, ] 992, pp. 183-188. meters are passed to procedural scrip~s or these atomic elements. 4. Fishldn, K.R, T. R Moran and B. L. Harrison. which perform the real work. Our belief is that a significant part of the "Embodied User Interfaces: Towards However, we are constrained by using only the complexity of current systems lies within the Invisible User Interfaces," To appear in structure of the data created by the user parameters that are exposed to the user and Proceedings of" EHCI'98 (Heraldion, Greece), by the components which are available for rather than in questions like "what does this 1998. button do?" The problem is, when building assembling. Nevertheless, we are now dealing 5. Fitzmaurice, G.W. Graspable User with higher level components with meaningful from scratch, one often doesn't know what Interfaces, Ph.D. dissertation, of handles. structural features are needed until long after Toronto, htt p:l/~nNw.dgp.toronto.ed ul decisions have been made which make peopledGeorgeFitzmaurice/home.hlurnl, Ul to Manipulate Surface Structure -- High changing that structure, or working within it, 1996. overly difficult, expensive or even impossible. As with the GUI and direct manipulation 6. Fitzmaurice, G.VV., H. IshJi and VV. Buxton. example above, there is a high compatibility We see the same thing in "Bricks: Laying the Foundation for with manipulating the surface structure. where it often takes us multiple iterations to Graspable User Interfaces" ACNI Proceedings get our data structures correct_ In this paper Moreover, the user is augmented by the exten- of CHI'95, 1995, pp. 442-449. sions mentioned above with the handles on we have argued that we can address these 7. Goldberg,A~ and D. Robson. Smalltalk-80:The problems for our users in much the same way higher level components.This gives users capa- Language and its Implementation, Addison- bilities they would never consider doing by we have dealt with them for ourselves: Wesley, Reading, MA, 1983. increase the granularity of the building blocks hand (e.g., adjusting each bird in a flock of 200 8. Ishii, H. and B. Ullmer."Tangible Bits:Towards birds). and shift the balance from a "creation" Seamless Interfaces between People, Bits, approach to one of "clone, modify and The results of these interactive assemblage and Atoms," ACM Proceedings of CH1"97, assemble:' components are a consequence of complex 1997, pp. 234-241.

Computer Gt-dphlr.s November 1998 67 9. Maya 1.0. 1998, AliaslWavefront, htt p://www.aw.sgi.co m/. I 0. Reynolds, C.W."FIocks, herds and schools: A distributed behavioural model," SIGGRAPH87 Proceedings, 1987, pp. 25-34.

George Fitzmaurice is a Research Scientist at AliaslWavefront, Inc. His research interests include computer augmented reality, physical- virtual interfaces, interactive 3D graphics and haptic input devices. He holds a Ph.D. in from the University of Toronto, an M.Sc. in computer science from Brown University and a B.Sc. in mathematics with computer science from the Massachusetts Institute of .

GeorgeW. Fitzmaurice & Bill Buxton AliaslWavefront Inc. Toronto, Ontario Email: {gf, buxton}@aw.sgi.com Web: http://www.dgp.toronto.ed u/ people/GeorgeFitzmaurice/.

68 November1998 Computer Graphics FROM THE EDITOR

Explore the World of Computer Gaming and Computer Graphics

Gordon Cameron performing minor miracles to achieve effects SOFTIMAGE, Inc. that today may look somewhat dated bur in their time were bleeding edge, whilst still The February 1997 issue of Computer Graphics managing to keep in mind that most impor- contained a focus (expertly guest edited by tant, yet too-oft neglected, aspect --- game- Mike Milne) on the entertainment industry, play. but we chose to save an important area of James has done a superb job in gathering this industry for later investigation. It's with together a collection of thoughtful and per- great pleasure that I present that focus on the sonalarticles from both past and present computer games industry in this May 1998 which together form a snapshot of the world issue of Computer Graphics. of computer gaming and computer graphics. Back in the early 'B0s when I was still My thanks go out to all those who con- in school, I was enthusiastically coding away tributed, and especially to James for working on a variety of early machines such as the under extremely tight deadlines. Sinclair ZX8 I, Oric- I, 800XL and Atari ST. At the same time, I spent a great deal of my hard-earned paper-round cash on games for these machines, so it was with great Also, once again we have a tremendous excitement that I recently discovered an series of columns. If you have any comments, I on-line "shrine" to the games and their pro- encourage you to drop a note to the colum- gremmers. James Hague had painstakingly put nisl3. For any general questions, ideas, com- together a list of"classic game programmers" merits, etc, please feel free to contact me at and in addition had interviewed several of the one of the addresses listed below and I'll do more revered game designers for a fascinating my best to answer -- thank you so much for electronic publication entitled Halcyon Days. your letters over the last few months and, Around the same rime, I was trying to put please, keep them cominl! The majority of together an issue on computer graphics and notes from the last issue complimented the the games industry, and so contacted James content, for which I'm extremely grateful on to see if he might be interested in guest edit- behalf of the contributors. However, rather ing such an endeavour. Luckily, he accepted, than print only these, I've decided to wait and ~e issue in your hands now contains the until we have a broad of letters resulting focus. to use in the next Letters column. Over the past decades, computer games Until next issue, all the very besT,,and I have evolved at a remarkable pace. Many of look forward to seeing some of you at the the early titles pushed the platform capabili- upcoming SIGGRAPH 98 25th anniversary ties, but more recently the games industry is conference. proving one of the major factors in pushing Gordon Cameron computer graphics in feneral forward at a breakneck pace -- many of the new titles are SOFTIMAGE, Inc. generating groundbreaking research of their 3510 boul. St-~urent own, and forcing the hardware (and stan- Suite 400 dards) to evolve co keep up, You can pick up a Montn~, Quebec, H2X 2V2 consumer PC with graphics comparable (or Canada superior) to the workstations of a short time Tel: + I-51A,aA,5-1636 ~ 3445 ago, at a fraction of the cost today, and this Fmc + I-514-845-5676 trend is really shaking up our industry and Email:Eordon_cameron~sll~q-aph-ori forcing innovations at a startling rate. At the same time, it is worthwhile to Jook back at the amazing things people were doing in the earlier days of computer gaming, with far more limited resources (both techni- cal and human). These early pioneers were

Computer Graphics ~ 1998 3