Information Visualization Using 3D Interactive Animation

George G. Robertson, Stuart K. Card, andJock D. Mackinlay INFORMATION VISUALIZATION USING 3D INTERACTIVE ANIMATION

innovations are often driven by a combination of technology advances and application demands. On the technology side, advances in interactive computer graphics hardware, coupled with low-cost mass storage, have created new possibilities for information retrieval systems in which UIs could play a more central role. On the application side, increasing masses of information confronting a business or an individual have created a demand for information management applications. In the 1980s, text-editing forced the shaping of the desktop metaphor and the now standard GUI paradigm. In the 1990s, it is likely that information access will be a primary force in shaping the successor to the desktop metaphor. This article presents an experimental system, the Information Visual/mr (see Figure 1), which explores a UI paradigm that goes beyond the desktop metaphor to exploit the emerging generation of graphical personal computersand to support the emerging application demand to retrieve, store, manipulate, and understand large amounts of information. The basic problem is how to utilize advancing graphics cecnnology to lower the cost of finding Information and accessing it once found (the information's "cost structure").

I t b U S T ~ A T I O N : R I C O k a N $ S T U D I O ¢~NIIINUCAT, OI4$OpT~/April 1993/%1.36, No.4 sml models. At the same time, software on Software Architectures for Non- Figure 1. support for real-time operating sys- WIMP User Interfaces [9]. It is this Information tems and emerging industry stan- kind of technology change that is Visualizer Overview dard open graphics libraries (e.g., driving our research in the Informa- tion Visualizer. We take four broad strategies: mak- OpenGL and PEX) are simplifying ing the usel:'S immediate workspace the 3D programming task. The trend larger, enabfing user interaction with will bring these technologies to the Information Access vs. Document Retrieval multiple agents, increasing the real- mass market in the near future. These technology advances have Computer-aided access to informa- time interaction rate between user and created many possibilities for user tion is often thought of in the context system, and using visual abstraction to interface innovation. Yet the basic of methods for library automation. shift information to the perceptual Windows-Icons-Menus-Pointing In particular, document retrieval system to speed information assimila- (WIMP) desktop metaphor has not [19] is usually defined more or less as tion and retrieval. changed much since its emergence in follows: There exists a set of docu- Technology Advances the Alto/Smalltalk work. Nonethe- ments and a person who has an inter- Since the early development of the less, there is a great desire to explore est in the information in some of standard GUI, hardware technology new UI paradigms. Experiments them. Those documents that contain has continued to advance rapidly. with pen-based notebook metaphors, information of interest are relevant, Processor and memory technology virtual reality, and ubiquitous com- others not. The problem is to find all have far greater performance at far puting are proceeding and may and only the relevant documents. lower cost. Specialized 3D graphics eventually influence the mass mar- There are two standard figures of hardware has made it progressively ket. Brown University's Andy van merit for comparing and evaluating faster and cheaper to do 3D transfor- Dam, in several recent conferences retrieval systems: Recall is the per- mations, hidden-surface removal, has exhorted us to break out of the centage of all the relevant documents double-buffered animation, an- desktop metaphor and escape flat- found; and precision is the percentage tialiasing, and lighting and surface land and a recent workshop focused of the documents found that are rel-

58 April 1993/Vol.36, No.4 /|ONNUNICJI~'I'ION|OPTH|A|N evant. While this formulation has als is tuned to the requirements of ondary or tertiary storage with infor- been useful for comparing different the work process using them. mation access from Immediate Stor- approaches, we propose extending Computer screens provide a work- age for information in use, and (2) the document retrieval formulation space for tasks done with the com- considers information access part of to take the larger context into ac- puter. However, typical computer a larger work process. That is, in- count. From a user's point of view, displays provide limited working stead of concentrating narrowly on document retrieval and other forms space. For real work, one often wants the control of a search engine, the of information retrieval are almost to use a much larger space, such as a goal is to improve the cost structure always part of some larger process of dining room table. The Rooms sys- of information access for user work. information use [2]. Examples are tem [10] was developed to extend the With this system, we use four (building an interpreta- WIMP desktop to multiple work- sensemaking methods for improving the cost tion of understanding of informa- spaces that users could switch structure of information access: tion), design (building an artifact), de- among, allowing more information cision making (building a decision and to reside in the immediate work area. 1. Large Workspace. Make the Imme- its rationale), and response tasks (find- The added cost of switching and diate Workspace virtually larger, so ing information to respond to a finding the right workspace was re- that the information can be held in query). duced by adding the ability to share low-cost storage In each of these cases: the same information objects in dif- 2. Agents. Delegate part of the work- ferent workspaces. Rooms also had load to semiautonomous agents 1. Information is used to produce an overview and other navigational 3. Real-Time Interaction. Maximize more information, or to act di- aids as well as the ability to store and interaction rates with the human rectly retrieve workspaces, all to remove user by tuning the displays and re- 2. The new information is usually at the major disadvantages of multiple sponses to real-time human action a higher level of organization rel- desktops. constants ative to some purpose The essence of our proposal is to 4. Visual Abstractions. Use visual ab- If we represent the usual view of evolve the Rooms multiple desktop stractions of the information to information retrieval as Figure 2(a), metaphor into a workspace for infor- speed assimilation and pattern detec- we can represent this extended view mation access--an Information Work- tion as Figure 2(b). Framing the problem space [2]. Unlike the conventional in- in this way is suggestive: what the formation retrieval notion of simple user needs is not so much informa- access of information from some dis- Figure 2. (a) Tradi- tion retrieval itself, but rather the tal storage, an information work- tional Information retrieval formula- amplification of information-based space (1) treats the complete cost tion and (D) refor- work processes. That is, in addition structure of information, integrating mulatlon with to concern with recall and precision, information access from distant, sec- context of use we also need to be concerned with (b) Amplification of (a) Information Retrieval reducing the time cost of informa- Information-Intensive Work tion access and increasing the scale of information that a user can handle at one time.

Information Workspaces From our observations about the problem of information access [2], we were led to develop UI paradigms oriented toward managing the cost structure of information-based work. This, in turn, led us to be concerned not just with the retrieval of information from a distant source, but also with the accessing of that information once it is retrieved and in use. The need for a low-cost, immediate storage for accessing objects in use is common to most kinds of work. The common solution is a workspace, whether it be a woodworking shop, a laboratory, or an office. A workspace is a special environment in which the cost structure of the needed materi-

COMMUNICATIONS 011TNU ACU/April1993/Vol.36, No.4 ~ Table 1. Techniques used in the Information visualizer to increase These define the goals for our UI information access per unit cost paradigm. Each of these is intended to decrease the costs for performing information-intensive tasks, or, alter- natively, to increase the scope of information that can be utilized for the same cost. Figure 3 shows how these goals are applied to the reformulated information access problem shown in Figure 2(b). The Information Visualizer system is our experimental embodiment of the Information Workspace con- cept with mechanisms for addressing each of these system goals (see Table 1): 1) [Large Workspace]. We use two methods to make the workspace larger: We add more (virtual) screen Table 2. In|k)rmation Visualizer solutions to basic Ul problems. space to the Immediate Workspace by using a version of the Rooms system. We increase the density of information that can be held in the same screen space by using animation and 3D perspective. 2) [Agents]. To delegate part of the workload, we use agents to conduct searches, to organize information into clusters, or design presentations of information. We manage this by means of a schedul- ing architecture, called the Cognitive Informatk Coprocessor [17], that allows multi- workspa~ ple display and application processes to run together. A kind of user interface agent, called Interactive Objects, is used to control and communicate with the system. 3) [Real-time Inter- action]. To maximize human interaction rates, we use the properties of Make Larger the scheduler to provide highly interactive animation and communica- Offload Work tion with the Interactive Objects. To with Agents tune the system to human action times, we require certain classes of actions to occur at set rates. To en- force these rates under varying com- Rapid interact at real-time ~ putational load, we use a Governor human rates mechanism in our scheduler loop. 4) [Visual Abstractions]. To speed the user's ability to assimilate information and find patterns in it, we use Make faster visualization of different abstract in- witJ~ information formation structures, including lin- Visualization ear structures, hierarchical struc-

Figure 3. Improving the information cost structure In the Information access model

60 April 1993/Vol.36, No.4 /flMNIlUlIIII~'I'IONIllOPTIHINIIACN tures, continuous data, and spatial 1. The Multiple Agent Problem. How Even the UI may itself contain agents data. can the architecture provide a sys- (e.g., presentation agents). These There have been many systems tematic way to manage the interac- additional agents have their own that have supported interactive ani- tions of multiple asynchronous time constants and languages of in- mation-oriented UIs, starting with agents? teraction that must be accommo- Ivan Sutherland's thesis [23] at the 2. The Animation Problem. How can dated by the UI. dawn of computer graphics. As with the architecture provide smooth in- Impedance matching can be diffi- Sutherland's thesis, early examples teractive animation and solve the cult to accomplish architecturally required specialized and/or expen- Multiple Agent problem? because all agents want rapid interac- sive computing machinery and were 3. The Interaction Problem. How can tion with no forced waiting on other oriented toward specialized tasks. 3D widgets be designed and coupled agents, and the user wants to be able Cockpit simulation systems are a to appropriate application behavior? to change his or her focus of atten- good example. The architectures for 4. The Viewpoint Movement Problem. tion rapidly as new information be- such systems share the animation- How can the user rapidly and simply comes available. For example, if a loop core with our system. The drop move the point of view around in a user initiates a long search that pro- in cost for 3D animated systems and 3D space? vides intermediate results as they the increase in capability has acceler- 5. The Object Movement Problem. How become available, the user should be ated experiments in using this tech- can objects be easily moved about in a able to abort or redirect the search at nology as the basis of a new mass 3D space? any point (e.g., based on perception market user interface paradigm. One of the intermediate results), without 6. The Small Screen Space Problem. strategy has been to work up from waiting for a display or search pro- How can the dynamic properties of building blocks. A. Van Dam's group cess to complete. The UI architec- the system be utilized to provide the at Brown has been working on an ture must provide a systematic way to user with an adequately large work- object-oriented framework for inter- manage the interactions of multiple space? active animation, 3D widgets [3], and asynchronous agents that can inter- modeling time in 3D interactive ani- Many of these problems are well rupt and redirect one another's mation systems. Silicon Graphics has known. The Multiple Agent and work. recently introduced a high-level 3D Animation problems are less obvious, The Animation Problem. Over the toolkit, called Inventor. Another tack and since they define the basic orga- last 65 years, animation has grown has been to drive the development by nization of the Information Visual- from a primitive art form to a very focusing on applications, for exam- izer, we describe them in more detail. complex and effective discipline for ple, continuously running physical The Multiple Agent Problem. We communication. Interactive anima- simulations. M. Green's group at the want our architecture to support tion is particularly demanding archi- University) of Alberta has developed multiple agents to which the user can tecturally, because of its extreme a Decoupled Simulation Model for delegate tasks. In fact, we have previ- computational requirements. virtual reality systems [20]. Their ously argued [17] that T.B. Sheri- Smooth interactive animation is architectural approach is similar to dan's analysis of the supervisory con- particularly important because it can ours, but focuses more on continu- trol of semiautonomous embedded shift a user's task from cognitive to ously running simulations. D. Zeltzer systems [21] can be adapted to de- perceptual activity, freeing cognitive and colleagues at MIT [25] have built scribe the behavior of an interactive processing capacity for application a constraint-based system for interac- system as the product of the interac- tasks. For example, interactive ani- tive physical simulation. Our system, tions of (at least) three agents: a user, mation supports object constancy. by contrast, is oriented toward the a user discourse machine (the UI), and a Consider an animation of a complex access and visualization of abstract task machine or application. These object that represents some complex nonphysical information of the form agents operate with very different relationships. When the user rotates that knowledge workers would en- time constants. For example, a search this object, or moves around the ob- counter. process in an application and the ject, animation of that motion makes graphical display of its results may be it possible (even easy, since it is at the UI Architecture slow, while the user's perception of level of perception) for the user to In order to achieve the goals set forth displayed results may be quite fast. retain the relationships of what is dis- in Table 1 we have been led to a UI The UI must provide a form of "im- played. Without animation, the dis- paradigm involving highly interac- pedance matching" (dealing with dif- play would jump from one configu- tive animation, 3D, agents, and visu- ferent time constants) between the ration to another, and the user would alizations. This is one of the UI re- various agents as well as translate have to spend time (and cognitive gimes now being made practical by between different languages of inter- effort) reassimilating the new dis- current and predicted advances in action. The application itself may be play. By providing object constancy, hardware and software technology. broken into various agents that sup- animation significantly reduces the There are several problems, how- ply services, some of which may run cognitive load on the user. ever, which need to be addressed in on distributed machines (e.g., an The Animation Problem arises order to realize such a UI paradigm: agent to filter and sort your mail). when building a system that attempts

¢~MUNICAVlONg OP'm~| AcM/Apri11993/Vol.36, No.4 61[ ~ ~ E N~R~ ~ZOa

to provide smooth interactive anima- scheduler loop and double-buffered lated environments), called 3D/ tion and solve the Multiple Agent graphics. In order to maintain the il- Rooms; supports for navigating problem. The difficulty is that lusion of animation in the world, the around 3D environments; and sup- smooth animation requires a fixed screen must be repainted at least port for Interactive Objects, which pro- rate of guaranteed computational every 0.1 second [1]. The Cognitive vide basic input/output mechanisms resource, wlhile the highly interactive Coprocessor therefore has a Governor for the UI. The task machine (which, and redirectable support of multiple mechanism that monitors the basic for the Information Visualization asynchronous agents with different cycle time. When the cycle time be- application, is a collection of visualiz- time constants has widely varying comes too high, cooperating render- ers) couples with the Cognitive computational requirements. The UI ing processes reduce the quality of Copocessor in various ways. More architecture must balance and pro- rendering (e.g., leaving off most of details of this architecture can be tect these very different computa- the text during motion) so that the found in [17]. tional requirements. cycle speed is increased. In fact, the animation problem is The immediate response time constant. Interactive Objects one aspect of a broader Real-Time A person can make an unprepared The basic building block in the In- Interaction problem. Services need response to some stimulus within formation Visualizer, called Interac- to be delivered under real-time about a second [15]. If there is more tive Objects, forms the basis for cou- deadline, under varying load, while than a second, then either the listen- pling user interaction with simultaneously handling the Multi- ing party makes a back-channel re- application behavior and offloading ple Agent problem. sponse to indicate that he is listening work to an agent to handle user in- (e.g., "uh-huh") or the speaking teraction. Interactive Objects are a The Cognitive Coprocessor party makes a response (e.g., "uh...") generalization of Rooms Buttons Table 2 summarizes the Information to indicate he is still thinking of the [10]. They are used to build complex Visualizer's solutions to each of the next speech. These serve to keep the 3D widgets that represent informa- problems described earlier. The next parties of the interaction informed tion or information structure. few sections describe these solutions. that they are still engaged in an inter- Rooms Buttons are used for a vari- The heart of the Information Vis- action. In the Cognitive Coprocessor, ety of purposes, such as movement, ualizer arclhitecture is a controlled- we attempt to have agents provide new interface building blocks, and resource scheduler, the Cognitive status feedback at intervals no longer task assistance. A Button has an ap- Coprocessor architecture, which than this constant. Immediate re- pearance (typically, a bitmap) and a serves as an animation loop and a sponse animations (e.g., swinging the selection action (a procedure to exe- scheduler for Sheridan's three agents branches of a 3D tree into view) are cute when the Button is 'pressed'). and additional application and inter- designed to take about a second. If The most typical Button in Rooms is face agents. It manages multiple the time were much shorter, then the a door--when selected, the user asynchronous agents that operate user would lose object constancy and passes from one Room to another. with diffe~rent time constants and would have to reorient himself. If Buttons are abstractions that can be need to interrupt and redirect one they were much longer, then the user passed from one Room to another, another's work. These agents range would get bored waiting for the and from one user to another via from trivial agents that update dis- response. email. Interactive Objects are similar play state to continuous-running The unit task time constant. Finally, a to Buttons, but are extended to deal simulations and search agents. This user should be able to complete some with gestures, animation, 2D or 3D architecture provides the basic solu- elementary task act within about 10 appearance, manipulation, object- tion to the Multiple Agent and Ani- seconds (say, 5 to 30 seconds) [1, 15]. relative navigation, and an extensible mation problems. This is about the pacing of a point set of types. The Cognitive Coprocessor is an and click editor. Information agents An Interactive Object can have impedance matcher between the cog- may require considerable time to any 2D or 3D appearance defined by nitive and perceptual information complete some complicated request, a draw method. The notion of selec- processing requirements of the user but the user, in this paradigm, always tion is generalized to allow mouse- and the properties of these agents. In stays active. A user can begin the based gestural input in addition to general, these agents operate on time next request as soon as sufficient in- simple 'pressing'. Whenever a user constants different from those of the formation has developed from the gestures at an Interactive Object, a user. There are three sorts of time last request or even in parallel with it. gesture parser is involved that inter- constants for the human that we The basic control mechanism prets mouse movement and classifies want to tune the system to meet: per- (inner loop) of the Cognitive Copro- it as one of a small set of easily differ- ceptual processing (0.1 second) [1], cessor is called the Animation Loop entiated gestures (e.g., press, rubout, immediate response (1 second) [15], (see Figure 4). It maintains a Task check, and flick). Once a gesture has and unit task (10 seconds) [15]. Queue, a Display Queue, and a Gover- been identified, a gesture-specific The perceptual processing time con- nor. Built on top of the Animation method is called. These gesture stant. The Cognitive Coprocessor is Loop is an information workspace methods are specified when the In- based on a continuously running manager (and support for 3D simu- teractive Object is created. The ges-

62 April 1993/Vo1.36, No.4 /COlllEIUHIUTIOH|OF'i*HIIACEi .EN

ture parser can be easily extended to The heart of the allow additional gestures and gesture methods, as long as the new gestures Information Visualizer are easily differentiated from other gestures. _ architecture is a controlled There are a number of types of Interactive Objects. In the current resource scheduler, the implementation, these include static text, editable text, date entry, num- Cognitive Coprocessor ber entry, set selection, checkmark, simple button, doors, sliders, and architecture, which serves as an thermometers (for feedback and progress indicators). The basic set of 3D widgets supported for Interactive animation loop and a schedulerfor Shen'dan's Objects can be easily extended. Interactive Objects are generalized three agents and additional application to the point that every visible entity in the simulated scene can be an In- and interface agents. teractive Object (and should be, so that object-relative navigation is consistent across the scene). Thus, the surfaces of the 3D Room (the walls, floor, and ceiling) are Interactive Objects. All the controls (e.g., buttons, sliders, thermometers, text, and editable text) are Interactive Objects. And finally, the application-specific artifacts placed in the room are In- teractive Objects.

Search Agents Search agents are also used to offload user work. The Information Visualizer uses an indexing and search subsystem [5], which allows search for documents by keyword or by iterative "relevance feedback" (e.g., find the documents most like this document). Associative retrieval based on such linguistic searches can be used to highlight portions of an information visualization. Thus we can combine traditional associative searches with structural browsing. In addition, clustering agents are used to organize information. Using Figure 4. Cognitive a near-linear clustering algorithm Coprocessor [4], which allows interactive use of Interaction clustering, a structure can be in- architecture duced on an unstructured (or par- tially structured) body of information. There are several ways this can be of use. For unstructured information, a user can induce a subject hierarchy, which can then be browsed with our hierarchy visualization tools. For information that already has a structure, the clustering results sometimes reveal problems with the existing structure. In general, if a

¢OMNUNICA'I'IONIOFTHIIIACN/Apri]1993/Vo1.36, No.4 ~ user is unsure about the content of a Current techniques for moving the slower for control near the view- corpus, and therefore unsure of viewpoint [13] are not very satisfac- point. what kinds of queries to make, clus- tory for targeted movement. They POI logarithmic flight and object tering can provide an overview of the typically exhibit one or more of the of interest logarithmic manipulation content of ~that corpus. following three problems: (1) ineffi- both allow simple, rapid movement cient interactions and movement tra- of the viewpoint and of objects in a 3D Navigation and Manipulation jectories, typically caused by 2D 3D space over multiple degrees of In virtual 3D workspaces, techniques input devices; (2) difficulties control- freedom and scales of magnitude are required for moving the user (the ling high velocities when the tech- with only a mouse and two keyboard viewpoint) and objects around the nique is based on flying or steering keys. We believe these techniques space. The Information Visualizer the viewpoint through the work- provide a mouse-based solution for currently has five of these as building space; and (3) limits on human reach the viewpoint movement and object blocks, with others under develop- and precision when the technique is movement problems that are as good ment: based on directly positioning the or even better than those requiring viewpoint. special 3D devices. The chief advan- 1. The Walking Metaphor Most viewpoint movement tech- tage of a mouse-based solution is that 2. Point of Interest Logarithmic niques focus on schemes for directly mice are ubiquitous. Also, many Flight controlling the six degrees of free- users of information visualization 3. Object of Interest Logarithmic dom of viewpoint movement (3 posi- (office workers, for example) are not Manipulation tion and 3 orientation) or their rate likely to be willing to wear special 4. Doors derivatives--a complex control task. equipment (such as gloves and hel- 5. Overview Our solution is to have the user select mets). Even so, the techniques could Walking Metaphor. The 'Walking a point of interest (POI) on the sur- be adjusted to work with 3D devices Metaphor' [13] has virtual joystick face of an object and use the distance such as the glove. controls superimposed as heads-up to this POI to calculate a logarithmic Doors. The 3D/Rooms system sup- displays on the screen and controlled motion function. Two keys on the ports Doors that allow a user to move by the mouse. The controls are oper- keyboard are used to indicate loga- from one room (or workspace) ators related to the way a human rithmic motion along the ray toward through to a home position in an- body might be moved (one control and away from the POI. The view- other room. The Door is an Interac- for body motion forward, backward, point is automatically oriented dur- tive Object that supports either man- turn-left, or turn-right; a second for ing the flight to face the surface ual control or scripted animation of motion in tlhe plane of the body: left, being approached by using the sur- opening and walking through to the right, up, or down; and a third for face normal at the POI. Another other room. rotating the head left, right, up, or control allows movement perpendic- Overview. As with Rooms, 3D/ down). This scheme is fairly general ular to the surface normal. This al- Rooms contains an Overview (see Fig- and works well for exploratory lows for scrolling over extended ob- ure 1) allowing the user to view all movement, which has no particular jects (for example, a virtual the 3D workspaces simultaneously. object as its target. blackboard) or circumnavigation This is a navigation technique that Large information spaces, how- around spherical objects (for exam- lets the user view all the rooms and ever, involve numerous objects and/ ple, a virtual globe.) go to any room directly. In 3D/ or highly detailed objects that re- Object of Interest Logarithmic Rooms the user can also reach into quire the user to move back and Manipulation. Logarithmic motion the Rooms from the Overview, move forth from global, orienting views to can also be used to manipulate ob- about in them, and manipulate their manipulate detailed information. jects with the same UI as POI view- objects. Therefore, an important require- point movement. The mouse cursor ment for such systems is a movement is used to control a ray that deter- 3D/ROOmS technique that allows the user to mines the lateral position of the ob- 3D/Rooms extends the logic of our move the viewpoint (1) rapidly ject of interest (given the viewpoint Rooms system to three dimensions. through large distances, (2) with such coordinates) and the same keyboard In the classical desktop metaphor control that the viewpoint can ap- keys are used to control the position and the original Rooms system, the proach very close to a target without of the object on the ray. However, view of a Room is fixed. In 3D/ collision. We call this the problem of the user must be able to control ob- Rooms, the user is given a position rapid and controlled, targeted 3D move- ject position at a distance, where log- and orientation in the Room, and can ment [12]. arithmic motion is not effective. The move about the Room, zoom in to Point of lnterest (POI) Logarithmic solution is to use an acceleration mo- examine objects closely, look around, Flight. Our second navigation tech- tion clipped by a logarithmic motion. or even walk through doors into nique uses. a point of interest loga- The object moves slowly at first (al- other Rooms. Thus 3D/Rooms is the rithmic movement algorithm for lowing control at a distance), then same as Rooms, except that visualiza- very rapid, but precise movement accelerates toward the viewpoint, tion artifacts (implemented as Inter- relative to objects of interest [12]. and finally moves logarithmically active Objects) replace a collection of

64 April 1993/Vol.36, No.4 /COIlIHUlllCAT|OIdl OlU THE ACM windows, and users can have arbi- understanding the structure was still are visible in the 3D/Rooms Overview trary positions and orientations in quite difficult. of Figure 1. The visualizers attempt the Rooms. The n-Vision system [7] exploits to present abstractions of large The effect of 3D/Rooms is to make 3D to visualize n-dimensional busi- amounts of data tuned to the pattern the screen space for immediate stor- ness data. Multivariate functions are detection properties of the human age of information effectively larger displayed in nested coordinates sys- perceptual system. For example, they (in the sense that the user can get to a tems, using a metaphor called use color, lighting, shadow, transpar- larger amount of ready-to-use infor- worlds-within-worlds. Although n- ency, hidden surface occlusion, con- mation in a short time). The effect of Vision focuses on continuous multi- tinuous transformation, and motion rapid zooming, animation, and 3D is variate functions, it does exploit the cues to induce object constancy and to make the screen space effectively human 3D perceptual apparatus to 3D perspective. denser (in the sense that the same visualize abstract data. amount of screen can hold more ob- Silicon Graphics has recently re- Visualizing Hierarchical Structure: jects, which the user can zoom into or leased an unsupported system, called Cone Trees animate into view in a short time). By File System Navigator (FSN) [24], Hierarchies are almost ubiquitous, manipulating objects or moving in which explores what they call Infor- appearing in many different applica- space, the user can disambiguate mation Landscapes. In this system tions, hence are good information images, reveal hidden information, the file system hierarchy is laid out structures to exploit. In some cases, or zoom in for detail--rapidly ac- on a landscape, with each directory arbitrary graphs can be transformed cessing more information. Both the represented by a pedestal which has into hierarchies (with auxiliary links), techniques for making the Immedi- boxes representing individual files so the utility of hierarchy visualiza- ate Storage space virtually larger and on top of it. They effectively use the tion is further enhanced. the techniques for making the space 3D space to present structure, while Cone Trees are hierarchies laid virtually denser should make its ca- using box size to represent file size out uniformly in three dimensions. pacity larger, hence the average cost and color to represent age. FSN uses Figure 7 is a snapshot of a simple of accessing information lower, a technique called 'artificial perspec- Cone Tree. Nodes are drawn as 3- × hence the cost of working on large tive', a form of fisheye effect [8], to 5-inch index cards. The top of the information-intensive tasks lower. make more effective use of screen hierarchy is placed near the ceiling of space. the room, and is the apex of a cone Information Visualization In the Information Visualizer, we with its children placed evenly Recent work in scientific visualization have explored 3D visualizations for spaced along its base. The next layer shows how the computer can serve as some of the classical data organiza- of nodes is drawn below the first, an intermediary in the process of tions: with their children in cones. The as- rapid assimilation of information. pect ratio of the tree is fixed to fit the 1. Hierarchical: The Gone Tree visu- Large sets of data are reduced to room. Each layer has cones of the alization [18] (see following descrip- graphic form in such a way that same height (the room height di- tion). human perception can detect pat- vided by the tree depth). Cone base 2. Linear: The Perspective Wall visu- terns revealing underlying structure diameters for each level are reduced alization [14] (see following descrip- in the data more readily than by a in a progression so that the bottom tion). direct analysis of the numbers. Infor- layer fits in the width of the room. 3. Spatial: The spatial structure of a mation in the form of documents The body of each cone is shaded building (see Figure 5) can be used as also has structure. Information visuali- transparently, so that the cone is eas- a structural browser for people. Se- zation attempts to display structural ily perceived yet does not block the lecting an organization will produce relationships and context that would view of cones behind it. The display the names and pictures of its mem- be more difficult to detect by individ- of node text does not fit the aspect bers and select their offices. Clicking ual retrieval requests. Although ratio of the cards very well, hence on offices retrieves their inhabitants. much work has been done using 3D text is shown only for the selected 4. Continuous Data: In the Data graphics to visualize physical objects path. Figure 8 shows an alternative Sculpture (see Figure 6), the user can or phenomena, only a few systems layout, which is horizontally oriented walk around or zoom into this visual- have exploited 3D visualization for and has text displayed for each node. ization containing over 65,000 sam- visualizing more abstract data or in- When a node is selected with the pling points as if it were a sculpture formation structure. mouse, the Cone Tree rotates so that in a museum. The SemNet [6] system is an early the selected node and each node in 5. Unstructured: The Information example of the exploitation of 3D the path from the selected node up Grid [16] is a 2D visualization for visualization of information struc- to the top are brought to the front unstructured information. tures. The structures visualized in and highlighted. The rotations of SemNet were mostly large knowl- These visualizations use interac- each substructure are done in paral- edge bases, and were often arbitrary tive animation to explore dynami- lel, following the shortest rotational graphs. The results tended to be cally changing views of the informa- path, and are animated so the user cluttered, and the cognitive task of tion structures. More visualizations sees the transformation at a rate the

¢~HUNICATION80B THR ACM/Apri11993/Vol.36, No.4 65 D D

L~ Data: fSO _one

66 April 1993/Vol.36, No.4 /¢OIdllilUNICJil"IION|OFTIIlilCN ...... IUIIII il I1[11 Ilnill I I Ill II ~II ......

COImHUNICA~rloMIoluTIIIIIAClU/April1993/Vol.36, No.4 ~ perceptual system can track. Typi- fisheye view effect from their use of area network browsers. cally, the ,entire transformation is 3D perspective. Our fisheye view ef- done in about a second. The tree can fect is further enhanced by selection Visualizing Linear Structures: also be rotated continuously to help rotation, because the user can easily Perspective Wall the user understand substructure select a new object of interest and Case studies indicate that tasks often relationships. have the structure quickly reconfig- involve spanning properties (such as The hierarchy is presented in 3D ure to highlight it. time) that structure information line- to maximize effective use of available arly [ 14]. This linear structure results screen spaoe and enable visualization Cone Tree: Examples in 2D layouts with wide aspect ratios of the whol[e structure. A 2D layout In Figures 7 and 8, Cone Trees are that are difficult to accommodate in a of the same structure using conven- used for a file browser, showing one single view. The principal obstacles tional graph layout algorithms would user's directory hierarchy, with each to a visualization of linear informa- not fit on the screen. The user would node representing a directory in a tion structures are (1) the large have to either scroll through the lay- Unix file system. Information access amount of information that must be out or use a size-reduced image of is done on file names and file con- displayed and (2) the difficulty of the structure. Most hierarchies en- tents. We have also visualized an en- accommodating the extreme aspect countered in real applications tend to tire Unix directory hieffarchy, which ratio of the linear structure on the be broad and shallow. This typical contained about 600 directories and screen. These problems make it diffi- hierarchy aspect ratio is problematic 10,000 files. To our knowledge, cult to see details in the structure for 2D layouts (a size-reduced image when we did this in 1989, it was the while retaining global context. may look like a line with little detail). first time anyone had ever visualized A common technique for viewing A 3D layout uses depth to fill the an entire Unix file system. The direc- linear information while integrating screen with more information. tory hierarchy was surprisingly shal- detail and context is to have two si- To see dais effect analytically, con- low and unbalanced. Since then, both multaneous views: an overview with a sider the aspect ratio of a 2D tree, FSN [24] and TreeMaps [11] (a 2D scale-reduced version of a work- ignoring the size of the nodes. If visualization technique) have been space, and a detailed view into the there are l levels and the branching used to visualize entire file systems. workspace where work can be ac- factor is b, the width of the base is Cone Trees have also been used as complished. The overview typically bt-1 and the aspect ratio is bt-1/l. As- an organizational structure browser. contains an indication of the detailed pect ratio for 2D trees increases Search is done in a database of facts view's location that can be manipu- nearly exponentially, and is much about each person (e.g., title or office lated for rapid movement through worse as the branching factor gets location) and a database of autobiog- the workspace. However, a uniform larger. Figure 9 shows what happens raphies. Users can search for other scale reduction of the workspace for small branching factors (b = 2 people with biographies similar to a causes it to appear very small. Fur- and b = 3). In contrast, the Cone selected person's biography. We have thermore, important contextual in- Tree aspect ratio is fixed to fit the implemented several organization formation, such as the neighborhood room by adjusting level height and charts. The largest contained the top of the viewing region, is just as small cone diameters to fit. The line near 650 Xerox Corporation executives. as unimportant details. Finally, if the the bottom of Figure 9 is a typical Since this requires 80 pages on display space for the overview is in- aspect ratio of four to three. Al- paper, this is the first time the orga- creased to make the workspace ap- though fixing the aspect ratio intro- nization chart could be seen in one pear larger, the space for the work- duces a limitation on the number of visualization. ing view becomes too small. levels that can be effectively dis- We also have used Cone Trees to Rather than a uniform overview of played (about 10), it makes Cone visualize a company's operating plan. a workspace, an effective strategy is Trees independent of the number of Text narratives describe each portfo- to distort the view so that details and nodes, branching factor, and num- lio, program, and project, and are context are integrated. Fisheye views ber of levels (until the limit is augmented with project highlights [8] provide such distorted views by reached). (brief statements of milestones and thresholding with Degree of Interest In addition to perceptual effects achievements) from the previous functions to determine the contents already mentioned, the 3D perspec- year. A typical search finds all proj- of the display. However, threshold- tive view of Cone Trees provides a ects related to a selected project. ing causes the visualization to have fisheye view [8] of the information, Cone Tree manipulation mecha- gaps that might be confusing or diffi- without having to describe a degree nisms are used during early stages of cult to repair. Furthermore, gaps can of interest function, as in general operating plan definition to reorga- make it difficult to change the view. fisheye view mechanisms. The se- nize the plan to a desired structure. The desired destination might be in lected path is brighter, closer, and Other potential applications in- one of the gaps, or the transition larger than other paths, both because clude software module management, from one view to another might be of the 3D perspective view and be- document management (library confusing as familiar parts of the vis- cause of coloring and simulated structure and book structure), ob- ualization suddenly disappear into lighting. SemNet [6] also reported a ject-oriented class browsers, and local gaps.

68 April 1993/Vol.36, No.4/¢OMILIUIIICATIOllSOFTHIIACH Spence and Apperley developed an early system called the Bifocal Display that integrates detail and context through another distorted view [22]. This 2D design is a concep- tual ancestor of the Perspective Wall system. The Bifocal Display was designed for professional offices that contain information subdivided into a hierarchy of journals, volumes, issues and articles. Abstractly, the workspace consists of information items positioned in a horizontal strip. The display is a combination of a detailed view of the strip and two distorted views, where items on either side of the detailed view are distorted horizontally into narrow vertical strips. For example, the detailed view might contain a page from a journal and the distorted view might contain the years for various issues of the journal. The Perspective Wall integrates detailed and contextual views to support the visualization of linearly structured information spaces, using interactive 3D animation to address 1 2 3 4 5 6 7 8 9 10 the integration problems of the Bifo- cal Display. The Perspective Wall folds a 2D layout into a 3D wall that smoothly integrates a central region for viewing details with two perspective regions, one on each side, for viewing context (see Figure 10). This intuitive distortion of the layout provides efficient space utilization and allows smooth transitions of views. Space utilization analysis of the Per- spective Wall technique indicates at least a three-fold improvement over simple 2D visualization (see [14] for details).

Perspective Wall: Implementation The Perspective Wall's physical metaphor of folding is used to distort an arbitrary 2D layout into a 3D visualization (the wall), while automatically retaining any 2D task-specific fea- tures. More important, no special large- and small-scale versions of items must be designed (as in the Bi- focal Display). The perspective pan- els are also shaded to enhance the Figure g. Aspect perception of 3D. This intuitive visu- ratio of 2D and alization provides efficient space uti- 3D trees lization for 2D layouts with wide as- Figure 10. pect ratios. In addition, the vertical Perspective Wall dimension of the wall can be used to visualization of flies

¢OMMUn|cmqrloNs OPTHII ACM/Apri11993/Vol.36, No.4 69 visualize layering in an information tive animation, and the human per- 2. Card, S.K., Robertson, G.G. and Mac- space. The Perspective Wall in Fig- ceptual system can be effectively k.inlay, J.D. The Information Visual- ure 10 holds cards that represent exploited to improve management of izer: An information workspace. In files in a computer system that are and access to large information Proceedings of SIGCHI'91, 1991, pp. 181-188. structured by modification date (hor- spaces. There is a large class of appli- 3. Conner, D.B., Snibbe, S.S., Herndon, izontally) and file type (vertically). cations for which these techniques K.P., Robbins, D.C. and Zeleznik, The perspective view has the further work. It seems clear that interactive R.C. Three-dimensional widgets. In advantage that it makes the neigh- animation can effectively shift cogni- Proceedings of the 1992 Symposium on borhood of the detailed view larger tive processing load to the perceptual Interactive 3D Graphics, 1992, pp. 183- than more distant parts of the con- system. And it seems plausible (but 188. textual view. not yet proved) that 3D can be used 4. Cutting, D.R., Karger, D.R., Peder- A major advantage of the Perspec- to maximize effective use of screen sen, J.O. and Tukey, J.W. Scatter/ tive Wall is that its intuitive 3D meta- space. Formal user studies are Gather: A cluster-based approach to phor for distorting 2D layouts allows needed to verify and expand on hrowsing large document collections. In Proceedings of SIGIR'92, 1992, pp. smooth transitions among views. these conclusions. 318-329. When the user selects an item, the The Information Visualizer we 5. Cutting, D.R., Pedersen, J.O. and wall moves that item to the center have described is an experimental Halvorsen, P.K. An object-oriented panel with a smooth animation, as if system being used to develop a new architecture for text retrieval. In Pro- it were a sheet in a player piano mov- UI paradigm for information re- ceedings of RIAO'91, Intelligent Text and ing selected notes to the center of trieval, one oriented toward the Image Handling, 1991, pp. 285-298. view. This animation helps the user amplification of information-based 6. Fairchild, K.M., Poltrock, S.E. and perceive object constancy, which work. It is based on our analysis of Furnas, G.W. Semnet: Three-dimen- shifts to the perceptual system work several aspects of information use sional graphic representations of that would otherwise have been re- that have led us to reframe the infor- large knowledge bases. In Cognitive Science and its Applications for Human- quired of the cognitive system to mation retrieval problem as a prob- Computer Interaction, Guindon, R. Ed, reassimilate the view after it had lem in the cost structuring of an in- Lawrence Erlbaum, 1988. changed. Furthermore, the relation- formation workspace. This, in turn, 7. Feiner, S. and Beshers, C. Worlds ship between the items in the detail has led us to evolve the computer within worlds: Metaphors for explor- and contexl: is obvious. Items even desktop metaphor toward (1) the ing n-dimensional virtual worlds. In bend around the corner. Cognitive Coprocessor interaction Proceedings of the UIST'90, 1990, pp. The Perspective Wall has the addi- architecture (to support highly cou- 76-83. tional feature that the user can adjust pled iterative interaction with multi- 8. Furnas, G. W. Generalized fisheye the ratio of detail and context. This is ple agents), (2) 3D/Rooms (to man- views. In Proceedings of SIGCHI'86, quite important when the detailed age information storage cost 1986, pp. 16-23. view contains a lot of information. hierarchies), and (3) information vis- 9. Green, M. and Jacob, R. SIG- The metaphor is to stretch the wall ualization (to increase the level of GRAPH'90 Workshop Report: Soft- like a sheet of rubber. information abstraction to the user). ware architectures and metaphors for The Perspective Wall has been Collectively these techniques alter non-WIMP user interfaces. Comput. used to visualize various types of in- the cost of retrieving information Graph. 25, 3 (July 1991), 229-235. formation. ]Figure 10 represents files from secondary storage and the cost 10. Henderson, D.A. and Card, S.K. Rooms: The use of multiple virtual in a file system that are classified by of using it in workspace. Future work workspaces to reduce space conten- their modification date and file type. will focus on how these techniques tion in a window-based graphical user Vacations and other work patterns aid in sensemaking, and will continue interface. ACM Trans. Graph. 5, 3, are clearly visible. The technique has to explore the rich space of tech- (July 1986), pp. 211-243. also been used for corporate memo- niques for visualizing information. 11. Johnson, B. and Shneiderman, B. randa and reports, which also have a Space-filling approach to the visuali- useful linear structure. The tech- Acknowledgments zation of hierarchical information nique is particularly effective when We would like to thank a number of structures. In Proceedings IEEE Visu- combined with a retrieval technique people who have been involved in alization '91, 1991, pp. 284-291. that allows the user to select an item the use and development of the In- 12. Mackinlay, J.D., Card, S.K. and Rob- and find similar related items. The formation Visualizer: Doug Cutting, ertson, G.G. Rapid controlled move- Perspective Wall makes it easy to vis- Peter Pirolli, Kris Halverson, Walt ment through a virtual 3d workspace. ualize the resuhs of such retrievals Johnson, Jan Pedersen, Ramana In Proceedings of SIGGRAPH '90, because it shows all similar items si- Rao, Dan Russell, Mark Shirley, 1990, pp. 171-176. 15. Mackinlay, J.D., Card, S.K. and Rob- multaneously and in context. Mark Stefik, and Brian Williams. [] ertson, G.G. A semantic analysis of References the design space of input devices. Summary 1. Card, S.K., Moran, T.P. and Newell, Human-Computer Interaction, 5, 2-3, To summarize, we believe that the A. The Psychology of Human-Computer 1990, pp. 145-190. structure of information, the emerg- Interaction. Erlbaum, Hillsdale, New 14. Mackinlay, J.D., Robertson, G.G. and ing technologies of 3D and interac- Jersey, 1983. Card, S.K. Perspective wall: Detail

~O April 1993/Vol.36, No.4 /COMMUNICATIONS OP~ AGM and context smoothly integrated. In Information visualization, information Authors' Present Address: Xerox PARC, Proceedings of SIGCHI'91, 1991, pp. visualizer, information access, informa- 3333 Coyote Hill Road, Palo Alto, CA 173-179. tion cost structure 94304; emaih {robertson, card, mac 15. Neweli, A. Unified Theories of Cogni- About the Authors: kinlay} @parc.xerox.com tion. Harvard University Press, Cam- GEORGE G. ROBERTSON isa principal bridge, Mass., 1990. scientist at Xerox Palo Alto Research Cen- Permission to copy without fee all or part of 16. Rao, R., Card, S.K., JeUinek, H.D., ter. this material is granted provided that the Mackinlay, J.D. and Robertson, G.G. copies are not made or distributed for direct The information grid: A framework STUART K. CARD is a research fellow commercial advantage, the ACM copyright for information retrieval and re- and manager of the user interface re- notice and the title of the publication and its search group at Xerox Palo Alto Research date appear, and notice is given that copying trieval-centered applications. In Pro- is by permission of the Association for Center. ceedings of UIST'92, 1992, pp. 23-32. Computing Machinery. To copy otherwise, or 17. Robertson, G.G., Card, S.K. and Mac- JOCK D. MACKINLAY is a member of to republish, requires a fee and/or specific kinlay, J.D. The Cognitive Coproces- permission. the research staff at Xerox Palo Alto Re- sor architecture for interactive user search Center. ©ACM0002-0782/93/0400-056 $1.50 interfaces. In Proceedings of UIST'89, 1989, pp. 10-18. 18. Robertson, G.G., Mackinlay, J.D. and Card, S.K. Cone Trees: Animated 3D visualizations of hierarchical infor- in-ter-op-er-a-biM-ty \,int-or- mation. In Proceedings of SIGCHI'91, 1991, pp. 189-194. 19. Sahon, G. and McGill, M.J. Introduc- ,fip-(a-)ra-'bil-at-e\ n. (1993) tion to modem information retrieval. McGraw-Hill, New York, 1983. 1. the existence of consistent 20. Shaw, C., Liang, J., Green, M. and Sun, Y. The decoupled simulation model for virtual reality systems. In processes and interfaces Proceedings of S1GCHI'92, 1992, pp. 321-328. 21. Sheridan, T.B. Supervisory control of across disparate computer remote manipulators, vehicles and dynamic processes: Experiments in command and display aiding. Ad- platforms. Extremely vances in Man-Machine System Research 1, 1984, JAI Press, 49-137. 22. Spence, R. and Apperley, M. Data difficult to achieve in base navigation: An office environment for the professional. Behavior Inf. Tech. 1, 1 (1982), 43-54. distributed applications. 23. Sutherland, I.E. Sketchpad: A man- machine graphical communication Until now. ADA system. MIT Lincoln Laboratory DEVELOPERS Tech. Rep. 296, May 1965. Abridged version in SJCC 1963, Spartan Books, Inlroducing Network Broker. programming task, your time is spent Baltimore, Md., 329. The environment for developing building your application. So there's less 24. Tesler, J. and Strasnick, S. FSN: 3D distributed applications. coding in the front door, and reduced Information Landscapes. Man pages Network Broker manages the complex- maintenance in the back door. entry for an unsupported but pub- ities of inter-communication among licly released system from Silicon heterogeneous or homogeneous First implemented in 1988, Network Graphics, Inc. Mountain View, Calif. networks. Complexities that are often Broker is a proven development tool Apr. 1992. unavoidable such as different vendor's designed specifically for use with the ADA 25, Zeltzer, D., Pieper, S. and Sturman, protocols, operating systems, and host progratnming language. Network Broker D. An integrated graphical simula- computers. By using an intelligent for C - language developers will be tion platform. In Proceedings of Graph- software environment-- the Virtual available later this year. ics Interface '89, 1989. Network -- Network Broker exploits the inherent concurrency in distributed To find out how Network Broker can CR Categories and Subject Descrip- systems, and insulates your working simplify the design, implementation, tors: H.3.3 [Information Storage and applications from their underlying integration, and maintenance of your Retrieval]: Information Search and Re- platforms. distributed applications, call our Software trieval-retrieval models; H.5.2 [Informa- Engineering Division directly, (410) 312-2259. tion Interfaces and Presentation]: User The result-- connectivity and Interfaces--interaction styles; 1.3.6 [Com- 7120 ColumNa Gateway Dr. puter Graphics]: Methodology and Tech- interoperability. Pure and simple. And that means less development time. ColumNa, MD 21046 niques-interaction techniques. (410)312-2000 Additional Key Words and Phrases: Since Network Broker handles the communications aspect of the FAX(410)312-2250

Circle #97 on Reader Service Card

COMMUNICATIONC OP TNC mCM/Apri11993/Vol.36,No.4 71