30706 02 pp011-026 r1jm.ps 5/6/04 3:49 PM Page 11 CHAPTER 2 3D3D UserUser Interfaces:Interfaces: HistoryHistory andand RoadmapRoadmap Three-dimensional UI design is not a traditional field of research with well-defined boundaries. Like human–computer interaction (HCI), it draws from many disciplines and has links to a wide variety of topics. In this chapter, we briefly describe the history of 3D UIs to set the stage for the rest of the book. We also present a 3D UI “roadmap” that posi- tions the topics covered in this book relative to associated areas. After reading this chapter, you should have an understanding of the origins of 3D UIs and its relation to other fields, and you should know what types of information to expect from the remainder of this book. 2.1. History of 3D UIs The graphical user interfaces (GUIs) used in today’s personal computers have an interesting history. Prior to 1980, almost all interaction with com- puters was based on typing complicated commands using a keyboard. The display was used almost exclusively for text, and when graphics were used, they were typically noninteractive. But around 1980, several technologies, such as the mouse, inexpensive raster graphics displays, and reasonably priced personal computer parts, were all mature enough to enable the first GUIs (such as the Xerox Star). With the advent of GUIs, UI design and HCI in general became a much more important research area, since the research affected everyone using computers. HCI is an 11 30706 02 pp011-026 r1jm.ps 5/6/04 3:49 PM Page 12 12 Chapter 2 3D User Interfaces: History and Roadmap 1 interdisciplinary field that draws from existing knowledge in perception, 2 cognition, linguistics, human factors, ethnography, graphic design, and 3 other areas. 4 In a similar way, the development of the 3D UI area has to a large de- 5 gree been driven by technologies, including 3D graphics technology, aug- 6 mented and virtual reality (VR) technology, and flight simulator 7 technology, to name a few. As each of these technologies matured, they en- 8 abled new types of applications, leading in turn to previously unexplored 9 user tasks, new challenges in UI design, and unforeseen usability issues. 0 Thus, 3D UI research became necessary. In the rest of this section, we 1 chronicle the development of one of these areas—virtual reality, to be pre- 2 cise—and show how advances in VR produced a need for 3D UI research. 3 In the late 1960s, Ivan Sutherland developed the first head-tracked, 4 head-mounted display (Sutherland 1968) and a vision for the use of this 5 new technology to enable a whole new type of computing (Sutherland 6 1965). Sutherland was ahead of his time, but finally in the late 1980s 7 and early 1990s, it became more practical to build VR systems. The tech- 8 nologies that enabled this vision included 3D stereoscopic computer 9 graphics, miniature CRT displays, position-tracking systems, and inter- 0 action devices such as the VPL DataGlove. Interestingly, when VR first 1 entered the public consciousness through Jim Foley’s article in Scientific 2 American (Foley 1987), the cover image showed not a display system or a 3 complex graphical environment, but rather the DataGlove—a “whole- 4 hand” input device allowing users to interact with and manipulate the 5 virtual world. 6 At first, VR was strictly the domain of computer scientists and engi- 7 neers (mostly in the graphics community) and was used for relatively 8 simple applications. Visualization of 3D scientific datasets, real-time 9 walkthroughs of architectural structures, and VR games were interesting 0 and useful applications, and provided plenty of research challenges (faster, 1 more realistic graphics; more accurate head-tracking; lower latency; bet- 2 ter VR software toolkits; etc.). These applications, however, were rela- 3 tively impoverished when it came to user interaction. The typical 4 application only allowed the user to interactively navigate the environ- 5 ment, with a few providing more complex interaction such as displaying 6 the name of an object when it was touched. As VR technology continued 7 to improve, however, researchers wanted to develop more complex ap- 8 plications with a much richer set of interactions. For example, besides al- 9 lowing an architect to experience his building design in a virtual world, 0 we could allow him to record and play audio annotations about the 30706 02 pp011-026 r1jm.ps 5/6/04 3:49 PM Page 13 2.1. History of 3D UIs 13 design, to change the type of stone used on the façade, to move a win- dow, or to hide the interior walls so that the pipes and ducts could be seen. At this point, some VR researchers realized that they didn’t know enough about interface design to make these complex applications us- able. Technology had improved to a point where such applications were possible, but more than technology was needed to make them plausible. Fortunately, because of the earlier focus on interfaces for personal computers, the field of HCI was already relatively mature when VR ex- perienced its “interface crisis.” HCI experts had developed general prin- ciples for good interface design (e.g., Nielsen and Molich 1992), product design and development processes aimed at ensuring usability (e.g., Hix and Hartson 1993), and models that explained how humans process in- formation when interacting with systems (e.g., Card et al. 1986). The application of existing HCI knowledge to VR systems helped to improve their usability. But there were some questions about VR inter- faces on which traditional HCI was silent. Consider the architectural de- sign example again. Suppose the architect wants to move a doorway and change the door’s paint color. An HCI expert would tell us that the move- ment task would best be accomplished by direct manipulation (like drag- ging an icon on your computer desktop), since that would provide direct and continuous feedback to the user. But how should a direct manipula- tion interface be implemented in VR? How does the architect select the door? Once the door is selected, how does he move it? How do the archi- tect’s hand and body motions map to the movement of the door? Again, the HCI expert might suggest that for the painting task, we need a simple menu selection mechanism, perhaps based on a real-world metaphor such as paint sample cards. But what does a menu look like in VR? How does the user activate the menu and select an item within it? Where does the menu appear in the 3D world? Questions such as these indicated that a new subfield of HCI was needed to address the issues specific to the design of interfaces for 3D VEs. In parallel, other technologies, such as interactive 3D graphics for desktop computers, augmented reality, and wearable computers, were also being developed, and researchers found that developing UIs on these platforms was also problematic. The common theme of all of these interactive technologies is interaction in a 3D context. Thus, the new sub- area of HCI is termed 3D interaction, 3D user interface design, or 3D HCI. In the next section, we look at the types of research problems ad- dressed and approaches used in 3D UI work and position them with re- spect to related work in other fields. 30706 02 pp011-026 r1jm.ps 5/6/04 3:49 PM Page 14 14 Chapter 2 3D User Interfaces: History and Roadmap 1 2.2. Roadmap to 3D UIs 2 To help you understand the material in this book in its proper context, it’s 3 important to discuss what types of work are part of the 3D UI area, what 4 disciplines and areas make up the background for 3D UI work, and what 5 impact 3D UIs have on other areas. In this section, therefore, we present 6 brief snapshots of a wide variety of areas with some connection to 3D 7 UIs. Figure 2.1 illustrates our basic organizational structure. In the fol- 8 lowing lists of topics, we provide at least one important reference for each 9 topic and, if applicable, a pointer to a particular chapter or section of the 0 book where that topic is covered. 1 2 3 Areas influencing 3D UIs Areas impacted 4 3D UIs by 3D UIs 5 3D interaction 3D UI 6 Theoretical techniques and evaluation Application and social interface • Evaluation of areas 7 background components devices • Simulation and • Interaction • Evaluation of 8 • Human spatial training techniques for interaction 9 perception, • Education universal tasks techniques cognition, and • Entertainment 0 • Interaction • Evaluation of action •Art techniques for complete 3D UIs or 1 • HCI and UI • Visualization complex or applications Design • Architecture and 2 composite tasks • Specialized • Popular media construction • 3D interaction evaluation 3 • Medicine and techniques using approaches psychiatry 4 2D devices • Studies of • Collaboration 5 • 3D UI widgets phenomena 6 particular to 3D UIs 7 8 Technological 3D UI design 3D UI software tools Standards background approaches 9 • Development tools • For interactive 3D 0 • Interactive 3D • Hybrid interaction for 3D applications graphics 1 graphics techniques • Specialized • For UI description • 3D visualization • Two-handed development tools 2 • 3D input devices interaction for 3D interfaces 3 • 3D display • Multimodel • 3D modeling tools devices interaction Reciprocal 4 • Simulator • 3D interaction aids impacts 5 systems • 3D UI design • Telepresence strategies • On graphics 6 systems • On HCI 7 • Virtual and mixed • On psychology reality systems 8 9 0 Figure 2.1 Roadmap to 3D UIs. 30706 02 pp011-026 r1jm.ps 5/6/04 3:49 PM Page 15 2.2.