30706 01 pp001-010 r1jm.ps 5/6/04 3:48 PM Page 1

PART I FoundationsFoundations ofof 3D3D UserUser InterfacesInterfaces

Part I introduces you to the topic of 3D user interfaces (UIs). Chapter 1 ex- plains what 3D UIs are and why they are important. It also introduces some key terminology used throughout the book and describes some ap- plications that use 3D UIs. Chapter 2 provides a brief history of 3D UIs and a “roadmap” of related research areas, positioning the topics covered in this book within a larger context.

1 30706 01 pp001-010 r1jm.ps 5/6/04 3:48 PM Page 2 30706 01 pp001-010 r1jm.ps 5/6/04 3:48 PM Page 3

CHAPTER 1 IntrIntroductionoduction toto 3D3D UserUser InterfacesInterfaces

On desktop , good (UI) design is now almost universally recognized as a crucial part of the software and hardware de- velopment process. Almost every computing-related product touts itself as “easy to use,” “intuitive,” or “designed with your needs in mind.” For the most part, however, desktop user interfaces have used the same basic principles and designs for the past decade or more. With the advent of virtual environments (VEs), augmented reality, ubiquitous computing, and other “off-the-desktop” technologies, three-dimensional (3D) UI de- sign is now becoming a critical area for developers, students, and re- searchers to understand. In this chapter, we answer the question, What are 3D user interfaces? and provide an introduction to terminology that is used throughout the book. We describe the goals of 3D UI design and briefly look at some application areas for 3D user interfaces. Keeping these applications in mind as you progress through the book will help provide a concrete reference point for some of the more abstract concepts we discuss.

1.1. What Are 3D User Interfaces? Modern users have become intimately familiar with a specific set of UI components, including input devices such as the mouse and keyboard, output devices such as the monitor, interaction techniques

3 30706 01 pp001-010 r1jm.ps 5/6/04 3:48 PM Page 4

4 Chapter 1 Introduction to 3D User Interfaces

1 such as drag-and-drop, interface widgets such as pull-down menus, and 2 interface metaphors such as the . 3 These interface components, however, are often inappropriate for the 4 nontraditional computing environments and applications under devel- 5 opment today. For example, a wearable-computer user may be walking 6 down the street, making the use of a keyboard impractical. A head- 7 mounted display in an augmented reality application may have limited 8 resolution, forcing the redesign of text-intensive interface components 9 such as dialog boxes. A virtual reality application may allow a user to 0 place an object anywhere in 3D space, with any orientation—a task for 1 which a 2D mouse is inadequate. 2 Thus, these nontraditional systems need a new set of interface com- 3 ponents: new devices, new techniques, new metaphors. Some of these 4 new components may be simple refinements of existing components; 5 others must be designed from scratch. Most of these nontraditional envi- 6 ronments work in real or virtual 3D space, so we term these new inter- 7 faces 3D user interfaces (a more precise definition is given in section 1.3). 8 In this book, we describe and analyze the components (devices, tech- 9 niques, metaphors) that can be used to design 3D user interfaces. We also 0 provide guidance in choosing the components for your particular system 1 based on empirical evidence from published research, anecdotal evi- 2 dence from colleagues, and personal experience. 3 4 5 1.2. Why 3D User Interfaces? 6 7 Why is the information in this book important? We have five main moti- 8 vations for producing this book: 9 0 3D interaction is relevant to real-world tasks: Interacting in three dimen- 1 sions makes intuitive sense for a wide range of applications (see section 2 1.4) because of the characteristics of the tasks in these domains and their 3 match with the characteristics of 3D environments. For example, VEs can 4 provide users with a sense of presence (the feeling of “being there”—re- 5 placing the physical environment with the virtual one), which makes 6 sense for applications such as gaming, training, and simulation. If a user 7 is immersed and can interact using natural skills, then the application can 8 take advantage of the fact that the user already has a great deal of knowl- 9 edge about the world. Also, 3D UIs may be more direct or immediate; 0 that is, there is a short “cognitive distance” between a user’s action and 30706 01 pp001-010 r1jm.ps 5/6/04 3:48 PM Page 5

1.2. Why 3D User Interfaces? 5

the system’s feedback that shows the result of that action. This can allow users to build up complex mental models of how a simulation works, for example. The technology behind 3D UIs is becoming mature: User interfaces for computer applications are becoming more diverse. Mice, keyboards, windows, menus, and icons—the standard parts of traditional WIMP (Windows, Icons, Mouse, and Pointers) interfaces—are still prevalent, but nontraditional devices and interface components are proliferating rapidly. These include spatial input devices such as trackers, 3D pointing devices, and whole-hand devices that allow gesture-based input. Multi- sensory 3D output technologies, such as stereoscopic projection displays, head-mounted displays (HMDs), spatial audio systems, and haptic de- vices are also becoming more common. 3D interaction is difficult: With this new technology, new problems have also been revealed. People often find it inherently difficult to understand 3D spaces and to perform actions in free space (Herndon et al., 1994). Al- though we live and act in a 3D world, the physical world contains many more cues for understanding and constraints and affordances for action that cannot currently be represented accurately in a computer simulation. Therefore, great care must go into the design of user interfaces and inter- action techniques for 3D applications. It is clear that simply adapting tra- ditional WIMP interaction styles to 3D does not provide a complete solution to this problem. Rather, novel 3D UIs based on real-world inter- action or some other metaphor must be developed. Current 3D UIs are either simple or lack : There are already some applications of 3D user interfaces used by real people in the real world (e.g., walkthroughs, psychiatric treatment, entertainment, and training). Most of these applications, however, contain 3D interaction that is not very complex. More complex 3D interfaces (e.g., immersive design, edu- cation, complex scientific visualizations) are difficult to design and evalu- ate, leading to a lack of usability. Better technology is not the only answer—for example, 30 years of VE technology research have not en- sured that today’s VEs are usable. Thus, a more thorough treatment of this subject is needed. 3D UI design is an area ripe for further work: Finally, development of 3D user interfaces is one of the most exciting areas of research in human– computer interaction (HCI) today, providing the next frontier of innova- tion in the field. A wealth of basic and applied research opportunities are available for those with a solid background in 3D interaction. 30706 01 pp001-010 r1jm.ps 5/6/04 3:48 PM Page 6

6 Chapter 1 Introduction to 3D User Interfaces

1 It is crucial, then, for anyone involved in the design, implementation, or 2 evaluation of nontraditional interactive systems to understand the issues 3 discussed in this book. 4 5 1.3. Terminology 6 7 The technology sector loves acronyms and jargon, and precise terminology 8 can make life easier as long as everyone is in agreement about the meaning 9 of a particular term. This book is meant to be accessible to a broad audi- 0 ence, but we still find it useful to employ precise language. Here we pre- 1 sent a glossary of some terms that we use throughout the book. 2 We begin with a set of general terms from the field of HCI that are 3 used in later definitions: 4 human–computer interaction (HCI) The process of communication be- 5 tween human users and computers (or technologies in general). Users 6 communicate actions, intents, goals, queries, and other such needs to com- 7 puters. Computers, in turn, communicate information about the world, 8 about their internal state, about the responses to user queries, and so on 9 to the user. This communication may involve explicit dialog, or turn-tak- 0 ing, in which a user issues a command or query, the system responds, 1 and so on, but in most modern computer systems, the communication is 2 more implicit, freeform, or even imperceptible (Hix and Hartson, 1993). 3 4 user interface (UI): The medium through which the communication be- 5 tween users and computers takes place. The UI translates a user’s actions 6 and state (inputs) into a representation the computer can understand and 7 act upon, and it translates the computer’s actions and state (outputs) into 8 a representation the human user can understand and act upon (Hix and 9 Hartson, 1993). 0 input device A physical (hardware) device allowing communication from 1 the user to the computer. 2 degrees of freedom (DOF) The number of independent dimensions of the 3 motion of a body. DOF can be used to describe the movements of input 4 devices, the motion of a complex articulated object such as a human arm 5 and hand, or the possible movements of a virtual object. 6 output device 7 A physical device allowing communication from the com- 8 puter to the user. 9 interaction technique A method allowing a user to accomplish a task via 0 the UI. An interaction technique includes both hardware (input device) 30706 01 pp001-010 r1jm.ps 5/6/04 3:48 PM Page 7

1.3. Terminology 7

and software components. The interaction technique’s software compo- nent is responsible for mapping the information from the input device (or devices) into some action within the system. usability The characteristics of an artifact (usually a device, interaction technique, or complete UI) that affect the user’s use of the artifact. There are many aspects of usability, including ease of use, user task perfor- mance, user comfort, and system performance (Hix and Hartson 1993). usability evaluation The process of assessing or measuring some aspects of the usability of a particular artifact.

Using this HCI terminology, we define 3D interaction and 3D user interface:

3D interaction Human–computer interaction in which the user’s tasks are performed directly in a 3D spatial context. Interactive systems that display 3D graphics do not necessarily involve 3D interaction; for ex- ample, if a user tours a model of a building on her desktop computer by choosing viewpoints from a traditional , no 3D interaction has taken place. On the other hand, 3D interaction does not necessarily mean that 3D input devices are used; for example, in the same application, if the user clicks on a target object to navigate to that object, then the 2D mouse input has been directly translated into a 3D location, and thus 3D interaction has occurred. 3D user interface (3D UI): A UI that involves 3D interaction.

Finally, we define some technological areas in which 3D UIs are used:

virtual environment (VE): A synthetic, spatial (usually 3D) world seen from a first-person point of view. The view in a VE is under the real-time control of the user. virtual reality (VR) Synonymous with VE. We use the term VE in this book, since the term VR is associated with unrealistic hype generated by the media. augmented reality (AR) A real-world environment that is enhanced (aug- mented) with synthetic objects or information. mixed reality (MR) A continuum including both VEs and AR. An environ- ment’s position on the continuum indicates the level of “virtuality” in the environment (with the extremes being “purely virtual” and “purely 30706 01 pp001-010 r1jm.ps 5/6/04 3:48 PM Page 8

8 Chapter 1 Introduction to 3D User Interfaces

1 physical”). Mixed reality systems may move along this continuum as the 2 user interacts with them (Milgram and Kishino 1994). 3 ubiquitous computing (UbiComp) The notion that computing devices 4 and infrastructure (and access to them) may be mobile or scattered 5 throughout the real environment so that users have “anytime, anyplace” 6 access to computational power (Weiser 1991). 7 8 9 1.4. Application Areas 0 1 Before we discuss the components of 3D UIs, it will be helpful for you to 2 have an idea of the types of applications that employ 3D interaction. 3 Keep these application examples in mind as you read through the vari- 4 ous sections of the book (a more complete list of application areas can be 5 found in Chapter 2). Although many of the examples we discuss in the 6 book come from VEs, the principles we discuss can be applied in a wide 7 variety of applications. Here are some examples: 8 9 • Design and prototyping: A 3D UI can be used to allow designers 0 of real-world artifacts to work directly in a realistic 3D context. 1 For example, an architect can navigate through a proposed new 2 building and make changes to its design directly rather than 3 working in the traditional 2D medium of drawings and plans 4 (Bowman, Wineman, et al., 1998). The scenario in the preface 5 illustrates the types of design problems and tasks a 3D UI might 6 help to address. 7 • Psychiatric treatment: People with strong fears of certain objects 8 or situations are often treated with “exposure therapy” in which 9 they are gradually presented with increasing levels of the fear- 0 inducing stimulus. VEs can be used for “virtual exposure ther- 1 apy” (Hodges, et al., 1995), where the stimulus is synthetic rather 2 than real. This can be less expensive, less embarrassing, and less 3 dangerous. A 3D UI can be used to allow the patient to interact 4 with the environment. For example, someone with a fear of 5 snakes might be able to pick up and handle a virtual snake with a 6 combination of 3D input devices and a realistic toy snake. 7 • Scientific visualization: Scientists and engineers run experiments 8 and simulations that produce huge amounts of data. This data 9 can be visualized using 3D graphics, providing understanding 0 30706 01 pp001-010 r1jm.ps 5/6/04 3:48 PM Page 9

1.5. Conclusion 9

and insight that could not be obtained from looking at numeric results. With 3D UI components, the user can interactively navi- gate through the data, query various points in the visualization, or even steer the simulation computation (Bryson 1996). • Heritage/tourism: Visiting historical sites can often be disappoint- ing. Buildings have crumbled, cities have grown up around the site, and information is difficult to obtain. Augmented reality technology can address some of these issues by allowing a visi- tor to see directly what the site might have looked like in earlier times. The combination of real-world images and synthetic images seen from a first-person point of view can be quite com- pelling. For example, 3D UIs can be used to set the time period the user wants to view or to navigate through text, audio, or image information related to the site (Gleue and Dahne 2001). • Collaborative work: More and more of our work is done in groups or teams, and often these groups are geographically scattered rather than located in a single office. This situation has led to the rise of a whole new software industry focused on collaborative applications, including videoconferencing, online presentations and classes, and collaborative document editing. There are a number of ways 3D UIs can be used for collaborative work (Prince et al. 2002). For example, a virtual meeting can be held in a 3D environment, providing more of the spatial and visual richness of a face-to-face meeting, or collaborators can enter a 3D environment to work together on the design of a new car.

1.5. Conclusion In this chapter, we discussed briefly the area of 3D UI design—its impor- tance, terminology, and applications. In Chapter 2, we step back to look at the bigger picture—the history of and context for 3D UIs. 30706 01 pp001-010 r1jm.ps 5/6/04 3:48 PM Page 10