User Interfaces for Mobile Augmented Reality Systems Tobias Hans Hollerer¨ Submitted in partial fulfillment of the requirements for the degree of Doctor of Philosophy in the Graduate School of Arts and Sciences COLUMBIA UNIVERSITY 2004 ©2004 Tobias Hans H¨ollerer All Rights Reserved Abstract User Interfaces for Mobile Augmented Reality Systems Tobias Hans Hollerer¨ In this dissertation, we present typical components of, useful services associated with, and user interactions possible with mobile augmented reality systems, based on a comprehensive series of hardware and software infrastructures and application proto- types we developed. We define a practical taxonomy of user interface components for such systems and establish methodology for adaptive mobile augmented reality inter- faces that dynamically rearrange themselves in response to changes in user context. The research contributions to the state-of-the-art in augmented reality begin with the author’s participation in the design of the ”Columbia Touring Machine” in 1997, the first example of an outdoor mobile augmented reality system, and his lead in developing later prototypes. We develop a series of hardware and software infrastructures for proto- typing mobile augmented reality applications that allow multiple users to participate in collaborative tasks taking place indoors and outdoors. We present exploratory user interfaces for many different applications and user scenarios, including the Situated Documentaries application framework for experienc- ing spatially distributed hypermedia presentations. Based on these explorations, we de- velop a taxonomic categorization of mobile augmented reality interface components and their properties. Virtual and real world objects alike are considered part of the interface. We tag each component with information about its purpose, its intrinsic properties, its relationship to other objects, and its capabilities and flexibility with regard to various manipulations. Mobile augmented reality has until now faced a significant challenge: the com- plexity of the augmented views rapidly increases when many virtual objects fight for screen space to annotate physical entities in the dynamic views of multiple fast-paced roaming users. Responding to this, we develop user interface management techniques for mobile augmented reality. A rule-based reasoning architecture uses the taxonomic data classification mentioned above to automatically rearrange augmented reality views in dynamic situations; for example to remove clutter in the augmented view of the world or to react to infrastructural context changes, such as variations in tracking accuracy. Contents List of Figures v List of Tables xi Acknowledgments xiii Chapter 1 Introduction 1 1.1 Mobile Augmented Reality........................ 1 1.2 Thesis Overview . ............................ 3 1.2.1 Properties of MARS Interfaces . .............. 4 1.2.2 MARS UI Management ..................... 5 1.2.3 Scope of this Research . ..................... 6 1.2.4 Summary of Contributions . .............. 8 1.2.4.1 System Designs for Outdoor MARS . 9 1.2.4.2 Situated Documentaries and Other Application Proto- types . ..................... 10 1.2.4.3 Taxonomy of MARS UI Components . 11 1.2.4.4 MARS UI Management . .............. 11 1.2.4.5 Rule-based AR System . .............. 11 Chapter 2 Background and Related Work 13 2.1 Components of Mobile AR Systems . .............. 13 2.2 Historical Overview ............................ 14 2.3 Mobile AR: Applications, and Challenges . .............. 16 2.3.1 Applications . ..................... 16 2.3.1.1 Assembly and Construction .............. 16 2.3.1.2 Maintenance and Inspection .............. 18 2.3.1.3 Navigation and Path Finding . .......... 18 2.3.1.4 Tourism . ..................... 18 2.3.1.5 Architecture and Archaeology . .......... 19 i 2.3.1.6 Urban Modeling . .............. 20 2.3.1.7 Geographical Field Work . .............. 20 2.3.1.8 Journalism . ..................... 21 2.3.1.9 Entertainment ..................... 22 2.3.1.10 Medicine . ..................... 22 2.3.1.11 Military Training and Combat . .......... 23 2.3.1.12 Personal Information Management and Marketing . 24 2.3.2 Challenges ............................ 24 2.4 Mobile Augmented Reality Systems . .............. 26 2.4.1 Mobile Computing Platforms . .............. 26 2.4.2 Displays for Mobile AR ..................... 28 2.4.3 Tracking and Registration . .............. 33 2.4.4 Environmental Modeling ..................... 38 2.4.5 Wearable Input and Interaction Technologies .......... 40 2.4.6 Wireless Communication and Data Storage Technologies . 44 2.5 Existing Mobile AR Systems . ..................... 45 Chapter 3 System Designs 48 3.1 System Architectures, Hardware ..................... 49 3.2 System Architectures, Software . ..................... 51 3.2.1 COTERIE-based MARS ..................... 52 3.2.1.1 Campus Tour Architecture . .............. 53 3.2.1.2 Situated Documentaries Extensions .......... 54 3.2.2 IN-N-OUT AR – an Indoor/Outdoor Java/COTERIE Hybrid In- frastructure ............................ 56 3.2.2.1 Development Tools . .............. 58 3.2.3 JABAR—AJava-Based AR Infrastructure . .......... 59 3.2.4 CGUI Lab AR — A Central Architecture for AR and View Man- agement . ............................ 61 3.2.5 Ruby — Rule-based Control of AR interfaces .......... 61 3.3 Indoor Tracking for MARS . ..................... 62 3.3.1 Wide-Area Indoor Tracking using Dead-Reckoning . 63 Chapter 4 Analytical Foundation of MARS UIs 68 4.1 Scope of MARS User Interfaces ..................... 68 4.2 Taxonomy of MARS UI components . .............. 72 4.2.1 MARS Objects . ..................... 73 4.2.1.1 Environment Objects . .............. 74 4.2.1.2 UI Templates ..................... 75 ii 4.2.1.3 Container Objects . .............. 76 4.2.1.4 Special Objects and Examples . .......... 76 4.2.2 Object Attributes . ..................... 79 4.2.2.1 Type Attributes . .............. 79 4.2.2.2 State Attributes . .............. 81 4.2.2.3 Realization Attributes . .............. 83 4.2.2.4 Semantic Attributes . .............. 85 4.2.3 MARS Concepts . ..................... 86 4.2.3.1 Output Media ..................... 86 4.2.3.2 Reference Frames . .............. 88 4.2.3.3 Relationships ..................... 89 4.2.3.4 Goals . ..................... 90 4.2.3.5 Purposes . ..................... 90 4.2.3.6 Events . ..................... 91 4.2.4 MARS Interaction . ..................... 91 4.2.4.1 Interaction Tasks . .............. 92 4.2.4.2 Input Devices ..................... 93 4.2.4.3 Output Devices . .............. 95 Chapter 5 Implemented MARS UIs 97 5.1 Spatially Distributed Hypermedia . .............. 99 5.1.1 Situated Documentaries I & II . .............. 101 5.1.1.1 Navigating the Web of Presentations . 104 5.1.1.2 Multimedia Presentations . .............. 106 5.1.1.3 Exploratory UI Design . .............. 109 5.1.2 Situated Documentary III: Interactive Storytelling . 109 5.1.3 Summary of UI Concepts ..................... 113 5.2 Collaboration . ............................ 114 5.2.1 Indoor/Outdoor Collaboration . .............. 114 5.2.2 Hybrid UIs for Augmented Meetings .............. 118 5.2.3 Integrated Web Browser and Communication .......... 119 5.2.4 Summary of UI Concepts ..................... 120 5.3 Navigation . ............................ 121 5.3.1 Indoor Navigation UI . ..................... 121 5.3.2 Situational Awareness . ..................... 124 5.3.2.1 Intuitive Control of an AR World in Miniature . 124 5.3.2.2 Restaurant Guide . .............. 126 5.3.3 Summary of UI Concepts ..................... 129 5.4 MARS UI Design, Problems and Solutions . .............. 130 iii 5.4.1 MARS UI Design: Lessons Learned . .............. 132 5.4.2 Analysis of UI Alternatives . .............. 134 5.4.3 Adaptation ............................ 136 5.4.3.1 UI Management Pipeline . .............. 137 5.4.3.2 View Management for AR . .............. 139 Chapter 6 Rule-based Architecture for MARS UI Management 143 6.1 Ruby: a Rule-Based System Architecture . .............. 144 6.1.1 Ruby Architecture . ..................... 146 6.1.2 Ruby and Jess . ..................... 149 6.1.3 Ruby Data Formats . ..................... 152 6.1.3.1 Ruby Objects ..................... 153 6.1.3.2 Internal Attributes . .............. 155 6.1.3.3 External Attributes, Relationships, Events, and Goals 159 6.1.4 Ruby Rules ............................ 162 6.1.5 Ruby Control Flow . ..................... 164 6.2 Examples . ............................ 166 6.2.1 Exploring Display Modes for Occluded Infrastructure . 166 6.2.2 Adapting to Mobile User Context . .............. 168 6.2.3 Adapting to Tracking Accuracy . .............. 170 6.3 Discussion . ............................ 176 Chapter 7 Conclusions and Future Work 180 7.1 Summary of Results ............................ 182 7.2 Future Work . ............................ 183 7.2.1 Environment Management . .............. 183 7.2.2 Scalable Real-Time Knowledge Processing . .......... 184 7.2.3 Support for Hierarchical UI Descriptions . .......... 184 7.2.4 Usability Evaluation . ..................... 185 7.2.5 Adaptivity in Non-AR UIs . .............. 186 7.2.6 Additional Interaction Modalities . .............. 186 References 187 Appendix A Details of Hardware Prototypes 1996–2003 207 A.1 Columbia Touring Machine . ..................... 207 A.2 MARS 1999 . ............................ 209 A.3 MARS 2000 . ............................ 211 A.4 MARS 2001/2002 . ...........................