Augmented Reality Interfaces for Procedural Tasks

Augmented Reality Interfaces for Procedural Tasks Steven J. Henderson Submitted in partial fulfillment of the requirements for the degree of Doctor of Philosophy in the Graduate School of Arts and Sciences COLUMBIA UNIVERSITY 2011 © 2011 Steven J. Henderson All Rights Reserved ABSTRACT Augmented Reality Interfaces for Procedural Tasks Steven J. Henderson Procedural tasks involve people performing established sequences of activities while in- teracting with objects in the physical environment to accomplish particular goals. These tasks span almost all aspects of human life and vary greatly in their complexity. For some simple tasks, little cognitive assistance is required beyond an initial learning session in which a person follows one-time compact directions, or even intuition, to master a sequence of activities. In the case of complex tasks, procedural assistance may be continually required, even for the most ex- perienced users. Approaches for rendering this assistance employ a wide range of written, audi- ble, and computer-based technologies. This dissertation explores an approach in which procedural task assistance is rendered using augmented reality. Augmented reality integrates virtual content with a user’s natural view of the environment, combining real and virtual objects interactively, and aligning them with each other. Our thesis is that an augmented reality interface can allow individuals to perform procedural tasks more quickly while exerting less effort and making fewer errors than other forms of assistance. This thesis is supported by several significant contributions yielded during the exploration of the following research themes: What aspects of AR are applicable and beneficial to the procedural task problem? In an- swering this question, we developed two prototype AR interfaces that improve procedural task accomplishment. The first prototype was designed to assist mechanics carrying out maintenance procedures under field conditions. An evaluation involving professional mechanics showed our prototype reduced the time required to locate procedural tasks and resulted in fewer head move- ments while transitioning between tasks. Following up on this work, we constructed another prototype that focuses on providing assistance in the underexplored psychomotor phases of procedural tasks. This prototype presents dynamic and prescriptive forms of instruction and was eval- uated using a demanding and realistic alignment task. This evaluation revealed that the AR prototype allowed participants to complete the alignment more quickly and accurately than when using an enhanced version of currently employed documentation systems. How does the user interact with an AR application assisting with procedural tasks? The application of AR to the procedural task problem poses unique user interaction challenges. To meet these challenges, we present and evaluate a novel class of user interfaces that leverage natu- rally occurring and otherwise unused affordances in the native environment to provide a tangible user interface for augmented reality applications. This class of techniques, which we call Oppor- tunistic Controls, combines hand gestures, overlaid virtual widgets, and passive haptics to form an interface that was proven effective and intuitive during quantitative evaluation. Our evaluation of these techniques includes a qualitative exploration of various preferences and heuristics for Opportunistic Control-based designs. Table of Contents List of Figures .................................................................................................................. iii List of Tables .................................................................................................................... xi Acknowledgements ........................................................................................................ xii 1 Introduction ............................................................................................................... 1 1.1 Research questions and dissertation goals ................................................................ 5 1.2 Contributions............................................................................................................. 7 1.3 Structure of Dissertation ......................................................................................... 25 2 Related Work .......................................................................................................... 27 2.1 The Nature of Procedural Tasks ............................................................................. 27 2.2 Teaching, Learning and Assistance for Procedural Tasks ...................................... 29 2.3 Designing Assistance for Procedural Tasks ............................................................ 30 2.4 Non-automated Forms of Assistance ...................................................................... 32 2.5 Computerized Forms of Assistance ........................................................................ 37 2.6 AR Interfaces .......................................................................................................... 42 3 Leveraging Augmented Reality in Informational Phases of Procedural Tasks 45 3.1 Related Work .......................................................................................................... 47 3.2 Prototype ................................................................................................................. 50 3.3 User Study Design .................................................................................................. 64 3.4 Quantitative Results ................................................................................................ 74 3.5 Qualitative Results .................................................................................................. 93 3.6 Discussion ............................................................................................................... 98 4 Leveraging Augmented Reality in Psychomotor Phases of Procedural Tasks 101 4.1 Related Work ........................................................................................................ 105 4.2 Selecting a Psychomotor Task Environment ........................................................ 107 4.3 Psychomotor Phase Assistance ............................................................................. 115 4.4 User Study ............................................................................................................. 119 4.5 Quantitative Results .............................................................................................. 134 4.6 Qualitative Results ................................................................................................ 140 4.7 Comparison to Physical Labels ............................................................................. 142 4.8 Discussion ............................................................................................................. 147 5 Opportunistic Controls ......................................................................................... 148 5.1 Related Work ........................................................................................................ 151 5.2 Alternative User Interfaces ................................................................................... 152 5.3 Opportunistic Controls Definition ........................................................................ 154 5.4 Designing Opportunistic Control Interfaces ......................................................... 155 5.5 OC User Observation Study.................................................................................. 163 5.6 Prototype Implementation ..................................................................................... 180 5.7 OC Interface Technique User Study ..................................................................... 185 5.8 Discussion ............................................................................................................. 194 6 Conclusions and Future Work ............................................................................. 197 6.1 Summary of Contributions .................................................................................... 197 6.2 Lessons Learned.................................................................................................... 199 6.3 Future Work .......................................................................................................... 201 6.4 Final Thoughts ...................................................................................................... 209 Appendix A The ARMAR Architecture ............................................................... 210 A.1 Architecture Requirements ................................................................................... 211 A.2 Related Work ........................................................................................................ 212 A.3 ARMAR Software Architecture ........................................................................... 214 A.4 ARMAR Hardware Architecture .......................................................................... 224 A.5 Evolution of the ARMAR Architecture ................................................................ 225 A.6 Value Source-based Version of ARMAR ............................................................. 225 A.7 Goblin XNA-based Version of ARMAR .............................................................

Load more