Linköping Studies in Science and Technology Dissertation No. 1306 Augmentation in the Wild: User Centered Development and Evaluation of Augmented Reality Applications by Susanna Nilsson Department of Computer and Information Science Linköping University SE-581 83 Linköping, Sweden Linköping 2010 ( 1 ) c Susanna Nilsson 2010 ISBN 978-91-7393-416-9 ISSN 0345-7524 Typeset using LATEX Printed by LiU-Tryck, Linköping 2010 ii ( 2 ) The only way to discover the limits of the possible is to go beyond them into the impossible. Arthur C. Clarke ( 3 ) ( 4 ) Abstract Augmented Reality (AR) technology has, despite many applications in the research domain, not made it to a widespread end user market. The one exception is AR applications for mobile phones. One of the main reasons for this development is technological constraints of the non-mobile phone based systems - the devices used are still neither mobile nor lightweight enough or simply not usable enough. This thesis addresses the latter issue by taking a holistic approach to the development and evaluation of AR applications for both single user and multiple user tasks. The main hypothesis is that in order for substantial wide spread use of AR technology, the applications must be developed with the aim to solve real world problems with the end user and goal in focus. Augmented Reality systems are information systems that merge real and virtual information with the purpose of aiding users in different tasks. An AR system is a general system much like a computer is general; it has potential as a tool for many different purposes in many different situations. The studies in this thesis describe user studies of two different types of AR applications targeting different user groups and different application areas. The first application, described and evaluated, is aimed at giving users instructions for use and assembly of different medical devices. The second application is a study where AR technology has been used as a tool for supporting collaboration between the rescue services, the police and military personnel in a crisis management scenario. Both applications were iteratively developed with end user representatives involved throughout the process and the results illustrate that users both in the context of medical care, and the emergency management domain, are positive towards AR systems as a technology and as a tool in their work related tasks. The main contributions of the thesis are not only the end results of the user studies, but also the methodology used in the studies of this relatively new type of technology. The studies have shown that involving real end users both in the design of the application and in the user task is important for the user experience of the system. Allowing for an iterative design process is also a key point. Although AR technology development is often driven by technological advances rather than user demands, there is room for a more user centered approach, for single user applications as well as for more dynamic and complex multiple user applications. ( 5 ) ( 6 ) Acknowledgements First of all I need to clarify that the animal on the cover of this thesis is not a tame dog, but a wild dingo on the beach of Frasier Island. The choice of title is not about the wild dingo however, it is of course a reference to the original piece "Cognition in the Wild" by Edwin Hutchins. I do not in any way intend to claim that this thesis has, or will have, the same impact in academia as the original work has had. However, I do hope that the title will encourage others to bring Augmented Reality applications into the wild. By this I do not necessarily mean bringing Augmented Reality to the dingos, but to find real world problems that can be solved with the help of a little creativity and Augmented Reality technology. The ideas in this thesis formed during my undergraduate studies and has since continuously been shaped by the people who have been involved in my education. Torbjörn Gustafsson and Per Carleberg were the people who introduced me to the field of Augmented Reality by asking if I would like to be part of augmenting the visual world like the terminator’s visual system in Terminator 2 (and of course I wanted to). Erik Hollnagel was my first supervisor and he, in collaboration with my ever present co-supervisor (and co-author) Björn Johansson helped me figure out the ways of cognitive systems engineering and systemic approaches to end user applications. After Erik left IDA, Arne Jönsson took over as my main supervisor and he has since then been an excellent facilitator of my work and always supportive of my ideas (and desire to travel). I am also grateful for the support I have received from Christina Aldrin, Jenny Gustafsson and Betty Tärning at FMV. The financial support from FMV has allowed me not only to pursue my research interests but also to work in a place full of creative and supportive people. So thank you for many entertaining and inspiring fikas and lunches: Fabian, Jiri, Maria, Jody, Sara, Amy, Johan, Rogier and all the rest of HCS. And a special thank you to Arne Jönsson and Ola Leifler for all the last minute LaTex support and Dennis Netzell at LiU-tryck for being so enthusiastic about my cover design ideas. Needless to say there are more people involved in my life as a PhD student and I would especially like to thank Dana, Kattis and Felix for pushing me over the finish line (once again), and Martin for being you. It’s also difficult to imagine getting to this point if it wasn’t for my family - thank you mamma Anne, Fia vii ( 7 ) and Leo (som fortfarande är min spindelman), Staffan (whose help with SPSS and general methodological insights has been invaluable), Anna, Stina and Arvid. And without my friends∗ my life would surely have been far less entertaining. Finally - without the help of all willing participants there would not have been any end user studies at all, so thank you Margaretha Fredriksson and Carina Zetterberg at Örebro University hospital, Håkan Evertsson at the police department in Linköping, Christer Carlsson at the fire and rescue services in Linköping and Peter Hagsköld at Malmen, Linköping. Not only for your dedication to your professions, but for all your support and help in finding participants and of course helping me discover and understand the real world problems that we could use Augmented Reality to solve. ∗see end titles ( 8 ) Contents Abstract v Acknowledgements vii List of Figures xiii List of Tables xv 1 Introduction 1 1.1 Mixed Reality and Augmented Reality . 2 1.2 Evaluating Human Computer Interaction . 4 1.3 Research Question . 5 1.4 Research Contributions . 6 1.5 A note on the terminology . 7 1.6 Reading Instructions . 8 1.7 List of publications . 9 2 Framing the thesis 13 2.1 Augmenting reality . 13 2.1.1 Display solutions for merging visual realities . 14 2.1.1.1 Optic see-through augmented reality . 15 2.1.1.2 Video see-through augmented reality . 17 2.1.1.3 Visually based marker tracking . 18 2.1.2 Interacting with virtual information . 19 2.1.3 Augmented Reality projects and applications . 21 2.1.4 Single and multi-user applications . 24 2.2 Shared views and common ground for collaboration . 27 2.3 User centered development of systems . 30 2.3.1 Human Computer Interaction in theory . 31 2.3.2 Usability engineering and evaluations . 32 2.3.3 Problems of human computer interaction and usability engineering approaches . 35 2.4 Systemic approaches to systems development . 36 2.4.1 Activity theory . 38 ix ( 9 ) x 2.4.2 Situated cognition and distributed cognition . 39 2.4.3 Cognitive systems engineering and the use of artefacts . 40 2.5 Evaluating Augmented Reality systems . 44 2.5.1 Viewing Mixed Reality systems as Joint Cognitive Systems . 46 2.5.2 Choosing a perspective . 48 3 Research approach and methods 51 3.1 Iterative design process . 53 3.2 End user studies and evaluations . 56 3.2.1 Evaluation of systems and products . 56 3.2.2 Simulation and scenario for the collaborative Augmented Reality application . 58 3.3 Data collection methods used in the studies . 59 3.3.1 Questionnaires . 60 3.3.2 Interviews . 61 3.3.3 Observations . 62 3.4 Analysis of data in the studies . 64 3.5 Instruments used in the studies . 67 4 Single user studies 71 4.1 Iterative design of the application tasks . 72 4.2 Study 1 - the electro-surgical generator . 73 4.2.1 Methodology . 73 4.2.2 Participants . 74 4.2.3 Equipment . 74 4.2.4 The user task . 75 4.2.5 Data analysis . 76 4.2.6 Results . 76 4.2.7 Discussion of the results . 78 4.3 Study 2 - the troakar . 79 4.3.1 Methodology . 80 4.3.2 Participants . 80 4.3.3 Equipment . 81 4.3.4 User task . 82 4.3.5 Data analysis . 84 4.3.6 Results . 85 4.3.7 Discussion of the results . 90 4.4 Study 3 - the second troakar study . 90 4.4.1 Methodology . 91 4.4.2 Participants . 92 4.4.3 Equipment . 92 4.4.4 The end user task . 93 4.4.5 Data analysis . 93 4.4.6 Results . 94 ( 10 ) xi 4.4.7 Discussion . 100 4.5 General observations and conclusions of the single user studies . 103 5 Multi-user study 107 5.1 The design and development process .