Augmentation in the Wild: User Centered Development and Evaluation of Augmented Reality Applications

Linköping Studies in Science and Technology Dissertation No. 1306

by Susanna Nilsson

Department of Computer and Information Science Linköping University SE-581 83 Linköping, Sweden

Linköping 2010

( 1 ) c Susanna Nilsson 2010 ISBN 978-91-7393-416-9 ISSN 0345-7524

Typeset using LATEX

Printed by LiU-Tryck, Linköping 2010

( 2 ) The only way to discover the limits of the possible is to go beyond them into the impossible.

Arthur C. Clarke

( 3 ) ( 4 ) Abstract

Augmented Reality (AR) technology has, despite many applications in the research domain, not made it to a widespread end user market. The one exception is AR applications for mobile phones. One of the main reasons for this development is technological constraints of the non-mobile phone based systems - the devices used are still neither mobile nor lightweight enough or simply not usable enough. This thesis addresses the latter issue by taking a holistic approach to the development and evaluation of AR applications for both single user and multiple user tasks. The main hypothesis is that in order for substantial wide spread use of AR technology, the applications must be developed with the aim to solve real world problems with the end user and goal in focus.

Augmented Reality systems are information systems that merge real and virtual information with the purpose of aiding users in different tasks. An AR system is a general system much like a computer is general; it has potential as a tool for many different purposes in many different situations. The studies in this thesis describe user studies of two different types of AR applications targeting different user groups and different application areas. The first application, described and evaluated, is aimed at giving users instructions for use and assembly of different medical devices. The second application is a study where AR technology has been used as a tool for supporting collaboration between the rescue services, the police and military personnel in a crisis management scenario.

Both applications were iteratively developed with end user representatives involved throughout the process and the results illustrate that users both in the context of medical care, and the emergency management domain, are positive towards AR systems as a technology and as a tool in their work related tasks. The main contributions of the thesis are not only the end results of the user studies, but also the methodology used in the studies of this relatively new type of technology. The studies have shown that involving real end users both in the design of the application and in the user task is important for the user experience of the system. Allowing for an iterative design process is also a key point. Although AR technology development is often driven by technological advances rather than user demands, there is room for a more user centered approach, for single user applications as well as for more dynamic and complex multiple user applications.

( 5 ) ( 6 ) Acknowledgements

First of all I need to clarify that the animal on the cover of this thesis is not a tame dog, but a wild dingo on the beach of Frasier Island. The choice of title is not about the wild dingo however, it is of course a reference to the original piece "Cognition in the Wild" by Edwin Hutchins. I do not in any way intend to claim that this thesis has, or will have, the same impact in academia as the original work has had. However, I do hope that the title will encourage others to bring Augmented Reality applications into the wild. By this I do not necessarily mean bringing Augmented Reality to the dingos, but to ﬁnd real world problems that can be solved with the help of a little creativity and Augmented Reality technology.

The ideas in this thesis formed during my undergraduate studies and has since continuously been shaped by the people who have been involved in my education. Torbjörn Gustafsson and Per Carleberg were the people who introduced me to the field of Augmented Reality by asking if I would like to be part of augmenting the visual world like the terminator’s visual system in Terminator 2 (and of course I wanted to). Erik Hollnagel was my first supervisor and he, in collaboration with my ever present co-supervisor (and co-author) Björn Johansson helped me figure out the ways of cognitive systems engineering and systemic approaches to end user applications. After Erik left IDA, Arne Jönsson took over as my main supervisor and he has since then been an excellent facilitator of my work and always supportive of my ideas (and desire to travel).

I am also grateful for the support I have received from Christina Aldrin, Jenny Gustafsson and Betty Tärning at FMV. The financial support from FMV has allowed me not only to pursue my research interests but also to work in a place full of creative and supportive people. So thank you for many entertaining and inspiring fikas and lunches: Fabian, Jiri, Maria, Jody, Sara, Amy, Johan, Rogier and all the rest of HCS. And a special thank you to Arne Jönsson and Ola Leifler for all the last minute LaTex support and Dennis Netzell at LiU-tryck for being so enthusiastic about my cover design ideas.

Needless to say there are more people involved in my life as a PhD student and I would especially like to thank Dana, Kattis and Felix for pushing me over the ﬁnish line (once again), and Martin for being you. It’s also diﬃcult to imagine getting to this point if it wasn’t for my family - thank you mamma Anne, Fia

vii

( 7 ) and Leo (som fortfarande är min spindelman), Staﬀan (whose help with SPSS and general methodological insights has been invaluable), Anna, Stina and Arvid. And without my friends∗ my life would surely have been far less entertaining.

Finally - without the help of all willing participants there would not have been any end user studies at all, so thank you Margaretha Fredriksson and Carina Zetterberg at Örebro University hospital, Håkan Evertsson at the police department in Linköping, Christer Carlsson at the ﬁre and rescue services in Linköping and Peter Hagsköld at Malmen, Linköping. Not only for your dedication to your professions, but for all your support and help in ﬁnding participants and of course helping me discover and understand the real world problems that we could use Augmented Reality to solve. . .

∗see end titles

( 8 ) Contents

Abstract v

Acknowledgements vii

List of Figures xiii

List of Tables xv

1 Introduction 1 1.1 Mixed Reality and Augmented Reality ...... 2 1.2 Evaluating Human Computer Interaction ...... 4 1.3 Research Question ...... 5 1.4 Research Contributions ...... 6 1.5 A note on the terminology ...... 7 1.6 Reading Instructions ...... 8 1.7 List of publications ...... 9

2 Framing the thesis 13 2.1 Augmenting reality ...... 13 2.1.1 Display solutions for merging visual realities ...... 14 2.1.1.1 Optic see-through augmented reality ...... 15 2.1.1.2 Video see-through augmented reality ...... 17 2.1.1.3 Visually based marker tracking ...... 18 2.1.2 Interacting with virtual information ...... 19 2.1.3 Augmented Reality projects and applications ...... 21 2.1.4 Single and multi-user applications ...... 24 2.2 Shared views and common ground for collaboration ...... 27 2.3 User centered development of systems ...... 30 2.3.1 Human Computer Interaction in theory ...... 31 2.3.2 Usability engineering and evaluations ...... 32 2.3.3 Problems of human computer interaction and usability engineering approaches ...... 35 2.4 Systemic approaches to systems development ...... 36 2.4.1 Activity theory ...... 38

( 9 ) x

2.4.2 Situated cognition and distributed cognition ...... 39 2.4.3 Cognitive systems engineering and the use of artefacts . . . 40 2.5 Evaluating Augmented Reality systems ...... 44 2.5.1 Viewing Mixed Reality systems as Joint Cognitive Systems . 46 2.5.2 Choosing a perspective ...... 48

3 Research approach and methods 51 3.1 Iterative design process ...... 53 3.2 End user studies and evaluations ...... 56 3.2.1 Evaluation of systems and products ...... 56 3.2.2 Simulation and scenario for the collaborative Augmented Reality application ...... 58 3.3 Data collection methods used in the studies ...... 59 3.3.1 Questionnaires ...... 60 3.3.2 Interviews ...... 61 3.3.3 Observations ...... 62 3.4 Analysis of data in the studies ...... 64 3.5 Instruments used in the studies ...... 67

4 Single user studies 71 4.1 Iterative design of the application tasks ...... 72 4.2 Study 1 - the electro-surgical generator ...... 73 4.2.1 Methodology ...... 73 4.2.2 Participants ...... 74 4.2.3 Equipment ...... 74 4.2.4 The user task ...... 75 4.2.5 Data analysis ...... 76 4.2.6 Results ...... 76 4.2.7 Discussion of the results ...... 78 4.3 Study 2 - the troakar ...... 79 4.3.1 Methodology ...... 80 4.3.2 Participants ...... 80 4.3.3 Equipment ...... 81 4.3.4 User task ...... 82 4.3.5 Data analysis ...... 84 4.3.6 Results ...... 85 4.3.7 Discussion of the results ...... 90 4.4 Study 3 - the second troakar study ...... 90 4.4.1 Methodology ...... 91 4.4.2 Participants ...... 92 4.4.3 Equipment ...... 92 4.4.4 The end user task ...... 93 4.4.5 Data analysis ...... 93 4.4.6 Results ...... 94

( 10 ) xi

4.4.7 Discussion ...... 100 4.5 General observations and conclusions of the single user studies . . . 103

5 Multi-user study 107 5.1 The design and development process ...... 108 5.1.1 The ﬁrst iteration ...... 109 5.1.2 The second iteration of the Augmented Reality system . . . 111 5.2 The end user study ...... 113 5.2.1 Participants ...... 114 5.2.2 The Augmented Reality system used in the study ...... 115 5.2.3 User task and procedure ...... 117 5.2.4 Data analysis ...... 120 5.3 Results ...... 121 5.3.1 Comparing the three sessions ...... 121 5.3.2 Collaborating through the Augmented Reality system . . . . 126 5.3.3 Evaluating the Augmented Reality system ...... 130 5.4 Discussion of the results ...... 137 5.4.1 Information sharing and collaboration ...... 137 5.4.2 The iterative design process ...... 140 5.4.3 Using Augmented Reality to establish common ground . . . 140 5.5 General observations and conclusions of the multi-user study . . . . 141

6 Discussion of the user studies 143 6.1 General observations ...... 143 6.2 Introducing Augmented Reality in end user environments ...... 146 6.3 Concluding discussion of the studies ...... 148

7 Conclusions 151

Bibliography 155

Appendix 175

( 11 ) ( 12 ) List of Figures

1.1 The virtual continuum as described by Milgram and Kishino (1994). 2 1.2 A schematic view of a video see-through AR system...... 4

2.1 The optic see-through solution makes it difficult to project virtual information on different focal planes, thus creating the illusion that objects, which are meant to be at different distances, appear to be at the same distance from the user...... 16 2.2 A schematic view of a video see-through AR solution...... 17 2.3 An example of a fiducial marker...... 19 2.4 The human input-output machine...... 32 2.5 The Technology Acceptance Model after Davis (1989)...... 35 2.6 The Activity Theory model (after Engeström (1999); Kuutti (1996). 39 2.7 The traditional view of internal cognition in comparison with the distributed view of cognition...... 40 2.8 An example of a Joint Cognitive System in traffic Renner and Johansson (2006)...... 42 2.9 Examples of different levels of boundaries in a Joint Cognitive System (after Hollnagel and Woods 2005)...... 47

3.1 A general schematic view of the design and development process used in the included studies...... 53 3.2 The research cycle according to Miles and Huberman (1994). . . . . 64

4.1 An electro-surgical generator with markers for marker tracking. . . 73 4.2 The helmet-mounted AR system...... 74 4.3 The participant’s view of the AR instructions...... 75 4.4 A participant working with the AR system...... 76 4.5 A troakar as the one used in the study, the insert shows the diﬀerent parts...... 80 4.6 The marker used for the troakar study...... 81 4.7 A participant working with the troakar using voice input to the AR system...... 82 4.8 An example of the instructions as seen by a participant (rough translation - "screw on the smallest cap")...... 83 4.9 An example statement from the questionnaire...... 84 4.10 A participant wearing the AR system with voice input functionality during the second troakar study...... 92

xiii

( 13 ) xiv

4.11 An example of instructions as seen by the participant ("screw together with the cylindrical part")...... 93 4.12 Results from items 5, p=0.072 and 8, p=0.086 ...... 95 4.13 Result from items 12, p= 0.122 and 14, p=0.027 ...... 96 4.14 Result from item 9, p= 0.027 ...... 97 4.15 Comparison between the conditions, p=0.016 ...... 100

5.1 The study was conducted in a simulated natural setting in a hangar at a helicopter base...... 109 5.2 The HMD and redesigned interaction device, which allows the user to choose a virtual object and place it on the digital map...... 111 5.3 The users display showing the digital map with symbols and pointing used in the collaborative AR application ...... 112 5.4 The joystick interaction device used in the study...... 115 5.5 The simulated natural setting (a hangar at a helicopter base) in the end user study...... 118 5.6 A schematic view of the user study...... 119 5.7 Results from questionnaire items 1, It took a long time to start to cooperate and 5, I felt that the group controlled the situation. For further explanation, see text...... 123 5.8 Results from items 6, It was easy to mediate information between the organisations and 7, The map made it easy to achieve a common situational picture. For further explanation, see text...... 125 5.9 Results from items 8, The symbols made it easy to achieve a common situational picture and 9, The map became cluttered/messy. See text for further explanation...... 125 5.10 Results from items 12, The map helped me trust the situational picture and 13, The symbols helped me trust the situational picture. See text for further explanation...... 126 5.11 Results from Items 7, The map/AR system made it easy to achieve a common situational picture, 8, The symbols made it easy to achieve a common situational picture and 9, The map was cluttered/messy. The AR system scored signiﬁcantly higher than the paper map. See text for further details...... 129 5.12 Results from Items 12, The map/AR system helped me trust the situational picture, and 13 The symbols on the map helped me trust the situational picture. There are tendencies to diﬀerence between the sessions, see text for further details...... 130

( 14 ) List of Tables

3.1 A summary of the studies presented in the thesis...... 69

4.1 Statements in the closed response questionnaire...... 86 4.2 Overall scores on the questionnaire ...... 94 4.3 Overall score and completion time...... 95 4.4 Diﬀerence in statement responses...... 95

5.1 AR-system questionnaire, average score and standard deviation. As the statements in the questionnaire were both positively and negatively loaded (see for instance the first two items), the scores on the negatively loaded items were transformed in order to make the result easier to interpret. This means that in the table a high score is positive for the AR system/paper map and a low score is negative for the AR system/paper map...... 122 5.2 AR system questionnaire, average score and lower/upper confidence bound on the 6 point Likert scale. As the statements in the questionnaire were both positively and negatively loaded (see for instance the first two items), the scores on the negatively loaded items were transformed in order to make the result easier to interpret. This means that in the table a high score is positive for the AR system and a low score is negative for the AR system . . . . 131

7.1 A summary of the results and main contributions of the end user studies...... 152

( 15 ) ( 16 ) Chapter 1

Introduction

Augmented Reality (AR) was for a long time a technology limited to the few researchers who spent their time developing systems that allowed the merging of real and virtual worlds in order to enhance the reality of a human using (or wearing) the system. Recently, with the introduction of the smart mobile phones, the ﬁeld of AR has expanded drastically and today basically anyone with a relatively new mobile phone can download an application that allows them to superimpose virtual information over the real world (as seen through the camera and display of the phone). AR became a buzzword in 2009 and the technology was mentioned in numerous on-line articles, stories and blog posts. As with many new trends in technology though, the technology is much older than the latest hype. Overlaying virtual information on top of someone’s real world view is an old concept that has been used in binoculars and telescopic sights for weapons for a long time. Ivan Sutherland ﬁrst demonstrated the concept of head mounted displays to create a virtual, or semi virtual experience already in 1965 and since then one of the main display technologies used in AR development has been the head mounted displays. This thesis describes the development and evaluation of a number of AR applications that all are based on head mounted display technology.

The ﬁeld of AR has grown and expanded drastically in terms of commercially available applications during the last couple of years, but as previously noted, mainly mobile phone based applications. The stationary, projection based and 1

( 17 ) 2 Chapter 1. Introduction head-mounted AR applications are still noticeably absent in the commercial end user domain. This circumstance may be the result of the technical focus of the research, where emphasis has been placed on developing functioning rather than user-friendly applications. This is often the case in the earlier stages of any technical development as it is diﬃcult to conduct user studies on products that are not working properly. However, ensuring usability is in many ways the foundation for any user interface or product, including AR systems, and should be included in all stages of system development.

1.1 Mixed Reality and Augmented Reality

Mixed Reality (MR) is a collective term used for the technologies developed to merge real and virtual information. Milgram and Kishino (1994) describes Mixed Reality as a virtual continuum which illustrates the relation between Augmented Reality (AR), Virtual Reality (VR) and the stages in between, see Figure 1.1.

Figure 1.1: The virtual continuum as described by Milgram and Kishino (1994).

While VR are systems that are totally immersive and allows the user to experience a more or less complete virtual world, AR only amplifies certain features, or adds some virtual effects, to the world as it is experienced by the user. Usually the virtual elements are presented visually through a head mounted or a mobile display. The focus in many MR definitions is "the merging of worlds" but MR is also defined as the technology used to accomplish this merging of real and virtual worlds. The technology usually includes a computer, some form of small display (most

( 18 ) Chapter 1. Introduction 3 commonly used are PDAs, mobile phones or head mounted displays) and tracking software. Technically there are two primary solutions to the problem of visually merging virtual information into the perceptive field of a user. The most straight forward way is to use a see-through, head up, display which allows the user to see the real world through what can be described as a pair of glasses. In this solution the virtual information is projected directly onto the see-through display in the users field of view. This solution is rather difficult in terms of visibility and depth cues of the virtual information given to the user. Problems like these has lead to the development of the video see-through approach most commonly used in AR systems. In this approach the virtual and real world information is fused before it is presented in a display to the user. This is done by using a camera which delivers the real world image to the computer, and the application program then merges the virtual information into the real world image before the complete image is presented in the users field of view. This approach gives more control over the image presented to the user and also has considerably fewer problems with visibility compared to the optic see-through approach (Azuma, 1997). A more detailed description of the technology is given in chapter 2.

AR applications have been developed for numerous different purposes, such as instructional tools in manufacturing and processing industry, different types of games in mobile phones, for overlaying images during medical procedures and as information devices in head up displays in air planes etcetera. There are in general two main types of end user applications - for single user purpose or for multi- user purpose. In the AR domain there are examples of both types, but there are few applications that focus on interactive aspects and dynamic user driven tasks. The AR systems used in the studies presented in this thesis have been developed for different purposes. The first purpose was to give instructions on how to use medical equipment, which is an example of a rather static, straight forward task. The second purpose was to develop an application suitable for a task with dynamic and changing requirements, that also relied more on user input, than the previous application. The systems used are all based on video see-through AR technology and make use of head mounted displays and cameras as can be seen in Figure 1.2.

( 19 ) 4 Chapter 1. Introduction

Figure 1.2: A schematic view of a video see-through AR system.

The ﬁgure illustrates the AR concept with images from the second user study presented in the thesis. The user sees her own hands and the medical device in front of her, while at the same time the virtual animated instruction is overlaid on the projected image giving her a mixed real and virtual view.

1.2 Evaluating Human Computer Interaction

In order to improve an existing product or interface, such as an AR system, some type of evaluation is needed to find out what the problems are and how we can improve the product or interface to solve them. Usability evaluation is one type of product assessment that focuses on system functionality, users’ experience of the interaction and problem identification. There are different ways of explaining, understanding and predicting human behaviour in interaction. Some of the ideas have developed into frameworks, such as distributed cognition (Hutchins, 1995a; Suchman, 1987), participatory design (Schuler and Namioka, 1993), action research (Wood-Harper, 1985; Checkland, 1991), information processing (Card et al., 1983) and cognitive systems engineering (Hollnagel and Woods, 1983, 2005).

The most dominant framework regarding human action has been the information processing (IP) framework, in which the human is modeled in much the same way as a computer processor, with stimuli going in, being processed and then responses coming out as a result (Card et al., 1983). This was the dominating view on human behaviour and cognition during the early formation and development

( 20 ) Chapter 1. Introduction 5 of the cognitive science field of research. Usability evaluation methods, heuristics and guidelines have been heavily influenced by the information processing view on human cognition. Many usability guidelines and design recommendations are, however, focused on minimising the internal processing required by the user and usability evaluations are often designed to evaluate the factors identified by the guidelines prescribed for the design.

In contrast to the human processing model, cognitive systems engineering and distributed cognition approaches emphasise not only internal processes, but the cognition is seen as distributed between the human and artefacts in the interaction. That is, the cognition is distributed throughout the cognitive system which can consist of people, artefacts and the surrounding environment (Hollnagel and Woods, 1983; Suchman, 1987; Hutchins, 1995a). Applications developed in the AR domain are mainly developed and tested or evaluated using traditional usability methods, often with a focus on quantitative measures. This approach has a valid scientiﬁc and psychological foundation and the aim of this thesis is not to question knowledge about human cognition and human perception. However, there are fundamental diﬀerences between desktop computer applications and AR applications, which should be taken into account when performing user evaluations.

1.3 Research Question

This thesis aims at describing and evaluating both an Augmented Reality system, and the development process of that Augmented Reality system, by conducting studies in near real life situations with end users taking part in both the development and evaluation phase in order to give insight to future development of Augmented Reality systems. To meet this aim, three AR applications have been developed and evaluated; two single user applications with sequential instructional tasks; and one multi-user application with a dynamic and interactive task. One objective of the studies has been to apply a systemic method, the cognitive systems

( 21 ) 6 Chapter 1. Introduction engineering framework, to the design and development of the applications. The thesis also discusses how usability can be approached in the AR domain, both for single and multi-user applications.

1.4 Research Contributions

This thesis aims a contributing to the ﬁeld of Augmented Reality in describing how development and evaluation can be done in order to not only learn about technical solutions, but also how to tweak these solutions so that they actually match the expectations of the end users. The thesis also gives an example of how theories of systemic development processes can be applied to an actual system development process. The studies conducted and presented in this thesis should be seen as a practical application of the theories and methodologies presented. The thesis has contributed to the ﬁeld of Augmented Reality and Human Computer Interaction by

• presenting a methodological discussion in relation to Augmented Reality applications

• illustrating how iterative design and evaluation can be performed

• demonstrating how Augmented Reality is perceived and can be used in two examples of single user applications with static, sequential tasks

• describing how a multi-user Augmented Reality application can be developed and evaluated

• demonstrating how Augmented Reality can be used and perceived as support to collaboration in a dynamic task

• providing lessons learned for development and evaluation of Augmented Reality applications in real end user situations

• demonstrating a practical application of systemic theories

( 22 ) Chapter 1. Introduction 7

1.5 A note on the terminology

In any research description a certain selection of words will be reoccurring, as is the case in this thesis. Rather than explaining the terminology and the use of it as it comes this section presents the most frequent terms and abbreviations used and explains how they are used.

MR Mixed Reality, a collective name for all technology that merges real and virtual information.

MR System A technical system used to merge real and virtual information (predominantly visual and verbal information) and presenting it to a user. An MR system has capabilities of showing only virtual information as well as merging virtual with real world information.

VR Virtual Reality, a subset of MR which aims a giving the user a completely immersive virtual world experience.

AR Augmented Reality, a subset of MR which ads virtual information to a predominantly real image/representation of the world in order to enhance, or augment, the information the user receives.

AR system An MR system which is used only for AR applications, that is not completely immersive as the case with VR systems/applications.

Artefact A human made object.

Immersion Complete involvement or engagement in a task or experience. Often used in relation to VR and computer gaming describing a form of detachment from reality in favour of engagement in the virtual experience.

Application Used here as a term for the program that runs on the AR system.

System A system consists of several elements that coexist and interact for a speciﬁc purpose. It can be a technical system where the system consists of for instance hardware and software, but the term can also refer to socio- technical systems and cognitive systems.

( 23 ) 8 Chapter 1. Introduction

Socio-technical system A system consisting of humans and technology, often in relation to contextual work place research where the interaction between humans and technology are studied. Complex socio-technical systems refer to the interaction between complex societal infrastructure and human behaviour.

Cognitive system A system which has the ability to modify its pattern of behaviour on the basis of past experience in order to maintain control.

HCI Human Computer Interaction, a ﬁeld of research focused on the study of the interaction between humans and computers often in relation to design, development and evaluation of computer based systems and products.

Usability relates to the perceived or measured ease with which people can use a particular tool or other artefact in order to achieve a particular goal.

CSE Cognitive Systems Engineering is a ﬁeld of research which focuses on the study of complex socio-technical systems.

In the text the terms AR will be used when referring to the technical system used. By deﬁnition the system used is a MR system since it has the ability to run applications which are completely virtual, as well as applications that only augment the real world image. However, the applications presented in the thesis are all deﬁned as AR applications.

1.6 Reading Instructions

The introductory chapter has given a general introduction to the thesis in terms of defining the field of study, and presenting the research questions in focus. Chapter 2 begins with an introduction to the field of Mixed and Augmented reality, including some different technological solutions and examples of applications in different domains. The second part of the theoretical chapter introduces theory of human computer interaction (HCI) and the frameworks which have influenced

( 24 ) Chapter 1. Introduction 9 the research in usability. A selection of HCI related theories, such as the activity theory, distributed cognition and the cognitive systems engineering approach, as well as theories related to collaboration and common ground are presented to provide the backdrop for the research question.

In Chapter 3, the methodology which has framed the research is presented. The chosen method was user studies conducted in the users’ work environment and these studies are presented in detail in Chapters 4 and 5.

Chapter 6 aims at connecting the dots, by revisiting the research question and discussing approaches to development and evaluations of AR systems by describing the contribution of the end user studies and delivering a summarising discussion. The thesis ends with Chapter 7, which includes a brief summary and the ﬁnal conclusions and presents a possible future direction of the research in relation to user centred development and evaluation of AR systems.

1.7 List of publications

The studies in this thesis has previously been presented in parts in the following publications:

Nilsson, S., Johansson, B., and Jönsson, A. (2010). Cross-organisational collaboration supported by augmented reality. Submitted to: Transactions on Visualization and Computer Graphics.

Nilsson, S., Johansson, B., and Jönsson, A. (2009). Using AR to support cross-organisational collaboration in dynamic tasks. In Proceedings of IEEE ISMAR-09, Orlando, FL.

Nilsson, S., Johansson, B., and Jönsson, A. (2009). A holistic approach to design and evaluation of mixed reality systems. In Dubois, E., Gray, P., and Nigay, L., editors, The Engineering of Mixed Reality Systems. Springer.

( 25 ) 10 Chapter 1. Introduction

Nilsson, S., Johansson, B., and Jönsson, A. (2009). A co-located collaborative augmented reality application. In Proceedings of VRCAI 2009, Yokohama, Japan.

Nilsson, S., Gustafsson, T., and Carleberg, P. (2009). Hands free interaction with virtual information in a real environment: Eye gaze as an interaction tool in an augmented reality system. Psychnology Journal, 7(2):175–196.

Nilsson, S., Johansson, B., and Jönsson, A. (2008). Design of augmented reality for collaboration. In Proceedings of VRCAI 2008, Singapore.

Nilsson, S. and Johansson, B. (2008). Augmented reality as a tool to achieve common ground for network based operations. In Proceedings of 13th ICCRTS: C2 for complex Endeavours, Seattle, USA.

Nilsson, S. and Johansson, B. (2008). Acceptance of augmented reality instructions in a real work setting. In Extended Abstracts Proceedings of the 2008 Conference on Human Factors in Computing Systems, CHI 2008, Florence, Italy, pages 2025–2032.

Nilsson, S. (2008). Having fun at work: Using augmented reality in work related tasks. In Pavlidis, I., editor, Human-Computer Interaction. In-Tech.

Nilsson, S. and Johansson, B. (2007). A systemic approach to usability in mixed reality information systems. In Proceedings of Australia and New Zealand Systems Conference, Auckland, New Zealand.

Nilsson, S. and Johansson, B. (2007). Social acceptance of augmented reality in a hospital setting. In Proceedings of 7th Berlin Workshop on Human-Machine Systems, Berlin, Germany.

Nilsson, S., Gustafsson, T., and Carleberg, P. (2007). Hands free interaction with virtual information in a real environment. In Proceedings of COGAIN 2007 (Communication by Gaze Interaction IST FP6 European Project), Leicester, UK.

( 26 ) Chapter 1. Introduction 11

Nilsson, S. (2007). Interaction without gesture or speech - a gaze controlled AR system. In Proceedings of the 17th International Conference on Artiﬁcial Reality and Telexistence (ICAT), Esbjerg, Denmark.

Nilsson, S. and Johansson, B. (2007). Fun and usable: augmented reality instructions in a hospital setting. In Proceedings of the 2007 Australasian Computer- Human Interaction Conference, OZCHI 2007, Adelaide, Australia, pages 123–130.

Nilsson, S. and Johansson, B. (2006). User experience and acceptance of a mixed reality system in a naturalistic setting - a case study. In Proceedings of the International Symposium on Mixed and Augmented Reality, ISMAR’06, pages 247–248. IEEE.

Nilsson, S. and Johansson, B. (2006). A cognitive systems engineering perspective on the design of mixed reality systems. In ECCE ’06: Proceedings of the 13th European conference on Cognitive ergonomics, pages 154–161.

( 27 ) ( 28 ) Chapter 2

Framing the thesis

This chapter addresses the theoretical framework which has inﬂuenced the work and results presented in this thesis.

2.1 Augmenting reality

Mixed Reality (MR) is a general term for the technologies developed to merge real and virtual information. The deﬁnition includes any type of perceptual information - visual, haptic, sound or smell, but the currently most applied and researched type is the visual information domain. The visual MR domain is also the main application area of this thesis.

There are diﬀerent types of solutions to creating AR applications and the main types of visual applications include handheld, stationary, projection based and head mounted devices. In the early days of AR the head-mounted AR system was the dominating idea, inspired by Sutherland’s vision and prototypical "ultimate display" system built in the sixties (Sutherland, 1965, 1968). Head mounted solutions are still a large part of the research area for several reasons - they free the users hands for other tasks and they are also true to the idea of continuously enhancing, or augmenting the users perceptual ﬁeld. In stationary AR larger displays (desktop displays etcetera) are used and acts as windows to the virtually 13

( 29 ) 14 Chapter 2. Frame of reference augmented reality. Rather than being "attached" to the user, augmenting the world from the uses view point, the user sees the virtual world only when she/he actually looks at the display. In a similar way projection based AR is usually ﬁxed to one location as it demands an appropriate surface to project the virtual elements on. The MIT project the sixth sense explores the use of small "wearable" projector allowing the user to move around and see the projected image on top of any real surface (http://www.pranavmistry.com/projects/sixthsense/). Finally the handheld AR application area includes any type of mobile display such as tablet PCs (Träskbäck and Haller, 2004), mobile phones (Henrysson et al., 2005a,b), or PDAs (Wagner and Schmalstieg, 2003). This is the area of AR which to date has by far the most widespread and commercially available end user applications (for instance the Layar, www.layar.com, application for the Android mobile phones or iPhones). The amount of AR games for mobile phones has increased drastically the past year and there a numerous applications available for download for the most commonly used mobile phone platforms.

In general Augmented Reality is based on a few core technologies where tracking, registration and displays are the most prominent (Henrysson, 2007). Tracking is needed for the system to know where (position) and how (orientation) to place the virtual information in relation to the user and his/her viewpoint. Registration is the result of tracking - the alignment of the virtual information, and display technology is the basis for the end result - how the mixed information is presented to the user.

2.1.1 Display solutions for merging visual realities

There are several types of display solutions for presenting visual augmented reality to the user including projection based displays such as retinal displays (Kollin, 1993; Tidwell et al., 1995; Oehme et al., 2001) and spatial displays (Olwal and Henrysson, 2007), screen based displays such as head mounted displays (HMDs), handheld devices such as tablet PCs, PDAs and mobile phones (Möhring et al., 2004; Henrysson et al., 2005a,b) as well as more AR oriented handheld

( 30 ) Chapter 2. Frame of reference 15 displays (Grasset et al., 2007; Schmalstieg and Wagner, 2007). Up to 2005 the main display chosen for AR applications was of the HMD type (Bimber and Raskar, 2006), however more recently the handheld, more speciﬁcally the mobile phone based display, solution has grown drastically. These types of displays that are either worn or held by the user, and are the most common tool for presenting virtual information, but there are also examples of solutions which detach most of the technology from the user (Bimber and Raskar, 2006). Instead of being dependent on the user, these displays, known as spatial displays, are integrated into the environment. AR applications using projectors are examples of spatial display systems.

Regardless of display solution there are two principally diﬀerent ways of merging real and virtual worlds in real time today; video see-through and optic see- through (Azuma, 1997; Azuma et al., 2001; Kiyokawa, 2007; Gustafsson et al., 2004, 2005). Both types of systems were developed for HMDs, which are displays that located directly in front of the users eyes (Sutherland, 1968). The two principal technical solutions, however, are similar regardless of where the display is placed.

2.1.1.1 Optic see-through augmented reality

The ideal form of an Augmented Reality system would be a lightweight, non- intrusive solution, perhaps in the shape of a pair of glasses that are easy to fold up and put away when not in use. Ideally there would only be wireless communication in the system and barely noticeable batteries running the system. The technical solution that comes closest to this today is optic see-through AR. In optic see- through AR, the user has a head mounted see-through optical display which allows the user to see the real world as if through a glass lens (Kiyokawa, 2007) . The virtual information is then overlaid on the see-through display.

Although the technique of blending virtual and real information optically is simple and cheap compared to other alternatives, this technique is known to cause some problems. For one, the virtual projection cannot completely obscure the real world image - the see-through display does not have the ability to block oﬀ incoming light

( 31 ) 16 Chapter 2. Frame of reference to an extent that would allow for a non-transparent virtual object. This means that real objects will shine through the virtual objects, making them diﬃcult to see clearly. The problem can be solved in theory, but the result is a system with a complex conﬁguration (Azuma, 1997). There are also issues with placement of the virtual images in relation to the surroundings in optic see-through displays. Since the virtual objects presented to the user are semi-transparent they give no depth clues to the user. In real vision humans with two functioning eyes will have several important visual cues that help in judging where an object is placed. For instance the size of the object related to the size of other known objects around gives clues to the distance. Other important cues are if the object is partly occluded by another object, then you know that it is behind the occluding object. When virtual information is optically projected onto the see-through display the virtual objects seem to be aligned along the same focal plane, see Figure 2.1.

Figure 2.1: The optic see-through solution makes it difficult to project virtual information on different focal planes, thus creating the illusion that objects, which are meant to be at different distances, appear to be at the same distance from the user.

This gives the impression that all virtual objects are the same size, at the same distance from the viewer which is not an ideal situation for most Augmented Re- ality applications. In natural vision, objects are perceived in diﬀerent focal planes allowing the viewer to discriminate between objects at diﬀerent distances (Haller et al., 2006).

( 32 ) Chapter 2. Frame of reference 17

2.1.1.2 Video see-through augmented reality

A way to overcome some of the problems with optic see-through is by using a technique commonly referred to as video see-through AR, where a camera is placed in front of the users’ eyes, see Figure 2.2. The captured camera image is then projected to a small display in front of the users’ eyes (Azuma, 1997; Haller et al., 2006; Kiyokawa, 2007) Gustafsson 2004). The virtual images are added to the real image before it is presented in the display which solves the problem with the semitransparent virtual images described above, as well as gives control over where, in what focal plane, the virtual objects are placed.

Figure 2.2: A schematic view of a video see-through AR solution.

This in turn places other demands on the system. The video resolution of the camera and display sets the limit for what the user perceives, however the cameras and displays used today offer high resolution images. The main problem is not resolution but rather that the field of view is very limited compared to natural vision. Another issue is the eye offset; the cameras position is not exactly where the eyes are located, which gives the user a somewhat distorted experience, since the visual viewpoint is perceived to be where the camera is (Azuma, 1997). This means that rather than perceiving something as being where it actually is, for instance 20 cm from your hand, the user will experience the object as being 20 cm from where the camera is, thus grasping for the object in the wrong place. This type of offset is something humans tend to adjust to fairly quickly however,

( 33 ) 18 Chapter 2. Frame of reference as can be seen in Nilsson and Johansson (2006b, 2007). Even though humans are able to adjust to the offset, the difference between the bodily perceived movement and the visual movement as seen through the display can have effect on the user experience of the system. This is the kind of visual effect that in some cases cause motion sickness and nausea, what is also referred to as cybersickness (Stanney, 1995; Jones et al., 2004).

Despite these problems there are important vantage points with the video see- through solution. One has already been pointed out - the ability to occlude real objects. Another is that the application designer has complete control over the presented image in real time since it is run through the computer before it is presented to the user. In the optic see-through design, only the user will see the ﬁnal augmented image. To conclude; there is a trade-oﬀ between the optic see- through systems and the camera based systems, and the available resources often determine the choice of solution.

2.1.1.3 Visually based marker tracking

The previous section described the technical solutions used to present the visual information to the user, but there are other important features included in MR and AR systems. One of the most important issues when using AR technologies is to solve the problem of how, and where, to place the virtual image, regardless of display solution. In order to place the virtual information correctly, the AR system needs to "know" where the user and user view point is. This means that the system has to use some kind of tracking or registration of the surrounding environment. This is what is referred to as "tracking", and this is in itself a major research area with applicability to not only the MR domain but many other types of systems as well.

There are diﬀerent techniques to solving the tracking and registration problem, and several of them can be combined to ensure more reliable tracking of the environment. Usually diﬀerent sensors are used to register the surrounding environment and this information is then used as a basis for placing the virtual

( 34 ) Chapter 2. Frame of reference 19 information (Azuma et al., 2001). One of the most commonly used techniques today is vision based tracking, where the environment is prepared with markers that can be recognized by feature tracking software (Kato and Billinghurst, 1999, 2004). Figure 2.3 shows an example of a marker that can be used for this purpose.

Figure 2.3: An example of a ﬁducial marker.

By tracking and identifying markers placed in the environment the position of the camera in relation to the marker can be calculated, and hence the virtual information can be placed in the display relative to the marker position (Kato and Billinghurst, 1999).

When using a camera based AR system (video see-through AR) the visual tracker is already there – the video camera. In optic see-through systems the tracker system must be added, either in the shape of cameras for visual marker tracking or some other kind of tracking devices.

2.1.2 Interacting with virtual information

Presenting virtual information in three dimensional space also means that the user has to react to and interact with this information in that space. The way the AR application is built deﬁnes how the user can give input to the system, and the method of input can also aﬀect the user experience of the system. Examples of this type in input are tangible user interaction, multimodal input and mobile interaction (Billinghurst et al., 2009). Despite the fact that AR is a relatively

( 35 ) 20 Chapter 2. Frame of reference old field (dating back to Sutherland (1968)s first concepts) the focus of research has been to achieve the visual merging of realities and not so much emphasis has been placed on allowing the user to give input to the system (Billinghurst et al., 2008, 2009). This has lead to applications where the user is a more or less passive recipient of information with very limited ability to manipulate the virtual content. As a result many applications developed which require user input has used computer interaction tools such as keyboards or mouse interaction. Even many of the new mobile phone AR games use buttons or the joypad as only (or main) source of user input. This may limit the user user experience, and even though many applications are advanced they may appear as very primitive to the user. There are however some other alternatives to user input such as tangible user interaction devices using for instance force feedback (Ishibashi et al., 2009) or haptic interaction as developed in the SPIDAR platform (Sato, 2002). Another input method used for AR is gestural input as described by Buchmann et al. (2004), where tracking of the user’s finger tip movements makes it possible to interact with virtual information through natural hand and finger movements. Another example of tangible interaction is described in the Tangible AR concept (Billinghurst et al., 2008), where each virtual object that is presented in an AR environment has a physical, tangible counterpart that the user interacts with. The user can thus interact naturally with a virtual object by manipulating a real, tangible object.

Other applications use multimodal input, for instance both the use of physical buttons as well as voice input (Nilsson and Johansson, 2007), or voice input alongside with gestural input as described by among others Billinghurst (1998), Billinghurst et al. (2009) and Livingston et al. (2005). Cheok et al. (2002) describe a game, or what they refer to as a novel entertainment system, called Touch-Space, where the users physical context aﬀects the game, and the interaction is both social and physical in the form of touching. In the game the user can collaborate with both real and virtual objects using tangible interaction technology.

( 36 ) Chapter 2. Frame of reference 21

2.1.3 Augmented Reality projects and applications

Mixed and Augmented Reality applications can be found in diverse domains, such as medicine, military applications, entertainment and edutainment, distance operation, geographic applications, technical support and industrial applications. One of the earliest end user applications developed was the application described by (Caudell and Mizell, 1992) where Boeing manufacture workers used AR technology to receive instructions on how to draw the wiring of the aircraft.

Reitinger et al. (2005, 2006) describe an AR based tool for liver surgery planning, where AR is combined with real time volume calculation in order to allow both visual 3D perception and real 3D user interaction (a tracked panel and pencil).

Billinghurst et al. (2001) ﬁrst described their edutainment application called MagicBook in 2001 and in 2008 Grasset et al. (2008) describes the further development and expands the concept by adding visual and auditory enhancements to an already published book. The MagicBook concept is based on marker tracking techniques in order to give life to a story, in where each page of the book pops out to give a 3D representation of the story.

AR can also be used to augment cultural experiences or for creative learning environments. Stapleton et al. (2005) and Hughes et al. (2004) describe how they use AR for informal education, where they for instance augment diﬀerent aspects of a museum exhibition with virtual user experiences. This is an example of how to use AR for visualisation of historic or natural events. AR can also be used for visualisation of geographic data as described in the Tinmith project1 (Piekarski, 2006). AR prism is a geographic, scientiﬁc visualisation application which allows users to overlay and interact with virtual 3D geographic information onto a real map (Hedley et al., 2002). The interaction was based on physical markers and tangible interaction techniques and the users, wearing HMDs stood around a map and as they moved the marker across the map they saw a graphic overlay of longitudes and latitudes. Additional markers allowed them to explore other types

1see www.tinmith.net for further information.

( 37 ) 22 Chapter 2. Frame of reference of data, such as terrain models, related to each point in the real map. The users could also lift and rotate the models and view them from diﬀerent angles and pass them around. The informal study conducted showed that the users liked that they could see each other while at the same time seeing the virtual model and they felt that this was close to face-to-face interaction.

AR can also be used to support distance operation (Milgram and Ballantyne, 1997). Piekarski and Thomas (2004) describe a framework where they combine AR technology and CAD working planes to allow for action with virtual objects both within the user’s reach, and with objects at a distance. The concept described includes manipulation of 3D objects, display of virtual information and creation of new geometry in order to support 3D modelling tasks in a mobile outdoor environment. Another approach to distance operation or manipulation is described by Lawson et al. (2002) where they use AR technology for telerobotic inspection and characterisation of remote environments. They use a remote vehicle equipped with a stereoscopic camera, which gives information to the human operator, who in turn by using 3D virtual cursors can measure and model the environment displayed in the images from the remote vehicle.

In the military domain there are several examples of using AR technology to present information of different kinds. Livingston et al. (2002) and Julier et al. (2000) describe BARS, a Battlefield Augmented Reality System, which was used to provide users with the ability to select and get more information about objects in their environment. Gustafsson et al. (2005) describe how AR can be used to give instructions on how to perform maintenance on military aircraft. Using AR to give instructions has also been done in other domains, for instance in manufacturing as mentioned above (Caudell and Mizell, 1992), and to give instructions for object assembly (Tang et al., 2003). The latter study reported that the AR instructions actually improved the task performance compare to other means of instructions and that AR is effective in reducing mental workload in these types of assembly tasks.

( 38 ) Chapter 2. Frame of reference 23

Besides the already mentioned research examples, there are other significant projects involving several collaborating partners developing AR systems and applications. The ARVIKA project is one of these collaboration projects with participants from both academia and industry (Weidenhausen et al., 2003; Friedrich, 2004). It was funded by the German ministry of education and research and included sponsoring participants from Audi, Airbus, BMW, Zeiss, Daimler Chrysler, Ford, Siemens and several more during 1999 and 2003. In this project the focus was mainly on industrial applications of AR, developing AR applications mainly for production and service in automotive and aerospace industry as well as process and power plants etcetera. One of the main values of AR for industry in the ARVIKA project was described as the possibility for AR to increase productivity as the technology allows the presentation of virtual information, such as task descriptions or instructions, directly in the users field of view, when and where she/he actually needs them, and thus reducing the mental effort of matching instructions from one medium or place to another.

Another project is the ARTESAS project, which was coordinated by Siemens, and also funded by the German Ministry for Education and Research and supervised by the DLR (German Aerospace Center). The main focus of the project was to develop marker-less procedures for AR in complex industrial environments, with consideration taken to ergonomic and usability questions2. Another project of relevance to the field is AMIRE, an EU-funded project which existed between 2002 and 2004, and was continued in the shape of VIRTHUALIS after 2005. The main goal of the project was to enable non-expert researchers to use MR for their applications and to create and modify these MR applications with the support of dedicated tools that allow an efficient authoring process for MR. Within the project two demonstrators for different fields were developed, a museum application to enhance the visitor experience and a training application for an oil refinery3.

Although these projects and many more involving research in the AR domain include an end-user perspective, very few papers report on results of extensive end

2see www.artesas.de for further information. 3see www.amire.net for further information.

( 39 ) 24 Chapter 2. Frame of reference user studies or HCI evaluations (Livingston, 2005; Swan II and Gabbard, 2005). Despite the potential of the technology, the research has still primarily focused on prototypes in laboratories, mainly due to the constraints of the hardware currently available to implement the systems (Livingston, 2005). This is also a reason why there are so few end user studies of MR and AR techniques; the hardware constraints also limit the human factor research in the area. Many HMD based applications are still bulky and are not wireless which influences the ergonomic design choices available. Still there have been user studies published, and the results point in the same general direction; there are several usability problems that are normally explained by hardware limitations, and despite these problems users respond positively to the use of AR for several different applications, for example see Bach and Scapin (2004), Haniff and Baber (2003) and Nilsson and Johansson (2006b). However there are other issues than hardware that affects the user experience of the AR system, and these issues may become easier to identify when apparent hardware related issues (such as motion sickness, limited field of view and the lack of depth perception etcetera) are solved.

2.1.4 Single and multi-user applications

There has been a focus on developing applications for single user tasks, especially in the industrial domain, such as the early Boeing application where the AR system was used to show the user how to draw the wires in the airplanes, or other similar tasks such as for assembly tasks or other types of instructional tasks. In other application areas where focus has been more on fun or education rather than work related tasks the applications may be open for several users but the systems are still designed for one - only one person can wear the same HMD and mobile displays tend to be single user oriented as they are rather small and diﬃcult to view for more than one person at a time, thus requiring several AR systems with the ability to interact. Applications may, however, have been multi-user oriented such as educational applications for visualisation of scientiﬁc data (Schmalstieg

( 40 ) Chapter 2. Frame of reference 25 et al., 1996) and the MagicBook (Billinghurst et al., 2001) where objects pop out of the pages allowing 360 degrees views.

An issue with these applications is that they tend to be one-way communica- tional (Billinghurst et al., 2009). That is the MagicBook will show the reader diﬀerent objects, but not necessarily allow the user to manipulate those objects. The more scientiﬁc visualisation applications may allow the students to manipulate the objects in simple ways such as rotation etc, but the base of the interaction is a single user approach.

While single user applications are the most common, systems for multi-user applications are not rare. Virtual environments seem to be a natural place for computer supported collaborative work applications (Billinghurst and Kato, 1999) and have been used as tools for training and simulating collaborative work, for instance the CAVE system and the Virtual Workbench (Fuhrmann et al., 1997) and the Studierstube projects (Szalavári et al., 1998; Schmalstieg et al., 1996) of which several were developed with more than one user in mind. In this project the users can see each other in the real world while at the same time manipulate and view virtual objects. Shared Space is another interface which allows user to interact with virtual objects while seated around the same table, seeing and communicating with each other (Billinghurst et al., 1997; Billinghurst, 1999). This application was a very simple game allowing complete novice users to collaborate in order to ﬁnd matching virtual objects, that when placed together triggered an animation.

Another example of technology used for collaborative work is an immersive Virtual Reality (VR) training environment for emergency rescue workers (Li et al., 2005). However users tend to prefer non- immersive applications for collaboration (Kiyokawa et al., 2002; Billinghurst et al., 2002b). This means that AR may be more suited for collaborative tasks than immersive systems like the VR applications mentioned above (Kiyokawa et al., 2002; Billinghurst et al., 2002b). Inkpen (1997) reported that users perform better if they are huddled around one workstation, or computer, rather than being spread out, collaborating on separate workstations. This is mainly due to the fact that being separate

( 41 ) 26 Chapter 2. Frame of reference means that the collaborating people may not take notice of normal communication cues that take place during face-to-face collaboration. However this may be true for desktop applications, but not necessarily for other types of systems, such as AR systems. Billinghurst and Kato (2002) compares aﬀordances in relation to both interaction and user viewpoints, between a face-to-face condition, AR and projector screen condition and ﬁnds that the similarities between AR and face- to-face are far greater than the similarities between face-to-face and the more traditional desktop/workstation projector based condition. While the projector based condition uses a public display and thus one common view point for the user, the AR condition has individual displays and thus independent view points much like a natural face-to-face situation. This means that the approach of using single user display systems like AR for collaborative multi user applications still is a viable option.

Regenbrecht and Wagner (2002) describe an AR application they call "MagicMeet- ing" using head mounted displays with built in cameras, aiming at face to face collaboration. In the application they combine ordinary desktop items, interactive 2D desktop screens integrated into a 3D environment. In the application the users are seated around the same table with a 2D presentation screen and the "cake platter", which is the main device for the shared 3D display. The main goal of the system is to integrate 2D and 3D data into one shared environment. Other similar approaches to collaborative work through AR are based on the idea of teleconference systems and Billinghurst and Kato (1999) and Billinghurst et al. (2002a) describe how AR can be used to support workers in mobile occupations. Their system includes avatars and live video of remote collaborators, which can be superimposed over any real location. In this way the remote collaborators would appear as if they are in the same place.

( 42 ) Chapter 2. Frame of reference 27

2.2 Shared views and common ground for collaboration

Exercising command and control is an attempt to establish common intent to achieve coordinated action (McCann and Pigeau, 2000). Successful communication is obviously necessary to achieve this. When personnel from different organisations work together under stress, as in many crisis situations, there is always a risk that misunderstandings emerge due to differences in terminology or symbol use. These situations are also dynamic, meaning that the problem is constantly changing, thus requiring several actions or decisions in order to reach the goal (Brehmer, 1990). Another significant aspect of these types of events is that the decisions must be made in real time, and each actor must act in the present and cannot wait until he or she has enough information. This type of activity can be compared with more static situations, or scenarios, where a decision made is the starting point for a chain of events rather than the response to a chain of events.

In projects where aspects such as collaboration, dynamic work flow and being able to adapt to changing requirements are important, the technical solutions usually aim at enhancing the ability to work and communicate about the tasks at hand. By sharing the same view of situations, decision-makers are believed to be supported in their effort of maintaining the current state of the joint activity. There are several potential problem areas that have to be addressed for cooperation to be efficient in an environment like this, such as training, cultural differences, organisational practices and so forth. The design, based on the idea of a shared focus on a map, is however easily recognised from "traditional" command centers and poses the same difficulties as they always have. The essential difference lies in the idea that decision makers from different branches and even organisations should cooperate in the same environment, sharing the same map. In many system designs, it is often assumed that merely providing a shared representation is enough to facilitate a shared understanding of a situation when a team of decision makers work together. However, linguists and psychologists have observed that in reality, meaning is often negotiated or constructed jointly (Clark, 1996). Although providing the

( 43 ) 28 Chapter 2. Frame of reference same view of a situation to two or more people is a good starting point for a shared understanding, things like professional and cultural background, as well as expectations formed by beliefs about the current situation, clearly shape the individual interpretation of a situation. This means that for at least a foreseeable future, problems concerning communication and negotiation will arise due to the diﬀerences in background pointed out. The coupling between the use of language and objects in a shared space is the initial obstacle that has to be overcome. As noted in the previous section, virtual environments seem to be appropriate arenas for collaborative applications (Billinghurst and Kato, 1999).

For any computer supported collaborative work (including AR applications) to be successful it is important to develop new technology that is appropriate for real world collaborative activities (Crabtree et al., 2005). However, users tend to prefer non-immersive applications for collaboration (Kiyokawa et al., 2002; Billinghurst et al., 2002b). This means that AR may be more suited for collaborative tasks than immersive systems like the VR applications mentioned above (Kiyokawa et al., 2002; Billinghurst et al., 2002b). A general problem with many computer supported cooperative work systems is that they are often based on single user systems, rather than being designed initially for collaboration and taking into account aspects of collaborative work (Fjeld et al., 2002).

Kiyokawa et al. (2002); Billinghurst et al. (2002b) conducted an evaluation comparing using AR to support collaborative work with how similar work is conducted without AR. What they found was that their system did in fact exceed the expected outcome and that AR is a promising tool for collaborative work. However, few, if any, AR systems have actually been aimed for use in crisis or emergency management situations which are examples of real world situations where collaboration is essential. Emergency management often demand collaboration between diﬀerent organisations, not least at the command and control level.

As noted, in linguistics it is common knowledge that time has to be spent on establishing common ground, or a basis for communication, founded on personal

( 44 ) Chapter 2. Frame of reference 29 expectations and assumptions between the persons communicating with each other (Clark, 1996; Klein et al., 2005). Clark (1996) denotes the knowledge two or more individuals have when entering a joint activity common ground.

Common ground is the least shared understanding of the activity that the participants need to have in order to engage in a joint action with a higher goal than creating common ground in itself. A simple example is person A throwing a ball to person B. Unless person A’s intent is to throw the ball to person B, and person B’s intent is to catch it, we can not call the action joint. The common ground in this case is the shared understanding of the activity (catch-throw), and the joint activity emerges from the intent of fulﬁlling these obligations. The essential components of common ground are the following three (Clark, 1996, p.43):

(1) Initial common ground. The background knowledge, the assumptions and beliefs that the participants presupposed when they entered the joint activity.

(2) Current state of the joint activity. This is what the participants presuppose to be the state of the activity at the moment, and

(3) Public events so far. These are the events that the participants presuppose have occurred in public leading up to the current state.

The maintaining of common ground is thus an ongoing process, which demands both attention and coordination between the participants. Initial common ground is maybe the most difficult part to bridge. It is composed by a number of different attributes, like cultural background, gender, age, professional knowledge etc. Establishing this between two participants with very different history will be time consuming and effort demanding. The second component, current state of the joint activity depends both on the participants’ initial common ground and on the public events so far. Thus, according to Clark, using language means engaging in a joint activity, which strives towards a shared goal with an established common ground. It is not hard to see how this applies to collaborative work as well as command and control work.

( 45 ) 30 Chapter 2. Frame of reference

2.3 User centered development of systems

Humans have been interacting with technical devices and artefacts for centuries, but despite this fact it is only in the last 60-70 years that human machine interaction has become a field of research in its own right, under names such as ergonomics and human factors. Henry Ford’s introduction of the production assembly line in the early 1900’s is an example of how studies of man and machine can lead to new solutions based on the capabilities of both humans and machines (although the assembly line production may be claimed to be the product of economics and productivity rather than ergonomics and human factors). When the first computers began to be used by a general user population (that is, non scientists) the need for studies of human factors in relation to technical systems started to grow. Even before that, in the 50’s and 60’s the research field of cognitive psychology began to take shape with the crossover of research from artificial intelligence, linguistics, psychology and ergonomics (Stillings, 1995). The origin of Human Computer Interaction, or HCI, as a research domain has often been claimed to be in 1982 with the first general conference on "Human Factors in Computer Systems".

Human Computer Interaction as a domain of research stems back to the 1970’s and Hansen’s "Know thy user" approach to user engineering (Hansen, 1971). However, the roots of HCI go back much further and can be found in the field of ergonomics, which dates back to the beginning of the 20th century, where studies of workers in factories and industries began to emerge. As a research field ergonomics, or human factors as it is usually referred to in the US, has been around at least since the 1940’s (Dix et al., 2004; Nickerson and Landauer, 1997). According to Dix et al. (2004), HCI "involves the design, implementation and evaluation of interactive systems in the context of the user’s task and work", and in this definition all types of computers should be included, both traditional desk top computers and embedded, or ubiquitous, computers. The core concepts in all definitions of HCI are people, computers and tasks, and the purpose of the research is to improve aspects of usability and usefulness of technical systems (Nickerson and Landauer,

( 46 ) Chapter 2. Frame of reference 31

1997). The dominating framework within HCI has been the information processing view-point which originates from Broadbent’s view on cognition as a sequential series of processes (Broadbent, 1958) and the view of the human as an information processing unit. However this viewpoint has been challenged since the beginning of HCI within frameworks such as Cognitive Systems Engineering, CSE (Hollnagel and Woods, 1983) and distributed cognition (Suchman, 1987; Hutchins, 1995a), as well as embodied interaction within the design research domain (Dourish, 2004).

2.3.1 Human Computer Interaction in theory

Landauer (1997) states that there are four main goals of HCI; evaluation or comparison of existing systems or features in systems; invention or design of new systems; discovering and testing relevant scientiﬁc principles and; establishing guidelines and/or standards of HCI.

In general the theories within the HCI domain are of three main categories, of which one include the explanatory theories, which aim at explaining the interaction between human and computer. Second there are the predictive theories which often aim at quantifying the interaction in order to predict it. Many of the predictive theories precede the computer era and stem from basic research in ergonomics, psychology and physiology, for example Fitts law, which dates back to 1954 (Fitts, 1954). Fitts law is a model of human movement and in HCI it is used to describe speed in relation to object size, for instance designing for faster interaction by using larger icons. Third there are theories constituting of taxonomies and models such as GOMS (Goals, Operators, Methods, Selection rules), which are abstractions of reality (Shneiderman, 1998). Several principles, standards and guidelines have been developed to define the field of HCI. Among these the most widespread is possibly Shneiderman’s 8 golden rules of interface design (Shneiderman, 1998), Norman’s design principles (Norman, 1998) and Nielsen’s 10 usability heuristics (Nielsen, 1993). One assumption that has bound the different approaches in HCI together has been the general idea that the human user can be analysed and seen as a human, biological, version of a computing machine. The focus often lies on

( 47 ) 32 Chapter 2. Frame of reference the interaction between a technological system and the human user operating that system, often viewing the human as an input- output device much like the computer (see Figure 2.4, (Card et al., 1983).

Figure 2.4: The human input-output machine.

In this model of human cognition, information enters the mind in various forms, is then processed internally on several levels, and then the mind, or body, produces a response. The processing can consist of comparing and matching and the results are in form of reactions to what has been processed. This view is an example of how basic research in psychology and human cognition has been expanded into theories of how to design interfaces. For example, the information processing unit that is the human mind is limited to handling memory chunks of "7 plus minus 2" (Miller, 1956), a fact that has been incorporated in most usability guidelines. Other basic research ﬁndings that have been incorporated into HCI guidelines are ﬁndings on visual perception such as the use of colour and grouping of similar objects, the gestalt laws (Gazzaniga et al., 2004; Friedrich, 2000).

2.3.2 Usability engineering and evaluations

Usability Engineering is an attempt to improve usability aspects of a product or system in a methodological, quantitative and structured way during its’ development process (Gould, 1988). Usually this is achieved by product evaluations and testing during the development cycle, and the measurements used are in general the measurements for HCI summarized by (Shneiderman, 1998). These are the time it takes to learn, the speed of performance, error rates, retention over

( 48 ) Chapter 2. Frame of reference 33 time and the subjective satisfaction (Dix et al., 2004). There are three typical goals of evaluations; testing the systems functionality; rating the users’ experience of the system; and identifying usability problems (Dix et al., 2004). These three main categories of evaluation are usually described in terms of usability testing, ﬁeld studies and heuristic or analytical evaluation.

The main goal of a usability test is not to demonstrate or explain phenomena but simply to find problems in the system that is being tested (Dumas and Redish, 1999). In traditional usability testing methodology it is important that the evaluator is in complete control over the environment and the course of the tests, and the results should be quantifiable - that is, measuring performance is interesting if the measures are quantifiable. Performance can be measured in for instance time, amount of errors and type of errors, as previously mentioned. Field studies differ from the usability tests in that the users are not placed in a laboratory setting; instead they are preferably studied in their natural environment. The analytical evaluations are for instance studies of how well a system complies with usability guidelines such as the heuristics presented by Nielsen (1993). Usability studies are often used to find usability problems in user interfaces (the first goal of HCI as presented by Landauer (1997), see above) but there are problems with these types of usability evaluations as reported by among others Hertzum and Jacobsen (2003). In their study, Hertzum and Jacobsen reviewed 11 studies with these methods, and the results showed that evaluators evaluating the same interface with the same evaluation method detected a markedly different set of problems. This is what is referred to as the evaluator effect and illustrates how subjective these supposedly objective measures really can be.

There are other ways of assessing products, or systems usability, than the quantitative usability test approaches. Among these are qualitative user studies where observations and interviews are a vital part. These methods are often used as part of participatory design and/or evaluation methods, in which both expert evaluators and real end users are a part of the development and design process (Schuler and Namioka, 1993; Muller and Kuhn, 1993). These methods are often used as part of participatory design and/or evaluation methods, in which not

( 49 ) 34 Chapter 2. Frame of reference only expert evaluators are used, but the user is also a vital part of the usable design process Patton (2002); Schuler and Namioka (1993). The purpose of a usability evaluation is to improve utility as well as usability of a product or system. In iterative design processes the evaluation can be deﬁned as a formative evaluation, where the evaluation is an ongoing process that shapes and reshapes the product. A summative usability evaluation is often done in the form of end user studies, where researchers do follow-ups of the product and how it is actually used by the end user.

The concept of usability engineering is still based on the HCI assumptions about what constitutes usability in a product or system. Another issue that usability methods such as cognitive task design (Hollnagel, 2003) have to deal with is the so called "envisioned world problem" (Hollnagel and Woods, 1983, 2005; Dekker and Hollnagel, 2004). Cognitive task design and other methods with similar approaches are based on observations of how a user completes a task in which the system or artefact is involved. The "envisioned world” problem states that even if a good understanding of a task exists, the new design/tool will change the task, rendering the first analysis invalid. When new technologies are introduced into a domain it will affect the user and the task on both a practical and a social level. The process of change requires knowledge, not only about the system introduced but also about the domain. The technical system or interface which is introduced should have as much positive effect on the user and her/his work as possible, while at the same time minimizing the negative effects of the system both for the user and other individuals. Fundamental usability awareness implies that the interface or system should not be harmful or confusing to the user, but rather aid the user in her/his tasks. However, traditional usability guidelines, such as the ones presented by Nielsen (1993) or Shneiderman (1998) often do not include the context of use, the surrounding and the effect the system or interface may have in this respect.

( 50 ) Chapter 2. Frame of reference 35

2.3.3 Problems of human computer interaction and usability engineering approaches

One problem with usability engineering and the more quantitative usability evaluations and methods, is that they often deliver measurements of very speciﬁc aspects of a user task and situation; measurements that aim more to satisfy the usability speciﬁcations, rather than usability per se (Dix et al., 2004). An aspect of usability that should not be forgotten (but often is) is the actual acceptance of technology within the user community. A product or technology may get low "scores" on usability testing with for example heuristics or laboratory experiments, but can still be a popular user product. This has to do with the actual use of the product, and illustrates the fallacies and problems with usability testing in laboratory or research settings. Davis proposed the Technology Acceptance Model (TAM) in 1986 to explain this phenomenon (Davis, 1986). TAM was originally developed for understanding of the causal link between users acceptance of personal computer based systems and external variables such as system design features (Fenech, 1998). The model is an adaption of the Theory of Reasoned Action which models user acceptance of information systems in general (Hubona and Blanton, 1996).

Davis (1989, 1993) describes two important factors that inﬂuence the acceptance of new technology, or rather information systems, in organisations. The perceived usefulness of a system and the perceived ease of use both inﬂuence the attitude towards the system, and hence the user behaviour when interacting with the system, as well as the actual use of the system (see Figure 2.5).

Figure 2.5: The Technology Acceptance Model after Davis (1989).

( 51 ) 36 Chapter 2. Frame of reference

If the perceived usefulness of a system is considered high, the users can accept a system that is perceived as less easy to use, than if the system is not perceived as useful. For AR systems this means that even though the system may be awkward or bulky, if the applications are good, i.e. useful, enough, the users will accept it. Equally, if the AR system is not perceived useful, the AR system will not be used, even though it may be light-weight, non-intrusive and very easy to use.

Davis (1989) defines the term "perceived usefulness" as "the degree to which a person believes that using a particular system would enhance his or her job performance". On the other hand, "perceived ease of use" is defined as "the degree to which a person believes that using a particular system would be free of effort". Usability testing methods often emphasizes the ease of use rather than the usefulness of a product. But according to Davis (1993) perceived usefulness is actually more important to users than the perceived ease of use, which means that users will put up with a system that is not entirely easy to use or manage as long as it pays off in terms of the goals the system allows them to achieve.

2.4 Systemic approaches to systems development

Traditional HCI is based on a scientific tradition of hypothesis testing to ensure scientific validity of its claims by allowing repeatability of the results of usability studies. However, the repeatability of studies involving human activity has been seriously questioned by among others Simon (1956) as well as in the Soft Systems Methodology paradigm (Checkland and Holwell, 1998). The foundation of soft systems methodology is "learning from intervention in real-world problem situations" (ibid) rather than classical hypothesis testing. According to Checkland (1999) a systems approach is, much like a scientific approach, a meta discipline. Studying something scientifically implies using a certain paradigm of methods that are based on defined laws of nature. Studying something from a systems approach means assuming that the world contains "structured wholes which can maintain their identity under a certain range of conditions and which exhibit certain general

( 52 ) Chapter 2. Frame of reference 37 principles of wholeness" (ibid). This mode of thinking is not new, for example, already Carl von Linné divided plants and creatures in nature in a systemic fashion based on their characteristics and properties in his "Systema naturae" from the 18th century. This is just one of many ways of approaching our surrounding world systematically. Numerous ways of decomposing our surrounding world into system components have been developed, all with their own specific purpose. When it concerns information systems, the information processing approach has been, and is, prevailing. From this perspective, a system (human, animal or machine) is divided into a number of components that propagates or processes data. The approach was developed by, among others, Shannon and Weaver (1949). This idea had profound impact, not the least on the Macy conferences (Dupuy, 2000; Heims, 1991), and its long lasting effects on the theoretical foundation of cybernetics, cognitive science, computer science and design cannot be ignored by anyone working in these fields, although it is often forgotten in contemporary texts.

The most important consequence of the information processing paradigm is probably that it has focused largely on structural descriptions focusing on what something is rather than what it does. Although these two directions in many respects are inseparable, the way of approaching a problem, or even defining what the problem is, differs greatly. In "traditional" HCI, the interface and the user are almost always seen as separate entities that exchange input and output. The focus of design based on this approach has consequently been improving that interaction as much as possible. Little thought is given to the task that is actually going to be performed by the user when using the program/tool beyond the interface. In contrast, a functional perspective would instead begin by asking what the purpose(s) or goal(s) of the system is and what the pre-requisites for achieving that goal(s) are. based on observations of how a user completes a task in which the system or artefact is involved, also have to deal with the so called "envisioned world problem" (Woods and Roth, 1988; Hollnagel and Woods, 2005). The "envisioned world" problem states that even if a good understanding of a task exists, the new design/tool will change the task, rendering the first analysis invalid.

( 53 ) 38 Chapter 2. Frame of reference have become clearer with the advent of new types of computer systems that are not restricted to desktop computer applications. The most prominent system theories which diﬀer from the traditional HCI approaches are activity theory, theories of situated cognition and the distributed cognition framework.

2.4.1 Activity theory

Activity theory has its roots in Soviet cultural (Marxist) psychology as deﬁned and described by foremost Vygotsky and later Leontiev (Vygotsky, 1978; Kaptelinin and Nardi, 2006). Prior to the advent of activity theory human behaviour had been explained primarily by behaviourist, psychoanalytical and cognitive developmental psychology. Activity theory instead models human behaviour on what can be observed of human action and the motives and conditions for this action (Engeström et al., 1999). In this sense the theory is used to explain the use of artefacts and how people both shape, and are shaped by, their cultural context through the use of diﬀerent artefacts (Kaptelinin and Nardi, 2006). Artefacts can be of both physical nature, such as books and telephones, and of a more abstract meaning, such as symbols and rules. The general aim of activity theory is to explain human behaviour terms of activity which are described and modelled hierarchically (Kuutti, 1996). The activity theory describes activity in relation to three levels, of which the bottom level is the operational level where routine activities belong, such as type writing. The next level includes more conscious actions, such as the activity of determining what to type, and on the top level is the activity, which is the overall goal of the action - perhaps writing a thesis, see Figure 2.6.

When analyzing human behaviour, activities can be identiﬁed by the motives behind them, and actions by their goals and on the lowest level operations by the conditions necessary to reach the goals (Engeström, 1999; Engeström et al., 1999). Activity is not an isolated phenomena and the context in which it takes place should also be considered in the analysis, an approach shared by other perspectives as can be seen below.

( 54 ) Chapter 2. Frame of reference 39

Figure 2.6: The Activity Theory model (after Engeström (1999); Kuutti (1996).

2.4.2 Situated cognition and distributed cognition

Ways of using knowledge or assumptions of human cognition was introduced by for example Suchman (1987) (situated cognition) and Hutchins (1990, 1995b) (distributed cognition). In these deﬁnitions of human cognition the context in which the cognition takes place has an important role. The human is not an isolated creature but is always a part of an environment, and the interaction between the human, the environment and the artefacts s/he is using is equally important for the experience of usability, see Figure 2.7. The human is thus part of a system where cognition is not isolated in the mind but takes place throughout the system, in the form of both internal and external cognition (Rogers, 2006). This is also known as "distributed cognition" (Hutchins, 1995b; Norman, 1998; Hollan et al., 2000). Key concepts within the distributed cognition framework are the "cognitive system" and the "communicative pathways", which are the way in which people communicate through the system i.e. phones, email etc; and "propagation of representational states" which refers to how information is transformed across diﬀerent media such as maps, paper, human memory etcetera (Hutchins, 1995b).

Investigating a distributed cognitive system requires extensive ethnographic studies that identify the distributed problem-solving tasks, the communication networks, the verbal and non-verbal information transmissions and how knowledge

( 55 ) 40 Chapter 2. Frame of reference

Figure 2.7: The traditional view of internal cognition in comparison with the distributed view of cognition. is shared and accessed (Rogers, 2006). This approach has been used in many diﬀerent studies attempting to explain complex collaborative behaviour. One of the most referred to studies is that of Hutchins (1995b) in where he studied the navigation of a ship. In this study Hutchins focused on the communication and how information ﬂowed through the cognitive system. Other studies include analyses of communication and collaborative work in cockpits (Hutchins, 1995a) and underground control rooms (Garbis, 2002).

2.4.3 Cognitive systems engineering and the use of artefacts

Another theory questioning the traditional view of the human as a processing unit is Cognitive Systems Engineering, CSE (Hollnagel and Woods, 1983, 2005). The foundations of CSE can be found in ideas described in the field of cybernetics (Wiener, 1948; Ashby, 1956) rather than the activity theory framework. Cybernetics is the origin of control theory and aims at studying and identifying central aspects and characteristics of systems in order to find ways of improvement. The central issue of interest in cybernetics is control and communication in the animal and in the machine (Wiener, 1948). In approaching complex socio-technical systems, the basis of CSE does not differ much from the soft systems methodology. CSE is a systemic approach to control and variability, where a central unit of analysis is Joint Cognitive Systems, JCS, which are systems that "can modify its behaviour on the basis of experience so as to achieve specific anti-entropic ends", page 22 in Hollnagel and Woods (2005). In more general terms this refers to the

( 56 ) Chapter 2. Frame of reference 41 thermodynamic deﬁnition, meaning that all systems strive to minimise chaos while trying to reach balance, or homeostasis.

Hollnagel and Woods (1983, 2005) suggested that a "cognitive system" should be described in terms of functions rather than hypothetical structures of information processing. By approaching the study of humans and machines from this perspective, CSE avoids the basic problems of detailed modelling of internal structures and can focus on the performance of a controlling system in its context:

"Instead of viewing an MMS as decomposable by mechanistic principles, CSE introduces the concept of a cognitive system: an adaptive system which functions using knowledge about itself and the environment in the planning and modiﬁcation of actions." (page 583 in Hollnagel and Woods (1983))

The foundation of the CSE approach is thus functional decomposition, using the cybernetic concept of requisite variety (Ashby, 1956). By "requisite variety" Ashby refers to the ability of a system to exercise control over another system. According to him, a system can only control another system if it can present a higher degree of variety than the system that it tries to control (only variety can destroy variety). Another way of explaining the concept is that the "repertoire of performance" that a system can present explains if he/she/it can control a situation. Constraints are also put on the system by the environment, and everything that is beyond the control of the system, but puts constraints on it, is considered to be a part of the environment. The boundaries of a system can thus be deﬁned from the components ability to exercise control over another component, and vice versa from its inability to do so (Hollnagel and Woods, 2005). Hollnagel and Woods uses the example of traﬃc to illustrate the JCS concept, see Figure 2.8 .

A car and its driver are considered as a JCS. The driver-vehicle system can move in itself, but its behaviour is constrained by the road (if we assume that it is a normal two-wheel drive with a "sensible" driver). The road, and the traffic on it, is in turn a part of the traffic infrastructure, which is constrained by the topography in which it is located. Lastly, the weather will affect all of these parts and is also

( 57 ) 42 Chapter 2. Frame of reference

Figure 2.8: An example of a Joint Cognitive System in traffic Renner and Johansson (2006). completely beyond driver-vehicle control. There are thus a number of analytical levels that the researcher can chose from depending on what the research focus is. The important thing to remember is that any driver-vehicle system under study is part of a larger context that shapes its behaviour. The same conclusion is valid for any user-system configuration. Goals and variability takes place on several different system levels, and will thus affect and be affected by other levels. Where to begin and end the analysis must be judged both by the purpose of the investigation as well as pragmatic aspects.

The traditional definition of cognition as something only human beings possess is questioned in this approach. The idea of cognition as something not confined to the human mind relates to the concept of "cognitive artefacts" (Norman, 1998; Hutchins, 1999). Hutchins (1999) defines cognitive artefacts as "physical objects made by humans for the purpose of aiding, enhancing, or improving cognition". Depending on how an artefact is used it can be considered as a tool or as a prosthesis (Hollnagel and Woods, 2005). A tool is something that enhances the users’ ability to perform a task or solve problems. Prostheses are artefacts that take over an already existing function. A wheelchair can, for someone with no legs, be seen as prosthesis or a substitute for the lost legs. It can also be considered a

( 58 ) Chapter 2. Frame of reference 43 tool for getting around faster in the world than by for example crawling or using your arms, which would be the alternative for someone without legs. Yet another example is the computer, which is a very general tool for expanding or enhancing the human capabilities of computation and calculation, a tool for memory support and problem solving. But the computer can also be used to not only enhance these human capabilities but also to replace them when needed. A computer used for automating the locks of the university buildings after a certain time at night has replaced the human eﬀort of keeping track of time and at the appropriate time going around locking the doors.

As noted in the beginning of this chapter, other approaches to usability and HCI often define the systems involved in a human computer interaction situation by the structure of the system and not as in the CSE approach, by its function; "A joint cognitive system is defined by its function (i.e. what it does) rather than by its structure (i.e. what it is)" (Hollnagel and Woods, 2005). In the JCS approach it is important to see the system as a whole and not isolate the parts from each other. The JCS can be comprised of a human and a computer or a human, a computer, a telephone and another human and so forth, where the human brings in the "natural cognition" to the system and artefacts or technological systems may have "artificial cognition". Woods and Hollnagel (2005) define a set of core values in cognitive systems engineering; authenticity, abstraction, discovery and participation. Authenticity is linked to the method of observing work in context, and based on these observations theories and conclusions should be supported by authentic samples of the system at work. Abstraction is the step between the observed behaviour and the patterns that can be identified in the systems variability. "The end of observation and abstraction is discovery" (Woods and Hollnagel, 2005), page 6.

Instead of focusing on overcoming limits of the human-machine system, as is the case in traditional HCI theory, CSE focuses on the abilities of the system to support, adapt and maintain control (Woods and Hollnagel, 2005). The focus of analysis is the ability to handle complexity and variability rather than the issue of human versus technology.

( 59 ) 44 Chapter 2. Frame of reference

Another trait of a joint cognitive system is that it is not predictable, for instance a computer display may be very ambiguous to a user and it is not always possible to predict what interpretation the user will make of a button or control. A fundamental aspect of the joint cognitive systems idea is that is based on control and goal driven behaviour, and how the joint system works at maintaining control. The purpose of maintaining control is in many regards a necessity to reach the goals of the system. One important issue when discussing joint cognitive systems is the system boundaries, and how to determine what parts the system is made of. The boundaries between diﬀerent parts of the joint cognitive system are not set, but are dependent on the level of analysis. As described above, Renner and Johansson (2006) use the example of a car and its driver to describe a joint cognitive system. If the goal is to drive a car safely from point A to B, the driver vehicle system must maintain control in relation to surrounding traﬃc, the road, and weather circumstances, all sources of variability in the system. However the parts that constitute the driver-vehicle system are only the parts that the system can control. Since neither the driver nor the vehicle can control the weather, the weather is not considered to be part of the joint cognitive system (Hollnagel and Woods, 2005).

2.5 Evaluating Augmented Reality systems

A long term goal of AR research is for AR systems to become fully usable and user-friendly, and as noted by Livingston (2005) and Nilsson and Johansson (2006b) there are problems addressing usability issues in MR and AR systems. Research in AR including user studies usually involves quantifiable measures of performance (e.g. task completion time and error rate) and rarely focuses on more qualitative performance measures. Of course there are exceptions such as described by Billinghurst et al. (2003) where users’ performance in collaborative AR applications is evaluated by not only quantitative measures but also with gesture analysis and language use. Another example is the method of domain analysis and task focus with user profiles presented by Livingston et al. (2006). In general there are few examples of MR specific usability criteria and maybe there

( 60 ) Chapter 2. Frame of reference 45 is no need for such criteria – the usability should instead be based on the task and goals of the user in the actual context of use, and not based on the technicalities of the system.

Augmented Reality (AR) is a technology that aims at merging the real and the virtual world, and thereby enhancing, or augmenting, the user’s perception of the surrounding environment in varying aspects, as the examples in chapter 2.1 illustrate. As mentioned previously, there is a wide range of AR applications in many different domains. AR can be used in many different ways, not only to increase productivity in an assembly process (although there may be such effects), but also for the fun of it. However there is a lack of user focus in the development and evaluation of AR systems and applications today. Current methods within the field of human-computer interaction (HCI) are largely based on findings from cognitive and perceptual theories, focusing on performance and quantitative usability measures. An AR-system, or any other novel mode of interaction is likely to receive poor results in an evaluation based on traditional HCI theories and methods. Therefore, other ways of evaluating and measuring are more relevant. For example, user experience in terms of enjoyment is a large part of user acceptance of a product, deciding if it actually is going to be used or not. One important motivation for this thesis is to describe how new technologies such as AR can be evaluated from a holistic perspective, focusing on the subjective user experience of the system.

Billinghurst (2008) mentions that there are in general three categories of user studies of AR systems; perception studies, interaction studies and collaboration studies. The perceptional studies focus primarily of how users react to and perceive virtual information that is overlaid on the real world. The interaction focused studies aim at describing how users interact with the virtual information and how real world objects can be used for interaction. The collaboration studies on the other hand are mainly interested in how AR can be used to enhance collaboration, either face to face or remote (Billinghurst, 2008). The methods used to study AR systems described in the existing literature have mainly been based on usability methods used for graphical user interfaces, sometimes in combination

( 61 ) 46 Chapter 2. Frame of reference with usability for VR applications (Bowman et al., 2002; Träskbäck et al., 2003; Dünser et al., 2008). This approach has some complications since it is not based on the experiences from actual AR systems users in actual contexts (Nilsson and Johansson, 2006a, 2007). Usability criteria and heuristics that are considered to be useful for designing new AR systems tend to be general, broad criteria, such as the ones Nielsen presented in his list of usability heuristics in 1993 (Nilsson and Johansson, 2006a; Dünser et al., 2008). However, there are few examples of studies on how users actually perceive the system in actual use situations. During the user studies in Nilsson and Johansson (2006b, 2007), users were asked about their experience of the AR system, and none of them mentioned desktop or laptop computers, or programs when describing what they were interacting through or with. Instead, words like robot, video game and instructor were used to describe the interaction. The AR system was thus perceived as introducing other properties to interaction than "normal" desktop applications. This could hardly be attributed to the content of the interaction (which mainly was simple instructions of operation), but rather to the fact that the content was presented directly in the context of use. This of course raised questions of how useful it really is to base design of AR systems on desktop computer metaphors and usability criteria.

2.5.1 Viewing Mixed Reality systems as Joint Cognitive Systems

The focus in many deﬁnitions of Mixed and Augmented Reality is the "merging of worlds" but MR is also deﬁned in terms of the technical hardware components and software solutions used to accomplish this merging of real and virtual worlds. There is also a focus on the interaction between the system and the user in MR research (Mansoux et al., 2004). This type of separation between the human user and the technical MR system illustrates a relationship with the traditional human information processing theories of usability.

As mentioned previously, a joint cognitive system is deﬁned as a system which has the ability to change its behaviour in order to reach the purpose of the

( 62 ) Chapter 2. Frame of reference 47 system (Hollnagel and Woods, 2005). A joint cognitive system usually consists of one or more human beings and one or more technical artefacts. The system is not defined as cognitive only because there is a human involved, rather it is defined based upon the functions it has - cognitive functions in a system makes it a cognitive system. For instance, memory and problem solving are a cognitive functions and since computers have memory (not necessarily the same type of memory as humans) and the ability to solve problems (albeit according to pre-programmed rules), the computer can be seen as a technical system with cognitive abilities, hence a cognitive system. An AR system (which usually includes a computer) uses knowledge of the world to position virtual information; it manipulates symbols and changes the information shown to the user in order to reach the goals set by the user. From this perspective it would be appropriate to define the AR system as an artefact (either a tool or prosthesis) and a human user and the AR system as a joint cognitive system, see Figure 2.9.

Figure 2.9: Examples of diﬀerent levels of boundaries in a Joint Cognitive System (after Hollnagel and Woods 2005).

Figure 2.9 illustrates how a joint cognitive system can be defined on different levels based on where the boundaries are drawn. On one level of analysis, where the focus lies on how the software is integrated into the hardware for instance, only the technical system is part of the analysis. In user testing situations where the focus lies on finding flaws in the technical system, the user and AR system is the level of analysis and constitute a joint cognitive system. Other aspects, such as the social context are normally excluded in evaluations of technical and physical

( 63 ) 48 Chapter 2. Frame of reference attributes of the AR system. In contextual user studies the users are a vital part of the analysis, but they are not only inﬂuenced by the AR system; their behaviour is also motivated by the physical surrounding as well as the social context. If the level of analysis is organizational the joint cognitive system also includes the rules and regulations that govern their everyday work, which in turn aﬀects the use of the AR system.

2.5.2 Choosing a perspective

When approaching usability issues related to a specific system there is a need to define what theories are relevant and appropriate. In HCI there is a clear emphasis on computer based systems as the future scenario but, as Landauer (1997) comments, there is no certainty whatsoever that the computer based systems of the future will be similar to the computer based systems of the past, and there will most likely be new ways of interacting that do not resemble the traditional keyboard and mouse. Voice interaction, haptic interaction and touch screens are just a few of the interaction techniques readily available today. This has changed the way people interact and work with computers and this should also be reflected in the way that interaction is studied and analysed. The theories and methods presented here are far from all theories and methods used within HCI research today, but they provide a background to the studies conducted within this thesis. HCI theories that focus too much on internal structures of cognition will not give dependable information on how that system or interface will be received by real end users. A focus on the context, situation and goal of the interaction is needed to complete the user-system picture. This is where systemic approaches such as the cognitive systems engineering approach, distributed cognition and activity theory come in. They add a contextual dimension to usability research which may aid and deepen the analysis of usability in the MR/AR system studied in this thesis.

A holistic or systemic approach to usability implies a view on reality where reality is made up of a set of components, or parts, that are interdependent on each

( 64 ) Chapter 2. Frame of reference 49 other. The parts cannot simply be summed up; the whole is more than just the sum of the parts. In the case of the AR system, the system is more than only technical components, and the boundaries of the joint cognitive system have to be defined and explained in relation to the specific situation and task, and not only in respect to the physical boundaries of the technical components. Usability in AR systems should be measured and evaluated in the same way, with consideration to the specific situation and user, and the usefulness of the system may be more important to measure than the ease of use.

( 65 ) ( 66 ) Chapter 3

Research approach and methods

To investigate the user experience of Mixed/Augmented Reality (MR/AR), four user studies have been conducted, all based on the foundation of the cognitive systems engineering (CSE) perspective which advocates real users solving real problems in real world contexts (Hollnagel and Woods, 1983, 2005; Woods and Roth, 1988). There are some major diﬀerences between qualitative and quantitative research approaches, where the purpose of quantitative research is generalisation, prediction and causal explanations, while qualitative research aims at contextualising, interpreting and understanding the perspectives of the actors. While some researchers in social science view these approaches as incompatible (c.f. Lincoln and Guba (1985)) there are others who claim that quantitative and qualitative research approaches can be combined (c.f. Patton (1990); Crotty (1998); O’Leary (2004)). The research in this thesis is an example of a combination of the approaches where qualitative aspects, such as contextualisation, search for patterns and descriptives, are used alongside more quantitative data of end user studies and experimentation. The main focus of the analysis has been on subjective user experiences working in their regular environment, rather than objective performance measurements collected in laboratory settings. In this sense the method of the studies is inﬂuenced by the idea of naturalistic studies, striving for ecological validity of the results (Farrington-Darby et al., 2006).

( 67 ) 52 Chapter 3. Research approach and methods

The AR applications developed during the process of this thesis work, were of two different types. The first three user studies (presented in Chapter 4) address single user applications for instructional tasks, and the final application (presented in Chapter 5) is of a more complex, collaborative multi-user application for a dynamic command and control task. These different types of applications demand different things both in terms of development process and evaluation. The first two applications were instructional tasks in the health care domain. Technical devices have been an integral part of modern medicine for a long time, whether dealing with simple thermometers to measure body temperature or complex surgical equipment. The way hospital staff and personnel receive instructions on how to use these technological devices differ between regions, departments and individuals but in all cases some kind of instructions must be given the first time personnel need to operate the equipment. In many cases there is a shortage of staff and instructions to beginners is not always prioritised despite the fact that a personal instruction with trained and experienced personnel would be beneficial. One way to approach the issue of instructions is by letting new personnel watch instructional videos, or read manuals in order to understand how to use new equipment. However there are of course drawbacks with these approaches, since there is little flexibility in these methods. One way to present instructions in a more interactive, and perhaps intuitive, way is to use an AR system, where the instructions can be shown in real time in the real context of the task (sharing some qualities with the instructions from a co-worker). In the first two studies of the thesis the participants were instructed, through the AR system, how to perform specific tasks. Using AR to give a user instructions on how to perform a task or assemble an object has been done before which can be seen in studies like the one presented by Tang et al. (2003) and in the ARVIKA project (Friedrich, 2004) presented in the previous chapter.

The multi-user collaborative application used a slightly different approach, taking advantage from both the experiences of the previous single user applications and from the field of computer supported cooperative work. The end user task of this application is different from the previous instructional tasks for the health

( 68 ) Chapter 3. Research approach and methods 53 care personnel. The focus of the multi-user application is on collaboration rather than individual completion of tasks. In the study the AR system was used to facilitate establishing a common ground between participants from three diﬀerent civil service organisations, working jointly in a real time operation.

3.1 Iterative design process

The development of the AR applications used in the studies, was iterative and involved end users throughout the entire design process. An iterative design model based on CSE was used as illustrated in Figure 3.1.

Figure 3.1: A general schematic view of the design and development process used in the included studies.

Application and users. The most basic assumption made in this model is that the application needs to support the users in realistic tasks. This means that all symbols and information presented to the user of the system must be in adjusted to ﬁt the intended user group. This is part of the constraints and functional requirements that need to be identiﬁed in the beginning of the process. The development of a product is always limited to the technological resources available, i.e the technological options in Figure 3.1. In this project the technical system used for the applications was an Augmented Reality system with

( 69 ) 54 Chapter 3. Research approach and methods a video see-through solution. The procedures, or matching rules, are what informs the developer on how to combine the technology available, or chosen, with the requirements of the user group and context. In the case of the studies included in this thesis this part was done through discussions with representatives from the user groups. This, in combination with previous research in the area was the basis for the next step, where the design is speciﬁed in greater detail, i.e the design speciﬁcations in Figure 3.1.

The design is of course not only based on the chosen requirements but also on an underlying concept or idea about design in the project, i.e the design basis as seen in Figure 3.1. As mentioned previously the general concepts in this thesis are strongly inﬂuenced by the cognitive systems engineering approach in where the main principle is to focus on the goals of the joint user and AR system. For example in the singe user studies the main goals of user-system is to prepare medical equipment for operation. In the case of the multi-user study the main goal is to facilitate the creation of a common situational picture.

The design specification stage is part of the iteration cycle, where in the case of the multi-user study presented in chapter 5, the results from the first user focus group and follow-up communication was used to define symbols, objects and interaction solutions. In the single user studies this part of the iteration were the discussions with our user representative who guided the content of the instructions. In the second iteration the results from the first evaluation session was used to define new design specifications.

The next phase of the process was the actual prototype building, and these iteration phases are also described in full detail in chapters 4 and 5. After the ﬁnal design iteration the Augmented Reality System was implemented and evaluated through a full scale user study.

The user task in the single user applications were iteratively designed and the instructions received through the AR system in the ﬁrst three studies were developed in cooperation with an experienced operating room nurse at the hospital in a pre-study. The instructions were given as statements and questions that had

( 70 ) Chapter 3. Research approach and methods 55 to be conﬁrmed or denied via the input device, in this case a numeric keypad with only three active buttons – "yes", "no", and "go to next step".

The multi-user study, included a pre-design phase where field experts from the three different organisations (the fire and rescue services, the police department and the helicopter platoon in the local area) took part in a brainstorming session. This session was used to establish the parameters of the AR system based on what the problematic issues in this type of operational work is and what technological tools could be of use, and how technology could solve some of the problematic issues. Apart from the theoretical problem formulation the brainstorming session was also used to define the components of the software interface, such as what type of symbols to use, and what type of information is important and relevant in the task of creating common ground between the three participating organisations. Based on an analysis of the brainstorming session a first design was implemented. This design was evaluated using a scenario in which the participants, one from each of the three organisations, had to interact and work together as they would in a real situation. The exercise was observed and the participants also answered questionnaires pertaining to the AR system design, and finally a focus group discussion was held. As a result of this design evaluation the system was redesigned and this redesign was later evaluated through a second study consisting of a focus group discussion and observation of the users interacting with the redesigned system while performing simple tasks. The final system design was then later used in a user study in which the methodology was inspired by other related fields, such as command and control research. In line with the naturalistic approach a realistic scenario was created and simulated using a micro-world normally used to train for, and evaluate collaborative work (Johansson et al., 2007; Trnka et al., 2006; Lindgren and Smith, 2006; Trnka et al., 2008). The study was also partly based on a qualitative observational method that focuses on user feedback and naturalistic settings (Nilsson and Johansson, 2006b, 2008).

( 71 ) 56 Chapter 3. Research approach and methods

3.2 End user studies and evaluations

In theory, user studies are common practise in product development, and preferably the usability studies should be included as early as possible in the design development. In this thesis a basic AR system was already developed, so the main focus was to adapt it and create realistic end user applications.

3.2.1 Evaluation of systems and products

Scriven (1991, 1997) describes two main types of evaluation; formative and summative. Formative evaluation has the purpose of affecting the object of evaluation and is often conducted during development of a program or product. Summative evaluation is where the evaluation does not affect the object directly, but is conducted after the product or program is completed. Taking an example from usability studies of technical systems, a formative evaluation can be part of an iterative design process, where the results of the usability evaluation lead to redesign. Summative evaluation is the assessment done when the product has reached the end users. The users will have opinions and ideas about the usability of the specific product they are using but it does not affect the implementation of the product, at least not until the critique reaches the designers and is incorporated into the design process of the next generation or version of the product.

The role of the evaluator diﬀers depending on the character of the evaluation; in a summative evaluation the evaluator is expected to give "objective" input on whether the program or product development should continue, change direction or be terminated. In a formative evaluation on the other hand, the evaluator has to work closely with the people responsible for the program or product development, but still be able to relate to the program or product and the people representing it (Jerkedal, 2005). Cronholm and Goldkuhl (2003) describe how information systems can be evaluated both as systems in themselves, without regard to the user, and as systems in use, using three diﬀerent strategies of evaluation; goal- based evaluation, goal-free evaluation and criteria-based evaluation. Goal-based

( 72 ) Chapter 3. Research approach and methods 57 evaluation is driven by explicit goals, goal free evaluation does not use any specific goals, but instead aims at a deeper understanding where the evaluator is not influenced and directed by predefined goals. Criteria-based evaluation uses a defined set of criteria such as check-lists or heuristics, which are general and not as specific as goals.

The aim of a goal driven evaluation should be that the results of the evaluation are utilised; otherwise there is no point in doing the evaluation in the first place (Jerkedal, 2005). A utilisation focused evaluation is basically used as a tool for finding possible ways to improve the object of the evaluation. The focus is on the users and the intended use of the product, program or other object under evaluation. This is a pragmatic approach to evaluation, where the emphasis is on the goal and purpose of the evaluation. The results of the evaluation should be directly usable to the intended users of the product in its intended environment, and not general, non-specific results aimed at potential users in potential situations (Patton, 1997). Utilisation focused evaluation can be both summative and formative, however Patton (1997) argues that the users (which can be the end users of a product, but perhaps more likely, the developers of the product) must be involved in the evaluation process, and understand it, in order to use the results of it. This indicates that formative utilisation focused evaluation may be more useful than summative for the purpose of evaluating usability during product development. Cousins and Earl (1992) discuss participatory evaluation as an extension of utilisation focused evaluation and participatory evaluation has, as noted before, been a vital part of usability focused design for some time.

The evaluations conducted in this thesis are of several types, as they are part of an ongoing process during development, so they are both formative and summative in that the results of a ﬁrst design iteration inform a redesign of the system. In general the type of evaluation used is the utilisation focused evaluation, where the end users both are part of deﬁning and designing the application as well of the evaluation of the application (although not necessarily the same individual users in both parts of the process).

( 73 ) 58 Chapter 3. Research approach and methods

3.2.2 Simulation and scenario for the collaborative Augmented Reality application

While the single user studies in this thesis were focused on a none to complex, sequence of instructions, the multi-user study included a more complex set of tasks and addressed a diﬀerent type of action. It was conducted with the purpose of evaluating the AR system and application design in order to improve it as a tool for collaboration between organisations. In the single user studies it was rather easy to conduct the studies in near real life situations, as the real life situations where the user receives instructions is a fairly neutral and quiet setting (any operating room, or preparation room in the hospital). Receiving instructions on how to perform a task in the hospital is usually not done in the midst of any time critical and intense events, but rather done in between other tasks. This made it easy to conduct the study in a setting where the users would normally perform such a task.

In the fourth study, where the domain was collaborative command and control involving different organisations working in a more complex and demanding context the situation was very different. For obvious reasons, it was not possible to conduct experiments in real fire fighting situations. Instead, to create a realistic study, we used a scenario where the groups had to collaborate, distribute resources and plan actions in response to a simulated situation. In the scenario the participants act as they would as on-scene commanders in their respective organisation. The AR system was used as a tool for them to see and manipulate their resources and as a way to have an overall view of the situation.

Scenarios are descriptions of reality, where the aim is to recreate a natural context for participants in a study, in where they recognise both the content and events occurring (Klein, 1985). Creating a scenario is done by identifying and deﬁning the actions, events and general content of a situation that is to be simulated. In the fourth study this process was done in the pre-design phase, where discussions with representatives from both the end more operative end user group and other administrative parts of the organisation took part.

( 74 ) Chapter 3. Research approach and methods 59

3.3 Data collection methods used in the studies

One of the objectives of the studies in this thesis is to apply a systemic, holistic approach to development and evaluation of user experience in relation to AR use in the speciﬁc context of the target user groups. In order to get reliable data that describes the user experience of the AR applications developed within the cope of this thesis, data was collected by means of observations, interviews and questionnaires. In developing the applications and tasks it was important that the participants in the studies were prospective real end users of the applications.

As previously mentioned, the area chosen for the first three applications was technical support in a medical context, more precisely - the AR application aimed at providing an alternative to costly training and instruction procedures that take place when new staff is introduced to technical devices used during surgery. In order to achieve realism in the task and the instructions, the applications were developed in close cooperation with a nurse who could give insights into what specific aspects of the instructions were important. The second application area for the AR system, as described in the fourth study, was as technical support to facilitate collaboration between personnel from different organisations. This application was developed using an iterative design approach involving real end users throughout the entire process as described previously. As noted above the first three studies were conducted on site at a hospital, and the fourth study in a simulated command and control room environment at a helicopter base. The studies had an explorative approach, and as such the data was collected mainly through questionnaires, short interviews, observations and video recordings. The qualitative data was analysed by identifying themes of relevance to the user experience, and the quantitative data was explored and described through statistical analysis. Certain issues were targeted in the interviews and questionnaires (as can be seen in the appendices), and some of the responses from the participants were relatively different in content as well as coverage.

( 75 ) 60 Chapter 3. Research approach and methods

3.3.1 Questionnaires

Questionnaires are a form of query research in which the main idea is that in order to find out what people, or users, think about a subject or interface - simply ask them. The interview may be a more flexible query technique, but the questionnaire is somewhat easier to administer and analyse (Dix et al., 2004; Karat, 1988). There are different types of questions that can be included in a questionnaire; general demographic questions about the respondent; open-ended questions which leave room for the respondents own words and opinions and; scalar questions where a user is asked to rate a specific statement on some kind of scale, often numeric such as the Likert scale (Likert, 1932). Other types of questionnaires include multiple choice questions and questions where the respondents are asked to rank options in a list. A general problem with questionnaires is that the questions are inflexible, and that the phrasing of the question is very important since the administrator of the questionnaire usually is not available for clarifications should there be a need for this during the form filling (Dix et al., 2004).

Another issue that can affect the result of a questionnaire is the way questions and statements are presented and formulated. One bias in a questionnaire, as well as in an interview, can occur if the questions or statements imply an expected response. This can be done by for instance asking negative or positive questions or posing negative or positive statements. For instance a statement can be phrased "this system was easy to use" or "this system was difficult to use" as compared to "this system was not easy to use" or "this system was not difficult to use". By randomising the way the questions are formulated (as negative or positive statements or questions) this risk may be reduced. However it will still be difficult, or even impossible to determine if the responses given are true due to a range of error sources (Strickler, 1999). The first issue is of course if the respondent answers truthfully - perhaps the respondent actually does not know how to answer the question but still answers, or perhaps the response does not reflect the true feelings or thinking of the respondent. Even though the respondent answers truthfully, there is always the issue of how the question or statement has been interpreted

( 76 ) Chapter 3. Research approach and methods 61 by the respondent. Usually this problem can be addressed by validating the questionnaire before distributing it. That is, making sure that the question is phrased in a way so that several diﬀerent people interpret the question in the same manner.

3.3.2 Interviews

Interviews can be conducted for various reasons and have very different approaches. The four main categories of interviews are unstructured (open-ended), structured, semi-structured and group interviews, often referred to as focus groups (Patton, 2002). In unstructured interviews the interviewer asks open questions which allows the respondent to elaborate in any direction. This method results in a lot of data, which can be very time consuming to analyse. Another problem with unstructured interviews is that they are impossible to replicate - they are much like conversations, where one discussion thread leads to the next. In comparison to the unstructured interviews, structured interviews are easier to replicate in that the questions are set beforehand, and the respondent is often given a set of alternatives to chose from as answers, much like the questions in a questionnaire. These types of interviews are useful when there is a specified purpose with the interview, which is often the case in user tests aiming to find flaws in an interface for example. The semi-structured interview on the other hand will not have answer alternatives for the respondent to choose from. Instead the questions are predetermined but the respondent is left to answer freely, like in an unstructured interview. The group interview is what is usually referred to as a focus group and these are similar to open discussions on a predefined set of topics, but they can also be more structured. The general purpose of a focus group is to allow people to relate to subjects in a social setting. Being surrounded by other people may help the focus group participants express their opinion in a more relaxed situation than in a one-to-one interview situation (Kvale, 1997; Morgan, 1998). However there is also the opposite effect; that the respondents are unwilling to share their thoughts with strangers (Rubin and Rubin, 1995).

( 77 ) 62 Chapter 3. Research approach and methods

One of the most crucial aspects of using interviews for data collection is the choice of questions, and also how they are composed, so that they do not influence the response in any way. The interviewer must also be cautious of how the question is asked, as this can affect how the question is interpreted. Analysing interview data can be problematic in terms of time and bias. A common method of interview data analysis includes transcribing, re-reading, and categorising (Rubin and Rubin, 1995; Miles and Huberman, 1994; Kvale, 1997; Silverman, 1993). For semi-structured interviews the themes of the answers are usually defined by the questions asked, which simplifies the analysis.

3.3.3 Observations

In usability focused research, using observations means watching and listening to users perform tasks or interact with interfaces, either in their natural setting, i.e. naturalistic observations, or in a more controlled environment such as a laboratory (Goodwin, 1995; Dewalt and Dewalt, 2002). Observations can be documented either through video or voice recordings or through note taking (Hackos and Redish, 1998). Observational methods emerged from the field of ethnography where the main goal of the observation is to describe a culture or a group of people (Dewalt and Dewalt, 2002; Fetterman, 2008). Ethnographic field studies are normally conducted over a long period of time, sometimes as long as up to several years. In these field studies the researcher completely immerses in the studied culture to get complete insight (Agar, 1996). Observation as a method was also an important part of early psychological studies of people when the field of psychology took the step from introspection and philosophy towards more scientific approaches. Observations of people in laboratory settings, with controlled environments, were used to study behaviour (Wilkinson, 2000). The problem with these studies were of course that people placed in laboratory settings and controlled environments often do not display the same behaviours as they would in their natural surroundings (ibid). Taking a different approach the field of naturalistic observation developed, where the goal was to observe people in their natural

( 78 ) Chapter 3. Research approach and methods 63 environments rather than in artiﬁcial surroundings as "social work should be studied in it’s natural state" (Hammersly and Atkinson, 1995).

When studying people as users of technology observations can be used but are usually not of the length and depth that ethnography demands. Hutchins (1995b), however, used a long term participant observation during his studies of ship navigation. This type of observation clearly relies more on the ethnographic approach than the psychological laboratory approach to observation. There are several issues of concern when using observation as a tool for data collection. One issue is that of the external focus, the observer sees only what happens externally, not internally in people. Another constraint is that there is a selection process of what events are actually being observed (Patton, 2002). The observer’s role is very significant, since it is this person who will choose what to observe and how to do it. Problems can arise in that the observer is not an objective and neutral part, but brings a perspective on what is observed. What is observed can be just as much of a reflection on the observer and his/her theoretical and cultural expectations as a reflection on what is actually going on (Kvale, 1989). Observational data can be a main data source in some research, like social or psychological studies of human interactions and conversations for instance (see for instance Heath and Hindmarsh (2002), or the data can be used mainly as a way to support integration of social and organisational elements into an analysis (see for instance Farrington-Darby and Wilson (2009)). The observations used in this thesis are however of a more superficial nature, where the users were filmed while completing their tasks, or while collaborating. In analysing the observational data the aim was to focus on specific issues of how the participants interacted with and through the AR system. The observation was therefore not ethnographic in any sense other than the fact that the users performed the task in their natural environments (or close to natural settings) and not at a research lab.

( 79 ) 64 Chapter 3. Research approach and methods

3.4 Analysis of data in the studies

Both qualitative and quantitative data was collected in the user studies. The quantitative data was explored with descriptive statistics and analysed to ﬁnd correlations and diﬀerences of interest with regard to the research questions. There are many ways of approaching qualitative data, and there is not always a methodological tool to analyse data in the same manner that quantitative data can be approached through very clearly formulated techniques like statistical methods. There are however methodological orientations which the researcher can use as a starting point (Heath and Hindmarsh, 2002). Miles and Huberman (1994) describe an approach based on a classic hermeneutic cycle of data collection, reduction, display and conclusions, see Figure 3.2. This cycle is most commonly used when dealing with qualitative data such as the results from the open responses in the interviews and questionnaires, as well as the observational data gathered during the user studies.

Figure 3.2: The research cycle according to Miles and Huberman (1994).

Data reduction can be done in several steps, ﬁrst in the case of observations and interviews there is the issue of transcribing the data. This process can be time consuming and tedious and often the researcher has to choose a level of transcription suited for the purpose. In studies of conversational patterns or linguistic analyses the transcription must be very detailed. However, in the case of semi-structured interviews for the purpose of product evaluation the details of language use is not the main interest and therefore there is no need to transcribe every pause and utterance. Deciding at what level of detail to

( 80 ) Chapter 3. Research approach and methods 65 transcribe the observations is somewhat similar. To reduce the observational data the researcher can choose to focus on speciﬁc patterns or types of behaviour. Jordan and Henderson (1995) describe interaction analysis of video data, which is an in-depth micro analysis of how people interact. The distributed cognition framework is another approach to video analysis where the researcher focuses on the communication within the studied system (as described in the theoretical background above). For the studies included in this thesis, the distributed cognition reduction method is the one that comes closest to mind (Artman and Waern, 1999). The observation and interview data was studied with focus on the interaction between the participants and the AR system, but most importantly the interaction through the AR system with the actual user task. In the third study attention was also given to the interaction between the participants.

A different approach is that of grounded theory in which the focus of analysis is not predefined, instead the researcher seeks the theory during the analysis process (Strauss and Corbin, 1998). The main idea is to identify and define appropriate categories and then use these as the basis for constructing a theory. The categorisation of data is what is called coding(regardless if it is a grounded theory analysis or other), where the data is tagged according to different themes (Strauss and Corbin, 1998). The analysis method of grounded theory is in this way empirically driven rather than theory driven. The responses in the interviews as well as the open-ended questions in the questionnaires used for the studies in this thesis were coded by tagging and grouping into themes similar to the method of open coding described by Strauss and Corbin (1998) and Glaser and Strauss (1967). Thus the empirical findings informed the categorisation and evaluation of data rather than a hypothesis or underlying theory.

The usability of research ﬁndings can be expressed in terms of validity and reliability, which addresses to what extent the results are possible to replicate and generalise or transfer to other settings. Ensuring validity in a study, or experimental set-up, means making sure that what is actually measured is what was intended to be measured. For instance if the aim of the study is to ﬁnd out if brushing teeth helps prevent cavities, what should be measured is the relationship

( 81 ) 66 Chapter 3. Research approach and methods between teeth brushing and the occurrence of cavities, and not the relationship between intake of food and cavities. However the study must also be conducted, and described, in a manner that allows other researchers to replicate the study, and (hopefully) attain the same results. For most qualitative studies, the likelihood of another researcher conducting the same study coming up with exactly the same results is not very high, as qualitative studies per se are more subjective than quantitative studies. This makes documenting and reporting the methodology used all the more important.

For research results to be interesting in general it is also important that the results are possible to generalise to a larger population than only the subjects involved in the study (external validity). This demand is however diﬃcult to reach when conducting research with few participants, which is often the case in studies based on ethnographic tools, such as interviews and observation. This is not unique to qualitative research; even in larger scale quantitative studies with large samples it is still diﬃcult to draw general conclusions of the results applicability to other groups. Closely related to the concept of external validity is that of ecological validity which deals with the relation between the study and the real-life situation that is being studied. In order to have ecological validity the setting of the study, as well as the methods and artefacts etcetera used, must approximate the real life context (Shadish et al., 2002).

Regarding the validity of the included studies the main issue is that they have been conducted in order to ﬁnd results pertaining to the usability and more importantly, the end user experience of the AR system applications. The evaluations in this thesis have not been conducted in order to ﬁnd general user guidelines for AR system design or evaluation, as noted before. An underlying assumption in this thesis is that generalisations in the shape of user guidelines are not of a huge interest in relation to AR systems, as they will be so general that they are more commonsensical than practically useful. Or as Lincoln and Guba (1985) phrase it: "The trouble with generalisations is that they do not apply to particulars."

( 82 ) Chapter 3. Research approach and methods 67

3.5 Instruments used in the studies

The questionnaires used in this thesis were a combination of the first three types described above; they included general questions to establish user background, and had open-ended questions similar to the interview questions in the first study (see appendices). In the second, third and fourth study the open questions were complemented with a section of closed response questions with scalar responses. However in the case of these studies, the questionnaires were not distributed in a wide group of people, but directly in connection with the actual user study, hence the evaluator was present which offered the opportunity to clarify any concerns about the questions asked. Other issues that must be considered when designing a questionnaire is for instance the choice and format of the scale. In the questionnaires used in these studies the scale was a 6 point Likert scale as opposed to the standard 5 or 7 points. The reason for this is that by using an even numbered scale it is less likely that the responses are completely neutral, i.e. 3 on the 5 or 4 on the 7 point scale. In the case of these user studies the intent was to avoid neutral responses as they would not result in useful information for redesign issues of the evaluated system.

In the studies included in this thesis the users were observed while using the AR system. The goal of the observations was partly to triangulate the responses in the questionnaires and partly to see the users’ behaviour and movements while wearing the AR system and completing the tasks at hand. The observations in the studies were thus not a main data source, however they were an important complement to the more structured, subjective data. In this way the observational material was used to clarify results from the other data sources. During the multi- user study structured observation protocols were used to document any events of interest in relation to the research themes.

For the purpose of the ﬁrst study in this thesis a semi-structured interview technique was used, where the respondents all were asked the same set of questions. The interview questions can be found in appendix I. The interviews were transcribed and the answers collected according to themes found in the material.

( 83 ) 68 Chapter 3. Research approach and methods

In the second study presented in this thesis descriptive statistics were used, where the data was quantiﬁed and presented in tables and ﬁgures with average scores and distribution. The results from the closed response section in the questionnaires of the second and third studies were described and analysed statistically. The closed response questionnaire results have mainly been used to clarify the more qualitative data gathered through the open ended questions and observations.

In the multi-user study the questionnaire data was analysed statistically and the structured group discussions were used to expand on several issues in the questionnaires, allowing the participants to discuss these issues with each other. The main aim behind these focused group discussions was to capture the participants attitudes and experiences of the study and application in an open forum.

In order to create a dynamic scenario and realistic responses and reactions to the participants decisions in the final study, we used a gaming simulator, C3 Fire (Granlund, 2001). C3 Fire is a micro world in where individuals work together as a team to put out computer-simulated fires. Micro worlds in general are artificially created environments used to simulate different aspects of the real world (Brehmer and Dörner, 1993). The C3 Fire micro world used in the fourth study is described more detail in chapter 5.

Table 3.1 gives an overview of the user studies presented in the following chapters.

( 84 ) Chapter 3. Research approach and methods 69

Table 3.1: A summary of the studies presented in the thesis.

( 85 ) ( 86 ) Chapter 4

Single user studies

To investigate user acceptance and attitude toward AR technology in single user context, two user studies were conducted where the participants were observed using an AR system, receiving instructions on how to interact with an electro- surgical generator (ESG) in the ﬁrst study, and in the second study to assemble a relatively small surgical instrument, the troakar.

The basic problem for both applications was how to give instructions on equipment both to new users, and to users who only use the equipment at rare occasions. Normally a trained professional nurse would give these instructions but this kind of person-to-person instruction is time consuming and if there is a way to free up the time of these professional instructors this would be valuable in the health care organisation. The AR applications were therefore accordingly aimed at simulating human personal instructions. It is also important to note that the aim of these studies was not to compare AR instructions with either personal instructions or paper manuals in quantitative measures, not was the focus on speed of task completion or other quantiﬁable values. The focus was instead on user experience and whether or not AR applications such as the ones developed for these studies could be part of the every day technology used at work places like the hospital in the studies. The ﬁrst study of the ESG was exploratory and focused primarily on qualitative data such as the open ended questionnaires and observations. In the

( 87 ) 72 Chapter 4. Single user studies following studies of the troakar application, the data collection was expanded to also include included closed response questions.

4.1 Iterative design of the application tasks

An important aspect of the studies was user involvement throughout the entire process of design, development and evaluation of the applications. Therefore the instructional applications described below were all developed in close cooperation with end user representatives. In both applications, the ESG and the troakar, the ﬁrst step of development was an observation of how instructions are given today in these domains. After observing and recording an experienced nurse as she gave instructions on how to use the medical device, the AR instructions were constructed in a similar structure and sequence. After a ﬁrst version of the AR instructions was implemented in the system, the nurse giving the instructions had the opportunity to give feedback and corrections to the implementation. This input then informed the next iteration of instructions in the application.

The design of the system was iteratively refined throughout the different applications. The results from the first user study with the ESG application informed the redesign of the AR system which was used in the second end user study application. The second user study application focused of instructions on how to assemble a troakar, and these instructions were iteratively developed in the same manner as the ESG instructions. The troakar is considerably smaller and does not give any feedback to the user in terms of lights or blinking numbers like the ESG. Instead the instrument was delivered in parts and needed to be assembled according to the given instructions. The troakar instructions were studied twice with different user groups, the first study included professional end users at a hospital while the second troakar study involved "future professional users", i.e. students in medical school, in training to becoming nurses. The second troakar study also investigated the effect of receiving both visual and audio-based instructions.

( 88 ) Chapter 4. Single user studies 73

4.2 Study 1 - the electro-surgical generator

The aim of the ﬁrst study was to investigate user experience and acceptance of an AR system in an instructional application for an electro-surgical generator (ESG). The ESG is a tool that is used for electrocautery during many types of surgical procedures. In general electrocautery is a physical therapy for deep heating of tissues with a high frequency electrical current. The ESG used in this study is used for mono- or bipolar cutting and coagulating during invasive medical procedures, see Figure 4.1. When using this device it is very important to follow the procedure correctly as failing to do so could injure the patient. Part of the operator’s task is to set the correct values for the current passing through the device, but most important is the preceding check up of the patient before using the tool. For instance, the patient cannot have any piercings or any other metal devices on or in the body, should not be pregnant, and most importantly, the areas around the patient must be dry as water near electrical current can cause burn injuries.

Figure 4.1: An electro-surgical generator with markers for marker tracking.

4.2.1 Methodology

The study took place on site at a regional hospital. After the participants had been introduced to the project they were interviewed about their experience and attitudes both towards new technology and instructions for use. Then they were observed using the AR system, receiving instructions on how to perform their task. After the task was completed they ﬁlled out a questionnaire about the experience. The interviews, as well as the observation were recorded. During the task, the

( 89 ) 74 Chapter 4. Single user studies participants´ view through the video-see-through AR system was also logged with a digital video camera.

4.2.2 Participants

Eight participants, ages between 30 to 60, all employed at the hospital, participated in the study. Four of them had previous experience with the ESG, and four did not. All of the participants had experience with other advanced technology in their daily work.

4.2.3 Equipment

The AR system hardware part, which can be seen in Figure 4.2, included a tablet computer (Fujitsu Siemens Lifebook T with 1GHz Intel¨Pentium¨ M, 0.99GB RAM), a helmet mounted display (Sony Glasstron) with a fire wire camera attached, see Figure 4.2. A numpad was used for the interaction with the user. The AR system used a hybrid tracking technology based on marker tracking; ARToolKit, (available for download at www.hitl.washington.edu/artoolkit), AR- ToolKit Plus (Schmalstieg, 2005) and ARTag (Fiala, 2005). The software includes an integrated set of software tools such as software for camera image capture, fiducial marker detection, computer graphics software and also software developed specifically for AR-application scenarios Gustafsson et al. (2005).

Figure 4.2: The helmet-mounted AR system.

The users were given instructions on how to activate (and prepare for operation) an electro surgical generator, or diathermy apparatus, the ERBE ICC-350a with

( 90 ) Chapter 4. Single user studies 75 associated instruments and electrodes. In general an ESG is used for cauterisation and cutting by deep heating of tissues with a high frequency electrical current. The ERBE ICC 350a can be used for mono- or bipolar cutting and coagulating during invasive medical procedures. The image in ﬁgure 4.1 also shows the placement of the markers.

4.2.4 The user task

The instructions received through the AR system were developed in cooperation with an experienced operating room nurse at the hospital in a pre study. After observing and recording the nurse as she gave instructions on how to use the ESG, the AR instructions were constructed in the same manner. The instructions were given as statements and questions that had to be conﬁrmed or denied via the input device, in this case a num pad with only three active buttons – "yes", "no", and "go to next step". An example of the instructions from the participants’ ﬁeld of view can be seen in Figure 4.3 and a participant using the AR system to complete the task in Figure 4.4 .

Figure 4.3: The participant’s view of the AR instructions.

Data was collected both through observation and open ended response questionnaires. The questionnaire consisted of questions related to overall impression of the AR system, experienced diﬃculties, experienced positive aspects, what they would change in the system and whether it is possible to compare receiving AR instructions to receiving instructions from a teacher.

( 91 ) 76 Chapter 4. Single user studies

Figure 4.4: A participant working with the AR system.

4.2.5 Data analysis

The results of the study were documented by observations of the users, but mainly through interview questions before the task and a questionnaire after the task. The interview questions consisted mainly of demographic questions regarding the participant’s backgrounds and attitudes towards new technology, while the questionnaire targeted their experience of the AR system and task. The results were analysed with a qualitative approach, meaning that the responses were organised around certain recurring themes, and the results are described below.

4.2.6 Results

It was found that all participants but one (out of eight) could solve the task at hand without any other help than by the instructions given in the AR system. In general the interviewed responded that they preferred personal instructions from an experienced user, sometimes in combination with short, written instructions, but also that they appreciated the objective instructions given by the AR system. The problems users reported on related both to the instructions given by the AR system and to the AR technology, such as problems with a bulky helmet etc.

( 92 ) Chapter 4. Single user studies 77

Despite the reported problems, the users were positive towards AR systems as a technology and as a tool for instructions in this setting.

All of the respondents work with computers on a day to day basis and are accustomed to traditional MS Windowsa based graphical user interfaces but they saw no similarities with the MR system. Instead one respondent even compared the experience to having a personal instructor guiding through the steps:

"It would be if as if someone was standing next to me and pointing and then... but it’s easier maybe, at the same time it was just one small step at a time. Not that much at once." (participant 1) 1.

Even though the participants stated that they normally do not mind working with many diﬀerent technological devices on a daily basis occasions that cause problems occurs. For example when rarely using equipment – it is not easy to remember how it works if a long time passes in between use:

"I think that especially on this apparatus there are a lot of details that we get information about and then we forget it because we don’t use it for a while and then you need to be updated if you have to deal with something, otherwise you don’t use the apparatus to its full potential." (participant 3)

Generally, the respondents are satisﬁed with the instructions they have received on how to use technology in their work. One problem with receiving instructions from colleagues and other staﬀ members is however that the instructions are not objective, but more of "this is what I usually do". The only objective, or neutral, instructions available are the manual or technical documentation and reading this is time consuming and often not a priority. When asked about the general impression of the AR system one respondent answered:

"...totally clear, it wasn’t any – this is how I usually do it and it usually works ﬁne. Really clear..." (participant 6)

1All quotes have been are translated from Swedish to English by the author.

( 93 ) 78 Chapter 4. Single user studies

One important feature of the personal instruction situation is the interactivity – it is important to be able to ask questions or return to instructions, or adjust the pace. A majority of the participants stated that they prefer instructions from an experienced user, which allows interactivity.

The participants were also asked about in what situations or for what purpose they think an AR system could be useful, if in any. Two of them mentioned medically invasive procedures such as laparoscopy. Several of the participants responded that AR systems could probably be used in any situation where instructions are needed, such as in education or for guiding users on how to use technical equipment. The video based observation illustrated that the physical appearance of the AR system probably aﬀected the way the participants performed the task. Since the display was mounted on a helmet there were some issues regarding the placement of the display in from of the users’ eyes, so they spent some time adjusting it in the beginning of the trial. However since the system was head mounted it left the hands free for interaction with the ESG and the numpad used for answering the questions when responding to the instructions.

4.2.7 Discussion of the results

Even though receiving instructions from colleagues and coworkers may be the preferred method of introducing new technology, this approach does have a problematic issue as pointed out by one of the respondents above. The issue is that the instructions are not complete or neutral, but more of "this is what I usually do". Although this method is probably eﬀective there is always a risk of misguided instruction, and the only alternatives to this type of personal instruction are the manual or technical documentation. Reading manuals or watching tutorial videos may be a way to ensure that all available information is provided, however the chance of undermanned hospital staﬀ sitting down for a prolonged period of time to watch videos or reading manuals is not very high. One possible way to solve this issue is by using AR technology; the instructions will be the same every

( 94 ) Chapter 4. Single user studies 79 time much like the paper manual, but rather than being simply a paper manual AR is experienced as something more, like a virtual instructor.

As noted above, there were some ergonomic issues with the AR system and these issues most likely affected the experience of both the task and the system. The observations during the study made it clear that although the helmet-mounted display left the hands free for interaction, it would be a considerable gain to interact through some other means than the numpad, as the user sometimes need both hands for the interaction with the ESG. As a result of the study, the AR system has been redesigned to better fit the ergonomic needs of this user group. Among the changes made are a different HMD and a voice input modality which is described in the next study. Changes have also been made to the instructions and the way they are presented which is illustrated by the next study.

4.3 Study 2 - the troakar

The second study is a follow-up of study 1, but with a slightly different application and new participants. The AR system was also equipped with voice interaction capabilities which was a considerable change done as a result of the previous user study of the ESG instructions. In the first study the instructions were aimed at starting up an electrical device, in the following study the application was changed in order to investigate the AR systems potential for use in a different type of instruction. The instrument in focus in this study was a troakar, see Figure 4.5. As in the previous study this application development was inspired by the real tasks in the organisation – this is also an instrument which is often used and needs to be handled with care, and as such more experienced personnel usually instruct new personnel how to put it together and use it.

( 95 ) 80 Chapter 4. Single user studies

Figure 4.5: A troakar as the one used in the study, the insert shows the diﬀerent parts.

4.3.1 Methodology

The participants were ﬁrst introduced to the AR system. When the head mounted display (HMD) and headset was appropriately adjusted they were told to follow the instructions given by the system to assemble the device they had in front of them. After the task was completed the participants ﬁlled out a questionnaire about the experience. The participants were recorded with a digital video camera when they assembled the troakar. During the task, the participants’ view through the video see-through AR system was also logged on digital video. As in study 1, data was collected both through direct observation and through questionnaires.

4.3.2 Participants

Twelve professional (ages between 35 and 60) operating room (OR) nurses and surgeons at a hospital took part in the study. The participants all had some knowledge about the object of assembly (a troakar, see Figure 4.5), although not all of them had actually assembled one prior to this study. A majority of the participants stated that they have an interest in new technology, and that they interact with computers regularly, however a majority of them had very limited experience of video games and 3D graphics.

( 96 ) Chapter 4. Single user studies 81

4.3.3 Equipment

The AR system was upgraded and redesigned after study 1 was completed (see ﬁgure 6). It includes a Sony Glasstron Head Mounted Display (HMD) and an oﬀ the shelf headset with earphones and a microphone. The AR system runs on a Dell Inspiron 9400 laptop with a 2.00 GHz Intel¨Corea 2 CPU, 2.00 GB RAM and a NVIDIA GeForce 7900 graphics card. The AR system uses the same hybrid tracking technology based on marker tracking as described in the previous study. The marker used can be seen in Figure 4.6.

Figure 4.6: The marker used for the troakar study.

One significant difference between the redesigned AR system and the AR system used in study 1 is the use of voice input instead of key pressing. The voice input was given through a standard, off the shelf headset, as can be seen in Figure 4.7. The voice input is received through the headset microphone and is interpreted by a simple voice recognition application based on Microsoft’s Speech API (SAPI). Basic commands are OK, Yes, No, Backward, Forward, and Reset.

The software in the AR system is based on a C++ code, but is structured to allow general use, meaning that applications can be built and implemented from an XML-file. Tags in the XML file determine the actions of the AR system when the camera identifies a fiducial marker. For example, one tag of particular importance for this application is structured to allow several sequential texts, animations

( 97 ) 82 Chapter 4. Single user studies

Figure 4.7: A participant working with the troakar using voice input to the AR system. etcetera to be triggered by the same marker in a predetermined order. This sequence also allows the user to stay in control of the instructions. In the speciﬁc application tested in the study, the participants interacted with the AR system with voice commands, triggering the system to continue to the next instruction.

Observation equipment The user study was recorded with a digital video camera to allow later observation of the participants’ interactions with the object of assembly as well as their spoken interaction with the system. The video see through display activities were also recorded with a digital recorder, and these recordings allow observations of errors during assembly as well as time records of the task.

4.3.4 User task

The object the participants were given instructions on how to assemble was a common medical device, a troakar (see Figure 4.5). A troakar is used as a "gateway" into a patient during minimal invasive surgeries. The troakar is

( 98 ) Chapter 4. Single user studies 83 relatively small and consists of seven separate parts which have to be correctly assembled for it to function properly as a lock preventing blood and gas from leaking out of the patient’s body. The troakar was too small to have several different markers attached to each part. Markers attached to the object (as the ones in the ESG study) would also not be realistic considering the type of object and its usage – it needs to be kept sterile and clean of other materials. Instead the marker was mounted on a small ring with adjustable size which the participants wore on their index finger, see Figure 4.6 and Figure 4.8, where the user is wearing the marker on her left index finger.

Instructions and voice input Instructions on how to put together a troakar are not formalized; instead they are normally given by more experienced operating room nurses. To establish realism the instructions designed for the AR application were based on the instructions given by an OR nurse at a hospital. The OR nurse was interviewed, observed and video recorded while giving instructions and assembling a troakar. The video was the basis for the sequence of instructions and animations given to the participants in the study. An example of the instructions and animation can be seen in Figure 4.8.

Figure 4.8: An example of the instructions as seen by a participant (rough translation - "screw on the smallest cap").

Before receiving the assembly instructions the participants were given a short introduction to the commands they can use during the task; "OK" to continue to the next step, and "back" or "backwards" to repeat previous steps. These

( 99 ) 84 Chapter 4. Single user studies commands are part of the Logic10 sequence described above. The voice input is received through the headset microphone and is interpreted by a simple voice recognition application based on Microsoft’s Speech API (SAPI). Basic commands are OK, Yes, No, Backward, Forward, and Reset. Figure 4.7 shows a participant using the AR system while working with the troakar task.

4.3.5 Data analysis

Observations and questionnaires formed the basis for a qualitative analysis. The questionnaire consisted of a set of closed questions where the participants responded on a 6 point Likert scale. In total the questionnaire included 17 closed questions of which the ﬁrst 3 were demo-graphical in nature (prior technological experience, interest in technology and computer use). The remaining 14 questions focused on the user experience with statements where the user could check a box between "completely agree" to "completely disagree". The statements included topics such as learning time, understanding of instructions, ability to assemble new objects, the clumsiness of the system, experiences of nausea, experienced control and diﬃculty of use as well as fun to use. Three questions of particular interest for the study was whether or not the users would be prepared to use an AR system for similar tasks in their work, and in other situations than work, and if it was fun to use. Figure 4.9 below shows an example statement (all statements can be found in the questionnaire appendices).

Figure 4.9: An example statement from the questionnaire.

The second part or the questionnaire included 10 open questions where the participants could answer freely on their experience of the AR system. The questions related to overall impression of the AR system, experienced diﬃculties, experienced positive aspects, what they would change in the system and whether

( 100 ) Chapter 4. Single user studies 85 it is possible to compare receiving AR instructions to receiving instructions from a teacher.

For the analysis of the ﬁrst part of the questionnaire, the 6 point scale all responses was transformed so that 6 was used as the most positive answer for the AR system (e.g. the user for instance completely agrees that the system was fun to use, or completely disagrees that the system was clumsy to use) and 1 was the most negative. The open ended questions are discussed below in themes found in the data, relating to the diﬀerent issues raised in the questions.

4.3.6 Results

This section is divided into two parts, ﬁrst the results of the closed part of the questionnaire are presented and then the results of the open ended questions and observations.

Closed response questions The results from the closed part of the questionnaire can be seen in Table 4.1 below.

As can be seen in the table, the scores in general are on the upper part of the scale. The lowest score can be found in question 9, where the users responded to whether they would prefer to receive instructions from a teacher or tutor. It is perhaps no surprise that in general the participants would prefer human to human interaction to interacting with a technical system. The question that received the highest scores was a question regarding general ergonomics – none of the participants responded that they experienced any discomfort during the use of the system. The lowest grade was a 5, which shows that no one agreed to the statement of having experienced any discomfort. The questions most related to the general aim of this paper related to whether or not the participants would like to use this system in their profession, see question 6 in Table 4.1.

Looking more closely at the data, one participant, scored a 1 and thus deﬁnitely does not want to use this kind of system in their work, while four others scored

( 101 ) 86 Chapter 4. Single user studies

Table 4.1: Statements in the closed response questionnaire.

6 – deﬁnitely do want to use this kind of system in their work. One participant, who would like to use the AR system at work, did however not ﬁnd it fun to use and completely disagreed with the statement that the system was fun to use (questionnaire item 13 in Table 4.1).

Overall the participants did ﬁnd the system fun to use, the average score was 4.50 and seven of the twelve participants gave the score ﬁve or six.

Open ended response questions The open ended questions were transcribed and are reported here as a brief summary under separate themes. They are discussed in relation to observations made during the study as well as during analysis of the video recorded material.

( 102 ) Chapter 4. Single user studies 87

Marker problems Many of the participants commented on the marker as being problematic. The problems arise when the marker is obscured in one way or another (mostly due to movements of the hand) and the AR system looses contact with the marker and then when the marker is identiﬁed the last viewed instruction is repeated. That this is problematic is illustrated by the video data, which shows how the participants move their hands to get the marker in clear view. This may aﬀect the perceived ease of use of the AR system and contribute to the experienced clumsiness of it.

Multimodality Concerning the multimodal aspects of the instructions, receiving both aural and visual instruction seemed to increase the perceived ease of use of the system:

"The voice control + text and image. You just can’t do wrong." (participant 2)

However it also confused one of the users answering the question of there was anything that s/he experienced as troublesome:

"...to both listen and look, hard to see what part to use." (participant 11)

Instructions A majority (eight) of the participants experienced the instructions as clear and easy to follow:

"Simple, compared to if you by reading text are trying to obtain the same information."(participant 4)

"Easy to understand the instruction. Easy to use." (participant 3)

The quote by participant 3 illustrates the relation between good design of instructions and the overall impression of the system. Overall the participants

( 103 ) 88 Chapter 4. Single user studies seemed to cope well with working through the AR system, and most of them found the system fun to use (as also can be seen in question 13 in the closed questionnaire). The idea of a system being fun to use is a positive factor when introducing new technology into a work place. As noted above, when users of a system ﬁnd it entertaining, they may be able to cope with more complex and more diﬃcult systems. However, it still must be perceived useful for their task, or it becomes redundant.

Comparing the AR system to other systems All of the respondents work with computers on a day to day basis and are accustomed to traditional MS Windowsa based graphical user interfaces but they saw no similarities with the AR system. Consistent with the findings in a previous study (Nilsson and Johansson, 2006b), most of the respondents referred to interacting with or through a computer when asked what the interaction felt like. Most of them (nine out of the twelve, or 75 percent) did not find any comparison at all, but the few that did, did not mention other computer aided instructional systems, but instead laparoscopic surgery, and computer games. The respondents are usually introduced to new technology and equipment through more experienced staff, or sometimes by the company selling the product. One problem with receiving instructions from colleagues and other staff members are that the instructions are not "objective", but more of "this is what I usually do. The only "objective" instructions available are the manual or technical documentation and reading this is time consuming and often not a priority. When asked about the possibility to compare the AR system with receiving instructions from a teacher one response was this:

"What you can be sure about with this system is that everyone has gotten the exact same information." (participant 4)

Ergonomic issues None of the open ended questions were speciﬁcally about the placement of the marker, but the marker was mentioned by half of the participants as either troublesome or not functional in this application:

( 104 ) Chapter 4. Single user studies 89

"It would have been nice to not have to think about the marker." (participant 7)

Concerning the dual modality function in the AR instructions (instructions given both aurally and visually) one respondent commented on this as a positive factor in the system. But another participant instead considered the multimedial presentation as being confusing:

"I get a bit confused by the voice and the images. I think it’s harder than it maybe is" (participant 9)

Issues of parallax and depth perception is not a problem unique to this AR application. It is a commonly known problem for any video-see-through system where the cameras angle are somewhat distorted from the angle of the users’ eyes, causing a parallax vision, i.e. the user will see her/his hands at on position but this position does not correspond to actual position of the hand. Only one participant mentioned problems related to this issue:

"The depth was missing." (participant 7)

The instructions A majority among the participants (eight, or 67 percent) gave positive remarks on the instructions and presentation of instructions. One issue raised by two participants was the possibility to ask questions. The issue of feedback and the possibility to ask questions are also connected to the issue of the system being more or less comparable to human tutoring. It was in relation this question that most responses concerning the possibility to ask questions, and the lack of feedback were raised. The question of whether or not it is possible to compare receiving instructions from the AR system with receiving instructions from a human did get an overall positive response. Four of the twelve participants gave a clear yes answer and ﬁve gave more unclear answers like:

"Rather a complement; for repetition. Better? Teacher/tutor/instructor is not always available – then when a device is used rarely – very good." (participant 1).

( 105 ) 90 Chapter 4. Single user studies

Several of the respondents in the Yes category actually stated that the AR system was better than instructions from a teacher, because the instructions were "objective" in the sense that everyone will get exactly the same information. When asked about their impressions of the AR system, a majority of the participants gave very positive responses:

"Very interesting concept. Easy to understand the instructions. Easy to use." (participant 3)

Others had more concerns:

"So and so, a bit tricky." (participant 6).

One question speciﬁcally targeted the attitude towards using AR in their future professional life and all participants responded positively to this question.

4.3.7 Discussion of the results

The differences within the group, and the fact that there are only twelve participants, make it difficult to draw any general conclusions from the results of the responses, but the individual comments are nonetheless relevant to the further development of the AR system. All users in this study were able to complete the task with the aid of AR instructions. The responses in the questionnaire were diverse in content but a few topics were raised by several respondents and several themes could be identified across the answers of the participants. Issues, problems or comments that were raised by more than one participant have been the focus of the analysis.

4.4 Study 3 - the second troakar study

One major diﬀerence between the ESG instructional application and the ﬁrst troakar application was, apart from the apparent physical design of the AR system,

( 106 ) Chapter 4. Single user studies 91 the added auditory dimension of the system. In order to investigate the perceived differences between the first type of visual only application and second type of instructional application that includes verbal instructions, a third end user study was conducted. While the first troakar study is a direct follow up of the ESG study, this, the second troakar study, is a direct follow up in order to investigate the results of the first troakar study. In this study the AR application was evaluated by a slightly different user group, namely university students all in nursing school, i.e. future professional end users.

The main goal of this study was to evaluate the effects of multi-medial information presentation in an AR system. The specific aim of the study was to find out whether AR instructions given both aurally and visually to the participants are more effective than AR instructions presented only visually to the user.

4.4.1 Methodology

The user study was conducted at a nursing school and the participants were observed using the MR system, receiving instructions on how to assemble a troakar (described above). Two independent variables were constructed in the study; first the "sound condition" where the users received instructions both aurally, through headphones, and visually, both text and animation; secondly the "no sound" condition, where the users received only visual instructions in the shape of text and animation. As in the previous troakar study the participants were first introduced to the AR system. When the head mounted display (HMD) and headset was appropriately adjusted they were told to follow the instructions given by the system to assemble the device they had in front of them. After the task was completed the participants filled out a questionnaire about the experience. The participants were recorded with a digital video camera when they assembled the troakar. During the task, the participants’ view through the video-see-through MR system was also logged on digital video.

( 107 ) 92 Chapter 4. Single user studies

4.4.2 Participants

34 users took part in the study, all of whom were students at the medical faculty in training to become nurses. All participants were between 20 and 30 years old and had no prior knowledge or experience of the object of assembly used in the study. A majority of the participants stated that they have an interest in new technology, and that they interact with computers regularly, however a majority of them had very limited experience of video games and 3D graphics. The participants were randomly assigned to one of the two conditions; 15 in the "no sound" condition and 17 in the "sound" condition. Originally the groups were of equal size but the recording and logging of two trials failed so the data was not complete and therefore removed from the sample.

4.4.3 Equipment

The AR system (seen in Figure 4.10), as well as the observation equipment used in this study was the same as described in the previous study.

Figure 4.10: A participant wearing the AR system with voice input functionality during the second troakar study.

( 108 ) Chapter 4. Single user studies 93

4.4.4 The end user task

The users in this study were asked to perform the same task as described in the previous study. Figure 4.11 below shows an example of what the participants in the study could see.

Figure 4.11: An example of instructions as seen by the participant ("screw together with the cylindrical part").

4.4.5 Data analysis

Data was collected both through observation and through questionnaires. The questionnaire responses were described and analysed statistically. The observation data resulted in data on frequent errors, task completion time and general interaction with, and through the AR system. The questionnaires included a set of closed questions where the participants responded on a 6 point Likert scale (the same questionnaire as used in the previous troakar study) and a set of ten open questions. As mentioned above, the statements included topics such as learning time, understanding of instructions, ability to assemble new objects, the clumsiness of the system, experience of nausea, experience of control, difficulty of use as well as fun to use. The questionnaires were analysed in the same manner as in the first troakar study with the difference that there were two groups in this study allowing a comparison of the sound conditions.

( 109 ) 94 Chapter 4. Single user studies

4.4.6 Results

In this section the results of the study are described, ﬁrst the results of the closed questionnaire responses, and secondly the responses in the open questions.

The closed response questions Table 4.2 shows the overall scores in the questionnaires. None of the participants had any major diﬃculties with the assembly of the troakar. However a few of them did miss one instruction and were forced to step backwards in the instruction to complete the task.

Table 4.2: Overall scores on the questionnaire

There was no signiﬁcant diﬀerence in completion times or overall scoring of the system as can be seen in Table 4.3.

( 110 ) Chapter 4. Single user studies 95

Table 4.3: Overall score and completion time.

Sound No Sound ttest Average score (max 84). 68.35 64.27 p=0.47 Average completion time 3min 25s 3min 10s p=0.33

Table 4.4: Diﬀerence in statement responses.

Statement ttest 2. The text was diﬃcult to read due to color or size p=0.072 5. I felt in control of the system p=0.086 6.I would like to use this kind of system to get instructions in my work p=0,027 8. I feel conﬁdent that the system is giving me correct instructions p=0,122 10.This system is clumsy to use p=0.027

The overall score did not show any difference between the two conditions but in specific questions there were some differences, as can be seen in Table 4.4. As expected, the participants in the sound condition (participants given both aural and visual instructions) tended to rate the system more positively although both groups did give the system overall high scores. Significant differences at the 0.05 level (using Bonferroni post hoc tests) were found were found on statements 6 and 10 and tendencies (p= 0.1) were found in statements 2, 5 and 8 as can be seen in the table.

The results of the closed questionnaire indicate that there are differences between the two groups. As can be seen in in Table 4.4 there are significant differences on a few questions. Statement 5, on whether the text was difficult to read due to color or size, was answered differently between the groups (see Figure 4.12, left).

Figure 4.12: Results from items 5, p=0.072 and 8, p=0.086

( 111 ) 96 Chapter 4. Single user studies

The question whether or not the participants feel in control of the system also diﬀered signiﬁcantly, which can be seen in see Figure 4.12, right.

Another question that the participants in the diﬀerent conditions showed tendencies to rate diﬀerently was about trust in the system - whether or not they felt that the system gave them correct information (see Figure 4.13, left).

Figure 4.13: Result from items 12, p= 0.122 and 14, p=0.027

The difference is not large (mean=5.3 vs 4.9, ttest p=0,122) but comes near a statistical significance of p=0.1. Opinions on whether or not the system was clumsy to use also differed between the groups see Figure 4.13, right. As in the other questions the more positive score came from the group receiving both auditory and visual instructions, who rated the system as less clumsy to use than the participants in the other group.

A question relating to the overall experience of the system is whether the participants could imagine to use this kind of system in their work, and most of the participants were positive towards this, see Figure 4.14. Had the system not been experienced as easy to use, too clumsy etcetera, the result would likely have been indicating that the participants would not like to use it as a part of their work.

The open response questions The open questions in the second part of the questionnaire gave room to clarify the responses made in the closed response section. The responses in this section were diverse in content but a few topics were raised by several respondents. There were no major diﬀerences found between the groups in the open response questions. This section brieﬂy summarizes the responses according to themes found.

( 112 ) Chapter 4. Single user studies 97

Figure 4.14: Result from item 9, p= 0.027

Marker placement There was no question targeting marker placement, but several participants did comment on the placement of the marker, three of the seventeen participants in the sound condition and three out of the ﬁfteen in the no sound condition:

"It was a bit tricky to think about keeping the marker in the ﬁeld of view, don’t know if you could do it in another way" (participant 6, No sound)

All of the participants, in both conditions, mentioning the marker wanted to change the placement, possibly making it non-mobile.

Visual depth A common issue in video see-through AR is the problem of parallax; that the image presented diﬀers from the expected image. This is due to the cameras placement in front of the participant’s eyes. This was commented upon by participants in both conditions; three in the sound condition and two in the no sound condition:

"It was a bit diﬃcult to know how far away the things are in front of you" (participant 5, No sound)

Peripheral view A few respondents mentioned the peripheral view as a disturbance in the interaction (four out of the 17 in the no sound condition and one out of the 15 in the sound condition), for example:

( 113 ) 98 Chapter 4. Single user studies

"...annoying sometimes that you could see under the glasses, maybe a shield that would cover the ﬁeld of view so that you could only see the display." (participant 1, Sound)

Feedback and interactivity Two themes that were identiﬁed by some participants was the lack of feedback from the system (i.e acknowledging when a verbal command was registered). Three participants in the sound condition and one in the no sound condition. Another related theme is the lack of interactivity in that the user cannot ask questions and get responses from the system. This was mentioned by one participant in the sound condition and two in the no sound condition.

"I had some diﬃculties with parts of it and then I would have liked for the voice to tell me that I had done it correctly and that I could move on" (participant 2, Sound)

Experienced uncertainty and control A theme related to the lack of feedback and interactivity in the AR system is a feeling of uncertainty, which was experienced by two of the sound condition participants, and four of the no sound condition participants.

"It was diﬃcult to know whether I did something wrong or if the system took time to approve ok [the verbal command]"(participant 1, No sound)

Another aspect concerning interactivity is the ability to control the interaction. Participants in both groups had comments regarding this:

"No instructions were skipped due to browsing a text. That you could complete the assembly in your own phase" (participant 11, No sound, answering a question about positive aspects of the AR system)

Multimedial interaction Several comments were made about the presence of multimedial instructions. Five

( 114 ) Chapter 4. Single user studies 99 of the participants in the sound condition mentioned the positive eﬀect of having both auditory and visual instructions:

"Clear and simple instructions on how to navigate (OK/backwards). Good that the instructions were repeated if you looked away. Good to have both image and sound instructions" (participant 8, Sound)

Comments were also made about using both text and images by two participants in the no sound condition:

"Very good that you could see the construction, that images illustrated how to do it. Good that I had the things in front of me and instructions in the ﬁeld of view, easier to focus than with written paper information" (participant 14, No sound)

Clarity of instructions A majority (twelve, or 71 percent) of the participants in the sound condition experienced the instructions to be clear and easy to follow. However, two participants had a diﬀerent experience:

"Sometimes the images didn’t look exactly like the objects, which could cause some confusion.Ó (participant 4, Sound)

The result was similar in the no sound condition where eleven of the participants (73 percent) mentioned the clarity and that the instructions were easy to follow. In this condition none of the participants mentioned any problems with clarity in the instructions.

Human-like interaction? The ﬁnal question in the questionnaire was whether it is possible to compare receiving instructions in the AR system to receiving instructions from a human (teacher/tutor/instructor). The question was open, but the answers were translated into "yes", "no" and "unclear" where necessary. There was a diﬀerence in the response as can be seen in Figure 4.15

( 115 ) 100 Chapter 4. Single user studies

Figure 4.15: Comparison between the conditions, p=0.016

The translation was triangulated in that two observers did translations and these were then compared and correlated. Cohens Kappa was used to determine the inter-rater reliability and the result was 0.90, which is well above the satisfactory level of 0.70.

4.4.7 Discussion

In this section the quantiﬁed results of the closed response questionnaire are analysed and discussed. In the following section a more qualitative analysis and discussion of the results of the open questions are presented.

There were no signiﬁcant diﬀerences between the two conditions in terms of the overall interaction with the AR system, although there was a tendency indicating that the participants with both visual and aural instruction used the backwards command more often than the participants receiving only visual instructions.

The experience of readability problems does seem to have a close coupling to previous ﬁndings in multimodality research - that using more modalities can disambiguate information. So by both hearing instructions and seeing them does enhance the recognition of the instructions. Hearing the instructions makes it easier to read them at the same time, or at least disambiguates and clariﬁes the text. This may also be relevant for the feeling of being in control over the system, being able to recognise and understand it’s instructions.

( 116 ) Chapter 4. Single user studies 101

The open questions were not targeting specific issues, rather they were general, supplementing the closed response statements and allowing the participants to elaborate on issues they found relevant. Nonetheless, several themes could be identified across the answers of the participants. Issues, problems or comments that were raised by more than one participant have been the focus of the analysis. As noted, there were no significant differences found between the groups in what topics they raised, but comments made are of general interest for the usability of the system.

The issue of marker placement is interesting since, this application is the first known to the author, using a ring mounted marker. On the whole it did not cause any problems in relation to completing the task, however several participants did find it disturbing that the marker was not always in clear view, causing the instructions to restart when it was recognized by the AR system again. One user did however find this to be a good thing - allowing her/him to "step back" by just covering the marker. But as noted, most of the participants who commented were disturbed by it not being stuck in the field of view. This would probably not have been so easy to solve, seeing that placing the marker on the table in front of them for instance would cause the participants to occasionally loose it from their field of view, either by covering it with their hand or the troakar while assembling it, or by turning their heads in such a way that it is no longer in their field of view. Different applications require different solutions and for this application the ring based marker seemed to work, but perhaps an alternative marker placed on the table should have been included.

The problem of parallax, and the problems of not having regular depth perception when looking through the AR system is a commonly known in video see through AR. Although it was commented upon by some of the participants, the lack of depth perception did not make it impossible to assemble the rather small object. What was lacking in visual depth was relatively easy compensated by tactile perception and the participant did get used to the situation quite quickly.

( 117 ) 102 Chapter 4. Single user studies

Interactivity and feedback are important issues for usability in any system. Providing the user with some feedback, such as if the voice command has been acknowledged or not, is not diﬃcult, it is a matter of changing the settings. In this study the feedback option was deliberately turned oﬀ, which in retrospect may have been a bad choice. However it was a question of limited time; the voice output was the standard Windows speech module voice, speaking in English, and since this was not the language used in the application it was considered as a problem. Unfortunately there was no time to install another alternative language. In future applications of this AR system this problem will be addressed.

The statement by participant 14 in the no sound condition above illustrates one aspect of the system; the possibility to give instructions in real time and in the real context. Seeing animations overlaid, right next to or over what the user is doing may be beneﬁcial for the learning experience, as noted by the participant. This is an important aspect of using multi-medial information presentation as opposed to information using only one medium.

According to statistical standards there was not a significant difference between task completion time and average score. It can still be noted that there was a tendency towards more positive results (faster completion time, and higher score) in the sound condition. In general there were no extreme differences between the two conditions, and to be noted is that where the no sound group gave lower scores, the scores were still relatively high resulting in an average of 64 of the maximum 84.

One question of particular interest, and that was more targeted than the other open questions, was the final question: "Is it possible to compare the instructions given by the AR system to receiving instructions from a teacher/tutor?" This question relates to issues of systems seen as human-like or tool-like Qvarfordt et al. (2003). There was a significant difference in the responses between the groups as can be seen in Figure 4.15. The participants receiving both verbal and visual instruction were more prone to comparing the AR system with receiving instructions from a human, than participants receiving only visual instructions. This maybe related to

( 118 ) Chapter 4. Single user studies 103 issues of interactivity - a system that interacts with two modalities is more similar to a human than a system which only interacts in one.

To summarize, in analyzing the results of the study the eﬀect of multi-medial instructions seem to be of relevance. The participants were in general more positive towards the system when they had both verbal and visual instructions, and they gave the AR system more positive scores than the participants who received only visual instructions.

4.5 General observations and conclusions of the single user studies

The first ESG study was exploratory in nature, targeting the user experience and acceptance of the AR system in the context of health care. The results of this study were positive in that the participants quickly learned how to use the AR system and follow the instructions completing the task as required. The two following studies further explored the idea of giving AR instructions to professional health care staff, but for a somewhat different task. The ESG in the first study was in a sense more interactive in that it responded to the user’s actions - when it was turned on the lights started blinking, and when they changed the settings they got instant feedback from the apparatus. The troakar, on the other hand, is a "silent" device - it does not give any responses indicating whether the user was successful or not (other than the look and feel of course). In this study the interaction in focus was instead more on the AR system - the added feature of voice input gave another dimension of analysis. First the AR application was studied in the same environment as the ESG, with professional participants, and later it was studied in a somewhat more artificial environment with participants representing future health care professionals. The second group allowed us to study the effect of having instructions presented both visually and through sound. As seen in the result there were no major efficiency effects, but it became clear that for some users the ability to both hear the instructions and see the animations was experienced as very

( 119 ) 104 Chapter 4. Single user studies helpful. And one thing that all three studies have in common is just that - the user experience of the AR system and applications.

The issue of the system being interactive and able to answer questions, as well as noticing whether the user has completed an instruction correctly or not, is a much wider issue. These requests by the participants are very valid but they would require extensive and complex technological changes and development in the AR system, including artiﬁcial intelligence and object recognition, which will not easily be solved in the near future. However, this is a very interesting topic of discussion for the AR community as a whole. The possibility of integrating speech recognition, dialogue management and object recognition into AR systems would greatly improve their potential.

Ergonomic issues that were obvious in both the ESG and the troakar study was the parallax problem. This issue is however not speciﬁc to the applications described here but an issue related to the use of the video see-through AR solution. One way to solve the visual depth issue is to use a completely diﬀerent technical solution using optical see-though AR. Another approach which eliminates the commonly known problems of optic see through AR is by adding stereo vision to the video see through AR system. This would solve the depth issue, but not the parallax issue. The use of two cameras instead of one would allow depth perception, but the cameras would still have to be mounted somewhere leaving a gap between the cameras position and the user’s eye position.

When introducing new systems, like AR, in an activity, user acceptance is crucial. It the difference between conditions like the sound and no sound condition in terms of performance is not significant, it is probably tempting for a developer to aim for the less costly no sound solution. However, if a new system is to be accepted by its user, it is crucial that acceptance and trust in the system is high, and the findings from the last troakar study indicates that the perceived trust probably is higher in the sound condition. This is motivated by the fact that the participants in the sound condition reported fewer problems with understanding the instructions. Also, the clear difference regarding if the participants experienced

( 120 ) Chapter 4. Single user studies 105 the system as comparable to a human instructor strengthen the assumption that the sound condition has a higher acceptance among the users. Any aspect of the AR system that can make it more human-like such as adding more interactivity and the ability to verbally interact with it will probably be important for user acceptance in the future.

( 121 ) ( 122 ) Chapter 5

Multi-user study

The previous chapter explored the possibilities of introducing AR applications as tools for rather simple sequential instructional tasks. These applications were aimed at helping users complete a pre-deﬁned task in several steps, where one step naturally follows another. The participants in the studies responded positively to the technology and ideas in the project. In real life there are many situations where people work individually, as in the studies in chapter 4, but in many situations work practice involves collaboration with several other people. And many tasks are not as simple and straight forward as following a sequential instruction. To further explore the potential of AR in real life work situations there was a need to take it to a diﬀerent level, involving more than one user solving a more complex and dynamic task demanding both communication and collaboration.

Much like in the previous studies the aim was to ﬁnd a real world problem and apply and evaluate an AR solution to it. The domain chosen was crisis and emergency management as this is an area of work where people have to cooperate and communicate in order to solve diverse and changing, dynamic, problems. Since the challenge of developing and using AR for collaborative work in crisis management has not previously been explored to any length there are no other studies or methods to fall back on. This study has therefore drawn inspiration from other related ﬁelds and the method is partly based on a simulator used to train for, and evaluate collaborative work (Johansson et al., 2007; Trnka 107

( 123 ) 108 Chapter 5. Collaborative multi-user study et al., 2006; Lindgren and Smith, 2006; Trnka et al., 2008), and partly on a qualitative observational method that focuses on user feedback and naturalistic settings (Nilsson and Johansson, 2006b, 2008).

Like in the single user studies the basic approach used was user centered and learning from the results of the single user studies the iterative design of the application was crucial, as well as involving real end users from the very beginning of the project. The study included a pre-design phase where field experts from three different organisations (fire and rescue services, police department and the helicopter platoon in the local area) took part in a brainstorming session to establish the parameters of the AR system. This brainstorming session was used to define the components of the software interface, such as what type of symbols to use, and what type of information is important and relevant in the task of creating common ground between the three participating organisations. Based on an analysis of the brainstorming session a first design was implemented. This design was evaluated using a scenario in which the participants, one from each of the three organisations, had to interact and work together as they would in a real situation, cooperating in response to a forest fire. The exercise was observed and the participants also answered questionnaires pertaining to the AR system design, and finally a focus group discussion was held. As a result of this design evaluation the system was redesigned and this redesign was later evaluated through a second study consisting of a focus group discussion and observation of the users interacting with the redesigned system while performing simple tasks. The final outcome of this design process was then evaluated in a larger study.

5.1 The design and development process

The AR system was iteratively designed in three steps. The task for the participants was to collaborate in responding to a forest ﬁre. The roles the participants had were on a command and control level, meaning that they were responsible for planning resources and giving orders to people in the ﬁeld. The

( 124 ) Chapter 5. Collaborative multi-user study 109

AR system was used as a tool for them to see and manipulate their resources and as a way to have an overall view of the situation. As mentioned above, in the pre-design phase ﬁeld experts took part of a brainstorming session to establish the parameters of the AR system. This brainstorming session was used to deﬁne the components of the software interface, such as what type of symbols to use, and what type of information is important and relevant in the task for creating common ground between the three participating organisations.

5.1.1 The ﬁrst iteration

After the brainstorming session a ﬁrst design was implemented and evaluated in a study conducted with the purpose of evaluating the system design as a tool for collaboration between organisations. To promote realistic results, the participants were representatives from the three organisations in focus; the ﬁre department, the police and the military helicopter platoon. The setting was at a military helicopter base.

Figure 5.1: The study was conducted in a simulated natural setting in a hangar at a helicopter base.

The AR system was designed to support cooperation as advocated by Billinghurst and Kato (2002) and Gutwin and Greenberg (2001) and thus emphasised the need for actors to see each other. Therefore, it was equipped with hand-held displays that are easier to remove from the eyes than head mounted displays, see Figure 5.1. We used a digital map where participants had personal, individual views, allowing

( 125 ) 110 Chapter 5. Collaborative multi-user study them to see an organisation speciﬁc map and the symbols they normally use. In this way each actor has her own information mapping to the AR markers on the map to facilitate independence and individuality. A feature allowed each participant to send their view of the situation (i.e. their individual map) to the other participants when necessary. Hand pointing on the map was not possible as the hand was occluded by the digital image of the map in the display.

The tools used for data gathering were observation, focus groups and questionnaires. This design was then evaluated using a scenario in which participants, one from each of the three organisations, had to interact and work together to complete tasks in a dynamic scenario. The exercise was observed and the participants also answered questionnaires pertaining to the AR system design, and ﬁnally a focused group discussion was held. The evaluation revealed a number of issues regarding the design of the system as well as the scenario being used. In general, the participants were positive to the AR system. What they appreciated most was the easy overview of what was going on. Being able to see all resources placed on the map facilitates the joint task.

One function that was implemented in the system, but not used by the participants, was the possibility to send and receive the other organisations’ map views. The reason for them not using this functionality was probably that the instructions and practice they were given on the system were not clear enough on how to use this feature.

Several suggestions were given for redesign including a map with more details, more events in the scenario played and changing the physical interaction devices. Especially the design of the AR displays as a handheld device did not receive a positive response and the observations clearly illustrated this problem.

The participants also commented on more positive aspects of the system, such as the possibility of spatially distributed collaboration. Other ﬁndings in the ﬁrst evaluation questionnaires were that despite the relatively clumsy design of the prototype, all participants thought it was easy to use and that it was quick to

( 126 ) Chapter 5. Collaborative multi-user study 111

Figure 5.2: The HMD and redesigned interaction device, which allows the user to choose a virtual object and place it on the digital map. learn. Despite ﬂaws in the system, all participants could also see themselves using the AR system in their professional life as well as in other situations.

5.1.2 The second iteration of the Augmented Reality system

As a result of the design evaluation the system was redesigned. The handheld display was replaced with a head mounted display allowing freedom of movement. The interaction device was also considerably redesigned and in the new AR system the user can easily manipulate objects using only one hand as opposed to using both in the previous prototype, see Figure 5.2.

Another improvement made was a simpliﬁed interaction in which the user can use their hand to point at things in the digital map. In the previous design this pointing manoeuvre could not be seen as the digital map was superimposed over the pointing hand giving the impression that the user was pointing ’under’ the map rather than on the map. The ﬁrst prototype therefore had a pointing function in the interaction device. The new, improved technical design has eliminated the need for this pointing device as the system now allows the users hand to be superimposed over the digital map image using blue-screen technique, see Figure 5.3. This allows

( 127 ) 112 Chapter 5. Collaborative multi-user study the users to use deictic gestures like pointing since their hands are visible above the digital representation. The system thus presents several properties of a normal paper map with the added functionality of adding, moving and removing digital objects that carry information and can be manipulated by any user working with the system.

Figure 5.3: The users display showing the digital map with symbols and pointing used in the collaborative AR application

The redesigned AR system was evaluated in a focus group discussion where the participants first were asked to reflect on their experience in the previous study. Then the redesigned system was presented and the participants were observed using it to complete simple tasks from the scenario in the pre-study. After this the focus group discussion continued with reflections on the new design.

The results from the discussions were positive. The problems that they reported on previously had been addressed. The head mounted display was a big improvement and allowed them to move around and interact more freely. The new joystick interaction device was also appreciated and the participants found it very easy to use. The added possibility to see deictic gestures such as pointing, on the digital map has simpliﬁed the interaction considerably and also resulted in a more natural interaction and better communication between the participants. In the redesigned application the participants had exactly the same view allowing them to alter their personal image but still seeing the same map, and not as previously the

( 128 ) Chapter 5. Collaborative multi-user study 113 organisation speciﬁc map. As noted by one of the participants during the group discussion:

“A common picture, everything is better than me telling someone what it looks like...you need to see the picture and not to hear my words. “ (participant from the second evaluation)

5.2 The end user study

For the third iteration of the system a complete user study was conducted and participants from three different organisations involved in crisis management were recruited. The study was conducted with the purpose of evaluating the AR system and application design in order to improve it as a tool for collaboration between organisations. For obvious reasons it is not possible to conduct experiments in a real life fire-fighting situation. Instead, to create a realistic study, we used a scenario where the groups had to collaborate, distribute resources and plan actions in response to a simulated forest fire and other related or non–related events. In the scenario the participants act as they would as on–scene commanders in their respective organisation. This means that they together have to discuss the current situation and decide how to proceed in order to manage the fire, evacuate residents, redirect traffic, coordinate personnel as well as dealing with any unexpected events that may occur during such incidents. Normally discussions like this take place in a temporary control room (often a room at an appropriate location near the affected area) around a paper map of the affected area. The AR system was used as a tool for them to see and manipulate their resources on a digital map and as a way to have an overall view of the situation.

As noted previously, the CSE approach to studying human computer interaction advocates a natural setting. A natural setting in crisis management is a rather dynamic setting and the course of events depends on each participant’s actions, making it diﬃcult to quantify and compare the results from one trial to the next. In a natural setting, unforeseen consequences are inevitable and also desirable.

( 129 ) 114 Chapter 5. Collaborative multi-user study

This makes each trial different - the actions taken by the team in one trial are not necessarily taken by another team (even though they are presented with the same information). Unless you can gather a very large number of participants, this makes it very difficult to use time and other frequencies as a measure of successful performance or experience of the system. Instead the main source of data in this study are questionnaires addressing the participants experience of the AR system, collaboration and task. Unfortunately current AR systems are not developed enough for use in real life situations, especially not if used in situations where enormous values are on stake, such as large forest fires. Therefore we use simulations of events, cf. (Woods and Roth, 1988) in this study.

5.2.1 Participants

In total 30 participants took part of the study during ten sessions distributed over ten days, with three persons in each session. The participants were all at the level in their organisation where they in real life are assigned to team-coordinating situations. This means that they all either have experience from working in teams with partners from at least one of the other organisations, or have a position in their organisation which require that they have a minimal education and training in these types of command and control assignments. The groups formed here had never worked together before and they did not know each other prior to this study. They are, however, used to this type of simulation based training and exercises.

Of the ten trials, two were spoiled due to unforeseeable events (in one case one participant was called to active duty due to an emergency and in the other case external technical problems forced the trial to end prematurely). This resulted in a total of eight complete trials with 24 participants, of which 23 were male, one female and the ages ranged from 25 to 57 (median: 36, average: 39,1). The participants from the police ranged between 30 and 39 (median age: 36, average: 34,9), the participants from the rescue services between 35 and 57 (median: 48, average: 48,6), and the helicopter pilots ranged from 25 to 56 (median: 28,5, average: 34,4) in age.

( 130 ) Chapter 5. Collaborative multi-user study 115

5.2.2 The Augmented Reality system used in the study

The AR system used in the study was a high fidelity prototype, which allowed the users to interact with virtual elements. It includes a Z800 3DVisor from eMagin (http://www.3dvisor.com/) integrated with a firewire camera. The Mixed Reality system runs on a 2.10GHz laptop with 3 GB RAM and a 128 MB NVIDIA GeForce 8400M GS graphics card and the marker tracking software used is based on ARToolkit Kato and Billinghurst (1999). In order to interact with the AR system the users had a joystick-like interaction device allowing them to choose objects and functions affecting their view of the digital map, see Figure 5.4.

Figure 5.4: The joystick interaction device used in the study.

One important technical design feature, which was a result of the iterative design process, is the ability to point in the map. The system allows the users hand to be superimposed over the digital map image, see Figure 5.3.

The users have access to a personal, organisation-specific symbol library which they can use to create a situational picture. Examples of symbols are police vehicles, fire trucks, helicopters, and personnel. Other types of symbols are the function symbols, for instance the i symbol which when used allows the user to see additional organisation-specific information about the already placed symbols, such as information about how many hours personnel has been on duty, or how much water is left in the tank of a tank truck. The symbols are simplified to some degree in order to be understandable by users from other organisations. All symbols are three-dimensional and follows the users’ movements, e.g. if a user

( 131 ) 116 Chapter 5. Collaborative multi-user study kneels down the symbols are seen from the side. It is also possible to personalise the system by ﬁltering out symbols belonging to one or more organisation, thus, e.g. showing only symbols from the own organisation on the map.

If necessary, the users can manipulate each others symbols, e.g. a fire-fighter can place, delete and move a police vehicle. There are also a set of symbols that are common to all users of the systems, such as fires and smoke (this is particularly important in this case as the participants in the study are confronted with a forest- fire fighting task). The users thus have access to a digital playground where they can add symbols, move them or remove them freely. The symbols were placed in relation to a marker attached on a joystick, meaning that there was no fixed menu in the user’s field of view or related to the map. Instead the menu of symbols was related to the physical controller.

Users can use a command function to zoom in and out on the map to focus on a symbol or a part of the map. It is also possible to physically lean in over the map to get a closer look, as you would over a regular paper map. In order to place a symbol the user ﬁrst moves the joystick-attached marker to the chosen position on the map and then selects and places the symbol in the menu by using the buttons on the joystick. The same procedure is used to remove a symbol, to see additional information about a symbol or to zoom in the map.

In order for the users to share the same view the AR systems must be interlinked and responsive to what each system user does. In the three system setup the individual AR systems communicate through an internal Ethernet network. Each system listens for changes in the internal representation in the other AR systems and updates its own internal representation to reﬂect the changes made to the representation in the other two AR systems.

Observation equipment The user study was recorded with ﬁve digital video cameras to allow later observation of the participants’ interactions with the each other as well as with the AR system and paper map. Two cameras were directed towards the area in which the participants worked, and three cameras were used to log the activity in each

( 132 ) Chapter 5. Collaborative multi-user study 117 participant’s AR system. During the study observers took notes in an observation protocol in which observations were organised in relation to diﬀerent aspects of interest.

5.2.3 User task and procedure

In the study the participants collaborated in groups of three, with one commander from each organisation (the ﬁre and rescue services, the police department and the helicopter platoon) in every team. The participants had to interact and work together to complete assignments in the dynamic scenario. The team worked together around the same table to encourage face-to-face communication as this is an important part of collaborative planning processes (as noted above).

Three different scenarios were used, each describing a forest fire that had been going on for a couple of days. The description was rather detailed and included information on when the fire has started, where people had been seen, weather conditions etc. Each organisation had a number of units that they had to assign to different locations on the map as they would have done in a real situation. The participants all have the same digital map in their view. They can independently place symbols using the handheld interaction device and they can also discuss with each other how to place their own symbols and also shared symbols, such as the fire symbol and break points.

To create a baseline for the evaluation the scenario was also conducted using paper maps and pens (similar to how they would normally go about this type of task). Every team performed the exercise over a course of about three hours where time was spent accordingly: 30 minutes introduction to the project and procedure, 30 minutes of training where the participants did a test run of the procedure both using the paper map and using the AR system, and then three 20 minute sessions where the forest ﬁre scenario was played out with a number of diﬀerent events in every session.

( 133 ) 118 Chapter 5. Collaborative multi-user study

Figure 5.5: The simulated natural setting (a hangar at a helicopter base) in the end user study.

In the first session the participants worked with the AR system, in the second session they worked with a paper map, pens and transparency film, and in the third session they used the AR system again. After each of the three sessions the participants filled out a questionnaire regarding the session and after the final session they also filled out an additional questionnaire which focused on an overall evaluation of the AR system. Finally a focus group discussion was held where the participants could reflect on and discuss a series of topics relating to the exercises, the scenario and the AR system.

The study was conducted at a helicopter base, Figure 5.5. In order to create a dynamic scenario and realistic responses and reactions to the participants’ decisions in the three sessions, we used a gaming simulator, C3 Fire (Granlund, 2001). C3Fire generates a task environment where a simulated forest fire evolves over time. The simulation includes a map and features like houses, different kinds of vegetation, computer simulated agents, vehicles etc. that can be controlled by an experiment assistant. For the purpose of the this study the micro world was used to create a realistic chain of events in response to the participants actions, thus making the scenario situation more realistic. The simulator was run in the background by the research team where one member inserted information into the gaming simulator, for instance, that several police cars have been reallocated to attend to a traffic incident. The experiment manager acted as a feedback channel to the participants in order for them to carry out their work. For instance, when the reallocated police cars had reached their new destination the experiment leader

( 134 ) Chapter 5. Collaborative multi-user study 119

Figure 5.6: A schematic view of the user study.

returned with information to the participants. (In other words, the experiment leader was the communication system between the commanders in the control room and the personnel in the ﬁeld.) Other examples of information from the gaming simulator are weather reports, status and location of personnel and vehicles, the spread of the ﬁre etc. Figure 5.6 illustrates the set-up of the C3 Fire engine in the study.

To summarise the experiment:

Activity Duration Introduction to AR and the experiment 30 minutes AR practise using a predeﬁned set of tasks ≈ 15 minutes Paper map exercise using a predeﬁned set of tasks ≈ 15 minutes AR session 1 20 minutes Co-operation questionnaire ≈ 10 minutes Paper map session 20 minutes Co-operation questionnaire ≈ 10 minutes AR session 2 20 minutes Co-operation questionnaire ≈ 10 minutes AR questionnaire ≈ 15 minutes Focus group discussion ≈ 20 minutes

( 135 ) 120 Chapter 5. Collaborative multi-user study

5.2.4 Data analysis

The main data sources used in this study were the questionnaires, although observations were used to triangulate the responses. As in the previously described troakar studies, the ﬁrst three questionnaires, which focused on collaborative aspects of the application and task, consisted of a set of closed questions where the participants responded on a 6 point Likert scale (see Appendix). In total the questionnaire included 18 closed questions of which the ﬁrst 3 were demo-graphical in nature (prior technological experience, interest in technology and computer use). The remaining 15 questions focused on the user experience with statements where the user could check a box between "completely agree" to "completely disagree".

For the analysis of the ﬁrst part of the questionnaire, the 6 point scale, all responses were transformed so that 6 was used as the most positive answer for the AR system (e.g. the user for instance completely agrees that the system was fun to use, or completely disagrees that the system was clumsy to use) and 1 was the most negative. The open ended questions are discussed below in relation to the closed response questions. Several types of analyses have then been made on the data, both a comparison between the AR system sessions and the paper map session, as well as an analysis of the ﬁnal AR system evaluation as described in the following sections.

The study also included a ﬁnal evaluation questionnaire, which included 18 closed response questions, also on a 6 point Likert scale, and a set of open questions allowing the participants to expand on certain issues. This questionnaire speciﬁcally targeted the AR application and system and the interaction related

( 136 ) Chapter 5. Collaborative multi-user study 121 with it (see Appendix) and it was analysed mainly with descriptive statistics as described below. The results of the open ended questions are discussed in relation to the closed response questions.

In addition to the questionnaires the participants took part in a focused group discussion after the sessions, which allowed them to elaborate on the issues raised by the questionnaires as well as give any additional comments. This discussion was recorded and used as a secondary source of information in the analysis, guiding the observations made of the video recordings made during the study. To guide the video observation, observation protocols were used during the sessions. The protocols were organised according to themes of interest for the observation, for instance noting any potential interaction problems, or communication issues either with each other or the AR system (see Appendix). These protocols were not part of the quantitative analysis, but mainly used as a resource for the general observations.

5.3 Results

This section describes the results of the study. First a comparison between the three sessions is made, then the questions regarding the collaborative aspects in the study are addressed, and ﬁnally the overall evaluation of the AR system is presented.

5.3.1 Comparing the three sessions

The AR-system collaboration questionnaire consisted of 15 items, and 6 open- ended questions. The queries and data from the sessions are presented in Table 5.11. The open-ended responses are presented and discussed in relation to the relevant questionnaire items.

1The queries have been translated to English by the author

( 137 ) 122 Chapter 5. Collaborative multi-user study

Table 5.1: AR-system questionnaire, average score and standard deviation. As the statements in the questionnaire were both positively and negatively loaded (see for instance the ﬁrst two items), the scores on the negatively loaded items were transformed in order to make the result easier to interpret. This means that in the table a high score is positive for the AR system/paper map and a low score is negative for the AR system/paper map.

In general the results on the questionnaire were positive for the AR system. The average scores were all above 3 out of 6. Using one way ANOVA with Bonferroni post hoc tests we found signiﬁcant diﬀerences between the three session on items 1, 5, 6, 7, 8, 9, 12 and 13 as can be seen in Table 5.1 .

There is a significant difference between the first AR session (AR1) and the second AR session (AR2) on the first item, Item 1, regarding how long it took to begin

( 138 ) Chapter 5. Collaborative multi-user study 123

Figure 5.7: Results from questionnaire items 1, It took a long time to start to cooperate and 5, I felt that the group controlled the situation. For further explanation, see text.

working together. The participants felt that it took longer time to cooperate in the ﬁrst AR session, see Figure 5.7, left. In AR2 they felt that they began to collaborate as fast, or faster, as when they used the paper map (F(2,42)=12,8, p<0.05).

As one user commented:

"Since we were a bit used to it, we could use the technology in a better and more eﬀective way". (RS0924, Question 3:4)2

When asked if it was easy to collaborate, Item 2, the results were positive in all three sessions - the mean score was 4.7, 4.7 and 5.0 on the 6 grade scale. There was in this case a slight difference between the organisations, where the rescue service participants scored lower than the helicopter pilots (F(2,42)= 2.8, p<0.05). Although there were no significant effects between the first and second AR session there is a social effect of getting to know one another better and therefore being able to understand and collaborate better:

"It worked smoothly with suggestions and orders. Mainly because of the shared picture and also since we are beginning to ﬁnd our feet". (HP0926, question 3:4)

2In the following text quotes of the participants are coded as follows: the ﬁrst letter/s indicate organisation (P-Police, RS- Rescue services, HP- Helicopter Pilot), the following four numbers are the team number, and the ﬁnal number combination indicates which of the open-ended questions the quote is related to.

( 139 ) 124 Chapter 5. Collaborative multi-user study

When asked about the AR system as a tool for collaboration, Item 3, again we had high scores. There was no significant difference between the sessions. There were however some differences between the organisations, where the helicopter pilots appreciated the AR system slightly more than the rescue service participants (F(4,42)=5.1, p<0.05).

Concerning whether or not the participants enjoyed the collaboration, Item 4, the scores are very high, between 5.2 and 5.3. There was no signiﬁcant diﬀerence between the sessions, see Figure 5.1, all seemed to enjoy it and the means were 4.3 at the lowest.

On the item of feeling that the group had control over the situation, Item 5, we note the importance of training. We have a significantly lower value for the first AR session, (Figure 5.7) indicating that users have the same sense of control using the AR system as they have using a normal paper based map, after some training. In AR1 the average score was 4,2 while the average in the second AR session was 4.8 (F(2,42)=7.98, p<0.05). In the paper session the average was 5.0 and this was also significantly higher than in AR1 (F2(42)=7.98, p<0.05). There was no significant difference between the paper map session and AR2.

Another aspect of collaboration is sharing information, Item 6, and this activity was more difficult during the first AR session. The overall average score on the item regarding information sharing, Item 6, was high; 4.0 out of 6 in AR1 and 4.8 in AR2 and 5,0 in the paper session, see Figure 5.8, right. The difference was significant between AR1 and AR2 (F(2,42= 12.0, p<0.05) and between AR1 and the paper map (F(2,42)=12.0, p<0.05). However, there was no significant difference between the second AR session and the paper map session which may indicate that sharing information was not experienced as more difficult when working with the AR system after some training, as compared to working with the paper map.

A group of items speciﬁcally addressed the map and the symbols on the map; Item 7, Item 8 and Item 9. Here the scores for the AR system are higher than for the paper map, Figure 5.8 and Figure 5.9, suggesting that the use of the AR system made it easier to achieve a common situation picture. Regarding the map, Item 7,

( 140 ) Chapter 5. Collaborative multi-user study 125

Figure 5.8: Results from items 6, It was easy to mediate information between the organisations and 7, The map made it easy to achieve a common situational picture. For further explanation, see text.

we only see a tendency to difference between AR2 and the paper map (F(2,42)=6.1, p≈0.052), but regarding the symbols, Item 8, there is a significant difference. The symbols in AR2 made it easier to achieve a common situational picture compared to the paper map (F(2,42)=15.3, p<0.05). The map is also regarded as less messy when using the AR system, Item 9, with significant differences both the first and second time the AR system was used, AR1vs paper map (F(2,42)=12.7, p<0.05) and AR2 vs paper map (F(2,42)=12.7, p<0.05).

We also note that the users wanted even more symbols than we had on the map, Item 10, scoring rather low on this item in all three situations. The participants had, however, no problems to interpret the symbols, Item 11. When asked if the map and symbols helped the participants trust the situational picture, Item 12 and Item 13 there are diﬀerences, see Figure 5.10. Concerning whether the map helped the users trust the situational picture, Item 12, we have a tendency to diﬀerence

Figure 5.9: Results from items 8, The symbols made it easy to achieve a common situational picture and 9, The map became cluttered/messy. See text for further explanation.

( 141 ) 126 Chapter 5. Collaborative multi-user study

Figure 5.10: Results from items 12, The map helped me trust the situational picture and 13, The symbols helped me trust the situational picture. See text for further explanation.

between the paper map and the second usage of the AR system, AR2, on the map, Item 12 (F(2,42)=4.6, p≈0.051). The symbols, Item 13, helped the users more using the AR system, AR2, than the symbols on the paper map (F(2,42)=5.1, p<0.05). We also found a significant difference between the first and second use of the AR system, AR1 vs AR2 for Item 12 (F(2,42)=4.6, p<0.05) and for Item 13 (F(2,42)=5.1, p<0.05).

Finally we had two items, Item 14 and Item 15, where users had to provide a more subjective view of the situational picture. Our participants scored high on these items in all three situations, all above 4, but there were no signiﬁcant diﬀerences between the sessions or organisations.

5.3.2 Collaborating through the Augmented Reality system

The purpose of the AR application was to support collaboration, hence this section presents the result of the collaboration using the paper map compared to using the AR system the second time, see Table 5.1. As noted in the previous section the participants experienced the AR system as easier to work with and through the second time, which illustrates the importance of training and repeated use of the technology, which was also highlighted by one of the participants:

"Good! It gives a credible situational picture and when you are secure in using the system my hope is that you can focus more on

( 142 ) Chapter 5. Collaborative multi-user study 127

your task and less on verbal communication." (RS0924, question 3:6)

The participants all have signiﬁcant amount of training and practice on how to work with paper maps and the tools provided in the paper map session. This makes a comparison between the second time using the AR application with the paper map slightly more relevant than an overall comparison between both AR sessions and the paper map session. Using one way ANOVA with Bonferroni post hoc tests we found interesting diﬀerences between the paper session and the AR session on Items 7, 8, and 9, as well as a tendency on Item 13. In general the results on the questionnaire were positive for the AR system, see Table 5.1.

On Item 1, It took a long time to start to cooperate, the participants do not experience that the AR system inﬂuences the time it takes to start to cooperate, nor does the paper map, both have high scores, 5.33 for both the AR system and the paper map. That the paper map does not inﬂuence the time it takes to start to cooperate is not surprising as this is what they are used to.

When asked if it was easy to collaborate, Item 2, the results were positive in both sessions - the mean score was 4.70 (AR) and 5.09 (paper map) on a 6 grade scale, no signiﬁcant diﬀerence.

When asked about the AR system as a tool for collaboration, Item 3, the average scores were above 4 in both sessions. There was no signiﬁcant diﬀerence between the paper map and the AR system sessions.

Concerning whether or not the participants enjoyed the collaboration, Item 4 The co-operation was fun, the scores were 5.29 for the AR system and 5.21 for the paper map. There was no signiﬁcant diﬀerence between the sessions.

On Item 5, I felt that the group controlled the situation we note a lack of significant difference between the paper based session and the AR session. The use of the AR system does not seem to result in an experienced loss of control in comparison to the paper map, despite that it is more difficult to get eye contact with the head- mounted display. In the paper map session the average score was 5.0 in comparison

( 143 ) 128 Chapter 5. Collaborative multi-user study to an average of 4.79 in the AR session. The importance of presence in the task was also noted by one participant:

"I felt that we could cooperate in a good way with this technology since we could see each others units. It became one operation together instead of like it is today when we work in diﬀerent places although it is the same event." (P0930, question 2:4)

Closely related to the ability to see each others vehicles is general information sharing, which was targeted in Item 6, It was easy to mediate information between the organisations. The overall average score on the item was 4.83 using AR and 5,0 in the paper session. However, there was no signiﬁcant diﬀerence between the AR session and the paper map session which means that sharing information was experienced as easy to do while working on the paper map as with the AR system.

The real time overview given by the AR system is a major contribution to the creation of a common ground for the collaborative work as illustrated by this quote:

"It was easy and clear to see the others units. Good that you can point on the map with your hand and in that way show where you mean, good that you see the others point in order to help each other out. That you are in the same room, with the same map simpliﬁes tremendously." (HP0930, question 1:4)

A group of questionnaire items specifically addressed the map and the symbols on the map; Item 7, Item 8, and Item 9. Here the scores for the AR system are higher than for the paper map, Figure 5.11, suggesting that the use of the AR system made it easier to achieve a common situational picture. Regarding the map, Item 7 The map made it easy to achieve a common situational picture, we only see a tendency to difference between the AR systems mean score of 5.04, and the paper maps mean score of 4.18 (F(2,42)=6.1, p≈0.052). Regarding the symbols, Item 8 The symbols made it easy to achieve a common situational picture, there is a significant difference between the mean scores of 4.83 and 3.42. The symbols in

( 144 ) Chapter 5. Collaborative multi-user study 129 the AR system made it easier to achieve a common situational picture compared to the paper map (F(2,42)=15.3, p<0.05). The map is also regarded as less messy when using the AR system, Item 9 The map became cluttered, with a signiﬁcant diﬀerence (F(2,42)=12.7, p<0.05) between the mean score of 4.0 for the AR system and 2.38 for the paper map.

Figure 5.11: Results from Items 7, The map/AR system made it easy to achieve a common situational picture, 8, The symbols made it easy to achieve a common situational picture and 9, The map was cluttered/messy. The AR system scored signiﬁcantly higher than the paper map. See text for further details.

We also note that the users wanted even more information than we had on the map, Item 10 I would have liked to have had more information than what was available, scoring rather low (mean score of 3.29 for the AR system and 3.17 for the paper map) on this item. The participants had, however, no problems to interpret the symbols and the map, giving Item 11, I felt that I was certain that I could interpret what was on the map, a mean score of 4.54 when using the AR system and 3.75 with the paper map.

When asked if the map and symbols helped the participants trust the situational picture, Item 12 and Item 13 there are diﬀerences. Concerning whether the map helped the users trust the situational picture, Item 12, we have a tendency to a diﬀerence between the paper map (mean score 3.67) and the AR system (mean score 4.67), F(2,42)=4.6, p≈0.051. The symbols, Item 13 The symbols helped me trust the situational picture, helped the users more using the AR system (mean score 4.58) than the symbols on the paper map (mean score 3.50), F(2,42)=5.1, p<0.05, see Figure 5.12.

( 145 ) 130 Chapter 5. Collaborative multi-user study

Figure 5.12: Results from Items 12, The map/AR system helped me trust the situational picture, and 13 The symbols on the map helped me trust the situational picture. There are tendencies to diﬀerence between the sessions, see text for further details.

Finally we had two items, Item 14, I thought I had a good situational picture and Item 15, I thought the others had a good situational picture, where users had to provide a more subjective view of the situational picture. Our participants scored high on these items in all three situations, all above 4, but there were no signiﬁcant diﬀerences between the sessions. One contributing factor to the scores as pointed out by a participant, may be the co-location aspect of the application:

Overall the comparison between the AR system and the paper based map shows that the AR system is as good as or better than the paper map in many respects.

5.3.3 Evaluating the Augmented Reality system

The questionnaire used to evaluate the AR system contained items speciﬁcally addressing the use of the AR system and did not include other aspects such as collaboration, see Table 5.2. The queries are based on the questions used in the

( 146 ) Chapter 5. Collaborative multi-user study 131 previous single user application studies Nilsson and Johansson (2007, 2006a), and here modiﬁed to reﬂect the task carried out in this study.

Table 5.2: AR system questionnaire, average score and lower/upper conﬁdence bound on the 6 point Likert scale. As the statements in the questionnaire were both positively and negatively loaded (see for instance the ﬁrst two items), the scores on the negatively loaded items were transformed in order to make the result easier to interpret. This means that in the table a high score is positive for the AR system and a low score is negative for the AR system

The participants found the system easy to use and learn, as seen in Item 1, It was easy to use the AR system and Item 5, It took a long time to learn to use the system, with the mean scores of 4.21 and 4.963 respectively. They had only used

3Note that we have transformed the scores for consistency in the table meaning that a high score on this item is positive for the AR system, i.e. positive, indicating it did not take a long time

( 147 ) 132 Chapter 5. Collaborative multi-user study the AR system that day but had no diﬃculty using it. Learning to cooperate using the AR system was not a problem either, Item 18 It took a long time to learn how to cooperate using the AR system, scored 4.67.

The participants liked to use the system. They gave high scores, 4.46, on Item 9, I would like to use the AR system in my work. They were slightly less interested to use the system in other situations, Item 10, scored 3.79.

On the general open ended question on what they thought of using AR technology in these types of situations in their everyday professional life, What was the best about the paper MR system for collaboration (one or several things)?, several users pointed out that they do think that this technology can be a reliable help:

"Good! It gives a credible situational picture and when you are secure in using the system my hope is that you can focus more on your task and less on verbal communication" (RS0924, question 3:6) 4

The quote also highlights the key issue of training and practise in order to fully take advantage and feel secure in the technology they use:

"Fun, but takes practise so people are comfortable using it." (HP0925, question 3:6)

The participants trusted the system, Item 10 I felt conﬁdent that the AR system gave me correct information scored 3.86.

The questionnaire also addressed the AR system display, in Item 2, Item 3 and Item 4. The users had no problems reading the map due to the colours, Item 2 The map was hard to read due to the colours (mean score of 4.6), but on Item 3, The map was hard to read due to picture resolution, we see that the users are less positive (mean score of 3.1). We believe that this is due to the instability of the image,

4In the following text quotes of the participants are coded as follows: the ﬁrst letter/s indicate organisation (P-Police, RS- Rescue services, HP- Helicopter Pilot), the following four numbers are the team number, and the ﬁnal number combination indicates which questionnaire and open ended question the quote is related to.

( 148 ) Chapter 5. Collaborative multi-user study 133 i.e. when they focus on the outer parts of the map the system sometimes loses the marker and hence the image projected to the user, as explained in the open ended question What were the worst aspects of the AR system for collaboration?:

"That the map image disappeared when I looked at the edges of the map. I felt somewhat isolated from the surrounding." (P0925, question 4:2)

The users did not use the ability to switch between showing all symbols and only their own symbols as frequently as we had anticipated, and consequently they found the map a bit cluttered, Item 4 The map was hard to read due to the symbols scored a mean of 3.38. However, the AR system seems to be experienced as less cluttered than the paper based system, see Item 9, The map became cluttered, in the questionnaire on collaboration, Figure 5.11. We believe the main reason for not using the functionality allowing them to declutter the map by choosing to see only selected symbols, is that it was a bit too cumbersome to use the interaction device. Users had to manually select each organisation that they did not want to see in a menu.

When asked about speciﬁc features in the AR system in the open ended section of the questionnaire, 14 of the 27 participants said that they did use the feature allowing them to see additional information about the objects on the map How often did you use the possibility to see additional information?. Of the remaining participants several would have used it if they had had more time and/or training.

"My ambition initially was to to work with i . However, I chose not to focus on this in order to save time. It’s a necessary feature for the future in my opinion" (RS0924, question 4:6)

The interaction device was evaluated in Item 8, I thought that the interaction device was easy to use (mean score of 3.9), and to a certain extent in Item 12, The AR system was clumsy to use (mean score of 3.9), and Q13, The AR system had no major ﬂaws (mean score of 3.0). The rather low scores on these items can

( 149 ) 134 Chapter 5. Collaborative multi-user study to some extent be explained by the result from the responses to the open ended question What are your general impressions of the AR system?:

"It needs further development in order for it to be more user- friendly" (RS0929, question 4:3)

The main issues of concern for improved user-friendliness are related to the symbol menu and the lack of shortcut buttons on the joystick interaction device:

"Flipping through units and icons. There is no need for RS [rescue service] vehicles in the police joystick for example." (P0923 question 4:3)

The interaction device and the symbol management in the design is also the main concerns for the participants when asked about what they would change in the system What would you change in order to improve the collaboration over the paper map/MR system?:

"Make the symbol management simpler." (HP1001, question 3:5)

"Move the symbols by pointing and dragging the ﬁnger." (P0925 question 3:5)

It is evident that the open approach derived from the design phase, where the decision was made to make all symbols visible to all participants, was misguided. This illustrates one of the diﬃculties in all design processes - a feature may be desirable in one cycle of development but perhaps not in the next (the envisioned world problem). Including more iterations in the process will likely reduce this problem.

Two items addressed the number of symbols, Item 6 There were too many symbols in the AR system and Item 7 There were too few symbols in the AR system scored 4.58 and 3.86 respectively. Again positive results, maybe adding some symbols, which poses no problems.

( 150 ) Chapter 5. Collaborative multi-user study 135

Addressing the more ergonomic or physical aspects of the system were Items 14, 15 and 16. As can be seen in Table 5.2 the users did not experience feeling sick, Item 14, dizziness, Item 15, nor did they feel discomfort, Item 16, due to using the AR system. The discomfort they felt appears to be mainly due to the head-mounted system. It became too heavy after a while as illustrated by this quote in response to a question about the negative aspects of using the system Did you experience anything as troublesome, and if so, what?:

"Standing for a long time with a front-heavy headset." (HP0925, question 4:4)

One important aspect of interacting with systems is whether or not the users enjoy working with the system. The result indicate that they did, which is evident by viewing the score on Item 17, The AR system was fun to use. The result has an average score of 5.6 on a 6 point scale. In the open ended question regarding any positive aspects of the AR system, several issues were highlighted Did you experience anything as positive, and if so, what?:

"Easy to get an overview, good to get information about people, hours of every unit. Also the possibility to see other units information. Easy to move units." (HP0930, question 4:5)

"Good and clear situational picture that is shared and updated." (RS0930, question 4:5)

"You learn quickly. Good situational picture." (P1003, question 4:5)

These responses are important in relation to the purpose of the application. The aim of the study was to implement and develop a system supporting the establishment of a common situational picture among participant from diﬀerent organisations. The quotes above illustrate that in many aspects the AR system succeeded to aid the users in achieving an overall situational picture. However, as the results indicate there are several important issues to address in terms of

( 151 ) 136 Chapter 5. Collaborative multi-user study interaction design, symbol management and the general physical design of the system.

The open-ended part of the questionnaire also included a question regarding whether the participants could see any other potential use for AR technology than this application and several users did Is there any other situation where a system like this would be useful?:

"Education, team work which requires a separation of tasks." (HP0929, question 4:9)

"Practice navigation on your own ship but still in dock, i.e. you ﬁne tune the things you’re supposed to do in reality. Real time supervision of divers in the water, i.e. you have a sea chart with 3D (available in military environments)." (P1003, question 4:9)

A common theme for most suggestions of potential use of AR include training, simulation and strategy testing before an operation. The last quote above does however indicate another direction of real time use of AR in an ongoing operation or event as does this quote:

"An extremely interesting and exciting way to build an operation. I can already now see the advantages with placing diﬀerent groups and units in connection to a larger football [soccer] command." (P0930, question 4:3)

The final part of the user study was the focused group discussion during which the participants discussed the experiences of the study, the AR system and their collaboration. One interesting issue raised during several of these focus group discussions were how the AR system could be used to share and distribute the command team’s situational picture to other people outside the control room. For instance the potential to work collaboratively but distributed as they do now but instead of having only telephone contact with commanding strategic officers at ’home’ they could all share the same view of what is going on in the field, in real time.

( 152 ) Chapter 5. Collaborative multi-user study 137

5.4 Discussion of the results

On many items the participants scored the system higher in the second session with AR (AR2) as compared to the first session (AR1). This indicates the necessity of more than one trial or session with the AR system. This is probably valid in most studies examining new technologies. If the study had been designed with only one AR session (apart from the training session) the results would have been less positive for the system. This would not have been a fair comparison towards the baseline session as the participants are all familiar with paper maps but have never before encountered a system like the one in this study. Another aspect of several sessions is the social effect of collaborative work. As several participants pointed out in the questionnaire, it became easier to both use the system and communicate with each other in the second AR session. This is partly due to the training effect on the AR system, but also due to the fact that the participants got to know each other better.

The participants experienced that it was simple to understand the symbols provided, although they requested to have even more symbols to choose from. Adding symbols, and information to symbols, is not a big issue. Again, our focus in this study was not to evaluate a complete system, but the prospects of using AR for cooperation.

5.4.1 Information sharing and collaboration

The information sharing aspect of the system turned out to be equivalent in the AR system and the paper map which is a very promising result. The current technical solution, camera see-through, causes a lack of direct eye contact which could be a drawback as gazing is used frequently as an indicator of focus in face-to- face communication. Despite the lack of eye contact the participants felt that they could easily share information among each other. This could be explained by the AR system’s ability to present a common situational picture when everyone sees what everybody else does with their resources directly. This reduces the need to

( 153 ) 138 Chapter 5. Collaborative multi-user study actively request and present information as part of the information sharing process. The ability to see each other’s units may also have strengthened the perception of them being a team rather than just participants of their respective organisations as illustrated by the quotation (repeated from the previous section):

The diﬀerence between the organisations regarding the AR systems potential as a collaborative tool may perhaps be explained by the diﬀerent responsibilities and experiences of the organisations in operations like the one in the scenario. While the police participants in the focus group discussion commented on the lack and need of new technology in their organisation it seemed to be the opposite for the other two organisations. The rescue services had just recently installed new systems in their control room environment and the helicopter pilots are currently going through a process of changing the current helicopter technology into new. This could explain why the participants from the police department were more positive towards adding new technology to their workplace than the other participants.

There are of course problematic issues with the system, and primarily these problems were related to interaction aspects rather than the content of the information. This was a drawback in the design - the tested system was a prototypical system and not all design choices had been evaluated. In order to have openness in the system all organisations could see all available symbols in the menu, which means that they also needed to go through them while changing the resources and contents of the map. This was not experienced as a major problem in the pre study trial and was hence neglected in the design process. Clearly this is an aspect of the system that needs to be addressed in future development. However, the participants did not consider any of the symbols unnecessary, but in further development the interaction will be re-designed. Personalising the menu and symbol selection will make the interaction quicker and not as time consuming.

( 154 ) Chapter 5. Collaborative multi-user study 139

However, the positive aspects of the new technology seemed to outweigh the negative in most instances, and this quote can perhaps illustrate why:

"So many advantages to have a complete overview." (P0924, question 1:6)

In emergency management and collaborative command and control operations the general overview of the situation is important for the team to achieve a common picture of the ongoing event. Having a complete overview of available resources and where they are located is invaluable for information sharing and decision making. The real time overview given by the AR system is a major contribution to the creation of a common ground for the collaborative work.

The main goal of the study was to evaluate the potential of the technology as a collaborative tool, and not to evaluate a ﬁnished design, and as such the study was successful. The AR system was in many aspects comparable to working with a paper map and adding ability to see and manipulate digital symbols and objects seem to help create a common ground for the collaborative work. Many aspects of common ground can of course not be solved simply by adding symbols and objects to a map but the symbols and objects on the map can certainly help illustrate the current state of the joint activity (the second essential component of establishing common ground according to (Clark, 1996)). It could also be possible to support (although not tested in this study) the users by allowing them to ’playback’ the situational picture, thereby helping them to understand why the current situation looks the way it does (thus supporting also the third component of common ground). This could be helpful especially in shift work when handing over to a new team.

Even though the AR system is not yet ﬁnal in design and implementation, the professional opinions and comments gathered in this study are important for future design and implementation. The results also clearly show the importance of an iterative design process for AR applications. And as a new system it performed particularly well as the evaluation indicates that in many aspects the users experienced the AR application to be equally as good for collaboration as

( 155 ) 140 Chapter 5. Collaborative multi-user study the paper map. It may not be able to replace a paper map in the near future (for technical and regulatory reasons), but it deﬁnitely holds the potential to strengthen collaboration by augmenting the shared situational picture.

5.4.2 The iterative design process

Working iteratively with re-design and evaluation, involving real users is invaluable for several reasons. Firstly, it allows us to cope with the envisioned world problem – by moving from making major changes to the system to minor changes, the discrepancy between the envisioned usage and the actual usage slowly erodes. It is however important to remember that the process is two-sided; the original vision also changes as the designer begins to understand possibilities and limitations with the system.

The iterative design of this application is by no means ﬁnished with this end user study. As noted in the results, the interaction device needs to be carefully designed to facilitate a situation where users not only access and modify their own objects and information but also can access and modify objects and information from other users. Not only is the interaction more complex than in a single user application, as there are many more symbols to manipulate, users also manipulate their own symbols more frequently than the others’ symbols. Regarding speciﬁc design features the next iteration of this application will make sure to simplify the interaction modality even further, taking note to the participants comments. The click-and-drag feature requested by one participant was considered during the development phase but was unfortunately not implemented due to time/resource constraints at that time, but has been implemented since. The menu has also been restructured to allow faster navigation.

5.4.3 Using Augmented Reality to establish common ground

One of the most interesting ﬁndings in this study was the fact that the participants on most issues gave the AR system an equal or better score than the regular

( 156 ) Chapter 5. Collaborative multi-user study 141 paper map. The quote by one of the fire fighters above; "Good! It gives a credible situational picture and when you are secure in using the system my hope is that you can focus more on your task and less on verbal communication" (RS0924, question 3:6) gives a clear indication that the AR application designed has in one important aspect reached it’s purpose. One of the most problematic issues during command and control work in the field is the messy and cluttered map over ongoing events. As several different actors give their input, and their view of what is going on and what needs to be done in what order, the notes and sketches tends to pile up, leaving a very difficult to read map (or white board or paper) to interpret. The AR system allows all individuals to work on the same map, or in the same interaction space, both individually as well as collaboratively. But the added benefit of the AR system, compared to a paper map, is that it is possible to quickly switch perspectives and follow one organisation at a time, as well as see the overall view of all available resources and their status and distribution. The experience of the AR system as less messy and cluttered than the paper map (Item 9 in the first questionnaires) illustrates this issue. Even though the participants felt that they had a good situational picture in both settings, the clutteredness of the paper map compared to the AR map significantly affected their rating of the paper map versus the AR system. The participants see a great potential in the AR system to present them ’with a credible situational picture’ allowing them to focus more on their actual task at hand rather than spend time on verbal communication, talking about what resources are where and when.

5.5 General observations and conclusions of the multi-user study

Paper maps are something that these participants are very used to working with, whereas this study was the ﬁrst time they ever encountered AR technology. The high scores given to the AR system indicate that they actually can perform their tasks to the same level of satisfaction as they normally perform, i.e. with a paper

( 157 ) 142 Chapter 5. Collaborative multi-user study map. The participants did not consider it more diﬃcult to achieve a common situational picture with the AR system than when using an ordinary paper map, nor did they regard the AR system to interfere with their communication to any large extent.

Designing a single user AR application with a sequential interaction scheme is in many respects relatively easy compared to designing applications for multiple users performing dynamic interactive tasks. Simply converting a single user application into multiple users may be technically relatively easy but the complexities of the task demand careful consideration before adapting a single user system into a multiple user one (which is often the case in many computer supported collaborative applications (Fjeld et al., 2002)).

The results of the study clearly indicate that the AR system was experienced as a possible future system not only for the task used in the scenario but also for other tasks within the three organisations. AR technology is not limited to creating and maintaining a common situational picture, it also has potential for training and information sharing throughout the chain of command and control.

( 158 ) Chapter 6

Discussion of the user studies

When introducing new systems, like AR, in an activity, user acceptance is crucial. The overall results, from both the individual and the multi-user application studies, shows a system that the participants like rather than dislike and the studies indicate that the participants would like to use AR systems in their future professional life.

6.1 General observations

Interactivity is an important part of direct manipulating user interfaces and also seems to be of importance in an AR system of the kinds investigated in these studies. A couple of the participants who that in the single user studies did not think it was possible to compare AR and human instructions, motivated their response in that you can ask and get a response from a human, but the AR system in the study did not have the ability to answer random questions from the users. Adding this type of dialogue management and/or intelligence in the system would very likely increase the usability and usefulness of the system, and also make it more human-like than tool-like. However, this is not a simple task, but these responses from real end users indicate and motivate the need for research in this direction. Utilising knowledge from other ﬁelds, such as natural language

143

( 159 ) 144 Chapter 6. Discussion of the user studies processing, is one direction that would be interesting not only for HMD based AR systems but perhaps also for mobile handheld AR applications, especially since mobile phones already have the hardware needed for verbal interaction.

Human face-to-face interaction is perhaps the most common way to receive instructions on how to operate new equipment, and by imitating this personal interaction the AR system illustrates a big advantage to other forms of instructions. The instructions are presented in real time right in the user’s ﬁeld of view allowing users to experience the instructions in actual context. This has the potential to reduce cognitive load in that the user does not have to transfer information from for instance a manual to the real world task. By confronting a design for a new technology with tasks from real work rather than laboratory settings allows problems that probably otherwise would remain unknown to emerge. In a laboratory experiment or evaluation, the user is confronted with a situation based on an assumption about what is relevant that often originates from the researcher/evaluator. A common approach in many experimental studies is to present the experiment participant with a "nonsense" task, deliberately removing any connections to actual work. The CSE approach instead suggests that conclusions about pros and cons of a new system should be drawn in its intended use setting. This is especially true when concerned with tools that are to be used by specialists in high-risk environments. A product like the ESG or the troakar represents this kind of tool used for a high-risk task by skilled professionals. It is also clearly a case where improved functionality in terms of instructions is beneﬁcial, since objective instructions has a potential to reduce erroneously performed assemblies. Testing the concept of providing AR instructions together with these instruments in a real-world setting thus reveal the pros and cons of a design.

Another aspect of the system that becomes evident when observing users in their actual work environment is its physical appearance and bulkiness largely originating from the technical limitations of using video see-through HMDs. Despite some physical issues with the AR system the users did complete their respective tasks without any other assistance. However, eﬀects of the physical

( 160 ) Chapter 6. Discussion of the user studies 145 intrusion of the system upon the user’s normal task should not be ignored. Even if the system is lightweight and non–intrusive, it still may change the task and how it is performed. In the worst case users may abandon the system if they experience it as being too clumsy. Regardless of appearance, any system does affect the social context of the task simply by being introduced as a new element. However, this may not be a problem in the long run, if the system is a positive influence on the task, user and context, it will with time and experience grow to be a part of the task. This should be true for AR as it has been true for other technologies; for instance the use of computers for writing papers - few students and researchers today can image writing papers without the cut-and-paste functionalities of word processors. To fully see the effects of introducing an AR system into a hospital or emergency management environment a more longitudinal study may be required.

One aspect, not to be forgotten, is the positive results of the question of enjoying the use of the system that was included in all studies. A majority of the participants found the AR system fun to use and work with, and several of them also wanted to use it as a part of their work. The technology acceptance model implies that the perceived ease of use and perceived usefulness of a system is what determines if users will actually use it in the end. The experienced ease of use is most likely inﬂuenced by the enjoyment of using a system. If it is fun to use, it may be experienced as easier to use, and vice versa. Entertainment in varying degrees is an important usability factor for all user interfaces, even though there are obvious examples of experienced usefulness being more important than both ease of use and entertainment. Although user acceptance of the AR system in the studies presented was high, there are several issues to overcome before AR systems can be a reliable part of work in these environments. In the current setup the AR system has some important shortcomings - the lack of dialogue for one thing. As pointed out, the user must have the ability to interact more freely, for instance by asking questions or issue commands.

( 161 ) 146 Chapter 6. Discussion of the user studies

6.2 Introducing Augmented Reality in end user environments

AR is a relatively new field in terms of end user applications and as such, the technological constraints and possibilities have been the driving forces influencing the design and development. This techno-centred focus has most likely reduced the amount of reflection that has been done regarding any consequences, other than the technical, of introducing the technology in actual use situations. The impact of the way AR is envisioned (optic see-through and video see-through) has largely taken focus off the use situation and instead lead to a focus on more basal aspects, such as designing to avoid motion sickness and increasing the physical ergonomics of the technology. However, these areas are merely aspects of the AR platform, not of the applications it is supposed to carry and the situations in which they are supposed to be used. Studies of AR systems require a holistic approach where focus is not only on the ergonomics of the system or the effectiveness of the tracking solutions. The user and the task the user has to perform with the system need to be in focus throughout the design and evaluation process. It is also important to remember that it is not always about effectiveness and measures; sometimes user attitudes will determine whether or not a system is used and hence it is always important to look at the actual use situation and the user’s attitude towards the system.

The purpose of the system is another important issue when evaluating how useful or user-friendly it is – is it intended for pleasure and fun or is it part of a work setting? If it is somewhat forced on the user by it being part of everyday work and mandatory tasks, the system needs to reach efficiency standards that may not be equally important if it is used as a toy or entertainment equipment. If the system is a voluntary toy the simplicity factor is more important than the efficiency factor. However, if a system is experienced as entertaining, chances are it may actually also be perceived as being easier to use. It is not a bold statement to claim that a system that is fun and easy to use at work will probably be more appreciated than a system that is boring but still reaches the efficiency goals. However, as the

( 162 ) Chapter 6. Discussion of the user studies 147 technology acceptance model states, if the efficiency goals are reached (i.e. the users find it useful) the users will most likely put up with some hassle to use it anyway. In the case of the user studies presented in Chapter 4, this means that if the users actually feel that the AR instructions help them perform their task they may put up with some of the system flaws, such as the hassle of wearing a head mounted display or updating software etc, as long as the trade off in terms of usefulness is good enough.

As discussed previously, there is a chance that the usability focused methodology measures the wrong thing, many interfaces that people use of their own free will (like games etcetera) may not score high on usability tests with novice users as test participants, but are still used on a daily basis. It can be argued that other measures need to be developed which are adapted to an interface like AR. Meanwhile, the focus should not be on assessing usability but rather the experienced usefulness of the system. If the user sees what he/she can gain by using it they will most likely use it despite usability tests indicating the opposite.

The field of AR differs from standard desktop applications in several aspects, of which the perhaps most crucial is that it is intended to be used as a mediator or amplifier of human action, often in physical interaction with the surroundings. In other words, the AR system is not only something the user interacts with through a keyboard or a mouse. The AR system is, in its ideal form, meant to be transparent and more a part of the users perceptive system than a separate entity in itself. The separation between human and system that is common in HCI literature is problematic from this point of view. By using an AR system the user should perceive an enhanced or augmented reality and this experience should not be complicated. Although several other forms of systems share this end goal as well, AR is unique in the sense that it actually changes the user’s perception of the world in which she/he acts, and thus fundamentally affects the way the user behaves. Seeing virtual instructions in real time while putting a bookshelf together, or seeing the lines that indicate where the highway lanes are separated despite darkness and rain will most likely change the way the person assembles the furniture or drives the car. This is also why the need to study contextual

( 163 ) 148 Chapter 6. Discussion of the user studies eﬀects of introducing AR systems seems even more urgent. When evaluating an AR system, focus has to be on the goal fulﬁlment of the user-AR system rather than on the interface entities and performance measures gathered from evaluation of desktop applications. This approach is probably valid in the evaluation of any human machine system, but for historical reasons, focus often lays on only one part of the system.

AR as an interaction method for the future is dependent on a new way of addressing usability, if the focus is kept on scoring well in usability tests maybe we should give up novel interfaces straight away. But if the focus is on the user’s subjective experience and level of entertainment or acceptance, AR is an interactive user interface approach that surely has a bright future.

6.3 Concluding discussion of the studies

The user studies all illustrated that methods like usability evaluations and user studies in AR systems can be done with an approach that is usable for the specific AR applications in focus. A general comment on the single user studies is that even though they are set at a hospital, they still have similarities with a laboratory setting. The scenarios are instructional class room type tasks, and not a part of regular every day work although it is part of their introduction to the work place. But what is more important is that the participants in all studies, both the single and multi-user, are potential real end users. This makes the methodology used in the studies more contextual than other traditional usability evaluations where the user background and experience is not usually part of the participant criteria, but rather treated as a demographic fact in the study. The approach in the user evaluations in this thesis was more similar to participatory studies, where real end users were involved in the development of the applications as well as the evaluation. This approach to usability in AR systems has not been a dominating influence on research in the field, but as illustrated by the studies presented in this thesis, it is an approach worth using.

( 164 ) Chapter 6. Discussion of the user studies 149

In all studies, as well as the introductory sections of this thesis the problems asserted with using traditional HCI measures to assess and evaluate non-traditional systems such as AR systems have been discussed. The first single user study (the ESG) provided knowledge about the experience of a particular user group when they performed a task with the aid of AR instructions. The study was a user study, where the analysis of the results was made with a CSE approach. One finding in this study was that the participants did not relate to the AR system as a computer, but rather found the technique as a new way of interacting. This supports the arguments that approaching HCI issues in these systems is perhaps not best done by applying the traditional approaches and methods. The analysis of the system may give more insight to the user experience and future design solutions if the focus is on more abstract and qualitative measures, such as user experience, rather than having a focus on time, error rates and quantifiable data. The general aim of the interaction and the joint cognitive system was in focus throughout the setup of the study and the analysis of the user study. This did provide a basis for discussing the results not in terms of quantitative data, but rather in terms of qualitative observations about the AR joint cognitive system. In this respect the CSE approach was imperative.

The second and third study (the two troakar studies) revisits the issues raised by the first study, and presents a user study where a new case application is evaluated. One issue that could be identified in the first user study was that the participants preferred instructions form a human instructor, a co-worker or senior colleague at their work place. In relation to this, an aspect that was found lacking in the MR system was human-like traits such as interactivity – the ability to ask questions and get responses from the system. Ideally some kind of intelligence should be implemented in the MR system, with dialogue management and voice interaction capabilities. The dialogue management problem has proven to be too complex to implement in the current project, but the voice interaction implementation was a step that could be taken. So, as a result of the first study, the system was redesigned and enhanced with voice control capabilities and the system did not only present visual instructions but also audio. That is, the participants in the

( 165 ) 150 Chapter 6. Discussion of the user studies

ﬁrst troakar study received instructions both verbally and visually, and were able to respond verbally to the system. For some of the users this seemed to give a more human-like interaction with the system. In general, the results in the study were similar to those of the ﬁrst study, although the redesign of the system did result in improved user experience.

The multi-user study was based on the same methodological user centered approach as the single user studies and Chapter 5 describes the co-located collaborative AR application which enables the users to share information and actively take part in an ongoing scenario. The aim of this study was not to evaluate performance, but rather to evaluate the potential of the AR system in supporting the creation of a common situational picture in a collaborative team task and the user experience of the AR system. The analysis was focused on the collaborative aspects of AR. In general the results indicate that users are positive towards the system, and that they experience it as a support in building and maintaining a shared situational view. The results of the study illustrate that the participants could work as successfully with the AR application as with the paper map. The system is thus a good candidate for a future collaborative support system for helping teams create common ground while coping with dynamic tasks.

The particular problem of enabling a shared view of gestures at the same time as presenting a virtual background was perhaps a consequence of adapting single-user AR technology to a multi-user setting. The system now allows augmentation of not only the individual users view but it allows each user to eﬀect and change their team members view of the ongoing situation, which is fundamental to the deﬁnition of a collaborative AR system. This illustrates the potential of AR, not only in shaping collaborative real world tools, but also as a tool to study collaborative work on a more theoretical level for future applications. A future research focus could be to examine how AR can be used to support commanders working with a shared representation in a distributed setting. As the system supports deictic gesturing and collaborative manipulation of symbols even when the users are located separately, this could provide an interesting possibility.

( 166 ) Chapter 7

Conclusions

One aim of this thesis has been to illustrate how a systemic approach can be used in the development and evaluation process of AR applications. During the ﬁrst study it became apparent that the end users did not perceive the AR system as just another computer application. This insight was valuable for the development and evaluation of the following studies – both in terms of how the interaction was implemented but also in terms of evaluation methods. Table 7.1 gives an overview of the most important results from the user studies presented in this thesis, as well as illustrates how the earlier studies informed the later studies.

What can be concluded from the studies is also that the main area of improvement and development for these applications have been the interaction aspects. This is in line with the point made by Billinghurst et al. (2009) where they point out that this is the aspect that has been lost in the development of AR systems as much focus has been on more technical issues like tracking and registration while the interaction aspect of AR has been left out. AR users have for the larger part only been passive recipients of information, rather than an active participant in an interaction with the virtual information that augments their environment. In order to achieve the ideal visionary, non intrusive and intuitive Augmented Reality system, the interaction with the system must be seamless and natural much like everyday human communication. Technologies like gesture based, and haptic as well as speech based interaction need to be further incorporated into 151

( 167 ) 152 Chapter 7. Conclusions

Table 7.1: A summary of the results and main contributions of the end user studies.

AR systems. Another type of interaction that is being explored is the use of eye gaze interaction (Nilsson et al., 2007, 2009). Eye gaze is a very important part of human communication and can also be a part of the users communication with the system.

In the health care examples, where the application was rather static and sequential in nature, the results show that an instructional AR application, used as a support system for when no experienced professional is around, can very well be implemented as of today. With the expanding market of new capable platforms in the shape of smart phones, it should be relatively easy to create instructional AR applications for any assembly task. However, as the studies indicate, handheld devices are not ideal, as they are more intrusive on manual tasks limiting the users ability to interact naturally with their other tools, so perhaps the signiﬁcantly useful AR implementations will have to wait for the "ultimate HMDs" that will hopefully be available in the near future.

Although the main commercial area for AR applications now seems to be mobile phone based applications, much can be learned from studies of HMD systems.

( 168 ) Chapter 7. Conclusions 153

In regards to the multi-user application described in chapter 5, my conclusion is that before any implementation into the real user context can be done, there is a need for further studies on organisational effects of the technology. This is in order to prevent the new technology from interfering with effective work practices and to avoid undermining years of experience and training in how to collaborate in dynamic emergency management tasks. The study illustrated that AR has potential in this domain and that applications can be built to support this user group; The participants in the study appreciated the positive effects of the ability to access a shared, yet personalised, view of an ongoing complex scenario. In addition to this the AR application as it is today can also be used as a tool for training and relatively low-cost simulated exercises for cross-organisational collaboration.

This thesis has presented four end user studies of three diﬀerent AR applications and the lessons learned on design and development from all the studies can be summarised as follows:

• Involve real end users in the design of the system/application development process

• Involve real end users in the design of the user study task

• In order for real end users to feel involved in the task, make sure the task, or scenario, is realistic and not artiﬁcial

• Do several design iterations

• Involve real end users in the evaluation of the application

These conclusions are valid for both single user and multi-user applications and are perhaps the only general guidelines that can be identiﬁed for the development and evaluation of AR systems. All applications must be developed with the end user in focus, and in order for an application to be used in a real work environment it has to be part of a real world solution and not only a technical gimmick.

( 169 ) ( 170 ) Bibliography

Agar, M. (1996). The Professional Stranger: An Informal Introduction to Ethnography. Academic Press, San Diego, California.

Artman, H. and Waern, Y. (1999). Distributed cognition in an emergency coordination center. Cognition, Technology & Work, 4(1):237–246.

Ashby, W. (1956). An Introduction to Cybernetics. Chapman & Hall, London, England.

Azuma, R. (1997). A survey of augmented reality. Presence, 6(4):355–385.

Azuma, R., Baillot, Y., Behringer, R., Feiner, S., Julier, S., and MacIntyre, B. (2001). Recent advances in augmented reality. IEEE Computer Graphics and Applications, 21(6):34–47.

Bach, C. and Scapin, D. L. (2004). Obstacles and perspectives for evaluating mixed reality usability. In Dubois, E., Gray, P. D., Trevisan, D., and Vanderdonckt, J., editors, Proceedings of the IUI-CADUI Workshop on Exploring the Design and Engineering of Mixed Reality Systems, MIXER, Funchal, Island of Madeira.

Billinghurst, M. (1998). Put that where? voice and gesture at the graphics interface. Computer Graphics, 32(4):60–63.

Billinghurst, M. (1999). Shared space: collaborative augmented reality. In Proceedings of the 1999 SIGGRAPH annual conference, pages 178–178.

Billinghurst, M. (2008). Usability testing of augmented/mixed reality systems. In Proceedings of ACM SIGGRAPH ASIA 2008, pages 1–13.

155

( 171 ) 156 Bibliography

Billinghurst, M., Belcher, D., Gupta, A., and Kiyokawa, K. (2003). Communication behaviors in colocated collaborative AR interfaces. International Journal of Human-Computer Interaction, 16(3):395–423.

Billinghurst, M., Cheok, A., Prince, S., and Kato, H. (2002a). Real world teleconferencing. IEEE Computer Graphics and Applications, 22(6):11–13.

Billinghurst, M. and Kato, H. (1999). Collaborative mixed reality. In Proceedings of the First International Symposium on Mixed Reality (ISMR ’99), Yokohama, Japan, pages 261–284.

Billinghurst, M. and Kato, H. (2002). Collaborative augmented reality. Communications of the ACM, 45(7):64–70.

Billinghurst, M., Kato, H., Kiyokawa, K., Belcher, D., and Poupyrev, I. (2002b). Experiments with face-to-face collaborative AR interfaces. Virtual Reality, 6(3):107–121.

Billinghurst, M., Kato, H., and Myojin, S. (2009). Advanced interaction techniques for augmented reality applications. In Virtual and Mixed Reality, Third International Conference, VMR 2009, Held as Part of HCI International 2009, San Diego, CA, USA, pages 13–22.

Billinghurst, M., Kato, H., and Poupyrev, I. (2001). The magicbook – moving seamlessly between reality and virtuality. IEEE Computer Graphics and Applications., 21(3):6–8.

Billinghurst, M., Kato, H., and Poupyrev, I. (2008). Tangible augmented reality. In Proceedings of ACM SIGGRAPH ASIA 2008, pages 1–10.

Billinghurst, M., Weghorst, S., and III, T. A. F. (1997). Shared space: An augmented reality approach for computer supported collaborative work. Virtual Reality Research Development and Applications, 3(1):25–36.

Bimber, O. and Raskar, R. (2006). Modern approaches to augmented reality. Technical report, Mitsubishi Electric Research Laboratories.

( 172 ) Bibliography 157

Bowman, D. A., Gabbard, J. L., and Hix, D. (2002). A survey of usability evaluation in virtual environments: classiﬁcation and comparison of methods. Presence: Teleoperators and Virtual Environments, 11(4):404–424.

Brehmer, B. (1990). Dynamic decision making: Human control of complex systems. Acta Psychologica, 81:211–241.

Brehmer, B. and Dörner, D. (1993). Experiments with computer–simulated microworlds: Escaping both the narrow straits of laboratory and the deep blue sea of the ﬁeld study. Computers in Human Behaviour, 9:171–184.

Broadbent, D. (1958). Perception and Communication. Pergamon, Oxford, England.

Buchmann, V., Violich, S., Billinghurst, M., and Cockburn, A. (2004). Fingartips: gesture based direct manipulation in augmented reality. In GRAPHITE ’04: Proceedings of the 2nd international conference on Computer graphics and interactive techniques in Australasia and South East Asia, pages 212–221.

Card, S. K., Moran, T. P., and Newell, A. (1983). The Psychology of Human- Computer Interaction. Lawrence Erlbaum Associates, Hillsdale, New Jersey.

Caudell, T. P. and Mizell, D. V. (1992). Augmented reality: An application of heads-up display technology to manual manufacturing technology. Proceedings of Hawaii International Conference on System Sciences, pages 659–669.

Checkland, P. (1991). From framework through experience to learning: the essential nature of action research. In Nissen H.-E., Klein H., and Hirschheim R., editors, Information Systems Research: Contemporary Approaches and Emergent Traditions, pages 397–403. North–Holland.

Checkland, P. (1999). Systems Thinking, Systems Practice: a 30-year retrospective. John Wiley& Sons, Chichester.

Checkland, P. and Holwell, S. (1998). Information, Systems and Information Systems: Making Sense of the Field. John Wiley & Sons, Inc., New York, NY.

( 173 ) 158 Bibliography

Cheok, A. D., Yang, X., Zhou, Z., Billinghurst, M., and Kato, H. (2002). Touch- space: Mixed reality game space based on ubiquitous, tangible, and social computing. Personal and Ubiquitous Computing, 6(5/6):430–442.

Clark, H. H. (1996). Using Language. Cambridge University Press, Cambridge, England.

Cousins, J. and Earl, L. (1992). The case for participatory evaluation. Educational Evaluation and Policy Analysis, 14(4):397–418.

Crabtree, A., Rodden, T., and Benford, S. (2005). Moving with the times: IT research and the boundaries of CSCW. Computer Supported Cooperative Work, 14(3):217–251.

Cronholm, S. and Goldkuhl, G. (2003). Strategies for systems evaluation: Six generic types. Electronic Journal of Information Systems Evaluation, 6(2):65– 74.

Crotty, M. (1998). The Foundations of social Research: Meaning and perspective in the research process. Allen Unwin, Sydney, Australia.

Davis, F. D. (1986). A technology acceptance model for empirically testing new end-user information systems: Theory and results. PhD thesis, Sloan School of Management, Massachusetts Institute of Technology.

Davis, F. D. (1989). Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS Quarterly, 13(3).

Davis, F. D. (1993). User acceptance of information technology: System characteristics, user perceptions and behavioral impacts. International Journal of Man-Machine Studies, 38(3):475–487.

Dekker, S. and Hollnagel, E. (2004). Human factors and folk models. Cognition, Technology & Work, 6(2):79–86.

Dewalt, K. and Dewalt, B. (2002). Participant observation. A guide for ﬁeldworkers. Altamira Press, Walnut Creek, California.

( 174 ) Bibliography 159

Dix, A., Finlay, J., Abowd, G., and Beale, R. (2004). Human Computer Interaction. Third Edition. Pearson Prentice Hall, Harlow, England.

Dourish, P. (2004). Where the Action is: The Foundation of Embodied Interaction. MIT Press, Cambridge, Massachusetts.

Dumas, J. S. and Redish, J. C. (1999). A Practical Guide to Usability Testing. Revisited Edition. Intellect Books, Exeter, England.

Dünser, A., Grasset, R., and Billinghurst, M. (2008). A survey of evaluation techniques used in augmented reality studies. Technical Report Technical Report TR2008-02, Human Interface Technlogy Laboratory New Zealand.

Dupuy, J.-P. (2000). The Mechanization of the Mind. Princeton University Press, Princeton, New Jersey.

Engeström, Y. (1999). Learning, Working and Imagining: twelve studies in activity theory. Orienta-konsultit, Helsinki, Finland.

Engeström, Y., Miettinen, R., and Punamaki, R., editors (1999). Perspectives on Activity Theory (Learning in Doing Social, Cognitive and Computational Perspectives). Cambridge University Press, Cambridge, Massachusetts.

Farrington-Darby, T., Wilson, J., Norris, B., and Clarke, T. (2006). A naturalistic study of railway controllers. Ergonomics, 49:1370–1394(25).

Farrington-Darby, T. and Wilson, J. R. (2009). Understanding social interactions in complex work: a video ethnography. Cognition, Technology & Work, 11(1):1– 15.

Fenech, T. (1998). Using perceived ease of use and perceived usefulness to predict acceptance of the world wide web. Computer Networks, 30(1-7):629–630.

Fetterman, D. M. (2008). Ethnography. In The Sage Encyclopedia of Qualitative Research Methods. Sage Publications.

Fiala, M. (2005). Artag rev2 ﬁducial marker system: Vision based tracking for ar. In Workshop of Industrial Augmented Reality, Wienna Austria.

( 175 ) 160 Bibliography

Fitts, P. (1954). The information capacity of the human motor system in controlling the amplitude of movement. Journal of Experimental Psychology, 47:381–391.

Fjeld, M., Lauche, K., Bichsel, M., Voorhorst, F., Krueger, H., and Rauterberg, M. (2002). Physical and virtual tools: Activity theory applied to the design of groupware. Computer Supported Cooperative Work, 11(1/2):153–180.

Friedrich, W., editor (2000). Cognitive Psychology. Sixth Edition. Allyn and Bacon, Boston, Massachusetts.

Friedrich, W., editor (2004). ARVIKA, Augmented Reality für Entwicklung, Produktion und Service. Publicis Corporate Publishing, Erlangen, Germany.

Fuhrmann, A., Löﬀelmann, H., and Schmalstieg, D. (1997). Collaborative augmented reality: Exploring dynamical systems. In Proceedings of IEEE Visualization 97´ , pages 459–462.

Garbis, C. (2002). Exploring the openness of cognitive artifacts in cooperative process management. Cognition, Technology & Work, 4(1):9–21.

Gazzaniga, M. S., Ivry, R. B., and Mangun, G. A., editors (2004). Cognitive Neuroscience: The Biology of the Mind. Second Edition. Norton and Company Inc, New York, New York.

Glaser, B. G. and Strauss, A. L. (1967). The Discovery of Grounded Theory: Strategies for Qualitative Research. Aldine Publishing Company.

Goodwin, C. (1995). Research in Psychology. Methods and Design. John Wiley & Sons.

Gould, J. (1988). How to design usable systems. In Helander, M., Landauer, T. K., and Prabhu, P. V., editors, Handbook of Human Computer Interaction, pages 757–789. North–Holland, New York, New York.

Granlund, R. (2001). Web-based micro-world simulation for emergency management training. Future Generation Computer Systems.

( 176 ) Bibliography 161

Grasset, R., Dünser, A., and Billinghurst, M. (2007). Human-centered development of an AR handheld display. In Sixth IEEE/ACM International Symposium on Mixed and Augmented Reality, ISMAR 2007, 13-16 November 2007, Nara, Japan, pages 177–180. IEEE.

Grasset, R., Dünser, A., and Billinghurst, M. (2008). Edutainment with a mixed reality book: a visually augmented illustrative childrens’ book. In ACE ’08: Proceedings of the 2008 International Conference on Advances in Computer Entertainment Technology, pages 292–295.

Gustafsson, T., Carleberg, P., Nilsson, S., Svensson, P., and Sivertun, A. (2005). Mixed reality systems for technical maintenance and gaze-controlled interaction. Technical report, Swedish Defence Research Agency, Technical report ISSN 1650-1942.

Gustafsson, T., Carleberg, P., Nilsson, S., Svensson, P., Sivertun, A., and LeDuc, M. (2004). Mixed reality för tekniskt stöd. Technical report, Swedish Defence Research Agency, Technical report ISSN 1650-1942.

Gutwin, C. and Greenberg, S. (2001). The importance of awareness for team cognition in distributed collaboration. Technical Report Report 2001-696-19, Dept Computer Science, University of Calgary, Alberta, Canada.

Hackos, J. and Redish, J. C. (1998). User and Task Analysis for Interface Design. John Wiley & Sons.

Haller, M., Billinghurst, M., and Thomas, B. H. (2006). Emerging Technologies of Augmented Reality: Interfaces and Design. IGI Publishing, Hershey, Pennsylvania.

Hammersly, M. and Atkinson, P. (1995). Ethnography: Principles in Practice. Routledge, London, England.

Haniﬀ, D. J. and Baber, C. (2003). User evaluation of augmented reality systems. In Seventh International Conference on Information Visualization, IV 2003, London, UK, pages 505–511.

( 177 ) 162 Bibliography

Hansen, W. J. (1971). User engineering principles for interactive systems. In AFIPS Fall Joint Conference Proceedings, volume 39, pages 523–532.

Heath, C. and Hindmarsh, J. (2002). Analysing interaction: Video, ethnography and situated conduct. In Qualitative Research in Action, pages 99–121.

Hedley, N. R., Billinghurst, M., Postner, L., May, R., and Kato, H. (2002). Explorations in the use of augmented reality for geographic visualization. Presence, 11(2):119–133.

Heims, S. (1991). The Cybernetics Group. MIT Press, Cambridge, Massachusetts.

Henrysson, A. (2007). Bringing Augmented Reality to Mobile Phones. PhD thesis, Linköping University, Sweden.

Henrysson, A., Billinghurst, M., and Ollila, M. (2005a). Face to face collaborative AR on mobile phones. In Fourth IEEE / ACM International Symposium on Mixed and Augmented Reality (ISMAR 2005), Vienna, Austria, pages 80–89.

Henrysson, A., Ollila, M., and Billinghurst, M. (2005b). Mobile phone based AR scene assembly. In Proceedings of the 4th International Conference on Mobile and Ubiquitous Multimedia, MUM 2005, Christchurch, New Zealand, pages 95– 102.

Hertzum, M. and Jacobsen, N. E. (2003). The evaluator eﬀect: A chilling fact about usability evaluation methods. International Journal of Human-Computer Interaction, 15(1):183–204.

Hollan, J., Hutchins, E., and Kirsh, D. (2000). Distributed cognition: toward a new foundation for human-computer interaction research. ACM Transactions on Computer-Human Interaction, 7(2):174–196.

Hollnagel, E. (2003). Is aﬀective computing an oxymoron? International Journal of Human-Computer Studies, 59(1-2):65–70.

Hollnagel, E. and Woods, D. D. (1983). Cognitive systems engineering: New wine in new bottles. International Journal of Man-Machine Studies, 18(6):583–600.

( 178 ) Bibliography 163

Hollnagel, E. and Woods, D. D. (2005). Joint Cognitive Systems: Foundations of Cognitive Systems Engineering. CRC Press, Taylor& Francis, Boca Raton, FL.

Hubona, G. S. and Blanton, J. E. (1996). Evaluating system design features. International Journal of Human-Computer Studies, 44(1):93–118.

Hughes, C. E., Smith, E., Stapleton, C. B., and Hughes, D. (2004). Augmenting museum experiences with mixed reality. In Proceedings of Knowledge Sharing and Collaborative Engineering, pages 197–202.

Hutchins, E. (1990). The technology of team navigation. In Galegher, R. E. K. J. and Egido, C., editors, Intellectual Teamwork: Social Foundations of Cooperative Work, pages 191–220. Lawrence Erlbaum Associates, Hillsdale, New Jersey.

Hutchins, E. (1995a). Cognition in the Wild. MIT Press, London, England.

Hutchins, E. (1995b). How a cockpit remembers its speeds. Cognitive Science, 19(3):265–288.

Hutchins, E. (1999). Cognitive artefacts. In Wilson, R. and Keil, F., editors, The MIT Encyclopedia of the Cognitive Sciences, pages 126–127. Bradford Books, Cambridge, Massachusetts.

Inkpen, K. (1997). Adapting the Human Computer Interface to Support Collaborative Learning Environments for Children. PhD thesis, Dept. of Computer Science, University of British Columbia.

Ishibashi, K., Da Luz, T., Eynard, R., Kita, N., Jiang, N., Segi, H., Terada, K., Fujita, K., and Miyata, K. (2009). Spider hero: a VR application using pulling force feedback system. In VRCAI ’09: Proceedings of the 8th International Conference on Virtual Reality Continuum and its Applications in Industry, pages 197–202.

Jerkedal, A. (2005). Utvärdering steg för steg. Om projekt och programbedömning. Nordsteds Juridik, Stockholm, Sweden.

( 179 ) 164 Bibliography

Johansson, B., Trnka, J., and Granlund, R. (2007). The eﬀect of geographical information systems on a collaborative command and control task. In den Walle, B. V., Burghardt, P., and Nieuwenhuis, C., editors, 4th International Conference on Information Systems for Crisis Response and Management (ISCRAM), Delft, NL.

Jones, M. B., Kennedy, R. S., and Stanney, K. M. (2004). Toward systematic control of cybersickness. Presence, 13(5):589–600.

Jordan, B. and Henderson, A. (1995). Interaction analysis: Foundations and practice. The Journal of the Learning Sciences, 4(1):39–103.

Julier, S., Baillot, Y., Lanzagorta, M., Brown, D., and Rosenblum, L. (2000). BARS: Battleﬁeld augmented reality system. In Proceedings of the NATO Symposium on Information Processing Techniques for Military Systems.

Kaptelinin, V. and Nardi, B. A. (2006). Acting with Technology: Activity Theory and Interaction Design (Acting with Technology). The MIT Press, Cambridge, Massachusetts.

Karat, J. (1988). Software evaluation methodologies. In Helander, M., Landauer, T. K., and Prabhu, P. V., editors, Handbook of Human Computer Interaction, pages 891–903. North-Holland, New York, New York.

Kato, H. and Billinghurst, M. (1999). Marker tracking and hmd calibration for a video-based augmented reality conferencing system. In IWAR ’99: Proceedings of the 2nd IEEE and ACM International Workshop on Augmented Reality, page 85.

Kato, H. and Billinghurst, M. (2004). Developing AR applications with ARToolkit. In 3rd IEEE and ACM International Symposium on Mixed and Augmented Reality (ISMAR 2004), Arlington, VA, USA, page 305.

Kiyokawa, K. (2007). An introduction to head mounted displays for augmented reality. In Haller, M., Billinghurst, M., and Thomas, B., editors, Emerging Technologies of Augmented Reality: Interface and Design. IGI Publishing, Hershey, Pennsylvania.

( 180 ) Bibliography 165

Kiyokawa, K., Billinghurst, M., Hayes, S., Gupta, A., Sannohe, Y., and Kato, H. (2002). Communication behaviors of co-located users in collaborative AR interfaces. In Proceedings of International Symposium on Mixed and Augmented Reality, ISMAR’02, pages 139–148.

Klein, G., Feltovich, P. J., Bradshaw, J. M., and Woods, D. D. (2005). Common Ground and Coordination in Joint Activity, chapter 6, pages 139–184. John Wiley & Sons.

Klein, J. (1985). The abstraction of reality for games andsimulations. International Journal of the Operational Research Society, 39(6):527–535.

Kollin, J. S. (1993). The virtual retinal display. Proceedings of the Society for Information Display, 24:827–829.

Kuutti, K. (1996). Activity theory as a potential framework for human-computer interaction research. In Nardi, B., editor, Context and consciousness: activity theory and human-computer interaction, pages 17–44. Massachusetts Institute of Technology, Cambridge, Massachusetts.

Kvale, S. (1989). Issues of Validity in Qualitative Research. Studentlitteratur, Lund, Sweden.

Kvale, S. (1997). Den kvalitativa forskningsintervjun. Studentlitteratur, Lund, Sweden.

Landauer, T. K. (1997). Behavioural research methods in HCI. In Helander, M., Landauer, T. K., and Prabhu, P. V., editors, Handbook of Human Computer Interaction, pages 203–228. Elseviers Science Publishers, Amsterdam, Holland.

Lawson, S. W., Pretlove, J. R. G., Wheeler, A. C., and Parker, G. A. (2002). Augmented reality as a tool to aid the telerobotic exploration and characterization of remote environments. Presence, 11(4):352–367.

Li, L., Zhang, M., Xu, F., and Liu, S. (2005). ERT-VR: an immersive virtual reality system for emergency rescue training. Virtual Reality, 8(3):194–197.

( 181 ) 166 Bibliography

Likert, R. (1932). A technique for the measurement of attitudes. Archives of Psychology, 8(140):1–55.

Lincoln, Y. S. and Guba, E. G. (1985). Naturalistic Inquiry. Sage, Beverly Hills, California.

Lindgren, I. and Smith, K. (2006). Using microworlds to understand cultural inﬂuences on distributed collaborative decision making in c2 settings. In 11th International Command and Control Research and Technology Symposium (ICCRTS), Cambridge, UK.

Livingston, M., Brown, D., Gabbard, J. L., Rosenblum, L. J., Baillot, Y., Julier, S. J., II, J. S., and Hix, D. (2002). An augmented reality system for military operations in urban terrain. In Proceedings of the Interservice / Industry Training, Simulation, and Education Conference (I/ITSEC ’02), Orlando, FL, USA.

Livingston, M., Brown, D., II, J. S., Goldiez, B., Baillot, Y., and Schmidt, G. S. (2005). Applying a testing methodology to augmented reality interfaces to simulation systems. In Proceedings of the 2005 International Conference on Human-Computer Interface Advances for Modeling and Simulation (SIMCHI’05), New Orleans, LA, USA.

Livingston, M. A. (2005). Evaluating human factors in augmented reality systems. IEEE Computer Graphics and Applications, 25(6):6–9.

Livingston, M. A., Julier, S., and Brown, D. (2006). Situation awareness for teams of dismounted warﬁghters and unmanned vehicles. In Proceedings of SPIE, Enhanced and synthetic vision, volume 6226.

Mansoux, B., Nigay, L., and Traccaz, J. (2004). Abstract and concrete interaction with mixed reality systems. the case of mini–screen, a new interaction device in computer assisted surgery. In Proceedings of the IUI-CADUI Workshop on Exploring the Design and Engineering of Mixed Reality Systems, MIXER. Funchal, Madeira.

( 182 ) Bibliography 167

McCann, C. and Pigeau, R. (2000). The human in command. In C, M. and R, P., editors, The Human in Command; Exploring the Modern Military Experience. Kluwer Academic/Plenum Publishers; New York, New York.

Miles, B. and Huberman, M. (1994). Qualitative data analysis. Sage, Thousand Oaks, California.

Milgram, P. and Ballantyne, J. (1997). Real world teleoperation via virtual environment modelling. In International Conference on Artiﬁcial Reality & Tele-existence, Tokyo.

Milgram, P. and Kishino, F. (1994). A taxonomy of mixed reality visual displays. IEICE Transactions on Information Systems.

Miller, G. A. (1956). The magic number seven, plus or minus two: Some limits on our capacity for processing information. Psychological Review, 63:81–97.

Möhring, M., Lessig, C., and Bimber, O. (2004). Video see-through AR on consumer cell-phones. In 3rd IEEE and ACM International Symposium on Mixed and Augmented Reality (ISMAR 2004), Arlington, VA, USA, pages 252– 253.

Morgan, D. L. (1998). Focus group kit. Vol. 1, The focus group guidebook. Sage.

Muller, M. J. and Kuhn, S. (1993). Participatory design. Communications of the ACM, 36(6):24–28.

Nickerson, R. S. and Landauer, T. K. (1997). Human computer interaction. background and issues. In Helander, M., Landauer, T. K., and Prabhu, P. V., editors, Handbook of Human Computer Interaction, pages 2–32. Elseviers Science Publishers, Amsterdam, Holland.

Nielsen, J. (1993). Usability Engineering. Academic Press, San Diego, California.

( 183 ) 168 Bibliography

Nilsson, S. and Johansson, B. (2006a). A cognitive systems engineering perspective on the design of mixed reality systems. In ECCE ’06: Proceedings of the 13th Eurpoean conference on Cognitive ergonomics, pages 154–161.

Nilsson, S. and Johansson, B. (2006b). User experience and acceptance of a mixed reality system in a naturalistic setting - a case study. In Proceedings of the International Symposium on Mixed and Augmented Reality, ISMAR’06, pages 247–248. IEEE.

Nilsson, S. and Johansson, B. (2007). Fun and usable: augmented reality instructions in a hospital setting. In Proceedings of the 2007 Australasian Computer-Human Interaction Conference, OZCHI 2007, Adelaide, Australia, pages 123–130.

Nilsson, S. and Johansson, B. (2008). Acceptance of augmented reality instructions in a real work setting. In Extended Abstracts Proceedings of the 2008 Conference on Human Factors in Computing Systems, CHI 2008, Florence, Italy, pages 2025–2032.

Norman, D. A. (1998). The Design of Everyday Things. MIT, London, England.

Oehme, O., Wiedenmaier, S., Schmidt, L., and Luczak, H. (2001). Empirical studies on an augmented reality user interface for a head based virtual retinal display. In Proceedings of the Ninth International Conference on Human- Computer Interaction, volume 1, pages 1026–1030.

O’Leary, Z. (2004). The Essential Guide to Doing Research. Sage, London, England.

Olwal, A. and Henrysson, A. (2007). LUMAR: A hybrid spatial display system for 2D and 3D handheld augmented reality. In Advances in Artiﬁcial Reality

( 184 ) Bibliography 169

and Tele-Existence, 17th International Conference on Artiﬁcial Reality and Telexistence, ICAT 2007, Esbjerg, Denmark, pages 63–70.

Patton, M. Q. (1990). Qualitative Evaluation and Research Methods. Sage Publications, Thousand Oaks, California, 2 edition.

Patton, M. Q. (1997). Utilization–focused Evaluations. Sage Publications, Thousand Oaks, California, 2 edition.

Patton, M. Q. (2002). Qualitative Evaluation and Research Methods. Sage Publications, Thousand Oaks, California.

Piekarski, W. (2006). 3D modeling with the Tinmith mobile outdoor augmented reality system. IEEE Computer Graphics and Applications, 26(1):14–17.

Piekarski, W. and Thomas, B. H. (2004). Augmented reality working planes: A foundation for action and construction at a distance. In 3rd IEEE and ACM International Symposium on Mixed and Augmented Reality (ISMAR 2004), Arlington, VA, USA, pages 162–171.

Qvarfordt, P., Jönsson, A., and Dahlbäck, N. (2003). The role of spoken feedback in experiencing multimodal interfaces as human-like. In Proceedings of ICMI’03, Vancouver, Canada.

Regenbrecht, H. T. and Wagner, M. T. (2002). Interaction in a collaborative augmented reality environment. In Proceedings of ACM CHI 2002 Conference on Human Factors in Computing Systems, volume 2 of Demonstrations, pages 504–505.

Reitinger, B., Bornik, A., Beichel, R., and Schmalstieg, D. (2006). Liver surgery planning using virtual reality. IEEE Computer Graphics Applications, 26(6):36– 47.

Reitinger, B., Werlberger, P., Bornik, A., Beichel, R., and Schmalstieg, D. (2005). Spatial measurements for medical augmented reality. In Fourth IEEE / ACM International Symposium on Mixed and Augmented Reality (ISMAR 2005), Vienna, Austria, pages 208–209.

( 185 ) 170 Bibliography

Renner, L. and Johansson, B. (2006). Driver coordination in complex traﬃc environments. In ECCE ’06: Proceedings of the 13th Eurpean conference on Cognitive ergonomics, pages 35–40, New York, New York. ACM.

Rogers, Y. (2006). Distributed cognition and communication. In Brown, K., editor, The Encyclopedia of Language and Linguistics. Second Edition., pages 181–202. Elsevier, Oxford, England.

Rubin, H. and Rubin, L. (1995). Qualitative Interviewing: the art of hearing data. Sage, Thousand Oaks, California.

Sato, M. (2002). Spidar and virtual reality. In Automation Congress, 2002 Proceedings of the 5th Biannual World, volume 13, pages 17–23.

Schmalstieg, D. (2005). Rapid prototyping of augmented reality applications with the studierstube framework. In Workshop of Industrial Augmented Reality, Wienna Austria.

Schmalstieg, D., Fuhrmann, A., Szalavri, Z., and Gervautz, M. (1996). Studierstube: An environment for collaboration in augmented reality. In Collaborative Virtual Environments (CVE’96), Nottingham, UK.

Schmalstieg, D. and Wagner, D. (2007). Experiences with handheld augmented reality. In Sixth IEEE/ACM International Symposium on Mixed and Augmented Reality, ISMAR 2007, Nara, Japan, pages 3–18.

Schuler, D. and Namioka, A., editors (1993). Participatory Design: Principles and Practices. L. Erlbaum Associates, Hillsdale, New Jersey.

Scriven, M. (1991). Evaluation Thesarus. Sage Publications, Newbury Park, California.

Scriven, M. (1997). Types of evaluation and types of evaluator. Evaluation Practice, 17(2):151–161.

Shadish, W. R., Cook, T. D., and Campbell, D. T. (2002). Experimental and quasi- experimental design for generalized causal inference. Boston: Houghton-Miﬄin.

( 186 ) Bibliography 171

Shannon, C. E. and Weaver, W. (1949). The mathematical theory of communication. University of Illinois Press, Urbana, Illinois.

Shneiderman, B. (1998). Designing the User Interface. Addison Wesley, 3 edition.

Silverman, D. (1993). Interpreting Qualitative Data: Methods for Analysing Talk, Text and Interaction. Sage Publications.

Simon, H. A. (1956). Rational choice and the structure of the environment. Psychological Review, 63(2):129–38.

Stanney, K. (1995). Realizing the full potential of virtual reality: human factors issues that could stand in the way. Virtual Reality Annual International Symposium.

Stapleton, C. B., Smith, E. M., and Hughes, C. E. (2005). The art of nurturing citizen scientists through mixed reality. In Fourth IEEE / ACM International Symposium on Mixed and Augmented Reality (ISMAR 2005), Vienna, Austria, pages 2–11.

Stillings, N. A. (1995). Cognitive science : an introduction. MIT Press, Cambridge, Massachusetts.

Strauss, A. and Corbin, J. (1998). Basics of Qualitative Research. Sage Publications, Thousand Oaks, California, 2 edition.

Strickler, Z. (1999). Elicitation methods in experimental design research. Design Issues, 2(15):27–39.

Suchman, L. A. (1987). Plans and Situated Actions. Cambridge University Press, Cambridge, England.

Sutherland, I. (1965). The ultimate display. In Proceedings IFIP Congress, pages 506–508.

Sutherland, I. E. (1968). A head-mounted three-dimensional display. In AFIPS Conference Proceedings, volume 33, pages 757–764.

( 187 ) 172 Bibliography

Swan II, J. E. and Gabbard, J. L. (2005). Survey of user-based experimentation in augmented reality. In 1st International Conference on Virtual Reality, Las Vegas, NV, USA.

Szalavári, Z., Schmalstieg, D., Fuhrmann, A., and Gervautz, M. (1998). Studierstube - An Environment for Collaboration in Augmented Reality. Virtual Reality: Research, Development & Applications, pages 37–48.

Tang, A., Owen, C., Biocca, F., and Mou, W. (2003). Comparative eﬀectiveness of augmented reality in object assembly. In CHI ’03: Proceedings of the SIGCHI conference on Human factors in computing systems, pages 73–80, New York, New York. ACM.

Tidwell, M., Johnston, R. S., Melville, D., and III, T. A. F. (1995). The virtual retinal display - A retinal scanning imaging system. Proceedings of Virtual Reality World’95, Stuttgart, Germany, pages 325–333.

Träskbäck, M. and Haller, M. (2004). Mixed reality training application for an oil reﬁnery: user requirements. In Proceedings VRCAI 2004, ACM SIGGRAPH International Conference on Virtual Reality Continuum and its Applications in Industry, Nanyang Technological University, Singapore, pages 324–327.

Träskbäck, M., Koskinen, T., and Nieminen, M. (2003). User-centred evaluation criteria for a mixed reality authoring application. In Proceedings of the Tenth International Conference on Human-Computer Interaction, volume 1 of Human factors and ergonomics, pages 1263–1267.

Trnka, J., Granlund, R., and Granlund, H. (2008). Using low-ﬁdelity simulations to support design of decision-support systems for command and control applications. In Proceedings of the 14th International Conference on Distributed Multimedia Systems(DMS’2008), Boston, Massachusetts.

Trnka, J., Johansson, B., and Granlund, R. (2006). Information support in collaborative command and control work – an empirical research using a role- playing exercise approach. In Proc. of the 11th Int. Command and Control Research and Technology Symposium (ICCRTS), Cambridge, UK.

( 188 ) Bibliography 173

Vygotsky, L. S. (1978.). Mind In Society. Harvard University Press, Cambridge, Massachusetts.

Wagner, D. and Schmalstieg, D. (2003). First steps towards handheld augmented reality. In Seventh IEEE International Symposium on Wearable Computers (ISWC’03), pages 127–137.

Weidenhausen, J., Knoepﬂe, C., and Stricker, D. (2003). Lessons learned on the way to industrial augmented reality applications, a retrospective on ARVIKA. Computers and Graphics, 27(6):887–891.

Wiener, N. (1948). Cybernetics. Wiley, New York, New York.

Wilkinson, J. (2000). Direct observation. In Breakwell, G. M., Hammond, S., and Fife-Schaw, C., editors, Research Methods in Psychology, 2nd edition. Sage Publications.

Wood-Harper, A. (1985). Research methods in information systems: Using action research. In Mumford E., Hirschheim R., G. F., and Wood-Harper A., editors, The Engineering of Mixed Reality Systems. North–Holland.

Woods, D. D. and Hollnagel, E. (2005). Joint Cognitive Systems: Patterns in Cognitive Systems Engineering. CRC Press, Taylor& Francis, Boca Raton, Florida.

Woods, D. D. and Roth, E. M. (1988). Cognitive engineering: human problem solving with tools. Human Factors, 30(4):415–430.

( 189 ) ( 190 ) Appendix

Instruments used in the studies

Enclosed are the instruments used in the studies:

• Interview guide for study 1

• The questionnaire used in study 1

• The questionnaire used in study 2 and 3

• The three questionnaires used in study 4

• The guide used in the focused group discussion in study 4

• The observation protocol used in study 4

All documents have been translated to English by the author.

175

( 191 ) Questions before the trial (nurses with experience of the Diathermy Apparatus, DA)

All questions translated from Swedish

1. For how long have you been working with Diathermy apparatuses?

2. How often or to what extent?

3. How were you instructed before the first time you used it? (verbally, on paper, instructional video etc)

4. Have you experienced any problems during use of the DA?

5. How would you prefer to receive instructions on new equipment?

Questions before the trial (nurses with no prior experience of the DA)

1. Do you use technical equipment/technical tools in your work?

2. How often/to what extent do you use this equipment?

3. How were you introduced to it?

4. Have you experienced any problems in using technology in your work?

5. How would you prefer to receive instructions on new equipment?

Interview questions after the trial (both experienced and inexperienced DA users)

1. What were your impressions of the MR system?

2. Was there anything you felt was difficult or strenuous?

3. Was there anything you thought was good/positive?

7. How did you experience the interaction? Can you relate it to something else?

8. What would you change in the system?

9. What do you think about this way of giving instructions?

11. Can you think of any other situation where a system like this could be useful?

12. How would you design the system? (considering ergonomics, glasses etc)

( 192 )

Questions regarding the Augmented Reality system (translated from Swedish)

Below are a number of statements, consider them carefully and decide to what extent you agree or disagree. Cross in the check-boxes.

Statement Agree Disagree completely completely

1. I have experience with 3D graphic computer games.

2. I am interested in new technology.

3. I have a lot of computer experience.

4. It was easy to understand the instructions.

5. The text was difficult to read because of the colour or size.

6. It took a long time to learn how to use the system.

7. This system is difficult to use.

8. I felt I had control over the system.

9. I would like to use this type of system to recieve instructions in my work.

10. I would like to use this type of system in other situations than work.

11. With this system I can put objects that are new to me together.

12. I feel confident that the system gives me correct instructions.

13. I would rather receive instructions from a person (teacher/tutor).

14. This system is clumsy to use.

15. This system does not have any flaws.

16. During the trial I felt nausea, dizziness or discomfort.

17. This system is fun to use.

On the next page there is a set of open ended questions for you to answer. Turn the page!

( 193 )

Open questions (answer as lengthily as you like)

1. For how long have you been working as a nurse/physician?

2. Does your job often entail encountering new technology or equippment?

3. How were you introduced to the technology/equippment that you use in your job (please also answer when the introduction took place; during your education or at your place of work)?

4. What are your impressions of the AR system?

5. Was there anything you felt was difficult or strenuous?

6. Was there anything you thought was good/positive?

7. How did you experience the interaction? Can you relate it to something else?

8. What would you change in the system?

9. What do you think about this way of giving instructions?

10. How would you prefer to receive instructions on how to use new technology/equipment?

11. Can you think of any other situation where a system like this could be useful?

12. How would you design the system? (considering ergonomics, glasses etc)

13. Can this be compared with receiving instructions from a teacher/tutor/instructor?

( 194 ) Evaluation of collaboration with the paper map

Position: ______

Presented below are a number of statements. Consider if you agree or disagree with them and to what degree, then check the boxes.

Statement Completely Completely agree disagree

1. It took a long time to start to cooperate.

2. It was easy to cooperate.

3. I think that the map is a good tool for collaboration.

4. The collaboration was fun.

5. I felt that the group was in control of the situation.

6. It was easy to mediate information between the organisations.

7. The map made it easy to achieve a common sitaational picture.

8. The symbols made it easy to achieve a common situational picture.

9. The map became messy/cluttered.

10. I would have liked to have more information than what was available

11. I felt certain that I could interpret what was on the map.

12. The map helped me trust the situational picture.

13. The symbols helped me trust the situational picture.

14. I thought I had a good situational picture.

15. I thought that the others had a good situational picture.

On the following pages there are a number of questions that you can respond to in as much detail you want. Turn page!

( 195 ) Open questions (respond in as much detail you want)

1. What where the best aspects of the paper map for collaboration (feel free to mention several things)?

2. What where the worst aspects of the paper map for collaboration (feel free to mention several things)?

3. Did anything make it difficult to collaborate and if so, what?

4. Did you experience anything that was positive with the collaboration and if so, why?

5. What would you change in order to improve the collaboration?

6. What is your opinion on using paper maps in these types of situations (please motivate)?

( 196 ) Evaluation of collaboration with the AR system

Position: ______

Presented below are a number of statements. Consider if you agree or disagree with them and to what degree, then check the boxes.

Statement Completely Completely agree disagree

1. It took a long time to start to cooperate.

2. It was easy to cooperate.

3. I think that the AR system is a good tool for collaboration.

4. The collaboration was fun.

5. I felt that the group was in control of the situation.

6. It was easy to mediate information between the organisations.

7. The AR system made it easy to achieve a common sitaational picture.

8. The symbols made it easy to achieve a common situational picture.

9. The map became messy/cluttered.

10. I would have liked to have more information than what was available

11. I felt certain that I could interpret what was on the map.

12. The AR system helped me trust the situational picture.

13. The symbols helped me trust the situational picture.

14. I thought I had a good situational picture.

15. I thought that the others had a good situational picture.

On the following pages there are a number of questions that you can respond to in as much detail you want. Turn page!

( 197 ) Open questions (respond in as much detail you want)

1. What where the best aspects of the AR system for collaboration (feel free to mention several things)?

2. What where the worst aspects of the AR system for collaboration (feel free to mention several things)?

3. Did anything make it difficult to collaborate and if so, what?

4. Did you experience anything that was positive with the collaboration and if so, why?

5. What would you change in order to improve the collaboration?

6. What is your opinion on using paper maps in these types of situations (please motivate)?

( 198 ) Evaluation of the AR system

Position: ______

Presented below are a number of statements. Consider if you agree or disagree with them and to what degree, then check the boxes.

Statement Completely Completely agree disagree

1. It was easy to use the AR system.

2. The map was difficult to the colours.

3. The map was difficult to read due to image resolution.

4. The map was difficult to read due to the symbols.

5. It took a long time to learn to use the AR system.

6. There were too many symbols in the AR system.

7. There were too few symbols in the AR system.

8. I thought that the interaction tool was easy to use.

9. I would like to use the AR system in my work.

10. I would like to use the AR system in other situations than work.

11. I felt confident that the AR system gave me correct information.

12. The AR system was clumsy to use.

13. The AR system had no major flaws.

14. I felt sick during the trial.

15. I felt dizziness during the trial.

16. I experienced discomfort during the trial.

15. The AR system was fun to use.

16. It took a long time to learn how to cooperate using the AR system.

On the following pages there are a number of questions that you can respond to in as much detail you want. Turn page!

( 199 ) Open questions (respond in as much detail you want)

1. For how long have you worked in your current position?

2. Do you often come encounter new technology or new equipment in your work? (If yes, please describe what type and if you experience it as positive or negative)

3. What were your general impressions of the AR system?

5. Was there anything that you experienced as troublesome and if so, what?

6. Was there anything you experienced as positive and if so, why?

7. How did you experience the system? Is it comparable with anything else?

9. What do you think about this way of presenting information (please motivate)?

10. Is there any other situation where this type of system could be useful?

( 200 ) Focused group discussion (ca 40-60 min)

Issues for discussion:

- The scenario o Was the scenario realistic? . How important is realism in this type of exercise? o How can you get participants to act more – what type of tasks or events creates most interaction between the organisations according to your experience? o Did you think that the scenario demanded that you collaborate? o Did you feel a need for more/less information from the other organisations? o In what situations do/did you need access to the other organisations knowledge or information?

- The AR system o How do you think the interaction worked? . Easy? Difficult? Difficult to learn? o Do you have any suggestions for how the AR system could be changed? o How often did you use the possibility to retrieve extra information about the objects on the map? . Was there any information missing? o Was it fun to use the system? o Was it tiresome/exhausting to use the system? o How important is the realism of the symbols? . 3D? - The AR system and collaboration o How did you think that the AR system worked for collaboration? o Are there other things you would like to add to the map that would facilitate collaboration? o Is there anything else that you could do with the AR system to facilitate collaboration? o Is it easy to work with the AR system in a group? o Did you at any point experience that the AR system disturbed your work? o Did you at any point experience that the AR system facilitated your work?

( 201 ) - Communication/shared situational picture o Was it good that you did not see all information about all objects at the same time? o Was the system useful in creating a common/shared situational picture? . Does the ability to see each others symbols facilitate when trying to get an understanding of the situation? o Was it faster to reach an understanding of the situation with or without the AR- system? o Did you experience any difference in how long it took to get an overview of the situation when you used the AR system compared to a regular paper map? o Was it more or less messy with the paper map compared to the AR system? o Did you think that the AR system prevented natural communication?

( 202 ) Date: Observations protocol

Observation protocol – AR 1/ Paper map /AR3

System Interaction – the focus in this project is how the participants use the AR system for collaboration.

- How is the general atmosphere in the group?

- Do the participants seem to think it is fun/difficult/distressing/ simple/hard to work with the AR system?

( 203) - Do the participants seem to have any troubles using the interaction device (joystick)? - Do they seem to have difficulties seeing through the display? - Does the AR system seem to have tracking problems (i.e. do any of the participants indicate that it is difficult to see/the image is shaky etc)? - Do they ask the exercise leader or each other how the system works or how to interact with the system?

Date: Observations protocol

Information exchange – to use all available information from all possible sources, which means to actively search for information in order to achieve a good situational picture.

How do the participants act in order to get more information?

- What sources of information do they use that you can see? - Are they speaking with each other? ( 204) - How do they se the map (pointing, gestures)? - Do they use the AR system to find more information?

Do the participants convey information to each other without first being asked? I.e. do they anticipate each other’s information needs?

- Describe an occasion when someone conveys information that was not requested but is important for the collaboration.

- Are there occasions when you think that information that is needed is not conveyed?

- Is unnecessary information conveyed? Date: Observations protocol

Updates of the situational picture – when someone gives an overall summary of the current situation to the others in the group.

- Did any participant do such an open summary of the situation for the rest of the group?

- If so, was it useful for the collaboration?

- Were there occasions were a summary like that may have been needed but not made? ( 205) - Was the AR system used for updates of the situational picture, and if so, how?

Communication – When and how information is conveyed between the group members.

- When the participants talk with each other do they use some kind of internal language/abbreviations/slang/technical terms etc?

- If so, could that have created any confusion or made it difficult for the others to understand?

- Was the AR system used for information conveying, and if so, how?

Date: Observations protocol

There is often an underlying structure in how information is

transmitted.

- Is there any occasion when the participants do not use accepted structure, i.e leave fragmentary or incomplete information?

- If so – what is missing in the information? How does this create a problem?

- Is there any unnecessary communication? I.e. any group member talking about things not related to the task. If so, could this have affected the collaboration?

( 206)

An important aspect of collaboration is the support group members give each other in identifying and bringing to attention error and mistakes and correcting them.

- Is there any occasion when someone corrects the others by for instance saying “you mean…” or by commenting that a symbol has been placed in the wrong place etc?

- Is the AR system used to bring attention to mistakes or errors? The End

Cast (in order of appearance) Funding/Executive Producer...... FMV Technical Development...... XM Reality PhD Student...... Susanna Nilsson Supervisor (2005-2008)...... Erik Hollnagel Co-Supervisor (2005-2010)...... Björn Johansson Supervisor (2008-2010)...... Arne Jönsson Casting Directors...... Margaretha Fredriksson ...... Carina Zetterberg ...... Håkan Evertsson ...... Christer Carlsson ...... Peter Hagsköld

Songs (in no particular order)

“Valiant Visions Dawn” “Ghostss I-IV” “All the Love in the World” “Something I can Never Have” Magna Carta Cartel Nine Inch Nails Nine Inch Nails Nine Inch Nails

“747” “Celsius” “Vals för Satan” “Berlin” (Simon Branting remix) Kent Kent Kent Kent

“It’s No Good” “Photographic” “But Not Tonight” “Blasphemous Rumours” Depeche Mode Depeche Mode Depeche Mode Depeche Mode

“Kingdom” “Play With Fire” “Deep Red” “Satie: Pieces for Piano” Dave Gahan Rolling Stones Apoptygma Berzerk Yuji Takahashi

“Fake Empire” “Leviathan” “Heartbeats” “The Beach” The National Covenant The Knife Nick Cave & Warren Ellis

“Heroes” “Death” “Working Class Hero” “The Road” David Bowie White Lies John Lennon Nick Cave & Warren Ellis

“Tokyo” “Noir Desir” “Falling Slowly” “When Your Minds Made Up” Vive la Fête Vive la Fête The Frames The Frames

“We care a lot” “Midlife Crisis” “Seven” “Keep The Streets Empty for Me” Faith No More Faith No More Fever Ray Fever Ray

“Tiny Dancer” “Corporate Cannibal” “Society” “Destroy Everything You Touch” Elton John Grace Jones Eddie Vedder Ladytron

“In Your Honor” “Sam, as in Samantha” “Notas” “The Man Comes Around” Foo Fighters Tiger Lou Gotan Project Johnny Cash

“Blue Monday” “Wise Up” “Stop Making Speeches” “Voodoo People” New Order Aimee Mann Khoma The Prodigy

( 207 ) “Swallow It” (regurgitated) “Bizarre Love triangle” “What Else Is There?” Fad Gadget New Order Röyksopp

“Dead Souls” “Assimilate” “Fanfanfan” Joy Division Skinny Puppy Thåström

“Dance Pieces” “Faustline” “Beatific” Philip Glass The Mount Fuji Doomjazz Corporation Glass Candy

“Piano Works” “You’re So Vain” “Life After Sundown” Debussy Carly Simon Glass Candy

“Panzermensch” “Relator” “Daniel” And One Pete Yorn & Scarlett Johansson Lior

“LA Woman” “Love Her Madly” “Hello, I Love You” The Doors The Doors The Doors

“Youre Spekaing My Language” “Eraser” “Control (I’m Here)” Juliette & the Licks Nine Inch Nails Nitzer Ebb

“The Girl And The Robot” “Just Like Honey” “Monster Hospital” (Mnstrkrft remix) Röyksopp Jesus and Mary Chain Metric

“I Wish I Knew How It Would Feel To Be Free” “Running” “Poker Face” (Miami Vice mix) Nina Simone The Hate Game Magna Carta Cartel

“Dollhouse Decoration” “Red Dirt Girl” And finally, thank you Billie Holiday Magna Carta Cartel EmmyLou Harris for all your songs.

*A few whose importance cannot be exaggerated enough: Jeanette, Dana, Lollo, Kattis, Linda, Görel, Cartman, Minna, Martin M, Anna, Anette, Rebecca, Martin H, Magdalena, Felix, Andreas, Dena, Elin, Stewie, Anders, the latte at Prego, Haruki Murakami and of course the Dingo on Frasier Island, Mia the koala at Cleland Wildlife Park in Adelaide and the Australian green tree frog in Alice Springs...

Life is not easy for any of us. But what of that? We must have perseverance and above all confidence in ourselves. We must believe that we are gifted for something and that this thing must be attained. Marie Curie

( 208 ) Department of Computer and Information Science Linköpings universitet

Dissertations

Linköping Studies in Science and Technology

No 14 Anders Haraldsson: A Program Manipulation No 214 Tony Larsson: A Formal Hardware Description and System Based on Partial Evaluation, 1977, ISBN 91- Verification Method, 1989, ISBN 91-7870-517-7. 7372-144-1. No 221 Michael Reinfrank: Fundamentals and Logical No 17 Bengt Magnhagen: Probability Based Verification Foundations of Truth Maintenance, 1989, ISBN 91- of Time Margins in Digital Designs, 1977, ISBN 91- 7870-546-0. 7372-157-3. No 239 Jonas Löwgren: Knowledge-Based Design Support No 18 Mats Cedwall: Semantisk analys av process- and Discourse Management in User Interface beskrivningar i naturligt språk, 1977, ISBN 91- 7372- Management Systems, 1991, ISBN 91-7870-720-X. 168-9. No 244 Henrik Eriksson: Meta-Tool Support for Knowledge No 22 Jaak Urmi: A Machine Independent LISP Compiler Acquisition, 1991, ISBN 91-7870-746-3. and its Implications for Ideal Hardware, 1978, ISBN No 252 Peter Eklund: An Epistemic Approach to Interactive 91-7372-188-3. Design in Multiple Inheritance Hierarchies, 1991, No 33 Tore Risch: Compilation of Multiple File Queries in ISBN 91-7870-784-6. a Meta-Database System 1978, ISBN 91- 7372-232-4. No 258 Patrick Doherty: NML3 - A Non-Monotonic No 51 Erland Jungert: Synthesizing Database Structures Formalism with Explicit Defaults, 1991, ISBN 91- from a User Oriented Data Model, 1980, ISBN 91- 7870-816-8. 7372-387-8. No 260 Nahid Shahmehri: Generalized Algorithmic No 54 Sture Hägglund: Contributions to the Development Debugging, 1991, ISBN 91-7870-828-1. of Methods and Tools for Interactive Design of No 264 Nils Dahlbäck: Representation of Discourse- Applications Software, 1980, ISBN 91-7372-404-1. Cognitive and Computational Aspects, 1992, ISBN No 55 Pär Emanuelson: Performance Enhancement in a 91-7870-850-8. Well-Structured Pattern Matcher through Partial No 265 Ulf Nilsson: Abstract Interpretations and Abstract Evaluation, 1980, ISBN 91-7372-403-3. Machines: Contributions to a Methodology for the No 58 Bengt Johnsson, Bertil Andersson: The Human- Implementation of Logic Programs, 1992, ISBN 91- Computer Interface in Commercial Systems, 1981, 7870-858-3. ISBN 91-7372-414-9. No 270 Ralph Rönnquist: Theory and Practice of Tense- No 69 H. Jan Komorowski: A Specification of an Abstract bound Object References, 1992, ISBN 91-7870-873-7. Prolog Machine and its Application to Partial No 273 Björn Fjellborg: Pipeline Extraction for VLSI Data Evaluation, 1981, ISBN 91-7372-479-3. Path Synthesis, 1992, ISBN 91-7870-880-X. No 71 René Reboh: Knowledge Engineering Techniques No 276 Staffan Bonnier: A Formal Basis for Horn Clause and Tools for Expert Systems, 1981, ISBN 91-7372- Logic with External Polymorphic Functions, 1992, 489-0. ISBN 91-7870-896-6. No 77 Östen Oskarsson: Mechanisms of Modifiability in No 277 Kristian Sandahl: Developing Knowledge Manage- large Software Systems, 1982, ISBN 91- 7372-527-7. ment Systems with an Active Expert Methodology, No 94 Hans Lunell: Code Generator Writing Systems, 1983, 1992, ISBN 91-7870-897-4. ISBN 91-7372-652-4. No 281 Christer Bäckström: Computational Complexity of No 97 Andrzej Lingas: Advances in Minimum Weight Reasoning about Plans, 1992, ISBN 91-7870-979-2. Triangulation, 1983, ISBN 91-7372-660-5. No 292 Mats Wirén: Studies in Incremental Natural No 109 Peter Fritzson: Towards a Distributed Programming Language Analysis, 1992, ISBN 91-7871-027-8. Environment based on Incremental Compilation, No 297 Mariam Kamkar: Interprocedural Dynamic Slicing 1984, ISBN 91-7372-801-2. with Applications to Debugging and Testing, 1993, No 111 Erik Tengvald: The Design of Expert Planning ISBN 91-7871-065-0. Systems. An Experimental Operations Planning No 302 Tingting Zhang: A Study in Diagnosis Using System for Turning, 1984, ISBN 91-7372- 805-5. Classification and Defaults, 1993, ISBN 91-7871-078-2 No 155 Christos Levcopoulos: Heuristics for Minimum No 312 Arne Jönsson: Dialogue Management for Natural Decompositions of Polygons, 1987, ISBN 91-7870- Language Interfaces - An Empirical Approach, 1993, 133-3. ISBN 91-7871-110-X. No 165 James W. Goodwin: A Theory and System for Non- No 338 Simin Nadjm-Tehrani: Reactive Systems in Physical Monotonic Reasoning, 1987, ISBN 91-7870-183-X. Environments: Compositional Modelling and Frame- No 170 Zebo Peng: A Formal Methodology for Automated work for Verification, 1994, ISBN 91-7871-237-8. Synthesis of VLSI Systems, 1987, ISBN 91-7870-225-9. No 371 Bengt Savén: Business Models for Decision Support No 174 Johan Fagerström: A Paradigm and System for and Learning. A Study of Discrete-Event Design of Distributed Systems, 1988, ISBN 91-7870- Manufacturing Simulation at Asea/ABB 1968-1993, 301-8. 1995, ISBN 91-7871-494-X. Ulf Söderman: No 192 Dimiter Driankov: Towards a Many Valued Logic of No 375 Conceptual Modelling of Mode Quantified Belief, 1988, ISBN 91-7870-374-3. Switching Physical Systems, 1995, ISBN 91-7871-516- 4. No 213 Lin Padgham: Non-Monotonic Inheritance for an Andreas Kågedal: Object Oriented Knowledge Base, 1989, ISBN 91- No 383 Exploiting Groundness in Logic 7870-485-5. Programs, 1995, ISBN 91-7871-538-5.

( 209 ) No 396 George Fodor: Ontological Control, Description, No 522 Niclas Ohlsson: Towards Effective Fault Prevention Identification and Recovery from Problematic - An Empirical Study in Software Engineering, 1998, Control Situations, 1995, ISBN 91-7871-603-9. ISBN 91-7219-176-7. No 413 Mikael Pettersson: Compiling Natural Semantics, No 526 Joachim Karlsson: A Systematic Approach for 1995, ISBN 91-7871-641-1. Prioritizing Software Requirements, 1998, ISBN 91- No 414 Xinli Gu: RT Level Testability Improvement by 7219-184-8. Testability Analysis and Transformations, 1996, ISBN No 530 Henrik Nilsson: Declarative Debugging for Lazy 91-7871-654-3. Functional Languages, 1998, ISBN 91-7219-197-x. No 416 Hua Shu: Distributed Default Reasoning, 1996, ISBN No 555 Jonas Hallberg: Timing Issues in High-Level Synthe- 91-7871-665-9. sis, 1998, ISBN 91-7219-369-7. No 429 Jaime Villegas: Simulation Supported Industrial No 561 Ling Lin: Management of 1-D Sequence Data - From Training from an Organisational Learning Discrete to Continuous, 1999, ISBN 91-7219-402-2. Perspective - Development and Evaluation of the No 563 Eva L Ragnemalm: Student Modelling based on Col- SSIT Method, 1996, ISBN 91-7871-700-0. laborative Dialogue with a Learning Companion, No 431 Peter Jonsson: Studies in Action Planning: 1999, ISBN 91-7219-412-X. Algorithms and Complexity, 1996, ISBN 91-7871-704- No 567 Jörgen Lindström: Does Distance matter? On geo- 3. graphical dispersion in organisations, 1999, ISBN 91- No 437 Johan Boye: Directional Types in Logic 7219-439-1. Programming, 1996, ISBN 91-7871-725-6. No 582 Vanja Josifovski: Design, Implementation and No 439 Cecilia Sjöberg: Activities, Voices and Arenas: Evaluation of a Distributed Mediator System for Participatory Design in Practice, 1996, ISBN 91-7871- Data Integration, 1999, ISBN 91-7219-482-0. 728-0. No 589 Rita Kovordányi: Modeling and Simulating No 448 Patrick Lambrix: Part-Whole Reasoning in Inhibitory Mechanisms in Mental Image Description Logics, 1996, ISBN 91-7871-820-1. Reinterpretation - Towards Cooperative Human- No 452 Kjell Orsborn: On Extensible and Object-Relational Computer Creativity, 1999, ISBN 91-7219-506-1. Database Technology for Finite Element Analysis No 592 Mikael Ericsson: Supporting the Use of Design Applications, 1996, ISBN 91-7871-827-9. Knowledge - An Assessment of Commenting No 459 Olof Johansson: Development Environments for Agents, 1999, ISBN 91-7219-532-0. Complex Product Models, 1996, ISBN 91-7871-855-4. No 593 Lars Karlsson: Actions, Interactions and Narratives, No 461 Lena Strömbäck: User-Defined Constructions in 1999, ISBN 91-7219-534-7. Unification-Based Formalisms, 1997, ISBN 91-7871- No 594 C. G. Mikael Johansson: Social and Organizational 857-0. Aspects of Requirements Engineering Methods - A No 462 Lars Degerstedt: Tabulation-based Logic Program- practice-oriented approach, 1999, ISBN 91-7219-541- ming: A Multi-Level View of Query Answering, X. 1996, ISBN 91-7871-858-9. No 595 Jörgen Hansson: Value-Driven Multi-Class Overload No 475 Fredrik Nilsson: Strategi och ekonomisk styrning - Management in Real-Time Database Systems, 1999, En studie av hur ekonomiska styrsystem utformas ISBN 91-7219-542-8. och används efter företagsförvärv, 1997, ISBN 91- No 596 Niklas Hallberg: Incorporating User Values in the 7871-914-3. Design of Information Systems and Services in the No 480 Mikael Lindvall: An Empirical Study of Require- Public Sector: A Methods Approach, 1999, ISBN 91- ments-Driven Impact Analysis in Object-Oriented 7219-543-6. Software Evolution, 1997, ISBN 91-7871-927-5. No 597 Vivian Vimarlund: An Economic Perspective on the No 485 Göran Forslund: Opinion-Based Systems: The Coop- Analysis of Impacts of Information Technology: erative Perspective on Knowledge-Based Decision From Case Studies in Health-Care towards General Support, 1997, ISBN 91-7871-938-0. Models and Theories, 1999, ISBN 91-7219-544-4. No 494 Martin Sköld: Active Database Management No 598 Johan Jenvald: Methods and Tools in Computer- Systems for Monitoring and Control, 1997, ISBN 91- Supported Taskforce Training, 1999, ISBN 91-7219- 7219-002-7. 547-9. No 495 Hans Olsén: Automatic Verification of Petri Nets in No 607 Magnus Merkel: Understanding and enhancing a CLP framework, 1997, ISBN 91-7219-011-6. translation by parallel text processing, 1999, ISBN 91- No 498 Thomas Drakengren: Algorithms and Complexity 7219-614-9. for Temporal and Spatial Formalisms, 1997, ISBN 91- No 611 Silvia Coradeschi: Anchoring symbols to sensory 7219-019-1. data, 1999, ISBN 91-7219-623-8. No 502 Jakob Axelsson: Analysis and Synthesis of Heteroge- No 613 Man Lin: Analysis and Synthesis of Reactive neous Real-Time Systems, 1997, ISBN 91-7219-035-3. Systems: A Generic Layered Architecture No 503 Johan Ringström: Compiler Generation for Data- Perspective, 1999, ISBN 91-7219-630-0. Parallel Programming Languages from Two-Level No 618 Jimmy Tjäder: Systemimplementering i praktiken - Semantics Specifications, 1997, ISBN 91-7219-045-0. En studie av logiker i fyra projekt, 1999, ISBN 91- No 512 Anna Moberg: Närhet och distans - Studier av kom- 7219-657-2. munikationsmönster i satellitkontor och flexibla No 627 Vadim Engelson: Tools for Design, Interactive kontor, 1997, ISBN 91-7219-119-8. Simulation, and Visualization of Object-Oriented No 520 Mikael Ronström: Design and Modelling of a Models in Scientific Computing, 2000, ISBN 91-7219- Parallel Data Server for Telecom Applications, 1998, 709-9. ISBN 91-7219-169-4. No 637 Esa Falkenroth: Database Technology for Control and Simulation, 2000, ISBN 91-7219-766-8.

( 210 ) No 639 Per-Arne Persson: Bringing Power and Knowledge No 808 Klas Gäre: Tre perspektiv på förväntningar och Together: Information Systems Design for Autonomy förändringar i samband med införande av and Control in Command Work, 2000, ISBN 91-7219- informationssystem, 2003, ISBN 91-7373-618-X. 796-X. No 821 Mikael Kindborg: Concurrent Comics - No 660 Erik Larsson: An Integrated System-Level Design for programming of social agents by children, 2003, Testability Methodology, 2000, ISBN 91-7219-890-7. ISBN 91-7373-651-1. No 688 Marcus Bjäreland: Model-based Execution No 823 Christina Ölvingson: On Development of Monitoring, 2001, ISBN 91-7373-016-5. Information Systems with GIS Functionality in No 689 Joakim Gustafsson: Extending Temporal Action Public Health Informatics: A Requirements Logic, 2001, ISBN 91-7373-017-3. Engineering Approach, 2003, ISBN 91-7373-656-2. No 720 Carl-Johan Petri: Organizational Information Provi- No 828 Tobias Ritzau: Memory Efficient Hard Real-Time sion - Managing Mandatory and Discretionary Use Garbage Collection, 2003, ISBN 91-7373-666-X. of Information Technology, 2001, ISBN-91-7373-126- No 833 Paul Pop: Analysis and Synthesis of 9. Communication-Intensive Heterogeneous Real-Time No 724 Paul Scerri: Designing Agents for Systems with Ad- Systems, 2003, ISBN 91-7373-683-X. justable Autonomy, 2001, ISBN 91 7373 207 9. No 852 Johan Moe: Observing the Dynamic Behaviour of No 725 Tim Heyer: Semantic Inspection of Software Large Distributed Systems to Improve Development Artifacts: From Theory to Practice, 2001, ISBN 91 and Testing – An Empirical Study in Software 7373 208 7. Engineering, 2003, ISBN 91-7373-779-8. No 726 Pär Carlshamre: A Usability Perspective on Require- No 867 Erik Herzog: An Approach to Systems Engineering ments Engineering - From Methodology to Product Tool Data Representation and Exchange, 2004, ISBN Development, 2001, ISBN 91 7373 212 5. 91-7373-929-4. No 732 Juha Takkinen: From Information Management to No 872 Aseel Berglund: Augmenting the Remote Control: Task Management in Electronic Mail, 2002, ISBN 91 Studies in Complex Information Navigation for 7373 258 3. Digital TV, 2004, ISBN 91-7373-940-5. No 745 Johan Åberg: Live Help Systems: An Approach to No 869 Jo Skåmedal: Telecommuting’s Implications on Intelligent Help for Web Information Systems, 2002, Travel and Travel Patterns, 2004, ISBN 91-7373-935-9. ISBN 91-7373-311-3. No 870 Linda Askenäs: The Roles of IT - Studies of No 746 Rego Granlund: Monitoring Distributed Teamwork Organising when Implementing and Using Training, 2002, ISBN 91-7373-312-1. Enterprise Systems, 2004, ISBN 91-7373-936-7. No 757 Henrik André-Jönsson: Indexing Strategies for Time No 874 Annika Flycht-Eriksson: Design and Use of Ontolo- Series Data, 2002, ISBN 917373-346-6. gies in Information-Providing Dialogue Systems, No 747 Anneli Hagdahl: Development of IT-supported 2004, ISBN 91-7373-947-2. Interorganisational Collaboration - A Case Study in No 873 Peter Bunus: Debugging Techniques for Equation- the Swedish Public Sector, 2002, ISBN 91-7373-314-8. Based Languages, 2004, ISBN 91-7373-941-3. No 749 Sofie Pilemalm: Information Technology for Non- No 876 Jonas Mellin: Resource-Predictable and Efficient Profit Organisations - Extended Participatory Design Monitoring of Events, 2004, ISBN 91-7373-956-1. of an Information System for Trade Union Shop No 883 Magnus Bång: Computing at the Speed of Paper: Stewards, 2002, ISBN 91-7373-318-0. Ubiquitous Computing Environments for Healthcare No 765 Stefan Holmlid: Adapting users: Towards a theory Professionals, 2004, ISBN 91-7373-971-5 of use quality, 2002, ISBN 91-7373-397-0. No 882 Robert Eklund: Disfluency in Swedish human- No 771 Magnus Morin: Multimedia Representations of Dis- human and human-machine travel booking di- tributed Tactical Operations, 2002, ISBN 91-7373-421- alogues, 2004, ISBN 91-7373-966-9. 7. No 887 Anders Lindström: English and other Foreign No 772 Pawel Pietrzak: A Type-Based Framework for Locat- Linguistic Elements in Spoken Swedish. Studies of ing Errors in Constraint Logic Programs, 2002, ISBN Productive Processes and their Modelling using 91-7373-422-5. Finite-State Tools, 2004, ISBN 91-7373-981-2. No 758 Erik Berglund: Library Communication Among Pro- No 889 Zhiping Wang: Capacity-Constrained Production-in- grammers Worldwide, 2002, ISBN 91-7373-349-0. ventory systems - Modelling and Analysis in both a No 774 Choong-ho Yi: Modelling Object-Oriented Dynamic traditional and an e-business context, 2004, ISBN 91- Systems Using a Logic-Based Framework, 2002, ISBN 85295-08-6. 91-7373-424-1. No 893 Pernilla Qvarfordt: Eyes on Multimodal Interaction, No 779 Mathias Broxvall: A Study in the Computational 2004, ISBN 91-85295-30-2. Complexity of Temporal Reasoning, 2002, ISBN 91- No 910 Magnus Kald: In the Borderland between Strategy 7373-440-3. and Management Control - Theoretical Framework No 793 Asmus Pandikow: A Generic Principle for Enabling and Empirical Evidence, 2004, ISBN 91-85295-82-5. Interoperability of Structured and Object-Oriented No 918 Jonas Lundberg: Shaping Electronic News: Genre Analysis and Design Tools, 2002, ISBN 91-7373-479-9. Perspectives on Interaction Design, 2004, ISBN 91- No 785 Lars Hult: Publika Informationstjänster. En studie av 85297-14-3. den Internetbaserade encyklopedins bruksegenska- No 900 Mattias Arvola: Shades of use: The dynamics of per, 2003, ISBN 91-7373-461-6. interaction design for sociable use, 2004, ISBN 91- No 800 Lars Taxén: A Framework for the Coordination of 85295-42-6. Complex Systems´ Development, 2003, ISBN 91- No 920 Luis Alejandro Cortés: Verification and Scheduling 7373-604-X Techniques for Real-Time Embedded Systems, 2004, ISBN 91-85297-21-6.

( 211 ) No 929 Diana Szentivanyi: Performance Studies of Fault- No 1022 Peter Aronsson: Automatic Parallelization of Equa- Tolerant Middleware, 2005, ISBN 91-85297-58-5. tion-Based Simulation Programs, 2006, ISBN 91- No 933 Mikael Cäker: Management Accounting as 85523-68-2. Constructing and Opposing Customer Focus: Three No 1030 Robert Nilsson: A Mutation-based Framework for Case Studies on Management Accounting and Automated Testing of Timeliness, 2006, ISBN 91- Customer Relations, 2005, ISBN 91-85297-64-X. 85523-35-6. No 937 Jonas Kvarnström: TALplanner and Other No 1034 Jon Edvardsson: Techniques for Automatic Extensions to Temporal Action Logic, 2005, ISBN 91- Generation of Tests from Programs and 85297-75-5. Specifications, 2006, ISBN 91-85523-31-3. No 938 Bourhane Kadmiry: Fuzzy Gain-Scheduled Visual No 1035 Vaida Jakoniene: Integration of Biological Data, Servoing for Unmanned Helicopter, 2005, ISBN 91- 2006, ISBN 91-85523-28-3. 85297-76-3. No 1045 Genevieve Gorrell: Generalized Hebbian No 945 Gert Jervan: Hybrid Built-In Self-Test and Test Algorithms for Dimensionality Reduction in Natural Generation Techniques for Digital Systems, 2005, Language Processing, 2006, ISBN 91-85643-88-2. ISBN: 91-85297-97-6. No 1051 Yu-Hsing Huang: Having a New Pair of Glasses - No 946 Anders Arpteg: Intelligent Semi-Structured Informa- Applying Systemic Accident Models on Road Safety, tion Extraction, 2005, ISBN 91-85297-98-4. 2006, ISBN 91-85643-64-5. No 947 Ola Angelsmark: Constructing Algorithms for Con- No 1054 Åsa Hedenskog: Perceive those things which cannot straint Satisfaction and Related Problems - Methods be seen - A Cognitive Systems Engineering and Applications, 2005, ISBN 91-85297-99-2. perspective on requirements management, 2006, No 963 Calin Curescu: Utility-based Optimisation of ISBN 91-85643-57-2. Resource Allocation for Wireless Networks, 2005, No 1061 Cécile Åberg: An Evaluation Platform for Semantic ISBN 91-85457-07-8. Web Technology, 2007, ISBN 91-85643-31-9. No 972 Björn Johansson: Joint Control in Dynamic No 1073 Mats Grindal: Handling Combinatorial Explosion in Situations, 2005, ISBN 91-85457-31-0. Software Testing, 2007, ISBN 978-91-85715-74-9. No 974 Dan Lawesson: An Approach to Diagnosability No 1075 Almut Herzog: Usable Security Policies for Runtime Analysis for Interacting Finite State Systems, 2005, Environments, 2007, ISBN 978-91-85715-65-7. ISBN 91-85457-39-6. No 1079 Magnus Wahlström: Algorithms, measures, and No 979 Claudiu Duma: Security and Trust Mechanisms for upper bounds for Satisfiability and related problems, Groups in Distributed Services, 2005, ISBN 91-85457- 2007, ISBN 978-91-85715-55-8. 54-X. No 1083 Jesper Andersson: Dynamic Software Architectures, No 983 Sorin Manolache: Analysis and Optimisation of 2007, ISBN 978-91-85715-46-6. Real-Time Systems with Stochastic Behaviour, 2005, No 1086 Ulf Johansson: Obtaining Accurate and ISBN 91-85457-60-4. Comprehensible Data Mining Models - An No 986 Yuxiao Zhao: Standards-Based Application Evolutionary Approach, 2007, ISBN 978-91-85715-34- Integration for Business-to-Business 3. Communications, 2005, ISBN 91-85457-66-3. No 1089 Traian Pop: Analysis and Optimisation of No 1004 Patrik Haslum: Admissible Heuristics for Distributed Embedded Systems with Heterogeneous Automated Planning, 2006, ISBN 91-85497-28-2. Scheduling Policies, 2007, ISBN 978-91-85715-27-5. No 1005 Aleksandra Tešanovic: Developing Reusable and No 1091 Gustav Nordh: Complexity Dichotomies for CSP- Reconfigurable Real-Time Software using Aspects related Problems, 2007, ISBN 978-91-85715-20-6. and Components, 2006, ISBN 91-85497-29-0. No 1106 Per Ola Kristensson: Discrete and Continuous Shape No 1008 David Dinka: Role, Identity and Work: Extending Writing for Text Entry and Control, 2007, ISBN 978- the design and development agenda, 2006, ISBN 91- 91-85831-77-7. 85497-42-8. No 1110 He Tan: Aligning Biomedical Ontologies, 2007, ISBN No 1009 Iakov Nakhimovski: Contributions to the Modeling 978-91-85831-56-2. and Simulation of Mechanical Systems with Detailed No 1112 Jessica Lindblom: Minding the body - Interacting so- Contact Analysis, 2006, ISBN 91-85497-43-X. cially through embodied action, 2007, ISBN 978-91- No 1013 Wilhelm Dahllöf: Exact Algorithms for Exact 85831-48-7. Satisfiability Problems, 2006, ISBN 91-85523-97-6. No 1113 Pontus Wärnestål: Dialogue Behavior Management No 1016 Levon Saldamli: PDEModelica - A High-Level Lan- in Conversational Recommender Systems, 2007, guage for Modeling with Partial Differential Equa- ISBN 978-91-85831-47-0. tions, 2006, ISBN 91-85523-84-4. No 1120 Thomas Gustafsson: Management of Real-Time No 1017 Daniel Karlsson: Verification of Component-based Data Consistency and Transient Overloads in Embedded System Designs, 2006, ISBN 91-85523-79-8 Embedded Systems, 2007, ISBN 978-91-85831-33-3. No 1018 Ioan Chisalita: Communication and Networking No 1127 Alexandru Andrei: Energy Efficient and Predictable Techniques for Traffic Safety Systems, 2006, ISBN 91- Design of Real-time Embedded Systems, 2007, ISBN 85523-77-1. 978-91-85831-06-7. No 1019 Tarja Susi: The Puzzle of Social Activity - The No 1139 Per Wikberg: Eliciting Knowledge from Experts in Significance of Tools in Cognition and Cooperation, Modeling of Complex Systems: Managing Variation 2006, ISBN 91-85523-71-2. and Interactions, 2007, ISBN 978-91-85895-66-3. No 1021 Andrzej Bednarski: Integrated Optimal Code Gener- No 1143 Mehdi Amirijoo: QoS Control of Real-Time Data ation for Digital Signal Processors, 2006, ISBN 91- Services under Uncertain Workload, 2007, ISBN 978- 85523-69-0. 91-85895-49-6.

( 212 ) No 1150 Sanny Syberfeldt: Optimistic Replication with For- Linköping Studies in Arts and Sciences ward Conflict Resolution in Distributed Real-Time No 504 Ing-Marie Jonsson: Social and Emotional Databases, 2007, ISBN 978-91-85895-27-4. Characteristics of Speech-based In-Vehicle No 1155 Beatrice Alenljung: Envisioning a Future Decision Information Systems: Impact on Attitude and Support System for Requirements Engineering - A Driving Behaviour, 2009, ISBN 978-91-7393-478-7. Holistic and Human-centred Perspective, 2008, ISBN 978-91-85895-11-3. Linköping Studies in Statistics No 1156 Artur Wilk: Types for XML with Application to No 9 Davood Shahsavani: Computer Experiments De- Xcerpt, 2008, ISBN 978-91-85895-08-3. signed to Explore and Approximate Complex Deter- No 1183 Adrian Pop: Integrated Model-Driven Development ministic Models, 2008, ISBN 978-91-7393-976-8. Environments for Equation-Based Object-Oriented No 10 Karl Wahlin: Roadmap for Trend Detection and As- Languages, 2008, ISBN 978-91-7393-895-2. sessment of Data Quality, 2008, ISBN 978-91-7393- No 1185 Jörgen Skågeby: Gifting Technologies - 792-4 Ethnographic Studies of End-users and Social Media Sharing, 2008, ISBN 978-91-7393-892-1. Linköping Studies in Information Science No 1187 Imad-Eldin Ali Abugessaisa: Analytical tools and No 1 Karin Axelsson: Metodisk systemstrukturering- att information-sharing methods supporting road safety skapa samstämmighet mellan informationssystem- organizations, 2008, ISBN 978-91-7393-887-7. arkitektur och verksamhet, 1998. ISBN-9172-19-296-8. No 1204 H. Joe Steinhauer: A Representation Scheme for De- No 2 Stefan Cronholm: Metodverktyg och användbarhet - scription and Reconstruction of Object en studie av datorstödd metodbaserad Configurations Based on Qualitative Relations, 2008, systemutveckling, 1998, ISBN-9172-19-299-2. ISBN 978-91-7393-823-5. No 3 Anders Avdic: Användare och utvecklare - om No 1222 Anders Larsson: Test Optimization for Core-based anveckling med kalkylprogram, 1999. ISBN-91-7219- System-on-Chip, 2008, ISBN 978-91-7393-768-9. 606-8. No 1238 Andreas Borg: Processes and Models for Capacity No 4 Owen Eriksson: Kommunikationskvalitet hos infor- Requirements in Telecommunication Systems, 2009, mationssystem och affärsprocesser, 2000, ISBN 91- ISBN 978-91-7393-700-9. 7219-811-7. No 1240 Fredrik Heintz: DyKnow: A Stream-Based Know- No 5 Mikael Lind: Från system till process - kriterier för ledge Processing Middleware Framework, 2009, processbestämning vid verksamhetsanalys, 2001, ISBN 978-91-7393-696-5. ISBN 91-7373-067-X. No 1241 Birgitta Lindström: Testability of Dynamic Real- No 6 Ulf Melin: Koordination och informationssystem i Time Systems, 2009, ISBN 978-91-7393-695-8. företag och nätverk, 2002, ISBN 91-7373-278-8. No 1244 Eva Blomqvist: Semi-automatic Ontology Construc- No 7 Pär J. Ågerfalk: Information Systems Actability - Un- tion based on Patterns, 2009, ISBN 978-91-7393-683-5. derstanding Information Technology as a Tool for No 1249 Rogier Woltjer: Functional Modeling of Constraint Business Action and Communication, 2003, ISBN 91- Management in Aviation Safety and Command and 7373-628-7. Control, 2009, ISBN 978-91-7393-659-0. No 8 Ulf Seigerroth: Att förstå och förändra No 1260 Gianpaolo Conte: Vision-Based Localization and systemutvecklingsverksamheter - en taxonomi för Guidance for Unmanned Aerial Vehicles, 2009, ISBN metautveckling, 2003, ISBN91-7373-736-4. 978-91-7393-603-3. No 9 Karin Hedström: Spår av datoriseringens värden – No 1262 AnnMarie Ericsson: Enabling Tool Support for For- Effekter av IT i äldreomsorg, 2004, ISBN 91-7373-963- mal Analysis of ECA Rules, 2009, ISBN 978-91-7393- 4. 598-2. No 10 Ewa Braf: Knowledge Demanded for Action - No 1266 Jiri Trnka: Exploring Tactical Command and Studies on Knowledge Mediation in Organisations, Control: A Role-Playing Simulation Approach, 2009, 2004, ISBN 91-85295-47-7. ISBN 978-91-7393-571-5. No 11 Fredrik Karlsson: Method Configuration method No 1268 Bahlol Rahimi: Supporting Collaborative Work and computerized tool support, 2005, ISBN 91-85297- through ICT - How End-users Think of and Adopt 48-8. Integrated Health Information Systems, 2009, ISBN 978-91-7393-550-0. No 12 Malin Nordström: Styrbar systemförvaltning - Att organisera systemförvaltningsverksamhet med hjälp No 1274 Fredrik Kuivinen: Algorithms and Hardness Results av effektiva förvaltningsobjekt, 2005, ISBN 91-85297- for Some Valued CSPs, 2009, ISBN 978-91-7393-525-8. 60-7. No 1281 Gunnar Mathiason: Virtual Full Replication for No 13 Stefan Holgersson: Yrke: POLIS - Yrkeskunskap, Scalable Distributed Real-Time Databases, 2009, motivation, IT-system och andra förutsättningar för ISBN 978-91-7393-503-6. polisarbete, 2005, ISBN 91-85299-43-X. No 1290 Viacheslav Izosimov: Scheduling and Optimization No 14 Benneth Christiansson, Marie-Therese of Fault-Tolerant Distributed Embedded Systems, Christiansson: Mötet mellan process och komponent 2009, ISBN 978-91-7393-482-4. - mot ett ramverk för en verksamhetsnära No 1294 Johan Thapper: Aspects of a Constraint kravspecifikation vid anskaffning av Optimisation Problem, 2010, ISBN 978-91-7393-464-0. komponentbaserade informationssystem, 2006, ISBN No 1306 Susanna Nilsson: Augmentation in the Wild: User 91-85643-22-X. Centered Development and Evaluation of Augmented Reality Applications, 2010, ISBN 978-91- 7393-416-9

( 213 )