The Air Defence Task: Understanding what motivates automation usage to support classification decisions in practice Thesis submitted in accordance with the requirements of the University of Liverpool for the degree of Doctor in Philosophy by Chlo¨eBarrett-Pink June 2018 2 Abstract With the growing use of automated systems in military environments it remains vital that research continues to explore the use of such systems operationally. The recent literature has tended to take a systems focused approach, which has concentrated on features of the system and what impact alterations have upon task performance. How- ever, research has begun to see the value in taking a human-centred perspective to understanding the use of automated systems in military environments; asking research questions that remain focused on the human operators that are required to utilise auto- mated systems in increasingly complex environments. Therefore, this thesis contributes to the literature on human-machine-interaction through exploring the operational use of automated systems in the maritime environment. Research into sociotechnical sys- tems is complex, therefore this thesis adopted a Naturalistic Decision Making (NDM) approach utilising mixed-methods to elicit understanding and knowledge from unique access to Royal Navy (RN) Subject Matter Experts (SME). Privileged access to a large number of RN experts (N =53) enabled novel and interesting findings to be drawn from two qualitative surveys. The first explored the stages of the air defence task conducted by RN personnel to better understand where uptake of automation may be beneficial. The findings of this questionnaire revealed that the high-level stages of the air defence task (Observe, Identify and Classify, and Decide and Act) have remained unchanged over the last 20 years and the areas that have previously been identified as potentially benefiting from automated system support remain the same. These findings raised pertinent questions as to why the same areas are still in need of support. Therefore, the second study of this thesis aimed to explore where automated systems have been brought into service to support RN operations to understand how the current procure- ment process functions. A second questionnaire was developed which allowed RN SME to discuss how automated systems are currently used across all operational settings and where they may be used in the future. Crucially, this second questionnaire explored RN SME opinions towards the existing procurement process. Of concern was that the findings of this study revealed the disconnect that often exists between end user and system designer which has a negative effect on the development of systems being fit for purpose at time of release. This in turn can have severe negative consequences to capability, appropriate system use and can increase the financial costs of developing i and implementing new systems. The findings from the first two studies presented in this thesis highlighted the need for, and recommended an increase in use of, immer- sive simulation environments to support automated system development and research. Therefore, the third part of this thesis presents the development and validation of a simulated microworld, the Automatic Radar Classification Simulation (ARCS). ARCS was designed by the author of this thesis in collaboration with a software engineer to replicate aspects of the air defence task conducted by RN personnel. This design pro- cess included 2 pilot studies, the results of which informed developments and changes to ARCS. Overall, this design process took 8 months with several iterations of ARCS be- ing developed. Following the development stage, an experiment was conducted (N =42 university students) to validate the utility of ARCS as a microworld using a holistic real-time scenario to explore individuals rationales for using a generic automated system when performing a threat detection task. In line with previous research, participants cited workload and managing uncertainty as reasons for selecting to use the automated decision support system. However, unexpectedly task performance was not significantly improved with access to the support system and strong learning effects were observed. Overall, this thesis supports the newly proposed move away from traditional \levels of automation" approaches, advocating for taking a more holistic approach to research into human-machine-interaction. This can be achieved through promoting long-term and continuous engagement between end users and system designers, ensuring that a human-human relationship is maintained throughout the life-cycle of the automated system. Additionally, this thesis highlights the importance of effective communication within and between the military, industry and academia, and the negative implications that ineffective communication has upon naval capability. Finally, this thesis supports the literature that highlights the importance of training in immersive environments and has provided academia with a high-fidelity microworld with which to explore operator use of automated decision support systems in the maritime environment. ii Contents Abstract i Contents vii List of Figures xii List of Tables xii Acronyms xiii Acknowledgement xv Publications xvii 1 When above water warfare meets human-machine-interaction: an overview of what is known and what remains to be answered 1 1.1 Introduction . .1 1.2 Problem Space . .1 1.2.1 Current and Future Capabilities . .2 1.2.2 The Importance of Synergy between Man and Machine . .8 1.2.3 Historical research into use of automated systems . .9 1.2.4 Barriers to automation uptake and use . 12 1.2.5 Overview of the literature on decision making . 15 1.2.6 Naturalistic decision making and decision centred design . 19 1.2.7 Thesis Structure . 23 2 A methodological approach to interdisciplinary research in real-world and experimental settings 27 2.1 Introduction . 27 2.2 Challenges characteristic to mixed-methods research and interdisciplinary scholarship . 28 2.2.1 Methodologies . 28 2.2.2 Taking a mixed-methods approach . 28 iii 2.2.3 Advantages of taking a mixed-methods approach to interdisci- plinary scholarship . 31 2.2.4 Framework of the mixed-method approach developed for this thesis 32 2.2.5 Conclusions of the challenges typical to mixed-method and inter- disciplinary research . 33 2.3 Interdisciplinary Scholarships: why NDM provides a robust approach to research . 34 2.3.1 Stage 1 - Adaptation of the critical decision method into a vi- gnette scenario . 35 2.3.2 Stage 2 - Questionnaire development . 37 2.3.3 Summary . 38 2.4 Stage 3 - Microworld Design and Development . 38 2.5 Automatic Radar Classification Simulation (ARCS) . 39 2.6 Questionnaire battery designed to explore individual differences and au- tomation usage decisions . 42 2.7 Measures collected and thesis hypothesis . 43 2.8 Chapter Summary . 45 3 Understanding the Air Defence Task: A Descriptive Decision Model from Perspectives of Royal Navy personnel 47 3.1 Abstract . 47 3.2 Introduction . 47 3.3 Historical literature on the air defence task . 48 3.4 Method . 50 3.4.1 Participants . 50 3.4.2 Vignette Survey . 51 3.4.3 Procedure . 52 3.4.4 Analysis . 53 3.5 Results . 55 3.5.1 Air defence task . 55 3.5.2 Observe . 55 3.5.3 Identify and Classify . 59 3.5.4 Decide and Act . 62 3.6 Discussion . 65 3.6.1 Conclusions . 68 4 On the bridges: insight into the current and future use of automated systems as seen by Royal Navy personnel. 71 4.1 Abstract . 71 4.2 Introduction . 71 iv 4.2.1 Current doctrine around automated systems . 73 4.2.2 Future visions of 2025 and 2045 . 74 4.3 Method . 75 4.3.1 Participants . 75 4.3.2 Materials . 75 4.3.3 Analysis . 76 4.4 Results . 77 4.4.1 Participants . 77 4.4.2 Within subject differences . 77 4.4.3 Questionnaire . 77 4.4.4 Results summary . 78 4.4.5 Inter-rater reliability . 79 4.4.6 Current operational use of automated systems . 79 4.4.7 Future operational use of automated systems . 81 4.4.8 Exploration of the level of engagement between practitioners and system designers in the development of new automated systems . 83 4.4.9 Exploration of the opinions of RN personnel of the current con- sultation process . 84 4.5 Discussion . 86 4.5.1 Acknowledgements . 91 5 The design and development of ARCS 93 5.1 Abstract . 93 5.2 Introduction . 93 5.3 ARCS Development . 95 5.3.1 Materials . 95 5.3.2 Data collection . 96 5.3.3 Procedure . 97 5.3.4 Pilot Study Participants . 102 5.4 Analysis . 103 5.5 Results . 103 5.5.1 Use of tool . 103 5.5.2 Workload . 104 5.5.3 Task Performance . 108 5.5.4 Experience . 108 5.6 Discussion . 109 5.6.1 Recommended task changes/ARCS changes . 110 5.6.2 Conclusions of the development of ARCS and the benefit to research111 v 6 Using ARCS to explore automation usage decisions 113 6.1 Abstract . 113 6.2 Introduction . 113 6.2.1 How automation use can impact on task performance . 114 6.2.2 Motivational influences on automation usage decisions . 116 6.2.3 Cognitive processing styles and how they may modulate automa- tion uptake and use . 118 6.2.4 Summary . ..