VOICE INTERACTION GAME DESIGN AND GAMEPLAY Fraser Allison ORCID: 0000-0002-7005-7390 Submitted in total fulfilment of the requirements of the degree of Doctor of Philosophy January 2020 School of Computing & Information Systems Melbourne School of Engineering The University of Melbourne Abstract This thesis is concerned with the phenomenon of voice-operated interaction with characters and environments in videogames. Voice interaction with virtual characters has become common in recent years, due to the proliferation of conversational user interfaces that respond to speech or text input through the persona of an intelligent personal assistant. Previous studies have shown that users experience a strong sense of social presence when speaking aloud to a virtual character, and that voice interaction can facilitate playful, social and imaginative experiences of the type that are often experienced when playing a videogame. Despite this, the user experience of voice interaction is frequently marred by frustration, embarrassment and unmet expectations. The aim of this thesis is to understand how voice interaction can be used in videogames to support more enjoyable and meaningful player experiences. Voice- operated videogames have existed for more than three decades, yet little research exists on how they are designed and how they are received by players. The thesis addresses that knowledge gap through four empirical studies. The first study looks at player responses to a videogame character that can be given commands through a natural language interface. The second study is a historical analysis of voice-operated games that examines the technological and cultural factors that have shaped their form and popularity. The third study develops a pattern language for voice game design based on a survey of 471 published videogames with voice interaction features. The fourth study compares player responses to videogames that feature speech-based voice interaction and non-verbal voice interaction, and applies the theoretical perspective of frame analysis to interpret their reactions. Through these studies, the thesis makes two main contributions to the human- computer interaction and games studies literature. First, it identifies five genres of voice gameplay that are based upon fundamentally different types of vocal activities, and details the design patterns and design goals that are distinctive to each genre. Second, it presents an empirically grounded theoretical model of gameplay that accounts for players’ feelings of engagement, social presence, frustration and embarrassment during voice gameplay. Overall, the thesis demonstrates that the fictional framing a videogame presents is a crucial factor in determining how players will experience its voice interaction features. ii Declaration This is to certify that: i. the thesis comprises only my original work towards the PhD, except where indicated in the Preface ii. due acknowledgement has been made in the text to all other material used iii. the thesis is fewer than 100,000 words in length, exclusive of tables, figures, bibliographies and appendices. Fraser Allison January 2020 iii Preface Peer-reviewed publications included in the thesis This dissertation includes four published research articles, in accordance with The University of Melbourne guidelines for a Thesis with Publication.1 Chapter 4 presents the following peer-reviewed journal article: Allison, F., Luger, E., & Hofmann, K. (2018). How Players Speak to an Intelligent Game Character Using Natural Language Messages. Transactions of the Digital Games Research Association, 4(2), 1–50. https://doi.org/10.26503/todigra.v4i2.88 I led all stages of the research and writing process. The research design and data collection was conducted at Microsoft Research Cambridge, UK, during a 12-week research internship for which I took leave from my PhD studies. My supervisors for that research internship, Dr Ewa Luger and Dr Katja Hofmann, contributed feedback on the study design and written drafts. Chapter 5 presents the following peer-reviewed journal article: Allison, F., Carter, M., & Gibbs, M. (2017). Word Play: A History of Voice Interaction in Digital Games. Games and Culture, 1555412017746305. https://doi.org/10.1177/1555412017746305 I led all stages of the research and writing process. My thesis supervisors, Dr Marcus Carter and Assoc. Prof. Martin Gibbs, contributed feedback on the study design and written drafts. Chapter 6 presents the following peer-reviewed conference paper: Allison, F., Carter, M., Gibbs, M., & Smith, W. (2018). Design Patterns for Voice Interaction in Games. Proceedings of the 2018 Annual Symposium on Computer-Human Interaction in Play, 5–17. https://doi.org/10.1145/3242671.3242712 I led all stages of the research and writing process. My thesis supervisors, Dr Marcus Carter, Assoc. Prof. Martin Gibbs and Assoc. Prof. Wally Smith, contributed feedback on the study design and written drafts. Chapter 7 presents the following peer-reviewed conference paper: Allison, F., Newn, J., Smith, W., Carter, M., & Gibbs, M. (2019). Frame Analysis of Voice Interaction Gameplay. Proceedings of the 2019 CHI 1 The University of Melbourne. (2019). Graduate Research Training Policy (MPF1321). Retrieved from https://policy.unimelb.edu.au/MPF1321 iv Conference on Human Factors in Computing Systems, 393:1–393:14. https://doi.org/10.1145/3290605.3300623 I led all stages of the research and writing process. My fellow PhD candidate, Joshua Newn, contributed to the technical setup, data analysis and writing of the paper. My thesis supervisors, Wally Smith, Marcus Carter and Martin Gibbs, contributed feedback on the study design and written drafts. Peer-review publications based upon the thesis research Two further publications drew upon the research conducted for this thesis, but are not included in the thesis itself: Allison, F. (2016). A History of Voice Interaction in Games. Abstract Proceedings of the First International Joint Conference of DiGRA and FDG. DiGRA/FDG ’16. http://www.digra.org/digital-library/publications/a- history-of-voice-interaction-in-games/ Allison, F., Luger, E., & Hofmann, K. (2017). Spontaneous Interactions with a Virtually Embodied Intelligent Assistant in Minecraft. Proceedings of the 2017 CHI Conference Extended Abstracts on Human Factors in Computing Systems, 2337–2344. https://doi.org/10.1145/3027063.3053266 For both of these papers, I led all stages of the research and writing process. My co-authors on the second paper, Ewa Luger and Katja Hoffmann, contributed feedback on the study design and written drafts, and assisted with the data collection. Additional peer-reviewed publications produced during candidature The following papers were produced during my candidature, but did not draw upon or contribute directly to the research conducted for this thesis: Allison, F. (2015). Whose mind is the signal? Focalization in video game narratives. Proceedings of the 2015 DiGRA International Conference, 12. http://www.digra.org/digital-library/publications/whose-mind-is-the- signal-focalization-in-video-game-narratives/ Carter, M., Allison, F., Downs, J., & Gibbs, M. (2015). Player Identity Dissonance and Voice Interaction in Games. Proceedings of the 2015 Annual Symposium on Computer-Human Interaction in Play, 265–269. https://doi.org/10.1145/2793107.2793144 Allison, F., Carter, M., & Gibbs, M. (2015). Good Frustrations: The Paradoxical Pleasure of Fearing Death in DayZ. Proceedings of the Annual Meeting of the Australian Special Interest Group for Computer Human Interaction, 119–123. https://doi.org/10.1145/2838739.2838810 v Allison, F. (2016). Are You Out of Your Mind? Focalization in Digital Games. Transactions of the Digital Games Research Association, 2(3), 31-60. Carter, M., & Allison, F. (2017). Fear, loss and meaningful play: Permadeath in DayZ. Journal of Gaming & Virtual Worlds, 9(2), 143–158. https://doi.org/10.1386/jgvw.9.2.143_1 Newn, J., Velloso, E., Allison, F., Abdelrahman, Y., & Vetere, F. (2017). Evaluating Real-Time Gaze Representations to Infer Intentions in Competitive Turn-Based Strategy Games. Proceedings of the 2017 Annual Symposium on Computer-Human Interaction in Play, 541–552. https://doi.org/10.1145/3116595.3116624 Carter, M., & Allison, F. (2019). Guilt in DayZ. In K. Jørgensen & F. Karlsen (Eds.), Transgression in Games and Play. MIT Press. Newn, J., Allison, F., Velloso, E., & Vetere, F. (2018). Looks Can Be Deceiving: Using Gaze Visualisation to Predict and Mislead Opponents in Strategic Gameplay. Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, 1–12. https://doi.org/10.1145/3173574.3173835 Newn, J., Singh, R., Allison, F., Madumal, P., Velloso, E., & Vetere, F. (2019). Designing Interactions with Intention-Aware Gaze-Enabled Artificial Agents. In D. Lamas, F. Loizides, L. Nacke, H. Petrie, M. Winckler, & P. Zaphiris (Eds.), Human-Computer Interaction – INTERACT 2019 (pp. 255–281). Springer International Publishing. https://doi.org/10.1007/978-3-030-29384-0_17 Acknowledgement of funding This thesis was financially supported by an Australian Government Research Training Program Scholarship, top-up scholarships from the Microsoft Research Centre for SocialNUI and the Melbourne Networked Society Institute (both at The University of Melbourne) and a paid research internship at Microsoft Research Cambridge, UK. Thank you to everyone who facilitated this support. vi Acknowledgements I am enormously grateful to an enormous number of people. This thesis would