SYNTHESIZING PLAY exploring the use of artificial intelligence to evaluate game user experience by samantha nicole stahlke A thesis submitted to the School of Graduate and Postdoctoral Studies in partial fulfillment of the requirements for the degree of master of science in computer science Faculty of Business and Information Technology university of ontario institute of technology (ontario tech university) oshawa, ontario, canada March 2020 Copyright © Samantha Nicole Stahlke, 2020 THESISEXAMINATIONINFORMATION Submitted by: Samantha Nicole Stahlke Master of Science in Computer Science Thesis Title: Synthesizing Play: Exploring the Use of Artificial Intelligence to Eval- uate Game User Experience An oral defence of this thesis took place on March 25, 2020 in front of the follow- ing examining committee: Examining Committee: Chair of Examining Committee Dr. Faisal Qureshi Research Supervisor Dr. Pejman Mirza-Babaei Examining Committee Member Dr. Jarek Szlichta Thesis Examiner Dr. Shahram Heydari The above committee determined that the thesis is acceptable in form and con- tent and that a satisfactory knowledge of the field covered by the thesis was demonstrated by the candidate during an oral examination. A signed copy of the Certificate of Approval is available from the School of Graduate and Postdoctoral Studies. ABSTRACT Digital games are a complex interactive medium providing a multitude of dif- ferent experiences. The field of games user research (GUR) is dedicated to in- vestigating and optimizing user experience in games. Such inquiries are of both commercial and academic importance, enhancing product quality and our under- standing of human behaviour. A common GUR methodology is usertesting, where researchers gain insights from human users interacting with products. However, usertesting is expensive in terms of expert labour, time, and resource costs. To address these concerns, we developed PathOS, a free, open-source tool for game testing with AI agents. PathOS simulates player navigation in games using a basic model of human behaviour. We conducted an evaluation of PathOS with develop- ers, finding that it provides valuable predictions of user behaviour in the iterative design process. Ultimately, we aim to give the game development community a useful and versatile augmentation to their testing processes. Keywords: games user research; human-computer interaction; user experi- ence; video games; artificial intelligence iii AUTHOR’SDECLARATION I hereby declare that this thesis consists of original work which I have authored. This is a true copy of the thesis, including any required final revisions, as accepted by my examiners. I authorize the University of Ontario Institute of Technology to lend this thesis to other institutions or individuals for the purpose of scholarly research. I further authorize the University of Ontario Institute of Technology to reproduce this thesis by photocopying or by other means, in total or in part, at the request of other institutions or individuals for the purpose of scholarly research. I understand that my thesis will be made electronically available to the public. The research work in this thesis that was performed in compliance with the regulations of the university’s Research Ethics Board under REB Certificate #15453. Samantha Stahlke iv For Victoria, from Elizabeth (and for loops) v STATEMENTOFCONTRIBUTIONS Some ideas and portions of figures have appeared previously in the following publications: Samantha Stahlke, Atiya Nova, and Pejman Mirza-Babaei. “Artificial Play- fulness: A Tool for Automated Agent-Based Playtesting”. In: Proceedings of CHI 2019 Extended Abstracts. Glasgow, Scotland, UK: ACM, 2019, LBW0176:1– LBW0176:6. doi: 10.1145/3290607.3313039 Samantha Stahlke and Pejman Mirza-Babaei. “Usertesting Without the User: Opportunities and Challenges of an AI-Driven Approach in Games User Re- search”. In: ACM Comput. Entertain 16.2 (2018), 18 pp. doi: 10.1145/3183568 Samantha Stahlke and Pejman Mirza-Babaei. “Usertesting Without the User: Introducing PathOS, a Framework for AI-Based Games User Research”. In: Proceedings of the 2017 CHI PLAY Workshop on Exploiting Players. Amsterdam, Netherlands, 2017, 5 pp. The prototype developed for this work (described in Chapter 3) was created with the help of Atiya Nova, who worked as a research assistant on this project. I wrote the majority of the code for the prototype presented here. A full history of all code contributions is available on the project’s Github page: https://github.com/samanthastahlke/pathos. The user study performed was conducted both remotely and on the university campus. I performed all of the interviews and was the primary contact for study participants. I hereby certify that I am the sole author of this thesis. All ideas and prior work from others referenced in this work has been attributed appropriately following standard practices. Parts of this work have been heavily edited and adapted for submission as a paper at ACM CHI PLAY 2020. The writing of this thesis predates the preparation of this submission. vi ACKNOWLEDGEMENTS Completing this research has been a long and exciting journey, which, while not without its difficulties, has been incredibly rewarding. I would not have been able to complete this work without the support of those around me. I would like to thank my supervisor, Dr. Pejman Mirza-Babaei, for his mentor- ship and advice. When I told him three years ago that I wanted to "make an AI stupid, like a person" to try and test games, he encouraged me to run with the idea and has helped me in this work ever since. I have learned so much from Pejman over the past five years, and I hope we can continue to collaborate in the years to come. I would also like to thank Atiya Nova, whom I have worked with on this project among many others. Her dedication and research ability are admirable, and her involvement in this work has made it all the better. For all the time and effort she put into this project, I am very grateful. To all my friends and colleagues in the graduate game lab and our faculty, thank you for putting up with my constant running around and non-stop chatter. For everyone who participated in this study, who shall remain anonymous for ob- vious reasons, I am incredibly appreciative of the fact that you were willing to lend your time to our research. This work would not exist without your participation. Ontario Tech University is a wonderful place, and I am proud to have called it my home for the past six years. I am immensely grateful for the university’s sup- port in providing the resources necessary for this work, in particular through the Dr. Deborah Saucier Early Researcher Award. I would also like to thank the Nat- ural Sciences and Engineering Research Council of Canada, the Vector Institute, and the Government of Ontario for additional funding supporting this work. On a personal note, I am most thankful for the support of those I love, through- out this process and always. My friends have been with me through many long nights of writing, coding, and knocking my head against a keyboard. Emilian, comrade, thank you for helping me turn every moment into an opportunity for laughter, and for being the best partner I could imagine. Helen, I’m glad we ended up in graduate school at the same time, for our joyful commiseration has been a lovely addition to our typical shenanigans. Josh and Owen, I couldn’t have made it through undergrad without you, and I’ll never forget all our adventures. Lastly, I would like to thank my mother, Svetlana, who has held me up through the past twenty-odd years of schooling it’s taken me to get here. Through all of the laughing, and the crying, and the inevitable yelling at submission systems into the odd hours of the morning, you’ve been there for me. Your love and patience has made me into the person that I am today, and none of this would have been possible without you. Thank you, for everything. vii CONTENTS i introduction1 1 introduction2 1.1 Games User Research . 3 1.2 AI and Games . 6 1.3 GUR and AI . 8 1.4 PathOS . 11 2 related work 14 2.1 AI for Analysis . 15 2.1.1 Evaluating User States . 15 2.1.2 Intelligent Data Analysis and Visualization . 17 2.1.3 Player Modelling and Clustering . 18 2.1.4 Behaviour Prediction . 20 2.2 Simulation-Driven Testing . 21 2.2.1 QA and Bugtesting . 21 2.2.2 Evaluating Playability . 23 2.2.3 Behaviour Simulation and UX Evaluation . 25 2.3 Evaluating AI GUR Tools . 28 2.3.1 Computational Performance . 28 2.3.2 Accuracy and Quality of Output . 29 2.3.3 Usability . 30 2.3.4 User Performance Benefit . 31 2.3.5 Other Considerations . 32 2.4 Summary . 33 ii the pathos framework 35 3 system design and development 36 3.1 System Overview . 37 3.1.1 Design Process and Justification . 38 3.1.2 System Prototype . 39 3.2 Agent Perception and Memory . 40 3.2.1 Agent "Sight" . 42 3.2.2 Agent Spatial Memory . 43 3.2.3 Agent Entity Memory . 45 3.3 Agent Decision-Making . 47 3.3.1 Game Entity Types . 48 3.3.2 Agent Motives . 51 3.3.3 Destination Evaluation . 54 3.4 User Interface and Data Visualization . 57 viii 4 system evaluation 64 4.1 Method . 64 4.1.1 Webinar . 64 4.1.2 User Study . 65 4.2 Results . 71 4.2.1 Participant Design Process and Broad Impressions . 72 4.2.2 Feature-Level Impressions . 79 5 discussion 83 5.1 User Study Results . 83 5.2 How can PathOS contribute to the game design process? . 84 5.2.1 How is PathOS useful? .