Echo: Reconstructing Gameplay Sessions for Analysis by Daniel MacCormick A thesis submitted to the School of Graduate and Postdoctoral Studies in partial fulfillment of the requirements for the degree of Master of Science in Computer Science Faculty of Business and Information Technology University of Ontario Institute of Technology (Ontario Tech University) Oshawa, Ontario, Canada April 2021 © Daniel MacCormick, 2021 THESIS EXAMINATION INFORMATION Submitted by: Daniel MacCormick Master of Science in Computer Science Thesis title: Echo: Reconstructing Gameplay Sessions for Analysis An oral defense of this thesis took place on April 7, 2021 in front of the following examining committee: Examining Committee: Chair of Examining Committee Dr. Faisal Qureshi Research Supervisor Dr. Loutfouz Zaman Examining Committee Member Dr. Pejman Mirza-Babaei Thesis Examiner Dr Andrew Hogue The above committee determined that the thesis is acceptable in form and content and that a satisfactory knowledge of the field covered by the thesis was demonstrated by the candidate during an oral examination. A signed copy of the Certificate of Approval is available from the School of Graduate and Postdoctoral Studies. ii ABSTRACT We created Echo, a tool designed to help bridge the gap between game analytics and video footage analysis. Echo reconstructs gameplay sessions from recorded data and presents them with their original graphics. A comparative evaluation to video footage analysis revealed that users preferred Echo overall and found it to be less frustrating. We later created Echo+, an expanded and improved version of the tool. A comparative evaluation of Echo+ across four popular genres - kart racing, first-person shooter (FPS), platformer, and tower defense – revealed that Echo+ was useful in them all to some degree, but there were no clear genres for which it was most or least useful. Echo+ was used differently across genres, with the camera features and visibility toggling being used most in the FPS game and tower defense game, respectively. User suggestions on how to improve Echo+ further included better representing player user interfaces within the visualization. Keywords: Game Analytics; Reconstruction; Playtest; Game Evaluation; Track; Genre ii AUTHOR’S DECLARATION I hereby declare that this thesis consists of original work of which I have authored. This is a true copy of the thesis, including any required final revisions, as accepted by my examiners. I authorize the University of Ontario Institute of Technology (Ontario Tech University) to lend this thesis to other institutions or individuals for the purpose of scholarly research. I further authorize University of Ontario Institute of Technology (Ontario Tech University) to reproduce this thesis by photocopying or by other means, in total or in part, at the request of other institutions or individuals for the purpose of scholarly research. I understand that my thesis will be made electronically available to the public. The research work in this thesis that was performed in compliance with the regulations of Research Ethics Board/Animal Care Committee under REB #15473 and #15975. Daniel MacCormick iii STATEMENT OF CONTRIBUTIONS Parts of the work described in Chapter 1, Chapter 2, and Chapter 7, as well as large sections of the work in Chapter 3 and Chapter 4, have appeared previously in the following published paper about Echo: MacCormick, D. and Zaman, L. 2020. Echo: Analyzing Gameplay Sessions by Reconstructing Them From Recorded Data. Proceedings of the Annual Symposium on Computer-Human Interaction in Play (New York, NY, USA, Nov. 2020), 281–293. Parts of Chapter 1, Chapter 2, Chapter 5, Chapter 6, and Chapter 7 (the sections pertaining to Echo+) are being adapted for submission to a journal. I conceptualized, designed, and developed the prototypes for Echo and Echo+, as described in Chapter 3 and Chapter 5, respectively, almost entirely on my own. The only exceptions being that I used a free asset, called Quick Outline, as part of the implementation, and some additions to Echo+ were suggested by participants in evaluation one. Quick Outline | Particles & Effects | Unity Asset Store: https://assetstore.unity.com/packages/tools/particles-effects/quick-outline-115488. Accessed: 2020-04-20. The source code for the locked version of Echo that was submitted to CHI PLAY 2020 can be found at this link: https://github.com/dmaccormick/Echo_CHIPLAY2020. The source code for the project in general, including the full history of commits and the additions to Echo+, can be found at this link: https://github.com/dmaccormick/Thesis. The NASA-TLX questionnaire software (see Appendix F and Appendix K) used in both evaluations was written by David Arppe. I modified it slightly for both evaluations and added the additional page in evaluation two. Phase one of the first evaluation was performed on the university campus, while all others were performed remotely online. I conducted all the sessions with participants. I hereby certify that I am the sole author of this thesis. I have used standard referencing practices to acknowledge ideas, research techniques, or other materials that belong to others. iv ACKNOWLEDGEMENTS Looking back on the two years of this master’s, I can pick out so many moments I will remember forever. It was challenging and exhausting at times, but also incredibly rewarding. I would not go back and change any of it. I would like to extend my sincere gratitude to my supervisor, Dr. Loutfouz Zaman. I have had the distinct pleasure of working with Loutfouz since midway through my undergraduate degree, for a total of around four years now. We have gotten to work together on a number of projects over the years, both large and small, and every single one has been exciting and valuable in its own way. Loutfouz has been incredibly supportive of me and has patiently guided me through this research at every step. I cannot thank him enough for everything he has done for me over the years, and I will be forever grateful. Ontario Tech was the perfect place for me throughout my undergraduate and graduate degrees. I would like to thank all of the professors over the years who got me this far and provided me with wonderful opportunities, especially Dr. Pejman Mirza-Babaei and Dr. Andrew Hogue. Thank you to all my friends in the lab. I really enjoyed spending time together there and chatting about all sorts of things, research-related or otherwise. I hope to visit everyone someday soon when we can all go back there in person. I would also like to thank my family for all of their love and support over the years. I could not have done this without them. Finally, I would like to thank my amazing girlfriend, Sam, for being an absolute rock over the years and for always being there to help when things got tough. This would never have been possible without her. Thank you all. For everything. v TABLE OF CONTENTS Chapter 1. Introduction ................................................................................................. 1 1.1 Background .......................................................................................................... 1 1.2 Motivation ............................................................................................................ 3 1.3 Echo and Echo+ ................................................................................................... 3 Chapter 2. Related Work .............................................................................................. 6 2.1 Game Analytics .................................................................................................... 6 2.1.1 Analytic Frameworks .................................................................................... 6 2.1.2 Temporal Analytics ....................................................................................... 6 2.1.3 Analytics Using Graphs ................................................................................ 7 2.1.4 Analytics using Trajectories.......................................................................... 8 2.1.5 Analyzing Trends ........................................................................................ 11 2.1.6 Analyzing Using Aggregation .................................................................... 12 2.1.7 Analyzing Using Animation ....................................................................... 13 2.2 Replay Systems in Commercial Games ............................................................. 15 2.3 Outside Digital Games ....................................................................................... 16 2.4 Game Genres ...................................................................................................... 18 2.4.1 Defining Game Genres ............................................................................... 18 2.4.2 Genres and Player Experience .................................................................... 18 2.4.3 Evaluations Within Different Game Genres ............................................... 20 2.5 Summary ............................................................................................................ 20 Chapter 3. Echo ............................................................................................................ 23 3.1 Basics of Echo .................................................................................................... 23 3.1.1 Static vs Dynamic Objects .........................................................................