Projective Replay Analysis: a Reflective Approach for Aligning Educational Games to Their Goals

PROJECTIVE REPLAY ANALYSIS: A REFLECTIVE APPROACH FOR ALIGNING EDUCATIONAL GAMES TO THEIR GOALS ERIK HARPSTEAD Human-Computer Interaction Institute School of Computer Science Carnegie Mellon University Pittsburgh, PA 15213 [email protected] CMU-HCII-17-107 August 2017 Submitted in partial fulfillment of the requirements for the degree of Doctor of Philosophy COMMITTEE: Vincent Aleven, HCII, Carnegie Mellon University, Chair Jodi Forlizzi, HCII, Carnegie Mellon University Jessica Hammer, HCII / ETC, Carnegie Mellon University Sharon Carver, Psychology, Carnegie Mellon University Jesse Schell, ETC, Carnegie Mellon University The research reported here was supported, in whole or in part, by the DARPA ENGAGE research program under ONR Contract Number N00014-12-C-0284 and by the Institute of Education Sciences, U.S. Department of Education, through grant R305B090023 to Carnegie Mellon University. All opinions expressed in this document are those of the author and do not necessarily reflect the position of the sponsoring agencies. Copyright © 2017 Erik Harpstead Keywords: Replay Analysis, Educational Game Design, Alignment For my Grandfather Dale and my Nephew Durinn ABSTRACT Educational games have become an established paradigm of instructional practice; however, there is still much to be learned about how to design games to be the most beneficial for learners. An important consideration when designing an educational game is whether there is good alignment between its content goals and the instructional behaviors it makes in order to reinforce those goals. Existing methods for measuring alignment are labor intensive and use complex auditing procedures, making it difficult to define and evaluate this alignment in order to guide the educational game design process. This thesis explores a way to operationalize this concept of alignment and demonstrates an analysis technique that can help educational game designers to both measure the alignment of current educational game designs and predict the alignment of prototypes of future iterations. In my work, I explore the use of Replay Analysis, a novel technique that uses in-game replays of player sessions as a data source to support analysis. This method can be used to capture gameplay experience for the evaluation of alignment, as well as other forms of analysis. The majority of this work has been performed in the context of RumbleBlocks, an educational game that teaches basic structural stability and balance concepts to young children. Using Replay Analysis, I leveraged replay data during a formative evaluation of RumbleBlocks to highlight some misalignments the game likely possesses in how it teaches some concepts of stability to players. These results led to suggestions for several design iterations. Through exploring these design iterations, I further demonstrate an extension of Replay Analysis called Projective Replay Analysis, which uses recorded student replay data in prototypes of new versions of a game to predict whether the new version would be an improvement. I implemented two forms of Projective Replay: Literal Projective Replay, which uses a naïve player model that replays past player actions through a new game version exactly as they were originally recorded; and Flexible Projective Replay, which augments the process with an AI player model that uses prior player actions as training data to learn to play through a new game. To assess the validity of this method of game evaluation, I performed a new replication study of the original formative evaluation to validate whether the conclusions reached through virtual methods would agree with those reached in a normal playtesting paradigm. Ultimately, my findings were that Literal Projective Replay was able to predict a new and unanticipated misalignment with the game, but Flexible Projective Replay, as currently implemented, has limitations in its ability to explore new game spaces. This work makes contributions to the fields of human-computer interaction by exploring the benefits and limitations of different replay paradigms for the evaluation of interactive systems; learning sciences by establishing a novel operationalization of alignment for instructional moves; and educational game design by providing a model for using Projective Replay Analysis to guide the iterative development of an educational game. i ii ACKNOWLEDGEMENTS A thesis is a funny thing. When you are writing one, it feels like it is the most consequential thing you have ever done—and at the time maybe it is—but at the same time there is a running joke, based on kernel of truth, that no one will ever read it. So, in a way it is like writing the most important footnote to history ever. I felt a similar way about writing my first CHI paper. All mountains seem like the tallest thing in the world until you get to the top and discover there are even bigger mountains even farther away. Adding to these complex feelings about the concept of a dissertation, is my opinion that the best research is always the product of collective effort. If I had it my way, single author publications would not even be allowed but that is not how this process of getting a PhD works. At the very least I want to take an opportunity to acknowledge all the other people who I feel contributed in some way to the ultimate production of this document and say thank you for helping me get here. I wouldn’t be here writing this in a very real sense if it weren’t for the assistance and advocacy of a few people. My path to CMU started while I was studying abroad in Japan and figured I should have a plan for what I was going to do that summer. I looked at a backlog of opportunity emails from the psychology department of IIT and found an opportunity to be a Pittsburgh Science of Learning Center intern working on educational technology at a school called Carnegie Mellon University, which, I’ll be honest, I hadn’t much heard of at the time. The good news was I still had time to apply because the deadline wasn’t until the next day, though being in Japan I really had 2 days. I quickly shot an email to Jo Bodnar who was the admin listed for the program and asked if I could have an extension if I could get her all of the required materials within a week and she agreed. Thanks to the heroic efforts of the IIT study abroad coordinator Louis Berends and my academic advisors at IIT Ruthana Gordon and Matt Bauer I managed to get everything in and was accepted to the program. When I got to CMU I joined the CTAT team working mainly with Martin Van Velsen, and Jonathan Sewall under Vincent Aleven. I had had a vague idea of going to graduate school after finishing my undergrad degree but no real plan for where to go or what to study. After my internship experience I knew CMU could be a place I could thrive and, even though I didn’t know much about it at the time, HCI would be the discipline I would pursue. With some guidance from Vincent, I applied to the PhD program in my senior year at IIT. The acceptance deadline came and went and I was concerned when I hadn’t heard anything. I learned that due to Vincent’s insistence I had been put on a wait list, which is something the department rarely does with PhD admissions. After another agonizing month of waiting I finally received the notification that a slot had opened up and I was officially accepted to the program. In joining the HCII PhD program I have had the opportunity and privilege to meet, work with, and learn from many great people: The fellow members of My PhD Cohort. In particular, my wife Kelly Rivers (who I met through the program) whose passion for teaching and making the world a better place has always inspired me; and whose support throughout my own dissertation work (to the point of taking dictation when I could not bring myself to write anymore) is honestly one of the reasons this document got completed in the first place. Sauvik Das, my longtime officemate and co-conspirator, who was always up for commiserating over the absurdity of academia (instead of working), and inventing myriad unrealized side-projects. Robert Xiao, the technical wizard, who was always willing (or at the very least able) to be distracted by crazy ideas and implement prototypes within hours. Samantha Finkelstein, the social adventurer, who kept all our lives interesting with her intense passion and excitement. My longtime collaborator Christopher MacLellan who has probably shaped my research trajectory more than anyone else. After discovering a synergy between his interest in studying students learning in design tasks and iii my corpus of students designing towers in RumbleBlocks (at a TG sponsored by Schell Games no less) we would go on to develop Conceptual Feature Extraction, TRESTLE, and the Apprentice Leaner Architecture together. For much of this work it has been hard to determine which ideas belong to who for the purposes of writing our respective dissertations. I have tried to denote some separation throughout this document but in truth I see most if not all of it as appropriately attributable to both of us. The many advisors and mentors I have found in the faculty of CMU and beyond. Vincent Aleven, my long-term advisor, who provided both guidance and counterpoint to my research ideas, as well as the freedom to explore new threads and possibilities. Brad Myers, my early co-advisor, helped me learn how to position an HCI contribution and whose voice I still hear in my head when writing and editing papers. The core members of the PIER steering committee, David Klahr, Sharon Carver, and Ken Koedinger, who helped me to appreciate the history, practice, and theory of learning science and education.

Load more