The Gamification Inventory An Instrument for the Qualitative Evaluation of Gamification and its Application to Learning Management Systems Jan Broer October 25, 2017 A thesis submitted for the degree of Doctor of Engineering (Dr.-Ing.) to the faculty of Mathematics and Computer Science at the University of Bremen. 1st supervisor: Prof. Dr. Andreas Breiter 2nd supervisor: Prof. Dr. Thomas Köhler Abstract Gamification has risen meteorically in popularity since the beginning of the decade, both in practitioner circles and among researchers. We show that empirical results of gamification’s effects do not match the hype around it as studies have largely failed to prove any effects. We posit that a proper evaluation of gamification requires an understanding of how gamification can be expressed in real-world applications and employ Wittgensteinian family resemblances as a basis for such a definition. We have collected a set of properties that gamified applications can have through the analysis of goals and means of gamification mentioned in the literature and through an expert survey. We then used those results to create the Gamification Inventory, an instrument for the qualitative assessment of gamification in a given system. We have tested the instrument with a set of evaluators in the field of learning management systems (LMSs), informing both a refinement of the instrument and the preparation of an experiment with the intent of testing the effectiveness of common forms of gamification. The analysis of these LMSs led to results very similar to what our analysis of pre- vious empirical studies in gamification, and especially gamification in education, have shown: most gamification is concentrated on using points, badges, levels and leader- boards as game design elements. We argue that a large-scale, long-term experiment with a proper factorial design is needed to evaluate the effectiveness of gamification and have prepared such a study. Having identified points and badges as two major elements to be tested, we developed an extension to a competency grid add-on for the LMS Moodle that allows for a 2x2 factorial design of using points and badges. The system is designed for large-scale distribution among schools using the competency grid in Moodle, with minimal invasiveness in mind. We briefly discuss the challenges that come with such large-scale experiments, especially in German schools. As a result, we present a new, tested, and refined instrument for the qualitative as- sessment of gamification in a given system, an overview over gamification as it is being used in the most popular LMSs, and an experimental setup to test the effectiveness of points and badges in schools, using custom add-ons to the competency grid for Moodle and to the corresponding mobile application. Zusammenfassung Seit Anfang des Jahrzehnts sind Bekanntheit und Nutzung von Gamification in Anwender- und Forscherkreisen stark angestiegen. Es wird aufgezeigt, dass empirische Studien zur Effektivität von Gamification diesen Hype bisher nicht unterstützen. Um diese messen zu können, ist ein Verständnis der zu Grunde liegenden Mechaniken notwendig. Auf den folgenden Seiten wird ein Ansatz beschrieben, in dem Familienähnlichkeiten nach Wittgenstein die Basis für die Definition und Analyse von Gamification bilden. Über eine Kombination von Literaturanalyse, mit einem Fokus auf die Ziele, die mit Gamifica- tion verfolgt werden, und die Mittel, mit denen Systeme gamifiziert werden, sowie von Expertenmeinungen konnte eine Liste von Eigenschaften generiert werden, die indikativ für die Mitgliedschaft in der Familie gamifizierter Anwendungen sind. Aus dieser Liste wurde das Gamification Inventory erstellt, ein qualitatives Instrument für die Evaluation von Gamification in einem System. Das Instrument wurde von Evaluatoren an Lern- managementsystemen (LMS) getestet und damit sowohl das Inventory weiterentwickelt als auch eine Wissensbasis für ein Experimentaldesign geschaffen, mit dem die Effek- tivität von Gamification getestet werden kann. Die Ergebnisse der Analyse dieser LMS sind denen von existierenden Studien zum Thema sehr ähnlich: in den meisten Fällen ist Gamification auf Punkte, Abzeichen, Level, und Ranglisten limitiert. Es wird argumentiert, dass groß angelegte Experimente über einen längeren Zeitraum mit faktoriellen Designs notwendig sind, um die Effektivität von Gamification zu messen; daraufhin wird das Design eines solchen Experiments beschrieben, mit Punkten und Abzeichen als experimentellen Faktoren. Das LMS Moodle mit einem speziellen Kompe- tenzraster wurde entsprechend erweitert. Das resultierende System ist so gestaltet, dass ein großflächiger Einsatz an verschiedenen Schulen möglich ist, die Moodle mit dem Kompetenzraster verwenden, ohne dabei in den Schulalltag einzugreifen. Kurz wird die Herausforderung diskutiert, groß angelegte Experimente durchzuführen, insbesondere an deutschen Schulen. Als Ergebnisse der Arbeit werden ein neues, getestetes Instrument zur qualitativen Evaluierung von Gamification, eine Übersicht über den Einsatz von Gamification in LMS, und ein Experimentaldesign inklusive Software zur Messung der Effektivität vom Einsatz von Punkten und Abzeichen in Schulen präsentiert. Acknowledgments I would like to thank my supervisor Prof. Dr. Andreas Breiter for enabling me to follow my own research question and for guiding and supporting me in those years. Without his support and his contacts, this thesis would not have been possible. I would further like to thank Michael Stütz and the other members of the Konzeptgruppe Moodle as well as Martin Kattner and GTN Solutions for giving me the opportunity to expand the competency grid employed in Baden-Württemberg with my own experimental features and supporting me whenever I had questions. My colleagues at the Research Group Information Management (AGIM) and the Institute for Information Management (ifib) deserve all the thanks I can give for always being there for me when I needed them most. Unfortunately, my thanks to the participating experts and evaluators have to remain anonymous. However, this research would not have been possible without their patience and active participation in forming the Gamification Inventory. To everyone who has participated: Thank you so much, you know who you are. Finally, I would like to thank all my friends and family members who quite literally kept me alive all this time. Without you, there would never have been a thesis. Contents 1 Introduction 1 1.1 Gamification in Human Computer Interaction .................. 6 1.2 Education as an Application Context for Gamification ............. 10 1.3 Research Approach ................................ 12 2 Theory 15 2.1 Definitions of Games ............................... 17 2.1.1 Structured and Unstructured Play . 19 2.1.2 Characteristics of Games . 23 2.1.3 Game Definitions in Detail . 25 2.2 Gamification .................................... 36 2.2.1 Distinguation between Gamification and Related Terms . 40 2.2.2 Structuring Gamification . 45 2.2.3 Gamification Elements and Design Principles . 48 2.3 Overview on Theories of Motivation ....................... 64 2.3.1 Self-determination Theory . 65 2.3.2 Other Commonly Referenced Theories of Motivation . 71 2.3.3 Motivations to Play . 75 3 Related Work 79 3.1 Empirical Studies of Gamification ........................ 79 3.2 Gamification in Education ............................. 87 3.3 Summary of Previous Empirical Findings .................... 94 4 Analyzing and Categorizing Implementations of Gamification 99 4.1 Step 1: Goals and Means Analysis ......................... 102 4.1.1 Analysis of Definitions of Gamification . 105 v vi CONTENTS 4.1.2 Comparison of Definitions . 112 4.2 Step 2: Expert Survey ............................... 119 4.2.1 Methods: Expert Survey . 119 4.2.2 Results: Expert Survey . 123 4.3 Step 3: Creating the Gamification Inventory ................... 132 4.3.1 Methods: Creating the Gamification Inventory . 132 4.3.2 Results: Creating the Inventory . 133 4.4 Step 4: Test for Reliability and Refinement ................... 135 4.4.1 Methods: Test for Reliability and Refinement . 135 4.4.2 Results: Test for Reliability and Refinement . 138 5 The Effectiveness of Gamification in Learning Management Systems145 5.1 Qualitative Evaluation of Gamification in LMS .................. 147 5.2 Measuring the Effectiveness of Gamification .................. 151 5.2.1 Target Audience and System . 152 5.2.2 Experimental Design . 155 5.2.3 Data Collection . 159 5.2.4 Data Analysis . 163 5.2.5 Gamification Plug-in for DAKORA and Moodle . 165 5.2.6 Field Access . 175 6 Conclusion 177 6.1 Limitations ..................................... 181 6.2 Future Work .................................... 184 References 187 Appendices 213 Appendix A Starting Terms for the Expert Survey 215 Appendix B Terms with High Expert Ratings 217 Appendix C Gamification in LMS- Result Tables 221 Appendix D The Gamification Inventory 235 Appendix E Gamification Plug-ins for Moodle and DAKORA 255 Appendix F German Manual for our Plug-Ins 263 1 Introduction The playing of games has mostly been seen as an activity of children in the past, with much focus in academic studies on training and learning (c.f. Grubbauer, 2011, p.21ff). Famously, Piaget (1975) studied the early development of children with a focus on play and Huizinga (2004) studied the element of play in culture, also focusing on the play of children and young animals. Caillois (1979, p.57), discussing the study of games, noted that “they have generally been regarded as simple and insignificant pastimes