University of California,Irvine Assessing Understanding of Complex Causal Networks Using an Interactive Game Dissertation submitted in partial satisfaction of the requirements for the degree of Doctor of Philosophy in Information and Computer Science by Joel Ross Dissertation Committee: Professor Bill Tomlinson, Chair Professor Paul Dourish Professor Bonnie Nardi 2013 © 2013 Joel Ross All Rights Reserved Table of Contents Page LIST OF FIGURES v LIST OF TABLES vii ACKNOWLEDGMENTS viii CURRICULUM VITAE x ABSTRACT OF THE DISSERTATION xv 1 Introduction 1 1.1 Assessing Causal Understanding . .8 1.2 Summary and Research Contributions . 14 1.3 Outline of this Research . 15 2 Background and Related Work: Causality 17 2.1 What is Causality? . 17 2.1.1 The Metaphysics of Causality . 18 2.2 Causality and Systems Thinking . 22 2.2.1 Emergence and Complexity . 23 2.2.2 Systems Thinking vs. Reductionist and Linear Thinking . 27 2.3 Causal Semantics . 30 2.3.1 Relata: Factors . 32 2.3.2 Relationships . 33 2.3.2.1 Causal Direction . 34 2.3.2.2 Causal Strength . 36 2.3.2.3 Causal Chains . 38 2.3.2.4 Other Causal Parameters . 40 2.4 Causality and Cognitive Mapping . 41 2.4.1 Visualizing Causality . 45 2.4.2 Constructing Causal Maps . 49 2.5 Conclusion . 52 ii 3 Background and Related Work: Games 53 3.1 Games and Play . 53 3.1.1 Engagement in Games . 59 3.1.1.1 Evaluating Engagement in Games . 63 3.1.2 Serious Games and Gamification . 64 3.2 Games and Education . 68 3.2.1 Games and Systems Education . 72 3.2.2 Games and Assessment . 76 3.3 Gamification Approaches . 79 3.3.1 Human Computation Games . 80 3.3.2 Other Gamification Approaches . 83 3.4 Conclusion . 87 4 Causlings Implementation 88 4.1 The Causlings Game System . 89 4.1.1 Causlings as Assessment . 97 4.1.2 Collecting Map Construction Data . 100 4.2 Implementation Details . 102 4.2.1 Architecture . 102 4.2.2 Prototype System Content . 103 4.2.3 Game Framing and Narrative . 105 4.2.4 Appearance . 107 4.3 Conclusion . 108 5 Evaluating Causlings 110 5.1 Study Participants . 112 5.1.1 High School Students . 114 5.1.2 University Students . 117 5.1.3 Mechanical Turkers . 118 5.2 Study Procedures . 120 5.2.1 Playing Causlings .............................. 120 5.2.2 In-Person Observations and Interviews . 122 5.2.3 Closing Survey . 124 5.3 Analysis Methodology . 126 5.3.1 How does the Causlings interactive system function as an assessment tool? 127 5.3.2 What forms do the causal maps constructed by players take? . 129 5.3.3 What patterns can be found in players’ sequence of map construction? . 131 5.3.3.1 Do players select islands or create bridges in a particular order? . 132 5.3.3.2 Do players construct causal maps hierarchically? . 132 5.3.3.3 Do players have conscious plans or strategies for constructing causal maps? . 133 5.3.4 What characteristics make Causlings effective or ineffective as an engag- ing game? . 134 5.4 Conclusion . 135 iii 6 Results and Analysis 136 6.1 How does the Causlings interactive system function as an assessment tool? . 138 6.1.1 Correct and Incorrect Bridges . 142 6.1.2 Supporting Learning . 149 6.2 What forms do the causal maps constructed by players take? . 151 6.2.1 Graph Structure . 154 6.2.2 Organization and Visual Structure . 160 6.3 What patterns can be found in players’ sequence of map construction? . 167 6.3.1 Do players select islands or create bridges in a particular order? . 167 6.3.2 Do players construct causal maps hierarchically? . 172 6.3.3 Do players have conscious plans or strategies for constructing causal maps? 177 6.4 What characteristics make Causlings effective or ineffective as an engaging game? 183 6.4.1 Game Narrative and Scoring . 186 6.4.2 Confusion and Instructions . 188 6.5 Conclusion . 190 7 Discussion 192 7.1 Causal Map Construction Reveals Failure of Systems Thinking . 193 7.1.1 Structural Forms Suggest Reductionist Thinking . 193 7.1.2 Construction Sequence Suggests Linear Thinking . 197 7.2 Games May Be Less Effective For Stand-Alone Assessment . 200 7.2.1 Playing a Game Requires Understanding the Complex Game System . 201 7.2.2 Assessment Contexts Do Not Condone Playful Feedback . 204 7.3 Contributions . 209 7.4 Broader Impacts . 212 7.5 Conclusion . 216 8 Future Work 217 8.1 Addressing Study Limitations . 217 8.2 Integration with The Causality Project . 219 8.3 Crowdsourcing Knowledge of Causal Relationships . 221 8.4 Conclusion . 223 9 Conclusion 224 References 230 Games Cited 251 iv List of Figures Page 1.1 A causal map, describing the causes and effects of and on faculty production. Map adapted from Burgess et al. (1992). 10 2.1 An example cycle of the Lotka-Volterra predator-prey model. Graphic used with permission from Wikimedia Commons. 25 2.2 A positive feedback loop. This example is adapted from Plate (2006), who adapted it from former Vice President Al Gore. 27 2.3 Effects of Public Transportation on Fossil Fuel Usage. 31 2.4 The four possible causal directions for a causal relationship. 35 2.5 An extended causal chain (not including any other causal relationships) that shows how biking leads to more adorable polar bear cubs. 39 2.6 Examples of Hasse (a) and an Ishikawa (b) diagrams. Images from Elmqvist and Tsigas (2004) and Kadaba et al. (2009), respectively. 46 2.7 Michotte’s launching effect. For animated demonstrations of this effect, see http: //www.yale.edu/perception/Brian/demos/causality.html......... 48 3.1 The flow channel is the "sweet spot" between an activity that is too easy and one that is too hard (graphic from Schell 2008). 61 4.1 An illustration of the Causlings (by Alicia Chan). 90 4.2 Players are shown an example of using increasing relationships ("positive bridges"), decreasing relationships ("negative bridges"), and causal loops–the last of which is not explicitly called such. 91 4.3 The Causlings game interface at the start of a new game. The help instruction tooltip disappears once the player begins playing. 93 4.4 The Causlings game being played (by me) in timed mode. The bright green dots are Causlings waiting to move, the yellow dot is a hesitating Causling, and the red dots are the disappearing remains of fallen Causlings. 95 5.1 The MTurk HIT, as seen by Turker participants. 120 6.1 Trendlines for changes in scores over multiple games for each player. 141 6.2 A player-built causal map using all 36 islands. This map has higher-than-average link density and WCI, and earned a rubric score of 22 points in basic mode. 156 v 6.3 A causal map with very high link density (3.17) and WCI (1.42). This map earned a rubric score of -17 points, as many of the large number of increasing bridges were incorrect. Constructed in basic mode. 157 6.4 A causal map with relatively low link density (0.8) and WCI (0.3). This player in timed mode was not able to finish connecting islands, resulting in the low density. This map earned an overall rubric score of 1 point. 157 6.5 A causal map that is a single linear chain, producing a WCI of 0.0. This player pro- duced this chain after 11 games of trials and continuous expansion (basic mode)— all the causal relationships are correct, earning an overall rubric score of 16 points. 158 6.6 A causal map with clearly grouped islands. One group surrounds "Samaki popula- tion" and deals with ecological issues, and the other group surrounds "Sales price per unit catch" and deals with economic issues. Map constructed in timed mode, and earned a rubric score of 65 points. 161 6.7 A causal map that is spatially organized (topics positioned from left to right). Only a few players laid out their causal maps in clean grids—and even maps that started on a grid often become more chaotic over time. Map constructed in basic mode, and earned a rubric score of 8 points. 162 6.8 A causal map coincidentally organized by category. Each branch of the map con- sists of a different topic: e.g., the branch on the far left deals with fish oil and Omega-3, while the branch on the right deals with the public controversy over the samaki. Map constructed in basic mode, and earned a rubric score of 29 points. 162 6.9.