A Framework for Exploring and Evaluating Mechanics in Human Computation Games Kristin Siu Alexander Zook Mark O. Riedl School of Interactive Computing School of Interactive Computing School of Interactive Computing Georgia Institute of Technology Georgia Institute of Technology Georgia Institute of Technology [email protected] [email protected] [email protected] ABSTRACT to successfully complete the underlying human computation task. Human computation games (HCGs) are a crowdsourcing approach Balancing these two goals is dicult, oen resulting in conicting to solving computationally-intractable tasks using games. We out- design decisions. Unfortunately, very lile design knowledge exists line a formal representation of the mechanics in HCGs, providing a beyond a small number of simple paerns from examples or take- structural breakdown to visualize, compare, and explore the space aways from successful games (e.g., [6, 25]). As a result, most HCGs of HCG mechanics. We present a methodology based on small-scale to date are built around specic kinds of templates, leaving the design experiments using xed tasks while varying game elements space of possible HCG designs limited and relatively unexplored. to observe eects on both the player experience and the human To facilitate broader adoption and ease of game development, computation task completion. Ultimately, we wish enable easier HCG design needs the tools and frameworks to study and com- exploration and development of HCGs, leing these games provide municate about these games in a consistent manner. We need to meaningful experiences to players while solving dicult problems. understand precisely what game elements make certain HCGs suc- cessful, that is both eective at engaging players and solving tasks. CCS CONCEPTS A common language and structure for HCGs would allow us to talk about and explore the space of possible HCG designs. •Human-centered computing →Computer supported coop- In this desiderata, we describe a formal representation of HCG erative work; •Applied computing →Computer games; mechanics that provides us with a common vocabulary and struc- KEYWORDS ture to visualize, compare, and explore the space of game mechanics in HCGs. We advocate a methodology for building up HCG design human computation games; games with a purpose; scientic dis- knowledge, which uses small-scale, controlled design experiments covery games; game design; game mechanics on tasks with known solutions to understand how variations of ACM Reference format: game elements aect the player experience and the completion of Kristin Siu, Alexander Zook, and Mark O. Riedl. 2017. A Framework for human computation tasks. Further details and illustrative examples Exploring and Evaluating Mechanics in Human Computation Games. In of how our framework enables the comparative study of existing Proceedings of FDG’17, Hyannis, MA, USA, August 14-17, 2017, 4 pages. HCGs and the exploration of novel HCG mechanics can be found DOI: 10.1145/3102071.3106344 in an extended version of this paper [21]. 1 INTRODUCTION 2 BACKGROUND Games are everywhere. For human computation games (HCGs), Human computation games have been developed as an alternative games which harness the computational potential of the human to traditional crowdsourcing systems, providing players an engag- crowd, this diverse, increasing audience of players presents new ing gameplay experience while utilizing game mechanics to enable opportunities to solve complex, computationally-intractable tasks task completion. e original Game With a Purpose, the ESP Game, or generate data through gameplay. Already, HCGs—also known addressed the problem of labeling images [24]. Since then, HCGs as Games With a Purpose (GWAPs), scientic discovery games, and have been used to annotate or classify other kinds of information, citizen science games—have been used to solve a variety of problems from music [2, 14], to relational information [12, 19], to protein func- such as image labeling, protein folding, and data collection. tion recognition [18]. Other HCGs have leveraged human players However, one hurdle compounding HCG development compared as alternatives to optimization functions for “scientic discovery” with that of mainstream games for entertainment is that these problems such as protein [5] and RNA folding [15], DNA multiple games suer the design problem of serving two dierent goals. On sequence alignment [10], and soware verication [7]. Addition- the one hand, an HCG must provide a suciently-engaging experi- ally, HCGs have been used to collect or generate new information, ence for its players. On the other hand, an HCG must enable players such as creative content or machine-learning datasets. Examples in- clude photo collection [23], location tagging [3], and commonsense Permission to make digital or hard copies of part or all of this work for personal or knowledge acquisition [13]. Comprehensive taxonomies [11, 17] classroom use is granted without fee provided that copies are not made or distributed for prot or commercial advantage and that copies bear this notice and the full citation detail a wide breadth of HCGs and their tasks. on the rst page. Copyrights for third-party components of this work must be honored. HCG design has been primarily guided by examples of successful For all other uses, contact the owner/author(s). games. ese include von Ahn and Dabbish’s templates for classi- FDG’17, Hyannis, MA, USA © 2017 Copyright held by the owner/author(s). 978-1-4503-5319-9/17/08...$15.00 cation and labeling tasks [25] and the design anecdotes of Foldit [6] DOI: 10.1145/3102071.3106344 rather than systematic study of HCG elements. While these are FDG’17, August 14-17, 2017, Hyannis, MA, USA Kristin Siu, Alexander Zook, and Mark O. Riedl action, verication, and feedback mechanics. Further examples and player(s) discussion can be found the extended version of this paper [21]. 3.1 Action Mechanics action verification feedback Action mechanics are the interface for players to complete a human computation task through in-game actions or gameplay. ese mechanics align with the process of solving the human computation Figure 1: Breakdown of HCG mechanics. Players provide in- task, oen asking players to utilize skills necessary for solving the puts to take actions (in blue), which are veried (in orange), task during play. Such mechanics may be as simple as entering and receive feedback (in gray) from the game. Solid lines text input or as complicated as piloting a space ship in a virtual represent transitions through the gameplay loop. environment, and tend to vary based on the nature of the task. Examples. In the ESP Game, players provide labels through text useful, we do not understand what specic elements of these par- entry to solve the task of labeling given images. In Foldit, players ticular design choices work and how to appropriately generalize are given a variety of spatial actions, such as handling or rotating them or consider new alternatives. Confounding this issue is the components of a protein structure, to solve the task of “folding” a fact that HCG research remains divided on how game elements, given protein into a minimal energy conguration. In PhotoCity, in particular game mechanics, can ensure both engaging player players navigate to a desired location and take pictures using their experiences and successful completion of tasks. Some argue that camera phones, which are later uploaded to a database and used to HCG game mechanics should be isomorphic or non-orthogonal to construct a 3D representation of the buildings in that location. the underlying task [9, 22] while others argue that incorporation or adaptation of game mechanics from successful digital games 3.2 Verication Mechanics designed for entertainment can keep players more engaged [12]. Verication mechanics combine the output of player actions to com- Controlled studies utilizing quantitative and qualitative methods pute task-relevant outcomes. ese mechanics can support task to study the inuence of general game design elements have been completion outcomes including the quality, volume, diversity, and widely used in analogous, dual-purpose domains such as educa- the rate at which the data are acquired. tion [1, 16]. It is only recently that researchers have conducted similar studies on specic game elements of HCGs that jointly Examples. For many human computation tasks, consensus on address aspects of the player experience and the completion of player input oen serves as verication. e ESP Game (and other the human computation task [8], and advocated for their use [20]. structurally-similar games) verify using an online agreement check Combined with formal crowdsourcing research, we posit that these that lters correct answers from incorrect answers using agreement approaches can enable a formal study of HCG design. between players (Figure 2). e ESP Game later added “taboo word” mechanics to promote data diversity through banning words once 3 FORMALIZING HCG MECHANICS consensus on existing data was reached. By contrast, both Foldit and PhotoCity accomplish verication We outline a formal representation of the mechanics of human com- through task-based evaluation functions. Foldit’s protein congu- putation games. is representation serves three core functions: ration energy function determines the quality of player solutions (1) Provides a common vocabulary and visual organization of online. In PhotoCity, the game does not explicitly evaluate the HCG elements provided photos; photos are instead