Evaluating Singleplayer and Multiplayer in Human Computation Games Kristin Siu Mahew Guzdial Mark O. Riedl School of Interactive Computing School of Interactive Computing School of Interactive Computing Georgia Institute of Technology Georgia Institute of Technology Georgia Institute of Technology [email protected] [email protected] [email protected] ABSTRACT alternative to traditional crowdsourcing platforms while providing Human computation games (HCGs) can provide novel solutions to potentially-entertaining experiences for their participants. intractable computational problems, help enable scientic break- However, the process of developing a human computation game throughs, and provide datasets for articial intelligence. However, is oen daunting. Choosing to build a game requires signicant our knowledge about how to design and deploy HCGs that appeal time and investment, especially for scientists and task providers to players and solve problems eectively is incomplete. We present who may not have experience designing and developing games. an investigatory HCG based on Super Mario Bros. We used this Complicating this is the fact that, unlike other games designed game in a human subjects study to investigate how dierent social primarily for entertainment, HCGs serve dual purposes. First, they conditions—singleplayer and multiplayer—and scoring mechanics— must provide an engaging player experience. Second, they must collaborative and competitive—aect players’ subjective experi- solve the corresponding human computation task eectively. ese ences, accuracy at the task, and the completion rate. In doing so, two design goals reect the preferences of players and the needs of we demonstrate a novel design approach for HCGs, and discuss the human computation task providers (e.g., scientists or researchers) benets and tradeos of these mechanics in HCG design. respectively. Ideally, HCGs aord both. Optimizing only for an engaging player experience may result in a game that does not prop- CCS CONCEPTS erly or eectively solve the underlying task. However, optimizing only for solving the underlying task may result in an uninteresting •Human-centered computing →Computer supported coop- game that will not aract or retain players, thus yielding few or erative work; •Applied computing →Computer games; poor task results. KEYWORDS When executed poorly, these games garner a reputation for be- ing unengaging for players and ineective for task providers [28]. human computation games; multiplayer; singleplayer; collabora- is is in spite of evidence that HCGs are an eective interface tion; competition; games with a purpose; game design for crowdsourced work, even when compared directly with other ACM Reference format: crowdsourcing platforms [13]. Understanding how gameplay me- Kristin Siu, Mahew Guzdial, and Mark O. Riedl. 2017. Evaluating Single- chanics aect both the player experience and completion of the player and Multiplayer in Human Computation Games. In Proceedings of human computation task is imperative in order to create eec- FDG’17, Hyannis, MA, USA, August 14-17, 2017, 10 pages. tive games. However, current HCG design is generally limited to DOI: 10.1145/3102071.3102077 templates and anecdotal examples of successful games, lacking generalized design knowledge or guidelines about how to develop 1 INTRODUCTION games for new kinds of tasks and changing player audiences. We need a beer understanding of how to design and develop Human computation games (HCGs) have been used to tackle vari- human computation games. is means exploring the HCG design ous computationally-intractable problems, such as classifying in- space and understanding how the elements of these games aect formation and discovering scientic solutions, by leveraging the both the player experience and completion of the underlying task. skills of human players. ese games, also known as Games with Specically, we want to understand and focus on game mechanics, a Purpose, scientic discovery games, or citizen science games, the rules that dictate what interactions players can have with the ask players to complete crowdsourcing problems or tasks by inter- game, as these are directly related to both the player experience acting with the mechanics of a game. e successful solutions to and the process of solving the human computation task. problems such as image labeling, protein folding, and more have In this paper, we explore the question of how human computation helped to demonstrate how games can be considered an eective games could benet from co-located multiplayer game mechanics Permission to make digital or hard copies of all or part of this work for personal or through a study comparing mechanical variations. Many mechanics classroom use is granted without fee provided that copies are not made or distributed in HCGs are designed to facilitate consensus and agreement as a for prot or commercial advantage and that copies bear this notice and the full citation on the rst page. Copyrights for components of this work owned by others than the way to verify the results (e.g., two players providing answers to author(s) must be honored. Abstracting with credit is permied. To copy otherwise, or the same problem and geing rewarded with in-game points if republish, to post on servers or to redistribute to lists, requires prior specic permission they agree). Traditionally, HCG players are isolated (i.e., prohibited and/or a fee. Request permissions from [email protected]. FDG’17, Hyannis, MA, USA from real-time communication during play) to prevent collusion © 2017 Copyright held by the owner/author(s). Publication rights licensed to ACM. that may impact results. On the other hand, co-located multiplayer 978-1-4503-5319-9/17/08...$15.00 experiences have been shown to be strongly-engaging for players DOI: 10.1145/3102071.3102077 FDG’17, August 14-17, 2017, Hyannis, MA, USA Kristin Siu, Mahew Guzdial, and Mark O. Riedl [32], including in domains such as education (where games are While human computation games have been shown to be an intended to both teach and engage) [12, 22]. Ultimately, we want eective interface for solving crowdsourcing tasks, current design to understand if co-located gameplay mechanics, based on their knowledge for developing these games remains limited. Commonly- success in other dual-purpose domains, might generalize to HCGs utilized guidelines for designing these games are oen based on while verifying if, and how, collusion may negatively impact human templates or anecdotal examples from successful games. ese computation task results. include von Ahn and Dabbish’s three game templates for classica- To beer examine these questions, we built a human compu- tion and labeling tasks [31] derived from their early successes with tation game based on the classic game Super Mario Bros. Using HCGs, and the design of the game mechanics in the protein-folding this game, we ran a user study to compare singleplayer and co- game Foldit [4]. However, it is not clear how these generalize to located (i.e., local) multiplayer gameplay experiences. Within the new kinds of tasks or changing player audiences. multiplayer experience, we also compare two variants: one using a Furthermore, there are no guidelines for ensuring game me- collaborative scoring system and one using a competitive scoring chanics will guarantee both an engaging player experience and system. successful completion of the task. Some researchers claim that hu- We evaluate these variations in gameplay mechanics using a hu- man computation games should use mechanics that directly map to man computation task with a known solution to see how changes in the process of solving the underlying task [11, 28], while others ar- mechanics aect the aspects of the player experience and the results gue that incorporating familiar or recognizable mechanics popular of the human computation task (which we refer to as task comple- in digital games will keep players more engaged [15]. is ongoing tion). We also report on the results of a survey asking experts in debate highlights the challenge of designing HCGs that optimize HCGs for their opinions and perceptions of these specic mechan- for both the player experience and the human computation task. ics. Using our study results and reported expert opinions, we high- light and discuss four design implications of our work—adaptation 2.2 Singleplayer and Multiplayer of successful gameplay mechanics to HCGs, use of direct player Researchers have studied the eects of singleplayer and multiplayer communication, synchronous competitive play, and synchronous in the context of mainstream digital games, but not human compu- collaborative play—and how these impact the player experience tation games. e gameplay of most HCGs is limited to singleplayer and the task completion. or networked multiplayer experiences; HCG players are generally e remainder of this paper is broken down as follows. First, not allowed to directly communicate during gameplay to avoid we review relevant work in the space of HCGs, game design, and potential collusion [31]. studies of player behavior. Next, we describe our Super Mario Bros.- Research on co-location in games validates that players respond inspired HCG, Gwario, and the methodology of our study. We then dierently when playing with or against other human players, present our results of the study, and report on expert opinions compared with singleplayer experiences or play against an articial on relevant HCG mechanics. is is followed by discussion of agent.