CHI 2018 Paper CHI 2018, April 21–26, 2018, Montréal, QC, Canada Video Game Selection Procedures For Experimental Research April Tyack, Peta Wyeth, Madison Klarkowski Queensland University of Technology (QUT) Brisbane, Australia [a.tyack, peta.wyeth, m.klarkowski]@qut.edu.au ABSTRACT likely to produce a desired PX. To our knowledge, no Videogames are complex stimuli, and selecting games that guidelines exist to aid researchers in this search. Studying consistently induce a desired player experience (PX) in an the wider literature for pointers is generally unhelpful: only experimental setting can be challenging. The number of a few publications describe this aspect of experiment design relatively high-quality games being released each year in detail, and some omit their reasons for selecting games continues to increase, which makes deriving a shortlist of entirely [45]. Concerns have long been raised regarding the plausible candidate games from this pool increasingly degree to which researchers understand the videogames problematic. Despite this, guidance for structuring and they study, and the corresponding validity of experimental reporting on the game selection process remains limited. research in the field (see [16] for a review). However, the This paper therefore proposes two approaches to game extent of the issue is obfuscated by a tendency for selection: the first leverages online videogame databases unsuccessful papers to go unpublished (i.e., the "file-drawer and existing PX research, and is structured with respect to problem" [57]). An apparent lack of justified reasoning for widely-applicable videogame metadata. The second process videogame selection is particularly jarring in the context of applies established game design theory to serve researchers a literature that tends to extol the medium when introducing when insufficient connections between desired PX new research. Videogames are complicated – for many, this outcomes and recognisable game elements exist. Both is indeed part of their appeal [32] – and richer descriptions methods are accompanied by example reports of their are hence required for readers to understand how candidate application. The present work aims to assist experimental games suit the goals of a study. Clear reporting practices researchers in selecting videogames likely to meet their more broadly are crucial for study replication, and assist in needs, while encouraging more rigorous standards of demonstrating the maturity of the field. reporting in the field. Indeed, conducting and reporting on a well-justified Author Keywords videogame selection process has a number of benefits. Video games; experiments; player experience; game Applying more rigorous criteria to game selection gives selection. researchers better chances of predictably inducing desired ACM Classification Keywords PX outcomes. Similarly, papers outlining this process are K.8.0 [Personal Computing]: General - Games better equipped to defend the validity of their results, and are more likely to invite further interest towards the subject. INTRODUCTION Well-designed, executed, and reported experiments provide Experimental and quasi-experimental studies constitute a reliable evidence about the nature, quality, and key segment of player experience (PX) research. characteristics of videogame experiences. Publications that Videogames themselves are useful experimental stimuli due feature sound methods are more likely to be positively to their potential in maintaining a balance between external received by the academic community and external funding validity and experimental control [44]. bodies alike; they contribute to authors’ reputations for However, the sheer volume of yearly releases presents an quality work, while demonstrating the merit of the broader obstacle for researchers attempting to select games most field and its suitability for further investment. A lack of versatile game selection procedures is a considerable barrier Permission to make digital or hard copies of all or part of this work for to adopting more detailed reporting practices, although personal or classroom use is granted without fee provided that copies are others (e.g., publication page limits) may also pose an issue. not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for How researchers can efficiently and accurately create a components of this work owned by others than the author(s) must be shortlist of appropriate stimulus games remains an open honored. Abstracting with credit is permitted. To copy otherwise, or question. The present work proposes two alternative republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from approaches to address this gap. The first involves applying [email protected]. videogame metadata to structure the game selection CHI 2018, April 21–26, 2018, Montreal, QC, Canada process. In this approach, links between individual © 2018 Copyright is held by the owner/author(s). Publication rights metadata elements and PX are made using existing licensed to ACM. ACM 978-1-4503-5620-6/18/04...$15.00 research, thus forming game selection criteria. Public https://doi.org/10.1145/3173574.3173760 Paper 186 Page 1 CHI 2018 Paper CHI 2018, April 21–26, 2018, Montréal, QC, Canada videogame databases also list a number of these metadata and confounding factors in their own work. In Study 1, for items, which allows for more efficient refinement in the example, they ignore factors included in the original study’s selection process. [3] pilot test (cell n=13), and instead observe a more obvious difference in control complexity: the non-violent The second approach to deriving game selection criteria is game uses only two keys, while the violent game requires based on the MDA framework [25]. This method assists mouse and keyboard input, with up to 20 keys. Based in researchers in translating a desired PX into more Self-Determination Theory, the researchers justified their recognisable game elements. Applying game design theory choice of covariate by applying existing work linking basic in this way is valuable in the event that relationships psychological need thwarting and aggression, together with between metadata elements and PX outcomes are not their more recent study findings connecting mastery of readily forthcoming, or when metadata-based criteria fail to controls in videogame play to competence need satisfaction. produce a shortlist of manageable length. This approach appears more successful than using the A paucity of published information exists to guide PX results of an underpowered pilot test: their hypotheses are researchers in selecting videogames likely to suit study supported both in the original text and the more recent aims. Authors rarely describe game characteristics in Bayesian re-analysis [22, 53]. We note, however, that sufficient detail for readers to understand the reasons for further evaluation of existing work linking game elements their inclusion in the research. We propose two videogame to PX would strengthen experiment design. selection methods based in existing user-centred McMahan et al. [44] highlight that stimulus games should frameworks and demonstrate their separate application to provide researchers with control over the experience of address these interrelated research gaps. play. Their own experiment involved Mario Kart Wii’s [47] BACKGROUND AND RELATED WORK Time Trial mode, which lacks the series’ random power- Improving the quality of experimental studies involving ups and game-controlled opponents, either of which could humans remains a high priority in fields such as health, introduce wild variation into participants’ experiences [44]. education, and psychology, particularly with respect to However, they also considered relatively normal parts of general study design [11, 19, 43], scale development and racing (e.g., crashing) as potential confounds, and we selection [41, 56, 59], and intervention design, selection, consider this excessive: some degree of variation across and use [7, 12, 35, 52]. In Human-Computer Interaction play sessions is desirable to avoid confounding the effect of (HCI), research has reviewed experimental design for videogame play with that of a constant stimulus [61]. usability testing [20, 21, 28], and the framing of user experience measures [5, 36], with a view towards Applying game modifications (“mods”) – external changes improving research quality. Recent work in the field has to a game’s assets – can be an effective way to exert control examined common issues in the reporting of statistical over unwanted aspects of a game’s design, or to create results in HCI publications [9, 15, 34]; for example, the stimuli more directly tailored to research goals [16]. While limits of null hypothesis significance testing. Videogame modding can be difficult and time-consuming, it may be researchers are presently concerned with improving necessary when desired game content is unavailable or measurement of game enjoyment [45] and PX more broadly otherwise unsuitable for research. However, the relatively [60]. small number of games that feature modding tools may restrict game selection when customised content is required. In proposing videogame selection methods, we aim to Researchers should strongly consider the trade-offs between complement this broad movement towards more robust commercial and custom-made (or modded) games