Player-AI Interaction: What Neural Network Games Reveal About AI as Play Jichen Zhu∗† Jennifer Villareale∗ Nithesh Javvaji∗ [email protected] [email protected] [email protected] Drexel University Drexel University Northeastern University Sebastian Risi Mathias Löwe Rush Weigelt [email protected] [email protected] [email protected] IT University Copenhagen IT University Copenhagen Drexel University Casper Harteveld∗ [email protected] Northeastern University Figure 1: Selection of neural network (NN) games. (From left to right, Top: How to Train Your Snake [108], Idle Machine Learning Game [141], Evolution [115], EvoCommander [114], Machine Learning Arena [117], Hey Robot [120]; Middle: Quick, Draw! [125], Semantris [126], Dr. Derks Mutant Battlegrounds [140], Forza Car Racing [139], Democracy 3 [122], Darwin’s Avatars [113], AudioinSpace [105]; Bottom: NERO [128], Black & White [106], Creatures [127], MotoGP19 [131], Supreme Commander 2 [121], Galactic Arms Race [119], Petalz [118]) ABSTRACT survey of neural network games (n = 38), we identified the dom- The advent of artificial intelligence (AI) and machine learning (ML) inant interaction metaphors and AI interaction patterns in these games. In addition, we applied existing human-AI interaction guide- arXiv:2101.06220v2 [cs.HC] 18 Jan 2021 bring human-AI interaction to the forefront of HCI research. This paper argues that games are an ideal domain for studying and exper- lines to further shed light on player-AI interaction in the context of imenting with how humans interact with AI. Through a systematic AI-infused systems. Our core finding is that AI as play can expand current notions of human-AI interaction, which are predominantly ∗These authors contributed equally to this research. productivity-based. In particular, our work suggests that game and † Currently at IT University of Copenhagen, Denmark. UX designers should consider flow to structure the learning curve of human-AI interaction, incorporate discovery-based learning to Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed play around with the AI and observe the consequences, and of- for profit or commercial advantage and that copies bear this notice and the full citation fer users an invitation to play to explore new forms of human-AI on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, interaction. to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]. CHI ’21, May 8–13, 2021, Yokohama, Japan © 2021 Association for Computing Machinery. ACM ISBN 978-1-4503-8096-6/21/05...$15.00 https://doi.org/10.1145/3411764.3445307 1 CHI ’21, May 8–13, 2021, Yokohama, Japan Zhu, Villareale and Javvaji, et al. CCS CONCEPTS gameplay. A neural network (NN) is a computational model that • Human-centered computing ! Human computer interac- includes nodes (i.e., neurons) and connections between these nodes tion (HCI); • Applied computing ! Computer games; • Com- that transmit information [76]. The strengths of these connections puting methodologies ! Artificial intelligence. (i.e., weights) are typically adjusted through some learning process. For our paper, this definition covers both NNs that control agents / KEYWORDS non-player characters (NPCs) in games [43, 88] and generative NN models that produce game content [33, 71]. Human-AI Interaction; Neural Networks; User Experience; Game We chose NN games for two key reasons. First, given the wide Design adoption of AI in games, we had to constrain our systematic (quali- ACM Reference Format: tative) review. Second, and more importantly, NN games provide Jichen Zhu, Jennifer Villareale, Nithesh Javvaji, Sebastian Risi, Mathias insights into some of the most pressing open problems in human-AI Löwe, Rush Weigelt, and Casper Harteveld. 2021. Player-AI Interaction: interaction. For example, NNs are notorious for UX designers to What Neural Network Games Reveal About AI as Play. In CHI Conference on work with because of NNs’ low interpretability of the underlying Human Factors in Computing Systems (CHI ’21), May 8–13, 2021, Yokohama, Japan. ACM, New York, NY, USA, 17 pages. https://doi.org/10.1145/3411764. process and the frequent unpredictability of its outcome. Studying 3445307 NN games can thus provide valuable information on how game designers have to work with these challenges. 1 INTRODUCTION We collected 38 NN games and applied a two-phased qualitative 1 analysis to examine them. In the first phase, we use close read- With the recent boom in artificial intelligence (AI) technology, peo- ing and grounded theory to identify the overarching interaction ple are interacting with a growing number of AI-infused products metaphors and patterns of how NNs are represented in the game in many aspects of everyday life. Already, these AI systems influ- user interface (UI). In the second phase, we apply current human-AI ence our decisions (e.g., recommendation systems [74]), inhabit our interaction design guidelines [6], compiled from a wide range of households (e.g., robotic appliances [80]), and accompany us in our productivity-based domains, to our dataset. From these analyses, playful experiences [54, 102, 103] and educational games [85, 99]. we derive design lessons for where games do well and identify The technological development has precipitated renewed in- open areas that can expand our current notion of human-AI inter- terest in the Human Computer Interaction (HCI) community. In action. A key design insight is that reframing AI as play offers a addition to improving the usability of individual products [58], HCI useful approach for considering human-AI interaction in games researchers synthesized design guidelines for human-AI interaction and beyond. from the past decades [6]. There is a growing recognition that, com- The core argument of this paper is that games are a rich and pared to traditional interactive systems, AI-infused products impose currently overlooked domain for advancing human-AI interaction. additional challenges (e.g., technical barrier, low interpretability) to The design space afforded by structuring AI as play, as game de- the current user experience (UX) design process [23]. Furthermore, signers have been exploring, can point out new opportunities for new interdisciplinary research areas have emerged around topics AI-infused products in general. At the same time, insights of the such as explainable AI [9, 66, 83, 101], ethics & fairness [12, 23, 38], generalized guidelines from other domains can be adapted to im- and machine learning (ML) as a design material for UX [93, 94]. prove player-AI interaction. The key contributions of this paper Among the fast-growing body of literature on human-AI inter- are as follows: action in the CHI community, one overlooked area is the context of play. With few exceptions, most recent literature focuses on • We propose the new research area of player-AI interaction. productivity-related domains such as e-commerce, navigation and Through the first systematic review on player-AI interaction autocomplete [6]. While these are important domains for human-AI in the context of NN games, we showcase how player-AI interaction, the history of AI and human-AI interaction has long interaction can expand the current productivity-based dis- been associated with play. For instance, ELIZA, one of the first AI cussions around human-AI interaction. programs designed to interact with lay users, was a playful satire • We adapted existing design guidelines for human-AI inter- of a certain school of psychotherapy [90]. Games such as Chess, action to the context of games. Currently, there are no syn- Poker, Go, and StarCraft have continued to serve as benchmarks thesized metrics to evaluate player-AI interaction. that propelled the development of AI since the beginning of the • We provide several insights from NN games (e.g., flow, ex- field. Since games naturally focus on end-user experience, gameAI ploration) to improve current challenges in human-AI inter- research has accumulated valuable knowledge related to human-AI action (e.g., learnability of AI). interaction [52, 54, 71, 86, 97, 100]. In this paper, we propose the new construct of player-AI interac- 2 RELATED WORK tion to highlight how people interact with AI in the context of play, In this section, we summarize related work in human-AI interac- especially through computer games. To provide an overview of tion and AI-based games research. We aim to bridge these two existing work in this area, we conducted the first systematic review disconnected areas. of player-AI interaction in the scope of Neural Network games — com- puter games in which players interact with an NN as part of the core 2.1 Human-AI Interaction 1Unless otherwise specified, we use the term AI broadly to include a wide rangeof The HCI community has developed a body of work on how to artificial intelligence and machine learning techniques. design user interactions to improve productivity through AI-based 2 Player-AI Interaction: What Neural Network Games Reveal About AI as Play CHI ’21, May 8–13, 2021, Yokohama, Japan applications [35, 39, 41, 78, 92]. Thanks to increasingly sophisticated co-creator, adversary, villain, or spectacle, and whether AI is vis- big data and deep neural networks (i.e., deep learning), AI-infused ible or guided. Cook et al. [16] further examined design patterns products have started to enter the consumer market, prompting in procedural content generation (PCG)-based games and derived a new surge of interest in human-AI interaction [6, 8, 63] and its different AI design patterns. In the context of assisting thegame societal impact in topics such as explainability and transparency [9, development process, Riedl and Zook [68] proposed that AI plays 66, 83, 101].