Empirical Analysis of User Data in Game Software Development The Story of Project Gotham Racing 4 Kenneth Hullett* Nachiappan Nagappan Eric Schuh John Hopson UC Santa Cruz Microsoft Research Microsoft Game Studios Bungie Studios Santa Cruz, CA, USA Redmond, WA, USA Redmond, WA, USA Bellevue, WA, USA [email protected] {nachin, eschuh}@microsoft.com [email protected] ABSTRACT For several years empirical studies have spanned the spectrum of General Terms research from software productivity, quality, reliability, Design, Measurement performance to human computer interaction. Analyses have Keywords involved software systems ranging from desktop software to Game design, Game development, Game metrics telecommunication switching systems. But surprising there has been little work done on the emerging digital game industry, one of the fastest growing domains today. To the best of our knowledge, our work is one of the first empirical analysis of a 1. EMPIRICAL RESEARCH AND GAMES large commercially successful game system. In this paper, we Empirical research in software engineering has typically focused introduce an analysis of the significant user data generated in the on software systems ranging from the traditional gaming industry by using a successful game: Project Gotham telecommunication systems to more recent web services. There Racing 4. has been little research on the software engineering aspects of More specifically, due to the increasing ubiquity of constantly digital games (a.k.a. video games, computer games, electronic connected high-speed internet connections for game consoles, games, etc.; referred to simply as games for the remainder of this developers are able to collect extensive amounts of data about paper). Games are increasingly becoming an important part of the their games following release. The challenge now is to make mainstream software development industry. sense of that data, and from it be able to make recommendations PricewaterhouseCoopers (PwC) report Global Entertainment and to developers. This paper presents an empirical case study Media Outlook: 2007-2011 estimates that the video game market 1 analyzing the data collected from a released game over a three will increase from $31.6 Billion in 2006 to $48.9 Billion in 2011 . year period. The results of this analysis include a better Games require significant software engineering effort and have understanding of the differences between long-term and short- become increasingly complex as games get more sophisticated term players, and the extent to which various options in the game [3]. Many of the issues in the development, production, and are utilized. This led to recommendations for future development testing of games reflect those of the general software engineering ways to reduce development costs and to keep new players community, and in many cases represent the state of the art. engaged. A secondary goal for this paper is to introduce software Research communities exist for specialized aspects of game game development as a topic of importance to the empirical development, such as SIGGRAPH’s game track [15] for graphics software engineering community and discuss research results on a or AAAI’s Artificial Intelligence and Interactive Digital key difference area: data analytics on user data to customize user Entertainment for game AI [2]. There are already workshops in and development experiences. this regard that have been held co-located to the International Conference of Software Engineering [12]. That said, games are a Categories and Subject Descriptors significantly wide field and in this paper our goals are twofold: D.2.8 [Software]: Metrics, K.8.0 [Personal Computing]: Games Identify a specific area of research and characterize its operation in the gaming community Investigate via data analytics the ability to improve game design Permission to make digital or hard copies of all or part of this work for There are several differences between software development for personal or classroom use is granted without fee provided that copies are games compared to software development for traditional software not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, * Kenneth Hullett was an intern at Microsoft Research when this work requires prior specific permission and/or a fee. was performed. ESEM’12, September 19–20, 2012, Lund, Sweden. 1http://www.businessweek.com/innovate/content/aug2007/id20070813_12 Copyright 2012 ACM 978-1-4503-1056-7/12/09...$15.00. 0384.htm 89 systems. It is beyond the scope of this paper to assess these Dixit and Youngblood performed user tests and used the data differences. In this paper we focus on one particular aspect, the collected to create visualizations showing where users’ attention importance of user testing. In recent years there has been a rise in was focused during gameplay [4, 5]. This information was then interest in the collection and analysis of game metrics, and how used to determine the best places to put relevant information to they can be used to inform the game development process. As improve recall. The implications for game design are to give games have gotten larger and more complex, the need for such designers a better idea of where to place important clues and other metrics to make sense of player behavior has increased. The information so that players are most likely to see it. number of reachable states in a modern commercial game title is enormous; without some way to simplify and represent collected Kim et al. presented TRUE, a system for collection and data development teams would be unable to act on it in a timely visualization of data from user studies, and presented a case study matter. of its use in Halo 2 [8], a popular First-Person Shooter (FPS) game. Their studies specifically looked for unintended difficulty Telemetry, or the collection of metrics, has become increasingly increases introduced during development. Their tests collected common in game development. As games have become more data on player deaths and surveyed the subject's opinions on complex traditional playtesting is no longer able to provide difficulty. They were able to identify several unbalanced elements sufficient coverage of all possible gameplay states or reveal in the game and correct them before release. potential emergent elements. Playtesting refers to the user testing wherein data is collected from players playing the game to Weber and Mateas used data mining techniques on large amounts identify defects and improve customer experience. This makes of collected data to understand player strategies in the game long-term metrics collection the only viable means to understand StarCraft. Over 5000 replays of expert matches were used as players and how they interact with the game. training data for a machine learning algorithm that predicted player strategies [20]. This predictor became a component of an But just collecting data is not enough, the data has to be distilled AI bot that played StarCraft better than most other available and interpreted before it can be used to inform development techniques, thus helping to improve game AI. decisions. A detailed accounting of players’ in-game actions is difficult to interpret even for a developer who is intimately Weber et al. used analysis of large datasets generated by players familiar with the game. Simply knowing what a player did at a of the game Madden NFL 11 to understand player retention [19]. certain time means nothing without the context of what they did They selected gameplay features to use as input to a machine previously, what they did afterwards, and how that relates to the learning algorithm that tried to predict how many games a player larger patterns of behavior throughout the game. would play overall. The selected features were then studied to make recommendations that would increase player retention in the Previous academic work studied data on smaller scales in limited game. domains, and case studies from industry have shown ways various types of data can be used in to aid the development process. Our Lewis and Wardrip-Fruin presented a case study of large-scale aim is to unite and advance these traditions by presenting a case data collection and interpretation of World of Warcraft study of analysis of large-scale data collected from Project repositories for better understanding of player behavior [9]. They Gotham Racing 4, a popular commercially released game. analyzed how long it took players from each class to reach level 80 (the highest level) in order to empirically evaluate whether the This paper presents our case study by explaining the domain, our game design is balanced, and confirm or refute common folklore analyses, the conclusions we drew, and recommendations we were surrounding the game. able to make. Some areas we explore include: Miller and Crowcroft also studied World of Warcraft, but instead Factors that hinder a player’s advancement looked at player movement [10]. They analyzed several gameplay Differences between long-term and short-term players traces that utilized the same battleground to find patterns in player Differences between multi- and single player usage traffic. They were able to identify distinct patterns of player How players interact with the game in their first ten behavior, including patrollers, who moved between multiple races and how this relates to long term behavior points along standard routes, and guards who tended to remain Utilization rates of various game play options and within a small area. This analysis is useful to developers by factors that contribute to them showing how players are interacting with the game environment. This paper is organized as follows. In Section 2 we discuss the B. Industry related work. In Section 3 we explain the various sources of data Articles in industry-focused publications like Gamasutra are a in games via quantitative (testing) and qualitative (subjective good source of information for ways in which data is used in the evaluations) aspects to characterize the process on how the game game industry.