Measuring Inflation within Virtual Economies using Deep Reinforcement Learning Conor Stephens1;2 and Chris Exton1;2 1Computer Systems and Information Science, University of Limerick, Ireland 2Lero, Science Foundation Ireland Centre for Software Research, Ireland Keywords: Reinforcement, Learning, Games Design, Economies, Multiplayer, Games. Abstract: This paper proposes a framework for assessing economies within online multiplayer games without the need for extensive player testing and data collection. Players have identified numerous exploits in modern online games to further their collection of resources and items. A recent exploit within a game-economy would be in Animal Crossing New Horizons a multiplayer game released in 2020 which featured bugs that allowed users to generate infinite money (Sudario, 2020); this has impacted the player experience in multiple negative ways such as causing hyperinflation within the economy and scarcity of resources within the particular confines of any game. The framework proposed by this paper can aid game developers and designers when testing their game systems for potential exploits that could lead to issues within the larger game economies. Assessing game systems is possible by leveraging reinforcement learning agents to model player behaviour; this is shown and evaluated in a sample multiplayer game. This research is designed for game designers and developers to show how multi-agent reinforcement learning can help balance game economies. The project source code is open source and available at: https://github.com/Taikatou/economy research. 1 INTRODUCTION our to avoid these taxes and attempt to amass vast amounts of currency and items as quickly as possi- Video game economies suffer from a massive inter- ble. nal problem when compared to real-world economies. Game systems with MMO titles have been shown The creation of the currency in-game is tied to player to implode due to un-intended player behaviour. Ul- mechanics; an example would be when players re- tima Online, an incredibly successful early MMO re- ceive money when they defeat a monster. Systems leased in 1997 was an early example of the issues or mechanics that result in the creation of economic players could present to game designers. The cre- value is effectively printing money. This core issue ator Richard Garriott explained in an interview how has been the cause of rampant inflation within Mas- players destroyed the virtual ecology within seconds sively multi-player online games (MMO) (Achter- by over-hunting (Hutchinson, 2018). Within the game bosch et al., 2008). An early example was in Asherons where the spawning of new monsters was regulated to Call: the in-game currency became so inflated that a specific frequency; when the game went live prob- shards were used instead of money. Similarly, the de- lems started to surface, resources were being depleted velopers of Gaia Online would donate $250 to char- at an alarming rate, destroying the game’s ecosystem ity if the players discarded 15 trillion Gold (Gold was due to players over hunting herbivores for fun. To the in-game currency used for transactions) (Haufe, solve this problem future games did not restrict the 2014). Game designers would create a counterweight rate that enemies would spawn, MMO games such as to prevent problems within the game’s internal bal- World of Warcraft and Star Wars: The Old Repub- ance. Sinkholes are key tools for game’s designers lic uses sinkholes to manage the imbalance set within when designing economies within the game’s systems their economies. that remove money from the economy. Sinkholes The framework that we propose assesses inflation are mechanics that take currency out of the games within an in-game economy by measuring the price system, sometimes referred to as ”Drains” within of resources within the game over time. In this frame- machination diagrams (Adams and Dormans, 2012). work the economy is simulated by using reinforce- Sinkholes have taken the form of Auction Fees and ment learning agents playing as both the supplier and Consumable Items only available from NPC’s (Non- consumer within a simplified game economy both Playable Characters). However, players will endeav- 444 Stephens, C. and Exton, C. Measuring Inflation within Virtual Economies using Deep Reinforcement Learning. DOI: 10.5220/0010392804440453 In Proceedings of the 13th International Conference on Agents and Artificial Intelligence (ICAART 2021) - Volume 2, pages 444-453 ISBN: 978-989-758-484-8 Copyright c 2021 by SCITEPRESS – Science and Technology Publications, Lda. All rights reserved Measuring Inflation within Virtual Economies using Deep Reinforcement Learning aiming to gain more wealth through the game sys- within the well established framework and brought tems. This research aims to show the power of re- public acknowledgement to the suitability and eligi- inforcement learning for both Game Balance (Beyer bility of the pursuit of further creation and exploration et al., 2016) ensuring the game is challenging and fair within this space. for different types of players and also assessing the After the mid 2000’s the study, development and game’s economy for potential inflation allowing key research into modern video games ballooned in cor- stakeholders to better understand how changes within relation to their growing popularity and financial suc- the virtual economy can influence player interactions cess of the wider games industry. In the beginning such as trade without releasing the game or extensive of the 2000’s proceduralism was becoming success- user testing. This paper is built upon two clear objec- ful in games such as The Marriage and Braid (Tre- tives: anor et al., 2011) Games with proceduralism as a core 1. Measure the upper limits of inflation within a mul- mechanism of the narrative are process-intensive. In tiplayer game economy; these games, expression is found primarily in the player’s experience as it results from interaction with 2. Provide a tool for simulating different changes to the game mechanics and dynamics (Hunicke et al., a game’s economy for analysis. 2004). This marked the period of an explosion of AGD systems emerging from UC Santa Cruz. The first 2 RELATED WORKS would be Untitled System which is capable of creat- ing mini-games from natural text input. This system This work builds upon many different fields of game’s was the first AGD system to change the visuals of the design such as automated game design, game balance game based on the input by having access to an im- and game theory as well as concepts from artificial in- age database. Game-o-matic was the second AGD telligence and simulation such as deep reinforcement system that created a buzz in short succession (Tre- learning and Monte Carlo simulations. The history of anor et al., 2012). Game-O-Matic is a videogame these fields are briefly described below with relevance authoring tool and generator that creates games that to this paper. represent ideas. Through using a simple concept map input system, Game-O-Matic is able to assemble sim- 2.1 Automated Game Design ple arcade style game mechanics into videogames that represent the ideas represented in the concept map. Game Generation systems perform automated, intel- Game-O-Matic gave a higher quality experience pro- ligent design of games (Nelson and Mateas, 2007), viding a more interactive and engaging process when whilst having a lot of similarities with procedural con- seeding ideas (seeding refers to a number or vector tent generation; Game Generation is a sub discipline used to initialize a random number generator, within of Automated Games Design with the latter being a Game-O-Matic. The seed refers to the terms used to larger collection of tools, techniques and ambitions, describe the desired game). such as collaborative design tools and testing frame- works for games designers. Automated Game Design 2.2 Automated Game Testing (AGD) strives not to create content and assets for pre- existing games systems which comprises of goals, re- sources and mechanics (Sellers, 2017); AGD strives Developers and researchers have been experimenting to create these systems allowing unique experiences with machine learning to test their games. This testing and interactions to be generated. has a lot of different use cases and contexts depending on the department spearheading the initiative. Qual- 2.1.1 History of Game Design Systems ity assurance departments may consider employing trained bots/learning agents to test the game levels AGD was created to develop rulesets for games given and stories to detect bugs. Examples include: miss- a common structure such as chess and strategy board ing collision within the level geometry; finding holes games. The first example of such a system was in the narrative events that can hard/soft lock players Metagame (Pell, 1992). This was partly due to break- from progressing; modl.ai is an excellent example of throughs in Artificial Intelligence (AI). It was also a company which offers this service to games devel- partly due to the extensive history of these games and opers and publishers. Dev-Ops and the monitization the recognition they received in academia, highlight- groups of the game company could employ A/B test- ing the importance of such a genre and further explo- ing to assess the quality of tutorials, in-game-currency ration of the different design possibilities contained and menu systems on different user groups. Game de- 445 ICAART 2021 - 13th International Conference on Agents and Artificial Intelligence signers may also use artificial intelligence and simu- 2.3.1 Cost Curve Analysis lations as a way of assessing the quality metrics such as fairness (of level design) or session-length of the Game Balance was traditionally achieved using ana- game they are developing and by extension testing, lytical methods such as cost-curve analysis, spread- such efforts are made easier by using modern tool sheet and Excel macros.