ISBN: 978-989-8704-20-7 © 2020 BALANCING INTRANSITIVE RELATIONSHIPS IN MOBA GAMES USING DEEP REINFORCEMENT LEARNING Conor Stephens and Chris Exton1 Lero – The Science Foundation Ireland Research Centre for Software, Computer Science & Information Systems (CSIS), University Of Limerick, Ireland 1Dr ABSTRACT Balanced intransitive relationships are critical to the depth of strategy and player retention within esports games. Intransitive relationships comprise the metagame, a collection of strategies and play styles that are viable, each providing counterplay for other viable strategies. This work presents a framework for testing the balance of massive online battle arena (MOBA) games using deep reinforcement learning to identify the synergies between characters by measuring their effectiveness against the other compositions within the games character roster. This research is designed for game designers and developers to show how multi-agent reinforcement learning (MARL) can accelerate the balancing process and highlight potential game-balance issues during the development process. Our findings conclude that accurate measurements of game balance can be found with under 10 hours of simulation and show imbalances that traditional cost curve analysis approaches failed to capture. Furthermore, we discovered that this approach reduced imbalance in each character's win rate by 20% in our example project a key measurement that would be impossible to measure without collecting data from hundreds of human-controlled games previously. The project's source code is publicly available at https://github.com/Taikatou/top- down-shooter. KEYWORDS Deep Reinforcement Learning, Game Balance, Design 1. INTRODUCTION The game balancing process aims to improve a game's aesthetic quality's and ensure the consistency and fairness of the game's systems and mechanics. Traditionally this process was achieved through a combination of data collected from playtesting analytical tools such as measuring the risk-reward ration of an item. This process is getting progressively more time consuming with rising levels of complexity in the game's design. To the constant updates that result in shifts in the game's balance. Game Designers have looked for options when testing the balance of games with reinforcement learning being a strong contender for the new solution. Reinforcement learning could potentially evaluate the quality of the game every evening when the developers are asleep, accelerating the project's timeline and giving designers more confidence when carrying out playtesting with participants. The sample problem this paper is focused on is an example project based on the popular MOBA game genre an asymmetric multiplayer game that is played in a square arena with a top-down perspective. We will evaluate the effectiveness of reinforcement learning when evaluating the balance of the game by measuring the effectiveness of different team compositions using accelerated simulated play controlled by deep reinforcement learning agents. 2. GAME DESIGN The hope of any game's design that is optimising the rules and content of a game to further progress it towards its aesthetic goals (Hunicke, Leblanc and Zubek, 2004). Common goals of Game balance is to ensure entertaining and fair games for the players (Adams, 2009). This can take the form of understanding various metrics about game mechanics and comparing them with other content in the game's systems. An example of the game balance proccess would be to balance a revolver gun; the designer could record the mean, median 126 International Conferences Interfaces and Human Computer Interaction 2020; and Game and Entertainment Technologies 2020 and standard deviation of the damage of the gun and compare it to the other weapons. This damage by itself may not highlight the revolver as being too strong; the gun may carry a cost to compensate for the damage it is doing, e.g. the player's speed is reduced by half. Understanding the benefit of the weapon vs the cost is the principle of a cost curve in a game, cost curves are a type of cost-curve. One of the best-known examples of cost-curve analysis is the Mana Curve in Magic: The Gathering (Flores, 2006) which can be described as the relationship the card game has with mana as input and power as the output. Where the best gameplay options are. As a player, you will choose these options and disregard options that are not as good, as a designer, you should pay attention to where new weapons and cards sit on this curve. 2.1 Game Balance Game balance can be described as the numeric properties of a game that makes players perceive the play as fair and remain enjoyable and challenging. Game Balance has traditionally been an analytical process of understanding data collected from play or using character stats to understand more transient game properties such as minimum time to kill within a game to the session length. Esports titles feature and tune multiple types of balance using a variety of tools and approaches; Situational Balance is the how different strategies are more favourable depending on the map or against the opponent's strategies. 2.2 Intransitive Relationships Intransitive Relationships consist of game rules involving the type of mechanic used. The most often used example is Rock, Paper, Scissors where the intransitive relationship consists of which class beats which other class (Adams, 2009). Traditional approaches to balancing games with intransitive relationships are to use the probability of how likely the character in question is to beat other characters; Rock Paper Scissors has a ratio of 1:1:1. This is simple with equal scoring, but games have an intransitive relationship with unequal scoring. These computations are traditionally calculated using rulesets but are not possible in MOBA games as the relationships are defined by playstyle and the different effectiveness of weapons and abilities. Currently, designers rely on player statistics and playtesting to compute the win rates and probabilities of these characters. Cost curve analysis is also an option; however, it is designed for single-character interactions and does not account for the situational balance of the game. 2.3 Metagames Early research on the topic of metagames coined the term metagame, paragame and orthogame (Carter, Gibbs and Harrop, 2012). The metagame is how players use outside influences to gain an advantage in the metagame; this is possible due to an externally sourced strategy and an understanding of some of the hidden information within the orthogame. Metagaming has different meanings regarding the different genres and games it can be featured in examples include playing the game differently that how your character would be able to play in tabletop role-playing games such as Dungeons and Dragons, where metagaming can give the player an advantage but breaks the aesthetic of the experience. 3. RELATED WORK The most relevant research in this area was carried out by King and evaluated the possibility of using Deep Learning to achieve Human performance playtesting on their match 4 "Crush Saga" games (Stefan Freyr Gudmundsson, 2018). This research showed the power of deep learning to simulate play, especially within such a high profile game; our research differs from this for two main reasons. Firstly it is a multiplayer game the outcomes of the next state are dependent on all four agents in the environment. Secondly, this research focuses on testing the game's level design we will be focusing on mechanics an area of the game that is defined much earlier on in development. Exploratory research for evaluating game balance in an adversarial game has been shown as possible using optimal agents (Jaffe et al., 2012). This research pioneered simulating games to evaluate the balance of card 127 ISBN: 978-989-8704-20-7 © 2020 games. The research identified the power differences between the intransitive relationship in the game, this being between (Green, Red and Blue). This research was carried out in symmetric perfect information game different from MOBA titles such as League of Legends and Dota II. New research into balancing decks of cards within the collectable card game Hearthstone (Silva, 2019) showed how genetic algorithms alongside simulated play could show how many viable options there are for players and to identify key compositions and decks that would be used if a card within the deck changes. This research shows how genetic algorithms can create fair games by changing the available mechanics given to both players within a very dense strategy space (over 2000 cards to evaluate). This was achieved by using an evolutionary algorithm to search for a combination of changes to achieve balance within the strategy space. This was measured by optimising the decks to have as close to a 50% win rate over a variety of conditions such as balancing the game with as few possible changes to each deck. 4. CHARACTER DESIGN The characters were designed as a simple intransitive relationship similar to other MOBA games; we have DPS (Damage-per-second), Tanks and Healers all bringing their unique utility. The DPS does the most damage and has greater mobility, the tanks are slower but have more sustain in the form of additional health, and the healers provide utility to themselves and other DPS roles. Each character has strengths and weaknesses Tanks cannot outrun DPS, DPS cannot survive by themselves for long periods due to a lack of sustain and healers cannot out damage the other roles. The design ambition of an asymmetric multiplayer game involves character roles complement each other. Players must work together as a team to overcome their opponents by choosing the most applicable characters in the current situation. Each character has an assigned weapon and different stats and abilities, as shown below. 4.1 Weapons Guns: Standard projectile weapon can fire every second, can hold ten bullets as ammo and takes 1 second to reload.