Towards Playing Full MOBA Games with Deep Reinforcement Learning
Total Page：16
File Type：pdf, Size：1020Kb
Towards Playing Full MOBA Games with Deep Reinforcement Learning Deheng Ye1 Guibin Chen1 Wen Zhang1 Sheng Chen1 Bo Yuan1 Bo Liu1 Jia Chen1 Zhao Liu1 Fuhao Qiu1 Hongsheng Yu1 Yinyuting Yin1 Bei Shi1 Liang Wang1 Tengfei Shi1 Qiang Fu1 Wei Yang1 Lanxiao Huang2 Wei Liu1 1 Tencent AI Lab, Shenzhen, China 2 Tencent TiMi L1 Studio, Chengdu, China {dericye,beanchen,zivenwzhang,victchen,jerryyuan,leobliu,jaylahchen,ricardoliu, frankfhqiu,yannickyu,mailyyin,beishi,enginewang,francisshi,leonfu, willyang,jackiehuang}@tencent.com; [email protected] Abstract MOBA games, e.g., Honor of Kings, League of Legends, and Dota 2, pose grand challenges to AI systems such as multi-agent, enormous state-action space, complex action control, etc. Developing AI for playing MOBA games has raised much attention accordingly. However, existing work falls short in handling the raw game complexity caused by the explosion of agent combinations, i.e., lineups, when expanding the hero pool in case that OpenAI’s Dota AI limits the play to a pool of only 17 heroes. As a result, full MOBA games without restrictions are far from being mastered by any existing AI system. In this paper, we propose a MOBA AI learning paradigm that methodologically enables playing full MOBA games with deep reinforcement learning. Specifically, we develop a combination of novel and existing learning techniques, including curriculum self-play learning, policy distillation, off-policy adaption, multi-head value estimation, and Monte-Carlo tree-search, in training and playing a large pool of heroes, meanwhile addressing the scalability issue skillfully. Tested on Honor of Kings, a popular MOBA game, we show how to build superhuman AI agents that can defeat top esports players. The superiority of our AI is demonstrated by the first large-scale performance test of MOBA AI agent in the literature. 1 Introduction Artificial Intelligence for games, a.k.a. Game AI, has been actively studied for decades. We have witnessed the success of AI agents in many game types, including board games like Go [30], Atari series [21], first-person shooting (FPS) games like Capture the Flag [15], video games like Super Smash Bros [6], card games like Poker [3], etc. Nowadays, sophisticated strategy video games attract attention as they capture the nature of the real world [2], e.g., in 2019, AlphaStar achieved the grandmaster level in playing the general real-time strategy (RTS) game - StarCraft 2 [33]. As a sub-genre of RTS games, Multi-player Online Battle Arena (MOBA) has also attracted much attention recently [38, 36, 2]. Due to its playing mechanics which involve multi-agent competition and cooperation, imperfect information, complex action control, and enormous state-action space, MOBA is considered as a preferable testbed for AI research [29, 25]. Typical MOBA games include Honor of Kings, Dota, and League of Legends. In terms of complexity, a MOBA game, such as Honor of Kings, even with significant discretization, could have a state and action space of magnitude 1020000 [36], while that of a conventional Game AI testbed, such as Go, is at most 10360 [30]. MOBA games are further complicated by the real-time strategies of multiple heroes (each hero is uniquely 34th Conference on Neural Information Processing Systems (NeurIPS 2020), Vancouver, Canada. designed to have diverse playing mechanics), particularly in the 5 versus 5 (5v5) mode where two teams (each with 5 heroes selected from the hero pool) compete against each other 1. In spite of its suitability for AI research, mastering the playing of MOBA remains to be a grand challenge for current AI systems. State-of-the-art work for MOBA 5v5 game is OpenAI Five for playing Dota 2 [2]. It trains with self-play reinforcement learning (RL). However, OpenAI Five plays with one major limitation 2, i.e., only 17 heroes were supported, despite the fact that the hero-varying and team-varying playing mechanism is the soul of MOBA [38, 29]. As the most fundamental step towards playing full MOBA games, scaling up the hero pool is challenging for self-play reinforcement learning, because the number of agent combinations, i.e., 10 lineups, grows polynomially with the hero pool size. The agent combinations are 4,900,896 (C17 × 5 10 5 C10) for 17 heroes, while exploding to 213,610,453,056 (C40 × C10) for 40 heroes. Considering the fact that each MOBA hero is unique and has a learning curve even for experienced human players, existing methods by randomly presenting these disordered hero combinations to a learning system can lead to “learning collapse” [1], which has been observed from both OpenAI Five [2] and our experiments. For instance, OpenAI attempted to expand the hero pool up to 25 heroes, resulting in unacceptably slow training and degraded AI performance, even with thousands of GPUs (see Section “More heroes” in [24] for more details). Therefore, we need MOBA AI learning methods that deal with scalability-related issues caused by expanding the hero pool. In this paper, we propose a learning paradigm for supporting full MOBA game-playing with deep reinforcement learning. Under the actor-learner pattern [12], we first build a distributed RL infrastruc- ture that generates training data in an off-policy manner. We then develop a unified actor-critic [20] network architecture to capture the playing mechanics and actions of different heroes. To deal with policy deviations caused by the diversity of game episodes, we apply off-policy adaption, following that of [38]. To manage the uncertain value of state-action in game, we introduce multi-head value estimation into MOBA by grouping reward items. Inspired by the idea of curriculum learning [1] for neural network, we design a curriculum for the multi-agent training in MOBA, in which we “start small” and gradually increase the difficulty of learning. Particularly, we start with fixed lineups to obtain teacher models, from which we distill policies [26], and finally we perform merged training. We leverage student-driven policy distillation [9] to transfer the knowledge from easy tasks to difficult ones. Lastly, an emerging problem with expanding the hero pool is drafting, a.k.a. hero selection, at the beginning of a MOBA game. The Minimax algorithm [18] for drafting used in existing work with a small-sized hero pool [2] is no longer computationally feasible. To handle this, we develop an efficient and effective drafting agent based on Monte-Carlo tree search (MCTS) [7]. Note that there still lacks a large-scale performance test of Game AI in the literature, due to the expensive nature of evaluating AI agents in real games, particularly for sophisticated video games. For example, AlphaStar Final [33] and OpenAI Five [2] were tested: 1) against professionals for 11 matches and 8 matches, respectively; 2) against the public for 90 matches and for 7,257 matches, respectively (all levels of players can participate without an entry condition). To provide more statistically significant evaluations, we conduct a large-scale MOBA AI test. Specifically, we test using Honor of Kings, a popular and typical MOBA game, which has been widely used as a testbed for recent AI advances [36, 38, 37]. AI achieved a 95.2% win-rate over 42 matches against professionals, and a 97.7% win-rate against players of High King level 3 over 642,047 matches. To sum up, our contributions are: • We propose a novel MOBA AI learning paradigm towards playing full MOBA games with deep reinforcement learning. • We conduct the first large-scale performance test of MOBA AI agents. Extensive experiments show that our AI can defeat top esports players. 1In this paper, MOBA refers to the standard MOBA 5v5 game, unless otherwise stated. 2OpenAI Five has two limitations from the regular game: 1) the major one is the limit of hero pool, i.e., only a subset of 17 heroes supported; 2) some game rules were simplified, e.g., certain items were not allowed to buy. 3In Honor of Kings, a player’s game level can be: No-rank, Bronze, Silver, Gold, Platinum, Diamond, Heavenly, King (a.k.a, Champion) and High King (the highest), in ascending order. 2 2 Related Work Our work belongs to system-level AI development for strategy video game playing, so we mainly discuss representative works along this line, covering RTS and MOBA games. General RTS games StarCraft has been used as the testbed for Game AI research in RTS for many years. Methods adopted by existing studies include rule-based, supervised learning, reinforcement learning, and their combinations [23, 34]. For rule-based methods, a representative is SAIDA, the champion of StarCraft AI Competition 2018 (see https://github.com/TeamSAIDA/SAIDA). For learning-based methods, recently, AlphaStar combined supervised learning and multi-agent reinforcement learning and achieved the grandmaster level in playing StarCraft 2 [33]. Our value estimation (Section 3.2) shares similarity to AlphaStar’s by using invisible opponent’s information. MOBA games Recently, a macro strategy model, named Tencent HMS, was proposed for MOBA Game AI [36]. Specifically, HMS is a functional component for guiding where to go on the map during the game, without considering the action execution of agents, i.e., micro control or micro- management in esports, and is thus not a complete AI solution. The most relevant works are Tencent Solo [38] and OpenAI Five [2]. Ye et al. [38] performed a thorough and systematic study on the playing mechanics of different MOBA heroes. They developed a RL system that masters the micro control of agents in MOBA combats. However, only 1v1 solo games were studied without the much more sophisticated multi-agent 5v5 games. On the other hand, the similarities between this work and Ye et al.