Heuristic Search Techniques for Real-Time Strategy Games by David Churchill A thesis submitted in partial fulfillment of the requirements for the degree of Doctor of Philosophy Department of Computing Science University of Alberta c David Churchill, 2016 Abstract Real-time strategy (RTS) video games are known for being one of the most complex and strategic games for humans to play. With a unique combination of strategic thinking and dexterous mouse movements, RTS games make for a very intense and exciting game-play experience. In recent years the games AI research community has been increasingly drawn to the field of RTS AI research due to its challenging sub-problems and harsh real-time computing constraints. With the rise of e-Sports and professional human RTS gaming, the games industry has become very interested in AI techniques for helping design, balance, and test such complex games. In this thesis we will introduce and motivate the main topics of RTS AI research, and identify which areas need the most improvement. We then describe the RTS AI research we have conducted, which consists of five major contributions. First, our depth-first branch and bound build-order search algorithm, which is capable of producing professional human-quality build-orders in real-time, and was the first heuristic search algorithm to be used on-line in a starcraft AI competition setting. Second, our RTS combat simulation system: SparCraft, which contains three new algorithms for unit micromanagement (Alpha-Beta Considering Dura- tions (ABCD), UCT Considering Durations (UCT-CD) and Portfolio Greedy Search), each outperforming the previous state-of-the-art. Third, Hierarchical Portfolio Search for games with large search spaces, which was implemented as the AI system for the online strategy game Prismata by Lunarch Studios. Fourth, UAlbertaBot: our starcraft AI bot which won the 2013 AIIDE starcraft AI competition. And fifth: our tournament managing software which is currently used in all three major starcraft AI competitions. ii Acknowledgements I would like to extend my sincere thanks to my supervisor Michael Buro for introducing me to the topic of Artificial Intelligence and for always pushing me to be a better researcher. Thanks to my committee members Vadim Bulitko, Martin M¨uller, H´ector Mu˜noz-Avlia, and Scott Dick for their feedback and guidance, and to the members of the Real-Time Strategy games research group and the Heuristic Search group at the University of Alberta for the constant stream of ideas and motivation that kept me excited about my research. None of this work would have been possible without the love and support of my family and friends. Thank you to my mother who (almost) never com- plained about me being on my computer at all hours of the night, allowing me to find and develop my true passion in life which led me to pursuing grad school. I can never thank you enough for how supportive you were throughout my life and for how much that has contributed to the person I am today. Also, my warmest thanks to Natasha Bulat for being my better half, encouraging me every step of the way, and for always being there for me. iii Table of Contents 1 Introduction 1 1.1Real-TimeStrategyGames.................... 1 1.2Motivation............................. 2 1.2.1 CreatingBetterAIAgents................ 3 1.2.2 RTSAICompetitions.................. 4 1.2.3 GameDesign,Balance,andTesting........... 5 1.3 Thesis Outline ........................... 6 2 RTS Sub-Problems, Background, and Literature Survey 7 2.1Strategy.............................. 9 2.1.1 Knowledge and Learning ................. 10 2.1.2 Opponent Modeling and Prediction . ........ 10 2.1.3 StrategicStance...................... 12 2.1.4 ArmyComposition.................... 14 2.1.5 Build-Order Planning ................... 14 2.2Tactics............................... 15 2.2.1 Scouting.......................... 16 2.2.2 Combat Timing and Position .............. 16 2.2.3 Building Placement .................... 17 2.3ReactiveControl......................... 18 2.3.1 UnitMicro........................ 19 2.3.2 Multi-Agent Pathfinding and Terrain Analysis ..... 21 3 Build-Order Optimization 23 3.1Background............................ 24 3.2 Build-Order Planning Model for Starcraft ........... 25 3.2.1 Abstractions........................ 26 3.2.2 Algorithm......................... 27 3.2.3 ActionLegality...................... 27 3.2.4 Fast Forwarding and State Transition .......... 28 3.2.5 ConcurrentActionsandActionSubsetSelection.... 29 3.2.6 HeuristicsandMacroActions.............. 30 3.3Experiments............................ 31 3.4 Summary ............................. 38 4 RTS Combat Micromanagement 39 4.1 Modeling RTS Combat: SparCraft ............... 40 4.2SolutionConceptsforCombatGames.............. 42 4.2.1 ScriptedBehaviours................... 43 4.2.2 GameTheoreticApproximations............ 44 4.3FastSearchMethodsforCombatGames............ 46 4.3.1 SimultaneousMoveSequentialization.......... 46 4.3.2 EvaluationFunctions................... 48 iv 4.3.3 MoveOrdering...................... 49 4.4 Alpha-Beta Considering Durations ............... 50 4.4.1 ExperimentSetup..................... 51 4.4.2 InfluenceoftheSearchSettings............. 52 4.4.3 EstimatingtheQualityofScripts............ 52 4.4.4 Discussuion........................ 54 4.5UCTConsideringDurations................... 55 4.6PortfolioGreedySearch..................... 57 4.6.1 Algorithm......................... 58 4.6.2 Experiments........................ 60 4.6.3 Results........................... 64 4.6.4 Discussion......................... 71 4.7IntegrationIntoRTSAIAgents................. 72 4.7.1 StarCraftExperiments.................. 74 5 Hierarchical Portfolio Search and the Prismata AI 79 5.1AIDesignGoals.......................... 80 5.2HierarchicalPortfolioSearch................... 80 5.2.1 ComponentsofHPS................... 81 5.2.2 StateEvaluation..................... 82 5.3Prismata.............................. 84 5.3.1 Game Description ..................... 84 5.3.2 AIChallenges....................... 86 5.4PrismataAISystem....................... 88 5.4.1 AI Environment and Implementation .......... 88 5.4.2 HierarchicalPorfolioSearchinPrismata........ 89 5.4.3 AIConfigurationandDifficultySettings........ 90 5.5Experiments............................ 90 5.5.1 AIvs.HumanPlayers.................. 91 5.5.2 DifficultySettings..................... 91 5.5.3 UserSurvey........................ 94 5.6 Summary ............................. 95 6 Software Contributions 97 6.1UAlbertaBot........................... 97 6.1.1 Design........................... 97 6.1.2 StrategyandAISystems................. 99 6.1.3 CompetitionResultsandMilestones.......... 102 6.1.4 Impact and Research Use ................ 104 6.2TournamentManagerSoftware................. 105 6.2.1 Server........................... 106 6.2.2 Client........................... 108 7 Conclusion 110 7.1Contributions........................... 110 7.1.1 Build-Order Optimization ................ 110 7.1.2 RTSCombatMicromanagement............. 111 7.1.3 HierarchicalPortfolioSearch............... 111 7.1.4 Software Contributions . ................. 112 7.2 Directions for Future Research .................. 113 7.2.1 “Goal-less” Build-Order Search ............. 113 7.2.2 ImprovedCombatSimulation.............. 115 7.2.3 Machine Learning State Evaluations .......... 115 Bibliography 117 v List of Tables 4.1ABCDvs.Script-scoresforvarioussettings.......... 53 4.2Playout-basedABCDperformance............... 53 4.3 Real-time exploitability of scripted strategies. ......... 53 4.4 Sequence of events occurring after an attack command has been given in StarCraft. Also listed are the associated values of isAtk and atkFrm, the results of BWAPI unit.isAttacking() and unit.isAttackFrame() return values for the given step. This shows the non-triviality of something as intuitively simple of having frame-perfect control of unit actions in starcraft... 73 4.5 Results from the micro AI experiment. Shown are scores for Micro Search, AttackWeakest, and Kiter decision policies each versus the built-in starcraft AI for each scenario. Scores are shown for both the micro simulator (Sim) and the actual BWAPI-basedimplementation(Game).............. 77 5.1 Prismata Player Ranking Distribution ............. 93 5.2Searchvs.DifficultiesResults(RowWin%).......... 93 5.3 Search Algorithm Timing Results (Row Win %) ........ 93 6.1 UAlbertaBot results for major starcraft AI Competitions. Question mark indicates values that are unknown or not appli- cable................................ 104 vi List of Figures 2.1 The main sub-problems in RTS AI research categorized by their approximate time scope and level of abstraction. Arrows indi- cate the direction that information flows hierarchically through the different sub-problems, similar to a military command struc- ture................................. 8 3.1 Makespan vs. nodes searched for late-game goal of two carriers, comparing optimal search (K = 1) and approximate search with macro actions (K = 2). Macro actions make complex searches tractable while maintaining close to optimal makespans. 32 3.2 A sample search episode of BOSS applied to starcraft using the Protoss race, starting with 8 Probes and 1 Nexus, with the goal of building two Dragoon units in the quickest way possible. The left-most path is the first build-order found by algorithm 1 which satisfies the goal (makespan listed below in starcraft game frames). Each other leaf from left to right represents