Applying Artificial Intelligence and Machine Learning Techniques to Create Varying Play Style in Artificial Game Opponents Nicholas John Sephton Doctor of Engineering University of York Computer Science October 2016 Abstract Artificial Intelligence is quickly becoming an integral part of the modern world, employed in almost every modern industry we interact with. Whether it be self-drive cars, integra- tion with our web clients or the creation of actual intelligent companions such as Xiaoice1, artificial intelligence is now an integrated and critical part of our daily existence. The application of artificial intelligence to games has been explored for several decades, with many agents now competing at a high level in strategic games which prove challenging for human players (e.g. Go [127] and Chess [73]). With artificial intelligence now able to produce strong opponents for many games, we are more concerned with the style of play of artificial agents, rather than simply their strength. Our work here focusses on the modification of artificial game opponents to create varied playstyle in complex games. We explore several techniques of modifying Monte Carlo Tree Search in an attempt to create different styles of play, thus changing the experience for human opponents playing against them. We also explore improving artificial agent strength, both by investigating parallelization of MCTS and by using Association Rule Mining to predict opponent’s choices, thus improving our ability to play well against them. 1http://www.msxiaoice.com/ 2 Contents Abstract 2 List of Contents 3 List of Figures 9 Acknowledgements 13 Author’s Declaration 15 1 Introduction 16 1.1 Background . 19 1.2 Research Questions . 21 1.2.1 Can modification of MCTS create artificial agents that affect playstyle? 22 1.2.2 Can we create an artificial agent which displays modified behaviour without compromising play strength? . 23 1.2.3 Can the application of game knowledge improve rule artificial intel- ligence techniques for modelling opponent knowledge? . 23 1.3 Publications . 23 1.4 Thesis Overview . 24 2 Literature Review 25 2.1 Introduction . 25 2.2 Games & Game terminology . 26 2.2.1 Game Terminology . 26 3 CONTENTS 2.2.2 Formal Game Definition . 27 2.2.3 Combinatorial Games . 27 2.2.4 Games of Imperfect Information . 28 2.2.5 Nash Equilibrium . 29 2.2.6 Pure versus Mixed Strategies . 30 2.3 Basic AI techniques & Concepts . 31 2.3.1 Classic AI Methods . 31 2.3.2 Move Pruning . 32 2.4 Monte Carlo Tree Search (MCTS) . 33 2.4.1 Multi-armed bandit . 33 2.4.2 Monte Carlo Tree Search . 35 2.4.3 Upper Confidence Bound applied to Trees (UCT) . 40 2.4.4 Flat Monte Carlo . 43 2.4.5 Counterfactual Regret (CFR) . 43 2.5 MCTS Enhancements & Variations . 44 2.5.1 Perfect Information Monte Carlo (PIMC) . 44 2.5.2 Transposition Tables . 47 2.5.3 Progressive Strategies . 48 2.5.4 Value Approximation Techniques . 49 2.5.5 Information Set MCTS . 51 2.5.6 Genetic Programming in MCTS . 51 2.5.7 Parallelization of MCTS . 52 2.6 Entertaining Play Experiences . 55 2.6.1 Making more human-like moves . 55 2.6.2 Entertaining Play . 56 2.6.3 Human-like play . 57 2.6.4 The importance of challenge to an entertaining experience . 58 2.7 Opponent Modelling . 59 2.8 Data Mining Techniques . 60 2.8.1 Data Preparation . 60 4 CONTENTS 2.8.2 Association Rule Mining . 61 2.8.3 The a priori Algorithm . 61 2.8.4 Data Mining for Games . 62 2.9 Summary . 63 3 Game Domains and Experimentation Software 64 3.1 Game Domains . 64 3.1.1 Lords of War . 64 3.1.2 Android: Netrunner . 66 3.2 Experimentation Hardware & Software . 67 3.2.1 MCTSTree . 68 3.2.2 Move & Game State interface . 68 3.2.3 Memory Management . 69 3.2.4 Engine Profiling . 69 3.3 Summary . 71 4 Parallelization of Information Set MCTS 73 4.1 Experimental Methodology . 75 4.1.1 Root Parallelization . 75 4.1.2 Tree Parallelization . 76 4.1.3 Leaf Parallelization . 77 4.1.4 Iteration Budget Experimentation . 78 4.1.5 Win Percentage Experimentation . 81 4.1.6 Experimentation Environment . 82 4.2 Results . 82 4.2.1 Iteration Budget Results . 82 4.2.2 Win Percentage Results . 85 4.3 Summary & Discussion . 87 4.3.1 Contributions . 90 5 CONTENTS 5 Modifying MCTS to Create Different Play Styles 92 5.1 Move Pruning . 93 5.2 Action Selection Mechanisms . 94 5.3 Experimental Methodology . 94 5.3.1 State Evaluation . 94 5.3.2 Single Heuristic Experimentation . 98 5.3.3 Multi-Heuristic Experimentation . 99 5.3.4 State-Extrapolation . 100 5.3.5 Action Selection Mechanism . 100 5.3.6 Complexity Measurements . 102 5.3.7 Online tuning . 104 5.4 Results . 104 5.4.1 Single Heuristic Results . 104 5.4.2 Multi-heuristic Results . 107 5.4.3 State Extrapolation . 108 5.4.4 Max Techniques . 110 5.4.5 Robust Techniques . 111 5.4.6 Further action selection mechanisms . 113 5.4.7 Varying Iterations . 114 5.4.8 Complexity Measure . 116 5.4.9 Online tuning . 118 5.5 Summary & Discussion . 119 5.5.1 Heuristic Move Pruning . 119 5.5.2 Action Selection Mechanisms . 120 5.5.3 Contributions . 121 6 Rule Association Mining for Opponent Modelling in Games 124 6.1 Experimental Methodology . 125 6.1.1 Netrunner Deck Data . 125 6.1.2 Apriori Rule Generation . 126 6 CONTENTS 6.1.3 Apriori Prediction . 127 6.2 Results . 134 6.2.1 Default Apriori (a1)......................... 135 6.2.2 Apriori with duplicates (a2)..................... 136 6.2.3 Apriori with Influence Priority (a3)................. 136 6.2.4 Apriori with Influence Filtering (a4)................. 137 6.2.5 Rule Generation including duplicate cards (a5)........... 137 6.2.6 Prioritising by rulesize (a6)...................... 137 6.2.7 Making confident predictions (a7).................. 138 6.2.8 Varied Size Observation Set . 138 6.2.9 Deck Creation . 141 6.3 Summary & Discussion . 142 6.3.1 Contributions . 142 7 Conclusions & Further Work 144 7.1 Research Contributions . 145 7.1.1 Parallelization of Monte Carlo Tree Search and Information Set Monte Carlo Tree Search . 145 7.1.2 Modification of Monte Carlo Tree Search using Heuristic Hard Prun- ing .................................. 146 7.1.3 Modification of Monte Carlo Tree Search play style using Action Selection Mechanisms . 147 7.1.4 Application of Data Mining Techniques to prediction of opponent decks in card games . 149 7.2 Summary and General Observations . 150 7.3 Future Work . 151 7.3.1 Parallelization of MCTS . 151 7.3.2 Heuristic Pruning . 152 7.3.3 Action Selection Mechanisms . 153 7.3.4 Rule Association Mining . 153 7 CONTENTS Appendices 155 A Experimental Results 156 A.1 Chapter 4 . ..