Monte-Carlo Tree Search for Multi-Player Games Monte-Carlo Tree Search for Multi-Player Games PROEFSCHRIFT ter verkrijging van de graad van doctor aan de Universiteit Maastricht, op gezag van de Rector Magnificus, Prof. dr. L.L.G. Soete, volgens het besluit van het College van Decanen, in het openbaar te verdedigen op maandag 2 december 2013 om 14.00 uur door Joseph Antonius Maria Nijssen Promotor: Prof. dr. ir. R.L.M. Peeters Copromotor: Dr. M.H.M. Winands Leden van de beoordelingscommissie: Prof. dr. ir. J.C. Scholtes (voorzitter) Prof. dr. P.I. Cowling (University of York) Prof. dr. M. Gyssens (transnationale Universiteit Limburg) Prof. dr. J.A. La Poutre´ (Technische Universiteit Delft) Prof. dr. A.J. Vermeulen This research has been funded by the transnational University Limburg (tUL). Dissertation Series No. 2013-40 The research reported in this thesis has been carried out under the auspices of SIKS, the Dutch Research School for Information and Knowledge Systems. Printed by Optima Grafische Communicatie, Rotterdam, The Netherlands ISBN 978-94-6169-452-2 ©2013 J.A.M. Nijssen, Maastricht, The Netherlands. All rights reserved. No part of this publication may be reproduced, stored in a retrieval sys- tem, or transmitted, in any form or by any means, electronically, mechanically, photocopying, recording or otherwise, without prior permission of the author. Preface Two topics that interested me the most since I was a child were computers and board games. When I found out, during my bachelor’s study Knowledge Engineering at the Maastricht University, that there exists a research area that combines these two top- ics, I immediately knew in which direction I would want to do my bachelor’s thesis. Letting computers play board games as strongly as possible, developing or investigat- ing new techniques or heuristics along the way, certainly caught my attention. After achieving my bachelor’s degree, I chose to continue with the master Artificial Intelli- gence. For my master’s thesis, I decided to stay in the same research direction. After applying Monte-Carlo based techniques to the game of Othello, I decided to investi- gate search techniques for the two-player chess-like board game Khet. While working on my master’s thesis, I got the idea to do Ph.D. research in the area of games and AI. Luckily, a new Ph.D. position in the Games and AI group opened a few months after I finished my master’s thesis. This Ph.D. thesis presents the research that I have performed in the past four years at the Department of Knowledge Engineering at the Maastricht University. Performing research and writing a thesis, however, is not a solitary effort. Therefore, there are various people I would like to thank. First of all I would like to thank my daily supervisor, Mark Winands. He taught me a lot about doing research and writing articles. Thanks to his support, this thesis was made possible. The second person I would like to thank is Jos Uiterwijk. He supervised my bachelor’s and my master’s theses, and it was thanks to his efforts that I was able to get my position as a Ph.D. student. Next, I would like to thank Ralf Peeters, for agreeing to be my promotor, and for looking at my thesis from a different point of view. Of course, my thanks also go to the people with whom I have shared an office dur- ing my time as a Ph.D. student. Maarten Schadd, Jahn Saito, David Lupien St-Pierre, Hendrik Baier, and Mandy Tak, thank you for the interesting discussions we had and for making the ‘games office’ a great place to work. In particular, I want to thank Marc Lanctot for proofreading my thesis, and raising some interesting points that certainly improved its quality. I would also like to thank all my other colleagues at the Depart- ment of Knowledge Engineering. I want to thank Guillaume Chaslot, Steven de Jong, Nyree Lemmens, Marc Ponsen, Philippe Uyttendaele, Frederik Schadd, Daan Bloem- bergen, Michael Kaisers, Pietro Bonizzi, Nela Lekic, and many others, with whom I had the pleasure to have lunch breaks, including both the interesting and slightly less interesting discussions. vi And last but not least, my thanks go to my parents Harry and Josine, and my sisters Iris and Yonne. Without their love and support, all of this would never have been possible. Bedaank veur alles. Pim Nijssen, 2013 Acknowledgments The research has been carried out under the auspices of SIKS, the Dutch Research School for Information and Knowledge Systems. This research has been funded by the transnational University Limburg (tUL). Table of Contents Preface v Table of Contents vii List of Figures xi List of Tables xii List of Algorithms xiv 1 Introduction 1 1.1 Games and AI . 1 1.2 Game Properties . 3 1.3 Multi-Player Games . 4 1.3.1 Perfect-Information Multi-Player Games . 4 1.3.2 Multi-Player Hide-and-Seek Games . 5 1.4 Search Techniques . 6 1.5 Problem Statement and Research Questions . 7 1.6 Thesis Overview . 9 2 Search Techniques 11 2.1 Tree Search . 11 2.1.1 The Game Tree . 12 2.1.2 The Search Tree . 12 2.2 Searching in Two-Player Games . 12 2.2.1 Minimax Search . 13 2.2.2 ®¯ Search . 14 2.2.3 Expectimax Search . 16 2.3 Searching in Multi-Player Games . 18 2.3.1 Maxn Search . 19 2.3.2 Paranoid Search . 21 2.3.3 Best-Reply Search . 22 2.4 Enhancements for Minimax-Based Techniques . 24 2.4.1 Dynamic Move Ordering . 24 2.4.2 Transposition Tables . 25 viii Table of Contents 2.4.3 Iterative Deepening . 26 2.5 Monte-Carlo Methods . 26 2.5.1 Monte-Carlo Evaluation . 26 2.5.2 Multi-Armed Bandit Problem . 27 2.6 Monte-Carlo Tree Search . 28 2.6.1 Basic MCTS Algorithm . 28 2.6.2 MCTS for Multi-Player Games . 30 2.7 Related Work in MCTS . 32 3 Test Environment 33 3.1 Chinese Checkers . 34 3.1.1 Rules . 34 3.1.2 Complexity . 35 3.1.3 Domain Knowledge . 36 3.2 Focus . 36 3.2.1 Rules . 36 3.2.2 Complexity . 37 3.2.3 Domain Knowledge . 38 3.3 Rolit . 39 3.3.1 Rules . 39 3.3.2 Complexity . 40 3.3.3 Domain Knowledge . 40 3.4 Blokus . 41 3.4.1 Rules . 41 3.4.2 Complexity . 42 3.4.3 Domain Knowledge . 43 3.5 Comparison of Complexities to Other Games . 43 3.6 Game Engine: MAGE ............................... 44 4 Search Policies for Multi-Player MCTS 47 4.1 Search Policies: Related Work . 48 4.2 Alternative Search Policies . 49 4.2.1 MCTS-Paranoid . 49 4.2.2 MCTS-BRS . 51 4.3 Experimental Results for Search Policies . 53 4.3.1 Experimental Setup . 53 4.3.2 Comparison of Minimax-Based Techniques . 53 4.3.3 Comparison of MCTS-Based Techniques . 55 4.3.4 MCTS-Based Techniques versus BRS . 58 4.3.5 Minimax-Based Techniques versus MCTS-maxn . 60 4.4 Background of MCTS-Solver . 62 4.5 Multi-Player MCTS-Solver . 62 4.6 Experimental Results for Multi-Player MCTS-Solver . 64 4.7 Chapter Conclusions and Future Research . 65 Table of Contents ix 5 Progressive History for MCTS 67 5.1 Related Work . 68 5.1.1 Rapid Action-Value Estimation . 68 5.1.2 Progressive Bias . 70 5.2 Progressive History . 70 5.3 Experiments and Results . 72 5.3.1 Experimental Setup . 72 5.3.2 Parameter Tuning and General Strength . 72 5.3.3 Dividing by the Number of Losses . 74 5.3.4 Tree-Only Progressive History . 75 5.3.5 Application of AMAF Values . 76 5.4 Progressive History in Havannah . 76 5.5 Chapter Conclusions and Future Research . 78 6 Search-based Playouts for Multi-Player MCTS 79 6.1 Related Work . 80 6.2 Search-Based Playouts in Multi-Player MCTS . 81 6.2.1 Two-Ply Maxn ............................... 81 6.2.2 Two-Ply Paranoid . 82 6.2.3 Two-Ply BRS . 82 6.2.4 Search Enhancements . 83 6.3 Experiments and Results . 84 6.3.1 Experimental Setup . 84 6.3.2 Fixed Number of Playouts per Move . 85 6.3.3 Overhead . 87 6.3.4 Fixed Amount of Time per Move . 88 6.4 Chapter Conclusions and Future Research . 93 7 MCTS for a Hide-and-Seek Game 95 7.1 Scotland Yard . 96 7.1.1 Background . 96 7.1.2 Rules . ..