MQP CDR#GXS 1501 UCT-Enhanced Deep Convolutional Neural Network for Move Recommendation in Go a Major Qualifying Project Report submitted to the faculty of the WORCESTER POLYTECHNIC INSTITUTE in partial fulfillment of the requirements for th e Degree of Bachelor of Science by Sarun Paisarnsrisomsuk Pitchaya Wiratchotisatian April 30, 2015 Professor Gábor N. Sárközy, Major Advisor Abstract Deep convolutional neural networks (CNN) have been proved to be useful in predicting moves of human Go experts [10], [12]. Combining Upper Confidence bounds applied to Trees (UCT) with a large deep CNN creates an even more powerful artificial intelligence method in playing Go [12]. Our project introduced a new feature, board patterns at the end of a game, used as inputs to the network. We implemented a small, single-thread, 1-hidden-layer deep CNN without using GPU, and trained it by supervised learning from a database of human professional games. By adding the new feature, our best model was able to correctly predict the experts’ move in 18% of the positions, compared to 6% without this feature. In practice, even though the board pattern at the end of the game is invisible to Go players until the very end of the game, collecting these statistics during each simulation in UCT can approximate the likelihood of which parts of the board belong to the players. We created another dataset by using UCT to simulate a number of playouts and calculating the average influence value of Black and White positions in those boards at the end of the simulations. We used this dataset as the input to the deep CNN and found that the network was able to correctly predict the experts’ move in 9% of positions. i Acknowledgements We would like to express our gratitude to the people whose assistance, hospitality, and advice helped the completion of this project: Levente Kocsis, Project Advisor and SZTAKI liaison Gabor Sarkozy, MQP Advisor Worcester Polytechnic Institute MTA-SZTAKI SZTAKI Colleagues Pachi and Gnu Go Development Team Ryan Harris on his YouTube videos about the back-propagation algorithm i Table of Contents Abstract ........................................................................................................................................... i Acknowledgements ........................................................................................................................ i Table of Contents .......................................................................................................................... ii Table of Figures............................................................................................................................ iv 1 Introduction ................................................................................................................................ 1 2 Background ................................................................................................................................ 4 2.1 Game of Go .......................................................................................................................... 4 2.1.1 Rules of Go .................................................................................................................... 4 2.1.2 Go Rank System............................................................................................................ 6 2.1.3 Go: An Ultimate Challenge for Artificial intelligence ............................................... 6 2.2 Computer Go Techniques ................................................................................................... 8 2.2.1 Monte Carlo Tree Search (MCTS) ............................................................................. 8 2.2.2 Upper Confidence Bounds Applied to Trees (UCB applied to trees / UCT) ......... 10 2.2.3 Deep Convolutional Neural Networks (Deep CNN) ................................................ 13 3 Deep Convolutional Neural Networks ................................................................................... 16 3.1 Artificial Neural Network (ANN) .................................................................................... 16 3.2 Deep Convolutional Neural Network (Deep CNN) ........................................................ 19 3.3 Back-propagation Algorithm ........................................................................................... 22 ii 4 Methodology ............................................................................................................................. 26 4.1 Existing Codebases ............................................................................................................ 26 4.1.1 Gnu Go......................................................................................................................... 26 4.1.2 Pachi ............................................................................................................................. 27 4.1.3 GTP (Go Text Protocol) ............................................................................................. 27 4.2.1 Go Data ........................................................................................................................ 28 4.2.2 Deep Convolutional Neural Network ........................................................................ 29 4.2.2.1 Weight Symmetries .................................................................................................. 31 4.2.2.2 Activation Functions ................................................................................................ 31 4.2.2.3 Mini-Batches ............................................................................................................ 32 4.2.3 UCT-simulated Final Board Pattern ........................................................................ 33 5 Experiments and Results ......................................................................................................... 34 6 Conclusion and Future Work ................................................................................................. 39 References .................................................................................................................................... 42 Appendix ...................................................................................................................................... 45 A. Smart Game Format (SGF) ............................................................................................... 45 B. Parameters used in our best DNN ..................................................................................... 48 C. Translational invariance .................................................................................................... 48 iii Table of Figures Figure 1. The standard 19x19 board after several moves ............................................................... 5 Figure 2. Performance ranks in Go, in increasing order of strength from left to right ................... 6 Figure 3. One iteration of a general MCTS approach ..................................................................... 9 Figure 4. Structure of a neural network in the human brain ......................................................... 16 Figure 5. Architecture of an artificial neural network .................................................................. 17 Figure 6. Convolutional Neural Network in Image Recognition .................................................. 20 Figure 7. Features used as inputs to the CNN in the work of Maddison et al. ............................. 21 Figure 8. A CNN with one hidden layer, and three convolutional kernels ................................... 21 Figure 9. The features we used in our deep network .................................................................... 28 Figure 10. 8-fold weight symmetry group .................................................................................... 31 Figure 11. The accuracy comparison over different numbers of trainings (move positions) for different types of experiments ...................................................................................................... 36 Figure 12. The accuracy of ranking top k move comparison over different types of experiments ....................................................................................................................................................... 37 Figure 13. Coordinate system for points and moves in SGF files ................................................ 46 iv 1 Introduction There have been countless developments of artificial intelligence that improved the ability to play games with computers. Computers are now expert players in many popular, two-player games, including Backgammon, Checkers, Chess, and Othello. Still the game of Go, an ancient board game, continues to elude the efforts of computer science researchers. Go is remarkable because of its simple rules but complex strategies. Because the size of the search space in the game is enormous and the game has the long-term influence of moves, creating a move prediction function in Go is a challenging task for computer science researchers. In the past, researchers had failed to combine brute force tree-search with human knowledge or reinforcement learning [5]. The efforts to equalize the performance of computer Go to the performance of other two-player games were in vain, until 2006 when the Monte-Carlo tree search (MCTS) was introduced to the game of Go. The performance