Better Computer Go Player with Neural Network and Long-term Prediction Yuandong Tian1 Yan Zhu1,2 1Facebook AI Research 2Rutgers University hp://yuandong-an.com Arxiv Link: hp://arxiv.org/abs/1511.06410

KGS: Online amateur games Introducon Rules Experiments GoGoD: Professional Games Go, originated in ancient China more than 2,500 years ago, is a two-player zero-sum Dataset: 170k KGS / 80k GoGoD Go Rating system board game with full information. The Name Description KGS rank possible board situations of Go are much Darkforest Trained on KGS 1d more than the atoms of universe, rendering Darkfores1 Trained on GoGoD with nstep=3 2d any brute-force search intractable. Darkfores2 Trained on GoGoD with nstep=3 and fine tuning 3d 4-connecetd group Black and white take The player with 30k 1k 1d 7d 1p 9p turns to place stones dies if surrounded by more territory wins Darkfores3 Trained on KGS with nstep=3 and fine tuning 3d on a 19*19 board. enemy Method Test Top-1 performance Rather than brute-force search, human plays Go with both intuitions and reasoning: first think about a few possible KGS GoGoD alternatives, and then find the best move by careful analysis. With the advancement of Deep Learning, it is now Training with 3 steps possible to model human’s intuition more precisely, yielding a better Computer Go player. Darkforest 53.9% 48.6% Darkfores1 54.2% 51.8% shows better Darkfores2 55.2% 53.3% performance than Conv layer Conv layers x 10 Conv layers Long-term Prediction training with 1-step Darkfores3 57.6% 52.0% 92 channels 384 channels k maps Policy Network Current Board 25 feature planes k parallel softmax 5 x 5 kernels 3 x 3 kernels 3 x 3 kernel Feature Name Win rate for Pure DCNN

Our/enemy liberties (6) Our next move (next-1) GnuGo Pachi 10k Pachi 100k Fuego 10k Fuego 100k Ko location (1) Clark & Storkey 91.0 - - 14.0 Our/enemy/empty/(3) (ICML 2015) Opponent move (next-2) Maddison et al. Our/enemy history (2) 97.2 47.4 11.0 23.3 12.5 Enemy rank (9) (ICLR 2015) Our counter move (next-3) Darkforest 98.0 ± 1.0 71.5 ± 2.1 27.3 ± 3.0 84.5 ± 1.5 56.7 ± 2.5 • Relative coding: our/enemy x 10 • Easy-to-compute features Darkfores1 99.7 ± 0.3 88.7 ± 2.1 59.0 ± 3.3 93.2 ± 1.5 78.0 ± 1.7 Darkfores2 100 ± 0.0 94.3 ± 1.7 72.6 ± 1.9 98.5 ± 0.1 89.7 ± 2.1 Monte Carlo Tree Search (MCTS) Original Design: • Top 3-5 moves are picked from DCNN AlphaGo* (RL) - - 85 - - • No prior once the moves are selected. (a) 22/40 #blackwin/#game (b) 22/40 (c) 23/41 • Add noise to win rate estimate to diverge Win rate for DCNN + MCTS 2/10 21/31 threads. 2/10 2/10 20/30 Vs Pachi 10k DF + MCTS DF1 + MCTS DF2 + MCTS 20/30 1/1 1/1 1/1 Vs Pure DCNN DF + MCTS DF1 + MCTS DF2 + MCTS Pure DCNN 71.5 88.7 94.3 2/10 Current Design: 11/19 1000 rollout (top5) 89.6 76.4 68.4 1000 rollout (top5) 88.4 94.4 97.6 2/10 10/12 10/18 2/10 10/12 10/18 10/12 • Top 7 moves from DCNN. • Use PUCT and virtual loss. Remove 1000 rollout (top3) 91.6 89.6 79.2* 1000 rollout (top3) 95.2 98.4 99.2 9/10 1/8 10/11 1/8 1/8 9/10 win rate noise. 5000 rollout (top5) 96.8 94.3 82.3 5000 rollout (top5) 98.4 99.6 100 • 5% of threads skip DCNN evaluation Tree policy 1/1 Synced 1/1 and run the default policy. Default policy *79.2: With PUCT and virtual loss, this win rate becomes 94.2%. Other win rates also increase. DCNN server

Default Policy DarkForest AlphaGo* Compeons • Local 3x3 pattern matching with Zobrist hashing Dataset Tygem Tygem • Incremental board status update with heap structure Top-1 Accuracy 30.1% 24.2% Stable KGS 5d (kgs id: darkfmcts3) • Handle special but critical cases with rules (nakade. etc) Speed 6-7 microsecond 2 microsecond 3rd in KGS January Go Tournament MSE after move 280 26% ~17% 2nd in 9th UEC Cup for Computer Go

0.41 Pachi 0.39

Games Simple 0.37 Default policy DarkForest Our Go engine will be open-sourced! nd th 0.35 in DarkForest: ~26% 2 place in 9 UEC Cup GoGoD 0.33 See: hps://github.com/facebookresearch/darkforest 0.31 0.29 • Standalone project with little dependency. 0.27 • 0.25 Efficient Go/MCTS libraries written in C/Lua. MSE on 10000 100 120 140 160 180 200 220 240 260 280 • Runnable on a single machine with 1-4 GPUs. Move number From Fig. 2(b) in AlphaGo [Silver et al, Nature 2016] • Much stronger than existing open source engines.

th AlphaGo*: all performances are based on [Silver et al., Nature 2016]. The performance of current AlphaGo might be higher. 4 Denseisen vs. Kobayashi (9p)