CSC 311: Introduction to Machine Learning Lecture 12 - AlphaGo and game-playing
Roger Grosse Chris Maddison Juhan Bae Silviu Pitis
University of Toronto, Fall 2020
Intro ML (UofT) CSC311-Lec12 1/59 Recap of different learning settings
So far the settings that you’ve seen imagine one learner or agent.
Supervised Unsupervised Reinforcement
Learner predicts Learner organizes Agent maximizes labels. data. reward.
Intro ML (UofT) CSC311-Lec12 2/59 Today
We will talk about learning in the context of a two-player game.
Game-playing
This lecture only touches a small part of the large and beautiful literature on game theory, multi-agent reinforcement learning, etc.
Intro ML (UofT) CSC311-Lec12 3/59 Game-playing in AI: Beginnings
(1950) Claude Shannon proposes explains how games could • be solved algorithmically via tree search (1953) Alan Turing writes a chess program • (1956) Arthur Samuel writes a program that plays checkers • better than he does (1968) An algorithm defeats human novices at Go •
slide credit: Profs. Roger Grosse and Jimmy Ba
Intro ML (UofT) CSC311-Lec12 4/59 Game-playing in AI: Successes
(1992) TD-Gammon plays backgammon competitively with • the best human players (1996) Chinook wins the US National Checkers Championship • (1997) DeepBlue defeats world chess champion Garry • Kasparov (2016) AlphaGo defeats Go champion Lee Sedol. •
slide credit: Profs. Roger Grosse and Jimmy Ba
Intro ML (UofT) CSC311-Lec12 5/59 Today
Game-playing has always been at the core of CS. • Simple well-defined rules, but mastery requires a high degree • of intelligence. We will study how to learn to play Go. • The ideas in this lecture apply to all zero-sum games with • finitely many states, two players, and no uncertainty. Go was the last classical board game for which humans • outperformed computers. We will follow the story of AlphaGo, DeepMind’s Go playing • system that defeated the human Go champion Lee Sedol. Combines many ideas that you’ve already seen. • supervised learning, value function learning... •
Intro ML (UofT) CSC311-Lec12 6/59 The game of Go: Start
Initial position is an empty 19 19 grid. • ×
Intro ML (UofT) CSC311-Lec12 7 / 59 The game of Go: Play
2 players alternate placing stones on • empty intersections. Black stone plays first. (Ko) Players cannot recreate a former • board position.
Intro ML (UofT) CSC311-Lec12 8 / 59 The game of Go: Play
Capture (Capture) Capture and remove a • connected group of stones by surrounding them.
Intro ML (UofT) CSC311-Lec12 9/59 The game of Go: End
Territory (Territory) The winning player has • the maximum number of occupied or surrounded intersections.
Intro ML (UofT) CSC311-Lec12 10 / 59 Outline of the lecture
To build a strong computer Go player, we will answer: What does it mean to play optimally? • Can we compute (approximately) optimal play? • Can we learn to play (somewhat) optimally? •
Intro ML (UofT) CSC311-Lec12 11 / 59 Why is this a challenge?
Optimal play requires searching over 10170 legal positions. • ∼ It is hard to decide who is winning before the end-game. • Good heuristics exist for chess (count pieces), but not for Go. • Humans use sophisticated pattern recognition. •
Intro ML (UofT) CSC311-Lec12 12 / 59 Optimal play
Intro ML (UofT) CSC311-Lec12 13 / 59 Game trees
Organize all possible games into a • tree. Each node s contains a legal position. • Child nodes enumerate all possible • actions taken by the current player. Leaves are terminal states. • Technically board positions can • appear in more than one node, but let’s ignore that detail for now. The Go tree is finite (Ko rule). •
Intro ML (UofT) CSC311-Lec12 14 / 59 Game trees
black stone’s turn
white stone’s turn
Intro ML (UofT) CSC311-Lec12 15 / 59 Evaluating positions
We want to quantify the utility of a • node for the current player. Label each node s with a value v(s), • taking the perspective of the black stone player. +1 for black wins, -1 for black loses. • Flip the sign for white’s value • (technically, this is because Go is zero-sum). Evaluations let us determine who is • winning or losing.
Intro ML (UofT) CSC311-Lec12 16 / 59 Evaluating leaf positions Leaf nodes are easy to label, because a winner is known.
s
-1 +1 +1 +1 -1 -1 +1 +1 +1 -1 -1 -1 -1 -1 +1 -1 black stones win white stones win
Intro ML (UofT) CSC311-Lec12 17 / 59 Evaluating internal positions
The value of internal nodes depends on • the strategies of the two players. s
regardless of white’s strategy. a
Intro ML (UofT) CSC311-Lec12 18 / 59 Evaluating positions under optimal play
s
min
-1 +1 +1 +1 -1 -1 +1 +1 +1 -1 -1 -1 -1 -1 +1 -1
Intro ML (UofT) CSC311-Lec12 19 / 59 Evaluating positions under optimal play
s
max
min
-1 +1 +1 +1 -1 -1 +1 +1 +1 -1 -1 -1 -1 -1 +1 -1
Intro ML (UofT) CSC311-Lec12 20 / 59 Evaluating positions under optimal play
max
min
max
min
-1 +1 +1 +1 -1 -1 +1 +1 +1 -1 -1 -1 -1 -1 +1 -1
Intro ML (UofT) CSC311-Lec12 21 / 59 Value function v ∗
v∗ satisfies the fixed-point equation • v∗ maxa (child(s, a)) black plays { ∗ } mina v (child(s, a)) white plays v∗(s) = { } +1 black wins 1 white wins − Analog of the optimal value function of RL. • Applies to other two-player games • Deterministic, zero-sum, perfect information games. •
Intro ML (UofT) CSC311-Lec12 22 / 59 Quiz!
1 -1?
2 +1? +1 Recall: black plays first and is trying to maximize, -1 +1 whereas white is trying to minimize.
Intro ML (UofT) CSC311-Lec12 23 / 59 Quiz!
max
min
Intro ML (UofT) CSC311-Lec12 24 / 59 In a perfect world
So, for games like Go, all you need is v∗ to play optimally in • the worst case:
a∗ = arg max v∗(child(s, a)) a { } Claude Shannon (1950) pointed out that you can find a∗ by • recursing over the whole game tree. Seems easy, but v∗ is wildly expensive to compute... • Go has 10170 legal positions in the tree. • ∼
Intro ML (UofT) CSC311-Lec12 25 / 59 Approximating optimal play
Intro ML (UofT) CSC311-Lec12 26 / 59 Depth-limited Minimax
In practice, recurse to a • small depth and back off to a static evaluation vˆ∗. vˆ is a heuristic, designed • ∗
by experts.
For Go (M¨uller,2002).
Intro ML (UofT) CSC311-Lec12 27 / 59 Progress in Computer Go
9 dan 8 dan 7 dan 6 dan 5 dan Minimax search for Go 4 dan 3 dan
Professional Dan 2 dan 1 dan 9 dan 8 dan 7 dan 6 dan 5 dan 4 dan 3 dan Amateur Dan 2 dan 1 dan 1 kyu 2 kyu 3 kyu 4 kyu 5 kyu GnuGo 6 kyu Go++ Many Faces of Go Amateur Kyu 7 kyu Many Faces of Go 8 kyu 2000 2005 2010 2015
adapted from Sylvain Gelly & David Silver, Test of Time Award ICML 2017 Intro ML (UofT) CSC311-Lec12 28 / 59 Expected value functions
Designing static evaluation of v∗ is very challenging, • especially so for Go. Somewhat obvious, otherwise search would not be needed! • Depth-limited minimax is very sensitive to misevaluation. • Monte Carlo tree search resolves many of the issues with • Minimax search for Go. Revolutionized computer Go. • To understand this, we will introduce expected value • functions.
Intro ML (UofT) CSC311-Lec12 29 / 59 Expected value functions
If players play by rolling fair dice, outcomes will be random.
-1 +1 -1 -1 -1 -1
This is a decent approximation to very weak play.
Intro ML (UofT) CSC311-Lec12 30 / 59 Expected value functions
Averaging many random outcomes expected value function. → v(s)= 1/9
-1 +1 +1 +1 -1 -1 +1 +1 +1 -1 -1 -1 -1 -1 +1 -1
Contribution of each outcome depends on the length of the path.
Intro ML (UofT) CSC311-Lec12 31 / 59 Quiz!
1 1/3?
-1 +1 2 1/2?
Intro ML (UofT) CSC311-Lec12 32 / 59 Quiz!
-1 +1 2 1/2?
Intro ML (UofT) CSC311-Lec12 33 / 59 Expected value functions
Noisy evaluations v are cheap
n 1 X 0 vn(s) = n o(si) i=1 0 E[o(s ) := v(s)] ≈ o(s) = 1 if black wins / loses. ± Longer games will be • underweighted by this evaluation o(s0 ) o(s0 ) o(s0 )
Intro ML (UofT) CSC311-Lec12 34 / 59 Monte Carlo tree search
Ok expected value functions are easy to approximate, • but how can we use vn to play Go? v is not at all similar to v . • n ∗ So, maximizing v by itself is probably not a great strategy. • n Minimax won’t work, because it is a pure exploitation strategy • that assumes perfect leaf evaluations. Monte Carlo tree search (MCTS; Kocsis and Szepesv´ari, • 2006; Coulom, 2006; Browne et al., 2012) is one way. MCTS maintains a depth-limited search tree. • Builds an approximation vˆ of v at all nodes. • ∗ ∗
Intro ML (UofT) CSC311-Lec12 35 / 59 MonteIEEE TRANSACTIONS ON Carlo COMPUTATIONAL INTELLIGENCEtree AND search AI IN GAMES, VOL. 4, NO. 1, MARCH 2012 6 Selection Expansion Simulation Backpropagation
Tree Default Policy Policy
Fig. 2. One iteration of the general MCTS approach. (Browne et al., 2012)
6 AlgorithmSelect 1 Generalan MCTS existing approach. leaf or expandrithm 1. aHere newv0 is the leaf. root node corresponding to state function• MCTSSEARCH(s0) s0, vl is the last node reached during the tree policy createEvaluate root node v0 leafwith state withs0 Monte Carlostagesimulation and corresponds tov staten. sl, and is the reward •while within computational budget do for the terminal state reached by running the default v TREEPOLICY(v ) policy from state sl. The result of the overall search Noisyl values0 v are backed-up the tree to improve DEFAULTPOLICY(s(vn)) a(BESTCHILD(v0)) is the action a that leads to the best • l BACKUP(vl, ) ∗ child of the root node v0, where the exact definition of “best” is defined by the implementation. returnapproximationa(BESTCHILD(v0)) vˆ . Note that alternative interpretations of the term “sim- Intro ML (UofT) CSC311-Lec12ulation” exist in the literature. Some authors take it 36 / 59 to mean the complete sequence of actions chosen per the tree until the most urgent expandable node is iteration during both the tree and default policies (see for reached. A node is expandable if it represents a non- example [93], [204], [94]) while most take it to mean the terminal state and has unvisited (i.e. unexpanded) sequence of actions chosen using the default policy only. children. In this paper we shall understand the terms playout and 2) Expansion: One (or more) child nodes are added to simulation to mean “playing out the task to completion expand the tree, according to the available actions. according to the default policy”, i.e. the sequence of 3) Simulation: A simulation is run from the new node(s) actions chosen after the tree policy steps of selection and according to the default policy to produce an out- expansion have been completed. come. Figure 2 shows one iteration of the basic MCTS al- 7 4) Backpropagation: The simulation result is “backed gorithm. Starting at the root node t0, child nodes are up” (i.e. backpropagated) through the selected recursively selected according to some utility function nodes to update their statistics. until a node tn is reached that either describes a terminal These may be grouped into two distinct policies: state or is not fully expanded (note that this is not necessarily a leaf node of the tree). An unvisited action 1) Tree Policy: Select or create a leaf node from the a from this state s is selected and a new leaf node tl is nodes already contained within the search tree (se- added to the tree, which describes the state s0 reached lection and expansion). from applying action a to state s. This completes the tree 2) Default Policy: Play out the domain from a given policy component for this iteration. non-terminal state to produce a value estimate (sim- A simulation is then run from the newly expanded ulation). leaf node tl to produce a reward value , which is then The backpropagation step does not use a policy itself, but updates node statistics that inform future tree policy 6. The simulation and expansion steps are often described and/or decisions. implemented in the reverse order in practice [52], [67]. 7. Each node contains statistics describing at least a reward value These steps are summarised in pseudocode in Algo- and number of visits. Monte Carlo tree search
Selection strategy greedily descends tree. • MCTS is robust to noisy misevaluation at the leaves, because • the selection rule balances exploration and exploitation:
( s ) 2 log N(s) a∗ = arg max vˆ∗(child(s, a)) + a N(child(s, a))
vˆ (s) = estimate of v (s), N(s) number of visits to node s. • ∗ ∗ MCTS is forced to visit rarely visited children. • Key result: MCTS approximation vˆ∗ v∗ (Kocsis and • → Szepesv´ari,2006).
Intro ML (UofT) CSC311-Lec12 37 / 59 Progress in Computer Go
9 dan 8 dan 7 dan 6 dan 5 dan Monte Carlo tree search for Go 4 dan 3 dan
Professional Dan 2 dan 1 dan 9 dan 8 dan 7 dan 6 dan Zen 5 dan Zen Zen, CrazyStone 4 dan Zen 3 dan Amateur Dan 2 dan MoGo Zen 1 dan Zen 1 kyu CrazyStone CrazyStone 2 kyu MoGo 3 kyu MoGo 4 kyu CrazyStone 5 kyu Traditional Search 6 kyu
Amateur Kyu Indigo 7 kyu Indigo 8 kyu 2000 2005 2010 2015
adapted from Sylvain Gelly & David Silver, Test of Time Award ICML 2017 Intro ML (UofT) CSC311-Lec12 38 / 59 Scaling with compute and time
The strength of MCTS bots scales with the amount of • compute and time that we have at play-time. But play-time is limited, while time outside of play is much • more plentiful. How can we improve computer Go players using compute • when we are not playing? Learning! You can try to think harder during a test vs. studying more • beforehand.
Intro ML (UofT) CSC311-Lec12 39 / 59 Learning to play Go
Intro ML (UofT) CSC311-Lec12 40 / 59 This is where I come in
2014 Google DeepMind internship on neural nets for Go. • Working with Aja Huang, David Silver, Ilya Sutskever, I was • responsible for designing and training the neural networks. Others came before (e.g., Sutskever and Nair, 2008). • Ilya Sutskever’s (Chief Scientist, OpenAI) argument in 2014: • expert players can identify a good set of moves in 500 ms. This is only enough time for the visual cortex to process the • board—not enough for complex reasoning. At the time we had neural networks that were nearly as good • as humans in image recognition, thus we thought we would be able to train a net to play Go well. Key goal: can we train a net to understand Go? •
Intro ML (UofT) CSC311-Lec12 41 / 59 Neural nets for Go
Neural networks are powerful parametric function approximators.
Idea: map board position s (input) to a next move or an evaluation (output) using simple convolutional networks.
Intro ML (UofT) CSC311-Lec12 42 / 59 Neural nets for Go
We want to train a neural policy or • neural evaluator, but how? An expert move (pink) Existing data: databases of Go • games played by humans and other compute Go bots. The first idea that worked was • learning to predict expert’s next move. Input: board position s • Output: next move a •
Intro ML (UofT) CSC311-Lec12 43 / 59 Policy Net (Maddison et al., 2015)
Dataset: KGS server games split ⇡ (a s, x) •
Use trained net as a Go player: •
s
Intro ML (UofT) CSC311-Lec12 44 / 59 Like learning a better traversal
+1 As supervised accuracy improved, searchless play improved.
Intro ML (UofT) CSC311-Lec12 45 / 59 Progress in Computer Go
9 dan 8 dan 7 dan 6 dan 5 dan Progress in my internship 4 dan 3 dan
Professional Dan 2 dan 1 dan 9 dan 8 dan 7 dan 6 dan MCTS 5 dan 4 dan 3 dan Amateur Dan 2 dan RL net 12 layer 1 dan SL net 12 layer 1 kyu SL net 10 layer 2 kyu SL net 6 layer 3 kyu SL net 3 layer 4 kyu 5 kyu Traditional Search 6 kyu
Amateur Kyu 7 kyu SL net 3 layer 8 kyu 2000 2005 2010 2015
adapted from Sylvain Gelly & David Silver, Test of Time Award ICML 2017 Intro ML (UofT) CSC311-Lec12 46 / 59 Can we improve MCTS with neural networks?
These results prompted the • formation of big team inside DeepMind to combine MCTS and neural networks. To really improve search, we • needed strong evaluators.
Recall: an evaluation function
v (s, x)
(si, o(si)) Loss: squared error, • N X 2 (o(si) vnet(si, x)) . − i=1
Problem: Effective sample size of •
160,000 games was not enough. s
(Silver et al., 2016) Intro ML (UofT) CSC311-Lec12 48 / 59 Value Net (Silver et al., 2016)
v (s, x)
(Silver et al., 2016) Intro ML (UofT) CSC311-Lec12 49 / 59 ARTICLE RESEARCH a b Rollout policy SL policy network RL policy networkValue network Policy network Value network Neural network p p p p (a⎪s) (s′)
Policy gradient
Classi!catio Self Play n
essio n Classi!cation Regr Data
s s′ Human expert positions Self-play positions Figure 1 | Neural network training pipeline and architecture. a, A fast the current player wins) in positions from the self-play data set. rollout policy pπ and supervised learning (SL) policy network pσ are b, Schematic representation of the neural network architecture used in trained to predict human expert moves in a data set of positions. AlphaGo. The policy network takes a representation of the board position A reinforcement learning (RL) policy network pρ is initialized to the SL s as its input, passes it through many convolutional layers with parameters policy network, and is then improved by policy gradient learning to σ (SL policy network) or ρ (RL policy network), and outputs a probability maximize the outcome (that is, winning more games) against previous distribution paσ(|s) or paρ (|s) over legal moves a, represented by a versions of the policy network. A new data set is generated by playing probability map over the board. The value network similarly uses many games of self-play with the RL policy network. Finally, a value network vθ convolutional layers with parameters θ, but outputs a scalar value vθ(s′) is trained by regression to predict the expected outcome (that is, whether that predicts the expected outcome in position s′. sampled state-action pairs (s, a), using stochastic gradient ascent to and its weights ρ are initialized to the same values, ρ = σ. We play maximize the likelihood of the human move a selected in state s games between the current policy network pρ and a randomly selected previous iteration of the policy network. Randomizing from a pool ∂log pa(|)s ∆σ ∝ σ of opponents in this way stabilizes training by preventing overfitting ∂σ to the current policy. We use a reward function r(s) that is zero for all non-terminal time steps t < T. The outcome zt = ± r(sT) is the termi- We trained a 13-layer policy network, which we call the SL policy nal reward at the end of the game from the perspective of the current network, from 30 million positions from the KGS Go Server. The net- player at time step t: +1 for winning and −1 for losing. Weights are work predicted expert moves on a held out test set with an accuracy of then updated at each time step t by stochastic gradient ascent in the 57.0% using all input features, and 55.7% using only raw board posi- direction that maximizes expected outcome25 tion and move history as inputs, compared to the state-of-the-art from other research groups of 44.4% at date of submission24 (full results in ∂log paρ(tt|)s Extended Data Table 3). Small improvements in accuracy led to large ∆ρ ∝ zt improvements in playing strength (Fig. 2a); larger networks achieve ∂ρ better accuracy but are slower to evaluate during search. We also trained a faster but less accurate rollout policy pπ(a|s), using a linear We evaluated the performance of the RL policy network in game softmax of small pattern features (see Extended Data Table 4) with play, sampling each move att∼psρ (⋅| ) from its output probability weights π; this achieved an accuracy of 24.2%, using just 2 µs to select distribution over actions. When played head-to-head, the RL policy an action, rather than 3 ms for the policy network. network won more than 80% of games against the SL policy network. We also tested against the strongest open-source Go program, Pachi14, Reinforcement learning of policy networks a sophisticated Monte Carlo search program, ranked at 2 amateur dan The second stage of the training pipeline aims at improving the policy on KGS, that executes 100,000 simulations per move. Using no search 25,26 network by policy gradient reinforcement learning (RL) . TheAlphaGo RL at all,(Silver the RL et policy al., 2016) network won 85% of games against Pachi. In com- policy network pρ is identical in structure to the SL policy network, parison, the previous state-of-the-art, based only on supervised abThe Value Net was a very strong evaluator. • 70 128 !lters 0.50 60 192 !lters 0.45 256 !lters 50 or 0.40 384 !lters 40 0.35 Uniform random ed err 0.30 rollout policy 30 Fast rollout policy 0.25 20 Value network 0.20 on expert games
AlphaGo win rate (%) SL policy network 10 Mean squar 0.15 RL policy network 0 0.10 50 51 52 53 54 55 56 57 58 59 15 45 75 105 135 165 195 225 255 >285 Training accuracy on KGS dataset (%) Move number
Figure 2 | Strength and accuracy of policy and value networks. The finalPositions version and of outcomes AlphaGo were used sampled rollouts, from Policy human Net, expert and games. Each a, Plot showing the playing strength of policy networks as a function • position was evaluated by a single forward pass of the value network vθ, of their training accuracy. Policy networks with 128, 192, 256 and 384 Valueor Net by together.the mean outcome of 100 rollouts, played out using either uniform convolutional filters per layer were evaluated periodically during training; Rolloutsrandom and rollouts, Value the Net fast as rollout evaluators. policy pπ, the SL policy network pσ or • the plot shows the winning rate of AlphaGo using that policy network Policythe RL Net policy to bias network the exploration pρ. The mean strategy. squared error between the predicted against the match version of AlphaGo. b, Comparison of evaluation • value and the actual game outcome is plotted against the stage of the game accuracy between the value network and rollouts with differentIntro policies. ML (UofT) (how many moves hadCSC311-Lec12 been played in the given position). 50 / 59
28 JANUARY 2016 | VOL 529 | NATURE | 485 © 2016 Macmillan Publishers Limited. All rights reserved Progress in Computer Go
9 dan 8 dan AlphaGo (Lee Sedol) 7 dan 6 dan 5 dan AlphaGo Team (Silver et al., 2016) 4 dan 3 dan
Professional Dan 2 dan 1 dan AlphaGo (Nature) 9 dan 8 dan 7 dan 6 dan MCTS 5 dan 4 dan 3 dan Amateur Dan AlphaGo 2 dan 1 dan Neural nets 1 kyu 2 kyu 3 kyu 4 kyu 5 kyu Traditional Search 6 kyu
Amateur Kyu 7 kyu 8 kyu 2000 2005 2010 2015
adapted from Sylvain Gelly & David Silver, Test of Time Award ICML 2017 Intro ML (UofT) CSC311-Lec12 51 / 59 Impact
Intro ML (UofT) CSC311-Lec12 52 / 59 Go is not just a game
Go originated in China more than 2,500 years ago. Reached • Korea in the 5th century, Japan in the 7th. In the Tang Dynasty, it was one of the four arts of the • Chinese scholar together with calligraphy, painting, and music. The aesthetics of Go (harmony, balance, style) are as • essential to top-level play as basic tactics.
Intro ML (UofT) CSC311-Lec12 53 / 59 2016 Match—AlphaGo vs. Lee Sedol
Best of 5 matches over the course of a week. • Most people expected AlphaGo to lose 0-5. • AlphaGo won 4-1. • Intro ML (UofT) CSC311-Lec12 54 / 59 Human moments Lee Sedol is a titan in the Go world, and achieving his level of play requires a life of extreme dedication.
It was humbling and strange to be a part of the AlphaGo team that played against him.
Intro ML (UofT) CSC311-Lec12 55 / 59 Game 2, Move 37
Intro ML (UofT) CSC311-Lec12 56 / 59 Thanks!
I played a key role at the start of AlphaGo, but the success is owed to a large and extremely talented team of scientists and engineers. David Silver, Aja Huang, C, Arthur Guez, Laurent Sifre, • George van den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, Sander Dieleman, Dominik Grewe, John Nham, Nal Kalchbrenner, Ilya Sutskever, Timothy Lillicrap, Madeleine Leach, Koray Kavukcuoglu, Thore Graepel & Demis Hassabis.
Intro ML (UofT) CSC311-Lec12 57 / 59 Course Evals
Use this time to finish course evals.
Intro ML (UofT) CSC311-Lec12 58 / 59 C. B. Browne, E. Powley, D. Whitehouse, S. M. Lucas, P. I. Cowling, P. Rohlfshagen, S. Tavener, D. Perez, S. Samothrakis, and S. Colton. A survey of monte carlo tree search methods. IEEE Transactions on Computational Intelligence and AI in games, 4(1):1–43, 2012.
R. Coulom. Efficient selectivity and backup operators in monte-carlo tree search. In International conference on computers and games, pages 72–83. Springer, 2006.
L. Kocsis and C. Szepesv´ari. Bandit based monte-carlo planning. In European conference on machine learning, pages 282–293. Springer, 2006.
C. J. Maddison, A. Huang, I. Sutskever, and D. Silver. Move Evaluation in Go Using Deep Convolutional Neural Networks. In International Conference on Learning Representations, 2015.
M. M¨uller.Computer go. Artificial Intelligence, 134(1-2):145–179, 2002.
D. Silver, A. Huang, C. J. Maddison, A. Guez, L. Sifre, G. van den Driessche, J. Schrittwieser, I. Antonoglou, V. Panneershelvam, M. Lanctot, S. Dieleman, D. Grewe, J. Nham, N. Kalchbrenner, I. Sutskever, T. Lillicrap, M. Leach, K. Kavukcuoglu, T. Graepel, and D. Hassabis. Mastering the game of Go with deep neural networks and tree search. Nature, 529(7587):484 – 489, 2016.
I. Sutskever and V. Nair. Mimicking go experts with convolutional neural networks. In International Conference on Artificial Neural Networks, pages 101–110. Springer, 2008.
Intro ML (UofT) CSC311-Lec12 59 / 59