Evaluation of Simulation Strategy on Single-Player Monte-Carlo Tree Search and its Discussion for a Practical Scheduling Problem Shimpei Matsumoto∗, Noriaki Hirosuey, Kyohei Itonagaz, Kazuma Yokooz and Hisatomo Futahashiz Abstract|Monte-Carlo Tree Search (MCTS) is a the total of benefits (zero/nonzero-sum) [2]. Among best-first search where the pseudorandom simulations them, perfect-information games do not have accidental guide the solution of problem. Recent improvements factor, so it is theoretically possible to foresee the moves on MCTS have produced strong computer Go pro- to the end. However, the computational effort actually gram, which has a large search space, and the suc- explodes with following the situations of game, so it is cess is a hot topic for selecting the best move. So virtually impossible to calculate optimal moves for most far, most of reports about MCTS have been on two- of games. To solve this problem, positional evaluation player game, and MCTS has been applied rarely in one-player games. MCTS does not need an admis- functions are given to decide the moves as criterion to de- sible heuristic, so the application of MCTS for one- termine the search range and to measure the situations. player games might be an interesting alternative. Ad- As one of the typical search algorithm using an evalua- ditionally, one-player games changed its situation by tion function to foresee the opponent decision, Mini-Max player's decision like puzzles are describable as net- method is a powerful tool for the design of game algo- work diagrams like PERT with the representation of rithm. Many algorithm refined Mini-Max method such interdependences between each operation. Therefore as αβ method have been proposed. However giving a if MCTS for one-player games is developed as a meta- proper evaluation function is extremely difficult for prob- heuristic algorithm, we would use this for not only lems with large search space. many practical problems, but also combinatorial op- timization problems. This paper investigated the ap- Recently in computer Go program, a revolutionary search plication of Single Player MCTS (SP-MCTS) intro- algorithm without an evaluation function, Monte-Carlo duced by Schadd et al. to a puzzle game called Bubble Tree Search (MCTS) was proposed by Coulom [5]. MCTS Breaker. Next this paper showed the effectiveness of performs many play-outs (playing until a simulation en- new simulation strategies on SP-MCTS by numerical experiments, and found the differences between the counters the end of the game) with pseudorandom moves, search methods and their parameters. Based on the and each situation is evaluated based on the results [1]． results, this paper discussed the application potential- Moves are selected in self-play until the end of the game. ity of SP-MCTS for a practical scheduling problem. It is well-known that the use of an adequate simulation strategy improves the level of play significantly [6, 4], Keywords: one-player game, bubble breaker, monte- and the main idea is to use heuristics. To control sim- carlo tree search, scheduling problem ulations, UCT (Upper bound Confidence for Tree) is mainly used, which is one of the most effective algo- 1 Introduction rithm and is based on UCB1 (Upper Confidence Bounds) for multi-armed bandit problem [7]. Schadd et al. pro- Games are separated into several classes mathematically posed a new MCTS variant, called Single-Player Monte- according to the characteristics such as the number of Carlo Tree-Search (SP-MCTS) for SameGame, a puzzle players, completeness of information, uncertainty, and game [3]. They mentioned A* and IDA* to compare the performance of SP-MCTS, and SP-MCTS obtained the ∗ Shimpei Matsumoto is with the Department of Computer and highest score than the others for benchmark problems, Control Engineering, Oita National College of Technology, 1666 Oaza-Maki, Oita City, Oita 870-0152 Japan; Tel/Fax: +81-97-552- so it turned out that SP-MCTS is able to be consid- 7421; Email: [email protected] ered as a new method for NP-complete puzzles. Bas- yNoriaki Hirosue is with the Department of Artificial Intelli- ing on the results, SP-MCTS is thought to be a meta- gence, Kyushu Institute of Technology, Kawazu 680-4, Iizuka 820- heuristic algorithm, so it might be applied for not only 8502, Japan; Email: [email protected] zKyohei Itonaga, Kazuma Yokoo, and Hisatomo Futahashi are perfect/imperfect information one-player games but also with the Department of Computer and Control Engineering, Oita other problems, as long as search spaces of the problems National College of Technology, 1666 Oaza-Maki, Oita City, Oita are describable as tree structure, and the problems have 870-0152 Japan; Email: fs0506, s0540, [email protected] termination of operations. where no adjacent blocks have the same color. In the first case, some bonus points are usually rewarded, and The purpose of this paper is to evaluate SP-MCTS for in the second case, points are deducted. The formula for one-player perfect-information games, Bubble Breaker deduction is similar to the formula for rewarding points, given similar rules with SameGame, and to examine ap- and applied for blocks left on the board. During deduc- plication potentiality of the solution as a meta-heuristic tion it is assumed that all blocks of the same color are algorithm. The one-player perfect-information game is connected. so-called puzzle games, and most of them give closely- defined rules. Previous researches have shown that most Bubble Breaker, the problem in this paper, adopts the of puzzle games are equal to optimization problems be- following scoring function, which the score Sk of kth turns longing to the class of NP-Complete, which are difficult is given as to find an optimal solution, and puzzle games are able − 2 to be considered as typical combinatorial optimization Sk = (n 2) : (1) problems. Therefore if we can clarify the effectiveness of SP-MCTS for such problems, we will open up new possi- The final score is given as follows. bilities of SP-MCTS for many practical problems that is ( P able to be described as combinatorial optimization prob- kmax i=1 Sk + 1000 ; if R = 0; lems. This paper firstly applies SP-MCTS for Bubble Ts = P (2) kmax S − (R − 2)2 ; otherwise: Breaker already reported NP-Completeness [9], and eval- i=1 k uates its effectiveness. This paper develops a softawre of Bubble Breaker with Action Script 3.0 to present the where R is the number of blocks left on the board, and sequence of moves visually until a game is solved. This kmax is the number of turns at the termination of the paper develops SP-MCTS based on concept of Schadd et problem. al., and proposes new heuristics. This paper shows the results of numerical experiments, and discusses the char- In SameGame given the similar rule with Bubble Breaker, acteristic of SP-MCTS. Finally based on the efforts of this the number of leaf notes for a random initial position is paper, this paper examines the availability of SP-MCTS estimated to be approximately 1085 in average, and the for practical problems by generalizing the procedures of total number of possible states is approximately 10159, SP-MCTS for one-player games. This paper mentions a which is calculatedP by the number of combinations for k r n reentrant scheduling problem as an example of practical columns, C = n=0 c where r is the height of the col- problem [8]. umn, c is the number of colors, and k is the number of columns. As the similar problems, Clickomania with 5 colors and 2 Columns was proven to be NP-complete by 2 Problem [9] where no points are rewarded and the only objective is to clear the board. In SameGame, a variant of Click- There is a rectangular playing screen initially filled with omania, there are two terminal positions with no blocks several, typically 4 or 5, kinds of blocks (colors) at ran- or some blocks on the board. SameGame is proven as a dom. By selecting one of a group of adjoined blocks, a harder problem than Clickomania by Schadd et al. and player may remove them from the screen. A move con- shown as NP-complete, too [3]. The problem in this pa- sists of removing a group of (at least two) orthogonally per, Bubble Breaker adopts the same rule of SameGame adjacent blocks with the same color. The blocks on top of except scoring formula, so we can understand that Bub- the removed group will fall down, and a column without ble Breaker is also NP-complete. any blocks will be trimmed away by other columns sliding to the left. Usually, there will be no time constraints in 3 Algorithm the game, however some implementations gradually push the rows upward or drop blocks from above. The game 3.1 Concept is over if no more blocks can be removed. In MCTS, one new node on search tree is expanded per For each removed group points are rewarded. As the most iteration. Usually the following four steps are repeated versions of the game, the amount of points is dependent until the time runs out, but this paper iterates the steps on the number of blocks removed and is given by the for- until the number of expanded nodes reaches the prede- mula (n−k)2 for removing n tiles (the size of the removed termined threshold. One-player games do not have the group) at once, where k = 1 or 2 depending on the im- opponent unlike two-player games, so we do not have to plementation.