Published as a conference paper at IFSA-SCIS 2017______FML-based Prediction Agent and Its Application to Game of Go

Chang-Shing Lee Chia-Hsiu Kao Yusuke Nojima Nan Shuo Mei-Hui Wang Sheng-Chi Yang Ryosuke Saga Naoyuki Kubota Dept. of Computer Science and Information Engineering Dept. of Electrical Engineering Dept. of System Design Knowledge Application and Web Service Research Center and Information Science Tokyo Metropolitan University National University of Tainan Osaka Prefecture University Tokyo, Japan Tainan, Taiwan Osaka, Japan [email protected] [email protected] [email protected]

Abstract- In this paper, we present a robotic prediction agent Many different real-world applications are with a high-level including a darkforest Go engine, a fuzzy markup language (FML) of uncertainty. A lot of researches proved the good performance assessment engine, an FML-based decision support engine, and a of using fuzzy sets. Being an IEEE Standard in May 2016, fuzzy robot engine for game of Go application. The knowledge base and markup language (FML) provides designers of intelligent rule base of FML assessment engine are constructed by referring decision making systems with a unified and high-level the information from the darkforest Go engine located in NUTN methodology for describing systems’ behaviors by means of and OPU, for example, the number of MCTS simulations and rules based on human domain knowledge [10, 11]. The main winning rate prediction. The proposed robotic prediction agent advantage of using FML is easy to understand and extend the first retrieves the database of Go competition website, and then the implemented programs for other researchers. FML is with FML assessment engine infers the winning possibility based on the understandability, extendibility, and compatibility of information generated by darkforest Go engine. The FML-based decision support engine computes the winning possibility based on implemented programs as well as efficiency of programming the partial game situation inferred by FML assessment engine. [10]. There are considerable research applications based on Finally, the robot engine combines with the human-friendly robot FML and Genetic FML (GFML) such as game of Go [12, 13] partner PALRO, produced by Fujisoft incorporated, to report the and diet [14, 15]. Additionally, Acampora et al. [16] proposed game situation to human Go players. Experimental results show FML Script to make it with evolving capabilities through a that the FML-based prediction agent can work effectively. scripting language approach. Akhavan et al. [17] used FML- based specifications to validate and implement fuzzy models. Keywords—Fuzzy markup language; prediction agent; decision support engine; robot engine; darkforest Go engine The objective of this paper is to use FML to construct the knowledge base and rule base of the proposed agent and then I. INTRODUCTION predict the winner of the game based on the information from darkforest Go open source and extracted sub-games. In this The game of Go originated from China. There are 381 paper, in addition to information provided by darkforest positions to play for a 19×19 board game [2]. Normally, the program, we have an additional system to show the information weaker player plays Black and starts the game. Stones are of the game such as current game situation. According to the consecutively located by two players, Black and White, on the first-stage prediction results of darkforest Go engine [8, 9] and points where the lines cross [3]. Black is also allowed to start the second-stage inferred results of the FML assessment engine with some handicap stones located on the empty board when the [18], we further introduce the third-stage FML-based decision difference in strength between the players is large [3]. Moreover, support engine to predict the winner of the game. Next, we the Go rules contain parts of the ko rule, life and death, suicide, choose seven games from 60 games ( vs. top professional and the scoring method [3]. In the end, the player who controls Go players in Dec. 2016 and in Jan. 2017) and three games of most territory wins the game. The level of amateur Go players is amateur Go players to evaluate the performance. Finally, we ranked as Kyu (1K is the highest one) and Dan (1D – 7D and ID combine playing Go with the fourth-stage robot engine to report is the lowest one). Professional Go players are ranked from 1P real-time situation to Go players. The experimental results show to 9P and the highest level is 9P [19]. The strongest professional the proposed approach is feasible. Go player in the world is Jie Ke from China in Feb. 2017 [20]. The remainder of this paper is organized as follows: Section Competing with top human players in the ancient, game of II introduces Dynamic DarkForest Go (DyNaDF) cloud Go has been a long-term goal of artificial intelligence [1]. Go’s platform for game of Go application. Section III describes the high branching factor makes traditional search techniques or proposed FML-based prediction agent by introducing the system even on cutting-edge hardware ineffective [1]. Additionally, the structure and the FML-based decision support engine. The evaluation function of Go could change drastically with one experimental results are shown in Section IV. Finally, stone change [1]. Combining a Deep Convolutional Neural conclusions are given in Section V. Network (DCNN) with (MCTS), Google DeepMind AlphaGo has sent shockwaves throughout II. DYNADF CLOUD PLATFORM FOR GAME OF GO Asia and the world since Challenge Match with in 2016 [3, 5, 6, 19]. Moreover, the new prototype version of A. Structure of Dynamic Darkforest Cloud Platform for Go AlphaGo played as Master and Magister on the online servers Fig. 1 shows the structure of the DyNaDF Cloud Platform and FoxGo defeated more than 50 of the top Go players and its brief descriptions are as follows: (1) The DyNaDF cloud in the world in Dec. 2016 and in Jan. 2017 [7]. platform for game of Go application is composed of a playing- The authors would like to thank the financially support sponsored by the Ministry of Science and Technology of Taiwan under the grant MOST 105- 2622-E-024-003-CC2 and MOST 105-2221-E-024-017. Published as a conference paper at IFSA-SCIS 2017______Go platform located at National University of Tainan (NUTN) / Machine Recommendation (MR) Logout Taiwan and National Center for High-Performance Computing Game Finish (NCHC) / Taiwan, a darkforest Go engine located at Osaka Recommendation Admin Prefecture University (OPU) / Japan, and the robot PALRO from Black (62.50%) Black Recommendation White (57.81%) Move 1 (B Q16) Darkforest Move 2 (W D16) Tokyo Metropolitan University (TMU) / Japan; (2) Human Go Move 3 (B R4) • suggest 1: H2 Move 4 (W D3) Move 5 (B C5) (Num simulation: 689, Win rate: 0.320) Move 6 (B P3) Move 7 (B M4) • suggest 2: A3 Move 8 (W C9) players surf on the DyNaDF platform located at NUTN / NCHC Move 9 (B H3) (Num simulation: 531, Win rate: 0.227) Move 10 (W C4) Move 11 (B E4) • suggest 3: M9 Move 12 (W D5) to play with darkforest Go engine located in OPU; (3) The FML Move 13 (B D4) (Num simulation: 415, Win rate: 0.248) Move 14 (W C6) Move 15 (B C3) • suggest 4: N19 Move 16 (W B5) Move 17 (B B4) (Num simulation: 329, Win rate: 0.308) Move 18 (W C5) assessment engine infers the current game situation based on the Move 19 (B E3) • suggest 5: H13 Move 20 (W B3) Move 21 (B D2) (Num simulation: 263, Win rate: 0.225) Move 22 (W N4) prediction information from darkforest and stores the results into Move 23 (B Q6) Move 24 (W M5) Move 25 (B L4) Move 26 (W N3) Move 27 (B Q3) Move 28 (W O17) Move 30 (W Q2) the database; (4) PALRO receives the game situation via the Move 29 (B R14) White Recommendation Move 31 (B F17) Move 32 (W S3) Internet and reports to the human Go players; (5) Human can Move 33 (B P17) Move 34 (W R8)

Move 103 (B L13) learn more information about game’s comments via Go eBook. Move 104 (W S16) Move 105 (B S15) Move 106 (B S13) Move 107 (B S17) Move 108 (W D13) Move 109 (B E13) Move 110 (W D12) Move 111 (B E12) Move 112 (W D11) Move 113 (B E11) Move 114 (W E10) Move 115 (B F10) Move 116 (W E9) Move 117 (B R10) Move 118 (W S10) Move 119 (B Q9) Move 120 (W S9) Move 121 (B F18) Move 122 (W F9) Move 123 (B G10) Move 124 (W L2) Move 125 (B B2) Tokyo Tainan Move 126 (W J2) Move 127 (B K3) Move 128 (W K2)

Osaka (a) OPU / Japan TMU / Japan NUTN / NCHC Human Go Players

Darkforest Go Engine PALRO Black: Jie Ke (9P) White: Master Komi: 6.5 Simulation Number: 3000 Date: Dec. 31, 2016

100

75 Playing-Go Platform FML Assessment Engine (%)

50 Black White

25 Go eBook Rate Winning

0 Fig. 1. Structure of dynamic darkforest cloud platform for Go. 1 11 21 31 41 51 61 71 81 91 101 111 121 Move No.

B. Introduction to Dynamic Darkforest Cloud Platform Black: Jie Ke (9P) White: Master Komi: 6.5 Simulation Number: 3000 Date: Dec. 31, 2016 4k

This subsection introduces the developed DyNaDF platform, 3k including a demonstration game platform, a machine 2k Black White

recommendation platform, and an FML assessment engine. 1k Number of of Simulations Number GUID 0 1 11 21 31 41 51 61 71 81 91 101 111 121 WCCI-2016-GO-466 Move No. Description Number of Simulations Winning Rate Difference FML Assessment-1 FML Assessment-2 Black: Jie Ke (9P) White: Master Komi: 6.5 Simulation Number: 3000 Date: Dec. 31, 2016 Difference

Move 1 (UncertainSituation) Move 1 (UncertainSituation) Move 1 (UncertainSituation) Move 1 (UncertainSituation) Move 2 (UncertainSituation) Move 2 (UncertainSituation) Move 2 (UncertainSituation) Move 2 (UncertainSituation)

Number of MCTS Simulations Komi Move 123 (WhiteObviousAdvantage) Move 123 (UncertainSituation) Move 123 (BlackObviousAdvantage) Move 123 (WhitePossibleAdvantage) Move 124 (WhiteObviousAdvantage) Move 124 (UncertainSituation) Move 124 (BlackObviousAdvantage) Move 124 (UncertainSituation) 3000 6.5 Move 125 (WhiteObviousAdvantage) Move 125 (UncertainSituation) Move 125 (BlackObviousAdvantage) Move 125 (UncertainSituation) Move 126 (WhiteObviousAdvantage) Move 126 (UncertainSituation) Move 126 (BlackObviousAdvantage) Move 126 (UncertainSituation) (Maximum: 50000) Move 127 (WhiteObviousAdvantage) Move 127 (UncertainSituation) Move 127 (BlackObviousAdvantage) Move 127 (UncertainSituation) Mode Color Move 128 (WhiteObviousAdvantage) Move 128 (UncertainSituation) Move 128 (BlackObviousAdvantage) Move 128 (WhitePossibleAdvantage) Multiple Black (b) Submit (a) Fig. 3. Captured screenshots, including (a) predicted next moves and (b) Human & Machine vs. Human Demonstration Game history of winning rrate, the number of simulations, and each-move Logout inferred current game situation provided by the machine Black: Jie Ke (9P) White: Master Komi: 6.5 Simulation Number: 3000 Date: Dec. 31, 2016 / simulation: 3000 / komi: 6.5 / mode: multiple / black recommendation platform. Black: Jie Ke (9P) White: Master Komi: 6.5 Simulation Number: 3000 Date: Dec. 31, 2016 Download SGF 128 / 128 The game between Jie Ke (9P) as Black and Master as White on Dec. 31, 2016 is used as an example of Figs. 2 and 3. Fig. 2 (a) shows the screen of setting game like the number of MCTS simulations, komi, playing mode, and human is as White or as Black. Fig. 2 (b) shows the game record provided by the demo game platform. Figs. 3 (a)–3 (b) show the predicted next moves as well as history information like winning rate, the number of simulations, and each-move inferred current game situation provided by the machine recommendation platform. FML assessment engine consists of four different methods, including winning rate difference, simulation difference, FML Assessment-1, and FML Assessment-2 [18]. In Fig. 2 (a), we set the number of MCTS simulations to 3000 for darkforest Go (b) engine and acquire the predicted and inferred results from Fig. 2. Captured screenshots, including (a) setting game and (b) game record darkforest and the FML assessment engine, respectively. From provided by the demo game platform. Published as a conference paper at IFSA-SCIS 2017______Fig. 3 (b), we can observe that the top-move rates of Black and  Stage III: The proposed FML-based decision support engine White are 62.5% and 57.81%, respectively. Fig. 3 (b) shows that computes the winning possibility based on the partial game FML Assessment-2 predicts “White is Possible Advantage situation inferred by FML assessment engine (see Section (WhitePossibleAdvantage)” for the last move. III.B) and stores the predicted results into the database.

III. FML-BASED PREDICTION AGENT  Stage IV: The robot engine retrieves information from the database to comment on the game situation, including (1) A. System Structure Black and White’s move numbers that appear the first 3 Fig. 4 shows the system structure of the proposed four-stage highest and the last 3 lowest number of simulations as well FML-based prediction agent, including Stage I: darkforest Go as the highest and lowest winning rates, (2) Black and engine, Stage II: FML assessment engine, Stage III: FML-based White’s average winning rates and top-move rates, and (3) decision support engine, and Stage IV: robot engine. The overall game situation. It also reports the real-time predicted followings are its short descriptions: top-3-move positions to the human Go player to think carefully before playing his /his next move [18]. Stage I: Stage II: FML Assessment Engine Darkforest Go Engine FML Assessment-2 B. FML-based Decision Support Engine

Move 1 (UncertainSituation) Winrate Fuzzy Inference Move 2 (UncertainSituation) SN Mechanism This subsection mainly focuses on the FML-based decision TMR DyNaDF Platform Game record with N moves Move N-5 (WhitePossibleAdvantage) support engine (DSE) whose diagram is shown in Fig. 5. Move N-4 (UncertainSituation) Human Go Move N-3 (UncertainSituation) Move N-2 (UncertainSituation) Players Darkforest Move N-1 (UncertainSituation) Stage III: FML-based Decision Support Engine (DCNN + MCTS) Move N (WhitePossibleAdvantage) x1/K B Stage III: FML-based Decision Support Engine B Current Game Situation x1 of a game with N moves w1 U x2 MAX PALRO Stage IV: Robot Engine FML Assessment-2

2 Move 1 (UncertainSituation) W w x11 Fig. 4. System structure of four-stage FML-based prediction agent. Move 2 (UncertainSituation) Decision Process y

W Move N-5 (WhitePossibleAdvantage) w3 Move N-4 (WhitePossibleAdvantage) x2/K  Stage I: In this paper, we apply Facebook darkforest Go Move N-3 (WhitePossibleAdvantage) Move N-2 (WhitePossibleAdvantage) x1 B MAX Move N-1 (WhitePossibleAdvantage) x2 wK-1 open source that trains a DCNN to predict the next top-k Move N (WhitePossibleAdvantage)

moves [8]. Darkforest Go engine powered by x11 W

has been developed mainly by Tian and Zhu from Facebook xK/K ) wK B : Black is obvious advantage x1 AI Research (FAIR) since May 2015 and was open to the B B : Black is possible advantage x2 U : Both are in an uncertain situation MAX public in 2016 [8, 9]. Darkforest has a stable rank of 5D on 11 x W W : Black is obvious advantage the KGS servers and pure policy network achieves a stable W : White is obvious advantage rank of 3D on KGS. It received the third place and second Fig. 5. Diagram of the FML-based decision support engine. place in the KGS Go Tournament 2016 and in the ninth UEC Cup Computer 2016, respectively [8, 9]. Darkforest relies on a DCNN designed for long-term predictions and has been able to substantially improve the winning rate for pattern

matching approaches against MCTS-based approaches, even Move No. with looser search budgets [8]. Tian and Zhu [9] proposed a

12-layered full convolutional network architecture for 1 15 30 45 60 75 85 90 105 120 135 150 165 178

darkforest where (1) each convolution layer is followed by a K = 4 Neighborhood 1/4 Neighborhood x1/4 = (CGSm40, CGSm41, …, CGSm50) 2/4 ReLU nonlinearity, (2) all layers use the same number of x2/4 = (CGSm85, CGSm86, …, CGSm95) Neighborhood3/4 x3/4 = (CGSm130, CGSm131, …, CGSm140) filters at convolutional layers (w = 254) except for the first Neighborhood4/4 x = (CGS , CGS , …, CGS ) 4/4 m168 m169 m178 layer, (3) no weight sharing is used, (4) pooling is not used Fig. 6. Example of a game with 178 moves when K = 4. owing to negatively affecting the performance, and (5) only one softmax layer is used to predict the next move of Back  and White to reduce the number of parameters. DSE ++ + + ++  Stage II: FML assessment engine adopts each-move- B B U W W position, darkforest-predicted top-5-move number of simulations and winning rate to decide each-move number of simulations, winning rate, and top-move rate (TMR) [18]. After that, the FML assessment engine infers each-move current game situation, including “Black is obvious x ++ -2 -1 0 1 2 advantage (BlackObviousAdvantage, B ),” “Black is Fig. 7. Fuzzy sets for B++, B+, U, W+, and W++. + possible advantage (BlackPossibleAdvantage, B ),” “Both are in an uncertain situation (UncertainSituation, U),” For example, if the input vector is x = (x1, x2, …, x11), then “White is possible advantage (WhitePossibleAdvantage, vector xi / k denotes the input 11 current game situations (CGSs) + W ),” and “White is obvious advantage in the Neighborhoodi / k. Fig. 6 gives an example of a game with ++ (WhiteObviousAdvantage, W ).” 178 moves when K = 4, where the input information of Neighborhood1/4, Neighborhood2/4, Neighborhood3/4, and Published as a conference paper at IFSA-SCIS 2017______

Neighborhood4/4 is the current game situations for experiments, including the adopted test models (TM1–TM4), Neighborhood of fuzzy number Move-45 (moves 40-50), collected games (G1–G13), machines that executed darkforest Neighborhood of fuzzy number Move-90 (moves 85-95), Go engine, namely (OPU, 2), (NUTN, 2), and (NCHC, 4), and Neighborhood of fuzzy number Move-135 (moves 130-140), the setting of the number of MCTS simulations (500 / 1000 / and Neighborhood of fuzzy number Move-Last (moves 168- 5000 / 10000 / 20000). 178), respectively. Fig. 7 is the fuzzy sets for B++, B+, U, W+, and W++. Table I shows the algorithm of the FML-based DSE. Taiwan

TABLE I. ALGORITHM OF FML-BASED DECISION SUPPORT ENGINE. ... Input: NCHC / 4GPUs NCHC / 4GPUs NUTN / 2GPUs NCHC 56 NCHC 57 Tesla K80 x 2 GeForce GTX 1080 x 4 Quadro K2200 x 1 All moves’ game situations inferred by FML assessment engine of a game Game Records Quadro M2000 x 1 Output: Taiwan Game result provided by FML-based decision support engine Japan DyNaDF Method: Platform Step 1: Divide a game into sub-games Human Go Players Step 2: For ← to Tainan Kaohsiung OPU / 2GPUs OPU / 1GPU Step 2.1: ndex ← / ) /* denotes the total number of moves of the GeForce GTX 1060 x 2 GeForce GTX 1060 x 1 input game*/ Fig. 8. Scenario of the experiments on DyNaDF platform. Step 2.2: If < Step 2.2.1: Find the current game situations CGSs) for moves [ ndex TABLE II. BASIC PROFILE OF ADOPTED MACHINES. ) − 5, ndex ) 5] in the window of e ghborhood / Location / GPU Number (Abbreviation) GPU Card Model Step 2.3: If equals OPU (Osaka, Japan) / 2 (OPU, 2) GeForce GTX 1060 2 Step 2.3.1: Find the CGSs for moves [ - 0, ] in the window of Quadro K2200 1 NUTN (Tainan, Taiwan) / 2 (NUTN, 2) e ghborhood / Quadro M2000 1 Step 2.4: Count individually the number of CGSs’ linguistics, including NCHC (Tainan, Taiwan) / 4 (NCHC, 4) GeForce GTX 1080 4 B , B , W , and W , in the window of e ghborhood / and store the counted results into array CGSary by ASIC PROFILE OF COLLECTED GAMES CGSary ← [countB , countB , countW , countW ] TABLE III. B 13 . Game Step 2.5: countmax ← MAX CGSary) /*Find the maximum of CGSary*/ Black / Level White / Level Date Winner Step 2.6: sum ← SUM CGSary) /*Total the CGSary */ No. Step 2.7: If sum equals 0 G1 Master Jie Ke / 9P 2016/12/30 Black Step 2.7.1: / ← 0 G2 Jie Ke / 9P Master 2016/12/30 White Step 2.8: If sum does not equal 0 G3 Master Yuta Iyama / 9P 2017/1/2 Black Step 2.8.1: CGS ndex←index of countmax /*Set the index of countmax to G4 Jie Ke / 9P Master 2017/1/3 White CGS ndex whose value is 0, , , or */ G5 Chun-Hsun Chou / 9P Master 2017/1/4 White Step 2.8.2: Outputary ←[- , - , , ] G6 Master Weiping Nie / 9P 2017/1/4 Black Step 2.8.3: / ←Outputary [CGS ndex] G7 Li Gu / 9P Master 2017/1/4 White

Step 3: yCGS ← ∑ / × G8 Shuji Takemura / 1D Darkforest 2017/2/9 White Step 4: Implement decision process G9 Minoru Ueda / 5K Darkforest 2017/2/9 White Step 4.1: If yCGS ≤ − G10 Lu-An Lin / 6D Darkforest 2017/2/20 Black ++ Step 4.1.1: y ← B Yi-Min Hsieh / 6P G11 Chun-Hsun Chou / 9P 2016/7/25 White Step 4.2: If 0 > yCGS > − Darkforest + PALRO + Step 4.2.1: y ← B Yi-Min Hsieh / 6P G12 Chun-Hsun Chou / 9P 2016/7/25 Uncertain Step 4.3: If yCGS equals 0 Darkforest + PALRO Step 4.3.1: y ← U Lu-An Lin / 6D G13 Darkforest 2016/11/16 Black Step 4.4: If > yCGS > 0 Darkforest + PALRO + Step 4.4.1: y ← W Step 4.5: If yCGS ≥ TABLE IV. INFORMATION OF EXPERIMENTS 1 TO 8. ++ Step 4.5.1: y ← W Test Model / (Machine Name, No. of GPU) / Exp. No. Step 5: End Collected Game No. No. of MCTS Simulations Exp. 1 TM1 / G1–G7 (OPU, 2), (NUTN, 2) / 1500 Exp. 2 TM1 / G1–G7 (OPU, 2), (NUTN, 2) / 3000 IV. EXPERIMENTAL RESULTS Exp. 3 TM1 / G6 (OPU, 2) / 500/5000/10000 Exp. 4 TM2 / G8 (OPU, 2) / 500 A. Experiment Scenario and Game Information Exp. 5 TM2 / G9 (OPU, 2) / 1000 Fig. 8 shows the scenario of the experiments whose brief Exp. 6 TM2 / G10 (NCHC, 4) / 20000 descriptions are as follows: (1) the invited human Go players Exp. 7 TM3 / G11–G12 (Facebook FAIR) surf on the DyNaDF cloud platform to play with darkforest Exp. 8 TM4 / G13 (NUTN, 2) / 5000 Note: located in NUTN, NCHC, or OPU, (2) the game records on the  Test Model No. TM ) denotes that we fed the collected game records Internet are downloaded and fed into the DyNaDF cloud downloaded from the Internet, for example, Master vs. top professional platform, and then (3) the predicted each-move information is Go players into our developed DyNaDF cloud platform. generated by darkforest during playing. Table II shows the  TM denotes that we invited the Go player to play with darkforest via information of three machines that we installed darkforest Go our developed DyNaDF cloud platform. engine such as machine’s location and number & model of GPU.  TM denotes that the weaker human cooperated with PALRO to challenge the stronger human by referring to the recommended top- - Table III shows the basic profile of the collected games, of next moves from darkforest. which 9 games are from professional Go players, and 4 games  TM4 denotes that the human cooperated with PALRO to challenge from amateur ones. Table IV shows the information of 8 darkforest by referring to the recommended top- -next moves from darkforest. Published as a conference paper at IFSA-SCIS 2017______

96.72 96.4 B. Darkforest-Predicted Winning Rate and TMR 100 95.4 94.47 95.96 94.09 90 Computer 80 The objective of experiments 1 and 2 is to evaluate the Human 70 64

(%) 62 variance in winning rate under the different number of MCTS 60 58 60 56.67 49.47 simulations and Neighborhood of various fuzzy numbers. The 50 total moves of G1–G7 is 228, 128, 135, 178, 118, 254, and 235, 40 respectively. Fig. 9 shows the darkforest-predicted winning rate Rate Winning 30 20 for G1–G7 on Neighborhood of fuzzy number Move-100, 5.84 6 10 5.1 3.1 3.95 3.48 Neighborhood of fuzzy number Move-200, and Neighborhood 0 500 / 100 500 / 200 500 / 254 5000 / 100 5000 / 200 5000 / 254 10000 / 100 10000 / 200 10000 / 254 of fuzzy number Move-Last. In Fig. 9, the values with a cross Number of Simulations / Move No. (+) denote the human’s winning rate of Exp. 2 (OPU, 2) / 3000, (a) while a star (*) is the ones of computer’s. Fig. 9 also shows that 100 90 Computer Human darkforest successfully predicted that “computer won the game 80 for G1–G7,” and “the winning rate difference between computer 70 62 62 64

(%) 60.63 60 60 60 58 Go and human is the smallest when G6 was already played 100 60 56 56 56 56.69 56.69 57 56.69 56 56.69 56.39

moves.” 50 Move Rate Move

- 40 100 95.94* Top 30 * * 90 89.45 92.97 * 20 80 * 87.53 * 62.37 * 73.91 10 70 65.02 74.79 * * * * * 73.51 73.18 * * 72.49 * * 0 (%) 60 70.99 * 57.80 73.76 51.86 66.97 69.65 + 500 / 100 500 / 200 500 / 254 5000 / 100 5000 / 200 5000 / 254 10000 / 100 10000 / 200 10000 / 254 50 + 43.93 + 49.55 Number of Simulations / Move No. 40 38.06 47.80 + 31.14 + + (b) 30 + + 24.96 Winning Rate Rate Winning 43.09 34.63 + + 14.75 33.25 + Fig. 11. Bar charts of (a) winning rate and (b) top-move rate for Exp. 3. 20 + + 27.57 23.93 30.84 + 10 15.05 + + 8.99 3.92 6.11 0 + C. Accuracy of FML-based Decision Support Engine This subsection describes the accuracy of FML-based Exp. 1 / (OPU, 2) / 1500 : Computer Exp. 1 / (OPU, 2) / 1500 : Human decision support engine. We first used the collected data from Exp. 2 / (OPU, 2) / 3000 : Computer Exp. 2 / (OPU, 2) / 3000 : Human Exps. 1 and 2, namely, (OPU, 2) / 1500, (NUTN, 2) / 1500, Fig. 9. Winning rate curves for Exps. 1 and 2. (OPU, 2) / 3000, and (NUTN, 2) / 3000, to evaluate G1–G7. After that, we inferred the game result based on the current game Fig. 10 shows the bar chart of darkforest-predicted winning situations from the last 10 moves to the last one move. The rate and TMR by two machines, namely (OPU, 2) and (NUTN, results show each game is correct except G7 of (NUTN, 2) / 2), when the number of MCTS simulations equals to 1500 and 1500. Next, we adopted the collected data of G1–G7 from (OPU, 3000. It indicates that TMR is roughly 50% and there is no 2) / 3000 plus G8–G10 (Exps. 4–6) to evaluate the performance obvious difference in TMR when changing the number of of FML-based DSE. Table V shows the inferred result of each MCTS simulations from 1500 to 3000 for both machines (OPU, game which indicates that G8 has the problem and the accuracy 2) and (NUTN, 2). Additionally, the winning rates of human and using K = 4 is higher than the one using K = 3. computer are roughly 20%–30% and 70%–80%, respectively, which exactly meets the actual game result (computer won G1– TABLE V. RESULTS OF FML-BASED DECISION SUPPORT ENGINE G7). Figs. 11 (a) –11 (b) show the bar charts of winning rate and Game No. top-move rate of Exp. 3, respectively. Fig. 11 (a) indicates that (Machine Name, No. of GPU) / Winner K = 3 K = 4 the winning rate is still close to each other even human’s is No. of MCTS Simulations G1: (OPU, 2) / 3000 Black B+ B+ higher than computer’s at the time of playing move 00. G2: (OPU, 2) / 3000 White W+ W+ However, the winning rate of computer increases after more G3: (OPU, 2) / 3000 Black B+ B+ moves were played. Observe Fig. 11 (b) that the top-move rate G4 (OPU, 2) / 3000 White W+ W+ is kept 55%–62% whatever the setting of the number of MCTS G5: (OPU, 2) / 3000 White W+ W+ simulations is 500, 5000, or even 10000. G6: (OPU, 2) / 3000 Black B+ B++ G7: (OPU, 2) / 3000 White W+ W+ 100 G8: (OPU, 2) / 500 White B+ W+ 90 72.7 81.5 + ++ 82.2 81.3 G9: (OPU, 2) / 1000 White W W 80 G10: (NCHC, 4) / 20000 Black B+ B+ 70 49.4 50.9 53.6 53.2 Accuracy 90% 100% 60 53.3 53.5 49.1 50.2 50

Percentage (%) Percentage 40 27.2 17.9 D. Human viewpoint on learning with PALRO 30 This subsection is to describe the human’s viewpoint about 18.7 18.1 20 Go learning with the PALRO. In Exp. 7, we have two human Go 10 players, 6P Yi-Min Hsieh Black) and 9P Chun-Hsun Chou 0 Winrate of Human Winrate of Computer TMR of Human TMR of Computer White), who played games with the darkforest and PALRO at the events of IEEE WCCI 0 6 and ICIRA 0 6. After (OPU, 2) / 1500 (OPU, 2) / 3000 (NUTN, 2) / 1500 (NUTN, 2) / 3000 demonstration game at IEEE WCCI 0 6 for example, G ), Fig. 10. Bar chart of darkforest-predicted winning rate and TMR by machines (OPU, 2) and (NUTN, 2) for Exps. 1 and 2. Yi-Min Hsieh commented: “At the cr t cal moment, PALRO ll Published as a conference paper at IFSA-SCIS 2017______cheer for me and somet me t ll rem nd me of hav ng some tea [3] E. W. D. Werf, “AI techniques for the game of Go,” Satawyse b. v., to relax. It seems to have a partner to f ght together rather than Maastricht, The Netherlands , 2004. f ght alone. If PALRO can be appl ed to educat on or dynam c [4] C. Clark and A. Storkey, “Teaching deep convolutional neural networks to play Go,” Dec. 0 4, [Online] Available: assessment, t ll be helpful for ch ldren’s learn ng.” Fig. a) http://arxiv.org/abs/1412.3409. shows the picture when Yi-Min Hsieh commented on G at [5] D. Silver, A. Huang, C. J. Maddison, A. Guez, L. Sifre, G. van den ICIRA 0 6. Fig. b) shows the picture that Lu-An Lin played Driessche, J. Schrittwieser, I. Antonoglou, V. Panneershelvam, M. G with darkforest in Exp. 8. Meanwhile, she also cooperated Lanctot, S. Dieleman, D. Grewe, J. Nham, N. Kalchbrenner, I. Sutskever, with darkforest and PALRO reporting the top- next moves via T. Lillicrap, M. Leach, K. Kavukcuoglu, T. Graepel, and D. Hassabis, Facebook. Lin said that “Play ng th darkforest s helpful and “Mastering the game of Go with deep neural networks and tree search,” nterest ng to me. In add t on to PALRO’s report ng pred cted Nature, vol. 529, pp. 484–489, Jan. 2016. pos t ons, a short t me of s ng ng and danc ng dur ng the game [6] E. Gibney, “Google masters Go: deep-learning software excels at complex ancient board game,” Nature, vol. 529, pp. 446, Jan. 2016. refreshes me very much.” [7] E. Gibney, “Google secretly tested AI bot: updated version of Google DeepMind’s AlphaGo program revealed as mystery online player,” Nature, vol. 541, pp. 142, Jan. 2017. [8] Y. Tian and Y. Zhu, “Better player with neural network and long-term prediction,” Jan. 2016, [Online] Available: https://arxiv.org/abs/1511.06410. [9] Y. D. Tian, “Facebookresearch / darkforestGo,” Jan. 0 6, [Online] Available: https://github.com/facebookresearch/darkforestGo. (a) (b) [10] IEEE Computational Intelligence Society, “ 855-2016-IEEE Standard for Fig. 12. (a) Exp. 7: ICIRA 2016 event and (b) Exp. 8: Lu-An Lin played with Fuzzy Markup Language,” May 0 6, [Online] Available: darkforest and PALRO @ NUTN on Facebook. http://ieeexplore.ieee.org/servlet/opac?punumber=7479439. [11] G. Acampora, V. Loia, C. S. Lee, and M. H. Wang, “On the Power of V. CONCLUSIONS Fuzzy Markup Language,” Springer-Verlag, Germany, Jan. 2013. This paper presents a robotic prediction agent, including a [12] C. S. Lee, M. H. Wang, M. J. Wu, Y. Nakagawa, H. Tsuji, Y. Yamazaki, darkforest Go engine, a fuzzy markup language (FML) and K. Hirota, “Soft-Computing-based emotional expression mechanism assessment engine, an FML-based decision support engine, and for game of Computer Go,” Soft Computing, vol. 17, no. 7, pp. 1263–1282, a robot engine for game of Go application. The proposed FML- Jul. 2013. based decision support engine computes the winning possibility [13] C. S. Lee, M. H. Wang, M. J. Wu, O. Teytaud, and S. J. Yen, “T FS- based adaptive linguistic assessment system for semantic analysis and based on darkforest’s prediction and the partial game situation human performance evaluation on game of Go,” IEEE Transactions on inferred by FML assessment engine. Additionally, professional Fuzzy Systems, vol. 23, no. 2, pp. 400–420, Apr. 2015. Go player commented that “PALRO will be helpful for [14] C. S. Lee, M. H. Wang, H. Hagas, Z. W. Chen, S. T. Lan, S. E. Kuo, H. children’s learning if it can be applied to the education.” In the C. Kuo, and H. H. Cheng, “A novel genetic fuzzy markup language and future, we will apply machine learning to the proposed platform its application to healthy diet assessment,” International Journal of and collect more data to further evaluate its accuracy and extend Uncertainty, Fuzziness, and Knowledge-Based Systems, vol. 20, no. 2, pp. 247–278, Oct. 2012. the proposed approach to achieve the goal of on-line real-time [15] C. S. Lee, M. H. Wang, and S. T. Lan, “Adaptive personalized diet Go prediction platform. Additionally, the proposed approach linguistic recommendation mechanism based on type-2 fuzzy sets and will be pertinent to generate play-comments on a game to genetic fuzzy markup language,” IEEE Transactions on Fuzzy Systems, highlight the positional strategic plan followed by a player vol. 23, no. 5, pp. 1777–1802, Oct. 2015. during a sequence of moves. [16] G. Acampora, M. Reformat, and A. Vitiello, “Extending FML with evolving capabilities through a scripting language approach,” in ACKNOWLEDGMENT Proceeding of 2014 IEEE International Conference on Fuzzy Systems (FUZZ-IEEE 2014), Beijing, China, Jul. 6-11, 2014, pp. 857–864. The authors would like to thank all invited Go players for [17] M. P. Akhavan, M. RobatJazi, and M. Reformat, “Validation and their kind help, especially for Chun-Hsun Chou, Ping-Chiang implementation of fuzzy models using FML-based specifications,” in Chou, Yi-Min Hsieh, Lu-An Lin, Shuji Takemura, and Minoru Proceeding of 2016 IEEE International Conference on Fuzzy Systems Ueda. The authors also thank Dr. Yuandong Tian and Yan Zhu (FUZZ-IEEE 2016), Vancouver, Canada, Jul. 24-29, 2014, pp. 1088–1095. for FAIR opening darkforest Go engine. Finally, the authors [18] C. S. Lee, M. H. Wang, S. C. Yang, P. H. Hung, S. W. Lin, N. Shuo, N. would like to thank National Center of High Performance Kubota, C. H. Chou, P. C. Chou, and C. H. Kao, “FML-based dynamic assessment agent for human-machine cooperative system on game of Go,” Computing (NCHC), especially for Sheng-Hsien Chen, and Internationl Journal of Uncertainty, Fuzziness & KnowledgeBased Kaohsiung City Government, Taiwan for computing resource Systems (IJUFKS), Jan. 2017. (Submitted Version). and technical support. [19] C. S. Lee, M. H. Wang, S. J. Yen, T. H. Wei, I. C. Wu, P. C. Chou, C. H. Chou, M. W. Wang, and T. H. Yang, “Human vs. computer Go: review REFERENCES and prospect,” IEEE Computational Intelligence Magazine, vol. 11, no. 3, [1] Wikipedia, “Darkforest,” Feb. 2017, [Online] Available: pp. 67-72, Aug. 2016. https://en.wikipedia.org/wiki/Darkforest. [20] R. Coulom. “Go ratings,” Feb. 2017, [Online] Available: [2] Go game, Nov. 2016, [Online] Available: https://www.goratings.org/. https://en.wikipedia.org/wiki/Go_(game).