A Q-Learning Approach to Developing an Automated Neural Computer Player for the Board Game of CLUE
Chenghui Cai and Silvia Ferrari
Abstract— The detective board game of CLUE can be minimizing the cost associated with the use of sensor and viewed as a benchmark example of the treasure hunt problem, platform resources, such as, energy and distance. in which a sensor path is planned based on the expected The board game of CLUE is a detective game with the value of information gathered from targets along the path. The sensor is viewed as an information gathering agent that makes objective of inferring three hidden cards, representing the imperfect measurements or observations from the targets, and guilty suspect, the weapon, and the room of an imaginary uses them to infer one or more hidden variables (such as, murder. The pawns navigate the game board in order to visit target features or classification). The treasure hunt problem the rooms of the CLUE mansion, where they can make arises in many modern surveillance systems, such as demining suggestions and gather information about the hidden cards, and reconnaissance robotic sensors. Also, it arises in the board game of CLUE, where pawns must visit the rooms which are viewed as the treasure. Similarly to sensor systems, of a mansion to gather information from which the hidden the information gathered by each player depends on the cards can be inferred. In this paper, Q-Learning is used to location of his/her pawn, because a suggestion must always develop an automated neural computer player that plans the include the room that is presently occupied by the pawn. path of its pawn, makes suggestions about the hidden cards, Also, the distance traveled by the pawn must be minimized and infers the answer, often winning the game. A neural network is trained to approximate the decision-value function in order to minimize the number of turns needed to infer representing the value of information, for which there exists the hidden cards and beat the adversaries. Recently, several no general closed-form representation. Bayesian inference, test authors have demonstrated that computer games are useful (suggestions), and action (motion) decision making are unified for developing and demonstrating computational intelligence using an MDP framework. The resulting computer player is methods and algorithms [5], [6], [7], [8], [9], [10]. This paper shown to outperform other computer players implementing Bayesian networks, or constraint satisfaction. uses the game of CLUE to illustrate the key challenges and characteristics of the treasure hunt problem, and develops a methodology that can be extended to sensor path planning I. INTRODUCTION research. A graphical-user-interface and an interactive simu- HE game of CLUE is a benchmark example for the lation of the game were developed by the authors in [11] to T treasure hunt, which is a fundamental problem that enable a Bayesian network (BN) intelligent computer player arises in many modern sensor and surveillance systems to compete against human players, and to test decision- comprised of sensors installed on mobile platforms. The making strategies that have been developed for robotic sensor treasure hunt problem is a coupled problem of robot path systems [3], [12], [13]. planning and inference that can be approached using pre- The approach presented in this paper is based on a Markov posterior decision analysis. Since the sensors are installed decision process (MDP) [14] representation of the game, on mobile platforms, their path must be planned for the which can be utilized to obtain a policy that optimizes the purpose of gathering sensor measurements and infer one or expected sum of a discounted reward, representing the value more hidden variables about the targets. Since the targets are of information minus its cost. The main difficulties associated distributed throughout the workspace, the path determines with MDPs are that the transition and reward functions may what measurements can be obtained by the sensor. Exam- be unknown, and that large state and/or decision spaces ples of relevant applications include landmine detection and may be intractable due to the curse of dimensionality [14]. classification by sensors installed on ground robots [1], [2], Reinforcement learning is able to obtain a decision-value [3]. These applications constitute a shift with respect to the function by using observed rewards to learn an optimal or traditional paradigm of a sensor that is employed in order nearly optimal policy from data. Function approximation by to enable robot motion planning [4]. Thus, new decision neural networks (NNs) and sampling techniques may be and control techniques need to be developed to optimize combined to deal with continuous state spaces [15]. Since the overall system performance by planning sensing or the game of CLUE presents both of these difficulties, in information-gathering strategies that maximize the expected this paper Q-Learning [16] [15] is used to learn the reward benefit of information of the measurement sequence, while function, and a neural network is used to represent the Q function. The approach presented in this paper is related C. Cai is a graduate student of Mechanical Engineering at Duke Uni- to [17], where a NN is trained by supervised learning to versity, Durham, NC, 27708, USA [email protected]. S. Ferrari is with the evaluate board positions with hand-labeled data in the game Faculty of Mechanical Engineering at Duke University, Durham, NC, 27708, USA [email protected]. of Backgammon. Also, in [18], an approach is presented for CLUE & c 2006 Hasbro, Inc. Used with permission. training a NN in Backgammon using time consuming self-
2347 978-1-4244-1821-3/08/$25.00c 2008 IEEE play with temporal difference. own deck. Clearly, if nobody can disprove the suggestion, The Q-Learning approach presented in this paper employs then the suggested cards are the hidden ones. When a player next state estimation to avoid learning from a large number is confident in inferring the hidden cards, he/she can make of real game trials, and save computation time. The current an accusation. If the accusation is correct, the player wins state estimation is based on probabilistic inference by BNs. the game, otherwise he/she loses and exits the game. The CLUE BN developed by authors in [11] is implemented not only to infer the hidden cards, but also to estimate the MDP state at the current time step, based on the latest available evidence obtained from the game. In fact, the posterior probability mass function (PMF) of the hidden room cards obtained by BN inference is viewed as the MDP state to satisfy the Markov property, and, together with the action (motion) decision, is used as the input to the NN approximation of the Q function. The paper is organized as follows. Section II reviews the background and the CLUE game rules. Section III presents the methodology used to develop the neural computer player and, in Section IV, the player is tested against other computer players of CLUE developed by the authors, which, to the best of our knowledge, are the only automated computer games that have been developed for this game. Fig. 1. CLUE game board. CLUE & c 2006 Hasbro, Inc. Used with II. BACKGROUND permission. A. The Game of CLUE B. Review of Markov Decision Processes (MDPs) The CLUE mansion includes nine rooms, the dining M {S, A,T,R} room, library, billiard room, hall, kitchen, lounge, ballroom, An MDP is a tuple = representing a S {x ,...,x } study and conservatory, and connecting hallways, as illus- sequential decision model. = 1 n is a finite A trated on the game board in Fig. 1. The six suspects, Col. set of possible state values, called the state space. = {a ,...,a } T Mustard, Miss Scarlet, Prof. Plum, Mr. Green, Mrs. White 1 m is the set of feasible action decisions. is the T S×A→P S and Mrs. Peacock are represented by pawns and can use any transition probability function : ( ),which of the six weapons, a knife, rope, candlestick, lead pipe, describes the MDP state transitions, such that whenever the k X x U revolver, and wrench, to commit the murder. Since each state at time has value k = i and the decision is k = a P X x | X x ,U a suspect, weapon, and room is represented by an illustrated j, there is a probability ( k+1 = l k = i k = j) X x card, there are a total of twenty-one cards in the deck. Three that the next state value is k+1 = l. The reward function, R S×A → R hidden cards representing the murder weapon, room, and : , specifies the value of the immediate r R X ,U guilty suspect are selected randomly and removed from the reward, k = ( k k), received after executing the action U X deck at the beginning of the game. The remaining cards are decision k in state k. A policy is a mapping of state values π S→A V π X then dealt to the players. During the game, players move their to actions, : . Let the value function ( k) denote the expected discounted return of policy π pawns from room to room and, upon entering a room, make a “suggestion” about the hidden cards, which must include ∞ π i the room they presently occupy. There are three ways for a V (Xk)=E γ rk+i | π, Xk (1) pawn to enter a room: (i) through one of the doors, (ii) via a i=0 secret passage (Fig. 1), or (iii) by another player’s suggestion. Where, rk+i is the reward received at i steps into future, and Both (i) and (ii) are decided by the player, whereas (iii) is the discount factor 0 ≤ γ<1 modulates the effect of future beyond the player’s control. For instance, the player enters rewards on present decisions, with small values emphasizing the ballroom and makes a “suggestion” represented by a set near-term gain and larger values emphasizing later rewards. ∗ π of {Miss Scarlet, knife, ballroom}, which shows that at that Then, an optimal policy π is one that maximizes V (Xk) time, he/she believes Miss Scarlet commits the imaginary for all possible states Xk ∈S. The Markov property murder with knife in the ballroom. guarantees that an optimal policy exists, though it may not Players gather information about the adversaries’ cards by be unique, and, thus, it is associated with an optimal value ∗ π making suggestions from which the three hidden cards are function V (Xk)=maxπ V (Xk). The optimal policy of an inferred. Every suggestion by player i must be disproved by MDP, M, is a fixed point of the Bellman’s equation, which player (i+1) by showing player i one of the suggested cards. can be determined iteratively using policy iteration or value If player (i+1) has none of the suggested cards, it is the turn iteration algorithms [19]. of player (i +2) to disprove the suggestion by player i,and In value iteration, the value of a state V (Xk) is the total so on, until someone can disprove it using a card from their expected discounted reward accrued by a policy starting at
2348 2008 International Joint Conference on Neural Networks (IJCNN 2008) X ∈S Q Q X ,U k .The function of a state-action pair, ( k k), U & +LGGHQ&DUGV V is the total expected discounted reward accrued by a policy & Z that produces Uk = π(Xk) [19]. The Bellman equation can & U &DUGVRI & 6 V 3OD\HU be formulated in terms of the aforementioned functions, such & U that the state-action value function is & Q X ,U E {R X ,U γV X } ( k k)= ( k k)+ ( k+1) (2) & V & Z U : & V (Xk+1)= max Q(Xk+1,Uk+1). (3) Uk+1∈A &DUGVRI Z & & U 3OD\HU If two functions Q(·) and V (·) satisfy the above Bellman equation, then they specify an optimal greedy policy & U 5 ∗ π (Xk)=argmaxQ(Xk,Uk) (4) Uk ∈A Value-iteration algorithms use eq. (2) to iteratively determine Fig. 2. Architecture of CLUE BN model, taken from [11] Q(·) and V (·) and, subsequently, determine π∗(·). Value iteration can be used to determine the optimal policy of an MDP, M, provided the transition probability function from which to infer the hidden cards. The architecture of T is known. If T is unavailable, Q-Learning can be utilized to the CLUE BN is shown in Fig. 2. For simplicity, it is learn an approximate state-action value function Q(Xk,Uk) assumed that there are three players in the game, but the same that is iteratively updated by the rule, approach can easily be extended to any number of players. The hidden cards are represented by the guilty suspect node Q(Xk,Uk) ← (1 − α)Q(Xk,Uk) S, the murder weapon node W , and the murder room node + α[rk + γ max Q(Xk+1,Uk)] (5) R Uk ∈A . The remaining eighteen cards are randomly dealt to the players, such that every time a card is dealt, it influences where, α is the learning rate, and 0 <α≤ 1. th the cards that are dealt later. Node Cj,i denotes the j card C. Review of Bayesian Network Inference in CLUE in the deck, dealt to player ith, and belonging to category s, w, r s w A Bayesian network is comprised of a directed acyclic = ,where, denotes the suspect category, the r graph (DAG) and a parameter structure that can be obtained weapon category, and the room category. Thus, the possible {s s from expert knowledge or learned from data [20], [21]. The values of the suspect cards are 1 (Col. Mustard), 2 (Ms. s s s s BN nodes XC = {X1,...,XN } represent discrete and ran- Scarlet), 3 (Prof. Plum), 4 (Mr. Green), 5 (Mrs. White), 6 (Mrs. Peacock) }, those of the weapon cards are {w1 (knife), dom variables that each have a finite range {xi,1,...,xi,ni }, th w w w w where xi,j denotes the j state value or instantiation of 2 (rope), 3 (candlestick), 4 (lead pipe), 5 (revolver), w } {r variable Xi. The union of the ranges of all nodes in XC 6 (wrench) , and those of the room cards are 1 (dining r r r r is the state space of the BN. The BN parameters are called room), 2 (library), 3 (billiard room), 4 (hall), 5 (kitchen), r r r r } conditional probability tables (CPTs). Each CPT is attached 6 (lounge), 7 (ballroom), 8 (study), 9 (conservatory) . to a node, Xi ∈ XC , and lists in tabular form the conditional Thus, the CPTs of the BN model in Fig. 2 can be obtained probability mass function (PMF), P(Xi | pa(Xi)],where using basic probability theory, as explained in [11]. pa(Xi) denotes the set of parents of Xi. If in the BN DAG At every player turn, indexed by a subscript k,new there is an arc from Xj to Xi, node Xj is said to be a parent evidence may become available about the adversaries’ cards. of Xi,andXi is said to be a child of Xj [20]. After a BN Information about BN nodes is referred to as hard evidence model is determined, it specifies the joint PMF of XC in when it refers to exact knowledge of a variable’s instanti- terms of the recursive factorization, ation. It is referred to as soft evidence, when it refers to an observed probability distribution based, for example, on P X ≡ P X ,...,X P X | pa X ( C ) ( 1 N )= ( i ( i)) (6) the negation of one or more values in a variable’s range. Xi∈XC Both types of evidence are obtained during the game of which is represented by the BN architecture. The above BN CLUE , and are organized into an evidence table, Ek,thatis factorization is convenient for inferring any node(s) in XC updated at every turn. Ek contains the observed probability from information or evidence about the other nodes in XC. before turn k for every possible value of every node card In particular, a junction-tree BN inference algorithm [22] can Cj,i in the CLUE BN (see [11] for more details). For be used to efficiently compute the posterior PMF of one or example, if player 2 shows Mrs. Peacock’s card to the more hidden nodes from the BN CPTs in (6). neural computer player, then the entry corresponding to the s In [11], the authors presented a BN approach to automating first (uninstantiated) suspect card C3,2 and s6 is assigned s suggestions in CLUE . A BN model of the relationships probability one, and all the other values of C3,2, s1 through between the cards is obtained by representing every card s5, are assigned probability zero. The neural computer player in the network by a BN node, and by using the outcomes developed in this paper utilizes the CLUE BN to infer the of the suggestions obtained during the game as evidence hidden cards and develop suggestions at every one of its
2008 International Joint Conference on Neural Networks (IJCNN 2008) 2349 turns, based on the latest evidence table. In the neural game player architecture (Fig. 3), the con- The next section presents the design of a neural computer tinuous state Xk is inferred from the CLUE BN using a player that uses an MDP representation of the game and Q- junction-tree inference algorithm (Section II-C). Xk is fed to Learning to determine an optimal policy for deciding pawn the NN approximator to estimate the Q function and output movements and suggestions. Qˆ(Xk,Uk). The MDP state is augmented by the pawn’s 2 position qk ∈ R in the game board. The distance between III. DEVELOPMENT OF MDP-BASED NEURAL qk and any room ri is defined as the minimum Manhattan COMPUTER PLAYER distance [23]. Since the distance traveled to ri is the cost of A. Design Overview the decision ai, the distance is denoted by D(qk,Uk = ai) The problem of deciding pawn’s movement and sugges- and is subtracted from the value of information Qˆ(Xk,Uk) tions in the game of CLUE is formulated as an MDP in (Fig. 3). An additional cost, N(Uk = ai) is used to which the reward to be maximized is the value of information measure the number of times the player enters room ri in obtained by visiting the mansion’s rooms, minus the distance order to limit visits to the same room. The total reward traveled by the pawn. Since the room occupied by the pawn function R(Xk,Uk,qk) is defined as a weighted function must be included in the suggestion, the pawn’s path deter- of Qˆ(Xk,Uk), D(qk,Uk) and N(Uk). Then, the optimal ∗ mines the information that will be gathered over one or more policy Uk = π (Xk,qk) determines which room the pawn turns by the computer player. To satisfy the Markov property, must move to next, given its position qk and the posterior the MDP state is formulated using posterior probabilities of PMF of the hidden room card Xk. A simple path planner the hidden room card. The architecture of the MDP-based implements the decision and moves the neural player’s pawn neural computer player is shown in Fig. 3, where the design to the chosen room. Subsequently, the neural player makes blocks are named based on their functionalities. a suggestion that includes the room occupied, and a weapon and suspect chosen by means of the CLUE BN. The 9DOXH)XQFWLRQ$SSUR[LPDWLRQ objective is to obtain the best possible evidence about the 3ODQW 6WDWH VWDWH ; hidden cards, by moving to rooms that offer the optimal 6DPSOLQJ 4)XQFWLRQ tradeoff between distance traveled and value of information. 1H[W6WDWH (VWLPDWRU UHLQIRUFHPHQW GDWD 4 ;8 B. Value of Information Function Approximation GHFLVLRQ 8 VXSHUYLVHG The value of information is the expected utility of perform- 1HXUDO*DPH3OD\HU SOD\HUSRVLWLRQ ' T8 'LVWDQFH ing a test or observation that may be used to infer hidden T )XQFWLRQ variables in a stochastic process. This terminology was first ¼ 8 4)XQFWLRQ %D\HVLDQ ; 4 ;8 5 ;8T *UHHG\'HFLVLRQ $SSUR[LPDWRU introduced by Howard in [24], who used utilities to guide 1HWZRUN 6HOHFWRU 1HXUDO1HWZRUN 1 8 the test selection in partially-observable MDPs. Although 2SWLPDO'HFLVLRQ several functions have been proposed for assessing the value HYLGHQFH( $FFXPXODWRU S ;T 0RYHPHQW of information in dual control [25], and sensor planning [13], &RQWUROOHU the best functional representation often cannot be determined apriori. Therefore, in this section, a NN is trained to Fig. 3. Architecture of MDP-based Neural Computer Player approximate the Q function associated with the value of information from the data obtained via Q-Learning. The Let each player be denoted by Pi,wherei represents objective is to estimate the reward associated with making the turn sequence, and assume the neural computer player a suggestion that includes room ri prior to visiting ri.The is P1. The set of feasible action decisions for P1 is A = utility of the suggestion is that its outcome may be used for {a1,...,a9},whereai ∈ A denotes the decision of entering inference, while its cost is the distance traveled to the room. room ri. Let the MDP state Xk be defined as the posterior The main drawback of Q-Learning is that its convergence T PMF P(R | Ek), such that Xk =[xk,1 ··· xk,9] , speed may be too low due to the large size of the state space where xk,i ≡ P(R = ri | Ek). Then, the Markov created by complex or multiple goal tasks. For instance, property P(Xk+1 | Xk,Uk,Xk−1,Uk−1,...,X0,U0)= learning the Q function for CLUE may require thousands 9 P(Xk+1 | Xk,Uk) holds, and i=1 xk,i = i P(R = of learning trials (where, every trial is a game) before conver- ri | Ek)=1for ∀k. Although time is discrete, the MDP gence to an optimal state-action value function is achieved. state is continuous, with S⊂[0, 1]9. Therefore, as illustrated Every simulated CLUE game takes approximately 10 to in Fig. 3, a finite subspace of n possible states, Sn,isfirst 20 minutes. Thus, a next-state estimator is presented that sampled from the continuous state space, i.e., Sn ⊂S. Then, estimates Xk by mimicking the outcome of a decision Uk, a next-state estimator is developed to estimate the next state leading to significant savings in learning times. Suppose P1 Xk+1 from the executed action Uk and current state Xk,as is in state Xk ∈Sn and makes a decision Uk = ai to enter required by Q-Learning. Equation (5) is utilized to obtain room ri and make a suggestion Gk = {SG,WG,RG = ri}, the Q function over the domain Sn ×A, which can be used where the subscript G denotes the suggested card. Then, there to train the corresponding neural network, as explained in are four possible outcomes: (i) Gk cannot be disproved and Section III-B. the value of Xk indicates that P(R = ri | Ek)=0, thus
2350 2008 International Joint Conference on Neural Networks (IJCNN 2008) TABLE I P1 has ri and Xk+1 = Xk; (ii) Gk cannot be disproved Q-LEARNING ALGORITHM and the value of Xk indicates that P(R = ri | Ek) > 0, P r X P R thus 1 does not have i,andthenewstate k+1 is ( = Inputs: Sn, A,R(·), 0 ≤ γ<1, 0 <α≤ 1 ri | Ek+1)=1and P(R = rj,j = i | Ek+1)=0; (iii) Outputs: Q(·) Q · one of the adversaries shows a suspect or weapon card to initialize ( ) to be a zero matrix initialize Q1(·) to be a matrix with elements →∞ disprove G, thus Xk+1 = Xk; (iv) one of the adversaries initialize learning convergence counter count ← 0 shows the room card ri to disprove G, thus Xk+1 changes for each epoch randomly select current state Xk from Sn to include P(R = ri | Ek+1)=0. In this case, the system randomly select Uk from {Uk | R(Xk,Uk) ≥ 0} R ri state is updated by eq. (7). The fact that = is the new Xˆk+1 ← next−state−estimator(Xk ,Uk, Sn) evidence obtained at turn k,andthenEk+1 = {Ek,R= ri}. update Q(Xk,Uk) using eq. (5) Also, P(R = ri | R = rj,Ek)=1,j = i. Eq. (7) can be /* Check convergence /* if |Q − Q1|
2008 International Joint Conference on Neural Networks (IJCNN 2008) 2351 25 24 a room during his/her turn, the player makes a suggestion 23 102 103 22 68 and waits for other players’ responses. If the player’s pawn
21 69 100 97 98 99 is taken into a room by another player, he/she must wait for 20 101 94 95 96 19 91 92 93 61 64 67 his/her turn to make a suggestion. 18 89 90 74 73 72 71 70 57 58 60 63 66 17 87 88 55 56 59 62 65 An accusation can be made at the beginning or at the 16 85 86 15 end of a player’s turn. Each player can see his/her cards, 82 83 84 53 54 14 81 78 79 80 13 and his/her decisions are observed and recorded by the other 51 12 75 76 77 52 11 50 players during the game via the interfaces developed in
10 18 19 21 23 25 27 29 30 MATLAB . He/she can also keep a record of all suggestions 9 16 17 20 22 24 26 28 31 32 33 49 8 9 11 13 15 34 35 36 48 that take place during the game, and of how they are 7 8 10 12 14 37 38 39 46 47 6 675 40 41 disproved, gathering evidence even during the adversaries’ 5 4 turns. The neural computer player is tested by letting it 4 3 42 43 3
2 compete against a human player, a BN computer player 2 44 1 1 45 developed by the authors in [11], a constraint satisfaction 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 player (CSP), and a random computer player.
Fig. 4. Convex polygonal decomposition (CPD) of the CLUE workspace, A. Competing BN and CSP Computer Players where cells of a door to room are shown in grey, other cells in white, and obstacles in black. The BN player computer player presented in [11] imple- ments the same BN model used in this paper to infer the posterior PMFs P(S | Ek), P(W | Ek),andP(R | Ek) connectivity graph in which each node represents a polygon at every turn k. If any of these posterior probabilities are that is obstacle free and decomposable into the square bins greater than or equal to 0.9 the BN computer player makes illustrated on the board. The connectivity graph is searched an accusation, otherwise, it rolls the die and moves its pawn. ∗ by the A algorithm to determine the shortest path from qk The pawn’s path planning is performed based on a heuristic ∗ to ri,i =argπ (Xk,qk). After rolling the dice, the neural rule, such that if P(R = rj | Ek) ≥ 0.9 for some room value computer player moves its pawn along the shortest path, from rj , then the pawn enters room rj as often as possible, and bin to bin in the two-dimensional grid illustrated on the game makes a suggestion by the same rule used by the MDP-based board (Fig. 1), until the dice value is met. neural player (Section III-C). Once the neural computer player, P1, enters room ri,say The constraint satisfaction problem (CSP) player imple- at turn (k + t), the BN model described in Section II-C ments the approach reviewed in [19, Chapter 5], with two s w r is used to compute the posterior PMFs P(S | Ek+t) and types of constraints: (C1) S = Cji,W = Cji and R = Cji, P W | E ( k+t) or, in other words, to infer the hidden cards for ∀ i, j, and (C2) Cji = Cnm,for = s, w, r, i, m =1, 2, S and W based on the latest available evidence. Then, a j, n =1, 2, 3,andi = m or j = n. When the CSP th suggestion G = {SG,WG,RG = ri} is produced according player makes a suggestion and the i player disproves it, an s to the following heuristic rule: if there is a suspect card j instantiation Cji = cji,l is obtained for some j and included with 0 < P(S = sj | Ek+t) < 0.9,thenSG = sj,otherwise in constraint (C2). Then, in order to make a suggestion, the SG is set equal to a suspect card owned by P1;ifthereisa CSP searches an assignment for S, W ,andR that satisfies weapon card wl with 0 < P(W = wl | Ek+t) < 0.9,then (C1)-(C2). Based on a heuristic strategy recommended by WG = wl,otherwiseWG is set equal to a weapon card owned the CLUE game rules, the CSP pawn moves to the nearest by P1. At the end of its turn, when G has been disproved room to begin making suggestions. Then, it moves back and ∗ ∗ ∗ by an adversary, P1 makes an accusation {si ,wj ,rl } only forth between two adjacent rooms to determine S, W ,and ∗ ∗ if P(S = si | Ek+t+1), P(W = wj | Ek+t+1),and subsequently navigates the board to determine R. ∗ P(R = rl | Ek+t+1) are all greater than or equal to 0.9, The random computer player performs random moves and otherwise the game continues. suggestions and, thus, is used as a place holder that hardly ever wins the game. Since the NN and BN models are trained IV. GAME SIMULATION AND PERFORMANCE before the game playing, decision making at every turn by COMPARISON RESULTS any of NN, BN, CSP and random computer players is very An interactive computer simulation of CLUE has been fast with the time less than several seconds on a Pentium 4 developed by the authors in [11] using the MATLAB CPU 3.06 GHz computer. Graphical User Interface (GUI) toolbox. The simulated game consists of three phases: the players choose pawns, the B. Simulated Games Results computer deals the cards, and the players start the game. For The efficiency of the BN and CSP players were tested simplicity, in this paper, the neural computer player plays by making both of them compete against an impartial against two adversaries who may be computer or human human player who is not familiar with the simulation or players. Every player’s turn consists of three steps: roll the the computer players designs. As shown in Table II, the die, move the pawn a distance no greater than the die number, results obtained from 25 games demonstrate that the BN and transfer the turn to next player. If a player’s pawn enters player performs almost as well as the human player, and the
2352 2008 International Joint Conference on Neural Networks (IJCNN 2008) former outperform the CSP player, which wins only 20% players of CLUE obtained by Bayesian networks and of the times. As shown in Table III, when the MDP-based constraint satisfaction approaches, which do not account for neural computer player competes against the CSP and the the value of information in deciding the pawn’s motion. random computer players it wins 75% of the times. Instead, VI. ACKNOWLEDGMENTS when the neural player competes against the BN and the random computer player, it wins 65% of the times (Table CLUE & c 2006 Hasbro, Inc. Used with permission. IV). Thus, it can be concluded that the neural player is the REFERENCES most effective of the computer players. The reason is that [1] J. MacDonald, et al., Alternatives for Landmine Detection. RAND, the other computer players do not account for the value of Science and Technology Policy Institute, 2003. information and the game evidence in deciding the pawn’s [2] R. Popoli, “The Sensor Management Imperative,” Chapter 10 in motion. Since the BN player performs almost as well as the Multitarget-Multisensor Tracking: Advanced Applications, Vol. II, Y. Bar-Shalom, Ed., Artech House, 1992. human player (Table II) the neural player is likely to perform [3] S. Ferrari and A. Vaghi, “Demining Sensor Modeling and Feature-level as well or, perhaps, better than human players. This test will Fusion by Bayesian Networks,” IEEE Sensors Journal, vol. 6, pp. 471- be the subject of future research. 483, 2006. [4] J. C. Latombe, Robot Motion Planning. Kluwer Academic Publishers, 1991. TABLE II [5] J.E. Laird, “Using a Computer Game to Develop Advanced AI,” GAME RESULTS OF BN, HUMAN AND CSP PLAYERS PLAYING TOGETHER Computer, vol. 34, no. 7, pp. 70-75, July 2001. [6] N. Baba and L. C. Jain, Computational Intelligence in Games. Heidel- berg: Physica-Verlag, 2001. [7] X. Z. Pang and P. J. Werbos, “Neural Network Design for J Function Winning Player: BN Player Human Player CSP Approximation in Dynamic Programming,” Math. Modelling and Sci- Winning Times / entific Computing (a Principia Scientia journal), vol. 5, no. 2/3, 1996. Total Playing Times 9/25 11 / 25 5/25 [8] K. Chellapilla and D. Fogel, “Evolution, neural networks, games, and intelligence,” Proc. of the IEEE, vol. 87, no. 9, pp. 1471-1496, Winning Rate 36.0% 44.0% 20.0% September, 1999. [9] D. Fogel, T. J. Hays, S. L. Hahn and J. Quon, “A self-learning evolutionary Chess program,” Proc. of the IEEE, vol. 92, no. 12, pp. TABLE III 1947-1954, December, 2004. [10] X. Cai and D. C. Wunsch II, “Computer Go: A grand challenge to GAME RESULTS OF NEURAL PLAYER PLAYING AGAINST CSP PLAYER AI,” in Studies in Computational Intelligence, W. Duch and J. Mandziuk Eds. Springer, vol. 63, pp. 443C465, 2007. [11] C. Cai and S. Ferrari, “On the Development of an Intelligent Computer Winning Player: Neural Player CSP Player for CLUE , the Great Detective Boardgame: A Case Study in Winning Times / Preposterior Decision Analysis,” Proc. of the 2006 American Control Conference, Minneapolis, MN, pp. 4350-4355, June 2006. Total Playing Times 15 / 20 5/20 [12] S. Ferrari, C. Cai, R. Fierro and B. Perteet, “A Multi-Objective Winning Rate 75.0% 25.0% Optimization Approach to Detecting and Tracking Dynamic Targets in Pursuit-Evasion Games,” Proc. of the 2007 American Control Confer- ence, New York, NY, pp. 5316-5321, July 2007. TABLE IV [13] C. Cai and S. Ferrari, “Comparison of Information-Theoretic Objective Functions for Decision Support in Sensor Systems,” Proc. of the 2007 GAME RESULTS OF NEURAL PLAYER PLAYING AGAINST CSP PLAYER American Control Conference, New York, NY, pp. 63-133, July 2007. [14] M. L. Puterman, Markov decision processes: discrete stochastic dy- namic programming. Wiley, New York, 1994. Winning Player: Neural Player BN Player [15] S. Rajneesh and G. Madan, “Game-theoretic-reinforcement-adaptive neural controller for nonlinear systems,” Proc. of the 2006 American Winning Times / Control Conference, Minneapolis, MN, pp. 2975-2980, June 2006. 12 / 20 8/20 Total Playing Times [16] M. L. Littman, “Markov Games as a framework for Multi-agent Winning Rate 60.0% 40.0% Reinforcement Learning”, Proc. of Eleventh International Conference on Machine Learning, Morgan Kaufman, pp. 157-163, 1994. [17] G. Tesauro, “eurogammon: a neural network Backgammon program,” Proc. of Int. Joint Conf. On NN, vol. 3, pages 33–39, 1990. V. C ONCLUSIONS [18] G. Tesauro, “Temporal difference learning and TD-Gammon,” Com- An MDP-based neural computer player is developed for munications of the ACM, vol. 38, no. 3, pp. 58-68, March 1995. Q [19] S. Russell and P. Norvig, Chapter 5, 17 and 21 in Artificial Intelligence: the board game of CLUE using a combination of - A Modern Approach. Prentice Hall, NJ, 2003. Learning and BN inference to estimate the value of infor- [20] F. V. Jensen, Chapter 1 in Bayesian Networks and Decision Graphs. mation. The objective of the neural player is to solve a Springer, 2001. [21] R. Cowell, “Introduction to inference for Bayesian networks,” in benchmark example of the treasure hunt problem, which Learning in Graphical Models. M. Jordan, Ed., the MIT press, pp. arises in several robotic sensor applications, such as demining 9-26, 1998 and reconnaissance. Sampling techniques and NN function [22] K. Murphy, How to Use Bayes Net Toolbox. [Online]. Available: http://www.cs.ubc.ca/ murphyk/Software/BNT/bnt.html, 2004. approximation are used to address the continuous state space, [23] E. F. Krause, Taxicab geometry: an adventure in non-Euclidean defined as the posterior PMF of the hidden variables to geometry. Dover, New York, 1986. maintain the Markov property. Thus, Bayesian inference [24] R. A. Howard, “Information Value Theory,” IEEE Transactions on Systems Science and Cybernetics, vol. 2, pp. 22-26, 1966. and action (motion) decision making can be unified by the [25] Y. Bar-Shalom and E. Tse, “Caution, Probing, and the Value of MDP framework. The simulated game statistics show that Information in the Control of Uncertain Systems,” Annals of Economic the neural computer player outperforms existing computer and Social Measurements, Vol. 5, No. 2, pp. 323-337, 1976.
2008 International Joint Conference on Neural Networks (IJCNN 2008) 2353