A Gamut of Games

AI Magazine Volume 22 Number 3 (2001) (© AAAI) Articles A Gamut of Games

Jonathan Schaeffer

In 1950, Claude Shannon published his seminal make this amazing feat possible. Often over- work on how to program a computer to play chess. looked, however, is that this result was also a Since then, developing game-playing programs testament to human abilities. Considering the that can compete with (and even exceed) the abil- formidable computing power that DEEP BLUE ities of the human world champions has been a used in its 1997 exhibition match against long-sought-after goal of the AI research commu- world chess champion Garry Kasparov nity. In Shannon’s time, it would have seemed (machine: 200,000,000 chess positions a sec- unlikely that only a scant 50 years would be need- ond; man: 2 a second), one can only admire ed to develop programs that play world-class backgammon, checkers, chess, Othello, and Scrab- the human champions for withstanding the ble. These remarkable achievements are the result technological onslaught for so long. of a better understanding of the problems being Computer game research was started by solved, major algorithmic insights, and tremen- some of the luminaries in computing science dous advances in hardware technology. Computer history. In 1950, Claude Shannon published games research is one of the important success sto- his seminal paper that laid out the framework ries of AI. This article reviews the past successes, for building high-performance game-playing current projects, and future research directions for programs (Shannon 1950). In 1951, Alan Tur- AI using computer games as a research test bed. ing (1953) did a hand simulation of his computer chess algorithm (a lack of resources pre- vented him from actually programming it); the algorithm lost to a weak human player. ames are ideal domains for exploring Around this time, Arthur Samuel began work the capabilities of computational intel- on his famous checkers-playing program, the Gligence. The rules are fixed, the scope of first program to achieve notable success the problem is constrained, and the interac- against human opposition (Samuel 1967, tions of the players are well defined. Contrast 1959). By 1958, Alan Newell and Herb Simon the game world with the real world—the game had begun their investigations into chess, of life—where the rules often change, the which eventually led to fundamental results scope of the problem is almost limitless, and for AI and cognitive science (Newell, Shaw, and the participants interact in an infinite number Simon 1958). An impressive lineup to say the of ways. Games can be a microcosm of the real least! world (for example, the role of game theory in In the half century since Shannon’s paper, economics, social interaction, and animal enormous progress has been made in con- behavior), and successfully achieving high structing high-performance game-playing pro- computer performance in a nontrivial game grams. In Shannon’s time, it would have can be a stepping stone toward solving more seemed unlikely that within a scant 50 years challenging real-world problems. checkers (8 8 draughts), Othello, and Scrab- Historically, games have been a popular ble programs would exist that exceed the abil- choice for demonstrating new research ideas in ities of the best human players,1,2 and back- AI. Indeed, one of the early goals of AI was to gammon and chess programs could play at a build a program capable of defeating the level comparable to the human world champi- human world chess champion in a match. This on. These remarkable accomplishments are the challenge proved to be more difficult than was result of a better understanding of the anticipated; the AI literature is replete with problems being solved, major algorithmic optimistic predictions. It eventually took insights, and tremendous advances in hard- almost 50 years to complete the task—a ware technology. The work on computer remarkably short time when one considers the games has been one of the most successful and software and hardware advances needed to visible results of AI research. For some games,

one could argue that the Turing test has been these enhancements include iterative deepen- passed (Krol 1999). ing, caching previously seen subtree results This article discusses the progress made in (transposition tables), successor reordering, developing programs for the classic board and search extensions and reductions, probabilistic card games. For a number of games, a short his- cutoffs, and parallel search. The results are tru- tory of the progress in building a world-class ly amazing. Even though there is an exponen- program for the game is given, along with a tial difference between the best case and the brief description of the strongest program. In worst case for an alpha-beta search, most high- each case, a single feature of the program that performance game-playing programs are Considering is a major contributor to the program’s searching within a small constant of the best strength is highlighted. The histories are neces- case (Plaat et al. 1996). the sarily brief. I apologize in advance to the many Sadly, the community of researchers in- formidable hard-working researchers and hobbyists whose volved in this work has done a relatively poor computing work is not mentioned here. job of selling the technology, resulting in many Enabling Technologies briefly summarizes of the ideas being reinvented for other power that some of the major advances in technology that domains. For example, many search tech- DEEP BLUE facilitated the construction of world-champi- niques pioneered with alpha-beta have become onship–caliber programs. Success in Computer standard in other search domains, with few used in its Games reports the past successes where comput- realizing the lineage of the ideas. 1997 ers have met or exceeded the best human play- At the heart of many game-playing programs ers (backgammon, checkers, chess, Othello, and is an evaluation function. Early on, game exhibition Scrabble). Current Research Efforts highlights developers quickly encountered the knowl- match against games of current academic interest (bridge, go, edge-acquisition bottleneck and traded quality world chess and poker). The Future of Computer Games dis- of knowledge for speed of the program. Simple cusses some of the future challenges of using evaluation functions, linear combinations of champion games as a research test bed for AI. easily identifiable features, were the mainstay Garry Although this article emphasizes the AI of computer game programs for many decades. viewpoint, one should not underestimate the Alternative approaches, such as modeling Kasparov engineering effort required to build these pro- human cognitive processes, turned out to be (machine: grams. One need only look at the recent suc- much harder to do than initially expected and cess of the DEEP BLUE chess machine to appreci- generally resulted in poor performance. Game 200,000,000 ate the effort required. This project spanned 8 programmers quickly learned that a little chess years (12 if one includes the pre-IBM time) and heuristic knowledge, when combined with positions a included several full-time people, extensive deep search, can produce amazing perfor- computing resources, chip design, and grand mance results. Indeed, one could argue that the second; man: master consultation. Some of the case studies viability of brute-force search, once a term with 2 a second), hint at the amount of work required to con- negative connotations in the AI community, is struct these systems. In all cases, the successes one of the main research results from games- one can only reported in this article are the result of consis- related research (Ginsberg 1996b). admire the tent progress over many years. In the last decade, new techniques have human moved to the forefront of games research. Two Enabling Technologies in particular are mentioned here because they champions for are likely to play a more prominent role in the withstanding The biggest advances in computer game play- near future: (1) Monte Carlo simulation and (2) ing have come as a result of work done on the temporal-difference learning. the alpha-beta search algorithm. This algorithm Monte Carlo simulation has successfully technological received the most attention because of the been applied to games with imperfect or non- research community’s preoccupation with deterministic information. In these games, it is onslaught for chess. With the DEEP BLUE victory over world too expensive to search all possible outcomes. so long. chess champion Garry Kasparov, interest in Instead, only a representative sample is chosen methods suitable for chess has waned and been to give a statistical proﬁle of the outcome. This replaced by activity in other games. One could technique has been successful in bridge, poker, argue that the chess victory removed a ball and and Scrabble. shackle that was stifling the creativity of Temporal-difference learning is the direct researchers who were building high-perfor- descendent of Samuel’s machine learning mance game-playing systems. research (Sutton 1988). Here, a database of The alpha-beta research led to a plethora of games (possibly generated by computer self- search enhancements, which significantly play) can be used to bootstrap a program to improved the efﬁciency of the search. Some of find a good combination of knowledge fea-

30 AI MAGAZINE Articles tures. The algorithm has successfully been ences” (Berliner 1980a, p. 215). Thus, assessing applied to backgammon and has recently the results of a five-game match is difficult. shown promise in chess and checkers (Schaef- Afterwards, Berliner (1980b, p. 71) analyzed fer, Hlynka, and Jussila 2001). the program’s play and concluded that The most obvious hardware advance is sim- There was no doubt that BKG9.8 played ply speed. To read about Samuel’s checkers- well, but down the line Villa played better. playing program running on a 1963 machine He made the technically correct plays that could execute 15 million additions a almost all the time, whereas the program minute (Pfeiffer 1964) starkly brings home the did not make the best play in eight out of Sadly, the point that orders of magnitude more comput- 73 non-forced situations. ing power makes many things possible. Indeed, community of BKG9.8 was an important first step, but major considering the paucity of computing power at work was still needed to bring the level of play researchers Samuel’s disposal, one can only be filled with up to that of the world’s best players. admiration at what he achieved. involved in In the late 1980s, IBM researcher Gerry Computer games research pioneered compe- Tesauro began work on a neural net–based this work has titions to assess the quality of the systems backgammon program. The net used encoded being developed. Since 1970, there have been done a backgammon knowledge and, training on data annual computer chess tournaments. There is relatively poor sets of games played by expert players, learned now an annual Computer Olympiad that the weights to assign to these pieces of knowl- job of selling brings together many of the top programs and edge. The program, NEUROGAMMON, was good their developers in head-to-head competition.3 the enough to win first place in the 1989 Comput- The competitive spirit has spread throughout er Olympiad (Tesauro 1989). technology, the AI community; competitions now exist for Tesauro’s next program, TD-GAMMON used a other applications, including theorem proving, resulting in neural network that was trained using tempo- planning, and natural language. ral difference learning. Instead of training the many of the program with data sets of games played by ideas being Success in Computer Games humans, Tesauro was successful in having the reinvented for program learn using the temporal differences In a number of games, computers have enjoyed from self-play games. The evolution in TD-GAM- other success that puts them on par or better with MON from version 0.0 to 3.0 saw an increase in the best humans in the world. In some sense, domains. For the knowledge used, a larger neural net, and these games are now the past, in that active the addition of small selective searches. The example, research to develop high-performance pro- resulting program is acknowledged to be on par grams for them is on the wane (or is now many search with the best players in the world and, possi- nonexistent). These include games where com- bly, even better. techniques puters are better than all humans (checkers, In 1998, an exhibition match was played Othello, Scrabble) and those where computers pioneered between world champion Malcolm Davis and are competitive with the human world cham- with alpha- TD-GAMMON 3.0 (at the AAAI-98 conference). To pion (backgammon, chess). reduce the luck factor, 100 games were played beta have Backgammon over 3 days. The final result was a narrow eight- become point win for Davis. Both Davis and Tesauro The first concerted effort at building a strong have done extensive analysis of the games, standard in backgammon program was undertaken by coming up with similar conclusions:4 Hans Berliner of Carnegie Mellon University. other search While this analysis isn’t definitive, it sug- In 1979, his program, BKG9.8, played an exhibi- domains, tion match against the then newly crowned gests that we may have witnessed a super- world champion Luigi Villa (Berliner 1980a, human level of performance by TD-GAM- with few 1980b). The stakes were $5,000, winner take MON, marred only by one horrible blunder realizing the all. The final score was seven points to one in redoubling to 8 in game 16, costing a lineage of the favor of the computer, with BKG9.8 winning whopping 0.9 points in equity and proba- four of the five games played (the rest of the bly the match! ideas. points came from the doubling cube). A notable feature of TD-GAMMON is its neural Backgammon is a game of both skill and net evaluation function. The net takes as input luck. In a short match, the dice can favor one the current board position and returns as out- player over another. Berliner writes that “in the put the score for the position (roughly, the short run, small percentage differences favor- probability of winning) (Tesauro 1995). The net ing one player are not too significant. However, has approximately 300 input values (Tesauro in the long run a few percentage points are 2001). The latest version, TD-GAMMON 3.0, con- highly indicative of significant skill differ- tains 160 hidden units. Each unit takes a linear

FALL 2001 31 Articles

sum of the weighted values of its input and 1990, the program earned the right to chal- then converts it to a value in the range –3 to 3 lenge for the human world championship. The (a backgammon is worth three points, a gam- checkers federations refused to sanction the mon two, and a win one). The conversion is match, leading to the creation of a new title: done with a sigmoid function, allowing the The World Man-Machine Championship. This output to be a nonlinear function of the input. title was contested for the first time in 1992, The resulting neural net has approximately with Marion Tinsley defeating CHINOOK in a 40- 50,000 weights that need to be trained. game match by a score of 4 wins to 2. CHINOOK’s The weights in the hidden units were trained wins were the first against a reigning world using temporal-difference learning from self- champion in a nonexhibition event for any play games. By playing the program against competitive game. itself, there was an endless supply of data for There was a rematch in 1994, but after six the program to train itself against. In a given games (all draws), Tinsley resigned the match game position, the program uses the neural net and the title to CHINOOK, citing health con- to evaluate each of the roughly 20 different cerns. The following week he was diagnosed ways it can play its dice roll and then chooses with cancer, and he died eight months later. the move leading to the maximum evaluation. CHINOOK has subsequently defended its title Each game is played to completion, and then twice and has not lost a game since 1994. The temporal-difference learning is applied to the program was retired from human competitions sequence of moves. Close to 1,500,000 self- in 1997 (Schaeffer 1997). play games were used for training TD-GAMMON The structure of CHINOOK is similar to that of 3.0. a typical chess program: search, knowledge, Tesauro’s success with temporal-difference database of opening moves, and endgame data- Tesauro’s learning in his backgammon program is a bases (Schaeffer 1997; Schaeffer et al. 1992). success with major milestone in AI research. CHINOOK uses alpha-beta search with a myriad of enhancements, including iterative deepen- temporal- Checkers ing, transposition table, move ordering, search difference Arthur Samuel began thinking about a check- extensions, and search reductions. CHINOOK was learning in his ers program in 1948 but did not start coding able to average a minimum of 19-ply searches until a few years later. He was not the first to against Tinsley (using 1994 hardware), with backgammon write a checkers-playing program; Christopher search extensions occasionally reaching 45 ply program is a Strachey (1952) predated him by a few months. into the tree. The median position evaluated Over the span of three decades, Samuel worked was typically 25-ply deep into the search. major steadily on his program, with performance tak- A notable feature in CHINOOK is its use of end- milestone in ing a back seat to his higher goal of creating a game databases. The databases contain all program that learned. Samuel’s checkers player checkers positions with 8 or fewer pieces, 444 AI research. is best known for its single win against Robert billion (4 1011) positions compressed into 6 Nealey in a 1963 exhibition match. From this gigabytes for real-time decompression. Unlike single game, many people erroneously con- chess programs, which are compute bound, cluded that checkers was a “solved’” game. CHINOOK becomes input-output bound after a In the late 1970s, a team of researchers at few moves in a game. The deep searches mean Duke University built a strong checkers-play- that the database is occasionally being hit on ing program that defeated Samuel’s program in the first move of a game. The databases intro- a short match (Truscott 1979). Early success duce accurate values (win/loss/draw) into the convinced the authors that their program was search (no error), reducing the program’s possibly one of the 10 best players in the world. dependency on its heuristic evaluation func- World champion Marion Tinsley effectively tion (small error). In many games, the program debunked that, writing that “the programs is able to back up a draw score to the root of a may indeed consider a lot of moves and posi- search within 10 moves by each side from the tions, but one thing is certain. They do not see start of a game, suggesting that it might be pos- much!’’ (Tinsley 1980). Efforts to arrange a sible to determine the game-theoretic value of match between the two went nowhere, and the the starting position of the game (one defini- Duke program was quietly retired. tion of “solving” the game). Interest in checkers was rekindled in 1989 CHINOOK is the first program to win a human with the advent of strong commercial pro- world championship for any game. At the time grams and a research effort at the University of of CHINOOK’s retirement, the gap between the Alberta—CHINOOK. CHINOOK was authored prin- program and the highest-rated human was 200 cipally by Jonathan Schaeffer, Norman Treloar, rating points (using the chess rating scale) Robert Lake, Paul Lu, and Martin Bryant. In (Schaeffer 1997). A gap this large means that

32 AI MAGAZINE Articles the program would score 75 percent of the pos- liant win in game two, handing Kasparov a sible points in a match against the human psychological blow from which he never recov- world champion. Since then, faster processor ered. In the final, decisive game of the match, speeds mean that CHINOOK has become Kasparov fell into a trap, and the game ended stronger, further widening the gap between quickly, giving DEEP BLUE an unexpected match man and machine. victory, scoring two wins, three draws, and a loss. Chess It is important to keep this result in perspec- The progress of computer chess was strongly tive. First, it was an exhibition match; DEEP BLUE influenced by an article by Ken Thompson that did not earn the right to play Kasparov.5 Sec- equated search depth with chess-program per- ond, the match was too short to accurately formance (Thompson 1982). Basically, the determine the better player; world-champi- paper presented a formula for success: Build onship matches have varied from 16 to 48 faster chess search engines. The milestones in games in length. Although it is not clear just chess program development become a state- how good DEEP BLUE is, there is no doubt that ment of the state of the art in high-perfor- the program is a strong grand master. mance computing: What does the research community think of 1978–1980: The pioneering programs from the DEEP BLUE result? Many are filled with admi- Northwestern University, most notably CHESS ration at this feat of engineering. Some are cau- 4.6 (Slate and Atkin 1977), ran on a top-of-the- tious about the significance. John McCarthy line Control Data computer and achieved the (1997) wrote that “in 1965, the Russian math- first major tournament successes. ematician Alexander Kronrod said, ‘Chess is 1980–1982: BELLE, the first program to earn a the Drosophila of artificial intelligence.’ Howev- U.S. master title, was a machine built to play er, computer chess has developed much as chess. It consisted of 10 large wire-wrapped genetics might have if the geneticists had con- boards using LSI chips (Condon and Thomp- centrated their efforts starting in 1910 on son 1982). breeding racing Drosophila. We would have 1983–1984: CRAY BLITZ used a multiprocessor some science, but mainly we would have very Cray supercomputer (Hyatt, Gower, and Nel- fast fruit flies.”6 son 1990). In retrospect, the chess “problem” turned 1985–1986: The HITECH chess machine was out to be much harder than was expected by based on 64 special-purpose VLSI chips (one to the computing pioneers. The DEEP BLUE result is a board square) (Ebeling 1987; Berliner and a tremendous achievement, and a milestone in Ebeling 1989). the history of computing science. 1985–1986: WAYCOOL used a 256-processor From the scientific point of view, it is to be hypercube (Felten and Otton 1988). regretted that DEEP BLUE has been retired, the 1987–present: CHIPTEST (and its successors hardware unused, and the programming team DEEP THOUGHT and DEEP BLUE) took VLSI technol- disbanded. The scientific community has a sin- ogy even further to come up with a full-board gle data point that suggests machine might be chess chip (Hsu 1999; Hsu et al. 1990a, 1990b). better than man at chess. The data are insuffi- In 1987, CHIPTEST shocked the chess world cient, and the sample size is not statistically sig- by tying for first place in a strong tournament, nificant. Moreover, given the lack of interest in finishing ahead of a former world champion DEEP BLUE from IBM, it is doubtful that this and defeating a grand master. The unexpected experiment will ever be repeated. Of what val- success aroused the interest of world champion ue is a single, nonrepeatable data point? Garry Kasparov, who played a two-game exhi- DEEP BLUE and its predecessors represent a bition match against the program in 1989. decade-long intensive effort by a team of peo- Man easily defeated machine in both games. ple. The project was funded by IBM, and the The DEEP BLUE team worked for seven years to principal scientists who developed the program improve the program, including designing a were Feng-Hsiung Hsu, Murray Campbell, and single-chip chess search engine and making Joe Hoane. significant strides in the quality of their soft- The notable technological feature of DEEP ware. In 1996, the chess machine played a six- BLUE is its amazing speed, the result of building game exhibition match against Kasparov. The special-purpose chess chips. The chip includes world champion was stunned by a defeat in the a search engine, a move generator, and an eval- first game, but he recovered to win the match, uation function (Cambell, Hoane, and Hsu scoring three wins and two draws to offset the 2001; Hsu 1999). The chip’s search algorithm is single loss. The following year, another exhibi- based on alpha-beta. The evaluation function is tion match was played. DEEP BLUE scored a bril- implemented as small tables on the chip; the

FALL 2001 33 Articles

values for these tables can be downloaded to tion. Bayesian learning was used to combine the chip before the search begins. These tables the evaluation-function features in a weighted are indexed by board features and the results quadratic polynomial. summed in parallel to provide the positional Statistical analysis of the program’s play score. indicated that it was a strong Othello player. A single chip is capable of analyzing over BILL won a single game against Brian Rose, the two million chess positions a second (using highest-rated American Othello player at the 1997 technology). It is important to note that time. In test games against IAGO, BILL won every this speed understates the chip’s capabilities. game. These results led Lee and Mahajan to Some operations that are too expensive to conclude that “BILL is one of the best, if not the implement in software can be done with little best, Othello player in the world.’’ As usual, or no cost in hardware. For example, one capa- there is danger in extrapolating conclusions bility of the chip is to selectively generate sub- based on limited evidence. sets of legal moves, such as all moves that can With the advent of the Internet Othello serv- put the opponent in check. These increased er (IOS), computer Othello tournaments capabilities give rise to new opportunities for became frequent. In the 1990s, they were dom- the search algorithm and the evaluation func- inated by Michael Buro’s LOGISTELLO. The pro- tion. Hsu (1999) estimates that each chess chip gram participated in 25 tournaments, finished position evaluation roughly equates to 40,000 first 18 times, second 6 times, and fourth once. instructions on a general-purpose computer. If The program combined deep search with an so, then each chip translates to a 100 billion extensive evaluation function that was auto- instruction a second chess supercomputer. matically tuned, which when combined with Access to the chip is controlled by an alpha- an extensive database of opening moves and a beta search algorithm that resides on the host perfect end-game player, is a winning recipe for computer (an IBM SP-2). Each of the 30 SP-2 Othello. processors could access 16 chips. The reported Although it was suspected that by the mid- cumulative performance, 200,000,000 posi- 1990s, computers had surpassed humans in tions analyzed a second, falls short of the peak their playing abilities at Othello, this was not speed (over 1 billion positions a second) properly demonstrated until 1997, when LOGIS- because of the inherent difficulty of getting TELLO played an exhibition match against world good parallel performance out of the alpha- champion Takeshi Murakami. In preparation beta algorithm. This massive amount of com- for the match, Buro (1997, p. 189) that puting allows the program to search deeper, BILL played a series of games against differ- significantly reducing the probability that it ent versions of LOGISTELLO. The results will make an error (as Kasparov found out to showed that bill, when playing 5-minute his regret). games running on a PentiumPro/200 PC, The AI community gave a collective sigh of is about as strong as a 3-ply LOGISTELLO, relief when DEEP BLUE defeated Kasparov. It was even though BILL searches 8 to 9 plies. time to move on to new challenges in the field. Obviously, the additional search is com- pensated for by knowledge. However, the Othello 3-ply LOGISTELLO can only be called The first major Othello program was Paul mediocre by today’s human standards. Rosenbloom’s (1982) IAGO, achieving impres- Two explanations for the overestima- sive results given its early-1980 hardware. It tion of playing strength in the past come dominated play against other Othello pro- to mind: (1) during the last decade human grams of the time but played only two games players have improved their playing skills against world-class human players, losing considerably, and (2) the playing strength both. The program’s ability to predict 59 per- of the early programs was largely overesti- cent of the moves played by human experts mated by using … nonreliable scientific was extrapolated to conclude that the pro- methods. gram’s playing strength was of world-champi- LOGISTELLO won all six games against Muraka- onship caliber. mi by a total disc count of 264 to 120 (Buro By the end of the decade, IAGO had been 1997), which confirmed what everyone had eclipsed. Kai-Fu Lee and Sanjoy Mahajan’s pro- expected about the relative playing strengths gram BILL represented a major improvement in of man and machine. The gap between the best the quality of computer Othello play (Lee and human players and the best computer pro- Mahajan 1990). The program combined deep grams is believed to be large and effectively search with extensive knowledge (in the form unsurmountable. of precomputed tables) in its evaluation func- Outwardly, LOGISTELLO looks like a typical

34 AI MAGAZINE Articles alpha-beta–based searcher (Buro 2001). Howev- (Shapiro and Smith 1977). In the 1980s, a num- er, the construction of the evaluation function ber of Scrabble programming efforts emerged, is novel. The program treats the game as having and by the end of the decade, it was apparent 13 phases: 13–16 discs on the board, 17–20 that these programs were strong players. With discs, …, and 61–64 discs.7 Each phase has a dif- access to the entire Scrabble dictionary in ferent set of weights in the evaluation function. memory (now over 100,000 words), the pro- The evaluation-function features are patterns of grams held an important advantage in any squares comprising combinations of corners, games against humans. diagonals, and rows. These patterns capture At the first Computer Olympiad in 1989, the important Othello concepts, such as mobility, Scrabble winner was CRAB written by Andrew stability, and parity. LOGISTELLO has 11 such pat- Appel, Guy Jacobson, and Graeme Thomas terns, which with rotations and reflections (Leavy and Beal 1989). Second was TYLER writ- yield 46. The patterns include a 3 3 and a 5 ten by Alan Frank. Subsequent Olympiads saw 2 configuration of stones anchored in a corner the emergence of TSP (Jim Homan), which and all diagonals of length greater than 3. edged out TYLER in the second and third The weights for each entry in each pattern Olympiads. All these programs were very good (46) for each phase of the game (11) are deter- and quite possibly strong enough to be a seri- mined by linear regression. More than 1.5 mil- ous test for the best players in the world. lion table entries need to be determined. The Part of their success was a result of the fast, data were trained using 11 million scored posi- compact Scrabble move generator developed tions obtained from self-play games and prac- by Andrew Appel (Appel and Jacobson 1988). tice games against another program (Buro Steven Gordon (1994) subsequently developed 1995). The evaluation function is completely a move generator that was twice as fast but table driven. Given a position, all 46 patterns used five times as much storage. are matched against the position, with a suc- Brian Sheppard began working on a Scrabble cessful match returning the associated weight. program in 1983 and started developing MAVEN These weights are summed to get the overall in 1986. In a tournament in December 1986, evaluation that approximates the final disc dif- MAVEN scored eight wins and two losses over an ferential. elite field, finishing in second place on a tie Michael Buro (1997, p. 193) comments on breaker. Sheppard describes the games against 8 the reasons why LOGISTELLO easily won the humans at this tournament: Murakami match: MAVEN reels off JOUNCES, JAUNTIER, and OVER- When looking at the games of the match TOIL on successive plays, each for exactly the main reasons for the clear outcome are 86 points, to come from behind against as follows: future national champion Bob Felt. MAVEN crushed humans repeatedly in offhand 1. Lookahead search is very hard for games. The human race begins to contem- humans in Othello. The disadvantage plate the potential of computers. becomes very clear in the endgame phase, In the following years, MAVEN continued to where the board changes are more sub- demonstrate its dominating play against stantial than in the opening and mid- human opposition. Unfortunately, because it dlegame stage. Computers are playing per- did not compete in the Computer Olympiads, fectly in the endgame while humans often it was difficult to know how strong it was com- lose discs. pared to other programs at the time. 2. Due to the automated tuning of the eval- In the 1990s, Sheppard developed a pre–end- uation functions and deep selective search- game analyzer (when there were a few tiles left es, the best programs estimate their win- in the bag) and improved the program’s ability ning chance in the opening and to simulate likely sequences of moves. These middlegame phase very accurately. This represented important advances in the pro- leaves little room for human innovations gram’s ability. It was not until 1997, however, in the opening, especially because the best that the opportunity arose to properly assess Othello programs are extending their the program’s abilities against world-class play- opening books automatically to explore ers. In 1997, a two-game match between MAVEN new variations. and Adam Logan at AAAI-97, one of the best players in North America, ended in two wins Scrabble for the human. Unfortunately, the match was The first documented Scrabble program not long enough to get a sense of who was real- appears to have been written by Stuart Shapiro ly the best player. and Howard Smith and was published in 1977 In March 1998, the New York Times sponsored

FALL 2001 35 Articles

an exhibition match between MAVEN and a team of QXI would be less preferable than leaving consisting of world champion Joel Sherman QUI; the latter offers more potential for play- and runner-up Matt Graham. It is not clear ing the Q effectively. whether the collaboration helped or hindered Bingo blocking: Playing all 7 letters in a sin- the human side, but the computer won con- gle turn leads to a bonus of 50 points (a bingo). vincingly by a score of six wins to three. The This move generator finds moves that reduce result was not an anomaly. In July 1998, MAVEN the chances of the opponent scoring a bingo played another exhibition match against Adam on his/her next turn. Sometimes it is worth sac- Logan (at AAAI-98), scoring nine wins to five. rificing points to reduce the opponent’s Shortly after the Logan match, Brian Shep- chances of scoring big. pard wrote:9 Immediate scoring: This generator finds the The evidence right now is that MAVEN is far moves with the maximum number of points stronger than human players.… I have (which becomes more important as the end of outright claimed in communication with the game nears). the cream of humanity that MAVEN should Each routine provides as many as 10 candi- be moved from the “championship cal- date moves. Merging these lists results in typi- iber’’ class to the “abandon hope’’ class, cally 20 to 30 unique candidate moves to con- and challenged anyone who disagrees sider. In the early part of the game, only the with me to come out and play. No takers score-and-rack generator is used. In the so far, but maybe one brave human will pre–end-game, there are four: the three listed yet venture forth. above plus a pre–end-game evaluator that No one has. “took years to tune to the point where it didn’t blunder nearly always.”11 In the end game, all MAVEN divides the game into three phases (Sheppard 2001): (1) early game, (2) pre–end possible moves are considered. game, and (3) end game. The early game starts The move-generation routines are highly at move one and continues until there are 9 or effective at filtering the hundreds or thousands 12 fewer tiles left in the bag (that is, with the of possible moves: opponent’s 7 tiles, there are 16 or fewer It is important to note that simply select- unknown tiles). In the pre–end-game and end- ing the one move preferred by the score- game phases, specialized searches are per- and-rack evaluator plays championship- formed, taking advantage of the limited caliber Scrabble. My practice of combining amount of unknown information. 10 moves from multiple generators is evi- In the early game phase, the program uses dence of developing paranoia on my part. simulations to get a statistical analysis of the “Massive overkill” is the centerpiece of likely consequences of making a move. Typical- maven’s design philosophy. ly, 1,000 three-ply simulations are done when Obviously, this move filtering works very making a move decision. The move leading to well, given the level of the program’s play. The the highest average point differential is select- Scrabble community has extensively analyzed ed. The issue with the simulations is move gen- MAVEN’s play and found a few minor errors in eration. On average, there are over 700 legal the program’s play. Postmortem analysis of the moves to a position, and the presence of 2 Logan match showed that MAVEN made mistakes blanks in the rack can increase this figure to that averaged nine points a game. Logan’s aver- more than 5000!10 Contrast this number, for age was 40 points a game. MAVEN missed seven example, with chess, where the average number fishing moves—opportunities to exchange of moves to consider in a position is roughly 40. some tiles (69 points lost), some programming Thus, MAVEN needs to pare the list of possible errors (48 points lost), and several smaller mis- moves down to a small list of likely moves. takes (6 points lost). The programming errors Omitting an important move from this list will have been corrected. If a future version of MAVEN have serious consequences; it will never be included fishing, the error rate would drop to played. Consequently, MAVEN uses multiple less than one point to a game. MAVEN would be move generators, each identifying moves that playing nearly perfect Scrabble. have important features that merit considera- Of the points lost because of programming tion. These move generators are as follows: errors, Brian Sheppard 13 Score and rack: This generator finds moves that result in a high score and a good rack (tiles It just drives me crazy that I can think up remaining in your possession). Strong players inventive ways to get computers to act evaluate their rack based on the likeliness of intelligently, but I am not smart enough the letters being used to aid upcoming words. to implement them correctly. For example, playing a word that leaves a rack The soliloquy of every games programmer!

36 AI MAGAZINE Articles

So, You Think You Are Good at Scrabble?

Figure A. MAVEN Plays BOS (j10), Scoring 26 Points. Figure B. MAVEN—Logan, Final Position.

At AAAI-98, MAVEN played an exhibition er bingo! pts, MAVEN = 330 Logan = 390. Maven’s sec- match against Adam Logan, one of the top MAVEN (AHINRTU) plays HURT at 4a, 34 ond brilliant move, choosing AX over GOX Scrabble players in North America. Logan pts, MAVEN = 110 Logan = 183. (13G,36) and sacrificing 11 points. won three of the first four games of the Logan (DDEEMMN) plays EMENDED at Logan (EIIILQU) plays LEI at o5, 13 pts, match, but MAVEN won six of the next seven. c7, 26 pts, MAVEN = 110 Logan = 209. MAVEN = 330 Logan = 403. Going into the critical twelfth game, MAVEN MAVEN (ABEINNP) plays IAMB at 8a, 33 MAVEN (?AHRTTW) plays WE at 9b, 10 led by a score of seven wins to four. The fol- pts, MAVEN = 143 Logan = 209. pts, MAVEN = 340 Logan = 390. lowing annotations are based on comments Logan (AILMTTU) plays MATH at a1, 27 Logan (AIIIOQU) plays QUAI at j2, 35 from Brian Sheppard. The columns of a pts, MAVEN = 143 Logan = 236. Strong play- pts, MAVEN = 340 Logan = 438. A 98-point Scrabble board are specified from left to ers also consider UTA (3a,20,ILMT), which lead and only a few moves are left in the right by the letters a to o. Rows are specified scores fewer points but gets rid of the game. Obviously, it’s all over. from top to bottom using the numbers 1 to annoying “U.” MAVEN (?AHRTTU) plays MOUTHPART at 15. Moves are specified by giving the square MAVEN (EFGNNPS) plays FEIGN at e10, 18 1a, 92 + 8 pts, MAVEN = 440 Logan = 438. See of the first letter of the word. If the coordi- pts, MAVEN = 161 Logan = 236. FENS figure B. Wonderful! MAVEN scores exactly nate begins with a number, then the word (j9,24,GNP) scores more points, but FEIGN 100 points, edging Adam Logan by 2. Shep- is placed horizontally. If the coordinate keeps better tiles. pard writes that “MAVEN steals the game on begins with a letter, then the word is placed Logan (AILORTU) plays TUTORIAL at the last move. Adam, of course, was vertically. The blank is referred to by “?.” 15h, 77 pts, MAVEN = 161 Logan = 313. stunned, as it seemed that there were no Follow along yourself. How good are the Adam Logan’s third bingo! places for bingos left on this board. If I moves that you find? MAVEN (?ABNOPS) plays BOS at j10, 26 hadn’t felt so bad for Adam, who played MAVEN (ACNTVYZ) plays CAVY at 8f, 24 pts, MAVEN = 187 Logan = 313. See figure A. magnificently, I would have jumped and pts, MAVEN = 24 Logan = 0. The alternative is Sheppard considers this to be a “fantastic cheered” (Brian Sheppard, personal com- ZANY, scoring 32 points, but leaving a poor move” and one of the most difficult moves munication, 1999). This game put MAVEN up selection of letters in the rack. in the game. by eight games to four, so winning the Logan (EGLNORY) plays YEARLONG at Logan (IILPRSU) plays PILIS at 15a, 34 match was no longer in doubt. g6, 66 pts, MAVEN = 24 Logan = 66. The only pts, MAVEN = 187 Logan = 347. PILIS, PULIS, How often do you score 438 points in a bingo! A 50 point bonus. PILUS, and PURIS are all good. game of Scrabble…and lose? MAVEN (ADNNOTZ) plays DOZY at 6d, 37 MAVEN (?AKNPRS) plays SPANKER at k5, Just in case some of the words used in pts, MAVEN = 61 Logan = 66. AZLON 105 pts, MAVEN = 292 Logan = 347. The only this game are not part of your everyday (10e,34,NTD) or ZOON (11e,26,ADNT) can bingo, reviving Maven’s chances despite vocabulary, here are a few useful definitions also be considered. the 160-point deficit. (taken from the commercial version of Logan (ADEFOTV) plays OFT at h13, 21 Logan (EEEORUS) plays OE at b1, 12 pts, MAVEN): pts, MAVEN = 61 Logan = 87. Of course, you MAVEN = 292 Logan = 359. The best move, Bos: a pal. also considered VOTED (5A,27,AF), OVA dumping extra vowels. Fens: marshes. (H13,21,DEFT), FOVEAL (10b,22,DT), and MAVEN (?HJTTWW) plays JAW at 7j, 13 Foveal: a shallow anatomical depression. ADVENT (12c,22,FO). pts, MAVEN = 305 Logan = 359. Gox: gaseous oxygen. MAVEN (AENNNOT) plays NEON at 5b, 15 Logan (AEEGRSU) plays GREASE at m3, Pilis: a Philippine tree. pts, MAVEN = 76 Logan = 87. 31 pts, MAVEN = 305 Logan = 390. AGER Uta: a type of lizard. Logan (ACDEEIV) plays DEVIANCE at (L9,24,ESU) also merits consideration. Zoon: whole product of one fertilized 12b, 96 pts, MAVEN = 76 Logan = 183. Anoth- MAVEN (?HRTTWX) plays AX at 6m, 25 egg.

FALL 2001 37 Articles

Other Games Zia Mahmood. In 1990, he offered a prize of Superhuman performance has been achieved £1,000,000 to the person who developed a pro- in several lesser-known games. For example, gram that could defeat him at bridge. At the for both the acient African game of Awari and time, this bet seemed safe for the foreseeable the recently invented Lines of Action, there future. seems little doubt that computers are signifi- In the 1990s, several academic efforts began cantly stronger than all human players. In using bridge for research in AI (Frank 1998; Awari, databases containing all positions with Ginsberg 1999; Smith, Nau, and Throop 1998a, 38 or fewer stones on the board have been con- 1998b; Ginsberg 1996b). The commercial structed (the game starts with 48 stones), sug- BRIDGE BARON program teamed up with Dana gesting that a perfect computer player will Nau and Steve Smith from the University of soon be available. In Lines of Action, the pro- Maryland. The result was a victory in the 1997 gram MONA won the world mail-play champi- World Computer Bridge Championship. The onship in competition against most of the top program used a hierarchical task network for human players. The gap between the top pro- the play of the hand. Rather than build a grams and the best humans appears to be large search tree where each branch was the play of and growing. a card, they would define each branch as a For some games, computers have been able strategy, using human-defined concepts such to determine the result of perfect play and a as finesse and squeeze (Smith, Nau, and Throop sequence of moves to achieve this result (van 1998a, 1998b). The result was an incremental den Herik, Uiterwijk, and van Rijswijck 2001).14 improvement in the program’s card play, but it In these games, the computer can play perfect- was still far from being world-class caliber. ly, in the sense that the program will never Beginning in 1998, Matthew Ginsberg’s pro- make a move that fails to achieve the best-pos- gram GIB started dominating the computer sible result. Solved games include Nine Men’s bridge competition, handily winning the Morris (Gasser 1995), Connect-4 (Allis 1988), World Computer Bridge Championship. The Qubic (Allis 1994), Go Moku (Allis 1994), and 8 program started producing strong results in 8 Domineering (Breuker, Uiterwijk, and van competitions against humans, including an den Herkik 2000). impressive result in an exhibition match This article has not addressed one-player against world champions Zia Mahmood and games (or puzzles). Single-agent search (A*) has Michael Rosenberg (held at AAAI-98). The successfully been used to optimally solve match lasted two hours, allowing 14 boards to instances of the 24-puzzle (Korf 2000; Korf and be played. The result was in doubt until the last Felner 2001) and Rubik’s Cube (Korf 1997). hand, before the humans prevailed by 6.31 international match points (IMPs). This result was the first notable man-machine success for Current Research Efforts computer bridge-playing programs. Zia Mah- In the past decade, a number of games have mood, impressed by the rapid progress made become popular research test beds. These by GIB, withdrew his million pound prize. games are resistant to alpha-beta search, either GIB was invited to compete in the Par Con- because of the large branching factor in the test at the 1998 World Bridge Championships. search tree or the presence of unknown infor- This tournament tests the contestant’s skills at mation. In many respects, the research being playing out bridge hands. In a select field of 35 done for these games has the potential to be of the premier players in the world, the pro- much more widely applicable than the work gram finished strongly in twelfth place. done on the alpha-beta search-based programs. Michael Rosenberg won the event with a score of 16,850 out of 24,000; GIB scored 11,210. Of Bridge the points lost by GIB, 1,000 were because of Work on computer bridge began in the early time (there was a 10-point penalty for each 1960s (Berlekamp [1963], for example), but it minute spent thinking), 6,000 were because GIB wasn’t until the 1980s that major efforts were did not understand the bidding, and 6,000 undertaken. The advent of the personal com- were because GIB was unable to handle some puter spurred numerous commercial projects hands where the correct strategy involves com- that resulted in programs with relatively poor bining different possibilities (Ginsberg 1999). capabilities. Perennial world champion Bob The name GIB originally stood for “Goren in Hamman once remarked that the commercial a box,” a tribute to one of the pioneers of programs “would have to improve to be hope- bridge. Another interpretation is “Ginsberg’s less’’ (Ginsberg 1999). A similar opinion was Intelligent Bridge.’’ Previous versions of GIB shared by another frequent world champion, used a fast search to play out a hand. It simu-

38 AI MAGAZINE Articles lated roughly 50 different scenarios for the effort that is currently being addressed. Had Zia placement of the opponent’s cards and chose Mahmood not withdrawn his offer, he might the play that maximized the expected score eventually have lost his money. (Ginsberg 1999). For the play of the hand, Ginsberg has developed a new version of the Go algorithm that eliminates the simulations and The history of computer go has not been dom- replaces it with perfect information (Ginsberg inated by hardware advances, as seen in com- 2001). puter chess. Computer go tournaments prolif- A challenging component of the game is the erated in the 1990s, and the organizers had the bidding. Most previous attempts at bridge bid- benefit of the chess experience. Two tourna- ding have been based on an expert-defined set ment rules were instituted that had a signifi- of rules. This is largely unavoidable because cant impact on how program development bidding is an agreed-on convention for com- would occur. The first required all competitors municating card information. GIB takes this to run on a commercially available single- bidding one step further, building on the abil- processor machine, which had the advantage ity to quickly simulate a hand (Ginsberg 1999). of putting all the programs on a level playing The program has access to a large database of field by factoring out most hardware differ- bidding rules (7,400 rules from the commercial ences. The second rule required that an entire program MEADOWLARK BRIDGE). At each point in game had to be completed in 30 minutes for the bidding, GIB queries the database to find the each player. Because games could be as long as set of plausible bids. For each bid, the rest of 180 moves a side, programmers were faced the auction is projected using the database, with critical cost-benefit decisions in their and then the play of the resulting contract is implementations. The rules had the advan- simulated. GIB chooses the bid that leads to the tages of making tournaments easy to organize average best result for the program. (no expensive hardware setup or modem con- Although intuitively appealing, this ap- nections needed) and ensuring that competi- proach does have some problems. Notably, the tions could be completed quickly with lots of database of rules might have gaps and errors in games being played. it. Consider a rule where the response to the The first go program was written by Al bid 4 is incorrect in the database. GIB will Zobrist in 1970 (Zobrist 1970). Walter Reitman direct its play toward this bid because it and Bruce Wilcox began researching go pro- assumes the opponents will make the (likely grams in 1972 (Reitman et al. 1974), an effort bad) database response. As Ginsberg writes, “It that has continued for Wilcox to the current is difficult to distinguish a good choice that is day. These early efforts produced weak pro- successful because the opponent has no win- grams; there was no obvious single algorithm ning options from a bad choice that appears to build a program around, as alpha-beta had successful because the heuristic fails to identify done for chess. The difficulty in writing a go such options’’ (Ginsberg 1999, p. 588). program became evident; a strong program GIB uses three partial solutions to the prob- would need lots of patterns and knowledge, lem of an erroneous or incomplete bidding sys- with only a limited dependence on search. tem. First, the bidding database can be exam- Computer go tournaments began in 1984 ined by doing extensive offline computations with a short-lived series of annual tournaments to identify erroneous or missing bid informa- at the USENIX Conference. In 1987, the First tion. This is effective but can take a long time International Go Congress was held, and there to complete. Second, during a game, simula- have been annual events ever since. The mid- tion results can be used to identify when a 1990s were dominated by the program database response to a bid leads to a poor HANDTALK, written by Zhixing Chen. HANDTALK result. This may be evidence of a database remained stagnant for a few years while it was problem, but it could also be the result of effec- being rewritten. During this time, Michael tive disruptive bidding by GIB. Finally, GIB can Reiss’ GO4++ assumed front-runner status. be biased to make bids that are “close” to the Chen’s new program, GOEMATE, now appears to suggested database bids, allowing the program be the best. Although the top programs claim a the flexibility to deviate from the database. performance level of as many as 3 kyu on the To summarize, GIB is well on the way to go rating scale (a middle amateur level), most becoming a world-class bridge player. The pro- experts believe that the programs are much gram’s card play is already at a world-class level weaker than that (around 8 kyu). (as evidenced by the Par Contest result), and The Ing Prize has been set up as an incentive current efforts will only enhance the program’s to build strong go programs. The grand prize of abilities. The bidding needs improvement, an roughly $1.5 million will be won by the devel-

FALL 2001 39 Articles

opers of the first program to beat a strong Poker human player on a 19 19 board. To qualify to There are many popular poker variants. Texas play for the grand prize, a program must win a Hold’em is generally acknowledged to be the number of matches of increasing difficulty. most strategically complex variant of poker Currently, the programs have to defeat three that is widely played. It is the premier event at junior players (ages 11 to 13). Don’t let their the annual World Series of Poker.15 Until age fool you; they are very strong players! The recently, poker has been largely ignored by the winner of the annual International Go Con- computing academic community. There are gress gets the chance to play. To qualify for this two main approaches to poker research event, a program must finish in the top three (Billings 1995): One approach is to use simpli- in one of the North American, European, or fied variants that are easier to analyze. Howev- Asian championships. er, one must be careful that the simplification Go has been resistant to the techniques that does not remove challenging components of have been successfully applied to the games the problem. For example, Findler (1977) discussed in this article. For example, because worked on and off for 20 years on a poker-play- of the 19 19 board and the resulting large ing program for 5-card–draw poker. His branching factor, alpha-beta search alone has approach was to model human cognitive no hope of producing strong play. Instead, the processes and build a program that could learn, programs perform small, local searches that use ignoring many of the interesting complexities extensive application-dependent knowledge. of the game. David Fotland, the author of the MANY FACES OF The other approach is to pick a real variant GO program, identifies over 50 major compo- and investigate it using mathematical analysis, nents needed by a strong go-playing program. simulation, and ad hoc expert experience. The components are substantially different Expert players with a penchant for mathemat- from each other, few are easy to implement, ics are usually involved in this approach. None and all are critical to achieving strong play. In of this work has led to the development of effect, you have a linked chain, where the strong poker-playing programs. weakest link determines the overall strength. There is one event in the meager history of Martin Müller (author of EXPLORER) gives a computer poker that stands out. In 1984 Mike stark assessment of the reality of the current Caro, a professional poker player, wrote a pro- situation in developing go programs (Müller gram that he called ORAC (Caro spelled back- 1999, pp. 105–106): wards). It played one-on-one, no-limit Texas Given the complexity of the task, the sup- Hold’em. Few technical details are known porting infrastructure for writing go pro- about ORAC other than it was programmed on grams should offer more than is offered an Apple II computer in Pascal. However, Caro for other games such as chess. However, arranged a few exhibitions of the program the available material (publications and against strong players:16 source code) is far inferior. The playing It lost the TV match to casino owner Bob level of publicly available source code…, Stupak, but arguably played the superior though improved recently, lags behind game. The machine froze on one game of that of the state-of-the-art programs. the two-out-of-three set when it had Quality publications are scarce and hard moved all-in and been called with its to track down. Few of the top program- three of a kind against Stupak’s top two mers have an interest in publishing their pair. Under the rules, the hand had to be methods. Whereas articles on computer replayed. In the [world series of poker] chess or general game-tree search methods matches, it won one (from twice world regularly appear in mainstream AI jour- champion Doyle Brunson—or at least it nals, technical publications on computer had a two-to-one chip lead after an hour go remain confined to hard to find pro- and a quarter when the match was can- ceedings of specialized conferences. The celed for a press conference) and lost two most interesting developments can be (one each to Brunson and then-reigning learned only by direct communication world champion Tom McEvoy), but— with the programmers and never get pub- again—was fairly unlucky. In private, lished. preparatory exhibition matches against Although progress has been steady, it will top players, it won many more times than take many decades of research and develop- it lost. It had even beaten me most of the ment before world-championship–caliber go time. programs exist (Mueller 2001). Unfortunately, ORAC was never properly doc-

40 AI MAGAZINE Articles umented and the results never reproduced. It is culating a single set of weights to reflect reason- highly unlikely that ORAC was as good as this able behavior, and use them for all opponents. small sample suggests. No scientific analysis An offline simulation was used to compute the was done to see whether the results were the expected value for each possible hand; these result of skill or luck. As further evidence, none results closely approximate the ranking of of the current-day commercial efforts can hands by strong players. This is called generic claim to be anything but intermediate-level opponent modeling (GOM) (Billings et al. players. 1998). Although rather simplistic, this model is In the 1990s, the creation of an internet quite powerful in that it does a good job of relay chat (IRC) poker server gave the opportu- skewing the hand evaluations to take into nity for humans (and computers) to play inter- account the most likely opponent holdings. active games over the internet. A number of Obviously, treating all opponents the same is hobbyists developed programs to play on IRC. clearly wrong; each player has a different style. Foremost among them is R00LBOT, developed Specific opponent modeling (SOM) customizes by Greg Wohletz. The program’s strength the calculations to include opponent-specific comes from using expert knowledge at the information. The probability of an opponent beginning of the game and doing simulations holding a particular hand is adjusted by feed- for subsequent betting decisions. ing into a neural net the betting frequency sta- The University of Alberta program POKI, tistics gathered on this opponent from previ- authored by Darse Billings, Aaron Davidson, ous hands. These statistics usually provide Jonathan Schaeffer, and Duane Szafron, is the enough information to differentiate, for exam- first serious academic effort to build a strong ple, aggressive playing styles from conservative poker-playing program. POKI plays on the IRC ones. poker server and, like R00LBOT, is a consistent In competitive poker, opponent modeling is big winner. Unfortunately, because these games much more complex than portrayed here. For are played with fictitious money, it is hard to example, players can act to mislead their oppo- extrapolate these results to casino poker. nents into constructing an erroneous model. To play poker well, a program needs to be Early in a session, a strong poker player might able to assess hand strength (chances that you try to create the impression of being very con- have the current best hand), assess hand poten- servative, only to exploit this image later in the tial (chances that additional cards will improve session when the opponents are using an your hand), model the opponents (exploiting incorrect opponent model. A strong player has tendencies in their play), handle deception to have a model of each opponent that can (misleading information given by the oppo- quickly adapt to changing playing styles. nents), and bluff (deceive the opponents). In At best, POKI plays at the strong intermediate strategic games such as chess, the performance level (Billings et al. 2001). A considerable gap loss by ignoring opponent modeling is small; remains to be overcome before computers will hence, it is usually ignored. In contrast, not be as good as the best human players. Recent only does opponent modeling have tremen- research has focused on trying to build “opti- dous value in poker, it can be the distinguish- mal” playing strategies (Koller and Pfeffer ing feature between players at different skill 1997). levels. If a set of players all have a comparable knowledge of poker fundamentals, the ability Other Games to alter decisions based on an accurate model Several less well-known games are providing of the opponent can have a greater impact on interesting challenges. The following three success than any other strategic principle.17 examples all have one property in common: a To assess a hand, POKI compares its cards large branching factor. against all possible opponent holdings. Naive- Shogi, often referred to as Japanese chess, is ly, one could treat all opponent hands as equal- very popular in Japan, with major tournaments ly likely; however, this skews the hand evalua- each year culminating in a computer world tions compared to more realistic assumptions. championship. From the search point of view, Many weak hands are likely to have been fold- Shogi is more challenging than chess: 9 9 ed early on in the game. Therefore, for each board (versus 8 8 for chess), 40 pieces (32 for possible opponent hand, a probability (or chess), 8 piece types (6), 80 to 120 average weight) is computed that indicates the likeli- branching factor (40), and ability of the cap- hood that the opponent would have played tured pieces to be returned to the board the hand in the observed manner. (removed from the board). Checkmating The simplest approach to determining these attacks are critical in Shogi; the programs need weights is to treat all opponents the same, cal- specialized checkmate solvers. These solvers

FALL 2001 41 Articles

have had some spectacular successes. For and deep solution lengths (some optimal solu- example, programs are now capable of solving tions are over 700 moves) make for a daunting composed problems with a solution length of search. On a standard test set, the program over 1500 ply! Nevertheless, the best programs ROLLING STONE can only solve 57 of 90 problems play at the master’s level, but world-champi- (Junghanns and Schaeffer 2001, 1999). onship–level play is still a few decades away (Iida, Sakuta, and Rollason 2001). Hex is an elegant game with a simple rule The Future of Computer Games set: alternate placing a stone of your color on In the realm of board and card games, go will an empty square. One player tries to create a continue to taunt AI researchers for many chain of stones connecting the top to the bot- decades to come. As well, new games will come tom of the board. The other player tries to con- along to provide interesting challenges. For nect the left side to the right side. It can be example, the game of Octi was invented to be shown mathematically that the game is a ﬁrst- resistant to computer algorithms.18 It is charac- player win and that draws are not possible. terized by having a large branching factor, QUEENBEE was the ﬁrst program to achieve suc- making deep search impractical. However, Octi cess against strong programs (van Rijswijk has the additional dimension that a move can 2000). The program uses alpha-beta search change the capabilities of a piece, making it with a novel evaluation function. Hexy is cur- challenging to design an evaluation function. rently the strongest program in the world and The research into board and card games is, in is competitive with strong human players for some sense, historically motivated because smaller board sizes. The program uses a special- these challenges were interesting at the dawn ized search for virtual connections, utilizing a of the computing age. However, with the theorem-prover-like technique to prove that advent of home computers, new forms of com- two points not connected can be connected by puter games and a $20 billion (and growing) a series of moves (Anshelevich 2001, 2000). industry has emerged: interactive computer A recently invented game that has become games. There are numerous products on the popular for games researchers is Amazons. It is market covering the gamut of action games played on a 10 10 board, with each player (for example. shoot’em-up games such as having four queens. Pieces move like a queen Quake), role-playing games (for example, player in chess, but after moving, they shoot an goes on a quest, as in Baldur’s Gate), adventure arrow in any direction. The square on which games (for example, navigating through a the arrow lands now becomes a wall and can- scripted story, as in King’s Quest), strategy not be occupied by a queen. In effect, each games (for example, controlling armies in a move reduces the playing area available. If you war, such as in Command and Conquer), “God” run out of moves, you lose. In the opening games (for example, evolving a simulated pop- phase of the game, there can be several thou- ulation, as in SimCity), and sports (for example, sand moves to choose from. The best programs controlling a player or coaching a team, such typically search five ply ahead (deeper in the as FIFA’01) (Laird and van Lent 2000). Histori- end game). Because of the territorial nature of cally, these games have been long on graphics the game, Amazons is often touted as a and short on AI.19 research stepping stone between the search- John Laird has promoted interactive com- intensive approaches used in chess and the puter games as an opportunity for the AI knowledge-intensive approaches used in go. AI research community (Laird and van Lent research into this game is only three years old. 2000). Many interactive computer games The best programs play reasonably well but are require computer characters that need to inter- not yet competitive with strong human play- act with the user in a realistic, believable man- ers (van den Herik 2000). ner. Computer games are the ideal application Interesting research is also being done on for developing human-level AI. There is puzzles. Recently, major advances have already a need for it because human game occurred in building programs that can solve players are generally dissatisfied with comput- crossword puzzles. PROVERB (Michael Littman, er characters. The characters are shallow, too Greg Keim, and colleagues) scores remarkably easy to predict, and, all too often, exhibit arti- well (over 95 percent of the words correct) on ficial stupidity rather than artificial intelli- the New York Times crossword puzzles without gence. This has led to the success of online understanding the clues (Keim et al. 1999; games (such as Ultima Online), where players Keim, and Shazeer 2001)! compete against other humans. The current Another challenging puzzle is SOKOBAN. Here state of the art in developing realistic charac- the large branching factor (could be over 100), ters can be described as primitive, with simple

42 AI MAGAZINE Articles rule-based systems and finite-state machines domain for exploring new ideas in AI. the norm. The lack of sophistication is the result of the lack of research effort (and, cause and effect, research dollars). This is changing Conclusions because more game companies and researchers Shannon, Turing, Samuel, Newell, and Simon’s recognize that AI will play an increasingly early writings were pioneering, realizing that important role in game design and develop- computer games could be a rich domain for ment. The quality of the computer graphics exploring the boundaries of computer science might draw you to a product, but the play of and AI. Software and hardware advances have the game will keep you using the product (and led to significant success in building high-per- buying the sequel). AI is critical to creating a formance game-playing programs, resulting in satisfying gaming experience. milestones in the history of computing. With it Finally, the last few years have seen research has come a change in people’s attitudes. on team games become popular. The annual Whereas in the 1950s and 1960s, understand- RoboCup competition encourages hardware ing how to build strong game-playing pro- builders and software designers to test their grams was at the forefront of AI research, today 20 skills on the soccer field. it has been demoted to lesser status. In part, Although this article has emphasized build- this is an acknowledgment of the success One could ing games programs that can compete with achieved in this field—no other area of AI humans, there are many other AI challenges research can claim such an impressive track argue that that can use games as an interesting experi- record of producing high-quality working sys- the series of mental test bed. Some sample projects include tems. However, it is also a reflection on the data mining, learning, and annotators. nature of AI itself. It seems that as the solution computer- Data mining: There are large databases of to problems become understood, the tech- chess end-game positions for chess, checkers, and niques become less “AIish.” tournaments awari. It is dissatisfying that all a program can The work on computer games has resulted in do is look up a specific position in the database. advances in numerous areas of computing. that began in If the exact position is in the database, you get One could argue that the series of computer- useful information, else nothing. Surely there 1970 and chess tournaments that began in 1970 and must be some way of mining the data to learn continue to this day represents the longest run- continue to the principles of strong end-game play. As well, ning experiment in computing science history. there are large databases of chess opening this day Research using games has demonstrated the moves. Can these databases be analyzed to dis- benefits of brute-force search, something that represents the cover new opening ideas? Can one characterize has become a widely accepted tool for a num- longest an opponent’s strengths and weaknesses? Can ber of search-based applications. Many of the the data be extrapolated to similar positions? running ideas that saw the light of day in game-tree Learning: Using temporal-difference learn- search have been applied to other algorithms. experiment in ing to tune an evaluation function is just the Building world-championship–caliber games precursor to other exciting applications of computing programs has demonstrated the cost of con- learning technology to games. For example, structing high-performance AI systems. Games science research in applying learning algorithms can have been used as experimental test beds for result in more focused and informed game-tree history. many areas of AI. And so on. searches, better opponent modeling in poker, Arthur Samuel’s concluding remarks from and adaptive characters in commercial games. his 1960 paper are as relevant today as they Annotators: Developing annotators that were when he wrote the paper (Samuel 1960): can provide an interesting and informative analysis of a game is a challenging problem. Programming computers to play games is There have been some attempts at automating but one stage in the development of an the commentary for chess games (the Interna- understanding of the methods which tional Computer Chess Association has an must be employed for the machine simu- annual competition), but the results are lation of intellectual behavior. As we mediocre. It is hard to differentiate between progress in this understanding it seems the trivial and the interesting, the verbose and reasonable to assume that these newer the informative, all the while anticipating the techniques will be applied to real-life situ- questions humans would like answered in the ations with increasing frequency, and the commentary. An interesting example is the effort devoted to games … will decrease. work done on providing computer commen- Perhaps we have not yet reached this turn- tary to RoboCup games (Frank et al. 2001). ing point, and we may still have much to Games will continue to be an interesting learn from the study of games.

FALL 2001 43 Articles

Acknowledgments (rock, paper, scissors) competitions (www.cs.ualber- ta.ca/~games). I would like to extend my deepest admiration to the brave human champions who accepted 18. www.octi.net. the challenge of a computer opponent. In most 19. For example, path finding is a critical component cases, the champion had little to gain but of many games, yet it took until 1996 for the industry to “discover” A*. everything to lose. Malcolm Davis, Garry Kas- parov, Adam Logan, Zia Mahmood, Marion 20. www.robocup.com. Tinsley, Michael Rosenberg, and Takeshi 21. Portions of this article were published in Schaef- Murakami made it possible to scientifically fer (2000) and are reproduced with permission. measure the progress of game-playing pro- References grams. The initial impetus for this article came Allis, V. 1994. Searching for Solutions in Games and Artificial Intelligence. Ph.D. dissertation, Depart- almost two years ago when Marvin Zelkowitz ment of Computer Science, University of Limburg. suggested I write an article for Advances in Com- Allis, V. 1980. A Knowledge-Based Approach to Con- puters 50, reflecting back on the 40 years since nect-Four. The Game Is Solved: White Wins. M.Sc. Arthur Samuel wrote an article on computer thesis, Department of Computer Science, Vrije Uni- games in volume 1 of this series. This article versiteit. was eventually worked into a talk that was pre- Anshelevich, V. 2001. A Hierarchical Approach to sented at AAAI-00. I want to thank David Leake Computer Hex. Artificial Intelligence. Forthcoming. for encouraging me to write this article. Anshelevich, V. 2000. The Game of Hex: An Auto- Financial support was provided by the Nat- matic Theorem-Proving Approach to Game Program- ural Sciences and Engineering Research Coun- ming. In Proceedings of the Seventeenth National cil of Canada (NSERC) and Alberta’s Informat- Conference on Artificial Intelligence,189–194. Menlo ics Circle of Research Excellence (CORE).21 Park, Calif.: American Association for Artificial Intel- ligence. Notes Appel, A., and Jacobson, G. 1980. The World’s Fastest 1. Othello is a registered trademark of Tsukuda Orig- Scrabble Program. Communications of the ACM 31(5): inal, licensed by Anjar Co. 572–578, 585. 2. Scrabble is a registered trademark of the Milton Berlekamp, E. 1963. A Program for Playing Double- Bradley Company, a division of Hasbro, Inc. Dummy Bridge Problems. Journal of the ACM 10(4): 3. See www.msoworld.com. 357–364. 4. E-mail message from G. Tesauro, 14 August 1998. Berliner, H. 1980a. Backgammon Computer Program 5. To be fair, it is unlikely that the International Beats World Champion. Artificial Intelligence 14(2): Chess Federation will ever allow computers to com- 205–220. pete for the world championship. Berliner, H. 1980b. Computer Backgammon. Scientif- 6. The drosophila is the fruit fly. The analogy is that ic American 242(6): 64–72. the fruit fly is to genetics research as games are to AI Berliner, H., and Ebeling, C. 1989. Pattern Knowledge research. and Search: The supreme Architecture. Artificial Intel- 7. Note that there is no need for a phase for less than ligence 38(2): 161–198. 13 discs on the board because the search from the Billings, D. 1995. Computer Poker. M.Sc. thesis, first move easily reaches 13 or more discs. Department of Computing Science, University of 8. E-mail message from B. Sheppard, 9 March 1999. Alberta. 9. Personal communication with B. Sheppard, 1999. Billings, D.; Davidson, A.; Schaeffer, J.; Szafron, D. 2001. The Challenge of Poker. Artificial Intelligence. 10. As a frequent Scrabble player, I painfully admit Forthcoming. that the number of words that I find is considerably smaller than this! Billings, D.; Papp, D.; Schaeffer, J.; and Szafron, D. 1998. Opponent Modeling in Poker. In Proceedings 11. E-mail message from B. Sheppard, 1 June 1999. of the Fifteenth National Conference on Artificial 12. E-mail message from B. Sheppard, 1 June 1999. Intelligence, 493–499. Menlo Park, Calif.: American 13. Personal communication with B. Sheppard, 1999. Association for Artificial Intelligence. 14. This is in contrast to the game of Hex, where it is Breuker, B.; Uiterwijk, J.; and van den Herik, J. 2000. easy to prove the game to be a first-player win, but Solving 8 3 8 Domineering. Theoretical Computer Sci- computers are not yet able to demonstrate this win. ence 20(1–2): 195–206. 15. The 2000 winner of this event was Chris Fergu- Buro, M. 2001. Improving Heuristic Min-Max Search son, whose research career began in AI (he has pub- by Supervised Learning. Artificial Intelligence. Forth- lished with Richard Korf [Powley, Ferguson, and Korf coming. 1993]). Buro, M. 1997. The Othello Match of the Year: 16. E-mail message from M. Caro, 13 March 1999. Takeshi Murakami vs. LOGISTELLO. Journal of the Inter- 17. The importance of opponent modeling can be national Computer Chess Association 20(3): 189–193. seen in the First and Second International RoShamBo Buro, M. 1995. Statistical Feature Combination for

44 AI MAGAZINE Articles the Evaluation of Game Positions. Journal of Artificial Junghanns, A., and Schaeffer, J. 1999. Domain- Intelligence Research 3:373–382 Dependent Single-Agent Search Enhancements. In Campbell, M.; Hoane, J.; and Hsu, F.-H. 2001. DEEP Proceedings of the Sixteenth International Joint BLUE. Artificial Intelligence. Forthcoming. Conference on Artificial Intelligence, 570–575. Men- Condon, J., and Thompson, K. 1982. Belle Chess lo Park, Calif.: International Joint Conferences on Hardware. In Advances in Computer Chess 3, ed. M. Artificial Intelligence. Clarke, 45–54. New York: Pergamon. Keim, G.; Shazeer, N.; Littman, M.; Agarwal, S.; Ebeling, C. 1987. All the Right Moves. Cambridge, Cheves, C.; Fitzgerald, J.; Grosland, J.; Jiang, F.; Pol- Mass.: MIT Press. lard, S.; and Weinmeister, K. 1999. PROVERB: The Prob- abilistic Cruciverbalist. In Proceedings of the Six- Felten, E., and Otto, S. 1988. A Highly Parallel Chess teenth National Conference on Artificial Intelligence, Program. In Proceedings of the Conference on Fifth Gen- 710–717. Menlo Park, Calif.: American Association eration Computer Systems, 1001–1009. New York: for Artificial Intelligence. Springer-Verlag. Koller, D., and Pfeffer, A. 1997. Representations and Findler, N. 1977. Studies in Machine Cognition Solutions for Game-Theoretic Problems. Artificial Using the Game of Poker. Communications of the Intelligence 94(1): 167–215. ACM 20(4): 230–245. Korf, R. 2000. Recent Progress in the Design and Frank, I. 1998. Search and Planning under Incomplete Analysis of Admissible Heuristic Functions. In Pro- Information: A Study Using Bridge Card Play. New York: ceedings of the Seventeenth National Conference on Springer Verlag. Artificial Intelligence, 1165–1170. Menlo Park, Calif.: Frank, I.; Tanaka-Ishii, K.; Okuno, H.; Nakagawa, Y.; American Association for Artificial Intelligence. Maeda, K.; Nakadai, K.; and Kitano, H. 2001. And the Korf, R. 1997. Finding Optimal Solutions to Rubik’s Fans Are Going Wild! SIG plus MIKE. In Proceedings Cube Using Pattern Databases. In Proceedings of the of the Fourth International Workshop on RoboCup. New Fourteenth National Conference on Artificial Intelli- York: Springer-Verlag. Forthcoming. gence, 700–705. Menlo Park, Calif. : American Asso- Gasser, R. 1995. Efficiently Harnessing Computation- ciation for Artificial Intelligence. al Resources for Exhaustive Search. Ph.D. disserta- Korf, R., and Felner, A. 2001. Disjoint Pattern Data- tion, Institute of Theoretical Computer Science, ETH base Heuristics. Artificial Intelligence. Forthcoming. Zürich. Krol, M. 1999. Have We Witnessed a Real-Life Turing Ginsberg, M. 1999. GIB: Steps toward an Expert-Level Test. Computer 32(3): 27–30. Bridge-Playing Program. In Proceedings of the Six- teenth International Joint Conference on Artificial Laird, J., and van Lent, M. 2000. Human-Level AI’s Intelligence, 584–589. Menlo Park, Calif.: Interna- Killer Application: Interactive Computer Games. In tional Joint Conferences on Artificial Intelligence. Proceedings of the Seventeenth National Conference Ginsberg, M. 1996a. Do Computers Need Common on Artificial Intelligence, 1171–1178. Menlo Park, Sense? In Proceedings of the Fifth International Confer- Calif.: American Association for Artificial Intelli- ence on Knowledge Representation and Reasoning, gence. 620–626. San Francisco, Calif.: Morgan Kaufmann. Lee, K.-F., and Mahajan, S. 1990. The Development of Ginsberg, M. 1996b. Partition Search. In Proceedings a World Class Othello Program. Artificial Intelligence of the Thirteenth National Conference on Artificial 43(1): 21–36. Intelligence, 228–233. Menlo Park, Calif.: American Levy, D., and Beal, D., eds. 1989. Heuristic Program- Association for Artificial Intelligence. ming in Artificial Intelligence. New York: Ellis Hor- Gordon, S. 1994. A Faster Scrabble Move Generation wood. Algorithm. Software Practice and Experience 24(2): Littman, M.; Keim, G.; and Shazeer, N. 2001. Solving 219–232. Crossword Puzzles by Computer. Artificial Intelligence. Hsu, F. 1999. IBM’s DEEP BLUE Chess Grandmaster Forthcoming. Chips. IEEE Micro 19(2): 70–81. McCarthy, J. 1997. AI as Sport. Science 276: 1518– Hsu, F.; Anantharaman, T.; Campbell, M.; and 1519. Nowatzyk, A. 1990a. A Grandmaster Chess Machine. Mueller, M. 2001. Computer Go. Artificial Intelligence. Scientific American 263(4): 44–50. Forthcoming. Hsu, F.; Anantharaman, T.; Campbell, M.; and Müller, M. 1999. Computer Go: A Research Agenda. Nowatzyk, A. 1990b. Deep Thought. In Computers, Journal of the International Computer Chess Association Chess, and Cognition, eds. T. Marsland and J. Schaef- 22(2): 104–112. fer, 55–78. New York: Springer Verlag. Newell, A.; Shaw, J.; and Simon, H. 1958. Chess-Play- Hyatt, R.; Gower, A; and Nelson, H. 1990. Cray Blitz. ing Programs and the Problem of Complexity. IBM In Computers, Chess, and Cognition, eds. T. Marsland Journal of Research and Development 2(2): 320–335. and J. Schaeffer, 111–130. New York: Springer Verlag. Pfeiffer, J. 1964. Man vs. Machine in the Mechanical Iida, H.; Sakuta, M.; and Rollason, J. 2001. The State Age. Popular Mechanics, August, 52–57,172–173. of the Art in Computer Shogi. Artificial Intelligence. Plaat, A.; Schaeffer, J.; Pijls, W.; and de Bruin, A. 1996. Forthcoming. Exploiting Graph Properties of Game Trees. In Pro- Junghanns, A., and Schaeffer, A. J. 2001. Enhancing ceedings of the Thirteenth National Conference on Single-Agent Search Using Domain Knowledge. Art- Artificial Intelligence, 234–239. Menlo Park, Calif.: ficial Intelligence 129(1–2): 219–251. American Association for Artificial Intelligence.

FALL 2001 45 Articles

Powley, C.; Ferguson, C.; and Korf, R. E. 1993. Depth- Sutton, R. 1988. Learning to Predict by the Methods First Heuristic Search on a SIMD Machine. Artificial of Temporal Differences. Machine Learning 3:9–44. Intelligence 60(2): 199–242. Tesauro, G. 2001. Programming Backgammon Using Reitman, W.; Kerwin, J.; Nado, R.; Reitman, J.; and Self-Teaching Neural Nets. Artificial Intelligence. Wilcox, B. 1974. Goals and Plans in a Program for Forthcoming. Playing Go. In Proceedings of the Association of Tesauro, G. 1995. Temporal-Difference Learning and Computing Machinery National Conference, td-gammon. Communications of the ACM 38(3): 123–127. New York: Association of Computing 58–68. Machinery. Tesauro, G. 1989. Neurogammon Wins Computer Rosenbloom, P. 1982. A World-Championship-Level Olympiad. Neural Computation 1(3): 321–323. Othello Program. Artificial Intelligence 19(3): 279–320. Thompson, K. 1982. Computer Chess Strength. In Samuel, A. 1967. Some Studies in Machine Learning Advances in Computer Chess 3, ed. M. Clarke, 55–56. Using the Game of Checkers: Recent Progress. IBM Oxford, U.K.: Pergamon. Journal of Research and Development 11(6): 601–617. Tinsley, M. 1980. Letter to the Editor. Scientific Amer- Samuel, A. 1960. Programming Computers to Play ican, August. Games. In Advances in Computers, Volume 1, ed F. Alt, Truscott, T. 1979–1980. The Duke Checkers Program. 165–192. San Diego, Calif.: Academic. Journal of Recreational Mathematics 12(4): 241–247. Samuel, A. 1959. Some Studies in Machine Learning Turing, A. 1953. Digital Computers Applied to Using the Game of Checkers. IBM Journal of Research Games. In Faster Than Thought, ed. B. Bowden, and Development 3(2): 210–229. 286–295. London: Pitman. Schaeffer, J. 2000. The Games Computers (and Peo- van den Herik, J. 2000. The Fifth Computer ple) Play. In Advances in Computers 50, ed. M. Olympiad. International Computer Games Association Zelkowitz, editor, 189–266. San Diego, Calif.: Acade- Journal 23(3): 164–187. mic Press. van den Herik, J.; Uiterwijk, J.; and van Rijswijck, J. Schaeffer, J. 1997. One Jump Ahead: Challenging 2001 Games Solved: Now and in the Future. Artificial Human Supremacy in Checkers. New York: Springer- Intelligence. Verlag. van Rijswijk, J. 2000. Computer Hex: Are Bees Better Schaeffer, J.; Hlynka, M.; and Jussila, V. 2001. Than Fruit Flies? M.Sc. thesis, Department of Com- Termporal Difference Learning Applied to a High- puting Science, University of Alberta. Performance Game-Playing Program. Paper present- Zobrist, A. 1970. Feature Extractions and Representa- ed at the Seventeenth International Joint Conference tion for Pattern Recognition and the Game of Go. on Artificial Intelligence, 4–10 August, Seattle, Wash- Ph.D. dissertation, University of Wisconsin. ington. Schaeffer, J.; Culberson, J.; Treloar, N.; Knight, B.; Lu, P.; and Szafron, D. 1992. A World Championship Cal- Jonathan Schaeffer is a professor iber Checkers Program. Artificial Intelligence 53(2–3): of computing science at the Uni- 273–290. versity of Alberta. He received his B.Sc. from the University of Toron- Shannon, C. 1950. Programming a Computer for to (1979) and his M.Math (1980) Playing Chess. Philosophical Magazine 41(4): 256–275. and Ph.D. (1986) degrees from the Shapiro, S., and Smith, H. 1977. A Scrabble Cross- University of Waterloo. His word Game-Playing Program. Technical Report, 119, research interests are in AI and Department of Computer Science, State University of parallel/distributed computing. He New York at Buffalo. is best known for his work on computer games. He is Sheppard, B. 2001. World Championship–Caliber the creator of the checkers program CHINOOK, the first Scrabble. Artificial Intelligence. Forthcoming. program to win a human world championship in Slate, D., and Atkin, L. 1977. Chess 4.5—The North- any game. He is also a cofounder of the bioinformat- western University Chess Program. In Chess Skill in ics software company BioTools, Inc. His e-mail Man and Machine, 82–118. New York: Springer-Ver- address is [email protected]. lag. Smith, S.; Nau, D.; and Throop, T. 1998a. Computer Bridge: A Big Win for AI Planning. AI Magazine 19(2): 93–105. Smith, S.; Nau, D.; and Throop, T. 1998b. Success in Spades: Using AI Planning Techniques to Win the World Championship of Computer Bridge. In Pro- ceedings of the Fifteenth National Conference on Artificial Intelligence, 1079–1086. Menlo Park, Calif.: American Association for Artificial Intelligence. Strachey, C. 1952. Logical or Non-Mathematical Pro- grammes. In Proceedings of the Association for Com- puting Machinery Meeting, 46–49. New York: Associ- ation of Computing Machinery.

46 AI MAGAZINE