Playing Agents: A Problem Analysis

Francisco Mota, Alex Rottinghaus University of Northern Iowa [email protected], [email protected]

Abstract based off the positions of the pieces and then deciding In this article we discuss chess as a problem to be on a best move from some heuristic. solved by . We analyze chess and describe why it is a particularly popular AI 2. Background problem. This article also explores the playing agents and discusses their advance- The game of chess is played on an 8x8 board with ment over time. We also highlight the competition two sets of pieces representing fictional armies. Each that erupted between chess players and set consists of a , a , two bishops, two engineers. We also discuss different algorithms knights, two rooks, and eight pawns. Each type of piece and programming techniques for playing chess. has a specific pattern it can move within. The king also We then offer an opinion of the of has a special requirement in that it can’t move into a techniques that are most often used and seem to have the greatest success at making winning chess space that would allow another piece to take it. As agents. players take turns moving their pieces they can capture a piece if they can move into its square while abiding Keywords: Chess Agents, Greenblatt, Deep Blue, by the rules of movement for their attacking piece. The , Genetic Algorithms game is won when one player positions their pieces such that they can attack the enemy king with one of 1. Introduction their pieces and the king can’t move to a square where it would not be under attack. Another way the game Chess has been used as a demonstration of artificial may end is under . In a stalemate a player is intelligence since the 1700’s when Wolfgang von stuck so that they can make no legal moves, yet they are Kempelen created a machine he called the Turk (Not so not in . A stalemate is considered a and smart 2003). While Kempelen’s machine was run by a should be avoided in favor of a . person hidden inside, it was an example of someone When first playing the game, it may appear as though trying to solve the problem of creating a machine to taking an opponent’s piece at every opportunity would play chess. be the way to win. Obviously, the more pieces you have Since that time chess has continued to be a popular compared with your opponent, the easier it will be for method of measuring machine intelligence. As you to trap their king. However, playing only a few progress over time people and companies continually try to create faster and better chess playing machines. The reason chess has become so popular may be because it is a game that can be described with a simple set of rules, but yet playing it is so vastly complex that many humans don’t or can’t learn to play it. Computer programmers were interested in intelligence and they needed somewhere to start. So they looked around at who the smartest people were, and they were themselves, of course. They were all essentially mathematicians by training and mathematicians do two things -- they prove theorems and play chess. And they said, hey, if it proves a theorem or plays chess, it must be smart. – (Hamilton 2000)

The complexity is what today makes a great measure of computer intelligence. There are between 10^43 and 10^50 legal chess boards (Lassabe et al. 2006). To create an algorithm to examine a chess Figure 1. A standard chess board with pieces in their board and then reference a database to find the correct starting positions. following move would be impossible with today’s computers. This means that a more clever solution must chess games with an experienced player will quickly be implemented that involves making assumptions show that this is false. Winning the game of chess requires careful planning that focuses not simply on

taking enemy pieces, but on arranging your pieces so as improvement from evaluating the whole board every to set up a trap for the enemy king, all the while time a single piece was moved or taken, but it was still preventing a similar trap from being set on your own overkill, for the Los Alamos agent was still evaluating king. every single board position reachable within four ply. In 1958, a chess player and programmer at IBM named Alex Bernstein, came up with an alternative 3. Agents solution. Bernstein’s program, operational in 1958, was agents draw their roots back to a also modeled after Shannon’s ideas. Unlike the Los journal article by (Shannon, 1950) that Alamos program, Bernstein’s program did not consider describes a general framework of a what a chess agent every single move. Bernstein implemented “plausible may look like. At any given turn, a chess agent would move generators” that would only allow his program to have to look at the set of possible moves and calculate consider moves that seemed to make a difference. He which move was best. With simple games, such as Tic- had different generators for different aspects of the Tac-Toe, this calculation can be performed using the game: King safety, development, defending own men, minimax algorithm. Shannon was really the first to attacking opponent’s men, and so on (Newell et al., argue that the minimax algorithm as it was originally 1958). proposed was unsuitable for chess agents – the This is similar to how humans seem to play chess, branching factor is much too high and the games are far which perhaps explains why this selectivity doesn’t too long to calculate every possible combination. adversely affect the agent’s ability to play by much, Shannon suggested that a chess agent would have to despite the radical savings it affords. For every 300 stop branching at some arbitrary point and evaluate the boards the Los Alamos program evaluates, Bernstein’s current board based on some heuristic. program considers only one board, and they both go to However, Shannon did not devise an actual chess a depth of four ply. agent. This was left up to , who in 1953 In spite of this massive reduction of complexity, and published an article detailing the inner working of a in spite of the faster hardware available under which chess agent as Shannon had envisioned it. Alan Turing this Bernstein’s program was run, it played at a rate did not create a program – he created a model or a comparable to the Los Alamos agent: Bernstein tended scheme through which he could simulate a chess agent to make a move in eight minutes, which is close to the by hand. Consequently, Turing’s agent required very Los Alamos program’s move time (ten minutes). This little computational power, but it did not play chess effect can be attributed to the fact that Bernstein’s very well, and manual simulation was prone to mistakes program performed an extensive analysis on each board (Turing made an error in his simulation, resulting in a it considered, whereas the Los Alamos program hardly different move than the actual agent would have taken, looked at all the boards it was evaluating. Part of this in the very same paper where he presented this idea). In extensive analysis was Bernstein’s spite of all this, chess agents had begun to take on (which was much more sophisticated than the Los human proportions: Turing’s agent could play just as Alamos program’s evaluation function) and another well (or just as badly) as a beginning chess player, so part of this analysis was devoted to allowing the perhaps it was fitting that it was one of the first. plausible move generators to generate good moves. One of the first computer programs that played chess automatically was invented at Los Alamos in 1956. 3.1 Greenblatt and a Second Generation of Agents Running on the MANIAC I machine, this agent could There were many chess agents created after Los play chess on a six by six board, by removing the Alamos and Bernstein, but none were truly innovative bishops. It also eliminated special moves , reducing the or particularly good at playing chess until 1966 when complexity of the problem to a size manageable for the an undergraduate student at MIT, Richard Greenblatt, technology of the time. The Los Alamos agent worked impressed everyone in the field with his chess program, truly as Claude Shannon had envisioned: it explored IV. This was the first agent to win against every possibility to a depth of four ply (two moves for a professional chess player in an official competition. Black and two moves for White), evaluating these The Greenblatt program is fairly intricate, compared boards in terms of the material possessed by both sides, to the other agents mentioned above. One of the and by how much mobility each position afforded to the cornerstones for Greenblatt’s involved pieces. These values were crystallized into a decision providing the agent with chess specific information and by following the minimax algorithm up through the with chess lore. Many of its move generators were only tree. triggered in specific situations, such as when the king is The Los Alamos agent was reasonably efficient, in danger. A pre-compiled list of starting moves taking roughly ten minutes to calculate a single move. It allowed it to start with perfect play, so as to not give the managed to be this fast by simplifying and minimizing opponent an advantage immediately. the process of evaluating the boards four ply deep. It Greenblatt’s success stems from the sophistication of would keep track of that leaf node’s parent’s evaluation the algorithms and heuristics employed in his agent, but and would only revise this evaluation as material was it also stems from his pragmatic approach to developing caught or as mobility changed. This was an the program: Greenblatt’s program allow one to see the

internal state of all nodes in its trees, on a cathode-ray widely considered to be the best chess agent of the monitor. It allowed one to see all the factors that went time, because it won the First World Computer into evaluating a particular board position. In essence, Championship in Stockholm in 1974. According to the Greenblatt program had a series of interactive and David Levy, introspective aids, and this allowed Greenblatt to debug and to improve his program at an impressive rate . uses a complex evaluation function involving many features. In fact, it is so complex Greenblatt’s importance lies in how it managed to that when I asked Mikhail Donskoy[, one of the start a resurgence of interest in computer chess (Levy, 1976). This reinvigoration happened because, as we developers,] he replied “....I don’t even remember what is in it”. (Levy, 1976) said before, the program played in a tournament and managed to beat a human player for the first time in computer chess history. The Greenblatt program’s KAISSA brute forced a minimax search with alpha-beta pruning to a maximum depth of 30 ply (although in sophistication allowed it to find a particular 9-ply practice a move would be chosen before it even checkmate sequence that many US Masters seemed to miss. It was put on show at the International Federation approached this depth), and it evaluated only significant moves (moves that involved material loss for one side, for Information Processing in 1968, where it played or moves that got one player out of check). anyone and managed to win about half the time. These factors all captured researchers’ imagination and contributed to the aforementioned resurgence of interest 3.2 Deep Blue in computer chess. In recent history, one of the most famous chess With Greenblatt, chess agents had become viable agents is the computer created at IBM called Deep competitors in chess. By 1970, the ACM began hosting Blue. In 1997 Deep Blue won a match of chess against an annual where all the competitors , who at the time was the world chess were computers. The winner of the first four ACM champion (Dennett 2007). While 1997 was the year that tournament was a program named CHESS 3.0 from made Deep Blue famous, that was really the end of the . Both the Greenblatt program Deep Blue saga. The story of Deep Blue goes back ten and the Northwestern program performed depth-first years earlier to a machine known as Chiptest, also searches with alpha-beta pruning, although the developed at IBM. Northwestern program had a very complicated way of Chiptest competed in 1987 in a strong chess determining how to expand the tree (quite a bit more tournament and tied for first place. Upon learning of complicated than Greenblatt’s and Bernstein’s plausible Chiptest’s win at the chess tournament, Kasparov move generators). wished to play this machine. He did so in 1989, Around this time, an agent called TECH was being winning both games in the two game match against developed at Carnegie-Mellon University by James Chiptest (Schaeffer 2001). This was the beginning of a Gillogly. Perhaps TECH was a form of intellectual long battle between IBM and what would be their protest against the ever more complicated heuristics ultimate test of chess prowess, Garry Kasparov. IBM’s being implemented in chess agents, because TECH’s loss in 1989 did not deter them, and they continued to philosophy was much simpler: brute force as much of work on Chiptest, renaming it to Deep Blue as it the problem as possible. It used a simple heuristic to evolved. determine what board was best when it reached a fixed In 1996 IBM brought out their machine against depth (obviously it couldn’t brute force the problem Kasparov once more. In the first game, Deep Blue forever) – it picks the board with highest material score. stunned the world by beating the chess master. This is detailed further below in a section on evaluation Kasparov managed to take the match, but it was clear heuristics. that Deep Blue had made significant advancement from At first, TECH was unsuccessful – it had essentially its early days as Chiptest. gone back to the Los Alamos model. Eventually, its The very next year IBM and Kasparov went head to designer relented and added sophisticated move head one more time. Deep Blue was finally able to generating heuristics that propelled TECH into the top bring down the chess master by scoring two wins, three tier of computer chess play. draws, and one loss (Schaeffer 2001). One of the features of TECH that hadn’t been Deep Blue’s victory is significant not because it was implemented previously was that it would keep able to beat a human being – this had occurred during computing during the opponent’s turn. When it Chiptest’s maiden tournament, and had been occurring occasionally guessed the opponent’s next move for quite some time. The significance of the final Deep correctly, it could reply almost immediately with its Blue vs. Kasparov match was that a computer had beat own next move. the human being widely regarded as the best chess For the sake of historical completeness, we shall player in existence. Essentially, it was a demonstration mention KAISSA briefly. KAISSA was a program that IBM was capable of creating a computer that could developed in 1971 in the based off of a outperform humanity. previous program that had defeated an American agent An unfortunate consequence of Deep Blue’s victory, in a much publicized match in 1967. KAISSA was is that research into chess agents has stagnated. Many

researchers are not interested in working on a problem is reached, and then using an evaluation function to that has essentially been “solved,” and there is little determine the value of the leaf nodes. Over time reason to believe that the situation will change any time however, this approach has been refined. Bernstein’s soon. Rather, research that would have previously approach was to select only moves it considers focused on chess tends to focus on even more complex important, giving rise to the concept of a “plausible games, such as the game of Go. move generator.” However, chess is far from a solved problem! Due to We shall look at both evaluation functions and IBM’s secrecy, we do not even have a good view of plausible move generators. The beauty of the modified how Deep Blue operated. Even though some minimax algorithm is that these are all legitimate ways can beat all humans, knowing how to of improving a chess agent’s effectiveness and make a an efficient chess agent that can beat humans is efficiency (or deteriorate these attributes, as the case still an open question, as is the question of creating a may be). agent that can defeat other supercomputer agents. 4.1. Evaluation Function With this in , let us look at chess agents from a An evaluation function is meant to signal the more abstract perspective. Let’s look at how chess desirability of a certain board. Typically a higher value agents can be constructed from the bottom up, from the ideas and algorithms to the finished product (as implies the board is more desirable for the agent, while a lower or a negative value implies the board should be opposed to looking at finished products and talking avoided, or that it is more desirable for the opponent about the ideas behind them, as we have just done in this section). (these are equivalent statements in chess). In the original minimax algorithm, if the value was positive, the agent would win, whereas if the value was negative, 4. Tree Searching Techniques the opponent would win (a zero signaled a tie). Because of chess’s complexity, this is impossible to determine In the field of Artificial Intelligence, the de facto without expanding every single path, so instead we just standard approach to creating a game-playing agent is use the evaluation function to imply desirability. to use the minimax algorithm. However, As Claude The simplest possible evaluation function for chess is Shannon pointed out, this is beyond impractical when it to count the material for each player. The actual value comes to chess, due to the overwhelming complexity of in this case would be the number of living pieces owned the problem (Shannon 1950). Rather, what almost all by the agent, minus the number of living pieces owned chess agent designers attempt to use is a modified by the opponent. minimax algorithm that limits the number of boards A slightly more complicated but more traditional being considered to a manageable size. heuristic is to give the different types of pieces different weights. The tradition is to weight pawns as 1, bishops function modified_minimax(board) and horses as 3, rooks as 5, queens as 9, and the king is if board worth continuing? worth an infinity (because no side can afford to lose the king). This weighting scheme has been used for evaluate board centuries before chess agents were even on anyone’s else mind, so this scheme is popular. get list of moves Genetic algorithms are becoming popular for each move everywhere in computing and can be seen regularly in calculate new board the game playing field of AI (McPartland 2008). get list of opponent’s moves Genetic algorithms can be used at many points during for each opponent’s move the evaluation of a chess game. calculate new board calculate value of new board via modified_minimax In this perspective, genetic programming [10] has end been mainly used in two ways. The first one is the choose board with smallest value automatic production of new heuristics for alpha- beta pruning [7], and the second is to find new end evaluation functions to estimate the value of choose board with largest_value potential next moves [5, 8]. Though these uses of end GP were original and effective, they still rely on an end exhaustive exploration of possible moves to play in order to choose the best next one. A third approach Figure 2. A recursive definition of the modified is to use Genetic Programming to classify minimax algorithm in Ruby-like pseudocode. endgames situations according to the remaining moves to checkmate the opponent. (Lassabe et al. Shannon’s original proposal suggested expanding 2006) every possible move until some arbitrary limit or depth

Using genetic programming then is an alternative to Lassabe et al. explored the use of genetic having people value boards and board types. With algorithms for the playing of end game scenarios. They genetic programming, the computer learns from used the classic king king end game and trained a experience which moves work in which situations, and genetic algorithm by giving value to the scenarios of then creates its own decision tree based on this. ending in stalemate or check mate in addition to general Most agents also consider other attributes of the moves. board, such as how much control the agent has over the center of the board, how well the king is defended, and how many pieces are free to move. Creating a new 6. Conclusion evaluation function is one of the easiest things to do in building a chess agent, so there has been a high level of Looking at the different agents historically we see a sophistication in this area, as the history section shows. variety of algorithms and programming techniques used in combination. The most successful agents use very 4.2 Plausible Move Generators fast hardware specifically designed for processing chess moves, in combination with multiple Plausible move generators look at the current board approaches to speeding up search and finding the best and try to come up with likely moves to be evaluated move. Alpha-beta pruning is king (so to speak) making later. This is how most humans seem to play chess (by it possible to evaluate many more moves of a game in a looking at relevant moves), so it is particularly reasonable time. In addition other methods such as appealing for chess agent designers, and it works well. parallel processing of the search function can help A simple move generator would look at all possible increase the speed with which the next move is found. attacks. A slightly more complicated generator would Genetic algorithms can be applied to improve and look at moves that defend higher-valued pieces from generate search functions, evaluation functions and attacks by blocking with lower-valued pieces. even end game agents. While it seemed that the Typically, an agent has multiple move generators, computer chess agent problem had been cracked by which it combines when creating the move list for the Deep Blue, its successors showed that there is still room current board. Move generators are an ideal location to for improvement and humanity isn’t out of the game. add chess-specific information because much of chess lore is concerned with what kinds of moves are References important at what times. Berliner, H., Kopec, D., and Northam, E. 1990. A taxonomy of concepts for evaluating chess 5. Endgame Strategies strength. In Proceedings of the 1990 ACM/IEEE Conference on Supercomputing (New York, New Another aspect of chess agents is their approach to York, November 12 - 16, 1990). Conference on finishing a game. When chess agents reach a certain High Performance Networking and Computing. point in the game they may change how they move to IEEE Computer Society, Washington, DC, 336- use a separate end game procedure. In the early days of 343. chess playing machines a man named L. Torres y Dennett, D. C. "Higher games: on the 10th anniversary Quevedo invented a device that could play the king of Deep Blue's triumph over Garry Kasparov in rook king end game (Shannon, 1950). The king rook chess, a prominent philosopher of mind asks, what king end game is one in which one player has a king did the match mean?. ." Technology Review and a rook and the other player only their king. This is an important end game because it tests the agent’s (Cambridge, Mass.). 110.5 (Sept-Oct 2007): 98(3). ability to avoid a stalemate. Academic OneFile. Gale. University of Northern One of the most common methods for end Iowa. 28 Mar. 2009 Hamilton, S. N., The Last Chess Game: Computers, game is to reference a set of pre saved end game Media Events, and the Production of Spectacular scenarios with best moves remembered. This is possible Intelligence. Canadian Review of American because of the reduced number of pieces in the end Studies , 30(3), 339-360. EBSCOhost. March 28, game. Oddly enough though, the reduced number of 2009. pieces actually allows each piece more freedom Lassabe, N., Sanchez, S., Luga, H., and Duthen, Y. meaning that the searches per piece increases and can 2006. Genetically programmed strategies for chess actually reduce performance of a search algorithm endgame. In Proceedings of the 8th Annual (Lassabe et al. 2006). So to save time, tables are used Conference on Genetic and Evolutionary where each board is stored with the next move Computation (Seattle, Washington, USA, July 08 - associated with it. In this way a board only has to be 12, 2006). GECCO '06. ACM, New York, NY, looked up in the table and the next move is immediately 831-838. DOI= known. The problem with this then is generating the http://doi.acm.org/10.1145/1143997.1144144 end game tables. Levy D. Before the Jet Age. Computer Games I , Springer-Verlag, New York, 1988, 116-153.

McPartland M., Gallagher M. Learning to be a Bot: Reinforcement Learning in Shooter Games. In The Fourth Artificial Intelligence and Interactive Digital Entertainment Conference , (Stanford California, 2008), AAAI Press, Newell, A. Shaw, J. C. Simon, H. A. Chess-Playing Programs and the Problem of Complexity. Computer Games I , Springer-Verlag, New York, 1988, 89-115. Not so Smart. Economist , 336 (8309). 11 EBSCOhost. March 28, 2009. http://search.ebscohost.com/login.aspx?direct=true &db=afh&AN=9019906&site=ehost-live Schaeffer, J. "A Gamut of Games. " AI Magazine. 22.3 (Fall 2001): 29. Academic OneFile. Gale. University of Northern Iowa. 28 Mar. 2009 Shannon, C. E. A Chess-Playing Machine. in Levy D. Computer Games I , Springer-Verlag, New York, 1988, 81-88.