The Utility of Expert Knowledge
Total Page：16
File Type：pdf, Size：1020Kb
THE UTILITY OF EXPERT KNOWLEDGE Jonathan Schacffer TA. Marsland Computing Science Department, University of Alberta, Edmonton, Canada T6G 2H1 ABSTRACT 2. Experiment Design How useful is the knowledge we add to an expert A chess program has two distinct parts; the system? What is important knowledge? Can too framework for making and analyzing moves, and much knowledge be bad? These questions are the knowledge that allows the program to play examined by presenting the preliminary results of well. The former includes legal move generation experiments that paired programs with varying and tree searching, and is well understood. The amounts of chess knowledge against each other. latter, however, is vague and informal; it is the The experiments illustrate problems of interacting product of centuries of experiences that have been knowledge and give some insight into methodolo• condensed into rules and exceptions, few of which gies for "teaching" expert systems. can be formalized. The expertise in our chess program Phoenix t 1. Introduction was partitioned into the following 8 routines: Tac• With the increasing awareness of the potential tics (T), Space and Mobility (SM), Pawn for expert systems, knowledge engineering has Weaknesses (PW), King Safety (KS), Center Con• become a recognized discipline. The addition of trol (CC), Pawn Structure (PS), Incremental knowledge to an expert system raises some impor• Scores (IS), and Planner (PL). Details of the con• tant questions: What should one add? How much? tents of the routines can be found elsewhere [2] Can too much knowledge be a bad thing? The and are not essential to the points raised in this goal of answering these questions is to find a paper. To determine the utility of this knowledge, methodology for adding knowledge to an expert a pair of experiments have been performed; one to system, one that maximizes performance while show how the knowledge should be acquired and minimizing redundancy and inefficiency. one to see the consequences of its removal. The Consider the analogy between a student and experiments illustrate the (un)importance of the an expert system. Both go through a period of knowledge routines as they interact with each learning in which the objective is to raise abilities other. to a desired level of competence. However, the Each experiment followed an established tech• student attends schools in which the curriculum is nique [3-5] and consisted of a series of matches organized so that new knowledge builds upon the between versions of Phoenix with differing old. It would be absurd, for example, to teach amounts of knowledge. A match consisted of 20 quantum mechanics as part of a grade 1 course. In games, with each opponent playing the white and contrast, expert systems are "taught" in an ad hoc black side of 10 starting positions. The accumula• manner. There is no established method for tive knowledge experiment starts initially with the teaching an expert system, nor guidelines for basic tactics program, T, and uses it to play a organizing a curriculum. series of matches against T supplemented with a In this paper, a series of experiments with different knowledge routine for each match. This chess knowledge is reported illustrating some of allowed us to measure the effectiveness of each the difficulties with adding knowledge to an expert expert component relative to a program with no system. Many of the problems are analogous to such knowledge. This process was repeated by those a student encounters when enrolled in a gradually expanding the basic program T with poorly designed education program. The solutions more and more knowledge and using it to identify are often similar to the way they are solved by the next best piece to acquire. The removal exper• educators. The results of the experiments give iment starts with Phoenix and, using the same some insight into the difficulties of "teaching" technique, gradually eliminates knowledge. This expert systems. allows us to measure the importance of the rou• t A competitor in the most recent World Computer tines relative to a program with complete Chess Championship [1]. knowledge. 586 J. Schaeffer and T. Marsland To limit the scope of the problem, we have program's performance with human abilities. The restricted our attention to the acquisition of average club player has a rating of about 1400. middlegame knowledge. It was therefore neces• The details of the rating formula are not impor• sary to remove any influence that other phases of tant and are discussed elsewhere [2]. Pegging the the game might have. Opening specific informa• basic T program with a rating of 1110, experimen• tion was reduced by chosing diverse starting posi• tally determined and consistent with others [3], tions for the matches that were each ten moves yields Table 2. Note that the version of Phoenix into the game. Tree searching techniques were used has a tournament rating of 1840, close to the factored out of the experiment by having all pro- predicted 1786. grams use the same parameters and search to a The tables support the well-known result [6] depth of 5 ply. A game was considered over when that the most important heuristic is Space and it either ended in checkmate or draw, or when Mobility, since it gives Phoenix almost half its rat• Phoenix determined that an endgame had been ing points. Space and Mobility is the simplest reached. In the latter case, the final position was routine to implement and requires no real expert adjudicated. knowledge. In some sense, SM can be viewed as A difficult problem was posed by cases where the first lesson in the education of a chess pro• there was no clear-cut winner, but one side had gram. accumulated positional advantages that in the After SM, it appears that the law of diminish• long-term may prove decisive. Since the ing returns takes over. Additional knowledge pro• opponents have slightly different models of what is vides fewer rating points for increased effort. It is important, it was possible for both sides to think interesting to note that the three smallest gains that they had the advantage! The notion of a (IS, PL, and PS) were for the three largest routines superior or inferior position was introduced to containing the most heuristics. The inclusion of IS ensure that the advantage of long term factors appears to have a negative effect on performance, could be considered even though the material bal• although the difference between 9.65 and 10 is not ance was equal. To ensure impartiality, the adju• significant (as verified by other experiments). This dications were performed by the chess program is an example of knowledge used as a building block) Cray Blitzt. Adjudication resulted in a position IS by itself may not be significant, but its presence being assessed a value in the range 0 to 1 with a provides the environment necessesary for effective win (worth 1 point) defined as the side to move use of subsequent additions. The rating gains being up a full pawn. If the position was not won, obtained by adding knowledge appear to decrease a value was assigned reflecting how (un) favour• steadily, except for King Safety (see Table 2). able the position was, with a value of 0.5 for a bal• This anomaly may be explained by the observa• anced position. tion that KS is not an important factor in most positions, and in many games has little bearing on 3. Results the play. The results of the accumulative knowledge Table 3 presents the results of the diminishing experiment are summarized in Table 1. Each row knowledge experiment. Whereas SM is the first gives the result of the matches between a base piece of knowledge that one would give to a pro• variant of Phoenix (T with 0 or more pieces of gram, PW is the most valuable to retain. When knowledge added) and that program supple• Phoenix is supplemented by all the knowledge rou• mented by the piece of knowledge specified in the tines, PW plays a much more important role than column heading. For example, the entry in row 3 it does when working in an environment with little and column 6 says that the base program knowledge. This illustrates that some knowledge T + SM + CC lost by a score of 9.35 to 10.65 to needs the right environment with which to T + SM + CC + KS. Note, however, that the interact to achieve best results. An analogy might acquisition of knowledge cannot be done in an be teaching new material to students who do not arbitrary manner. Consider row 3 column 7, have the proper pre-requisites. where the base program won by a score of 11.15 to The results of matches involving PS in both 8.85 over itself supplemented by PS. This, and Tables 1 and 3 are interesting in that by removing similar apparent anomalies, illustrate that the PS the program often plays better! Most of the haphazard addition of knowledge may not be knowledge in PS is sophisticated, in the sense that effective until necessary basic knowledge is in it builds on many elementary concepts that would place. be taught early in any chess education. One possi• These results can be put into a more familiar ble explanation of the results then is that Phoenix form by expressing them as chess ratings, which does not know enough to use PS properly. provide a convenient means of equating the Another possibility is that PS has not been imple• mented correctly; either there is a bug in the rou• t A portable version of the current World Computer tine, or the knowledge has not been properly Chess Champion (l], but running on a VAX 11/780.