What Has the Loebner Contest Told Us About Conversant Systems?
Total Page：16
File Type：pdf, Size：1020Kb
What Has the Loebner Contest Told Us About Conversant Systems? Kenneth R. Stephens Operant WebSites, Inc. T he Standard Turing Test, proposed by Alan Turing in 1950 as a test of whether machines can actually think, is an exercise in making human judges think a computer program is actually a human. The annual competition for the Loebner Prize, which has been held every year since 1991 (with the Cambridge Center for Behavioral Studies as the original sponsor), was intended as the “first instantiation of the Turing Test.” No competitor has yet come close to winning the $100,000 Grand Prize, but each year a smaller award and the Bronze Prize are awarded to the most human-like contestant. After eleven years of contests, it is appropriate to look back at the most successful systems of the past, and in doing so, try to answer the following questions: 1. What have been the most successful techniques? 2. To what extent have these mirrored the state-of-the-art in computational linguistics and natural language processing? 3. Are we any closer to a truly intelligent computer, and is the Loebner competition the right way to find out? Contents: . Non-Loebner Conversant Systems . Loebner Contest Winners . Conclusions . References Non-Loebner Conversant Systems W e must begin our review of conversant systems several years before the first Loebner competition, as some of the winners were descendants of two early programs — ELIZA and PARRY — and the SHRDLU system is a benchmark for any intelligent conversational system. ELIZA ELIZA was a program written by Joseph Weizenbaum (Weizenbaum, 1966) from 1964 to 1966 while he was a researcher at M.I.T. ELIZA carried on a “conversation” in probably the most limited manner possible — by reacting as though it were a Rogerian therapist, it simplified its task considerably. Psychotherapists trained in the tradition of Carl Rogers tried to be “nondirective.” Rather than asking leading questions that might steer the dialog in the direction of the topics that interested Freudian and Jungian psychoanalysts (“Tell me more about your sexual feelings toward your mother”), Rogerians simply turned the clients' statements around to show some comprehension but to elicit more statements from the client which might eventually lead them to an understanding of their own conflicts and problems. (In practice, a videotape analysis by Truax (1966) showed that even Carl Rogers was directing the dialog through subtle verbal reinforcement of themes he found interesting.) At any rate, a Rogerian therapist did not have to initiate new themes in the conversation, and as the ELIZA program demonstrated, did not have to exhibit any knowledge or understanding in the traditional sense. It only had to maintain the illusion of carrying on its end of the conversation, which made it an ideal candidate for a system given the limited capabilities of computer programs in the 1960s. The computational task of ELIZA was 1) parse the input from the user to find keywords, and to identify the most important keywords to which it should respond 2) use rules and templates to transform the input (turn “I” into “you,” wrap key statements in such contexts as “in what way do you think __”, “does it please you to believe __”, “how long have you been __”, etc.) into a response that echoed the input statement or asked for more discussion around the keyword 3) if the user input contained no identifiable keywords or content, generate a reasonable response. There was no overall script to the conversation, other than a list of transformational and decomposition rules — ELIZA was purely reactive. Weizenbaum was surprised and dismayed by the extent that humans accepted ELIZA without question. In his 1976 book, he pointed out some of the concerns that led him to drop this line of research: psychoanalysts believed that this was the basis of an acceptable, automated therapeutic regime in the future, users quickly became emotionally attached to the program, and some learned people felt that ELIZA had demonstrated a general solution to man-machine communication. To some extent, perhaps these reactions reflected the lack of general understanding by most people at that time of the capabilities and limitations of computers. PARRY A few years after ELIZA was written, a group of computer scientists at Stanford AI Lab led by Kenneth Colby produced a system called PARRY that simulated the dialog between a therapist and a paranoid schizophrenic. PARRY modeled a paranoid with distorted belief systems and inability to stay on topic for very long, and was so successful it was used to train psychologists, many of whom were unable to distinguish PARRY's dialog from that of a real paranoid patient. In that sense, it might be said that PARRY satisfied a limited form of the Turing Test even in the mid-1970's. PARRY was more challenging to develop than ELIZA, because it could not simply reflect what the user had said, but had to have something to say as well. However, since the output was often gibberish or a non sequitur, it might fool psychologists into thinking they were interacting with a schizophrenic, but would not do well in a test in mimicking a normal conversation. Nevertheless, the PARRY system influenced some of the later Loebner contestants. In fact, a version of PARRY competed in the early Loebner contests. SHRDLU In 1972, Terry Winograd published his doctoral research at MIT on the SHRDLU system. SHRDLU simulated the actions of a robot interacting with a “blocks world” of different colored and shaped blocks which could be placed on a table or put in a box. It maintained an internal model of the state of the different blocks (where they were, their position relative to other objects, what manipulations had been done to them lately, etc.). SHRDLU could carry on a conversation about the blocks world with its user, and was very sophisticated as far as sentence parsing of input commands and questions (its replies, however, were fairly limited and even boring.) In the sense that it answered correctly about the status of the blocks world, it showed what could be called limited understanding of its virtual environment. It is indeed impressive to examine some of the early logs, as it appears that the level of discourse supported by SHRDLU is at a much higher level than contemporary systems. (However, it is important to note that if the commands or questions had been phrased a little differently, the system might not have appeared so robust. It is also important to note that it was relatively ignorant of the very things it was supposed to know about, so that it had no knowledge that blocks were heavy, that they dropped if moved off the table, that larger blocks weighed more than smaller, etc.) SHRDLU can be considered a knowledge-based system. It employs “procedural representations” of words and sentences, so that functions written in LISP and MICRO-PLANNER embodied the meaning of a command or question, and the reasoning capabilities of SHRDLU existed within these programs. A parsing module, called PROGRAMMAR, was written to implement an approach to “systemic grammar” which was a different method of parsing than methods like backtracking (in which grammar rules are applied with identification of choice points, and if parsing is blocked, the process backtracks to the choice point and tries a different rule) or parallel processing (in which many different possible structures are built concurrently, and then the best is chosen). The details of systemic grammar, as well as concepts such as context-free grammars, augmented transition networks, semantic networks, knowledge representation schemes such as frames, rules, etc., and parsing strategies are beyond the scope of the present paper, but the interested reader can find background on these techniques in artificial intelligence texts (e.g. the chapter on “Understanding Natural Language” in Barr and Feigenbaum (1981). SHRDLU employed much more powerful techniques than did ELIZA. Loebner Contest Winners H ugh Loebner has said “I believe that this contest will advance AI and serve as a tool to measure the state of the art.” By contrast, in a review which was critical of the Loebner Prize, Shieber (1994) said “...the Loebner prize competition neither satisfies its own avowed goals [of “pushing the envelope”], nor the original goals of Alan Turing.” Whatever was true in 1994, let us review the history of the entire competition and its winners to see whether the competition has become more challenging, and to what extent it reflects the state of the art. PC Therapist The winner of the first Loebner competition in 1991 was a program called PC Therapist III, written by Joseph Weintraub. It won in the category of “whimsical conversation,” which was perfect for the kind of non sequitur it would deliver. Several of the later winners of the competition have dismissed Weintraub's system as nothing more but an ELIZA clone, but in truth, Weintraub was so disappointed in the version of ELIZA which he had purchased, that he returned it and set about to write his own. Weintraub won the competition in 1991, 1992, 1993, and again in 1995. Two of his other systems were called PC Professor (which was prepared to talk about the differences between men and women) and PC Politician (which discussed liberals and conservatives.) PC Therapist was touted to employ “AI sentence parsing and knowledge base technology.” If it could not construct a reply based on keyword parsing, it would pull a relevant quote or phrase from its knowledge base, called KBASEK. TIPS In 1994, the winning program was written by Thomas Whalen of the Communications Research Centre, in Ottawa, Canada.