Reprinted by permission from lEEE TRANSACTIONS ON MAN-MACHINE SYSTEMS Vol. MMS-11, No. 4, December 1970 Copyright© 1970, by the Institute of Electrical and Electronics Engineers, Inc. PRINTED IN THE U.S.A.

AI in CAI: An Artificial-Intelligence Approach to Computer-Assisted Instruction

JAIME R. CARBONELL, member, ieee

Abstract The main purpose of the research reported here is to powerful than existing ones, to prove that it is feasible, show that —a new and more powerful type of computer-assisted and to demonstrate by example some of its major capa- instruction (CAI), based on extensive application of artificial- bilities. In the course of this investigation, a set of com- intelligence (AI) techniques, is feasible, and to demonstrate some puter programs, system was written. of its major capabilities. A set of computer programs was written the scholar [I], and given the name scholar. Due to its complexity, only the scholar is capable of reviewing the knowledge of a stu- conception and educational aspects of this system (including an dent in a given context (e.g., geography of South Amer- actual on-line protocol) are presented in this paper. ica) by maintaining a mixed-initiative dialogue with him (AFO) In what may be called conventional ad /ioc-frame-oriented in a rather comfortable subset of English. Fig. 1 presents the data base consists of many "frames" of specific pieces CAI, fragment of text, questions, and anticipated answers entered in advance by a of a protocol taken on-line, demonstrating the teacher. By contrast, an information-structure-oriented (ISO) some of the basic capabilities of scholar. In this proto- CAI system is based on the utilization of an information network col, scholar starts typing after being called. The stu- of facts, concepts, and procedures; it can generate text, questions, dent's turn comes each time an asterisk is printed. He and corresponding answers. Because an ISO CAI system can also may respond to a question by scholar, direct a question utilize information network answer questions formulated by its to or such the student, a mixed-initiative dialogue between student and to scholar, state a command as halting, or computer is possible with questions and answers from both sides. changing the mode of the interaction. The student re- turns control to scholar by typing another asterisk and a carriage I. Introduction and Overview return. scholar can prompt the student, indicate when it does HE MAIN purpose of the research reported in this not understand him, detect misspellings, and answer the document is to introduce a new type of computer- student's question using acceptable English, scholar (CAI), many respects more assisted instruction in can also generate questions and evaluate the student's answers, deciding when these are correct, wrong, or only Manuscript received July 20. 1970. This work was supported in approximately or partially correct, and then take some part by the Office of Naval Research under Contract NOOOl4- It -69-C-6223. by the Air Force Electronics Systems Division under conditional actions. keeps track of content and changes Contract Ki9628-69-C-0298. and by the Advanced Research it on the basis of relevancy and time considerations. Agency F19625-C-0125. Project under Contract scholar does all this without specific and detailed direc- J. R. Carbonell is with Bolt Beranck and Newman, Inc.. Cam- bridge. Mass. 02138. tions, but rather by applying general criteria and pro- oarbonell: ARTII'IOI.U.-INTELUOKNCE approach to CAI 191 cedures to a body of "knowledge" about the subject be- from the student, thus using its semantic network for ing discussed. question-answering purposes. This explains why mixed- Conventional tutorial CAI systems depend for their initiative dialogues between man and computer are now operation on the utilization of ad hoc blocks of material, possible. The use of a semantic network' also facilitates usually called frames, entered in advance by the teacher. the two-way communication in a rather large and free The material in the frames usually includes paragraphs subset of English. of English text to be presented, specific questions with The research leading into the development of scholar their expected correct answers, perhaps some expected can also be considered as an investigation on an aspect incorrect answers, keywords, and anticipated branching of man-computer interaction, namely, the possibility of for a limited closed set of predictable alternatives for having mixed-initiative man-computer discourse. This each student's answer. Questions are more often than not mode of interaction is of interest for areas of application of the multiple-choice type. The student has no initiative other than CAL and can ask no questions. We like to call the CAI sys- The subject matter, geography of a given region, was tems based on detailed frames ad hoc- frame-oriented selected as being representative of verbally oriented sub- (AFO) CAI systems. jects with comparatively little inherent logical structure In contrast to the AFO approach, the approach to CAI and contextual algorithms. This type of subject seemed presented in this document can be defined as being in- to be among the hardest to treat in a generative way, formation-structure-oriented (ISO), since it is based on since its structure isrepresented by its descriptive seman- the utilization of a complex but well-defined information tics. Algorithmically, formal subjects like algebra or structure in the form of a network of facts, concepts, and analytic geometry present considerably fewer difficulties procedures as a data base. The elements of this network in terms of both their internal representation and their are units of information defining words and events in the natural language input-output. Success in dealing with form of multilevel tree lists. The elements of those lists them would not have implied immediate translation to are other words that in turn point to their respective verbally oriented subjects [4]. units, and so on. Fig. 2 is a pictorial simplified repre- Changing the region (e.g., Africa) to which geography sentation of a portion of a network of this sort in the procedures are applied presents no problem to scholar context of geography of South America. Each rectangle since only part of the semantic network must be updated. or plane is a unit with a name (Uruguay, Argentina, Changing the application to a different subject (e.g., South America, country, latitude) and a set of symboli- anatomy of the circulatory system) would mean an al- cally coded properties. most complete revision of the content of the semantic The perspective from a given word pointing to the network, but not of the program. Shifting to a more com- corresponding unit, with the elements in the properties putational or algorithmic topic (such as aspects of Span- pointing to other units, and so on, represents the "mean- ish syntax or analytic geometry) would still require prac- ing" of that word. In this sense, the network can be said tically no changes in the program but would imply a to be a "semantic" network. Semantic networks were first semantic network much richer in procedures than that of introduced by the pioneering work of Quillian [2], [3]. more descriptive subjects and procedures, such as to Fig. 3 is a small portion of the actual internal configura- conjugate Spanish verbs or to determine an 'intersection' tion of scholar's semantic network, presenting the unit in analytic geometry. for "latitude," and fragments of the unit for "Argentina." Artificial-intelligence techniques have played a major Observe that no specific pieces of text, questions with role in the development of scholar through its natural their predicted answers, errors, or anticipated branching language processing, utilization of semantic networks, form part of this information structure, as is the case in and question-answering procedures. Existing techniques the data bases of conventional AFO CAI systems. have been combined with original procedures for question In ISO systems, an executive program, almost com- generation and for handling the problems of context and pletely independent of the subject matter to which it ap- relevancy. plies, is now capable of probing the semantic network After some extensions and refinements, we contemplate in order to generate the material to be presented and the the application of scholar to practical situations with questions to be asked of the student. As shown in Fig. 1, large data bases. Because the content-free nature of the this program is at the same time capable of generating executive program forces a precise definition of pro- the corresponding answers to its own questions and of a cedures, scholar constitutes an ideal environment to in- certain degree of branching conditional on the student's vestigate pedagogical questions. responses, while maintaining a continuity of contexts and scholar is currently implemented in the bbn-lisp subcontexts. language in an XDS-940 time-sharing computer. This Furthermore, the data base of ISO CAI systems re- environment was selected principally because of its flects basic "knowledge" about the subject under discus- rather large virtual memory obtained through paging. sion; therefore, scholar can at any time accept questions Conversion to a larger and faster Digital Equipment to to

REPHRASE YOUR STATEMENT. I CAN'T UNDERSTAND IT. "800300* " INCORRECT* "SCHOLAR) YOUR ANSWER IS INCORRECT. VERY GOOD. THE CORRECT ANSWER IS: 214000. THE AREA OF PARAGUAY IS APPROX 157000 SQUARE MILES. ♦♦♦♦+ + + +++ + + ++ + + + + + + + + + + + + + + + ++ + ++++++++ + + +++ + + + + +T-++ + + + THE POPULATION IN CHILE IS APPROX 8500000. PEOPLE TRUE OR FALSE? THE LANGUAGE IN BRAZIL IS FRENCH. HELLO. MY NAME IS SCHOLAR. I WILL HELP YOU IN REVIEWING RIGHT OR WRONG? THE GEOGRAPHY OF SOUTH AMERICA. THIS IS ABC REGIONAL "TRUE* HIGH SCHOOL. TODAY IS 04/29/70. "WRONG* YOU ARE CORRECT. YOU ARE CORRECT

USE ONE OF THE FOLLOWING! THE LANGUAGE IN BRAZIL IS PORTUGUESE. IF YOU HAVE CAREFULLY STUDIED YOUR INSTRUCTIONS, WE IS IT TRUE THAT THE CAPITAL IN BRAZIL IS BRAZILIA? MAY BEGIN THE REVIEW NOW. SUCRE FRENCH ♦YES« ARGENTINA WHEN YOU ARE READY TO START, TYPE READY REMEMBER SPANISH THAT. FINE. TO TERMINATE WITH AN ASTERISK FOLLOWED ---BY A CARRIAGE RETURN. * TO ANSWER THE QUESTION: APPROX WHAT IS THE POPULATION IN BRAZIL? WHAT IS THE LANGUAGE IN CHILE? "60000000. "READY* " APPROXIMATELY TRUE. BUT A MORE EXACT VALUE IS 70808800. PLEASE TYPE YOUR NAME MR.. MISS, OR MRS. FOLLOWED YOU ARE TAKING TOO MUCH TIME.. BY FIRST NAME. MIDDLE NAME--- OR MIDDLE INITIAL. AND LAST WHAT IS BRAZIL? NAME. ♦SPANICH* "TELL ME SOMETHING ABOUT PERU.* *MR. JAMES R. JONEEStStES* YOU SHOULD HAVE WRITTEN "SPANISH". NAME OF YOUR INSTRUCTOR, PLEASE: VERY GOOD. WAIT A MINUTE. I'M CHECKING MY FILES "MR. BROWN* LET'S PROCEED NOW. a THE CAPITAL IN CHILE IS SANTIAGO. _ _W.LICORRECT UKOR INI_KK-.17INCORRECT? H LET'S BEGIN OUR DISCUSSION NOW, MR. JONES. PERU IS A COUNTRY. P "CORRECT* IT IS IN SOUTH AMERICA. y, THE POPULATION IS APPROX 7000000. PEOPLE. » "f++++4^++++++++++++++4^+ ++++++ +++ +++ + + THAT'S(AT'S RIGHT. THE CAPITAL IS LIMA.LIMA q

HrrnuAAPPROX WHATwnHl ISl_ THEIn_ AREAhk_ upOF BRAZIL?BKAZIL7 IWHNOW ANSWERHN_w_K THEih_ QUESTIONUUbSIION YOU DIDN'T ANSWER BEFORE. 85 WHAT IS CHILE? W "2500000 SQUARE MILES* *Q/A* 0O "A COUNTRY* "4 WRONG. g THAT'S FINE. PLEASE INDICATE IF THE FOLLOWING STATEMENT IS CORRECT DO YOU WANT TO CHANGE THE GENERAL MODE OF THE INTERACTION > OR INCORRECT: TO O/A? 2 PLEASE ANSWER V OR N. > SELECT AN ALTERNATIVE FROM THE LIST: THE AREA OF PARAGUAY IS APPROX 47432 SQUARE MILES. > *V* n 214000. "WHAT IS THE AREA OF BRAZIL?* 2 800300. OK, MR. JONES. LET'S GO AHEAD. 49883. £ 63690. THE AREA OF BRAZIL IS APPROX 3600000. SQUARE MILES. "PLEASE, TELL ME MORE ABOUT PERU.* Zj" FOR THE QUESTION: NOW ANSWER THE QUESTION YOU DIDN'T ANSWER BEFORE. THE AREA OF PERU IS APPROX 480000. SQUARE MILES. " APPROX WHAT IS THE AREA OF CHILE? ♦FALSE* THE LANGUAGE IS SPANISH. D Fig. 1. A protocol of a student conversation with scholar. g »_ o

_ CARBONELL: ARTIFICIAL-INTELLIGENCE AI'I'ROACH TO CAI 193

IS IT CORRECT TO SAY THAT THE CITIES IN BRAZIL ARE SAO PAULO, RIO OE JANEIRO, BRAZILIA, PORTO ALEGRE. BAHIA, RECIFE, SANTOS, MANAOS, AND BELEM?

"YES* "HOW MANY SQUARE MILES ARE THERE IN BRAZIL?* THAT'S FINE. THE AREA IS APPROX 3600300. SQUARE MILES. THE CITIES IN BOLIVIA ARE ♦WHAT IS THE PRINCIPAL RELIGION IN URUGUAY?* *LA PAZ, SUCRE, BOGOTA, AND ACONCAGUA* THE PRINCIPAL RELIGION IN URUGUAY IS CATHOLICISM. AT LAST! ANYWAY, LET US SEE WHAT YOU HAVE JUST WRITTEN. * WELL, HAVE YOU MADE UP YOUR MIND? HAVEN'T YOU INCORRECTLY USED "BOGOTA" AND "ACONCAGUA"? ON THE OTHER HAND "LA PAZ" AND "SUCRE" IS CORRECT. *THE PRINCIPAL RELIGION IN PERU IS .* THE COMPLETE ANSWER ALSO INCLUDES POTOSI, COCHABAMBA, I'M AFRAID I CAN'T ANSWER THAT. AND SANTA CRUZ.

"IS IT TRUE THAT THE AREA OF URUGUAY IS 70000?* THAT'S ABOUT RIGHT. THE EXACT ANSWER IS 79000. SELECT AN ALTERNATIVE FROM THE LIST: 845600. 1033500. ♦COULD YOU PLEASE INDICATE THE PRINCIPAL RIVERS IN -8297216. ARGENTINA?* 4000000.

BELOW: I DON'T UNDERSTAND YOUR WORD "INDICATE". I'LL GIVE TO FILL THE BLANK YOU ANOTHER CHANCE. THE POPULATION IN BOLIVIA IS APPROX PEOPLE. ♦THE PRINCIPAL RIVERS IN ARGENTINA ARE " "40 THE PRINCIPAL RIVERS IN ARGENTINA ARE THE RIO DE LA PLATA. 00 YOU ARE TAKING TOO MUCH TIME.. THE PARANA, THE URUGUAY RIVER. THE SALADO, THE PARAGUAY RIVER, AND THE PILCOMAYO. "4000000.* THAT'S RIGHT. "MIXINIT* TYPE EITHER T OR F AFTER THE FOLLOWING STATEMENT: DO YOU WANT TO CHANGE THE GENERAL MODE OF THE INTERACTION TO MIXINIT? THE LANGUAGE IN BOLIVIA IS SPANISH. PLEASE ANSWER V OR N. "WHAT IS CORDOBA?* *V*«

OK. MR. JONES. LET'S GO AHEAD. CORDOBA IS A CITY. IT IS LOCATED IN ARGENTINA. WHERE IS BRAZIL? THE POPULATION IS 700000. PEOPLE. NOW ANSWER THE QUESTION YOU DIDN'T ANSWER BEFORE. PLEASE HURRY UP. I'M WAITING FOR YOUR STATEMENT. ♦♦♦♦ ♦♦ ♦♦♦♦+♦♦♦♦♦♦♦♦♦+♦♦♦♦+♦♦♦♦♦ +♦♦+♦♦♦+♦♦+ +«■«"♦♦♦■»" +♦ + + "IN SOUTHAMERICA* + + + TODAY'S REVIEW IS OVER. I BELIEVE YOU MEANT TO TYPE "SOUTH AMERICA". IT HAS BEEN A PLEASURE TO WORK WITH YOU, MR. JONES.

THAT'S RIGHT. ♦+♦♦♦♦♦♦+♦♦+++♦+♦♦♦++♦♦♦+♦♦♦♦+♦♦++♦+♦♦♦+♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦+

Fig. 1 (cont.).

Corporation PDP-10 time-sharing system with paging fers to an example of some diagnostic process (medical is under way. or otherwise) that the student must perform. The third category in Bryan's taxonomy is called con- 11. General Background trolled learning and implies detailed anticipation and branching in a Crowderian sense. Drill-and-practice and A. Traditional Approaches to CAI tutorial efforts involve the construction of frames of Computer-assisted instruction efforts have in the past questions with anticipated correct and wrong answers few years proceeded along several lines. Bryan's [5] and, perhaps, keywords to be extracted from the stu- » classification distinguishes three broad categories. In the dent's answer. Sequencing is traditionally deterministic. first, ad lib CAI, the student is given access to a com- We have called these systems ad /ioc-frame-oriented puter (including one or more languages and perhaps a (AFO) CAI systems, eliza, described by Taylor [B], is library of routines), but he is in full control: his input one of the most interesting CAI systems of this kind. is not controlled by the computer, logo, developed by If we examine the typical CAI tutoring systems, it is Feurzeig and Papert [6], provides one of the most inter- easy to detect some of their basic limitations. The student esting environments for efforts of the ad lib kind. has little or no initiative, he can not use natural language The second category is games and simulation, where in his responses, and systems usually look fairly rigid the student has some initiative but is constrained by the to him. The teacher has a considerable burden in the rules of the game or the logic of the simulation. The preparation of questions, answers, keywords, and branch- Socratic system |7] is a program where all possible ing. From a system's point of view, the system controls branches in a huge tree of alternatives (with possible the student but is in turn tightly ad hoc programmed })y loops) must be specifically programmed. That tree re- the teacher, the system has no real initiative or decision 194 lEEH TRANSACTIONS ON MAN-MACHINE SYSTEMS, DECEMBER 1970

(BPAOQ LATITUDE (((CN LATITUDE) (DET THE DEF 2) ) NIL (SUPERC NIL (DISTANCE NIL ANGULAR (fFOtl NIL EOUATOR) ) ) (SUPEBP (I 2) , LOCUTION) (VALUE (I 2) (RANGE NIL -90 90) ) (UNIT (I 2) DEGBEES) ) )

(RPAt>O ARGENTINA ({(XN ARGENTINA) (DET NIL CEF _) ) NIL (SUPERC NIL CO'JtiTHY) (SUPIBP (I b) SOUTH\ArtEHICA) (AREA (12) (APPROX \IL \ 1200000)) (LOCATION NIL SOUTH\AMEBICA (LATITUDE (I 2) (RANGE NIL -22 -55)) Fig. 2. Representation of a portion of a network on South (LONGITUDE (I 1) America. (range nil -57 -71 ) ) (BORDfcRI!)G\COUNTEIES (I 1) (NORTHERN (1 1) BOLIVIA PARAGUAY) power of its own, and, of course, it has no real "knowl- (EASTERN (I 1) ( (S.L BRAZIL UHUGUA edge." NIL In most CAI systems of the AFO type, the computer (SOUNDAKT NIL ÜB.UGUAYARIVEB) ) ) does little more than what a programmed textbook can do, and one may wonder why the machine is used at all. Some systems allow some degree of processing of unan- (CAPITJL (I 1) ticipated answers, time can be measured, and statistics HU£NOS\AIRES) are collected in most cases, but not much more. (CITIES (I 3) {PRINCIPAL .IIL (JL BUE»OS\AIR_S CORDOBA BOSA_IO tIttJUOZA LA\PLATA TUCUBANM) Computer Capabilities in CAI (TOfoaaApHT (i i) B. Better Use of the VARIED (MOUNTAINVCHAiNS NIL (PRINCIPAL NIL ANDES We think that a clue to the basic nature of important (LOCATION NIL (BOUNDARY NIL (KITH NIL problems can a examination of man- CHILE) ) ) be found in close (ALTITUDE NIL (HIGHEST NIL ACONCAGUA computer interaction ; especially the decision and control (APPROX NIL 22030) ) ) ) (SIEBRAS Nit (LOCATION NIL (SL CORDOBA aspects [9]. The control of the teacher-student interac- BUgoOS^ _BES) ) ) ) (PLAINS NIL (MSBTILE NIL USUALLY) tions can be shared in a mixed-initiative discourse, since ((tL EASTE3N CENTRAL) interaction is basically a NIL PArtPA) man-computer communication (NORTHERN NIL CKACO))) between two information structures, including their com- putational capabilities. This leads to a close examina- Fig. 3. The units for latitude and Argentina in scholar's seman- tion of the data bases and information-handling capa- tic network. bilities of CAI programs. Simmons and Silberman [10] concentrated on the prob- data base, e.g., "capital" and "profit." Shortest paths lem of natural-language processing as a means to handle are found and reported first; more intricate relations are unanticipated answers. They continued to use frames and added only on request of the user. detailed prespecification by the teacher of correct an- Uhr [13] was perhaps the first to use the word "gen- swers and actions. Recently, Simmons [11] has de- erative" for a system that could originate some of its scribed new work that includes processing of the instruc- questions. His early attempts included generating ran- tional data base as well as student answers, and still dom numbers for drill and practice in elementary arith- highly focused on natural language comprehension. metic operations and converting descriptive text ma- At the Massachusetts Institute of Technology Sloan terial into first-draft programmed instruction. Much School, Rockart et al. [12] have developed a sys- more elaborated programs that have been more recently tem using a semantic representation of elements of ac- created by this group follow. counting. It is capable of several modes of operation, Uttal et al. [4] designed and implemented an interest- including question answering and testing, but no real ing generative system for teaching analytic geometry, mixed-initiative mode exists. The analysis of student in- where generation of random values is used to determine put is based exclusively on keywords ; the program stores coefficients of quadratic equations. Equations are manipu- English text for definitions and other purposes; it uses lated, partitioned, and otherwise elaborately used for a detailed agenda and it is not generative but presents ex- presentation and diagnosis. In this system, there is no ercises entered as such by the author. The system makes information structure, since "the idea of generation an interesting use of the semantic representation to find hinges completely upon . . . algorithmic manipulation." relations via intersections between two concepts in the And later: ". . . we must exclude verbally oriented sub- carbonell: artificial-intku.igf.ncf. approach to cai 195 jects as possible items for a generative curriculum.'' A. Sonic Pedagogical Considerations scholar is an attempt to disprove this conclusion. The first point is fundamentally related to the differ- Very recently, Wexler [14], working with Uhr, com- ent philosophy in ISO CAI versus the traditional AFO pleted the development of an interesting system that in CAI. In the latter, the computer is given as its data base certain respects is related closely to our own effort in items that it must manipulate literally, with no latitude developing scholar. The main area of application used to draw inferences and generalizations. The computer in by Wexler has also been geography of a certain area. He AFO CAI systems has no "knowledge"; it merely re- also utilizes information nets. Wexler defines questions gurgitates text, questions, and answers that have been and related support for student-computer interactions by specifically entered in advance. It is unwise and even means of what he calls skeletons, which are functional unfair to pretend that, under those conditions, the com- frames with arguments that can be randomly generated puter could even approximate the behavior of a human within prescribed sets of alternatives. Skeletons are also teacher. In his teaching process, a human teacher is not used to compute whether or not the student answer is reciting specific questions, but he is utilizing and process- correct and to answer related questions by the student. ing knowledge he has stored in the form of a semantic information structure. Furthermore, his exploration re- C. Some Relevant Artificial-Intelligence Aspects quires the utilization of information not only about the specified subject on hand, but also about more general The goal of artificial-intelligence research has been de- knowledge. For instance, asking a question about the fined as "to construct computer programs which exhibit average temperature in Montevideo and processing the behavior that we call 'intelligent behavior' when we ob- corresponding student answer may require knowing that of serve it in human beings [15]." The development temperatures are measured in degrees Fahrenheit, and direction, scholar is, to a large extent, an effort in this that nowhere on earth is the average temperature above, field of and can be legitimately considered to be in the say, 120°F. artificial intelligence: answering questions not specifically The development of scholar is a step in the direction constructing questions on given topics, and anticipated, of CAI programs that know what they are talking about, carrying on a mixed-initiative contextual dia- generally the same way human teachers do. This necessary preoc- a and comfortable logue with a human in rather free cupation with properly representing and intelligently subset of English. utilizing knowledge has led us to the use of a semantic intelligence most important The area within artificial network for the data base and, generally, an artificial to knowledge structures, essen- for us is that related the intelligence approach in the program. tial basis for the ISO approach. In this sense, work on The second pedagogical consideration is that CAI can semantic information structures is highly relevant, and benefit from a close look at human teachers. Clearly, they Minsky's collection [16] is an important reference. But do not act on the basis of stored questions, answers, we have been specially influenced by the work of Quillian etc., but on knowledge, in the form of facts, concepts, on semantic networks. It became clear early in our re- and procedures. Also, human teachers are not prepro- some form of a semantic network provided search that grammed to the ultimate detail, as is, for example, the kind of data base capable of supporting an ISO CAI the Socratic system [7|. Instead, they have general pro- with the characteristics that scholar now exhibits. system cedural guidelines and criteria that depend on their in- area of relevance is natural language com- The second formation structures and also on recent events (such as munication with computers. Bobrow [17], Quillian [3], a student's response). and Simmons [18] have adopted an approach in which Following Minsky [16], we would like to argue against both syntax and semantics play an important role in absence of some rational criterion language comprehension. They have emphasized the se- the notion that, in the decision, teachers decide because of their own "voli- mantic aspects, i.e., what questions and other statements for tion." As Minsky says, ". people are incapable of mean rather than how they are structured. . . explaining how it (free will or volition) differs from I The third area of artificial intelligence that directly stochastic caprice. . . ." On the other hand, we do not relates to our work is that of question-answering sys- an understanding so complete as to explain all their tems. Kellogg [19] describes a rather complete system have on a rational deterministic basis. In designing a with good data-base building facilities. It also has fairly acts in the domain of artificial intelligence, we still interesting inferential capabilities for use in question program want to preserve the richness of variety of human answering, such as comparing, counting, finding the larg- often decisions, even if we can not or should not program all est element, etc. of them in their ultimate details. The behavior of scholar looks richer and "more intelligent" than it 111. Technical Discussion would if questions were consistently selected by some arbitrary) criterion In this section, we present some relevant technical deterministic (but, perhaps, equally within data base. problems in CAI, their role in the development of ISO like ordering the natural language systems in general, and of scholar in particular. A third point is the importance of 196 lEEE TRANSACTIONS ON MAN-MACHINE SYSTEMS, DECEMBER 1970 communication between man and computer. We think forts. The precise thinking and the generality required this is an important goal, and we have made a sub- by ISO CAI with its information-processing formulation stantial effort to achieve dialogues in a rather large and will translate, we hope, into our better understanding of comfortable subset of English. The results have been the processes of teaching, learnings and personal verbal surprisingly good, as protocols illustrate. In spite of this, communication. we that the importance of natural language consider B. Semantic Networks communication for CAI has been somewhat exaggerated. This is especially so in the case of researchers who have We will briefly describe here semantic networks and neglected other aspects to concentrate on natural lan- their general characteristics. As indicated above, semantic guage understanding. networks stem from the pioneering work of Quillian A fourth point concerns the processing of unanticipated [2] in natural language comprehension. answers and the associated and frequently mentioned Semantics is the science of meaning. In linguistics, need to construct a model of the student. This will make semantics is concerned with the "deep" structure of sen- it possible, it is argued, to process his errors, study his tences, i.e., with what the words and their modifiers stand misconceptions, and take some remedial action about for and how different words affect each other; on the them [B]. But that modeling task is not easy, and the other hand, the way they are organized sequentially great difficulty of constructing from scratch a model of within a sentence is in the domain of syntax. A semantic each student has been a major stumbling block for many information structure is an organization of units of in- investigators in the area of computers and education. formation in terms of their meaning and mutual rela- Our approach is different: having the semantic net- tionships. When each unit in the set may refer to other work as an information structure on the subject being units within the set, which in turn refer to other units dealt with, it seems natural to consider it as an input- in the set, and so on, with the possibility of loops and output model of the ideal student. It is so to the extent cross-references, we have a semantic information net- that the semantic network, when interrogated, would work. give the same answers a "perfect" student would. In Fig. 2 was presented in Section I as a pictorial repre- other words, we are not claiming that a perfect student sentation of a portion of the semantic network on the has his knowledge organized strictly the way the seman- geography of South America. Fig. 3 then represented a tic network is organized. We simply claim that both fragment of the unit corresponding to "Argentina" plus would produce, when interrogated, essentially the same the unit corresponding to "latitude." output. Units are the basic components of semantic networks The modeling of a student is made much easier by and may be thought of as pieces of information to which giving him the benefit of the doubt and assuming he is we usually associate a name. However, there is no one- correct until proven wrong. The practical advantage is to-one correspondence between units and names, since that we start with a complete model (the "ideal" struc- some units have no single word as a name and some ture), which is closer to a model of a real student than have several. Each unit is composed of semantic informa- starting from zero. From a practical point of view, we tion about the unit in the form of a set of properties. need to be much less worried about modeling with our In scholar, the first element of each property is the approach than with the classic and rather unsuccessful name of the property (attribute), the second element is "building-from-scratch" approach. a set of tags used by the executive program, and the The final point on pedagogical considerations concerns rest is the value of the property. A value can either be our perspective on the development of an environment a set of properties or a pointer to a unit (or a set of for further research and development. We are interested units) modified by other properties. This allows multiple in the investigation of some important pedagogical ques- embedding. In Figs. 2 and 3, already presented, proper- tions, such as the following. What is an effective tax- ties are delimited by sets of parentheses. Special symbols, onomy of question types, from both a semantic and a such as $L, are used to indicate that what follows is a syntactic point of view? What classification of errors list of pointers to other units. should be utilized, if it is to apply regardless of the Through its different properties and their constituents specific subject matter? What different efficient tech- (attributes and values) , each unit points to other units. niques can possibly be defined and used for diagnostic "Argentina" points to "latitude" since the latter is the and remedial purposes? attribute of a property of the former. The entry "lati- The human teacher, neither in his personal teaching tude" in "Argentina" points to all the information about nor in his preparation of conventional CAI frames, is latitude, and similarly, having "Buenos Aires" as the ever confronted with these questions since he is always value of the property "capital" of Argentina makes Ar- aware of the subject matter in the most specific form. gentina point to all the properties of Buenos Aires, its But a program like scholar needs answers for these capital. This avoids unnecessary repetitions since practi- questions. And we would further like to claim that a cally all information is stored only once. better understanding in this respect could eventually ben- Units form a complex network of facts, concepts, and efit education not only in computerized but general ef- sometimes procedures; the latter have been mixed in carbonell: aktificiai.-intf.li.ici:mi: approach to cai 197

scholar with descriptive information. An example of a time, it seems natural to put those two properties in procedure within the information structure is that for parallel formally, which implies the same formal depth. inferring the climate of a place given certain local con- The solution can be obtained through asking the person ditions like latitude, altitude, etc. In other words, if constructing the network to tag the properties in order the climate of a place is not given factually (in terms to modify their relevancy without changing their posi- of temperature, precipitation, etc.), it can be inferred tions within a unit. with good probability of success knowing the latitude, The "subjectivity" of the network evidenced by tag- altitude, etc. ging is not an artifact. Two equally knowledgeable teach- Storing information in a semantic network has dis- ers would create semantic networks with slightly differ- tinct advantages for CAI and for other interactive man- ent configurations when dealing with the same subject computer systems generally. Assuming that human sym- matter. This is not an exclusive characteristic of ISO bolic memory has an organization in the form of a CAI. In AFO CAI, the same two teachers would create semantic network, a machine using such an organization two different sets of questions. And for that matter, they will be working with the same kind of information struc- would have different behavior in a classroom. tures as a man. For an information-retrieval system, With respect to contextual continuity and topic cover- the advantage of this kind of compatibility is that the age, an AFO system has a large degree of anticipation. organization in the computer provides retrievability ac- In ISO systems, we also can build an adjustable degree cording to the dimensions that the users consider rele- of overall anticipation at a macrolevel using a more or vant. For a computer-assisted instruction system, the less detailed agenda. Similarly, if desired, anticipation advantage lies in the fact that the type of organization can be obtained in ISO systems at a more detailed level of the knowledge to be learned by the student is not by utilizing strategies for material presentation and ques- far from that of the information stored in the computer. tion generation that are more or less deterministic and sequential, with limited branching. We think that these macro- micro-strategies limit the power of ISO CAI C. Relevancy and Context and systems; therefore, in the current implementation of If we are going to let a program like scholar carry scholar, we have chosen to limit anticipation to a min- on its own mixed-initiative dialogue with a student with imum, with an agenda reduced to an overall context. no anticipation of the details of that dialogue, we must Natural Language in and Answers give such a program the capability to deal with the con- D. Questions cepts of relevancy and context. Quillian [3] has appro- In scholar, we have been able to achieve a large de- priately said that in a semantic network the meaning gree of freedom for expression in English, better than our of a word, phrase, sentence, or event is the whole net- early hopes, both in input and output (i.e., in compre- work as seen through it. hension and in generation by scholar). This pragmatic The notion of contextual relevancy is all important for approach has proved very successful ; instead of attempt- maintaining continuity and meaningfulness in the dia- ingto comprehend all classes of input, we have restricted logue, by asking contextually relevant questions, and by student answers to scholar questions to certain types, answering student's questions with relevant information namely, numerical, atomic, and lists of atoms, though and not everything that could possibly be said about the other elements like auxiliary words can also appear. questioned matter. For example, suppose the student The underlying reason has not been difficulties in pars- asks, what is Montevideo? We would say that it is a ing complete sentences, but in judgingtheir acceptability city, the capital of Uruguay, and perhaps give its pop- as answers. The above limitation has represented a trade- ulation size, but not details like the average precipitation off, since as a consequence of it, we had to be more in Montevideo during the month of January. demanding in the generation of questions in order to We would like to have a metric to define the relevancy produce expected answers only of the types mentioned of a property or fact in terms of a given concept; it is above. We now feel confident that an extension to sim- possible in a future version I easier to establish a metric for irrelevancy, e.g., the ple complete sentences will be distance in a graph-theoretic sense from one node to of SCHOLAR. another in the semantic network. Since all elements The case where we have allowed complete sentences within a given distance of a node could be said to be with a large degree of freedom is that of questions asked within the context of that node, that maximum distance by the student. We have decided to leave special cases thus acts as a threshold of relevancy. aside and concentrate on methods to solve most practical The graph-theoretic sense, however, docs not seem to ones. When scholar can not comprehend a student's us to be refined enough to be capable of handling all question, it so tells the student asking him to rephrase necessary cases. For example, it would not discriminate the question; if words unknown to scholar appear, it between two equally deep or distant properties of an ob- points them out. This is, after all, what a human would ject, one subjectively important, the other less so. For do. generation example, the latitude of a city seems subjectively more A similar approach has guided the of important or relevant than its longitude. At the same English sentences by the computer. The strategy adopted ON SYSTEMS. DECEMBER 1970 198 lEEE TRANSACTIONS MAN-MACHINE classify that answer in terms was that of using short sentences with no embedded memory, in an attempt to relevancy relation to the ques- clauses and a limited repertoire of verbs. It has proven of compatibility and in compatible and to be highly successful. Most programs producing ac- tion. The answer is accepted if it is both rele- ceptably constructed English output do so by a technique relevant. The determination of compatibility and a of replacement within formats, as Weizenbaum's eliza vancy of complex, constructed answers is, however, [20] does. All sentences and questions generated by very difficult and largely unsolved problem. This problem natural-language compre- scholar involve a complete processing from a semantic is related to the whole area of essay ques- internal representation into English. hension. Problems are especially serious when are Per- a mixed-initiative ISO system such as scholar, tions such as "tell me about Argentina" asked. In matching questions asked of the system must be interpreted in haps some solution intermediate between the strategy could terms of the data base; questions asked by the system strategy and the answer-comprehension must be generated from that data base. Therefore, it is represent the most promising approach. wrong in absolute necessary for us to have good understanding of what An answer may not be correct or as are the questions are and what types exist. terms. For example, a question such "what answered with a There are clearly two levels in each question: the countries in South America?" can be plus one semantic aspect (i.e., what the question is about) and list of only nine of them, or with most of them the syntactic aspect (i.e., how the question is formu- Central American country, etc. The error-analysis pro- wrong parts of lated). For example, the question "how many people are cedures must separate the correct and there in Brazil?" refers to the string "70 000 000 popula- the answer, as is done by scholar. tion Brazil" and tells us specifically that in this string, Beyond detecting errors, one would like to classify and under- we questioning the value 70 000 000. In terms of them, take proper actions to correct them, are a the form of the question, we see that it is a "how stand the reasons for those errors to occur. In sense, that many" question that perhaps could be considered as a errors can be considered as the symptoms of diseases then, particular case of many which use a "WH" word (i.e., are the reasons for their occurrences. We are faced, operate on what, which, where, who, etc.). Other types are fill-in, with a diagnostic task but one that should multiple-choice, and true-false questions. an open set of alternatives. It is clear that AFO systems have little or no capa- E. Error Detection, Diagnosis, and Remedial Action bility to perform diagnosis by themselves. They can only the teacher as a The whole area of error analysis, diagnosis, and con- follow specifically the directions left by on predict- sequent remedial action is one of the most promising result of his possible "prediagnostic" efforts possessing a data base avenues open to ISO CAI systems, in sharp contrast to able answers. Only ISO programs, knowledge about the subject limitations of classical AFO CAI systems. In classical organized on the basis of for probing AFO CAI, there is usually full anticipation of correct matter, will have the potential capability to find reasons for an answers as well as certain incorrect ones with their cor- that data base and utilizing it responding branching. Either there is no possibility of unanticipated observed student error, scholar is a pro- basic condi- unanticipated answers (because of restricting questions gram that has been constructed with the good diagnostic capa- to closed-set multiple-choice ones), or there is a category tions to eventually have some though, we have for all unanticipated answers, with a preestablished con- bilities; in the balance of priorities, sequent action. concentrated so far in developing those basic conditions the develop- In ISO systems, while generating a question, we can in scholar, and only to a minor extent, in at the same time generate a correct answer or a set of ment of its diagnostic capabilities. our point of view and answers. It seems natural to use that derived correct In Section 111-A, we presented answer as a standard for matching with the student's current solution to the problem of modeling the student, us answer. This is convenient and is the strategy adopted prompted by practical considerations. Similarly, let a student's in scholar. For this strategy to be effective, it is neces- think for a moment as to whether diagnosis of j for something s ary to have a more or less unique and well-defined cor- errors should be a goal or really a means I rect answer, or a well-defined closed set of correct else. Based on introspection and on observation of other that is the case. The answers, for example, a list of items. On the other hand, human teachers, we think the latter a free complex-constructed responses really represent open teacher usually wants his student to go through certain terminal state sets and can not be checked by matching techniques. learning process in order to achieve some knowledge or As we said before, in the present version of scholar, in which that student has acquired certain problems de- our question-generation routines are designed in such a skills or possibly new ways of thinking. If student to over- way that expected answers are either atoms, lists of velop, the ultimate objective is for the diagnose them. This is atoms, or numbers. come them, not for the teacher to teachers in The alternative to the matching technique, which clearly reflected in the behavior of most good teachers sometimes would remove restrictions, consists in a direct interpreta- their classes and discussions. Human students' confusions tion of the student's answer in terms of the semantic try to understand the nature of their CARBONELL: ARTIFICIAL-INTELLIGENCE APPROACH TO CAI 199 and problems, but at least as often, they go into explana- that the language in a country is usually the language tory and remedial sequences without a full understanding in the cities located in that country. The third kind of of the reasons for the students' errors. Many times, the misconception is very different from the two first ones, students will realize through an "aha" process that they and should be treated accordingly. were wrong, and that is all the teacher can hope for. ISO Without trying to be exhaustive, let us now present programs, of which scholar is an example, will provide a some classes of errors commonly encountered. rich environment for research on error types, detection, 1) Missing information, e.g., not knowing that Buenos and diagnosis, and on consequent teaching strategies. Aires is in Argentina or what its population size is. Students' problems in a certain learning process (as 2) Misfiled fact, e.g., belief that Buenos Aires is in reflected in their errors) may exist at three different Brazil. levels. These three levels would reflect on three cor- 3) Wrong entry, e.g., sayingthat the population size of responding levels of diagnosis. Buenos Aires is 500 000 people. 1) Diagnosis of misconceptions, i.e., modifications in 4) Lack of a concept, e.g., what a longtitude is in its the knowledge structure of facts, concepts, and pro- geographic seinse. cedures that are specific to the area being learned or 5) Wrong superordinate, e.g., the student says the discussed. This is the type of computerized diagnosis Nicaragua is part of South America (wrong superpart), most people have in mind; it seems feasible in a near or that Aconcagua is a city (wrong superconcept) . future through the use of ISO CAI. Some procedures in 6) Overgeneralization error, e.g., belief that all South scholar are attempts to provide capabilities for the American governments are military. diagnosis of misconceptions. 7) Failure to draw some superordinate inference, e.g., 2) Diagnosis of students' problems in terms of their that the language in a country (usually) implies the basic capabilities e.g., understanding and following in- language in its cities. structions, command of the language, handling spatial 8) Failure to draw some negative inference, in other relationships, equilibrium skills, inductive skills, deduc- words, to recognize that a piece of information contra- tive skills of different kinds, capability for making value dicts the rest. judgments, etc. Generally, it seems harder to program a F. Teacher Interactions computer in order to properly diagnose students' prob- lems in these more context-free aspects. Elsewhere [9] we compared CAI to remote manipula- 3) Diagnosis of very general problems coming from tion in which the teacher communicates with a computer, attitudes and other general attributes, e.g., industrious- which in turn, after going through a time-space barrier, ness of the student, motivation, initiative, curiosity, per- communicates with the student. severance, etc. Again, we feel that no immediate pros- The teacher-computer interaction is usually necessary pect exists for computers performing diagnosis in these at the following three levels. areas outside of spsrMc programs written just for test- 1) Preparation of the data base, be it in the form of questions and frames in AFO CAI, or a semantic network ing a particular attribute N In the following, we wdll restrict our analysis to the in ISO CAI. diagnosis of misconceptions. A taxonomy of errors is 2) Setting conditions for student-computer interaction, needed, first to provide a criterion for determining the i.e., defining the system parameters necessary to stimu- relative importance of different errors and a basis for late the conditions of that interaction. allocating time to be spent in discussion, clarification, 3) Collection ofresults, in the form of scores, statistics, and correction of the error. The second benefit from a and general history of the student-computer interaction taxonomy of errors is to determine in general terms what after it has taken place. strategies can be followed to clarify the student's prob- There is a possible fourth role for the teacher in CAI, lem. Finally, if we recognize that an error belongs to a that of a supervisor in real time of the actual operation certain class, we might search for similar related errors of a system with many terminals. When, for example, a I following the same pattern, e.g., missing the inference system like scholar is asked a question for which it has that the language of Argentina should apply to Buenos no answer, instead of answering something such as "sorry, Aires suggests that an inference about the predominant I don't know," it would ask the human supervisor for religion in Buenos Aires in terms of that in Argentina help. The system could also ask for help when compli- may also be missed. cated diagnoses appear needed. Suppose a student has been told that the language in In scholar, we have concentrated our efforts on the Argentina is Spanish, but when asked about the language student-computer interaction ; priorities have forced in Buenos Aires, he responds: "Portuguese." Three hy- postponement of the programming of most teacher-com- potheses can now be made about the student's miscon- puter interactions. A small conversational program exists ception. He may have forgotten what the language in to help the teacher set the student-computer interaction Argentina is, he may not know that Buenos Aires is in conditions [type 2) ] ; so far no implementation exists for Argentina, or he may not be able to draw the inference a teacher program of type 1). The problem is now more 200 IKKI-! TRANSACTIONS ON MAN-MACHINE SYSTEMS, DECEMBER 1970 important and interesting than before, since we are faced with the construction of a semantic network. Two pos- Name of unit: sible approaches should be noted. "Chile* The first approach follows the line of studies on Part of speech: natural-language comprehension by Quillian [3] and *XN* others. This consists of reading English text into the Superconcept, please machine: the program attempts to code that text in the format chosen for the internal structure. The program *contry* checks with the person entering the material to see if it Do you mean country? Answer yes or no k-» has been properly coded and can call his attenion to "Yes* undefined terms and other anomalies. In what continent is Chile? The second approach is the one we advocate for ISO *South America* CAI data bases and is one step more conservative than the first. It involves building a conversational program More about the location of Chile? to help the teacher encode the material in an incremental *No* way, saving parsing and other problems. Heavy use can Area of Chile in square miles: be made of semantically higher levels (through super- superordinates) if they have been concept and other If you want to add some properties, type first the name already coded. of the old property, if any, in which they are included; In Fig. 4, we illustrate a simulated teacher-computer if none, type none. computer its knowl- protocol. Observe that the can use "Population edge about country and its properties to help the teacher encode the unit Chile. In the case of Cordoba, the com- Name of new property: puter does not know what a province is and proposes to ♦Language* define that first. If the teacher overrules this, he would Value still be able to encode properties by giving their names. ♦Spanish* most important system parameters in- Some of the Property cluding complex symbolic lists can be set by the teacher *Done* by using a special interactive program. Fig. 5 is an on- line protocol, which shows that the teacher does not have to use any cryptic computer language. The program Names of properties and subproperties you want to tag makes suggestions in English and guides the teacher in for irrelevancy: each step. The only requirement is for him to have some "Location, area, population* very general understanding scholar and its param- of OK. Location: eters. *2* With respect to the set of parameters dealt with in Fig. 5, it must be said that they do not represent an exhaustive list of the adjustable system parameters in Name of unit: scholar. For example, the specific tolerances for accept- *Cordoba* numerical or a misspelled ing an approximate answer Part of speech: on adjustable parameters. The question- word depend *XN* generation routines select question modes according to preestablished weights which can be adjusted. Superconcept, please: "Province* IV. Future of AFO Versus ISO CAI Undefined. Do you want to define it now? Both the classic ad hoc frame-oriented (AFO) CAI *Yes* systems and the information-structure-oriented (ISO) Part of speech: ones can present material and questions to the student. AFO systems can at this time ask more involved ques- *CN* tions, but these must have been formulated in all detail Superconcept, please in advance by a human teacher. ISO systems have better capabilities for analyzing unanticipated answers, which Fig. 4. Simulated protocol of a teacher building scholar's they can relate to their semantic memory. Because of semantic network. this, ISO systems can be designed to have diagnostic capabilities that AFO systems can not possess beyond conversational capability with questions from both sides the specific errors anticipated by the teacher. that will depend on overall context, specific context, what AFO systems rarely allow students' questions; ISO has just happened in the previous question, etc. systems process and answer them. This leads to a true We have further shown that ISO systems exhibit the carbonell: artificial-intelligence approach to CAI 201 ■SETINTERACTION) capability for handling relevancy and determining how much information and which to present at a given time ************************************************************ either in response to a question or as new material. This THIS IS the program to set the conditions of the interaction between the student and scholar, do you you want is something completely alien to 'AFO systems. to change those conditions? please type y or questions, an- n. (remember to terminate your typing with an The teacher preparing frames of text, asterisk » followed by a carriage return.) swers, and branching for AFO systems is faced with an *V* extensive, rather boring, and unchallenging task. It is NAME OF INSTITUTION! known that teachers preparing AFO CAI courses can "ABC REGIONAL HIGH SCHOOL* barely keep up with the students who use up the ma- very Preparing question well TYPE NAME OF SUBJECT MATTER. 1.E.. CONTEXT TO BE DISCUSSED! terial fast. the 1000th takes the same time and effort as preparing the first. Finally, "GEOGRAPHY OF SOUTH AMERICA* in AFO systems the teacher is not necessarily led to TYPE OF INTERACTION. IT MUST BE ONE OF THE FOLLOWING! MIXINIT. TEST. OR 8/A. conceptualize his subject. On the contrary, the teacher's

"MIXINIT* role in an ISO system is a more conceptual one, with less repetitive examples. Adding a piece of INSTRUCTOR IN CHARGE OF -THE COURSE! concern for new information to the data base usually permits many pos- ECHEVERRIGARAY* "MR. JUAN sible new questions; the program can also use that piece TYPE YOUR NAME EVEN IF YOU HAVE TYPED IT ABOVE I of information to draw inferences and set relations. The "JAIME CARBONELL* larger the semantic network on a given context, the TODAY - DATE I greater will be the effect of an addition to that network. "4/15/1978* In terms of practical realization and use with students, objects MAX. DURATION OF STUDENT INTERACTION. IN MINUTES! ISO systems will still be of research and develop- ment for time to AFO systems can be and "60* some come. are being implemented in many computers now. Actually, MIN. DURATION OF STUDENT INTERACTION. IN MINUTES! the problem with them is that they are frequently using "40* facilities too powerful for what they require. Many of the MIN. NO. OF OUESTIONS TO BE PRESENTED! AFO systems are input-output bound and make little use " IS* of the computer as such; programmed books could per- IF YOU WANT FULL INITIAL INSTRUCTIONS PRESENTED AT haps replace most drill-and-practice AFO CAI systems. THE BEGINNING OF THE INTERACTION WITH THE STUDENT, TYPE 1. IF NOT. TYPE 0. ISO CAI systems, on the other hand, make heavy and balanced use of the different computer components, i.e., memory, processing, and input-output devices. LET US NOW DECIDE IF SCHOLAR SHOULD CALL THE STUDENT _ central ATTENTION ABOUT WORDS IT CAN NOT RECOGNIZE. PLEASE In order for ISO systems to be practical, they will re- TYPE OR V Nl quire, perhaps, more powerful computational facilities »V» than those existing now. TYPE PROBABILITY. IN PERCENT. FOR GENERATING A BUESTION With respect to costs, when teachers' time and effort ABOUT A SUBCONTEXT OF A GIVEN CONTEXT. WHEN DEALING are included, when educational objectives are taken WITH THE CONTEXT ITSELF! and as unitary measures, ISO systems might be, for certain "85 " applications and in the near future, quite competitive. TYPE THE NUMBER OF SECONDS TO WAIT BEFORE PRODUCING A PROMPTING MESSAGE! Let us conclude these brief remarks by emphasizing that we are not advocating the complete elimination of "2B« AFO CAI. It will have its role for some time to come. TYPE MAXIMUM SEMANTIC DEPTH ACCEPTABLE FOR SUBCONTEXT GENERATION! We see it convenient for cases in which the subject mat- ter is very diversified and the interactions with the stu- are brief. In SCHOLAR IS SET BOTH TO CHECK FOR MISPELLINGS IN THE dents planned to be those cases, the develop- STUDENT'S ANSWERS AND TO ACCEPT APPROXIMATE NUMERICAL ment of complex semantic networks is not justified. When ANSWERS. NORMALLY YOU WILL WANT TO LEAVE BOTH OF THESE CHECKS IN. YOU DO THIS BY TYPING discussion in depth is desired, when the student should WITHOUT THE OUOTATIONS. OF COURSE 1 "MISP APPROX". IF YOU ONLY WANT ONE OF THEM, TYPE ITS have some initiative, when detailed anticipation is un- NAME. IF YOU DESIRE NONE. TYPE NIL. wanted, then ISO systems are to be preferred. On the "MISP APPROX* other hand, when teaching sequences are extremely simple, perhaps trivial, one should consider doing away DO YOU WANT TO START THE STUDENT INTERACTION NOW? ANSWER V OR N. with the computer, and using other devices or techniques

"N* more related to the task.

THE YOU HAVE ENTERED HAVE BEEN STORED O. K. VALUES lkdc; ent IN THE SYMBOLIC FILE /SET INTER/. A ck xow m The author wishes to express his gratitude to Profs. D. C. Carroll, T. B. Sheridan, and (i. A, (iorry for their ************************************************************for sc iiolah- Fig. 5. On-line protocol of teacher selling conditions many comments student. interactions. valuable and suggestions and to Drs. 11)70 202 lEEE TRANSACTIONS ON M AN'-M ACHINE SYSTEMS, VOL. MMS-11, NO. 4, DECEMBER I). Bobrow, A. M. Collins for [10] R. F. Simmons and H. F. Silbcrman, "A plan for research G. M. R. Quillian, and toward computer-aided instruction with natural English," their sustained encouragement and constructive criticism. System Dcv. Corp. TM-3623, Santa Monica, Calif.. 1967. [11] R. F. Simmons, "On natural language for instructional com- Reference munication," Interlace, vol. 4, no. 3, pp. 29-30, June 1970. [12] J. F. Rockart, M. S. Scott Morton, and Z. S. Zannetos, 111 J. R. Carbonell, "Mixed-initiative man-computer instruc- "Associative learning project—phase-1 system," working tional dialogues," Ph.D. dissertation, Massachusetts Institute paper, A. P. Sloan School of Management, Massachusetts of Technology, Cambridge, June 1970. Institute of Technology, Cambridge, January 1970. [2] M. R. Quillian, "Semantic memory," Ph.D. dissertation, [13] L. Uhr, "The automatic generation of teaching machine pro- Carnegie Institute of Technology, Pittsburgh, Pa., 1966. grams" (to be published). [3] , "The teachable language comprehender: a simulation [14] J. D. Wexler, "A generative teaching system that uses in- program and theory of language," Commun. Ass. Comput. formation nets and skeleton patterns," Ph.D. dissertation, i*» Mach., vol. 12, pp. 459-476, Aug. 1969. University of Wisconsin, Madison, 1970. [4] W. R. Uttal, T. Pasich, M. Rogers, and R. Hieronymus, [15] E. A. Feigenbaum, and J. Feldman, Eds., Computers and "Generative computer-assisted instruction," Mental Health Thought. New York : McGraw-Hill, 1963. Res. Inst., Ann Arbor, Mich., Commun. 243, 1969. [16] M. Minsky, Ed.. Semantic Information Processing. Cam- 15] G. L. Bryan, "Computers and education," Comput. and bridge, Mass.: M'.I.T. Press, 1968. Automat., vol. 18, no. 3, pp. 1-4, 1969. [17] D. G. Bobrow, "Natural language input for a computer [6] W. Feurzeig et al., "Programming-languages as a conceptual problem-solving system," Massachusetts Institute of Tech- framework for teaching mathematics," Interface, vol. 4, pp. nology, Cambridge, Project MAC Rept. TR-1, 1964. 13-17, April 1970. [18] R. F. Simmons, "Natural language question-answering sys- 17] J. A. Swets, and W. Feurzeig, "Computer-aided instruction," tems: 1969," Commun. Ass. Comput. Mach., vol. 13, pp. 15- Science, vol. 150, pp. 572-576, 1965. -30, January 1970. [8] E. F. Taylor, "The eliza program : conversational tutorial," [19] C. H. Kellogg, "A natural language compiler for on-line data A Skimmable Report on the eliza Conversational Tutoring management," Syst. Dev. Corp. SP-3160, Santa Monica, Calif., System, Education Research Center, Cambridge, Massachu- August 1968. setts Institute of Technology, March 1968. [20] J. Weizenbaum, "eliza—A computer program for the study [9] J. R. Carbonell, "Interactive non-deterministic computer- of natural language communications between man and assisted instruction," Proc. Internal. Symp. on Man-Machine machine," Commun. Ass. Comput. Mach., vol. 9, pp. 36-45, Syst. (Cambridge, England), September 1969. January 1966.