Problem of Extracting the Knowledge of Experts Fkom the Perspective of Experimental Psychology
Total Page:16
File Type:pdf, Size:1020Kb
AI Magazine Volume 8 Number 2 (1987) (© AAAI) The ‘Problem of Extracting the Knowledge of Experts fkom the Perspective of Experimental Psychology RobertR.Hoffman or perceptual and conceptual and represent their special knowledge The first step in the development of an problems requiring the skills of . [It] may take several months of the expert system is the extraction and charac- an expert, expertise is rare, the expert’s time and even more of the terization of the knowledge and skills of an expert’s knowledge is extremely system builder’s” (p. 264). Three years expert. This step is widely regarded as the detailed and interconnected, and our later, Duda and Shortliffe (1983) major bottleneck in the system develop- scientific understanding of the echoed this lament: “The identifica- ment process To assist knowledge engi- expert’s perceptual and conceptual tion and encoding of knowledge is one neers and others who might be interested in the development of an expert system, I processes is limited. Research on the of the most complex and arduous offer (1) a working classification of meth- skills of experts in any domain affords tasks encountered in the construction ods for extracting an expert’s knowledge, an excellent opportunity for both of an expert system” (p. 265). (2) some ideas about the types of data that basic and practical experimentation. Some common phrases that occur the methods yield, and (3) a set of criteria My investigations fall on the experi- in the literature are “knowledge acqui- by which the methods can be compared mental psychology side of expert sys- sition is the time-critical component” relative to the needs of the system develop- tem engineering, specifically the prob- (Freiling et al. 1985), and “extracting er The discussion highlights certain issues, lem of generating methods for extract- and articulating the knowledge is the including the contrast between the empiri- ing the knowledge of experts. I do not most important phase” (Gevarter cal approach taken by experimental psy- review the relevant literature on the I984), and “the process of extracting chologists and the formalism-oriented approach that is generally taken by cognition of experts.l I want to share knowledge can be laborious” (Quinlan cognitive scientists a few ideas about research methods 1984). I encountered passages of this that I found worthwhile as I worked type in every major review article or with expert interpreters of aerial pho- textbook on expert systems that has tographs and other remotely sensed come to my attention (for example, data [Hoffman 1984) and on a project Bramer 1982; Denning 1986; Hayes- involving expert planners of airlift Roth, Waterman, and Lenat 1983; Mit- operations (Hoffman 1986). These tal and Dym 1985; Raulefs 1985; ideas should be useful to knowledge Weiss and Kulikowski 1984). Such engineers and others who might be articles dutifully assert that “the first interested in developing an expert sys- step is to get the knowledge” (Freiling tem. et al. 1985, p. 152). What follows, however, is typically a discussion of TheBottleneck abstract inference strategies, details of In generating expert systems, one Lisp code, and descriptions of system architecture rather than an answer to must begin by characterizing the the question of exactly how to get the knowledge of an expert. As far as I can knowledge. tell from the literature on expert sys- Some papers have titles that explic- tems, getting this characterization is itly suggest they deal with the nuts the significant bottleneck in the sys- and bolts of how to extract an expert’s tem development process. For exam- knowledge (for example, “Acquisition ple, from their experience in the of Knowledge from Domain Experts,” development of PROSPECTOR, Duda Friedland 1981; see also Nii and Aiello and Gasbnig (1981) concluded that ‘I... 1979; Politakis and Weiss 1984). How- something must be done to shorten ever, such papers deal mostly with the the time needed to interview experts SUMMER 1987 53 nosing (interpretation of data), plan- ning, designing, and explaining. Psychologically, the tasks that an expert typically performs involve at least the following: (1) the analysis of complex stimuli into relevant features or cues based on a process psycholo- gists call “perceptual learning,” (2) the analysis of conceptual categories in terms of the relevant features (the per- ception of similarities and differences), (3) the analysis of the features and the categories in terms of relevant under- lying causal laws (involving “concept- formation processes”), and (4) abilities to infer and test hypotheses. Although these (and probably other) psychological factors are involved, the products that result from familiar tasks bear on these tasks and might not actually be very informative about the expert’s reasoning. For example, the standard method of aerial-photo interpretation (terrain analysis) pro- vides lots of information about the land that appears in an aerial photo but says precious little about how the expert arrived at the description (cf. Mintzer and Messmore 1984; Way Table 1. Types of Methods That Can Be Used to Extract the 1978). Such is also probably true for Knowledge of an Expert. the kinds of diagnostic notations made by radiologists when they interpret X representation of knowledge. III short, type of method involves studying the rays (Feltovich 1981). Nevertheless, an apparently little or no systematic expert’s performance on the “tough analysis of familiar tasks can be very research has been conducted on the cases” that sometimes occur. The cate- beneficial because it can give the question of how to elicit an expert’s gorization shown in table 1 forms the knowledge engineer a feel for the kinds knowledge and inference strategies topical organization of this paper. Fol- of knowledge and skill involved in the (Duda and Shortliffe 1983; Hartley lowing a brief discussion of each of domain. 1981).2 How can one find good tasks these method types is an examination An analysis of familiar tasks (includ- for extracting the knowledge of an of some ways in which the data they ing an analysis of available texts and expert? How can various tasks be yield can be analyzed and the various technical manuals) can be used to gen- compared? Can tasks be tailored to methods compared. erate a “first pass” at a data base. What extract specific subdomains of knowl- the expert knows is represented as a edge within the expert’s broader The Method of Familiar Tasks categorized listing of statements cast domains of knowledge? My research in some sort of formal language (such The method of familiar tasks involves as propositions) using terms and cate- has addressed these questions. studying the expert while he or she is Methods for extracting expert gories that are meaningful and related engaged in the kinds of tasks that are to the domain at hand (Freiling et al. knowledge seem to fall neatly into a usually or typically engaged in. Look- handful of categories, as shown in 1985). Such propositions can express ing across a set of experts’ specific tac- observation statements or facts as well table 1. One obvious methodological tics and procedures, one should see category involves observing the perfor- as implications or potential if-then commonalities in terms of goals, the rules of inference.3 Table 2 presents mance of the expert at the kinds of information the experts like to have tasks the expert is familiar with or example excerpts from the data base of available, and the data or records that the aerial-photo-interpretation project. usually engages in. A second category are produced (Mittal and Dym 1985). of methods is the interview. Artificial In a number of reviews (for example, The Unstructured Interview tasks can be devised that depart from Duda and Shortliffe 1983; Stefik et al. what the expert usually does by limit- 1982), the various tasks that experts As far as I can tell, the development of ing the information that is available to engage in have been analyzed and cate- most existing expert systems started the expert or by constraining the prob- gorized into basic types, such as diag- with unstructured interviews of the lem the expert is to work on. Another 54 AIMAGAZINE experts. Indeed, most system develop- ers have apparently relied exclusively on the unstructured interview method. ROCK FORMS #3--DOME Some developers have even apparently Raised circular, linear or ellipsoid rock taken it for granted that an unstruc- Can be small, compound, or clustered tured interview is the only way to Can have radiating fractures extract expert knowledge (for example, Can be salt, gypsum, or intrusive bedrock Weiss and Kulikowski 1984, p. 105). Radial drainage pattern, annular pattern at base of the slope In an unstructured interview, the knowledge engineer asks more-or-less ROCK TYPE # 1 --FLAT SHALE spontaneous questions of the expert Gently rolling, irregular plain while the expert is performing (or talk- Symmetrical finger ridges ing about) a familiar task [see Freiling Branching rounded hills with saddle ridges et al. 1985). For instance, the inter- Scalloped hill bases viewer might ask the expert a question V- and U-shaped stream gullies such as “How do you know that?” Uniform slope gradients imply homogeneous rock whenever the expert seems to tap into Compound slope gradients imply thick bedding knowledge or make an inference, or Escarpments, very sharp ridges, steep slopes, and steep “Do you want a cup of coffee?” when- pinnacles imply sandy soils ever they get tired. HUMID CLIMATE Presumably, the knowledge engi- Implies rounded hills neer’s prior analysis of the familiar Implies fine dendritic drainage pattern tasks has turned the engineer from a ARID CLIMATE novice into an apprentice (or even a ImP lies steep, rounded hills and ridges with asymmetrical slopes master).