COGNITIVE REPRESENTATIONS OF DOS COMMANDS AS A FUNCTION OF EXPERTISE

Kathleen M. Snyder James R. Lewis IBM Corporation IBM Corporation Dept F3B, Div 20 T. J. Watson Research Center Room 12B174 Dept 564, Room Hl-E22A 360 Hamilton Avenue Route 9A White Plains, NY Hawthorne, NY

ABSTRACT that would be impossibly difficult with anything less adequate. [I, The purpose of this study was to examine P. 221 the cognitive networks derived from the similarity rating of IBM Personal Disk (IBM PC Although Winston's [l] comments were DOS) (TM) commands by computer users given in the context of the advantage of with varying levels of expertise. good representation in artificial Naive, novice, and intermediate networks intelligence, the argument is also were examined to determine which links applicable to the use of multivariate in their networks were also present in statistical methods in psychology. an expert network. Groups with a Different methods allow different greater level of DOS expertise had more representations of the structure hidden links in common with the experts. A in a complex dataset. Like any other core set of commands were identified craftsman, we must choose the which were linked in every network. As appropriate tools to achieve our the level of expertise changed, it was analytical goals. possible to show the order in which links in the experts' network became The goal of this study was to present in the novice and intermediate demonstrate the use of network analysis groups. for modeling cognitive representations of operating system commands as a function of expertise.

In some uninteresting sense, all computer-based representations are Expert-Novice Studies equivalent. This is so because computer-based representations are A number of studies have been conducted embedded ultimately in the symbolic which have examined the cognitive structures available in a computer differences between experts and novices language like LISP and thence down (.g., [2,3]). The findings of these into arrangements of bits in studies have been remarkably consistent. memory. Consequently, any representation that can be used to Chase and Simon [2] studied the ability represent arrangements of bits can of novice and expert chess players to indirectly bear the information in recall the board positions of chess any other representation. pieces. They found that experts outperformed novice players. The In a practical sense, however, some experts' advantage was attributed to representations emphasize things their ability to form larger chunks of that are important to solving a information in memory rather than above- class of problems. One scheme, average memory capacity. They also therefore;,.is more powerful than found that if the arrangement of chess another because it offers more pieces was random rather than a midgane convenience to the user even (and presumably meaningful) arrangement, though, theoretically, both can do then experts recalled no more piece the same work. Convenience, positions than novices. however, is perhaps too weak a word. In general, the much greater In an attempt to replicate these perspicuity and the inherent findings in a more applied setting, Egan thinking advantages of powerful and Schwartz [3] had both novice and representations enable progress expert electronics technicians attempt

447 0073-1129/89/0000/0447$01.W0 1989 EEE to recall briefly presented (5 - 10 sec) regression and discriminant analysis are circuit diagrams. Some diagrams were examples of dependence methods. meaningful and others were random re- Principal components analysis, factor constructions of the meaningful analysis, multidimensional scaling diagrams. As with the chess masters, (MDS), cluster analysis, and network the expert technicians recalled more of analysis are types of interdependence the meaningful diagrams, but had no methods. Dependence methods are used to performance advantage for the random explain or predict one or more dependent arrangements. Additional analyses variables as a function of a set of showed that neither more skilled independent variables. "Interdependence guessing by expert technicians nor methods, on the other hand, are less spatial proximity could account for the predictive in nature and attempt to experts' performance advantage. provide insights into the underlying structure of the data by simplifying the Alwood [4] has recently reviewed the complexities, primarily through data body of work examining the behavior of reduction" [7, p. 191. For this reason, novice with , interdependence methods have been used contrasting their behavior with that of to assess the relationship among experts to illustrate that which is concepts in memory. typical of novices. In his review, he points out that the expert advantage for A recent addition to the collection of the recall of meaningful information has interdependence methods is a link- been replicated several times with weighted network analysis procedure novice and expert programmers recalling called Pathfinder [8, 91. It can be lines of code (e.g., [SI). applied to the same types of data typically analyzed by MDS or cluster These results have been explained by analysis, i.e, estimates of pairwise hypothesizing that experts can process distance between entities. With global display properties or code a Pathfinder, distances between entities single chunk whose relation to a more are represented as links in a network if general category is known. This the resulting links form the shortest conceptual knowledge would allow the possible path between the entities. expert to search visually and recall Pathfinder solutions are affected by two from memory more systematically than variables: the value of the path length novices. Therefore, a critical function (defined by the Minkowski r- difference between novices and experts metric) and the number of links examined would be the organization of concepts in in constructing the network. With long-term memory. ordinal level measurement, g should be set to infinity to ensure a unique According to Bateson, Alexander, and network structure. To avoid triangle Murphy [6], another explanation is that ineqvality violations, N-1 links should experts may use higher-level knowledge be examined when developing the network. to understand problems while novices focus on specific details. Experts may Since their development and availability also use high-level plan knowledge to on computer systems powerful enough to direct their activities. Bateson et al. implement them, multivariate divided 50 computer science majors into interdependence methods have been used novice and expert groups, and as tools for psychological research. administered tests to assess syntactic Rips, Shoben, and Smith [lo] memory, semantic memory, tactical skill, demonstrated that MDS could be used to and strategic skill. The best model semantic distance in memory, and predictors of expertise were that these semantic distances could be semantic memory, tactical skill, and used to predict reaction times in a strategic skill. In particular, they categorization task. Schvaneveldt, concluded that semantic memory is an Durso, Goldsmith, Breen, and Cooke important cognitive factor in the [ll] used both MDS and Pathfinder to development of programming skill. This study the organization of flight-related research also supports the hypothesis concepts by fighter pilots of varying that the organization of concepts in degrees of expertise. They concluded memory should differ due to expertise. that both MDS and Pathfinder revealed the underlying structure of the data, but highlighted different aspects. Pathfinder focused more on local Multivariate Statistical Methods and relationships among concepts, while MDS Psychological Research provided more global information regarding the dimensions of the concept Dillon and Goldstein [7] have described space. In research comparing the serial two primary categories of multivariate recall of concept lists organized by statistics: dependence and ALSCAL-S MDS and Pathfinder interdependence methods. Multiple

448 representations, Cooke, Durso, and knowledge [15]. Their formal method Schvaneveldt [12j found that the network requires three stages: initial organization led to faster learning than elicitation of a set of concepts, the MDS organization. This implies that application of a psychological scaling the Pathfinder networks better captured technique (i.e., multivariate the relations important for recall. interdependence method), and the interpretation of the resulting These methods have also been applied in representation. For example, when using two areas of computer development: menu Pathfinder, the final stage is the organization and expert systems. interpretation of the nature of the McDonald, Stone, Liebelt, and Karat relationship between concepts, achieved [13] used MDS and cluster analysis to through either direct link-labeling or model experts' similarity ratings of cluster analysis of similarity ratings word-processing functions. They of the links (applying a second compared the performance of a different interdependence method to the analysis set of word-processing experts using a of the output of the first). menu based on experts' organization of the commands and one based on a random organization. Using a paired-associates Interdependence Analyses of Expert and learning paradigm, they found that fewer Novice Computer Users errors were made using the organized menu. Snyder, Paap, Lewis, Rotella, To draw together these two lines of Happ, Malcus, and Dyck [14] used research (expert-novice studies and the Pathfinder to develop IBM Personal application of multivariate Computer (IBM PC interdependence methods) in the context DOS) (TM) menus based on the similarity of computer use, we will review three ratings of novice and expert IBM PC DOS applicable studies. Since these users. A third menu had an alphabetical statistical methods have been used to organization. Three groups of naive model the relationship among concepts in participants used the menus with a long-term memory, they provide the means paired-associates learning task. by which one may model the change in the Participants using the Pathfinder menus relationships as a function of learning, learned at a significantly faster rate. i.e., the evolution from novice to expert. Knowledge acquisition has been frequently cited as a major bottleneck Using the method of ordered trees from in the development of expert systems, cued recall, McKeithen, Reitman, Rueter, and it is possible that the multivariate and Hirtle [5] developed hierarchical interdependence methods can be used to representations of ALGOL W reserved obtain expert knowledge more efficiently words for beginners (just starting their [15, 161. Olson and Rueter [16] have first ALGOL W course), intermediates recently reviewed methods for knowledge (just completed their first ALGOL W acquisition, classifying them into course), and experts (teachers of ALGOL direct (interview, questionnaires, task W courses). The method of ordered trees observation, protocol analysis, from cued recall requires the interruption analysis, and drawing participants to memorize the set of closed curves) and indirect (MDS, concepts. After learning the concept cluster analysis, network analysis, names, the participants are given a cue ordered trees from recall, and repertory (a concept) and asked to recall the grid analysis) methods. They point out words that go with it, followed by the that both classes have their rest of the words. Each concept serves limitations, since the direct techniques as a cue once. Then the participant's generally assume that experts can recall strings are searched for all verbalize what they know or how they groups of items that always appear solve problems, and the indirect methods contiguously, regardless of order. The are less rich and must make some final result is a hierarchical assumptions about the underlying struc- representation of the concepts. ture of the representation of objects and their relations. People cannot Using this method, McKeithen et al. always accurately verbalize what they [5] attacked three questions: Are know [17] and the assumptions required experts more organized than beginners? for indirect methods may not be met. Does skill affect the depth of organization? Are organizations within Cooke and McDonald [18] have reviewed a skill group similar? They determined the research issues associated with the that there was no strong evidence for a use of multivariate techniques as difference in the amount of organization knowledge-elicitation tools, and have among the groups. They also concluded argued for a formal methodology for that depth of organization did not seem acquiring and representing expert to increase with skill level. After

449 determining the amount of similarity to free recall as a method of between pairs of representations, they determining semantic distance. performed an MDS analysis and found that experts were more alike as a group than Kay and Black [20] examined the changes either beginners or intermediates. in the knowledge representations of text Beginners appeared to use general editing commands with increasing mnemonic techniques to memorize the experience. The novices in this words, such as alphabetical order or experiment were psychology students who story sequences. The different used the computer for text editing and organizations created by the application data analysis, while the experts were of different strategies probably computer science students. Participants accounts for the variability of position rated the pairwise similarity of fifteen of the beginners' representations in the text-editing commands. These similarity multidimensional space. Since the ratings were analyzed using MDS, experts fully understood the meaning of hierarchical cluster analysis, and the concept words in the context of additive clustering analysis (similar to ALGOL W, their representations were hierarchical cluster analysis except highly similar. The intermediates' that an item could belong to more than representations showed some common one group). As with the studies by understanding of the ALGOL W words, but McKeithen et al. [5] and Cooke [19], a also showed the lingering effects of three dimensional MDS solution was found some common-language sequences. to be best. Two dimensions were found to be the same between the groups: Cooke [19] divided her participants into formatting vs. non-formatting commands four groups: naive (no programming and specific vs. general functions. For experience), novice (up to one year of the expert users, the third dimension programming experience), intermediate seemed to be based on the dimension of (one to three years of programming destructive vs. non-destructive experience), and expert (over three commands. The novices' third dimension years of programming experience). Her was defined by commands that began or concepts were words that had meaning in ended editing sessions. The interpre- a programming context, but were not tation of the cluster analyses was restricted to a specific language. She consistent with that of the MDS. Kay had the participants both provide and Black [20] further concluded that similarity ratings for pairs of concepts the knowledge representation for experts and to practice free recall of the is more complex (contrary to the - concepts until they were recalled ings by McKeithen et al., [5]) and that, without error twice. (Note that this with experience, users combine indivi- recall task differs from the cued recall dual commands to form sequences or plans of the Reitman-Rueter technique used by to accomplish the various user goals. McKeithen et al. [5].) Distance estimates were obtained from the recalled lists by counting the number of Goals of the Present Study intervening concepts between each pair in a recalled list. Both the similarity The studies by McKeithen et al. [5], and the recall data were analyzed using Cooke [19], and Kay and Black [20] have MDS. Within-group correlations based on all demonstrated the utility of recalled lists were uniformly poor, multivariate interdependence methods in while within-group correlations based on examining structural changes in memory similarity ratings showed an increase as a function of expertise. The from novice to intermediate to expert, specific methods employed were ordered replicating the findings of McKeithen et trees from cued recall, MDS, and cluster al. [5]. However, the naive within- analysis. In the present study, we group correlation was as strong as the attempted to demonstrate the development expert within-group correlation. Cooke of computer expertise by examining [19] concluded that "the U-shaped network representations of IBM PC DOS function suggests that naive programmers commands by users at various levels of initially agree with each other on the IBM PC DOS expertise. MDS little that they know. As learning representations show the arrangement of takes place, this agreement declines at concepts in multidimensional space. first and then increases as the Ordered trees and cluster analysis show programmer develops expertise" (p. 28). the way in which users have organized Based on the lack of correlation between concepts into groups. A network the recall and rating data, and on the representation shows concepts as nodes consistency in results between the connected by links. Given this rating within-group correlations and representation, it should be possible to those of McKeithen et al. [5], she illustrate the development of expertise concluded that ratings were preferable by determining which links are common to groups at various levels of expertise

450 with the links present in an expert Descriptions of the commands were network: an examination that would not printed on 3x5 cards. Participants were be possible using the MDS or cluster asked to read the descriptions and to analysis modes of representation. One sort the cards into groups based on the would predict that as expertise similarity of the commands. Once the increases, the number of links in common commands were grouped, participants (and, correspondingly, the statistical assigned labels to the groups. agreement) with the most expert group should also increase. It should be If participants had been asked to possible to identify a "core" set of generate a similarity rating for each commands which are common to all groups, possible pair of commands, they would and to show specifically which commands have been required to make over 900 are added to this core set as a function paired comparisons. We had the of expertise. By studying a wide range participants group the commands since of expertise, the analysis of intragroup I'clustering prior to pairing is a way of agreement in this study may shed light reducing the number of objects so that on the discrepant results reported by using paired comparisons becomes a McKeithen et al. [5] and Cooke [19] reasonable technique" [21, p. 231. regarding the pattern of intragroup agreement as a function of expertise. A 9-point scale was used to collect similarity ratings between a participant's command groups. A 111" Method meant the command groups were perceived as highly similar and a "9" meant highly Participants dissimilar. A program written for the IBM Personal Computer AT (TM) was used Thirty-seven IBM employees took part in to capture this paired comparisons data. the study. Participants were assigned to one of six experience categories Cognitive networks were derived for each based on their performance on a recall participant and for each group. This test of the 43 commands in IBM PC DOS. resulted in 32 individual networks and 6 The six categories, the criterion for group networks. To generate the inclusion in the category, and the networks : number of participants assigned to each category are shown in Table 1. 1. A program was written to build a 43 by 43 similarity matrix for each participant's data. Table 1. Categories of Expertise The program looked at every possible pair of commands and Number of generated similarity ratings Experience Commands Number of for each pair. Pairs from Category Recalled Participants the same group of commands were assigned a value of 0. Pairs Naive 0 5 of commands from different Novice 1- 4 5 groups received the rating Intermediate 1 5 - io 9 assigned to the pair of groups. Intermediate 2 11 - 15 7 Intermediate 3 16 - 24 5 2. In deriving the group (various Expert Over 30 6 levels of expertise) networks a second program was written to sum all of the individual Materials Procedures matrices from the same group. All participants completed a 3. The matrices were then used as questionnaire before beginning the input to the Pathfinder algo- experimental task. The goal of the rithm. Pathfinder generates questionnaire was to identify the link-weighted networks from a participants' knowledge and use of set of distance data. In this computers, programming languages, study the distance data con- sisted of the matrices con- operating systems and applications. One 2. question asked participants to list as structed in steps 1 and The many IBM PC DOS operating system Pathfinder network consists of commands as they could. The a set of nodes and links that participants' responses to this question connect pairs of nodes that are were used to classify them into one of highly related. In the present the six experience categories. context the nodes are IBM PC DOS commands and the links re- present the perceived similar- ity between the commands.

45 1 the Naive and Expert groups to be Results significantly different (p<.05). The group networks were compared to uncover structural changes as a function of expertise, shown in Table 2. First, Table 2. Development of Links Common to the links common to all groups were All Higher Networks identified. These constituted the "core" set of commands. Next, links Level of that were common to all groups except Expertise Common Links the Naive group were identified. The Naive Backup-Restore Date-Time third comparison identified links common Find-Sort Mkdir-Rmdir to all Intermediate and the Expert groups only. Next, we identified the Novice -Restore links common only to the Intermediate 2, Intermediate 3, and Expert Groups. Int. 1 Break-Prompt Ctty-Keybxx Finally, the additional links common to -Erase - the Intermediate 3 and Expert groups were listed. (See Figure 1, the Expert Int. 2 Backup-Recover Chdir-Rmdir network, for comparison with the Chdir-Mkdir information in Table 2.) Int. 3 Attrib- Copy-Ren Using kappa [22], the intergroup Break-Verify Ctty-Prompt agreement with the Expert group was Chdir-Dir Erase-Ren measured for the Naive, Novice, and Comp-Copy Find-More Intermediate groups (see Figure 2). Comp-Erase Mode-Prompt Kappa is a measure of agreement or similarity which can be legitimately Comp-Ren applied to a 0-1 level of measurement. It is a more accurate measurement of Expert 47 additional links were similarity than the number of links in only found in the Expert common since it takes into account both network (see Figure 1). the way in which two 0-1 matrices match (links in common) and the ways in which they fail to match (links not in Discussion common). All values of kappa showed significant agreement with the Expert As predicted, the relationship between group (p<.05), and also showed a the expert group and the other groups monotonically increasing trend with the increased as a function of expertise. increasing expertise of the group. Over While this is not surprising, it this range of experience, kappa changed supports the use of Pathfinder for this from .16 to .30, almost doubling, with type of cognitive modeling and also the greatest change occurring between validates the use of free recall of the naive and novice groups. commands as a measure of expertise. However, the use.of free recall of The intragroup agreement was assessed by concepts is restricted to domains in calculating kappa for each pair of which the set of concepts to be recalled participants within a group. As shown has already been identified, such as in Figure 3, the average kappa for commands in an operating system or intragroup agreement was an S-shaped a 's reserved words. curve with the increase in experience. The evolution of expertise revealed in A one-way analysis of variance conducted Table 2 shows the way in which an on the kappas calculated for each pair initial nucleus of properly linked of participants within a group was commands (the core commands) grew with significant (F(5,101)=3.4, p=.OO7). A the increasing expertise of the groups. post-hoc comparison using Bonferroni &- While no studies have been conducted tests [23] showed significant differ- demonstrating the utility of such a ences (p<.05) between the Naive and representation in the development of Expert groups, and the Intermediate 2 training materials, it is reasonable to and Expert groups. suggest that modeling the change in network structure as a function of Figure 4 shows the relationship of expertise could provide valuable intragroup agreement and expertise with guidance to such an effort. This the three intermediate groups collapsed. representation would allow a developer In this case, the averaged group kappas to identify which concepts a naive user increased monotonically with increasing has properly related, and suggests the expertise. A one-way analysis of order in which new concepts should be variance was significant (F(3,101)=3.75, introduced and related to old concepts. p=.Ol). Bonferroni t-tests showed only However, one should be cautious when

452 , figurn 1. The expert network (Note that the length of a line does not necessarily mflect Its link-weight.)

453 Novice Int 1. Int 2 Int 3 Expertise Figure 2. Relationship of Naive, Novice. and Intermediate groups to the Expert group.

Os' 1h Naive Novice Int 1 Int 2 Int 3 Exper Expertise Figure 3. intragroup relationship aa a function of expertise.

0.3 0.2 t 0.1 0 Naive Novice IRt Expert Expertise Figure 4. intragroup relationship as a function of expertise. (intermediate groups combined.)

454 interpreting links which are present in As the slower learners catch up, the each network until determining if the intragroup agreement would increase nature of the link remains unchanged again. In general, a decrease in the between levels of expertise [15]. intragroup agreement may reflect the introduction of major new organizing The effectiveness of this type of principles, while an increase may representation for computer menu design represent the gradual assimilation of has been suggested [14]. Its potential the new principles. Only more studies utility in the development of expert examining a sufficient number of levels systems has also been documented [15]. of expertise and exploring other With the advent of more intelligent operational definitions of expertise computer interfaces, it may also be will resolve this issue. Also, future possible to use network representations research should focus to some extent on of various levels of expertise to longitudinal rather than cross-sectional control adaptive interfaces, interfaces experimental designs. which change as a function of user behavior. Some interface elements which In conclusion, we have demonstrated that could be controlled in this way are menu the analysis of similarity ratings of organization, the content of on-line computer-related concepts using and error messages, and hypertext. Pathfinder is consistent with previous findings in the relevant literature. We The analysis of intragroup agreement in have also shown a way in which a network this study support the results of representation of concepts with nodes McKeithen et al. [5] rather than Cooke and links can reveal changes in memory [19] since the data could be represented structure which would be more difficult as a monotonically increasing curve (see to study using other types of Figure 4), but not a U-shaped one. Both multivariate interdependence methods. McKeithen et al. and we used similar conceptual materials (ALGOL W reserved words and IBM PC DOS commands) while References Cook& used words that were more general in nature (i.e., could be found in a P. H. Winston, Artificial standard English dictionary). Since intelligence, Reading, MA: Cooke’s stimuli were more general, naive Addison-Wesley, 1984. participants could have better intra- group agreement since they would have a W. G. Chase and H. A. Simon, common basis for judgement of similari- “Perception in chess,If Cognitive ty, unaffected by knowledge of the use Psychology, Vol. 4, P- 55, 1973. of the words in the domain of computer programming. Naive participants rating D. E. Egan and B. J. Schwartz, words such as IBM PC DOS commands or IIChunking in recall of symbolic ALGOL W reserved words would have little drawings,” Memory and Cognition, common basis for judging similarity, and Vol. 7, p. 149, 1979. would show poor intragroup agreement. . M. Allwood, llNoviceson the It is difficult to explain the S-shaped computer: A review of the function (Figure 3) with the same simple literaturettf International mechanisms used to explain the monotonic Journal of Man-Machine Studies, or U-shaped functions. Despite the vol. 25, p. 633, 1986. statistical significance of the drop in the curve, it may simply be the result K. B. McKeithen, J. S. Reitman, H. of random variation. On the other hand, H. Rueter, and S. C. Hirtle, it is possible that the relationship I’Knowledge organization and skill between intragroup agreement and differences in computer expertise is more complex than programmers , Cognitive previously indicated. For example, if Psychology, vol. 13, p. 307, 1981. participants begin learning the relationships among concepts, but have A. G. Bateson, R. A. Alexander, no common basis for similarity and M. D. Murphy, I’Cognitive judgement, the initial intraqroup processinq differences between agreement will be poor. As they gain a novice and expert computer common basis through experience, the programmersIf1 International intragroup agreement would increase. ~-Journal of Man-Machine Studies, If, at some intermediate point in Vol. 26, p. 649, 1987. learning, new organizing principles must be learned, intragroup reliability might W. R. Dillon and M. Goldstein, decrease since participants may not Multivariate analysis: Methods and learn these principles at the same rate. applications. New York: John Wiley, 1984.

455 D. W. Dearholt, R. W. [17] R. E. Nisbett and T. D. Wilson, Schvaneveldt, and F. T. Durso, sfTellingmore than we can know: "Properties of networks derived Verbal reports on mental from proximities," New Mexico processes," Psychological Review, State University, Computing vol. 84, p. 231, 1977. Research Laboratory, Las Cruces, NM, Tech. Memorandum MCCS-85-14, [18] N. M. Cooke and J. E. McDonald, 1985. "The application of psychological scaling techniques to knowledge c91 R. W. Schvaneveldt, F. T. Durso, elicitation for knowledqe-based and D. W. Dearholt, "Pathfinder: systems ,'I International-Journal of Scaling with network structures," Man-Machine Studies, volm. New Mexico State Univeristy, 533, 1987. Computing Research Laboratory, Las Cruces, NM, Tech. Memorandum MCCS- [19] N. M. Cooke, Memory structures of 85-9, 1985. expert and novice computer programmers: Recall order vs. L. J. Rips, E. J. Shoben, and E. similarity ratings. Unpublished E. Smith, "Semantic distance and master's thesis, New Mexico State the verification of semantic University, Las Cruces, NM, 1983. relations, Journal of [20] D. S. Kay and J. B. Black, "Changes in knowledge representation of computer systems R. W. Schvaneveldt, F. T. Durso, with experience,Il in Proceedings T. E. Goldsmith, T. J. Breen, N. of the Human Factors Society 28th M. Cooke, R. G. Tucker, and J. C. Annual Meeting, p. 963, San De Maio, IIMeasuring the structure Antonio, TX: Human Factors of expertise," International Society, 1984. Journal of Man-Machine Studies, Vol. 23, p. 699, 1985. [2l] P. Dunn-Rankin, Scaling methods, Hillsdale, NJ: Erlbaum, 1983. N. M. Cooke, F. T. Durso, and R. W. Schvaneveldt, "Measures of [22] J. L. Fleiss, Statistical methods memory organization and recall,1f -__-for rates and pr oDortions. New New Mexico State University, York, NY: John Wil ey, 1981. Computing Research Laboratory, Las Cruces, NM, Tech. Memorandum MCCS- [23] J. L. Myers, Fundamentals of 85-11, 1985. experimental design. Boston: Allyn and Bacon, 1979. J. E. McDonald, J. D. Stone, L. S. Liebelt , and J. Karat , IIEvaluating a method for structuring the user- system interface," in Proceedings of the Human Factors Society 26th Annual Meeting, p. 551, Seattle, WA: Human Factors Society, 1982. K. M. Snyder, K. Paap, J. R. Lewis, J. Rotella, A. Happ, L. MalCUS, and J. Dyck, Wsing cognitive networks to create menus,11International Business Machines, Inc., Boca Raton, FL., Tech. Report 54.405, 1986. N. M. Cooke and J. E. McDonald, "A formal methodology for acquiring and representing expert knowledge,." Proceedings of the IEEE, vol. 74, p. 1422, 1986. J. R. Olson and H. H. Rueter, IIExtracting expertise from experts: Methods for knowledge acquisition," Expert Systems, vol. 4, p. 152, 1987.

456