On Analyzing Planning Applications

Yolanda Gil Marc Linster Information Sciences Institute Workplace Integration Technologies University of Southern California Digital Equipment Corporation 4676 Admiralty Way 200 Forest Road Marina del Rey, CA 90292 Marlboro, MA 01752 gil@isi, edu linst er@guess,enet. dec. com

Abstract illustrates, the world is a complex place and in the It is hard to evaluate in current planning ap- process of implementing an application, the designer plications what aspects of the approach address continually makes choices based on (1) the baseline ar- each of the complexities of the problem. This chitecture used to implement the application, (2) the results from the fact that the planning commu- characteristics of the problem itself, or (3) arbitrary de- nity is lacking a vocabulary to describe planning cisions to simplify the problem. All these decisions are tasks and apphcations. This work is an effort intertwined in the resulting application, and as a result, towards descriptions of planning applications in it is not easy to abstract a description of it in terms of terms that are useful 1) to extract conclusions architectural limitations. At the same time, we would from particular implementations, 2) to facilitate like to base our science on real-world applications that cross-comparisons among different planners ap- are subject to analyses that allow making predictions plied to the same problem, and 3) to facilitate comparisons amongdifferent tasks. We analyze and to controllable experiments whose parameters can the Sisyphus experience, a 3-year old and still be modified to obtain experimental results of our pro- ongoing effort in the knowledge acquisition com- grams’ behavior (Hanks, Pollack, & Cohen 1993). But munity to enable a cross-comparison of their ap- our applications rarely facilitate this task. Abstract- plication systems as they implement a common ing the design issues that lie behind the behavior of pre-stated problem description. Based on this our programs is a hard thing to do, yet it is crucial experience, we propose a set of dimensions to de- that we aim towards this goal. At the core of this scribe applications that distinghish between de- matter is how to achieve in practice Newell’s distinc- scriptions of the properties of the architecture, tion between descriptions of systems at the knowledge the type of problem, and the data sets. Weshow how they can be used to produce useful distic- level, abstracting from implementation concerns and tions in the context of the first Sisyphus task, focusing on reasoning-related issues, and descriptions which was an office assignment problem. Our of systems at the level of the symbols they use (Newell hope is that the same dimensions will be useful 1982). to other researchers in describing and character- izing their applications, as well as a useful point The expert systems community is an experienced of comparison for future Sisyphus efforts. one when confronted with real-world applications. There were efforts from the beginning to understand the applications of knowledge-based systems and ab- Introduction stract from the details the techniques used to address Planning applications should be an invaluable experi- conceptual issues. (Stefik el al. 1982) describes al- mental source of challenges for planning research. The ternative organizations of expert system architectures. real world always stretches the limitations of our plan- Each organization is appropriate for a class of prob- ning systems. These limitations point towards new re- lems. Different classes of problems require different search themes in all areas: knowledge representation, organizations, depending on the complexity of the task computational constraints, adaptability, instructabil- and the type of knowledge required. For example, ity, etc. Although the planning community has devel- if the data and knowledge are reliable, the data are oped formalisms to describe planning architectures, we static, and the search space is large but factorable, then are still lacking a formal language to describe applica- generate-and-test is an appropriate method. Although tions and problems that provides an understanding of many applications may be complex and require a com- implementations. We argue that this is a major defi- bination of these methods, this analysis makes useful ciency that stops feedback from applications back to re- distinctions between the techniques that are necessary search and that keeps us away from cross-comparisons (and sufficient) to address the complexity of classes of planning systems in the same task. As Figure 1 problems.

41 that were used in (Linster 1993b) to evaluate the first two rounds of the Sisyphus experiment and some of the conclusions that we drew using those dimensions. Then, we enumerate a set of distinctions that we deem useful to compare applications of planning architectures to the same problem.

Drawing from the Sisyphus Experience The goal of Sisyphus is to compare different methodological approaches to the engineering of knowledge- based systems, and to provide a set of meaningful data- points to enable further analysis. Each year, a Sisy- phus conference is held organized around a problems that should each be solved by the different research Figure 1: Applications are shaped by the architecture teams. Each participating team develops a solution used, the characteristics of problems they automate, using its own methodology or tool. The solutions are and assumptions about the problem made by the designers. presented to foster comparison, discussion, and to serve as data-points to relate different approaches. In 1991, two problems were proposed: one was concerned with the analysis of textual material related to the rigging of The knowledge acquisition community, continuing sailboats, and the other was an assignment problem of this quest for automating expertise into real-world ap- people to offices. In 1992, a slightly modified version plications has taken an alternative and perhaps more of the latter was used. In 1993, a complex elevator empirical approach towards the same goal of under- configuration task was chosen (Sandra Marcus 1988; standing application systems. In 1990, during one of Marcus & McDermott 1989; Yost 1993). We focus our their meetings (Wielinga et al. 1990), they decided to discussion on the office assignment problem, because 1) publish several problem statements and ask research it is the most manageablein size, 2) it is the one that groups from around the world to use their tools, meth- has been implemented by most groups to date, and ods, and approaches to provide solutions for the prob- 3) it has already produced some comparative studies. lems. The results would be presented at an annual The problem is described briefly in the Appendix. We conference baptised Sisyphus. Their goal was to use now summarize the dimensions for comparing systems the different solutions to a same problem as a basis in one of the studies mentioned in which we based the for developing a commonvocabulary for the descrip- discussion presented in the rest of the paper. tion and differentiation of knowledge acquisition approaches. Various previous camparisons (Karbach Dimensions for comparison VoB1991) had pointed out that the vocabulary used by Linster (Linster 1993b) compares several Sisyphus so- different research groups varied in terms and in mean- lutions along different dimensions to address the fol- ing. Using a commonproblem statement would hope- lowing questions: fully help clarify issues such as: what is the difference 1. What are the building blocks of the model? between what different systems call actions, schemas, knowledgesources, inference actions. Intuitively, there 2. Which components are generic, which are non- are bound to be commonalities between their defini- reusable? tions, but each approach describes its building blocks 3. What is the purpose of the tool/methodology (e.g., at a different grain size and using varying terminology. code generation, visualization, conceptualization, Additionally, relating systems implementedin different elicitation, knowledge management)? languages and using different algorithms forced the KA 4. Where in the development cycle is the tool or communityto use knowledge-level abstractions. Sisy- methodology most useful? phus allowed the KAcommunity to develop a better understanding of the different tools, approaches, and 5. Whois the user of the tool or methodology? representation schemes. We draw on their effort and These dimensions reflect an engineering point of propose a set of dimensions that can be used to de- view on knowledge modeling, instead of a cognition- scribe planning applications in a way that facilitates oriented one. For example the analysis does not con- identifying the architectural issues behind the imple- sider the adequacy of the models nor the efficiency of mentation. the approaches in capturing the righl kind of knowl- First, we will give a brief review of tile Sisyphus ex- edge (see (Burton e~ al. 1990) for such a study). periment, the problem statement, and the solutions do not look at the methodological aspects of the ap- submitted by different groups in the knowledge acqui- proaches either, that is, we do not study how much sition community.We will describe a set of dimensions guidance they provide for the practitioner developing

42 a model (see (Allemang & Rothenfluh 1992) for such refined and even undone if indicated by the ongoing a study). elaboration process. A more detailed description of this view on the developmentcycle is given in (Linster What are the building blocks used to model 1993a). knowledge? Building blocks were defined as discrete and identifiable constructs that the framework pro- 1. Initial knowledge acquisition. The knowledge vides to describe the knowledge that goes into an ap- engineer goes through initial interviews with the do- plication. Note that according to this definition inter- main expert, records first protocols, and if possible preters or other predefined and invariable elements of she uses techniques such as the knowledge acquisi- a system are not considered building blocks. Rules, tion grid (LaFrance 1987) to obtain initial structures objects, knowledge sources, agents, classes, are exam- and to get a first overview of the application task. ples of building blocks. The focus was on declarative 2. Data interpretation and knowledge structur- elements of the knowledge representation. This ap- ing. The knowledge engineer identifies recurring pears to be the only clear boundary between general- and potentially more abstract structures in the do- purpose programming languages, such as Lisp or Hy- main. These structures can be of different kinds. perTalk and other means of representation that are more commonlyreferred to as knowledge representa- (a) Identification of domain structures. The tion. Wewill distinguish three kinds of representa- knowledge engineer develops a structured termi- tional primitives: (1) domain-knowledge representa- nology for the domain, for example she can define a T-box in KL-ONE-like approaches (Brachman tion facilities; (2) method elements, that is the ele- & Schmolze1985) or a set of classes in an object- mentary generic problem-solving building blocks, their oriented approach. aggregation principles and the control description that combines building blocks into problem-solving meth- (b) Identification of inferences and roles that ods; and (3) primitives to connect method definitions knowledge elements play in the problem- with domain knowledge. solving process. The knowledge engineer defines the actions, goals, and decision criteria to give an What is the purpose of the tool/methodology? abstract (possibly knowledge level) description Four categories of usage were employed to discuss the the problem-solver. different approaches: knowledge conceptualization, vi- (c) Identification of other structures, such as, sualization, implementation, and knowledge manage- task sharing, task decomposition, data flow, or ment. A tool helps in the conceptualization phase if it modality. allows to represent observations and interpretations of (d) Integration and mapping of the different observations previous to their formalisation, and if it structures into a coherent model of the supports the user in the transition from informal to for- task. The different knowledge structures, iden- mal knowledge. Werefer to a tool as a knowledge visu- tified in the previous phases, must be merged into alisation tool if its interface emphasizesgraphical com- a coherent model. This phase is most important, munication of knowledge. For a tool to be categorized as it represents the creative interaction between as an implementation tool it must provide directly op- the different points of view represented by the dif- erational formalisms or strong methodological support ferent knowledge structures. to transform pre-operational knowledge into an operational representation. Simply attaching an editor for 3. Acquisition of the detailed knowledge in the Lisp code doesn’t do it. Moreover, delivering running framework defined by the task model. The systems must be the intention of the tool developers, model of the task provides structures that are now as opposed to tools built to deliver executable specifi- stuffed with detailed knowledge about the applica- cations or feasability studies. Knowledgemanagement tion. refers to repository, dictionary, and browsingcapabili- 4. Knowledge implementation. The task model is ties for the tool’s knowledgerepresentation primitives. transformed1. into an operational system An environment provides active knowledge acquisition 5. Testing and debugging. support if it derives guidance for the ongoing acquisition from the current contents of its knowledgebase. 6. Knowledge maintenance after system delivery. We distinguish two types of knowledge mainte- Where in the development cycle is the tool most nance: useful? The study used a description of system de- (a) Maintenance of the structures that consti- velopment as a cyclic process consisting of a set of tute the framework of the task model. If distinct activities. These activities are organized in structures are modified, then this changes the task a logical sequence, not to be confused with a waterfall- model. Such modifications require re-engineering oriented series of phases; we see them as being the central activities of a spiral developmentcycle, so that 1Obviouslythis phase is obsolete if the task model is all phases can be repeated and previous results can be defined using operational primitives.

43 of the task model, and if necessary, restructuring adaptation, and scaling up? We believe that the next of the detailed knowledge. step is to analyze each contribution along the lines by (b) Maintenance of the detailed knowledge in which they would be evaluated to determine their suc- the structures of the task model. Within the cess as office assignment tools. To do so, we propose a framework of the task model, maintenance of the set of dimensions that can makeuseful distinctions re- detailed knowledgeis similar to the acquisition of garding this kind of analysis. Building on these obser- the detailed knowledge. vations drawn from the Sisyphus experience, we hope that they will be useful to analyze planning domains This distinction depends strongly on the implemen- and planning applications. tation phase. It can only be drawn if the implementation maintains the distinction between structures Comparing Architectures~ Applications of the task model and the detailed knowledge. and Problems Who is the user? The study differentiated between In order to produce useful descriptions of planning ap- four classes of users: (1) domainexperts, that is, users plications, it is important to makedistinctions between with a lot of knowledgeof the area that the tool will the planning architecture, the problem that the appli- be used in, but without systems analysis skills and lit- cation is trying to address, and the application itself. tle or no programmingknowledge; (2) knowledge engi- A planner is any system that implements a planning neers, that is, people with good systems analysis and algorithm. SNLP(MeAllister &: Rosenblitt 1991) is programming skills; (3) analysts, such as workplace example of a planner. An architecture is a domain- analysts, who do not necessarily have programming independent system that combines different aspects of knowledge; and (4) teams consisting of domain experts reasoning. SOAR(Laird, Rosenbloom, &: Newell 1986) working under the guidance of knowledge engineers or is an example of an architecture. For planning applica- analysts. tions, we are interested in architectures that have some Some conclusions kind of planning capability. A problem states the constraints posed by a set of Manydifferent approaches were used to solve the Sisy- situations in a domain and the requirements of desired phus tasks, including configuration design, situated solutions. Typically a solution is found for particular classification, constraint satisfaction, and genetic algo- instances of a problem, which we will call data sets. Ex- rithms (Linster 1994). After categorizing the Sisyphus amples of problems are Blocks World and the STRIPS contributions in this framework, we drew conclusions robot problem (Fikes & Nilsson 1971). task is a de- such as the following ones: scription that encompasses a class of problems. For If one wants to support the creative aspects of example, robot navigation tasks. model building in the early phases of knowledgeac- Anapplication is the system that results from apply- quisition, then having independent languages for ing a problem-solving architecture to a problem. No- method and domain modeling appears to be cru- tice that the same architecture applied to the same cial. problem mayresult in a different application if the ar- If one wants support for initial knowledge ac- chitecture offers different alternatives for modelingthe quisition and bottom-up structure development, same problem. then one is bound to get little help for the acquisi- The dimensions presented in the previous section are tion and maintenance of the detailed knowledge. useful to define which aspects of a problem an application addresses. However, they are more focused on is- If one wants to have good support for the im- sues that are directly relevant to work on knowledgeac- plementation phase, the acquisition and mainte- quisition, such as the type of user and tool support dur- nance of the detailed knowledge, then one should ing the development of the application. Planning ap- use a shell. plications should address these issues to some degree, Given the wide variation in the approaches taken by since these are issues that arise with any real applica- the different participants, it is clear that any of the tion. But in order to understand the planning architec- approaches could successfully implement the Sisyphus ture itself, we need to abstract additional things from problem statements. The question is not whether we the applications that are more directly relevant to plan- can implement an application using a certain kind of ning research. The following subsections present some approach, since this seems to have a clearly positive dimensions that we believe are useful to describe plan- answer. Rather, the question is whether our imple- ning architectures, problems, and applications. Our mentations can support the additional functionalities aim is not to be comprehensive, but rather to present that real applications demand. Can these systems be what we believe are useful factors to analyze applica- corrected if the designers had any misconceptions or tions. misunderstandings about the task they were automating? Can these systems support additional functionali- Describing an Architecture ties that applications mayrequire, such as explanation, 1. Scope of Architecture ¯ task coverage -- types of tasks that the architec- ¯ User intervention -- Does the system share the ture can be used for task with the user? This is a dimension spawned ¯ capabilities -- types of reasoning that the system by the following extreme points: (1) the user keys is able to do: uncertainty, default reasoning, learn- in an instance of the problem statement; the sys- ing, etc. tem comes back with a ready-made solution; and 2. Representations (2) the user and the system cooperate in solving ¯ state descriptions -- what formalism is used to problems. ¯ inspection -- can the user understand how the sys- represent static descriptions of objects? is there a tem is solving a problem? representation of time? ¯ explanation -- can the system justify its behavior? ¯ procedural knowledge -- can the system represent deduction rules? can action changes be rep- Describing a Problem resented? ¯ decisions -- can preferences be expressed ex- 1. Entities plicitely? can constraints be represented? ¯ types of objects ¯ properties of individual objects 3. Finding solutions ¯ properties of classes of objects ¯ search strategy -- how does the system find solu- ¯ properties that relate objects tions? ¯ search control -- are there domain-independent 2. Actions and Change search-control mechanisms? how is domain- ¯ temporal considerations -- reasoning about events dependent search-control knowledge formulated? and durations ¯ abstraction -- is there an abstraction mechanism ¯ state transformations -- transitions and action in the system? models 4. Building blocks used to construe the problem and 3. Restrictions -- can be preferences or hard-set re- phrase the solution strictions ¯ generic building blocks ¯ restrictions on objects ¯ grain size of building blocks ¯ restrictions on actions ¯ knowledge-level vs. symbol-level building blocks 4. Solutions ¯ formal vs. operational vs. informal ¯ are there solutions to all the instances of the prob- ¯ process vs. structure oriented building blocks lem? 5. Activities supported in the development cycle of an ¯ criteria satisfaction -- does the problem require application satisficing solutions or are partially satisficing so- ¯ elicitation lutions sufficient? ¯ knowledge structuring ¯ solution quality -- is there a notion of better qual- ¯ acquisition of detailed knowledge ity solutions? can the user specify a range for the ¯ knowledge implementation quality of the resulting solution? ¯ testing A problem does not necessarily need to be modelled ¯ maintenance in these terms. Our intention in enumerating the above 6. Adaptability -- can the system evolve as its envi- set is rather to provide a list of issues that need to be ronment requires? addressed in modelling the problem. ¯ execution -- can the system handle unexpected Data sets instantiate the problem statement in all events during plan execution? the respects listed above. Data sets may consist of a ¯ learning -- does the system adapt autonomously, description of all the objects of each type and their or does it require manual adaptation? properties. ¯ factual knowledge -- how can objects and facts Describing an Application System be added/deleted/updated? can the same factual knowledge be used for a different task? 1. Representation -- the knowledge necessary for solv- ¯ task knowledge-- can the definition of the task be ing problems changed? can the goal be changed and in which ¯ object types -- how many different types of ob- ways? jects? 7. Prototyping ¯ constraints -- are there restrictions in the way the objects interact? ¯ fast prototyping -- howmuch effort is required to ¯ relations -- are there relations amongobjects? produce a working prototype? ¯ actions -- what changes and transitions are possi- ¯ prototype complexity -- how much complexity of ble? representation is required for simple problems? ¯ time -- does the system do some type of temporal 8. User-system communication reasoning? 2. Search Space -- is the system exploring a search 3. project heads should be close to the group head. space in finding a solution? 4. twin offices should be shared by people in different ¯ branching factor -- what is the branching factor? projects. ¯ search depth -- what is the depth of the search 5. researchers are not eligible for single offices. tree? ¯ goal interactions -- howcan the interactions be- 6. a non-smoker cannot share a room with a smoker. tween goals be characterized? (independent, seri- ¯ criteria satisfaction -- solutions that satisfy all the alizable vs non-serializable, positive vs negative) constraints are preferred if they exist. Otherwise, ¯ backtracking -- does the system ever need to back- solutions that satisfy most of the criteria are accept- track? able. ¯ solution density -- what is the average solution The data set provided made the problem relatively density when comparing the size of the search tree simple. For example, there were a lot of double-size to the number of solutions? and single-size offices compared to how many people 3. Efficiency -- how does the system address the com- needed to be assigned. There were only 4 smokers. binatorial complexity of the problem? There were 8 projects, each of 1-3 people. These data made the constraints relatively easy to satisfy. The ¯ quantitative measures -- empirical results on the participants in the Sisyphus conferences describe their system efficiency systems with a focus on how this particular problem ¯ qualitative measures -- a description of the mech- and data were implemented with their architecture. anisms used to make the system efficient While it is interesting to see that an architecture can 4. Delivery -- what happens when the system is put to represent and solve this problem with the data pro- use? posed, it may be more useful to focus on how the ar- ¯ users -- who are the users of the application? chitecture would implement alternative definitions of does the system achieve a useful task according this problem. This was tried out when the conference to users? organizers issued a second statement of this Sisyphus ¯ environment -- howdoes it fit in the overall work problem (described at the end of the Appendix) is environment? variant of the first data set but with one more smoker. This small change produced a data set that had no sat- The description of the application can also include a isficing solution, which madesome architectures fail to more detailed account of the dimensions for describing return any solution at all. This sort of exercise is very an architecture. useful to understand the strengths and weaknesses of an architecture. But it is nonetheless a small variation The Sisyphus Problem Revisited along one single dimension. It would be interesting to The Sisyphus problem is an office assignment task. The learn howthe different alternative architectures com- task consists on assigning a set of people into a set pare in other dimensions. For example, how would the of offices satisfying somegiven restrictions. An office system’s performace be affected if: assignment task is a kind of assignment task. ¯ More rooms, more people, more projects (this would As far as the particular office assignment problem test scaling up issues) posed, it can be described as follows: ¯ More (and more problematic) smokers (as in second ¯ types of objects: rooms, people, projects, job Sisyphus task, but more than one) types (group head, project head, manager, secretary, ¯ More managers project member) ¯ More people in each project ¯ properties of individual objects: person x is a smoker ¯ Absolutely no assignments with smoker/non-smoker or a non-smoker, person x is a hacker or a staff mem- ¯ Hackers cannot share rooms with non-hackers ber, roomx is double or single size ¯ Newtypes of workers (e.g., visiting researchers) ¯ properties that relate objects: person x has job of ¯ Required to respect officemate assignments as they type y, person x works with person y, person x works are currently stand in the building before the group with person y. movesto the castle ¯ restrictions ¯ The user has strong preferences on which constraints 1. single rooms accomodate one person, double to violate rooms accomodate two. Notice that the data determines whether a con- 2. there is only one group head straint in the problem definition is making the problem 3. there is only one manager hard to solve. In this case, the fact that the number 4. there is only a project head per project of smokers is even makes this data set simpler to solve with regard to the non-smoking constraint, compared ¯ preferences to data sets that include manysmokers. Andin the ex- 1. offices of the group head and staff should be close. treme, if there are no smokers at all then the constraint 2. group heads should be given a double-size office. becomes nonrelevant. Similarly, if there are manymore

46 rooms than people, then this constraint does not really Tasks. In Wetter, T.; Althoff, K.-D.; Boose, J.; Gaines, affect the complexity of the problem. B.; Linster, M.; and Schmalhofer, F., eds., Current devel- It is noticeable that the planning and knowledge ac- opments in knowledge acquisition-EKAW92, volume 599 quisition communities usually base their analyses in al- of LNAI, 353 - 372. Heidelberg: GI, ECCAI. most mutually disjoint items from the list above. The Brachman, R., and Sehmolze, G. 1985. An overwiev of planning community focuses mostly on computational the KL-ONEknowledge representation system. Cognitive efficiency, while the knowledge acquisition groups con- Science 9(11):216-260. centrate on modeling, knowledge representation, and Burton, A. M.; Shadbolt, N. R.; Rugg, G.; and Hedge- system development. Are our planning systems con- cock, A. 1990. The efficacy of knowledge elieitation tech- cerned only with execution issues? Are our knowledge niques: A comparison across domains and levels of exper- acquisition tools so submerged in the real world that tise. Knowledge Acquisition 2(2):167 - 178. that there is not much room for computational issues? Fikes, R. E., and Nilsson, N. J. 1971. STRIPS: A new We would expect the answers to be negative in both approach to the application of theorem proving to problem cases. Our analyses and evaluations in the two areas solving. Artificial Intelligence 2(3-4):189-208. should make an effort to- cover both grounds. Hanks, S.; Pollack, M. E.; and Cohen, P. R. 1993. Bench- marks, test beds, controlled experimentation, and the design of agent architectures. AI Magazine 14(4). Appendix: The Office Assignment Karbach, W., and Vofl, A. 1991. Glueing together small Problems of Sisyphus ~91 and ’92 solutions: An office planning model in MODEL-K.In The task is described in detail in (Linster 1994). Linster, M., ed., Sisyphus’91: Models of problem solving, volume 663 of Technical Report. St. Augustin: GMD. essence, the members of a research group need to be assigned to new offices. Their characteristics are as LaFrance, M. 1987. The knowledge acquisition grid: A method for training knowledge engineers. Int. Journal of follows: Man-Machine Studies 26:245-255.

Per~gn ~R91 Smoker Hacker Project Laird, J. E.; Rosenbloom, P. S.; and Newell, A. 1986. Thomu 0. groupleader no no EULISP Chunking in SOAR: The anatomy of a general learning MonlkaX. secretary no no mechanism. Machine Learning 1(1):11-46. Ulrike U, secretary no no Eva L man~Jer no no Linster, M. 1993a. Explicit and operational models as JochimI. pro| leader no no ASERTI ~’~ a basis for second generation knowledge-acquisition tools. Hans W. proJleader yes no BABYLON RoomSize KatharlnaN. proj leade¢ yes yes MLT C-113 S In David, J.-M.; Krivine, J.-P.; and Simmons, R., eds., Andl L resemehe¢ yes no TUTOR/2000 C-114 S Second generation expert systems. Springer Verlag. 477 - UweT. researcher yes yes AUT SYS C-115 S 506. Werner L researcheq no yes RESPECT C-11E Si C-117 L JuergenI_ resewcher no yes EULISP C-11fi L : Linster, M. 1993b. A review of Sisyphus 91 & 92: Models MarcM. researche¢ no yes KRITON C-12¢ L ! of problem-solving knowledge. In Aussenac, N.; Boy, G.; Angl W. reseerche~ no no RESPECT C-121 L Harry C. researche¢ no yes 8ABYLON C-122 L Gaines, B.; Linster, M.; Ganascia, J.-G.; and Kodratoff, MichaelT, remmarcher no yes BABYLON C-123 L Y., eds., Knowledge Acquisition for Knowledge-Based Sys- tems, volume 723. Toulouse: Springer Verlag. The offices in the floor plan are numbered counter- Linster, M. 1994. Special Issue on Sisyphus: Models of clockwise. In addition, the problem statement included Problem Solving. Int. Journal of Human-ComputerStud- some restrictions such as: ies M. Linster (Ed.) 40 (2). ¯ offices of the group head and staff should be close. Marcus, S., and McDermott, J. 1989. SALT: A knowl- ¯ group heads should be given a double-size office. edge acquisition language for propose-and-revise systems. ¯ project heads should be close to the group head. Artificial Intelligence 39 (1): 1-37. ¯ twin offices should be shared by people in different McAllister, D., and Rosenblitt, D. 1991. Systematic non- projects. linear planning. In Proceedings of the 1991 National Con- ference on Artificial Intelligence. ¯ researchers are not eligible for single offfices. ¯ a non-smoker cannot share a room with a smoker. Newell, A. 1982. The knowledge level. Artificial Intelli- gence 18:87 - 127. The problem statement contains additional informa- Sandra Marcus, Jeffrey Stout, J. M. 1988. VT: An expert tion that we do not include here because of lack of elevator designer that uses knowledge-based backtracking. space, including a sample protocol of an expert solv- AI Magazine 9(1). ing the problem. Stefik, M.; Aikins, J.; Balzer, R.; Benoit, J.; Birnbaum,L.; The second version of the problem stated that Hayes-Roth, F.; and Sacerdoti, E. 1982. The organization Katharina N. left the group and Christian I. joined of expert systems: A tutorial. Artificial Intelligence 18(2). as a researcher on the MLT project that smokes and Wielinga, B.; Boose, J.; Gaines, B.; Schreiber, G.; and hacks. Someren, M. V., eds. 1990. Current Trends in Knowl- edge Acquisition; Proceedings of EKAW90, Amsterdam: References ECCAI. Ailemang, D., and Rothenfluh, T. 1992. Acquiring knowl- Yost, G. R. 1993. Sisyphus 1993: Configuring elevator sys- edge of knowledge acquisition: A self-study of Generic tems. Technical report, Workplace Integration Technolo- gies Group, Digital Equipment Corporation, 111 Locke Drive (LM02/Kll), Marlboro, MA01752. Unpublished.