The First Law of Robotics (A Call to Arms)

Home , Runaround (story)

The First Law of Robotics (a call to arms) Oren Etzioni Daniel Weld* Department of Computer Science and Engineering University of Washington Seattle, WA98195 {etzioni, weld}@cs.washington.edu

Abstract ¯ A construction robot is instructed to fill a pothole Even before the advent of Artificial Intelligence, sci- in the road. Although the robot repairs the cavity, it leaves the steam roller, chunks of tar, and an oil ence fiction writer Isaac Asimov recognized that a slick in the middle of a busy highway. robot must place the protection of humans from harm at a higher priority than obeying human orders. In- ¯ A software agent is instructed to reduce disk utiliza- spired by Asimov, we pose the following fundamental tion below 90%. It succeeds, but inspection reveals questions: (1) Howshould one formalize the rich, but that the agent deleted irreplaceable IATEXfiles with- informal, notion of "harm"? (2) How can an agent out backing them up to tape. avoid performing harmful actions, and do so in a com- While less dramatic than Asimov’s stories, the sce- putationally tractable manner? (3) Howshould narios illustrate his point: not all ways of satisfying a agent resolve conflict between its goals and the need humanorder are equally good; in fact, sometimes it is to avoid harm? (4) When should an agent prevent better not to satisfy the order at all. As we begin to human from harming herself? While we address some deploy agents in environments where they can do some of these questions in technical detail, the primary goal real damage, the time has come to revisit Asimov’s of this paper is to focus attention on Asimov’sconcern: Laws. This paper explores the following fundamental society will reject autonomous agents unless we have questions: some credible means of making them safe! ¯ How should one formalize the notion of "harm"? In Sections and , we define dent-disturb and restore two domain- The Three Laws of Robotics: independent primitives that capture aspects of Asi- 1. A robot may not injure a human being, or, mov’s rich but informal notion of harm within the through inaction, allow a human being to come classical planning framework. to harm. ¯ How can an agent avoid performing harm- 2. A robot must obey orders given it by human ful actions, and do so in a computationally beings except where such orders would conflict tractable manner? We leverage and extend the with the First Law. familiar mechanismsof planning with subgoal inter- 3. A robot must protect its own existence as long actions [35, 7, 24, 29] to detect potential harm in as such protection does not conflict with the polynomial time. In addition, we explain how the First or Second Law. agent can avoid harm using tactics such as confronta- Isaac Asimov [,~]: tion and evasion (executing subplans to defuse the threat of harm). Motivation ¯ How should an agent resolve conflict between In 1940, Isaac Asimovstated the First Lawof Robotics, its goals and the need to avoid harm? We capturing an essential insight: a robot should not impose a strict hierarchy where dent-disturb con- slavishly obey human commands-- its foremost goal straints override planners goals, but restore con- should be to avoid harming humans. Consider the fol- straints do not. lowing scenarios: ¯ When should an agent prevent a human from harming herself? In section , we show how our *Wethank Steve Hanks, Nick Kushmerick, Neat Lesh, framework could be extended to partially address and Kevin Sullivan for helpful discussions. This research was funded in part by Office of Naval Research Grants 90- this question. J-1904 and 92-J-1946, and by National Science Foundation The paper’s main contribution is a "call to arms:" Grants IRI-8957302, IRI-9211045, and IRI-9357772. before we release autonomous agents into real-world environments, we need some credible and computation- formuli. This sidesteps the ramification problem [23], ally tractable means of making them obey Asimov’s since domain axioms are banned. Instead, we demand First Law. that individual action descriptions explicitly enumer- ate changes to every predicate that is affected. 3 Note, Survey of Possible Solutions however, that we are not assuming the STRIPS rep- To make intelligent decisions regarding which actions resentation; Instead we adopt an action language are harmful, and under what circumstances, an agent (based on ADL[26]) which includes universally quanti- requires some explicit model of harm. Wecould pro- fied and disjunctive preconditions as well as conditional vide the agent with an explicit model that induces a effects [29]. partial order over world states (i.e., a utility function). Given the above assumptions, the next two sections This framework is widely adopted and numerous re- ...... define the: primitives dour-disturb and restore, and searchers are attempting to render it computationally explain how they should be treated by a generative tractable [13, 31, 34, 32, 37, 17], but manyproblems planning algorithm. Weare not claiming that the ap- remain to be solved [36]. In many cases, the intro- proach sketched below is the "right" way to design duction of utility models transforms planning into an agents or to formalize Asimov’s First Law. Rather, optimization problem -- instead of searching for some our formalization is meant to illustrate the kinds of plan that satisfies the goal, the agent is seeking the best technical issues to which Asimov’s Law gives rise and such plan. In the worst case, the agent maybe forced .... how they might be solved. With this in mind, the pa- to examine all plans to determine which one is best. per concludes with a critique of our approach and a In contrast, we have explored a satisficing approach -- (long) list of open questions. our agent will be satisfied with any plan that meets its constraints and achieves its goals. The expressive Safety power of our constraint language is weaker than that Some conditions are so hazardous that our agent of utility functions, but our constraints are easier to should never cause them. For example, we might de- incorporate into standard planning algorithms. mandthat the agent never delete I~TEXfiles, or never By using a general, temporal logic such as that of handle a gun. Since these instructions hold for all [33] or [9, Ch. 5] we could specify constraints that times, we refer to them as dont-distuxb constraints, would ensure the agent would not cause harm. Before and say that an agent is safe when it guarantees to executing an action, we could ask an agent to prove abide by them. As in Asimov’s Law, dont-disturb that the action is not harmful. While elegant, this ap- constraints override direct humanorders. Thus, if we proach is computationally intractable as well. Another ask a software agent to reduce disk utilization and it alternative would be to use a planner such as ILP [3, 2] can only do so by deleting valuable IbTEXfiles, the or ZENO[27, 28] which supports temporally quantified agent should refuse to satisfy this request. goals but, at present, these planners seem too ineffi- We adopt a simple syntax: clout-disturb takes a cient1 for our needs. single, function-free, logical sentence as argument. For Instead, we aim to make the agent’s reasoning about example, one could commandthe agent avoid deleting harm more tractable, by restricting the content and files that are not backed up on tape with the following form of its theory of injury. 2 Weadopt the standard constraint: assumptions of classical planning: the agent has complete information of the initial state of the world, the dont-disturb(written.to.tape(f)V isa(f, file)) agent is the sole cause of change, and action execu- Free variables, such as f above, are interpreted as tion is atomic, indivisible, and results in effects which universally quantified. In general, a sequence of ac- are deterministic and completely predictable. Section tions satisfies dont-disturb(C) if none of the actions considers relaxing these assumptions. On a more syn- make C false. Formally, we say that a plan satisfies tactic level, we make the additional assumption that an dont-disturb constraint when every consistent, the agent’s world model is composed of ground atomic totally-ordered, sequence of plan actions satisfies the constraint as defined below. 1Wehave also examined previous work on "plan qual- ity" for ideas, but the bulk of that workhas focused on the Definition: Satisfaction of dour-disturb: Let w0 problemof leveraging a single action to accomplishmulti- be the logical theory describing the initial state of the ple goals thereby reducing the numberof actions in, and world, let A1,...,A, be a totally-ordered sequence of the cost of, the plan [20, 30, 39]. Whilethis class of op- actions that is executable in w0, let wj be the theory timizations is critical in domainssuch as database query describing the world after executing Aj in wj-1, and optimization, logistics planning, and others, it does not ad- let C be a function.free, logical sentence. We say that dress our concerns here. 2Looselyspeaking, our approach is reminiscent of clas- 3Althoughunpalatable, this is standard in the planning sical workon knowledgerepresentation, whichrenders in- literature. For example, a STrtWsoperator that movesblock ference tractable by formulating restricted representation A from B to C must delete on(A,B) and also add clear(B) languages[21]. even though clear(z) could be defined as Vy"-on(y,

18 AI,...,A, satisfies the constraint dent-disturb(C) vlolatlon(E, C) if/or all j E [1, n], for all sentences C, and for all l. LetR :-{} substitutions 0, 2. For each disjunction D E C do 3. Foreach literal e E E do /fw0 ~ CO then w i ~ CO (1) 4. If e unifies with f E D then add {’~ I = ~ (D- {/})} to R Note that unlike the behavioral constraints of [11, 5. Return R 12] and others, dent-disturb does not require the agent to make C true over a particular time interval; rather, the agent must avoid creating any additional Figure 1: violation computes the conditions (repre- violations of C. For example, if C specifies that all of sented in DNF) under which an effect consequent Gore’s files be read protected, then dent-disturb(C) .... ~will violate constraint C. Returning R = {} -_- false commands the agent to avoid making any of Gore’s means no violation, returning {... {}...} means nec- files readable, but if Gore’s .plan file is already read- essary violation. Weassume that E is a set of literals able in the initial state, the agent need not protect (implicit conjunction) and C is in CNF:i.e., a set of that file. This subtle distinction is critical if we want sets representing a conjunction of disjunctions. to make sure that the behavioral constraints provided to an agent are mutually consistent. This consistency as long as the planner commits to ensuring that ex- problem is undecidable for standard behavioral con- ecution will not result in E. This is achieved by straints (by reduction of first-order satisfiability) but adding --S as a new subgoal to be made true at the is side-stepped by our formulation, because any set of time5 when Ap is executed. dent-disturb constraints is mutually consistent. 3. Evade: Alternatively, by definition of violation it Synthesizing Safe Plans is ok to execute Ap as long as R = violation(E, C) To ensure that an agent acts safely, its planner must will not be true after execution. The planner can generate plans that satisfy every dent-disturb con- achieve this via goal regression, i.e. by computing straint. This can be accomplished by requiring that the causation preconditions [25] for -~R and Ap, to the planner make a simple test before it adds new ac- be6 made true at the time when Ap is executed. tions into the plan. Suppose that the planner is con- 4. Refuse: Otherwise, the planner must refuse to add sidering adding the new action Ap to achieve the sub- Ap and backtrack to find another way to to support goal G of action A~. Before it can do this, it must it- G for A~. erate through every constraint dent-disturb(C) and For example, suppose that the agent is operating every effect E of Ap, determining the conditions (if under the written, to. tape constraint mentioned ear- any) under which E violates C, as defined in figure 1. For example, suppose that an effect asserts -,P and lier, and is given the goal of reducing disk utilization. Suppose the agent considers adding arm paper.rex the constraint is dont-disturb(P V Q), then the ef- action to the plan, which has an effect of the form fect will violate the constraint if--Q is true. Hence, ~isa(paper.tex,file).Since violationreturns violation(-~P, PVQ)= "~Q. In general, if violation -~written. to.tape (paper. tex), the rm action threat- returns true then the effect necessarily denies the con- ens safety. To disarm the threat, the planner must straint, if false is returned, then there is no possible perform one of the options above. Unfortunately, dis- conflict, otherwise violation calculates a logical ex- 4 avowal (option one) isn’t viable since paper, rex exists pression specifying whena conflict is unavoidable. in the initial state (i.e., it is of type file). Option two Before adding Ap, the planner iterates through ev- (confrontation) is also impossible since the threatening ery constraint dont-disturb(C) and every effect con- effect is not conditional. Thus the agent must choose sequent E of Ap, calculating violation(E, C). between either refusing to add the action or evading violation ever returns something other than False, its undesired consequences by archiving the file. then the planner must perform one of the following four repairs: 5Notethat -,S is strictly weakerthan Pednault’s preser- 1. vation preconditions [25] for Apand C; it is moreakin to Disavow:If E is true in the initial state, then there preservation preconditions to a single effect of the action. is no problem and Ap may be added to the plan. °Whileconfrontation and evasion are similar in the sense 2. Confront: If Ap’s effect is conditional of the form that they negate a disjunct (S and R, respectively), they dif- when 5 then E then Ap may be added to the plan fer in two ways.First, confrontation’ssubgoal -,S is derived from the antecedent of a conditional effect while evasion’s 4If E contains "lifted variables" [24] (as opposedto uni- -~R comesfrom a disjunctive constraint, dont-disturb(C), versally quantified variables which pose no problem) then via violation. Second, the subgoals are introduced at dif- violation may return an overly conservative R. Sound- ferent times. Confrontation demandsthat -~S be madetrue ness and safety are maintained, but completeness could be before Ap is executed, while evasion requires that -~R be lost. Webelieve that restoring completeness would make true after execution of Ap. This is whyevasion regresses R violation take exponential time in the worst case. through Ap.

19 Analysis Definition 2 differs from Definition I in two ways: (1) Two factors determine the performance of a planning restore constraints need only be satisfied in w,, after system: the time to refine a plan and the number of the complete plan is executed, and (2) the goal takes plans refined on the path to a solution. The time per precedence over restore constraints. Our constraints refinement is affected only when new actions are added obey a strict hierarchy: dent-disturb takes priority to plan: each call to violation takes O(ec) time where over restore. Note also that restore constraints are e is the numberof consequent literals in the action’s guaranteed to be mutually consistent (assuming a con- effects and c is the number of literals in the CNFen- sistent initial state) since they merely restore aspects coding of the constraint. Whena threat to safety is of w0. detected, the cost depends on the planner’s response: disavowal takes time linear in the size of the initial :’:Synthesizing Tidy Plans state, refusal is constant time, confrontation is linear The most straightforward way to synthesize a tidy plan in the size of S, and the cost of evasion is simply the is to elaborate the agent’s goal with a set of "cleanup" time to regress R through Ap. goals based on its restore constraints and the initial It is more difficult to estimate the effect of state. If the agent’s control comes from a subgoal dent-disturb constraints on the number of plans ex- interleaving, partial order planner such as ucPoP [29], plored. RefusMreduces the branching factor while the then the modification necessary to ensure tidiness is other options leave it unchanged(but can add new sub- straightforward. The agent divides the planning pro- goals, as do confrontation and evasion). In some cases, cess into two phases: first, it plans to achieve the top the reduced branching factor may speed planning; how- level goal, then it plans to clean up as muchas pos- ever, in other cases, the pruned search space maycause sible. In the first phase, the planner doesn’t consider the planner to search muchdeeper (or even fail to halt) tidiness at all. Oncea safe plan is generated, the agent to find a safe solution. The essence of the task, how- performs phase two by iterating through the actions ever, is unchanged. Safe planning can be reformulated and using the violation function (figure 1) to test as a standard planning problem. each relevant effect against each constraint. For each non-false result, the planner generates new goals as Tidiness follows. (1) If the effect is ground and the correspond- Sometimes dont-disturb constraints are too strong. ing ground instance of the restore constraint, C0, is Instead, one would be content if the constraint were not true in the initial state, then no new goals are nec- satisfied when the agent finished its plan. We de- essary. (2) If the effect is ground and C0 is true in the note this weaker restriction with restore; essentially, initial state, then C0 is posted as a new goal. (3) it ensures that the agent will clean up after itself the effect is universally quantified, then a conjunction -- by hanging up phones, closing drawers, returning of ground goals (corresponding to all possible unifica- utensils to their place, etc. An agent that is guar- tions as in case 2) is posted/ After these cleanup goals anteed to respect all restore constraints is said to have been posted, the planner attempts to refine the be tidy. For instance, to guarantee that the agent previous solution into one that is tidy. If the planner will re-compress all files that have been uncompressed ever exhausts the ways of satisfying a cleanup goal, in the process of achieving its goals, we could say then instead of quitting altogether it simply abandons restore (eompressed (f) ). that particular cleanup goal and tries the next. As with dent-disturb constraints, we don’t require Note that in some cases, newly added cleanup ac- that the agent clean up after other agents -- the state tions could threaten tidiness. For example, cleaning of the world, when the agent is given a command,forms the countertop might tend to dirty the previously clean a reference point. However, what should the agent do floor. To handle these cases, the planner must continue when there is a conflict between restore constraints to perform the violationtest and cleanup-goal gener- and top level goals? For example, if the only way to ation process on each action added during phase two. satisfy a user commandwould leave one file uncom- Subsequent refinements will plan to either sweep the pressed, should the agent refuse the user’s commandor floor (white knight) or preserve the original cleanliness assume that it overrides the user’s background desire by catching the crumbs as they fall from the counter for tidiness? We propose the latter -- unlike matters (confrontation). of safety, the agent’s drive for tidiness should be sec- ondary to direct orders. The following definition makes Analysis these intuitions precise. Unfortunately, this algorithm is not guaranteed to Definition: Satisfaction of restore: Building on eliminate mess as specified by constraint 2. For ex- the definition of dour-disturb, we say that A1,..., An ample, suppose that a top level goal could be safely satisfies the constraint restore(C) with respect goal G if for all substitutions 0 rCase 3 is similar to the expansionof a universally quan- tiffed goal into the universal base [29], but case 3 removes /fw0 ~ C0 then (w. l= CO or G l= -~CO) (2) groundliterals that aren’t true in the initial state.

2O achieved with A~ or Ay and in phase one, the planner Remaining Challenges chose to use A=. If A= violates a restore constraint, A~ does not, and no other actions can cleanup the Some changes cannot be restored, and some resources mess, then phase two will fail to achieve tidiness. One are legitimately consumedin the service of a goal. To could fix this problem by making phase two failures make an omelet, you have to break some eggs. The spawn backtracking over phase one decisions, but this question is, "Howmany?" Since squandering resources will engender exhaustive search over all possible ways clearly constitutes harm, we could tag a valuable re- of satisfying top level goals. sources with a min-consume constraint and demand that the agent be thrifty- i.e., that it use as little as Remarkably, this problem does not arise in the cases possible when achieving its goals. Unfortunately, sat- we have investigated. For instance, a software agent isfying constraints of this form may require that the has no difficulty grepping through old mail files for agent examine every plan to achieve the goal in order a particular message and subsequently re-compressing to find the thriftiest one. Weplan to seek insights the appropriate files. There are two reasons why tidi- into this problem in the extensive body of research on ness is often easy to achieve (e.g., in software domains resource managementin planning [8, 10, 16, 39, 38]. and kitchens [1]): So far the discussion has focused on preventing an ¯ Most actions are reversible. The compress ac- agent from actively harming a human, but as Asimov tion has uncompress as an inverse. Similarly, a noted -- inaction can be just as dangerous. We say short sequence of actions will clean up any mess that an agent is vigilant when it prevents a human from harming herself. Primitive forms of vigilance are in the kitchen. Manyenvironments have been sta- already present in many computer systems, as the "Do bilized [18] (e.g., by implementing reversible com- you really want to delete all your files?" message at- mands or adding dishwashers) in a way that makes them easy to manipulate. tests. Alternatively, one could extenddont-disturb and ¯ Weconjecture that, for a partial-order planner, most restoreprimitives with an additionalargument that cleanup goals are trivially serializable [6] with re- specifiesthe class of agentsbeing restricted. By writ- spects to each other, ingself as the firstargument, one couldachieve the functionalitydescribed in sectionsand ; by writing When these properties are true of restorecon- Sam as the argument,the agentwill clean up after, straints in a domain, our tidiness algorithm does sat- and attemptto preventsafety violations by Sam.Fi- isfy constraint 2. Trivial serializability ensures that nally,by providingeveryone as the firstargument, backtracking over phase one decisions (or previously we demandthat the agentattempt to cleanup after achieved cleanup goals) is unnecessary. Tractability is allother agents and attemptto preventall safetyvi- another issue. Since demanding that plans be tidy is olations.Other classes (besides self and everyone) tantamount to specifying additional (cleanup) goals, couldbe defined,leading to morerefined behavior. requiring tidiness can clearly slow a planner. Further- Our suggestionis problematicfor severalreasons. more if a cleanup goal is unachievable, the planner (I)Since the agent has no representationof the goals might not halt. However, as long as the mess-inducing thatother users are trying to accomplish,it might try actions in the world are easily reversible, it is straight to enforcea generalizedrestore constraint with tidy- forward to clean up for each one. Hence, trivial serial- ingactions that directly conflict with the user’s goal. izability assures that the overhead caused by tidiness In addition,there is the questionof whenthe agent is only linear in the number of cleanup goals posted, shouldconsider the human"finished" -- withoutan which is linear in the length of the plan for the top adequatemethod, the agentcould tidy up whilethe level goals. humanis stillactively working. (2) Moregenerally, the humaninterface issues are complex-- we conjec- turethat users would find vigilance extremely annoy- SFormally,serializability [19] meansthat there exists ing. (3) Givena complexworld where the agentdoes a ordering amongthe subgoMswhich allows each to be not havecomplete information, any any attemptto for- solved in turn without backtracking over past progress. malizethe second half of Asimov’sFirst Law is fraught Trivial serializability meansthat every subgoal ordering withdifficulties. Theagent might reject direct requests allows monotonicprogress [6]. While goal ordering is often importantamong the top level goals, we observe that to performuseful work in favorof spendingall of its cleanup goals are usually trivially serializable once the timesensing to seeif somedangerous activity might be block of top level goals have been solved. For example, happeningthat it mightbe ableto prevent.A solution the goal of printing a file and the constraint of restoring to thisproblem appears difficult. files to their compressedstate are serializable. Andthe se- rialization ordering places the printing goal first and the cleanup goal last. As long as the pla~ner considers the Conclusion goals in this order, it is guaranteed to find the obvious This paper explores the fundamental question origi- uncompress-print-compress plan. nally posed by Asimov: how do we stop our artifacts

21 from causing us harm in the process of obeying our or- [9] E. Davis. Representations of Commonsense ders? This question becomes increasingly pressing as Knowledge. Morgan Kaufmann Publishers, Inc., we develop more powerful, complex, and autonomous San Mateo, CA, 1990. artifacts such as robots and software agents [15, 14]. [10] T. Dean, J. Firby, and D. Miller. Hierarchical Since the positronic brain envisioned by Asimovis not planning involving deadlines, travel times, and yet within our grasp, we adopted the familiar classical resources. Computational Intelligence, 4(4):381- planning framework. To facilitate progress, we have 398, 1988. focused on two well-defined primitives that capture [11] M. Drummond. Situated control rules. In Pro- aspects of the problem: dent-disturb and restore. ceedings of the First International Conference on Consequently, we argued that the well-understood, and Knowledge Representation and Reasoning, May computational tractable i mechanism of threat detec- 1989. tion can be extended to avoid harm. Other researchers have considered related questions. [12] O. Etzioni, S. Hanks, D. Weld, D. Draper, A precursor of dent-disturb is discussed in the work N. Lesh, and M. Williamson. An Approach to of Wilensky and more extensively by Luria [22] under Planning with Incomplete Information. In Proc. the heading of "goal conflict." Similarly, a precursor 3rd Int. Conf. on Principles of Knowledge Rep- of restore is mentioned briefly in Hammondet. a/’s resentation and Reasoning, October 1992. Avail- analysis of "stabilization" [18] under the heading of able via anonymous FTP from "~tp/pub/ai/at "clean up plans." Our advances include precise and cs. washington,edu. unified semantics for the notions, a mechanismfor in- [13] Oren Etzioni. Embedding decision-analytic con- corporating dent-disturb and restore into standard trol in a learning architecture. Artificial Intelli- planning algorithms, and an analysis of the computa- gence, 49(1-3):129-160, 1991. tional complexity of enforcing safety and tidiness. [14] Oren Etzioni. Intelligence without robots (a reply Even so, our work raises more questions than it an- to brooks). AI Magazine, 14(4), December 1993. swers: are constraintslike dont-disturb and restore the "right"way to representharm to an agent?Can we [15] Oren Etzioni, Neal Lesh, and Richard Segal. handletradeofls short of usingexpensive decision theo- Building softbots for UNIX(preliminary report). Technical Report 93-09-01, University of Wash- retictechniques? What guarantees can one provideon ington, 1993. Available via anonymous FTP from resourceusage? Most importantly, how do we weaken "ftp/pub/ai/atcs.washington, edu. theassumptions (laid out in section) of a staticworld and completeinformation? [16] M. Fox and S. Smith. ISIS -- a knowldges-based system for factory scheduling. Expert Systems, References 1(1):25-49, July 1984. [1] P. Agreand I. Horswill.Cultural support for im- [17] Peter Haddawy and Steve Hanks. Utility Mod- provisation.In Proc. 10th Nat. Conf. on Artificial els for Goal-Directed Decision-Theoretic Planners. Intelligence, 1992. Technical Report 93-06-04, Univ. of Washing- ton, Dept. of Computer Science and Engineering, [2] J. Allen. Planning as temporal reasoning. In Pro- ceedings of the Second International Conference September 1993. Available via anonymous FTP from "ftp/pub/ai/atcs. washington,edu. on Principles of Knowledge Representation and Reasoning, pages 3-14, 1991. [18]K. Hammond,T. Converse,and J. Grass.The sta- bilizationof environments. Artificial Intelligence, [3] J. Allen, H. Kautz, R. Pelavin, and J. Tenenberg. 1992.To appear. Reasoning about Plans. Morgan Kaufmann, San Mateo, CA, 1991. [19]R. Korf. Planningas search:A quantitative approach. Artificial Intelligence, 33(1):65-88, [4] Isaac Asimov. Runaround. Astounding Science September 1987. Fiction, 1942. Reprinted in [5]. [20] Amy Lansky, editor. Working Notes of the AAAI [5] Isaac Asimov. /, Robot. Ballantine Books, New Spring Symposium: Foundations of Automatic York, 1983. Planning: The Classical Approach and Beyond, [6] A. Barrett and D. Weld. Characterizing subgoal Menlo Park, CA, 1993. AAAIPress. interactions for planning. In Proc. 13th Int. Joint [21] H.J. Levesque and R. Brachman. A fundamental Conf. on Artificial Intelligence, pages 1388-1393, tradeoff in knowledge representation. In R. Brach- September 1993. man and H.J. Levesque, editors, Readings in [7] D. Chapman. Planning for conjunctive goals. Ar- Knowledge Representation, pages 42-70. Morgan tificial Intelligence, 32(3):333-377, 1987. Kaufmann, San Mateo, CA, 1985. [8] K. Currie and A. Tate. O-plan: the open planning [22] Marc Luria. Knowledge Intensive Planning. PhD architecture. Artificial Intelligence, 52(1):49-86, thesis, UCBerkeley, 1988. Available as technical November 1991. report UCB/CSD88/433.

22 [23] Ginsberg M. and D. Smith. Reasoning about ac- [38] D. Wilkins. Can AI planners solve practical prob- tion I: A possible worlds approach. Artificial In- lems? Computational Intelligence, 6(4):232-246, telligence, 35(2):165-196, June 1988. November 1990. [24] D. McAllester and D. Rosenblitt. System- [39] D. E. Wilkins. Practical Planning. Morgan Kauf- atic nonlinear planning. In Proc. 9th Nat. mann, San Mateo, CA, 1988. Conf. on Artificial Intelligence, pages 634-639, July 1991. internet file at ftp.ai.mit.edu: /pub/users/dam/aaai91c.ps. [25] E. Pednault. Synthesizing plans that contain actions with context-dependent effects. Computa- tional Intelligence, 4(4):356-372, 1988. [26] E. Pednault. ADL: Exploring the middle ground between STRIPS and the situation calculus. In Proceedings Knowledge Representation Conf.,, 1989. [27] J. S. Penberthy and Daniel S. Weld. A new approach to temporal planning (preliminary report). In Proceedings of the AAAI 1993 Symposium on Foundations of Automatic Planning: The Classi- cal Approach and Beyond, pages 112-116, March 1993. [28] J.S. Penberthy. Planning with Continuous Change. PhD thesis, University of Washington, 1993. Available as UWCSE Tech Report 93-12- 01. [29] J.S. Penberthy and D. Weld. UCPOP:A sound, complete, partial order planner for ADL. In Proc. 3rd Int. Conf. on Principles of Knowledge Rep- resentation and Reasoning, pages 103-114, Octo- ber 1992. Available via anonymous FTP from "ftp/pub/ai/at cs. washSngeon, edu. [30] Martha Pollack. The uses of plans. Artificial In- telligence, 57(1), 1992. [31] S. Russell and E. Wefald. On optimal game-tree search using rational-meta-reasoning. In Proceed- ings IJCAI-89, pages 334-340, aug 1989. [32] S. Russell and E. Wefald. Do the Right Thing. MIT Press, Cambridge, MA, 1991. [33] Y. Shoham. Reasoning about Change: Time and Causation from the Standpoint @Artificial Intel- ligence. MIT Press, Cambridge, MA,1988. [34] D. Smith. Controlling backward inference. Artifi- cial Intelligence, 39:145-208, 1989. [35] A. Tare. Generating project networks. In Proc. 5th Int. Joint Conf. on Artificial Intelligence, pages 888-893, 1977. [36] M. Wellman. Challenges for decision-theoretic planning. In Proceedings of the AAA1 1993 Sym- posium on Foundations of Automatic Planning: The Classical Approach and Beyond, March 1993. [37] M. Wellman and J. Doyle. Modular utility reprsentation for decision theoretic planning. In Proc. 1st Int. Conf. on A.L Planning Systems, pages 236-242, June 1992.