0 Jiang et al. / Front Inform Technol Electron Eng 2019 ??(?):0-0

Frontiers of Information Technology & Electronic Engineering www.jzus.zju.edu.cn; engineering.cae.cn; www.springerlink.com ISSN 2095-9184 (print); ISSN 2095-9230 (online) E-mail: [email protected]

Task Planning in Robotics: an Empirical Comparison of PDDL-based and ASP-based Systems

Yuqian Jiang1, Shiqi Zhang†2, Piyush Khandelwal3, Peter Stone1 1Department of Computer Science, The University of Texas at Austin 2Department of Computer Science, SUNY Binghamton 3Cogitai, Inc. E-mail: [email protected]; [email protected]; [email protected]; [email protected] Received mm. dd, yyyy; Revision accepted mm. dd, yyyy; Crosschecked mm. dd, yyyy

Abstract: Robots need task planning algorithms to sequence actions toward accomplishing goals that are impossible through individual actions. Off-the-shelf task planners can be used by intelligent robotics practitioners to solve a variety of planning problems. However, many different planners exist, each with different strengths and weaknesses, and there are no general rules for which planner would be best to apply to a given problem. In this article, we empirically compare the performance of state-of-the-art planners that use either the Planning Domain Description Language (PDDL), or Answer Set Programming (ASP) as the underlying action language. PDDL is designed for task planning, and PDDL-based planners are widely used for a variety of planning problems. ASP is designed for knowledge-intensive reasoning, but can also be used for solving task planning problems. Given domain encodings that are as similar as possible, we find that PDDL-based planners perform better on problems with longer solutions, and ASP-based planners are better on tasks with a large number of objects or in which complex reasoning is required to reason about action preconditions and effects. The resulting analysis can inform selection among general purpose planning systems for particular robot task planning domains.

Key words: Task planning; Robotics; PDDL & Answer Set Programming https://doi.org/10.1631/FITEE.1000000 CLC number: TP

1 Introduction delwal et al., 2017). It is convenient to achieve all these tasks using knowledge declared in a single de- In a general purpose planning system, task plan- scription of the domain, and general purpose plan- ning problems are tackled by problem-independent ning systems are well suited to the task. solvers based on a description of the domain in a arXiv:1804.08229v3 [cs.AI] 25 Feb 2019 declarative language. Such planning systems are ex- In order to design a general purpose planning tremely useful in application domains where many system, a declarative language for formalizing the different planning goals need to be accomplished, or domain first needs to be selected, followed by the the domain description evolves over time. For in- selection of a suitable solver which supports this lan- stance, in an application domain such as robotics, guage. Many different factors affect this selection a mobile service robot may need to solve plan- process. Every language has its limitations in rep- ning tasks such as collecting documents, making de- resenting task planning problems, and given a par- liveries, or providing navigation assistance to visi- ticular language, specific language-dependent tech- tors (Cambon et al., 2009; Erdem et al., 2012; Khan- niques may need to be employed to succinctly for- malize a particular planning problem. For instance, † Corresponding author c Zhejiang University and Springer-Verlag GmbH Germany, part not all languages support default reasoning, which of Springer Nature 2018 might bring difficulties in formalizing some planning Jiang et al. / Front Inform Technol Electron Eng 2019 ??(?):0-0 1 problems. A solver is also typically tied to a partic- soning about action preconditions and effects than ular language, but may not support all features in problems in the IPC, which focus on generating long that language, requiring careful construction of the plans efficiently. Various properties of the domain domain description using only supported features. or task in these benchmarks are also varied during Additionally, the properties of the domain can af- evaluation to analyze the effect on planning time. fect how quickly a given pair of language and solver The goal of this article is to help a robot planning can solve planning problems. For instance, some practitioner understand the effects of specific domain domains include many objects and their properties, properties to aid the choice of language selection for which can be challenging to some planning systems. general purpose planning. Finally, given a language, a solver and a planning We hypothesize that current state-of-the-art problem, there can be many ways of formalizing the PDDL-based planners perform better on tasks with problem using the language. For these reasons, care- long solutions, and ASP-based planners tend to per- ful consideration needs to be given to the selection form better on shorter tasks with a large number of language and solver. of objects. The hypothesis is confirmed in all three This article aims to help in the language se- benchmark domains. We also hypothesize that ASP- lection process, given a task planning problem at based planners outperform PDDL-based planners in hand. Specifically, we compare two declarative lan- domains where complex reasoning (specified in Sec- guages: the Planning Domain Definition Language tion 3.1) is required to reason about action precondi- (PDDL) (McDermott et al., 1998), the most pop- tions and effects. This is observed in a controlled ex- ular language in the planning community, and An- periment of the Robot Navigation and Blocks World swer Set Programming (ASP) (Gelfond and Kahl, domains. To the best of our knowledge, this is the 2014; Lifschitz, 2008), a popular general knowledge first work on empirical comparisons between PDDL- representation and reasoning (KRR) language that based and ASP-based planning systems, and can has been recently used in a variety of task planning serve as a useful reference to robot planning practi- problems (Lifschitz, 2002; Yang et al., 2014; Erdem tioners. et al., 2016), including robotics (Erdem and Patoglu, 2018). PDDL was created for the explicit purpose 2 Background of solving planning problems, whereas the develop- ment of ASP has focused on a broader set of reason- Research in task planning dates back to one of ing tasks, such as inference and diagnosis, as well as the earliest research areas in Artificial Intelligence. planning. Since the development of STRIPS (Fikes and Nils- The main contribution of this article is, within son, 1971) (as part of the Shakey robot project), the context of robotics, a comparison of planning many languages have been developed for represent- time between ASP-based and PDDL-based task ing task planning domains. Such languages typically planners when both are used to model the same need to describe actions’ preconditions and effects, domain. Evaluation is performed across three dif- and are commonly known as action languages.A ferent benchmark problems. While it may be pos- summary of some early action languages is avail- sible to construct ASP and PDDL planners specif- able (Gelfond and Lifschitz, 1998). ically suited to these specific benchmarks, domain- In order to plan for real-world problems (such independent solvers are compared in this article. Al- as robot systems), action languages first need to be though planner performance can be sensitive to do- capable of formally representing complex planning main encoding, we take care, to the extent possible, domains. Some of the recent research in task plan- to encode the domains similarly in each language. ning is focused on developing languages that im- The benchmark problems consist of the Blocks World prove the representation capability of action lan- and Hiking problems from the International Plan- guages (Giunchiglia et al., 2004; Lee et al., 2013; ning Competition (IPC) (Coles et al., 2012), as well Babb and Lee, 2015). Planning using these action as a variant of the Robot Navigation problem (Yang languages usually requires a translation to more gen- et al., 2014; Zhang et al., 2015). The Robot Navi- eral knowledge representation and reasoning (KRR) gation problem typically requires more complex rea- languages such as ASP. A planning paradigm for 2 Jiang et al. / Front Inform Technol Electron Eng 2019 ??(?):0-0

ASP was proposed (Lifschitz, 2002), and has been trajectories, such as to reason about history, non- used in many real-world applications (Chen et al., determinism, or both for diagnosis purposes. From 2010; Erdem et al., 2012; Yang et al., 2014). In this the perspective of design purposes, PDDL is an ac- article, we follow the same planning paradigm when tion description language, and ASP is an action encoding domains in ASP. query language, though their implementations often- In parallel, another line of research in task times support both description and query functional- planning aims at more efficient planning algorithms ities. At the same time, action language systems are and their implementations. PDDL (McDermott usually implemented by compilation. For instance, et al., 1998) was developed as a common formal- Coala (Gebser et al., 2010) is one such compilation ism with the goal of allowing more direct com- system that provides compilation techniques for sev- parison of planning algorithms and implementa- eral action languages. tions. Since then, many efficient search algo- There is existing research on predicting plan- rithms have been developed for task planning prob- ning time using features of domains and prob- lems, such as Fast-Forward (Hoffmann, 2001) and lems (Fawcett et al., 2014), or more generally on Fast-Downward (Helmert, 2006). These algorithms predicting time required to solve a problem (Leyton- have publicly available implementations including Brown et al., 2002). These methods can be used to SAYPHI (Rosa et al., 2007), LAMA (Richter et al., help a planning practitioner to estimate the difficulty 2011) and FDSS (Helmert et al., 2011). of a planning problem, after planning language and PDDL requires axioms, in the form of log- system have been selected. In contrast, this work ical formulas, for reasoning within a situation aims at analyzing what domain properties affect the (whereas action descriptions are used for reason- performances of existing planning systems, and can ing across successive situations). A fundamental serve as a reference on the selection of action lan- difference between PDDL and ASP is on their guages used for encoding planning problems. (non)monotonicity property. The axiom-based rea- soning in PDDL is monotonic in the sense of logical 3 Domain Formalization reasoning, meaning that previously achieved conclu- sions remain when new information becomes avail- In this section, we introduce the three bench- able. In contrast, ASP is nonmonotonic, so it allows mark domains, namely Robot Navigation, Blocks removal of previously achieved conclusions given new World and Hiking, and formally describe each in information. The nonmonotonic property of ASP both ASP and PDDL. makes it useful in tasks that require default rea- It may be possible that different styles of en- soning and reasoning about inertial facts. Existing coding would result in different planning times. To research has studied translating PDDL programs ensure the conclusion is general and fair, we se- to ASP (Gebser et al., 2011), and applying ax- lect the benchmark domains from a variety of ori- ioms extracted from PDDL programs to ASP-based gins, and follow the encoding in existing literatures. planning (Miura and Fukunaga, 2017). In particu- Blocks World and Hiking have been used in the lar, robotics researchers have developed robot nav- IPC, and PDDL-based planners have been specif- igation algorithms that switch between ASP-based ically designed to solve problems in the IPC. The and PDDL-based planning systems on mobile ser- PDDL versions are used as is, and translated into vice robots (Lo et al., 2018). However, none of this equivalent ASP code. Robot Navigation is a do- research conducted empirical comparisons over the main used to demonstrate planning using an ASP performances of the state-of-the-art PDDL-based description (Yang et al., 2014), and contains proper- and ASP-based planning systems. ties such as recursive action effects that are missing Action languages can be further categorized as from IPC domains. We translate the complex rea- Action Description Languages and Action Query soning rules of the Robot Navigation domain from Languages (Lifschitz, 1997). Action description ASP as PDDL axioms. languages focus on specifying the transition sys- The languages themselves require certain differ- tem. Given a transition system, action query lan- ences in domain encodings, and direct comparisons guages are used for reasoning about properties of are complicated by the fact that typically the people Jiang et al. / Front Inform Technol Electron Eng 2019 ??(?):0-0 3 doing the encoding have greater familiarity with one • hasdoor(R,D): This predicate specifies that language or the other. We acknowledge that different room R has door D to move to an adjacent lo- encodings may be more suitable to each planner, but cation. Statements expressed using hasdoor it is infeasible to control the encoding style in prac- are specified during initialization, and do not tice. In this work, we ensure the fairest comparison change over time. The PDDL expression is possible by enforcing in all translations that both the (hasdoor ?r - room ?d - door). ASP and PDDL versions for a given domain have exactly the same set of predicates and actions, along • connected(R1,R2): The connected predicate in- with the same preconditions and effects. Specifically, dicates that room R1 is directly connected to an action should be allowed to execute on the same room R2 without a door. Similar to hasdoor, set of states, and it should make the same change to this predicate is used during initialization to de- the state, regardless of the language. Consequently, scribe directly connected locations. In PDDL, statements are described as (connected ?r1 - as we use optimal ASP and PDDL planners in the experiments, they generate identical plans, and only room ?r2 - room). planning times need to be compared. A detailed ex- • acc(R1,R2): acc specifies that room R1 is acces- planation of how ASP can be used for planning is sible from room R2 via a single navigation action available in previous work (Lifschitz, 2002). executed by a low level controller. Intuitively, any two rooms that are not separated by a closed 3.1 Robot Navigation door are accessible, i.e. the low level controller The robot navigation domain differs from clas- can navigate from one room to another. sical planning domains in the IPC in that complex R1 is accessible from R2 if R1 is directly connected reasoning needs to be performed to ensure that ac- to R2, or R1 and R2 share the same door D which tion preconditions are met, and that action effects is open. Furthermore, acc is both commutative, are executed correctly. Specifically, this domain fea- i.e. R1 is accessible from room R2 if R2 is acces- tures action effects that require recursive reasoning sible from R1, as well as associative, i.e. if both to compute the final state of the world. In this do- R1 and R2 are accessible from r3, then they’re main, a mobile robot navigates an office floor which accessible from one another. This associative consists of a set of rooms that are connected to one property requires a recursive definition: another. Rooms can be connected to one another acc(R1,R2,n) :- connected(R1,R2). via doors, and closed doors need to be opened by acc(R1,R2,n) :- open(D,n), the robot before it can pass through. Alternatively, hasdoor(R1,D), hasdoor(R2,D). rooms can be directly connected to one another such acc(R1,R2,n) :- acc(R2,R1,n). that access is always possible from any location in acc(R1,R2,n) :- acc(R1,R3,n), one room to any location in the other. acc(R3,R2,n). The robot has the following perception and ac- The recursive formulation of acc can be ex- tuation modules available. Using a low-level con- pressed in PDDL using derived predicates: troller, the robot can traverse to any room from its current location if its path is not blocked by a closed (:derived (acc ?r1 - room ?r2 - room) (or (connected ?r1 ?r2) door, and this navigation can be encoded by a single (exists (?d - door) (and (open ?d) high-level symbolic action goto. Furthermore, the (hasdoor ?r1 ?d) (hasdoor ?r2 ?d))) robot has some means of opening a closed door when (acc ?r2 ?r1) it is next to it, either by enlisting human aid, or us- (exists (?r3 - room) (and (acc ?r1 ?r3) (acc ?r3 ?r2))))) ing a robot arm to open the door. On the perception side, the robot can sense its location, whether or not • at(R,n): at is used to specify that the robot is it is next to a door, and whether or not a door is at room R at timestep n (of the high-level plan). open. This predicate is inertial, i.e. the robot remains The domain knowledge can be formalized in in room R if there is no evidence showing it is ASP by statements defined using the following pred- not in room R anymore, and this property is icates: specified as: 4 Jiang et al. / Front Inform Technol Electron Eng 2019 ??(?):0-0

at(R,n) :- at(R,n-1), not -at(R,n). (:action goto :parameters (?r2 - room) :precondition (exists (?r1 - room) In PDDL, the predicate is expressed as (at ?r - (and (at ?r1) room), and all predicates are inertial by default. (acc ?r1 ?r2))) :effect (and (at ?r2) (forall (?r1 - room) • open(D,n): Door D is open at step n. door is (when (at ?r1) (not (at ?r1)))) inertial, i.e. the robot believes that a door will (visited ?r2) stay in the same state unless sensed differently. (forall (?d1 - door) In ASP, the inertial property for this predicate (not (canopen ?d1))))) is represented as: The action effects in PDDL have an additional canopen open(D,n) :- open(D,n-1), not -open(D,n). statement than ASP to indicate that is not inertial. In PDDL, open is expressed as (open ?d - door). • approach(D,n): This action specifies that the robot should approach door D. The action is only • canopen(D,n): The robot can open door D if it executable when the robot is in a room that has is right next to it. canopen is the action effect of door D. After executing this action, the robot can approaching a door, and a precondition before open door D. In ASP, this action is expressed as: the door can be opened. canopen is not iner- tial. In PDDL, the predicate is expressed as :- approach(D,n), at(R1,n-1), not hasdoor(R1,D). canopen(D,n) :- approach(D,n). (canopen ?d - door). In PDDL, this action is expressed as: • visited(R,n): Once the robot visits a room, the (:action approach visited fluent for that room remains true until :parameters (?d - door) the end of planning process. visited is used to :precondition (exists (?r1 - room) (and (at ?r1) describe goal conditions. The persistence prop- (hasdoor ?r1 ?d))) erty is expressed in ASP as: :effect (and (canopen ?d) (forall (?d1 - door) visited(R,n) :- visited(R,n-1). (when (not (= ?d1 ?d)) (not (canopen ?d1))))))

(visited ?r - room) expresses visited in The action effects in PDDL both express that PDDL. door d can be opened, and no other doors in the domain can be opened without approaching There are three actions in the domain: goto, them first. approach, and opendoor, with the following defini- tions: • opendoor(D,n): This action allows the robot to open door D if canopen(D) is true. Opening • goto(R2,n): This action specifies that the robot an open door does not change the state of the should navigate to room R2 at timestep n in the world. This action is represented in ASP as: high-level plan. The precondition for this action :- opendoor(D,n), not canopen(D,n-1). open(D,n) :- opendoor(D,n). is that robot must be in a room R1 from which R2 is accessible. Once the robot goes to room In PDDL, this action is represented as: R2, the goal condition visited is set to true for (:action opendoor that room as well. The ASP rules defining the :parameters (?d - door) action preconditions and effects are as follows: :precondition (canopen ?d) :effect (and (open ?d) (forall (?d1 - door) :- goto(R2,n), at(R1,n-1), not acc(R1,R2,n-1). (not (canopen ?d1))))) at(R2,n) :- goto(R2,n). -at(R1,n) :- goto(R2,n), at(R1,n-1), R1 != R2. visited(R2,n) :- goto(R2,n). In all three actions, a specific action effect in PDDL describes the non-inertial property of The same description in PDDL is expressed as canopen. This effect is not required in ASP because follows: the canopen is not inertial. Jiang et al. / Front Inform Technol Electron Eng 2019 ??(?):0-0 5

The goal in the Robot Navigation domain is to (:derived (above ?x ?y) (or (on ?x ?y) visit a randomly selected set of rooms. In order to (exists (?z) (and (on ?x ?z) (above ?z ?y))))) visit a room, the robot needs to recursively reason about which rooms are accessible from one another. The predicate is defined in ASP as: If a door in its path is closed, the robot needs to ex- above(X,Y,n) :- on(X,Y,n). opendoor plicitly approach the door and execute an above(X,Y,n) :- on(X,Z,n), above(Z,Y,n). action. Whenever opendoor is executed, the direct accessibility of rooms changes, and this action effect The complete domain description in PDDL3 needs to be computed recursively. This recursive and ASP4 are online. We use both the original and property of the domain differentiates it from tradi- the extended versions of the Blocks World domain in tional IPC planning domains such as Blocks World experiments. and Hiking. 3.3 Hiking 3.2 Blocks World We select the hiking domain, new in IPC-2014, In the Blocks World domain, the goal is to move as our third benchmark domain. The hiking domain a set of stackable blocks from one configuration to features negative preconditions. We use the official another using a robot hand. We use the official do- PDDL formalization in IPC, and an equivalent def- main definition in IPC-2011 as the PDDL domain inition in ASP for this study. description, and translate the domain to ASP. The In short, the purpose of this domain is to arrange full description of the IPC domain is available on- activities for a number of couples so each couple can 1 2 line in PDDL and ASP . We only describe the hike to their destination with a tent ready. A hiking translation of the pick-up action to ASP as an illus- problem specifies connections between places, and trative example; the other actions (put-down, stack, initial locations of couples, cars, and tents. A typical and unstack) follow similarly. pick-up allows a robot plan transports and sets up tents at the destination, to pickup a block that has no blocks underneath it drives each couple to the starting point of their hike, (designated by ontable). Furthermore, it should also and then has them walk together along the hike. have no blocks stacked on top of it (designated by The complete description of the Hiking domain is clear). Finally, a block can only be picked up if available online for both PDDL5 and ASP6. the robot hand is empty (designated by handempty). The effect of the action is that the robot hand is holding the block, and all preconditions for the ac- 4 Experiments tion become false. This action description translated The experiments in this section are designed to to looks as follows: ASP compare planning times when the domains and prob- :- pickup(B,n), not clear(B,n-1). lems formalized in the previous section are optimally :- pickup(B,n), not ontable(B,n-1). solved using state-of-the-art solvers. Our encoding :- pickup(B,n), not handempty(n-1). -ontable(B,n) :- pickup(B,n). strategy and the award-winning optimal planning -clear(B,n) :- pickup(B,n). systems ensure that all planners generate the same -handempty(n) :- pickup(B,n). holding(B,n) :- pickup(B,n). plan given the same pair of domain and problem. We select FastDownward (Helmert, 2006) with The first three statements specify the same ac- the setting FDSS-1 (Helmert et al., 2011), which tion preconditions specified in the PDDL descrip- had the highest score in the sequential optimization tion, and the last four statements specify the same track at IPC 2011. It should be noted that the ver- action effects as specified in the PDDL description. sions of IPC after 2011 do not take into account or The extended version of the Blocks World do- announce the time needed to solve each problem. main introduces a recursively defined predicate: Instead, a hard time constraint (e.g., 30 minutes) is above (Thiébaux et al., 2005a). The PDDL defi- 3 nition is as follows: http://pastebin.com/raw/FwZgGmZf 4http://pastebin.com/raw/9D2PuNze 1http://pastebin.com/raw/b07aMTJB 5http://pastebin.com/raw/v3wkv57W 2http://pastebin.com/raw/SAqM3xbF 6http://pastebin.com/raw/Dw1BwG0Z 6 Jiang et al. / Front Inform Technol Electron Eng 2019 ??(?):0-0

103

102

101

100

small domain: clingo Planning time (seconds) 10−1 small domain: downward large domain: clingo large domain: downward 10−2 0 2 4 6 8 10 12 Number of rooms need to be visited Fig. 1 Robot Navigation: Small Domain - 10 rooms, Large Domain - 15 rooms. 40% of rooms are connected via doors, and the rest are directly accessible from the corridor. given to all planners. As a result, most PDDL plan- formance of different planning paradigms. It should ners always use the maximum time allowed to avoid be noted that we are generally more interested in reporting suboptimal or incorrect solutions (which comparing sensitivities of planning systems (instead is greatly penalized), and are therefore unfit for a of comparing individual data points) given different comparison of planning times. The Robot Navi- domain characteristics. The sensitivity that can be gation domain and the extended Blocks World do- reflected by the “trend” of a series of data points main require derived predicates. Since none of the is typically more robust to implementation details planners in the optimization track support derived of planning systems such as programming languages predicates, we use FastDownward with the setting and compilers. The goal of these evaluations is to test LAMA-2011 (Richter et al., 2011), the winner of se- the hypothesis that PDDL-based approaches work quential satisficing track in 2011. We use version better in situations where the generated plans have 4.5.4 of Clingo (Gebser et al., 2014) in incremental many steps, and ASP-based approaches work better mode as the ASP solver. Clingo is an Answer Set in situations where the domain is large or substan- solving system that integrates Clasp, the winner of tial reasoning is required at every step of the plan to the fifth Answer Set Programming Competition in compute the world state. 2015 (Calimeri et al., 2016). Planner performances are evaluated on a Robot Navigation Results from the Robot Nav- general-purpose High Throughput Computing igation domain are shown in Figure 1. Two different (HTC) cluster that is operated by the Department versions of the domain have been created. The small of Computer Science at the University of Texas domain has 10 rooms, and each of them is connected at Austin. We filter out machines with less than a central corridor. 4 rooms are connected via doors, 8GB memory in the experiments, resulting in more and the rest are directly connected. In contrast, the than ten machines with different hardware configura- large domain has 15 rooms where 6 rooms are con- tions (e.g., memory ranging from 8GB to more than nected via doors to the corridor. 500GB). All data points are averaged across at least The robot is initially located in the corridor, and 10 trials to reduce noise, and standard deviations are as its goal, needs to visit a number of rooms (repre- reported. Given the wide range of hardware config- sented on the x-axis in Figure 1) that are randomly urations and statistical analysis, we aim to conclude selected in each trial. Since rooms that are connected with observations that are generally valid and useful by doors take more steps to visit, we increase the to practitioners. number of trials (to 50), on which we compute the During evaluations, various domain characteris- average planning time for each data point. Regard- tics are also changed to measure the difference in per- less, the number of rooms in the goal is positively Jiang et al. / Front Inform Technol Electron Eng 2019 ??(?):0-0 7

103

102

101

100

Planning time (seconds) small domain: clingo 10−1 small domain: downward large domain: clingo large domain: downward 10−2 1 2 3 4 5 6 7 8 9 10 11 Height Fig. 2 Blocks World: Small Domain - 15 blocks, Large Domain - 60 blocks. We use a timeout of 1800 seconds (same as IPC). The graph only plots configurations where all trials of both planners finished before timeout (same for all following experiments). correlated with the plan length. The planning time planners is to generate a plan that builds a single for each planner is represented on the y-axis (log stack of a specified height with randomly specified scale, same for all following figures). blocks. The plan length is proportional to the height We can observe that the red curves have smaller of the stack, and is represented as the x-axis in Fig- slopes, since they intersect (or will do so) with the ure 2. All results have been averaged across 10 trials. blue curves. This confirms our hypothesis that The two red lines are far apart and almost hor- PDDL planners are better at solving planning prob- izontal, i.e. the PDDL planner slows down sig- lems which require a large number of steps. We nificantly in the larger domain, but planning time can also observe that the gap between red curves is does not depend on plan length. The planning larger than the gap between blue curves, even on log- time of Clingo grows as the plan length increases, arithmic scale. This observation again supports the but the difference between different sized domains hypothesis that ASP-based planning is less sensitive is smaller than PDDL. These observations support to object scaling. our hypotheses that PDDL-based planning is fast Furthermore, ASP-based planning is much at producing long plans, while ASP-based planning faster than PDDL-based planning when the num- is faster in large domains where smaller plans are ber of rooms is less than eight, and finishes within necessary. a reasonable amount of time. This is especially use- Figure 3 compares the performance of each plan- ful in domains such as robotics where fast real-time ner between the regular Blocks World domain and operation is necessary. This better performance is the extended version. In this experiment, the do- probably a consequence of recursive action effects main has 10 blocks. The task is the same as before: embedded in the domain. We further verify this ob- building a tower of various heights from unstacked servation in a controlled experiment using the Blocks blocks. The difference of the extended domain is World domain. that the goal condition is expressed with the predi- cate above, where the planners reason about recur- Blocks World Figure 2 reports results from the sive effects. Similarly, the results are averaged across regular Blocks World domain. Similarly, two ver- 10 trials. sions of the domain are evaluated: a small domain In Figure 3, there is a small difference in ASP with 15 blocks in the environment, and a large do- solving time, but a significant change in the perfor- main with 60 blocks in the environment. Initially, mance of the PDDL planner. In contrast to Fig- all blocks are unstacked and on the table (so as to ure 2, we can observe that complex reasoning affects control the optimal plan length). The goal of the PDDL planning time in its slope with respect to plan 8 Jiang et al. / Front Inform Technol Electron Eng 2019 ??(?):0-0

103 extended blocksworld: clingo

2 extended blocksworld: downward 10 regular blocksworld: clingo regular blocksworld: downward 101

100 Planning time (seconds) 10−1

10−2 1 2 3 4 5 6 7 Height

Fig. 3 Blocks World: Regular Domain vs. Extended Domain. length, whereas object scaling shifts the curve up. even take two parameters of each. Therefore, in- The observation confirms that ASP-based planning creasing the number of these objects complicates the is better for domains that involve complex reason- grounding7 of ASP problems. We also observe that ing (reasoning about recursive action effects in this most of the increased planning time of Clingo is in case). grounding. For instance, when the number of couple increases from 5 to 10 – see Figure 4(c), average solv- Hiking Figure 4 and 5 show results from the Hik- ing time for one trip stays below 0.8 second, while ing domain. In each graph, the x-axis is the total average grounding time grows from 16.0 seconds to number of trips made by all couples, and the y-axis is 354.2 seconds. Based on these two observations, we the planning time in seconds. Figure 5 shows the per- hypothesize that if the domain has actions that check formance of each planner for two sizes of the domain. or change the state of many objects, Clingo’s advan- Although we can make similar observations that the tage at planning in large domains can be canceled PDDL planner becomes faster than the ASP plan- out by the extra grounding time. A further analysis ner at higher plan lengths but slower with more ob- of how parameter type and number affect grounding jects, the evidence is weaker than the results above. time is left for future research. Since the Hiking domain has four types of objects that affect planning in different ways, the domain Remark All of the above results support our hy- size has four dimensions. Figure 4 shows a more pothesis that PDDL-based approaches work better controlled set of experiments in which the larger do- in situations where the generated plans have many main increases only one type of objects. Objects are steps, and ASP-based approaches work better in sit- added in a way that does not affect the plan length or uations where the domain is large or substantial rea- the number of optimal plans at each point of x-axis. soning is required at every step of the plan to com- In all graphs the slope of blue curves is larger than pute the world state. the slope of red curves. So the hypothesis about the Given the wide range of hardware configurations plan length holds for all object types. of machines in the HTC cluster and the low standard When adding cars and tents to the domain, from deviation values reported in the results, we can see Figure 4(a) and 4(b), we can observe that the ASP machines of different configurations did not cause solver is less sensitive than the PDDL planner, but significant differences in planning time, which indi- there is no significant difference in the case of cou- cates that the trends are consistent across different ples and places – see Figure 4(c) and 4(d). We find types of computing platforms. that couples and places are more heavily used as 7Plan generation in ASP is a two-step process that in- action parameters than other two types of objects. cludes grounding and solving, where the grounding step out- Actions such as drive_passenger and walk_together puts a variable-free representation. Jiang et al. / Front Inform Technol Electron Eng 2019 ??(?):0-0 9

small domain: clingo small domain: downward large domain: clingo large domain: downward

10 3 10 3

10 2 10 2 Planning time (seconds)

10 1 10 1 1 2 3 4 5 1 2 3 4 5 (a) Small - 3 cars, Large - 10 cars (b) Small - 3 tents, Large - 10 tents

10 4 10 3

10 3

10 2

10 2

10 1 10 1 1 2 3 4 5 1 2 3 4 5 (c) Small - 5 couples, Large - 10 couples (d) Small - 4 places, Large, 8 places

Fig. 4 Varying individual domain attributes in the Hiking domain

5 Conclusions and Discussions ing complex reasoning such as the Robot Navigation domain, ASP-based planners can be considerably In this article, we empirically compared ASP- faster than PDDL-based planners for shorter plans. based and PDDL-based task planners using three Such observations can serve as a useful reference to robotic benchmark domains. PDDL is the domi- task planning practitioners in the process of action nant action language in the task planning commu- language selection. nity; ASP is widely used for knowledge representa- tion and reasoning, and can be used for task plan- This article, by no means, aims to provide a list ning. The analysis in this article demonstrates that of the best planners given a pair of planning domain PDDL-based planners perform better when tasks and planning problem, which is infeasible in prac- require long solutions. However, ASP-based task tice. From a high-level perspective, the observations planners are less susceptible to an increase in the shared in this article are intended to contribute to number of domain objects, as long as the growth in the community’s understanding of how properties of the number of objects does not explode the num- planning domains and problems affect the perfor- ber of grounded actions. Finally, in domains requir- mance of different planning systems. Based on one’s 10 Jiang et al. / Front Inform Technol Electron Eng 2019 ??(?):0-0

104

103

102

101

Planning time (seconds) small domain: clingo small domain: downward 100 large domain: clingo large domain: downward

0.5 1 1.5 2 2.5 3 3.5 4 4.5 5 5.5 Number of trips

Fig. 5 Hiking: Small Domain vs. Large Domain knowledge and intuition about properties of the plan- paradigms. One of the main reasons is that the orig- ning problems at hand, a practitioner can then make inal definition of the PDDL language includes only a more informed choice of the planning system. its syntax, and does not discuss its semantics. As In this work, we selected the Clingo system for a result, different PDDL solvers have different “in- evaluating the ASP-based planning formalism, and terpretations” of PDDL programs, although they all the FastDownward planning systems of FDSS-1 and aim at producing optimal solutions. Another direc- LAMA-2011 for the PDDL-based formalism. These tion for future work is to look into the semantics of award-winning optimal systems represent the state- PDDL (McDermott, 2003; Thiébaux et al., 2005b), of-the-art algorithms and implementations of the two and conduct formal analysis between the two plan- planning paradigms. As the first work on empirical ning paradigms. comparisons of PDDL-based and ASP-based task planners, we focus on a clear presentation of the Acknowledgements methodology and results. We carefully selected the three domains that are as distinct as possible for A portion of this work has taken place in a representative comparison. We believe the con- the Learning Agents Research Group (LARG) at clusions hold in most cases. In the future, we will UT Austin. LARG research is supported in part conduct further evaluations over the two planning by NSF (IIS-1637736, IIS-1651089, IIS-1724157), formalisms and their implementations on other task ONR (N00014-18-2243), FLI (RFP2-000), Intel, planning problems and using other more extreme Raytheon, and Lockheed Martin. Peter Stone serves computing platforms (such as low-end onboard com- on the Board of Directors of Cogitai, Inc. The puters on mobile robots). The experiments in this terms of this arrangement have been reviewed and article were conducted using an ASP-based and a approved by the University of Texas at Austin in PDDL-based planner that are meant to be represen- accordance with its policy on objectivity in research. tative of their respective classes. While we believe References that our results and observations will generalize to other such planners, we acknowledge that there is Babb J, Lee J, 2015. Action language BC+: preliminary report. Proceedings of the Twenty-Ninth AAAI Con- no way to establish that conclusively without empir- ference on Artificial Intelligence (AAAI), p.1424-1430. ically comparing with many other planners, which is Calimeri F, Gebser M, Maratea M, et al., 2016. Design and beyond the scope of this article. results of the fifth answer set programming competition. Artificial Intelligence, 231:151-181. This article does not include formal analysis or Cambon S, Alami R, Gravot F, 2009. A hybrid approach to comparisons between PDDL-based and ASP-based intricate motion, manipulation and task planning. The Jiang et al. / Front Inform Technol Electron Eng 2019 ??(?):0-0 11

International Journal of Robotics Research, 28(1):104- Lifschitz V, 1997. Two components of an action language. 126. Annals of Mathematics and Artificial Intelligence, 21(2- Chen X, Ji J, Jiang J, et al., 2010. Developing high-level cog- 4):305-320. nitive functions for service robots. Proceedings of the Lifschitz V, 2002. Answer set programming and plan gener- 9th International Conference on Autonomous Agents ation. Artificial Intelligence, 138(1):39-54. and Multiagent Systems: volume 1-Volume 1, p.989- Lifschitz V, 2008. What is answer set programming? Pro- 996. ceedings of the 23rd national conference on Artificial Coles A, Coles A, Olaya AG, et al., 2012. A survey of intelligence, p.1594-1597. the seventh international planning competition. AI Lo SY, Zhang S, Stone P, 2018. PETLON: Planning effi- Magazine, 33(1):83-88. ciently for task-level-optimal navigation. Proceedings Erdem E, Patoglu V, 2018. Applications of asp in robotics. of the 17th Conference on Autonomous Agents and KI-Künstliche Intelligenz, :1-7. MultiAgent Systems (AAMAS). Erdem E, Aker E, Patoglu V, 2012. Answer set programming McDermott D, Ghallab M, Howe A, et al., 1998. PDDL-the for collaborative housekeeping robotics: representation, planning domain definition language, . reasoning, and execution. Intelligent Service Robotics, McDermott D, 2003. The formal semantics of processes in 5(4):275-291. PDDL. Proceedings of ICAPS Workshop on PDDL, Erdem E, Gelfond M, Leone N, 2016. Applications of answer p.101-155. set programming. AI Magazine, 37(3). Miura S, Fukunaga A, 2017. Automatic extraction of ax- Fawcett C, Vallati M, Hutter F, et al., 2014. Improved ioms for planning. Proceedings of the Twenty-Seventh features for runtime prediction of domain-independent International Conference on Automated Planning and planners. Proceedings of the Twenty-Fourth Interna- Scheduling (ICAPS), p.218-227. tional Conference on International Conference on Au- Richter S, Westphal M, Helmert M, 2011. Lama 2008 and tomated Planning and Scheduling, p.355-359. 2011. International Planning Competition, p.117-124. Fikes RE, Nilsson NJ, 1971. Strips: A new approach to De la Rosa T, Olaya AG, Borrajo D, 2007. Using cases util- the application of theorem proving to problem solving. ity for heuristic planning improvement. International Artificial intelligence, 2(3-4):189-208. Conference on Case-Based Reasoning, p.137-148. Gebser M, Grote T, Schaub T, 2010. Coala: a compiler from Thiébaux S, Hoffmann J, Nebel B, 2005a. In defense of action languages to asp. European Workshop on Logics PDDL axioms. Artificial Intelligence, 168(1-2):38-69. in Artificial Intelligence, p.360-364. Thiébaux S, Hoffmann J, Nebel B, 2005b. In defense of Gebser M, Kaminski R, Knecht M, et al., 2011. plasp: PDDL axioms. Artificial Intelligence, 168(1-2):38-69. A prototype for PDDL-based planning in ASP. Logic Yang F, Khandelwal P, Leonetti M, et al., 2014. Planning in Programming and Nonmonotonic Reasoning, :358-363. answer set programming while learning action costs for Gebser M, Kaminski R, Kaufmann B, et al., 2014. Clingo mobile robots. 2014 AAAI Spring Symposium Series. = ASP + control: Preliminary report. CoRR, Zhang S, Yang F, Khandelwal P, et al., 2015. Mobile robot abs/1405.3694. planning using action language bc with an abstraction http://arxiv.org/abs/1405.3694 hierarchy. Proceedings of the 13th International Confer- Gelfond M, Kahl Y, 2014. Knowledge representation, reason- ence Logic Programming and Nonmonotonic Reasoning ing, and the design of intelligent agents: The answer-set (LPNMR), p.502-516. programming approach. Cambridge University Press. Gelfond M, Lifschitz V, 1998. Action languages, . Giunchiglia E, Lee J, Lifschitz V, et al., 2004. Nonmonotonic causal theories. Artificial Intelligence, 153(1-2):49-104. Helmert M, Röger G, Karpas E, 2011. Fast downward stone soup: A baseline for building planner portfolios. ICAPS 2011 Workshop on Planning and Learning, p.28-35. Helmert M, 2006. The fast downward planning system. Jour- nal of Artificial Intelligent Research (JAIR), 26:191-246. Hoffmann J, 2001. FF: The fast-forward planning system. AI magazine, 22(3):57. Khandelwal P, Zhang S, Sinapov J, et al., 2017. BWIBots: A platform for bridging the gap between AI and human– robot interaction research. The International Journal of Robotics Research, 36(5-7):635-659. Lee J, Lifschitz V, Yang F, 2013. Action language BC: preliminary report. Proceedings of the Twenty-Third international joint conference on Artificial Intelligence (IJCAI), p.983-989. Leyton-Brown K, Nudelman E, Shoham Y, 2002. Learning the empirical hardness of optimization problems: The case of combinatorial auctions. International Confer- ence on Principles and Practice of Constraint Program- ming, p.556-572.