The Thirteenth International Symposium on Combinatorial Search (SoCS 2020) Applying Monte-Carlo Tree Search in HTN Planning Julia Wichlacz, Daniel Holler,¨ Alvaro´ Torralba, Jorg¨ Hoffmann Saarland University, Saarland Informatics Campus, Saarbrucken,¨ Germany [email protected], [email protected], [email protected], [email protected] Abstract cost plans (sometimes with optimality guarantees depend- ing on the technique used, see e.g. Behnke, Holler,¨ and Bi- Search methods are useful in hierarchical task network undo, 2019b). Here we investigate Monte-Carlo Tree Search (HTN) planning to make performance less dependent on the (MCTS) as a new algorithmic alternative in HTN planning. domain knowledge provided, and to minimize plan costs. Here we investigate Monte-Carlo tree search (MCTS) as a MCTS is a well-known search paradigm, originating new algorithmic alternative in HTN planning. We implement in probabilistic planning problems (Kocsis and Szepesvari´ combinations of MCTS with heuristic search in Panda. We 2006; Keller and Helmert 2013), but successful also in furthermore investigate MCTS in JSHOP, to address lifted games (Silver et al. 2016) and in deterministic planning set- (non-grounded) planning, leveraging the fact that, in contrast tings (Trunda and Bartak´ 2013; Schulte and Keller 2014). to other search methods, MCTS does not require a grounded MCTS is essentially based on sampling the state space, feed- task representation. Our new methods yield coverage perfor- ing back rewards obtained at terminal states to inform the mance on par with the state of the art, but in addition can action decisions on earlier states. This converges to optimal effectively minimize plan cost over time. decisions in the limit, but its main prowess in practice is to take good action decisions in limited time, with an any- 1 Introduction time behavior where decisions improve over time. MCTS can also be combined with heuristic search, as systematized Hierarchical Task Network (HTN) planning complements by Keller and Helmert (2013) in their trial-based heuristic actions causing state transition like in classical planning tree search (THTS) framework. Here, the focus is to balance with a hierarchy on the things to do, the tasks. Abstract tasks exploration and exploitation during search. are not applicable directly and are decomposed in a pro- Our contribution here is the implementation and evalu- cess similar to the derivation of words from a formal gram- ation of MCTS in HTN planning. We do so in two HTN mar. Solutions need to satisfy constraints induced by de- planning systems, namely Panda (Bercher, Keen, and Bi- composition structure and state transition system. The mo- undo 2014) and SHOP (Nau et al. 1999). tivation behind the hierarchy is manifold: it allows the de- Panda is the state-of-the-art system incorporating heuris- scription of complex behavior (Erol, Nau, and Hendler 1994; tic search, thus facilitating a direct empirical comparison, Holler¨ et al. 2014), enables the communication with users on and allowing to leverage prior work on heuristic functions different levels of abstraction (see e.g. Behnke et al., 2019) (Holler¨ et al. 2018). We implement and evaluate THTS. or to add advice, i. e., to guide the search using human do- main knowledge (as e.g. exploited by Nau et al., 2003). SHOP (Nau et al. 1999; 2003) is the state-of-the-art sys- Traditionally, HTN planning systems relied mostly on the tem for lifted HTN planning, not requiring a grounded task informativeness of the domain knowledge provided, mak- representation. This serves to leverage the milder repre- ing do with simple search algorithms like depth-first search sentation requirements of MCTS: in difference to heuristic (Nau et al. 2003). Recent work lines have established more search and SAT, MCTS does not require a grounded task informed search mechanisms, namely sophisticated reacha- representation. This is a key advantage because grounded bility analysis (Bit-Monnot, Smith, and Do 2016; Behnke et representations quickly become prohibitively large when al. 2020), heuristic search (Bercher, Keen, and Biundo 2014; the object universe is large, and/or when predicate/action arity is high. This happens, for example, in natural lan- Bercher et al. 2017; Holler¨ et al. 2018), compilations to clas- guage generation (Koller and Stone 2007; Koller and Hoff- sical planning (Alford, Kuter, and Nau 2009; Alford et al. mann 2010) and in Minecraft planning (Roberts et al. 2017; 2016), and SAT (Behnke, Holler,¨ and Biundo 2018; 2019a; Wichlacz, Torralba, and Hoffmann 2019). Implementing Schreiber et al. 2019). These methods improve HTN plan- MCTS in JSHOP, a version of SHOP, with a number of adap- ning performance, make it less dependent on the domain tations for increased efficiency, we obtain the first non-blind knowledge, and provide it with the ability to search for low- HTN planner not requiring a grounded task representation. 1 Copyright c 2020, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved. 1Attempts have been made to compute heuristic functions with- 82 Evaluating our implementations on standard HTN bench- The problem definition contains an initial task network marks, we find that our new methods are generally on par tnI and an initial state s0. Some task network is a solution if with previous methods in terms of coverage, thus adding and only if it can be derived from tnI via the decomposition new technology options into the HTN arsenal. More specif- process as described before, it contains only primitive tasks, ically, in Panda, it turns out that the exploration term can and there is a (totally ordered) sequence of the tasks in line yield benefits beyond purely relying on heuristics to guide with its ordering definition that is executable in s0. There are search. In JSHOP, a na¨ıve implementation of MCTS lags be- two points that we want to emphasize: hind in coverage, but our adaptations – in particular, using • A planner is not allowed to insert additional actions apart backtracking on MCTS samples to escape shallow dead-end from those included due to decomposition. paths – remove that disadvantage. To evaluate plan-cost minimization over time without • Usually, there is no state-based goal definition given, the guidance of a heuristic function, we create a version of objective of the problem is defined in terms of the tasks JSHOP that continues search after the first plan is found. contained in tnI . Comparing MCTS against this system, we find that MCTS There are several restrictions that define subclasses of the is more effective at improving plan cost over time. This is es- given HTN definition (Bercher, Alford, and Holler¨ 2019), pecially the case on benchmarks aimed at featuring variance mostly on the ordering definitions in methods or on recur- in action costs, including a number of Minecraft benchmark sion allowed in decomposition. The two systems we build studies that we design for lifted planning. on differ in terms of the HTN variants they deal with. Panda The paper is organized as follows. Section 2 gives the re- allows for recursive, partially ordered models. The version quired background in HTN planning, MCTS, and THTS. of SHOP we build on allows for recursion but is restricted to Section 3 describes our THTS implementation in Panda, totally ordered models, i. e. models where all task networks which we empirically evaluate in Section 4. Section 5 de- in method definitions and the initial task network are totally scribes our adaptation of MCTS in JSHOP, Section 6 gives ordered. its evaluation. Section 7 concludes the paper. Monte-Carlo Tree Search (MCTS) 2 Background Monte-Carlo tree search (MCTS) is a family of algo- In this section we give a brief introduction to HTN planning rithms (Browne et al. 2012) that have been very success- and Monte-Carlo Tree Search search. ful in Games and probabilistic planning on MDPs settings. Many variants of MCTS, or algorithms containing elements HTN Planning thereof, have been proposed. Generally, MCTS considers a search tree. This is initialized with a root node, then itera- In Hierarchical Task Network (HTN) planning (see e.g. tively expanded, performing four operations at each step: Ghallab, Nau, and Traverso, 2004, Chapter 11, or Bercher, Alford, and Holler,¨ 2019), there are two types of tasks, ab- • Selection: Traverse the tree from the root to a leaf, select- stract tasks and primitive tasks (also called actions). Like in ing actions according to a strategy that balances explo- classical planning, actions are directly applicable to the en- ration (visit nodes that have been explored less often) and vironment and cause state transition. In its most basic defi- exploitation (visit more promising nodes more often). The nition, the environment is described using a finite set of state UCT algorithm defines a popular selection strategy tak- features, but there are also systems supporting numbers (see ing inspiration from multi-armed bandit problems (Kocsis e.g. Nau et al., 2003) and time (e.g. Bit-Monnot, Smith, and and Szepesvari´ 2006). Do, 2016). Tasks are maintained in task networks, which are • Expansion: Generate the successors of the selected leaf partially ordered multisets of tasks. node. Abstract tasks are not directly executable, but are decom- posed into other tasks (that may be primitive or abstract) • Evaluation: Estimate the solution cost (reward/quality) of using decomposition methods. A method definition includes a newly generated node. This is done by roll-outs, sim- the task it is applicable to and a task network that defines ulations following a random action policy, until a termi- its subtasks. Usually, there is more than one method for a nal state is reached. (But a link to heuristic search can be given abstract task, resulting in the combinatorial problem made here, see below.) of choosing the right method to apply to a given task.