AI Magazine Volume 25 Number 2 (2004) (© AAAI) Articles Incremental Heuristic Search in AI
Sven Koenig, Maxim Likhachev, Yaxin Liu, and David Furcy
■ Incremental search reuses information from previ- changes of their models of the world, for ex- ous searches to find solutions to a series of similar ample, because the actual situation turns out search problems potentially faster than is possible to be slightly different from the one initially by solving each search problem from scratch. This assumed or because the situation changes over is important because many AI systems have to adapt their plans continuously to changes in (their time. In these cases, the original plan might no knowledge of) the world. In this article, we give an longer apply or might no longer be good, and overview of incremental search, focusing on LIFE- one thus needs to replan for the new situation LONG PLANNING A*, and outline some of its possible (desJardins et al. 1999). Similarly, one needs to applications in AI. solve a series of similar search problems if one wants to perform a series of what-if analyses or if the costs of planning operators, their precon- ditions, or their effects change over time be- cause they are learned or refined. Overview In these situations, most search algorithms t is often important that searches be fast. AI replan from scratch, that is, solve the new has developed several ways of speeding up search problem independently of the old ones. Isearches by trading off the search time and However, this approach can be inefficient in the cost of the resulting path, which includes large domains with frequent changes and, using inadmissible heuristics (Pohl 1973, 1970) thus, limit the responsiveness of AI systems or and search with limited look ahead (Korf 1990; the number of what-if analyses that they can Ishida and Korf 1991; Koenig 2001), which is perform. Fortunately, the changes to the also called real-time or agent-centered search. search problems are usually small. A robot, for In this article, we discuss a different way of example, might have to replan when it detects speeding up searches, namely, incremental a previously unknown obstacle, a traffic rout- search. Incremental search is a search technique ing system might have to replan when it learns for continual planning (or, synonymously, re- about a new traffic jam, and a decision support planning, plan reuse, and lifelong planning) system for marine oil spill containment might that reuses information from previous searches have to replan when the wind direction to find solutions to a series of similar search changes. This property suggests that a com- problems potentially faster than is possible by plete recomputation of the best plan for the solving each search problem from scratch. Dif- new search problem is unnecessary because ferent from other ways of speeding up search- some of the previous search results can be es, it can guarantee to find the shortest paths. reused, which is what incremental search does. Notice that the terminology is unfortunately Incremental search solves dynamic shortest- somewhat problematic because the term incre- path problems, where shortest paths have to be mental search in computer science also refers to found repeatedly as the topology of a graph or both online search and search with limited its edge costs change (Ramalingam and Reps look ahead (Pemberton and Korf 1994). 1996b). The idea of incremental search is old. Most of the research on search has studied For example, an overview article about short- how to solve one-time search problems. How- est-path algorithms from 1984 already cites ever, many AI systems have to adapt their several incremental search algorithms, includ- plans continuously to changes in the world or ing several ones published in the late 1960s
Copyright © 2004, American Association for Artificial Intelligence. All rights reserved. 0738-4602-2004 / $2.00 SUMMER 2004 99 Articles
(Deo and Pang 1984). Since then, additional Uninformed Incremental Search incremental search algorithms have been sug- gested in the algorithms literature (Ausiello et We now discuss one particular way of solving al. 1991; Even and Gazit 1985; Even and fully dynamic shortest-path problems. As an Shiloach 1981; Feuerstein and Marchetti-Spac- example, we use route planning in known eight-connected gridworlds with cells whose camela 1993; Franciosa, Frigioni, and Giaccio traversability changes over time. They are ei- 2001; Frigioni, Marchetti-Spaccamela, and ther traversable (with cost one) or untraversa- Nanni 1996; Goto and Sangiovanni-Vincentel- ble. The route-planning problem is to repeatedly li 1978; Italiano 1988; Klein and Subramanian find a shortest path between two given cells of 1993; Lin and Chang 1990; Rohnert 1985; Spi- the gridworld, knowing both what the topolo- ra and Pan 1975) and, to a much lesser degree, gy of the gridworld is and which cells are cur- the AI literature (Al-Ansari 2001; Edelkamp rently traversable. It can be solved with con- 1998). They differ in their assumptions, for ex- ventional search, such as breadth-first search, ample, whether they solve single-source or all- by finding a shortest path every time some pairs shortest-path problems, which perfor- edge costs change. Conventional search typi- mance measure they use, when they update cally does not reuse information from previous the shortest paths, which kinds of graph searches. The following example, however, il- topologies and edge costs they apply to, and lustrates the potential advantage of reusing in- how the graph topology and edge costs are al- formation from previous searches. lowed to change over time (Frigioni, Marchetti- Consider the gridworlds shown in figure 1. Spaccamela, and Nanni 1998). If arbitrary se- The original gridworld is shown in figure 1a, quences of edge insertions, deletions, or weight and the changed gridworld is shown in figure changes are allowed, then the dynamic short- 1b. Only two blockages have moved, resulting est-path problems are called fully dynamic in four cells with changed blockage status. The shortest-path problems (Frigioni, Marchetti-Spac- figure shows the shortest paths in both cases under the assumption that every move has cost camela, and Nanni 2000). one. The shortest path changed because two The idea of incremental search has also been cells on the original shortest path became un- pursued in AI for problems other than path traversable. finding. For example, a variety of algorithms The length of a shortest path from the start have been developed for solving constraint-sat- cell to a cell is called its start distance. Once the isfaction problems (Dechter and Dechter 1988; start distances of all cells are known, one can eas- Verfaillie and Schiex 1994) or constraint logic ily trace back a shortest path from the start cell programming problems (Miguel and Shen to the goal cell by always greedily decreasing the 1999) where the constraints change over time. start distance, starting at the goal cell. The start In this article, however, we study incremental distances are shown in each traversable cell of search only in the context of dynamic shortest- the original and changed gridworlds (figures 1a path problems, where it has not been studied and 1b). Those cells whose start distances in the extensively in AI. changed gridworld (figure 1b) are different from We believe that four achievements are neces- the corresponding cells in the original grid- sary to make incremental search more popular world; they are shaded gray in figure 1b. Even in AI. First, one needs to devise more powerful though large parts of the shortest path have incremental search algorithms than those that changed, less than a quarter of the start distances currently exist. Second, one needs to study have changed. Thus, in such cases, there is a po- their properties more extensively, both analyt- tential advantage to recalculating only those ically and experimentally, to understand their start distances that have changed, which is an ar- gument for caching the start distances rather strengths and limitations better. Third, one than the shortest path itself. needs to demonstrate that they apply to AI ap- Caching the start distances is basically what plications and compare them to other search the incremental search algorithm DYNAMIC- algorithms for these applications to demon- SWSF-FP (Ramalingam and Reps 1996a) does.1 strate that they indeed have advantages over DYNAMICSWSF-FP was originally developed in them. Finally, the AI community needs to be the context of parsing theory and theoretical made more aware of incremental search. This computer science. It uses a clever way of iden- article addresses the last issue so that more re- tifying the start distances that have not searchers can address the first three. It describes changed and recalculates only the ones that one particular incremental search algorithm have changed. Consequently, it performs best and its potential applications and then discuss- in situations where only a small number of es its potential advantages and limitations. start distances change.
100 AI MAGAZINE Articles
A. Original Eight-Connected Gridworld
Sstart 1 2 3 4 5 6 7 8 9
1 192 3 4 5 6 7 8 2 292 3 4 5 6 7 8
3 4 4135 6 7 12 12 12 5 5137 11 11 12 6 6136 7 8 10 11 12
8 9 10 11 12 13 12 1110 9 9 11 13
12 1110 10 10 12 12 Sgoal
B. Changed Eight-Connected Gridworld
Sstart 1 2 3 4 5 6 7 8 9
1 192 3 4 5 6 7 8 2 292 3 4 5 6 7 8
3 9
4 4105 6 7 12 11 10 5 51111 11 11 6 6 6 7 8 10 11 12 12
7 8 9 10 11 12 13
9 8138 8 9 11
9 9129 9 9 Sgoal
Figure 1. Simple Gridworld.
Consider the gridworld in figure 2 to under- ue of the neighboring cell plus the cost of get- stand how it operates. The start cell is A3. As- ting from the neighboring cell to the cell in sume that one is given the values in the left question, which is indeed the case. For exam- gridworld, and it is claimed that they are equal ple, the value of cell B1 should be the mini- to the correct start distances. There are at least mum of the values of cells A0, A1, A2, and C1 two different approaches to verify this proper- plus one. Thus, the values are indeed equal to ty. One approach is to perform a complete the correct start distances. Both approaches search to find the start distances and compare need about the same run time to confirm this them to the given values. Another approach is property. Now assume that cell D1 becomes to check the definition of the start distances, untraversable, as shown in the right gridworld namely, that the value of the start cell is zero, of figure 2, and thus, the costs of all edges into and the value of every other cell is equal to the the cell become infinity, and it is claimed that minimum over all neighboring cells of the val- the values in the cells remain equal to the cor-
SUMMER 2004 101 Articles
Informed Incremental Search One way of speeding up searches is to use in- 0123 0123 cremental search. Another way of speeding up searches is to use heuristic search. The question A 3 2 1 0 A 3 2 1 0 arises whether incremental and heuristic search can be combined. Many of the start dis- B 2 1 B 2 1 tances that have changed in the example in fig- ure 1 are irrelevant for finding a shortest path C 4 3 2 2 C 4 3 2 2 from the start cell to the goal cell and thus do not need to be recalculated. Examples are the D 4 3 D 4 3 cells in the lower left corner of the gridworld. Thus, there is a potential advantage to using E 5 4 E 5 4 heuristics to avoid having to recalculate irrele- vant start distances. F 6 6 55 F 6 6 55 To summarize, there are two different ways to decrease the search effort of breadth-first search for finding the start distances for the changed gridworld (figure 1b): First, incremental search algorithms, such as Figure 2. An Example. DYNAMICSWSF-FP (Ramalingam and Reps 1996a), find shortest paths for series of similar search problems potentially faster than is pos- sible by solving each search problem from scratch (complete search) because they do not recompute those start distances that have not rect start distances. Again, there are at least two changed. different approaches to verify this property. Second, heuristic search algorithms, such as A* One approach is again to perform a complete (Nilsson 1971), use heuristic knowledge in the search to find the start distances and compare form of approximations of the goal distances them to the given values. The second approach to focus the search and find shortest paths for is again to check that the value of the start cell search problems faster than uninformed search is zero and that the value of every other cell is because they do not compute those start dis- equal to the minimum over all neighboring tances that are irrelevant for finding a shortest cells of the value of the neighboring cell plus path from the start cell to the goal cell. the cost of getting from the neighboring cell to Consequently, we developed a search algo- the cell in question. Because the values remain rithm that combines incremental and heuristic unchanged, each cell continues to have this search, namely, LPA* (LIFELONG PLANNING A*) (Koenig and Likhachev 2002b). We call it life- property unless its neighbors have changed. long planning in analogy to lifelong learning Thus, one needs to check only whether the (Thrun 1998) because it reuses information cells close to changes in the gridworld contin- from previous searches. LPA* repeatedly finds ue to have this property, that is, cells C1 and shortest paths from a given start vertex to a giv- E1. It turns out that cell C1 continues to have en goal vertex on arbitrary known finite graphs this property, but cell E1 does not. Thus, not all (not just gridworlds) whose edge costs increase values are equal to the correct start distances or decrease over time (which can also be used (which does not mean, of course, that all cells to model edges or vertices that are added or but E1 have correct start distances). The second deleted). The pseudocode of the simplest ver- approach now needs less run time than the sion of LPA*reduces to a version of A* that first one. Furthermore, the second approach breaks ties among vertices with the same f-val- provides a starting point for replanning, name- ue in favor of smaller g-values when used to search from scratch and to DYNAMICSWSF-FP ly, cell E1, because one needs to work on the when used with uninformed heuristics. In fact, cells that do not have this property, moving it differs from DYNAMICSWSF-FP only in the cal- outward from the starting point. The principle culation of the priorities for the vertices in the behind the second approach is the main idea priority queue (line {01} in the pseudocode of behind DYNAMICSWSF-FP. It shares this idea figure 3). It is unoptimized and needs consis- with other incremental search approaches, in- tent heuristics (Pearl 1985). We have also de- cluding some that apply to constraint satisfac- veloped more sophisticated versions of LPA* tion (Dechter and Dechter 1988). that are optimized (for example, recalculate the
102 AI MAGAZINE Articles various values much more efficiently than the simple version), can work with inadmissible S denotes the finite set of vertices of the graph. succ(s) ⊆ S denotes the set of successors heuristics, and can break ties among vertices of vertex s ∈ S. Similarly, pred(s) ⊆ S denotes the set of predecessors of vertex s ∈ S. with the same f-value in favor of larger g-val- 0
SUMMER 2004 103 Articles
1a 1b
2a 2b
old search tree new search tree
Figure 4. Old and New Search Trees.
drawn with uniform probability for each grid- to prefer LPA* over A*. For example, if this num- world. All edge costs are either one or two with ber is two, then LPA* solves one planning prob- uniform probability. We then change each lem and two replanning problems together gridworld 500 times in a row by selecting a giv- faster than A*. Additional experiments are re- en number of edges and reassigning them ran- ported in Koenig, Likhachev, and Furcy (2004). dom costs. We report the probability that the cost of the shortest path changes to ensure that Experiment 1 the edge cost changes indeed change the short- In the first experiment, the size of the grid- est path sufficiently often. A probability of 33.9 worlds is 101 by 101. We change the number of percent, for example, means that the cost of edges that get assigned random costs before the shortest path changes on average after 2.96 each planning episode. Table 1 shows our ex- planning episodes. We use the Manhattan dis- perimental results. The smaller the number of tances as heuristics for the cost of a shortest edges that get assigned random costs, the less path between two cells, that is, the sum of the the search space changes and the larger the ad- difference of their x- and y-coordinates. For vantage of LPA* in our experiments. The aver- each experiment (tables 1, 2, and 3), we report age run time of the first planning episode of the run time (in milliseconds) averaged over all LPA* tends to be larger than that of A*, but the first planning episodes (#1) and over all plan- average run time of the following planning ning episodes (#2), run on a PENTIUM 1.7-mega- episodes tends to be so much smaller (if the hertz PC. We also report the speedup of LPA* number of edges that get assigned random over A* in the long run (#3), that is, the ratio of costs is sufficiently small) that the number of the run times of A* and LPA* averaged over all replanning episodes that are necessary for one planning episodes. The first search of LPA* tends to prefer LPA* over A* is one. to be slower than that of A* because it expands more states and needs more time for each state Experiment 2 expansion. During the subsequent searches, In the second experiment, the number of edges however, LPA* often expands fewer states than that get assigned random costs before each A* and is thus faster than A*. We therefore also planning episode is 0.6 percent. We change the report the replanning episode after which the size of the square gridworlds. Table 2 shows our average total run time of LPA* is smaller than experimental results. The smaller the grid- that of A* (#4), in other words, the number of worlds, the larger the advantage of LPA* in our replanning episodes that are necessary for one experiments, although we were not able to pre-
104 AI MAGAZINE Articles
edge path cost A* LPA* cost changes changes #1 and #2 #1 #2 #3 #4 10.370؋ 1 0.029 0.386 0.299 3.0% 0.2% 5.033؋ 1 0.067 0.419 0.336 7.9% 0.4% 3.344؋ 1 0.108 0.453 0.362 13.0% 0.6% 2.603؋ 1 0.156 0.499 0.406 17.6% 0.8% 2.126؋ 1 0.174 0.434 0.370 20.5% 1.0% 1.858؋ 1 0.222 0.476 0.413 24.6% 1.2% 1.657؋ 1 0.282 0.539 0.468 28.7% 1.4% 1.507؋ 1 0.332 0.563 0.500 32.6% 1.6% 1.384؋ 1 0.328 0.497 0.455 32.1% 1.8% 1.249؋ 1 0.315 0.433 0.394 33.8% 2.0%
Table 1. Experiment 1.
path cost A* LPA* maze size changes #1 and #2 #1 #2 #3 #4 5.032؋ 1 0.015 0.098 0.077 7.3% 51 51 3.987؋ 1 0.050 0.258 0.201 10.7% 76 76 3.315؋ 1 0.104 0.437 0.345 13.0% 101 101 3.128؋ 1 0.220 0.789 0.690 16.2% 126 126 2.900؋ 1 0.322 1.013 0.933 17.7% 151 151 2.753؋ 1 0.564 1.608 1.553 21.5% 176 176 2.696؋ 1 0.682 1.898 1.840 22.9% 201 201
Table 2. Experiment 2.
dict this effect. This insight is important because longer decreases with the size of the grid- it implies that LPA* does not scale well in our worlds. The closer the edge cost changes are to gridworlds (although part of this effect could be the goal cell, the larger the advantage of LPA*is the result of the fact that more edges get as- in our experiments, as predicted earlier. This signed random costs as the size of the grid- insight is important because it suggests using worlds increases; this time is included in the run LPA* when most of the edge cost changes are time averaged over all planning episodes). We, close to the goal cell. We use this property therefore, devised the third experiment. when we apply LPA* to mobile robotics and control (see later discussion). Experiment 3 Although these experiments give us some in- In the third experiment, the number of edges sight into the behavior of LPA*, we need to im- that get assigned random costs before each prove our understanding of when to prefer in- planning episode is again 0.6 percent. We cremental search over alternative search change both what the size of the square grid- algorithms and which incremental search algo- worlds is and how close the edges that get as- rithm to use. The main criteria for choosing a signed random costs are to the goal cell. Eighty search algorithm are its memory consumption percent of these edges leave cells that are close and its run time. to the goal cell. Table 3 shows our experimen- With respect to memory, incremental search tal results. Now, the advantage of LPA*no needs memory for information from past
SUMMER 2004 105 Articles
80% of edge cost changes are Յ 25 cells away from the goal path cost A* LPA* maze size changes #1 and #2 #1 #2 #3 #4 51 51 13.5% 0.084 0.115 0.014 6.165 1 76 76 23.9% 0.189 0.245 0.028 6.661 1 101 101 33.4% 0.295 0.375 0.048 6.184 1 126 126 42.5% 0.696 0.812 0.084 8.297 1 151 151 48.5% 0.886 0.964 0.114 7.808 1 176 176 55.7% 1.353 1.450 0.156 8.683 1 201 201 59.6% 1.676 1.733 0.202 8.305 1
80% of edge cost changes are Յ 50 cells away from the goal path cost A* LPA* maze size changes #1 and #2 #1 #2 #3 #4 51 51 8.6% 0.086 0.115 0.017 5.138 1 76 76 15.7% 0.190 0.247 0.039 4.822 1 101 101 23.2% 0.304 0.378 0.072 4.235 1 126 126 31.3% 0.702 0.812 0.130 5.398 1 151 151 36.2% 0.896 0.959 0.173 5.166 1 176 176 44.0% 1.372 1.458 0.242 5.664 1 201 201 48.3% 1.689 1.742 0.313 5.398 1
80% of edge cost changes are Յ 75 cells away from the goal path cost A* LPA* maze size changes #1 and #2 #1 #2 #3 #4 76 76 12.1% 0.196 0.250 0.047 4.206 1 101 101 17.5% 0.306 0.391 0.088 3.499 1 126 126 26.0% 0.703 0.818 0.175 4.012 1 151 151 28.8% 0.893 0.972 0.225 3.978 1 176 176 36.8% 1.370 1.438 0.319 4.301 1 201 201 40.1% 1.728 1.790 0.408 4.236 1
Table 3. Experiment 3.
searches. LPA*, for example, needs to remember struction set of the processor, the optimiza- the previous search tree. This property of LPA* tions performed by the compiler, and the data tends not to be a problem for gridworlds, but structures used for the priority queues; it is the search trees of other search problems are thus hard to characterize. For example, when often so large that they do not completely fit the number of edges that get assigned random into memory. In this case, it might be possible costs was 0.2 percent in experiment 1, the to combine incremental search with memory- number of heap percolates of A* was 8213.04, limited search algorithms such as RBFS (Korf but the number of heap percolates of LPA*was 1993) or SMA* (Russell 1992), but this work is only 297.30. This result makes it difficult to de- for the future. termine whether there is a benefit to incremen- With respect to run time, our experiments tal search and, if so, how to quantify it. For ex- have demonstrated that LPA* expands fewer ample, one can decrease the speedup of LPA* vertices than A*, for example, when only a few over A* by using buckets to implement the pri- edge costs change, and these edge costs are ority queues rather than heaps, even though close to the goal vertex. These situations need this approach is more complicated. We imple- to be characterized better. LPA*also needs more mented A* with buckets and a simple FIFO tie- time for each vertex expansion than A*. This breaking strategy within buckets (which speeds time disadvantage depends on low-level imple- it up by more than a factor of two) but left the mentation and machine details, such as the in- implementation of LPA* unchanged. In this
106 AI MAGAZINE Articles case, LPA* needed more than one replanning Consequently, the calculation of the heuristics episode in experiment 1 to outperform A* if the comprises about 80 percent of its run time number of edges that got reassigned random (Bonet and Geffner 2001). costs before each planning episode was less HSP 2.0 thus repeatedly solves relaxed search than 1.0 percent and did not outperform A*at problems as it calculates the heuristics. The re- all if the number of edges that got reassigned laxed search problems that it solves to find the random costs before each planning episode heuristics of two states are similar if the two was 1.0 percent or more. Therefore, we are only states are similar, and two states whose heuristics willing to conclude from our experiments that it calculates consecutively are often similar be- incremental heuristic search is a promising cause they are both children of the same parent technology that needs to be investigated fur- in the A* search tree. Thus, incremental search ther. In general, the trade-off between the can solve a relaxed search problem by reusing in- number of vertex expansions and the time formation from the calculation of the previous needed for each vertex expansion might relaxed search problem. Our PINCH (prioritized, benefit incremental search algorithms that are incremental heuristics calculation) algorithm less sophisticated than LPA* and, thus, expand (Liu, Koenig, and Furcy 2002), for example, is more vertices but with less time for each vertex based on DYNAMICSWSF-FP and speeds up the expansion. One might also be able to develop run time of HSP 2.0 by as much as 80 percent in incremental search algorithms that apply only several domains, and in general, the amount of to special cases of graphs (such as gridworlds) savings grows with the size of the domains, al- and, thus, are faster but not as versatile as LPA*. lowing HSP 2.0 to solve larger search problems with the same limit on its run time and without changing its heuristics or overall operation. Applications Continual planning: So far, we have de- Our gridworld experiments suggest that incre- scribed how incremental search can solve sin- mental search can be beneficial for route plan- gle symbolic planning problems. However, ning in traffic or computer networks, where planning researchers realized a long time ago the congestion and, thus, the optimal routes that one often does not need to solve just one change over time. These applications are simi- symbolic planning problem but, rather, a series lar to our gridworld experiments. However, a of similar symbolic planning problems. Exam- variety of areas in AI could also potentially ples of practical significance include the benefit from incremental search. In the follow- aeromedical evacuation of injured people in ing subsections, we discuss some of the possi- crisis situations (Kott, Saks, and Mercer 1999) ble applications, many (but not all) of which and air campaign planning (Myers 1999). Re- involve gridworlds. We also discuss the current planning is necessary in these cases, for exam- state of the art, including opportunities for fu- ple, when a landing strip at an airfield becomes ture research on incremental search. unusable. Planning researchers have therefore studied replanning and plan reuse. Replanning Symbolic Planning (HSP) attempts to retain as many plan steps of the Symbolic planning is the most obvious appli- previous plan as possible. Plan reuse does not cation of incremental search. Interestingly, in- have this requirement. (We do not make this cremental search can be used not only to solve distinction and, thus, use the term replanning a series of similar symbolic (STRIPS-style) plan- throughout the text.) Examples include case- ning problems but also single symbolic plan- based planning, planning by analogy, plan ning problems. adaptation, transformational planning, plan- One-time planning: Heuristic search-based ning by solution replay, repair-based planning, planners solve symbolic planning problems. and learning search-control knowledge. These They were introduced by McDermott (1996) search algorithms have been used as part of and Bonet, Loerincs, and Geffner (1997) and systems such as CHEF (Hammond 1990), have become very popular. In its default GORDIUS (Simmons 1988), LS-ADJUST-PLAN configuration, HSP 2.0 (Bonet and Geffner (Gerevini and Serina 2000), MRL (Koehler 2000), for example, uses weighted A* searches 1994), NOLIMIT (Veloso 1994), PLEXUS (Alterman with inadmissible heuristics to perform for- 1988), PRIAR (Kambhampati and Hendler 1992), ward searches in the space of world states to and SPA (Hanks and Weld 1995). find a path from the start state to a goal state. HSP 2.0 with weight one and consistent However, the calculation of the heuristics is heuristics finds plans with minimal plan-exe- time consuming because HSP 2.0 calculates the cution cost. If HSP 2.0 solves a series of similar heuristic of each state that it encounters during symbolic planning problems, then it can use the search by solving a relaxed search problem. LPA*instead of A* to replan faster, resulting in
SUMMER 2004 107 Articles
the SHERPA (speedy heuristic search-based) re- Goal-directed navigation in unknown ter- planner (Koenig, Furcy, and Bauer 2002) for rain: Planning with the free-space assumption consistent heuristics. A difference between is a popular solution to the goal-directed navi- SHERPA and the other replanners described ear- gation problem where a mobile robot has to lier is that SHERPA not only remembers the pre- move in initially unknown terrain to given goal vious plans but also remembers the previous coordinates. For example, the characters in plan-construction processes. Thus, it has more popular combat games such as TOTAL ANNIHILA- information available for replanning than even TION, AGE OF EMPIRES, and WARCRAFT have to move PRIAR, which stores plans together with expla- autonomously in initially unknown terrain to nations of their correctness, or NOLIMIT, which user-specified coordinates. Planning with the stores plans together with substantial descrip- free-space assumption always plans a shortest tions of the decisions that resulted in the solu- path from its current coordinates to the goal co- tion. Another difference between SHERPA and ordinates under the assumption that the un- the other replanners is that its plan-execution known terrain is traversable. When it observes cost is as good as the plan-execution cost obstacles as it follows this path, it enters them achieved by search from scratch. Thus, incre- into its map and then repeats the procedure, mental search can be used for plan reuse if the until it eventually reaches the goal coordinates, plan-execution cost of the resulting plan is im- or all paths to them are untraversable. portant, but its similarity to the previous plans To implement this navigation strategy, the is not. robot needs to replan shortest paths whenever Inadmissible heuristics allow HSP 2.0 to solve it detects that its current path is untraversable. search problems in large state spaces by trading Several ways of speeding up the searches have off run time and the plan-execution cost of the been proposed in the literature (Barbehenn resulting plan. SHERPA uses LPA* with consistent and Hutchinson 1995; Ersson and Hu 2001; heuristics. Although we have extended LPA*to Huiming et al. 2001; Podsedkowski et al. 2001; use inadmissible heuristics and still guarantee Tao et al. 1997; Trovato 1990). FOCUSSED DYNAM- that it expands every vertex at most twice, it IC A*(D*) (Stentz 1995) is probably the most turns out to be difficult to make incremental popular solution and has been extensively used search more efficient than search from scratch on real robots, such as outdoor high-mobility with the same inadmissible heuristics, although multipurpose wheeled vehicles (HMMWVs) we have had success in special cases. This diffi- (Hebert, McLachlan, and Chang 1999; culty can be explained as follows: The larger the Matthies et al. 2000; Stentz and Hebert 1995; heuristics are, the narrower the A* search tree Thayer et al. 2000), and studied theoretically and, thus, the more efficient A* is. However, the (Koenig, Tovey, and Smirnov 2003). We believe narrower the A* search tree, the more likely it is that D* is the first truly incremental heuristic that the overlap between the old and new A* search algorithm. It resulted in a new applica- search trees is small and, thus, the less efficient tion for incremental search and a major ad- LPA* is. For example, situations 2a and 2b in fig- vance in robotics. LPA* and D* share similarities. ure 4 correspond to situations 1a and 1b respec- For example, we can combine LPA*with ideas tively, except that the old and new search trees from D* to apply it to moving robots, resulting are narrower and, thus, overlap less. in D* LITE (Koenig and Likhachev 2002a). D* LITE and D* implement the same navigation strate- Mobile Robotics and Games gy and are about equally fast, but D* LITE is al- (Path Planning) gorithmically simpler and, thus, easy to under- Mobile robots often have to replan quickly as stand, analyze, and extend. Both search the world or their knowledge of it changes. Ex- algorithms search from the goal coordinates to- amples include both physical robots and com- ward the current coordinates of the robot. Be- puter-controlled robots (or, more generally, cause the robot usually observes obstacles close computer-controlled characters) in computer to its current coordinates, the changes are close games. Efficient replanning is especially impor- to the goal of the search, which makes incre- tant for computer games because they often mental search efficient, as predicted earlier. simulate a large number of characters, and Mapping: Greedy mapping is a popular solu- their other software components, such as the tion to the problem of mapping unknown ter- graphics generation, already place a high de- rain (Koenig, Tovey, and Haliburton 2001; mand on the processor. In the following para- Romero, Morales, and Sucar 2001; Thrun et al. graphs, we discuss two cases where the knowl- 1998). The robot always plans a shortest path edge of a robot changes because its sensors from its current coordinates to a closest patch acquire more information about the initially of terrain with unknown traversability until unknown terrain as it moves around. the terrain is mapped.
108 AI MAGAZINE Articles
To implement this navigation strategy, the values that reinforcement learning algorithms robot needs to replan shortest paths whenever need to update. it observes new obstacles. Both D* and D* LITE can be used unchanged to implement greedy Control (PARTI-GAME Algorithm) mapping, although their advantage over A* is State spaces of control problems are often con- much smaller for mapping than for goal-direct- tinuous and sometimes high dimensional. The ed navigation in unknown terrain (Likhachev PARTI-GAME algorithm (Moore and Atkeson 1995) and Koenig 2002). finds control policies that move an agent in such domains from given start coordinates to Machine Learning given goal coordinates. It is popular because it (Reinforcement Learning) is simple and efficient and applies to a broad Reinforcement learning is learning from rewards range of control problems such as path plan- and penalties that can be delayed (Kaelbling, ning for mobile robots and robot arms (Al- Littman, and Moore 1996; Sutton and Barto Ansari and Williams 1999; Araujo and de 1998). Reinforcement learning algorithms, Almeida 1998, 1997; Kollmann et al. 1997). To such as Q-LEARNING (Watkins and Dayan 1992) solve these problems, one can first discretize the or online versions of value iteration (Barto, domains and then use conventional search to Bradtke, and Singh 1995), often use dynamic find plans that move the agent from its current programming to update state or state-action coordinates to the goal coordinates. However, values and are then similar to real-time search uniform discretizations can prevent one from (Ishida and Korf 1991; Koenig 2001; Korf finding a plan if they are too coarse grained (for 1990). The order of the value updates deter- example, because the resolution prevents one mines how fast they can propagate informa- from noticing small gaps between obstacles) tion through the state space, which has a sub- and result in large state spaces that cannot be stantial effect on their efficiency. The DYNA-Q searched efficiently if they are too fine grained. environment (Sutton and Barto 1998) has been The PARTI-GAME algorithm solves this dilemma used by various researchers to study ways of by starting with a coarse discretization and making reinforcement learning more efficient refines it during execution only when and by ordering their value updates. PRIORITIZED where it is needed (for example, around obsta- SWEEPING (Moore and Atkeson 1993) and QUEUE- cles), resulting in a nonuniform discretization. DYNA (Peng and Williams 1993) are, for exam- To implement the PARTI-GAME algorithm, the ple, reinforcement learning algorithms that re- agent needs to find a plan with minimal worst- sulted from this research. They concentrate the case plan-execution cost whenever it has value updates on those states whose values refined its model of the domain. Several ways of they change most. speeding up the searches with incremental Incremental search can order the value up- search have been proposed in the literature. Al- dates in an even more systematic way and uses Ansari (2001), for example, proposed an unin- concepts related to concepts from reinforce- formed incremental search algorithm that re- ment learning. For example, LPA* performs dy- stores the priority queue of a minimax search namic programming and implicitly uses the algorithm (similar to Dijkstra’s algorithm) to Bellman (1957) equations. It is currently un- the vertices it had immediately before the clear how incremental search can be applied to changed edge costs made it behave differently, minimizing the expected (discounted) plan-ex- and we proposed a combination of MINIMAX LPA* ecution cost in nondeterministic domains, the and D* LITE (Likhachev and Koenig 2003). typical objective of reinforcement learning. However, we have extended LPA* from mini- Conclusions mizing the plan-execution cost in determinis- tic domains to minimizing the worst-case plan- Incremental search reuses information from execution cost in nondeterministic domains, previous searches to find solutions to a series of resulting in MINIMAX LPA* (Likhachev and similar search problems potentially faster than Koenig 2003), which we believe to be the first is possible by solving each search problem incremental heuristic minimax search algo- from scratch. Although incremental search is rithm. It applies to goal-directed reinforcement currently not used much in AI, this article learning for minimizing the worst-case plan- demonstrated that there are AI applications execution cost. Although this is the only appli- that might benefit from incremental search. It cation of incremental search to reinforcement also demonstrated that we need to improve our learning that has been identified to date, it sug- understanding of incremental search, includ- gests that ideas from incremental search can ing when to prefer incremental search over al- potentially be used to reduce the number of ternative search algorithms and which incre-
SUMMER 2004 109 Articles
mental search algorithm to use. For example, College of Computer Science, Northeastern Universi- the search spaces of incremental search meth- ty. ods (for example, for computer games) are of- Al-Ansari, M., and Williams, R. 1999. Robust, Effi- ten small, and thus, their scaling properties are cient, Globally Optimized Reinforcement Learning less important than implementation and ma- with the PARTI-GAME Algorithm. In Advances in Neural chine details. At the same time, it is difficult to Information Processing Systems, Volume 11, 961–967. compare them using proxies, such as the num- Cambridge, Mass.: MIT Press. ber of vertex expansions, if they perform very Alterman, R. 1988. Adaptive Planning. Cognitive Sci- different basic operations. We therefore suggest ence 12(3): 393–421. that AI researchers study incremental search in Araujo, R., and de Almeida, A. 1998. Map Building more depth in the coming years to understand Using Fuzzy Art and Learning to Navigate a Mobile Robot on an Unknown World. In Proceedings of the it better; develop new algorithms; evaluate its International Conference on Robotics and Automa- potential for real-world applications; and in tion, Volume 3, 2554–2559. Washington, D.C.: IEEE general, determine whether incremental search Computer Society. is an important technique in the tool box of AI. Araujo, R., and de Almeida, A. 1997. Sensor-Based Acknowledgments Learning of Environment Model and Path Planning with a Nomad 200 Mobile Robot. In Proceedings of This research was performed when the authors the International Conference on Intelligent Robots were at Georgia Institute of Technology. and Systems, Volume 2, 539–544. Washington, D.C.: Thanks to Peter Yap, Rob Holte, and Jonathan IEEE Computer Society. Schaeffer for interesting insights into the be- Ausiello, G.; Italiano, G.; Marchetti-Spaccamela, A.; havior of LPA*. Thanks also to Anthony Stentz and Nanni, U. 1991. Incremental Algorithms for and Craig Tovey for helpful discussions and to Minimal Length Paths. Journal of Algorithms 12(4): Colin Bauer for implementing some of our 615–638. ideas. The Intelligent Decision-Making Group Barbehenn, M., and Hutchinson, S. 1995. Efficient is partly supported by National Science Foun- Search and Hierarchical Motion Planning by Dynam- dation awards to Sven Koenig under contracts ically Maintaining Single-Source Shortest Paths Trees. IIS-9984827, IIS-0098807, and ITR/AP- IEEE Transactions on Robotics and Automation 11(2): 0113881. Yaxin Liu was supported by an IBM 198–214. Ph.D. fellowship. The views and conclusions Barto, A.; Bradtke, S.; and Singh, S. 1995. Learning to contained in this article are those of the au- Act Using Real-Time Dynamic Programming. Artifi- cial Intelligence 73(1): 81–138. thors and should not be interpreted as repre- senting the official policies, either expressed or Bellman, R. 1957. Dynamic Programming. Princeton, N.J.: Princeton University Press. implied, of the sponsoring organizations, agen- cies, companies, or the U.S. government. Bonet, B., and Geffner, H. 2001. Planning as Heuris- tic Search. Artificial Intelligence 129(1): 5–33. Notes Bonet, B., and Geffner, H. 2000. HEURISTIC SEARCH PLAN- NER 2.0. AI Magazine 22(3): 77–80. 1. DYNAMICSWSF-FP, as originally stated, searches from the goal vertex to all other vertices and, thus, Bonet, B.; Loerincs, G.; and Geffner, H. 1997. A Ro- maintains estimates of the goal distances rather than bust and Fast Action Selection Mechanism. In Pro- the start distances. We, however, use it to search from ceedings of the Fourteenth National Conference on the start vertex to all other vertices. Also, to calculate Artificial Intelligence, 714–719. Menlo Park, Calif.: a shortest path from the start vertex to the goal ver- American Association for Artficial Intelligence. tex, not all distances need to be known. To make DY- Dechter, R., and Dechter, A. 1988. Belief Mainte- NAMICSWSF-FP more efficient, we changed its termi- nance in Dynamic Constraint Networks. In Proceed- nation condition so that it stops immediately after it ings of the Seventh National Conference on Artificial has found a shortest path from the start vertex to the Intelligence, 37–42. Menlo Park, Calif.: American As- goal vertex. sociation for Artificial Intelligence. 2. To be more precise, it is not only important that the Deo, N., and Pang, C. 1984. Shortest-Path Algo- trees are similar, but most start distances of its nodes rithms: Taxonomy and Annotation. Networks have to be the same as well. Thus, it is insufficient, for 14:275–323. example, that the tree remains unchanged if most of desJardins, M.; Durfee, E.; Ortiz, C.; and Wolverton, the start distances of its nodes change. M. 1999. A Survey of Research in Distributed, Con- 3. The code of our implementations is available at tinual Planning. AI Magazine 20(4): 13–22. www-rcf.usc.edu/~skoenig/fastreplanning.html. Edelkamp, S. 1998. Updating Shortest Paths. In Pro- ceedings of the Thirteenth European Conference on Arti- ficial Intelligence, 655–659. Chichester, U.K.: Wiley. Ersson, T., and Hu, X. 2001. Path Planning and Nav- References igation of Mobile Robots in Unknown Environ- Al-Ansari, M. 2001. Efficient Reinforcement Learning ments. In Proceedings of the International Confer- in Continuous Environments. Ph.D. dissertation, ence on Intelligent Robots and Systems, 858–864.
110 AI MAGAZINE Articles
Washington, D.C.: IEEE Computer Society. ternational Joint Conferences on Artificial 35–44. Technical Report, UMCS-97-4-1, De- Even, S., and Gazit, H. 1985. Updating Dis- Intelligence. partment of Computer Science, University tances in Dynamic Graphs. Methods of Op- Italiano, G. 1988. Finding Paths and Delet- of Manchester. erations Research 49: 371–387. ing Edges in Directed Acyclic Graphs. Infor- Korf, R. 1993. Linear-Space Best-First Even, S., and Shiloach, Y. 1981. An Online mation Processing Letters 28(1): 5–11. Search. Artificial Intelligence 62(1): 41–78. Edge- Deletion Problem. Journal of the ACM Kaelbling, L.; Littman, M.; and Moore, A. Korf, R. 1990. Real-Time Heuristic Search. 28(1): 1–4. 1996. Reinforcement Learning: A Survey. Artificial Intelligence 42(2-3): 189–211. Feuerstein, E., and Marchetti-Spaccamela, Journal of Artificial Intelligence Research Kott, A.; Saks, V.; and Mercer, A. 1999. A A. 1993. Dynamic Algorithms for Shortest 4:237–285. New Technique Enables Dynamic Replan- Paths in Planar Graphs. Theoretical Comput- Kambhampati, S., and Hendler, J. 1992. A ning and Rescheduling of Aeromedical er Science 116(2): 359–371. Validation-Structure-Based Theory of Plan Evacuation. AI Magazine 20(1): 43–53. Franciosa, P.; Frigioni, D.; and Giaccio, R. Modification and Reuse. Artificial Intelli- Likhachev, M., and Koenig, S. 2003. Speed- 2001. Semidynamic Breadth-First Search in gence 55(2): 193–258. ing Up the PARTIGAME Algorithm. In Ad- Digraphs. Theoretical Computer Science Klein, P., and Subramanian, S. 1993. A Fully vances in Neural Information Processing Sys- 250(1–2): 201–217. Dynamic Approximation Scheme for All- tems 15, eds. S. Becker, S. Thrun, and K. Frigioni, D.; Marchetti-Spaccamela, A.; and Pairs Shortest Paths in Planar Graphs. In Al- Obermayer. Cambridge, Mass.: MIT Press. Nanni, U. 2000. Fully Dynamic Algorithms gorithms and Data Structures: Third Workshop, Likhachev, M., and Koenig, S. 2002. Incre- for Maintaining Shortest-Path Trees. Jour- WADS’93, 443–451. Lecture Notes in Com- mental Replanning for Mapping. In Pro- nal of Algorithms 34(2): 251–281. puter Science 709. New York: Springer-Verlag. ceedings of the International Conference Frigioni, D.; Marchetti-Spaccamela, A.; and Koehler, J. 1994. Flexible Plan Reuse in a on Intelligent Robots and Systems, Nanni, U. 1998. Semidynamic Algorithms Formal Framework. In Current Trends in AI 667–672. Washington, D.C.: IEEE Comput- for Maintaining Single-Source Shortest- Planning, eds. C. Bäckström and E. Sandewall, er Society. Path Trees. Algorithmica 22(3): 250–274. 171–184. Amsterdam, The Netherlands: Lin, C., and Chang, R. 1990. On the Dy- Frigioni, D.; Marchetti-Spaccamela, A.; and IOS. namic Shortest-Path Problem. Journal of In- Nanni, U. 1996. Fully Dynamic Output Koenig, S. 2001. Agent-Centered Search. AI formation Processing 13(4): 470–476. Bounded Single Source Shortest Path Prob- Magazine 22(4): 109–131. Liu, Y.; Koenig, S.; and Furcy, D. 2002. lem. In Proceedings of the Seventh Annual Koenig, S., and Likhachev, M. 2002a. D* Speeding Up the Calculation of Heuristics Symposium on Discrete Algorithms, LITE. In Proceedings of the Eighteenth Na- for Heuristic Search-Based Planning. In Pro- 212–221. New York: Association of Com- tional Conference on Artificial Intelligence, ceedings of the Eighteenth National Con- puting Machinery. 476–483. Menlo Park, Calif.: American As- ference on Artificial Intelligence, 484–491. Gerevini, A., and Serina, I. 2000. Fast Plan sociation for Artificial Intelligence. Menlo Park, Calif.: American Association Adaptation through Planning Graphs: Lo- Koenig, S., and Likhachev, M. 2002b. INCRE- for Artificial Intelligence. cal and Systematic Search Techniques. In MENTAL A*. In Advances in Neural Information McDermott, D. 1996. A Heuristic Estimator Proceedings of the Fifth International Confer- Processing Systems 14, eds. T. Dietterich, S. for Means-Ends Analysis in Planning. In ence on Artificial Intelligence Planning Sys- Becker, and S. Ghahramani, 1539–1546. Proceedings of the Third International Confer- tems, eds. S. Chien, S. Kambhampati, and Cambridge, Mass.: MIT Press. ence on Artificial Intelligence Planning Sys- C. A. Knoblock, 112–121. Menlo Park, Koenig, S.; Furcy, D.; and Bauer, C. 2002. tems, ed. Brian Drabble, 142–149. Menlo Calif.: AAAI Press. Heuristic Search-Based Replanning. In Pro- Park, Calif.: AAAI Press. Goto, S., and Sangiovanni-Vincentelli, A. ceedings of the Sixth International Conference Matthies, L.; Xiong, Y.; Hogg, R.; Zhu, D.; 1978. A New Shortest-Path Updating Algo- on Artificial Intelligence Planning Systems, Rankin, A.; Kennedy, B.; Hebert, M.; rithm. Networks 8(4): 341–372. eds. M. Ghallab, J. Hertzberg, and P. Traver- Maclachlan, R.; Won, C.; Frost, T.; Hammond, K. 1990. Explaining and Re- so, 294–301. Menlo Park, Calif.: AAAI Press. Sukhatme, G.; McHenry, M.; and Goldberg, pairing Plans That Fail. Artificial Intelligence Koenig, S.; Likhachev, M.; and Furcy, D. S. 2000. A Portable, Autonomous, Urban 45(1–2): 173–228. 2004. LIFELONG PLANNING A*. Artificial Intelli- Reconnaissance Robot. In Proceedings of Hanks, S., and Weld, D. 1995. A Domain- gence Journal 155:93–146. the International Conference on Intelligent Independent Algorithm for Plan Adapta- Koenig, S.; Tovey, C.; and Halliburton, W. Autonomous Systems. Amsterdam, The tion. Journal of Artificial Intelligence Research 2001. Greedy Mapping of Terrain. In Pro- Netherlands: IOS. 2:319–360. ceedings of the IEEE International Confer- Miguel, I., and Shen, Q. 1999. Extending Hebert, M.; McLachlan, R.; and Chang, P. ence on Robotics and Automation, FCSP to Support Dynamically Changing 1999. Experiments with Driving Modes for 3594–3599. Washington, D.C.: IEEE Com- Problems. In Proceedings of IEEE Interna- Urban Robots. In Proceedings of the SPIE puter Society. tional Fuzzy Systems Conference, 1615–1620. Washington, D.C.: IEEE Com- Mobile Robots, 140–149. Bellingham, Koenig, S.; Tovey, C.; and Smirnov, Y. 2003. puter Society. Wash.: International Society for Optical En- Performance Bounds for Planning in Un- gineering. known Terrain. Artificial Intelligence Moore, A., and Atkeson, C. 1995. The PARTI- Huiming, Y.; Chia-Jung, C.; Tong, S.; and 147(1–2): 253–279. GAME Algorithm for Variable Resolution Re- inforcement Learning in Multidimensional Qiang, B. 2001. Hybrid Evolutionary Mo- Kollmann, J.; Lankenau, A.; Buhlmeier, A.; State Spaces. Machine Learning 21(3): tion Planning Using Follow Boundary Re- Krieg-Bruckner, B.; and Rofer, T. 1997. Nav- 199–233. pair for Mobile Robots. Journal of Systems igation of a Kinematically Restricted Architecture 47(7): 635–647. Wheelchair by the PARTI-GAME Algorithm. In Moore, A., and Atkeson, C. 1993. PRIORITIZED Ishida, T., and Korf, R. 1991. Moving Target Proceedings of the Society for the Study of SWEEPING: Reinforcement Learning with Search. In Proceedings of the Twelfth Inter- Artificial Intelligence and the Simulation of Less Data and Less Time. Machine Learning national Joint Conference on Artificial In- Behavior 1997 Workshop on Spatial Rea- 13(1): 103–130. telligence, 204–210. Menlo Park, Calif.: In- soning in Mobile Robots and Animals, Myers, K. 1999. CPEF: A Continuous Plan-
SUMMER 2004 111 Articles ning and Execution Framework. Artificial Search Methods. In Proceedings of the Tenth Reuse in Dynamic Constraint-Satisfaction Intelligence 20(4): 63–69. European Conference on Artificial Intelligence, Problems. In Proceedings of the Twelfth Nebel, B., and Koehler, J. 1995. Plan Reuse ECAI92, 1–5. Chichester, U.K.: Wiley. National Conference on Artificial Intelli- versus Plan Generation: A Theoretical and Simmons, R. 1988. A Theory of Debugging gence, 307–312. Menlo Park, Calif.: Ameri- Empirical Analysis. Artificial Intelligence Plans and Interpretations. In Proceedings can Association for Artificial Intelligence. 76(1–2): 427–454. of the Seventh National Conference on Ar- Watkins, C., and Dayan, P. 1992. Q-LEARN- Nilsson, N. 1971. Problem-Solving Methods tificial Intelligence, 94–99. Menlo Park, ING. Machine Learning 8(3–4): 279–292. Calif.: American Association for Artificial in Artificial Intelligence. New York: McGraw- Sven Koenig is an asso- Intelligence. Hill. ciate professor at the Pearl, J. 1985. Heuristics: Intelligent Search Spira, P., and Pan, A. 1975. On Finding and University of Southern Strategies for Computer Problem Solving. Read- Updating Spanning Trees and Shortest California. His research ing, Mass.: Addison-Wesley. Paths. SIAM Journal on Computing 4(3): interests are in the areas Pemberton, J., and Korf, R. 1994. Incremen- 375–380. of decision-theoretic tal Search Algorithms for Real-Time Deci- Stentz, A. 1995. The Focused D* Algorithm planning and robotics. sion Making. In Proceedings of the Second In- for Real-Time Replanning. In Proceedings He has coauthored over ternational Conference on Artificial of the Fourteenth International Joint Con- 70 publications in these Intelligence Planning Systems, ed. K. Ham- ference on Artificial Intelligence, areas and is the recipient of a National Sci- mond, 140–145. Menlo Park, Calif.: AAAI 1652–1659. Menlo Park, Calif.: Internation- ence Foundation CAREER award and an Press. al Joint Conferences on Artificial Intelli- IBM Faculty Partnership award. His e-mail Peng, J., and Williams, R. 1993. Efficient gence. address is [email protected]. Learning and Planning within the DYNA Stentz, A., and Hebert, M. 1995. A Com- Maxim Likhachev is a Ph.D. student at Framework. Adaptive Behavior 1(4): plete Navigation System for Goal Acquisi- Carnegie Mellon University. His research 437–454. tion in Unknown Environments. Au- interests are in robotics. Podsedkowski, L.; Nowakowski, J.; tonomous Robots 2(2): 127–145. Yaxin Liu is a Ph.D. student at the Georgia Idzikowski, M.; and Vizvary, I. 2001. A New Sutton, R., and Barto, A. 1998. Reinforce- Institute of Technology. His research inter- Solution for Path Planning in Partially ment Learning: An Introduction. Cambridge, ests are in the area of decision-theoretic Known or Unknown Environments for Mass.: MIT Press. planning. He is the recipient of an IBM Nonholonomic Mobile Robots. Robotics Tao, M.; Elssamadisy, A.; Flann, N.; and Ab- graduate fellowship. and Autonomous Systems 34:145–152. bott, B. 1997. Optimal Route Replanning David Furcy is a Ph.D. student at the Geor- Pohl, I. 1973. The Avoidance of (Relative) for Mobile Robots: A Massively Parallel In- gia Institute of Technology. His research in- Catastrophe, Heuristic Competence, Gen- cremental A* Algorithm. In Proceedings of terests are in the area of search. His-email uine Dynamic Weighting and Computa- the IEEE International Conference on Ro- address is [email protected]. tional Issues in Heuristic Problem Solving. botics and Automation, 2727–2732. Wash- In Proceedings of the International Confer- ington, D.C.: IEEE Computer Society. ence on Intelligent Robots and Systems, Thayer, S.; Digney, B.; Diaz, M.; Stentz, A.; 20–23. Washington, D.C.: IEEE Computer Nabbe, B.; and Hebert, M. 2000. Distributed Society. Robotic Mapping of Extreme Environ- Pohl, I. 1970. First Results on the Effect of ments. In Proceedings of the SPIE: Mobile Error in Heuristic Search. In Machine Intel- Robots XV and Telemanipulator and Telep- ligence, Volume 5, eds. B. Meltzer and D. resence Technologies VII, Volume 4195. Michie, 219–236. New York: American Else- Bellingham, WA: International Society for vier. Optical Engineering. Ramalingam, G., and Reps, T. 1996a. An In- Thrun, S. 1998. Lifelong Learning Algo- cremental Algorithm for a Generalization rithms. In Learning to Learn, eds. S. Thrun of the Shortest-Path Problem. Journal of Al- and L. Pratt, 181–209. New York: Kluwer gorithms 21(2): 267–305. Academic. Ramalingam, G., and Reps, T. 1996b. On Thrun, S.; Bücken, A.; Burgard, W.; Fox, D.; the Computational Complexity of Dynam- Fröhlinghaus, T.; Hennig, D.; Hofmann, T.; ic Graph Problems. Theoretical Computer Krell, M.; and Schmidt, T. 1998. Map Learn- Science 158(1–2): 233–277. ing and High-Speed Navigation in RHINO. In Rohnert, H. 1985. A Dynamization of the Artificial Intelligence– Based Mobile Robotics: All Pairs Least Cost Path Problem. In Pro- Case Studies of Successful Robot Systems, eds. ceedings of the Second Annual Symposium on D. Kortenkamp, R. Bonasso, and R. Mur- Theoretical Aspects of Computer Science, phy, 21–52. Cambridge, Mass.: MIT Press 279–286. New York: Springer-Verlag. Trovato, K. 1990. DIFFERENTIAL A*: An Adap- Romero, L.; Morales, E.; and Sucar, E. 2001. tive Search Method Illustrated with Robot An Exploration and Navigation Approach Path Planning for Moving Obstacles and for Indoor Mobile Robots Considering Sen- Goals and an Uncertain Environment. Jour- sor’s Perceptual Limitations. In Proceedings nal of Pattern Recognition and Artificial Intel- of the International Conference on Robot- ligence 4(2): 245–268. ics and Automation, 3092–3097. Washing- Veloso, M. 1994. Planning and Learning by ton, D.C.: IEEE Computer Society. Analogical Reasoning. New York: Springer. Russell, S. 1992. Efficient Memory-Bounded Verfaillie, G., and Schiex, T. 1994. Solution
112 AI MAGAZINE