Planning in Inhabited Environments Örebro Studies in Technology 42 Marcello Cirillo Planning in Inhabited Environments Human-Aware Task Planning and Activity Recognition © Marcello Cirillo, 2010 Title: Planning in Inhabited Environments: Human-Aware Task Planning and Activity Recognition Publisher: Örebro University, 2010 www.publications.oru.se [email protected] Printer: Örebro University, Repro 11/2010 issn 1650-8580 isbn 978-91-7668-756-7 Abstract Promised some decades ago by researchers in artificial intelligence and robotics as an imminent breakthrough in our everyday lives, a robotic assistant that could work with us in our home and our workplace is a dream still far from being fulfilled. The work presented in this thesis aims at bringing this future vision a little closer to realization. Here, we start from the assumption that an efficient robotic helper should not impose constraints on users’ activities, but rather perform its tasks unob- trusively to fulfill its goals and to facilitate people in achieving their objectives. Also, the helper should be able to consider the outcome of possible future ac- tions by the human users, to assess how those would affect the environment with respect to the agent’s objectives, and to predict when its support will be needed. In this thesis we address two highly interconnected problems that are essen- tial for the cohabitation of people and service robots: robot task planning and human activity recognition. First, we present human-aware planning, that is, our approach to robot high-level symbolic reasoning for plan generation. Human-aware planning can be applied in situations where there is a controllable agent, the robot, whose actions we can plan, and one or more uncontrollable agents, the human users, whose future actions we can only try to predict. In our approach, therefore, the knowledge of the users’ current and future activities is an important prerequi- site. We define human-aware as a new type of planning problem, we formal- ize the extensions needed by a classical planner to solve such a problem, and we present the implementation of a planner that satisfies all identified require- ments. In this thesis we explore also a second issue, which is a prerequisite to the first one: human activity monitoring in intelligent environments. We adopt a knowledge driven approach to activity recognition, whereby a constraint-based domain description is used to correlate sensor readings to human activities. We validate our solutions to both human-aware planning and activity recog- nition both theoretically and experimentally, describing a number of explana- tory examples and test runs in a real environment. i Acknowledgements First of all, I would like to thank my supervisors, Alessandro Saffiotti, Lars Karlsson and Federico Pecora for the patient guidance they offered me during the three and a half years of my PhD. I also profoundly thank my family, both the close and the extended one, for supporting me during this long time, in spite of the geographical distance. For the work presented here, I would particularly like to thank Federico Pecora and Jonas Ullberg, with whom I cooperated to design and develop one of the systems described in these pages. I would also like to thank CUGS, the Swedish National Graduate School in Computer Science, which has provided funding and arranged courses and workshops. It is not possible to remember all the useful (and sometimes enlightening) discussions I had over the years with the people at AASS, but I know that most of my colleagues have contributed, one way or another, to the final results presented in this thesis. Among the students I supervised for their own theses during my PhD, I have the pleasure to remember Francesca Dell’Osa, whose work has provided useful material for my own studies. There are two more people whom I gladly thank for their continuous friendly support: Barbro Alvin and Kicki Ekberg. Without them I probably would be homeless. Obviously, also the most interesting of jobs is unpleasant if it is not mixed with more fun and light moments and with good company. Many old friends are still in Italy and came to visit every now and then, but luckily I found a good deal of friends also here in Örebro, with whom I spent many a pleasant day indeed. The first person I want to thank is Ellen, whose presence pulled me back from the depths of insanity in my last year. Many of the most interesting stories I will still narrate in ten years time would have not been possible without the friendship of Trinchetto the mountain kid, Teo the chef, Pex, Thor, Roomie, Kevin (they killed him!), Karol and the buffalo soldiers, Luigi, Barbie, Uncle Mathias, the other Mattias, Marios, Achim, Henrik, Jay, Martin, Per and Jonas Ullgren. I really hope that the next generations of PhD students here will bond in such strangely assorted groups as well. iii Contents 1 Introduction 1 1.1 The Dream of a Robot Butler . .1 1.2 Two Motivating Scenarios . .2 1.2.1 Robot Task Planning . .3 1.2.2 Monitoring Human Activities . .5 1.3 Methodology . .5 1.4 Thesis Outline . .6 1.5 Publications . .7 2 Related Work 9 2.1 Planning for HRI . .9 2.1.1 Human-Robot Joint Task Planning . 10 2.1.2 Robot Task Planning with People Present . 11 2.2 Activity Recognition . 12 2.2.1 Data-driven Approaches . 12 2.2.2 Model-driven Approaches . 13 3 Human-Aware Planning 17 3.1 Overview . 17 3.2 States, Situations and Actions . 19 3.3 Interaction Constraints and Action Preconditions . 24 3.4 Partial Observability . 26 3.5 Planning Problem and Solution . 28 3.5.1 Problem . 28 3.5.2 Solution . 30 3.6 The HA-PTLplan Algorithm . 31 3.6.1 Termination Conditions . 33 3.6.2 Complexity . 33 3.6.3 Pruning the Search Space . 34 3.7 A Full Example . 35 3.7.1 Planning Problem . 35 v vi CONTENTS 3.7.2 Problem Solution . 36 3.8 Performance Evaluation . 38 3.8.1 The Vacuum Cleaner Scenario . 39 3.8.2 The Moving Table Scenario . 43 3.9 Summary . 44 4 Multi-Human Human-Aware Planning 47 4.1 Multi-Human HA-PTLPlan . 47 4.2 Interaction Constraints with Multiple Humans . 51 4.3 Partial Observability . 51 4.4 Policy Generation with Multiple Humans . 53 4.5 Algorithm . 55 4.5.1 Termination Conditions . 55 4.5.2 Complexity . 56 4.5.3 Pruning the Search Space . 56 4.6 Application Scenarios . 56 4.6.1 A Domestic Helper . 57 4.6.2 The Surgeon Helper . 61 4.7 Summary . 63 5 Activity Monitoring 65 5.1 Domain Representation . 66 5.1.1 Modeling Knowledge for Activity Recognition . 67 5.1.2 Modeling Knowledge for Plan Synthesis . 70 5.1.3 Combining Recognition and Plan Synthesis Knowledge . 71 5.2 Recognition and Proactive Execution . 71 5.2.1 Sensing Processes . 72 5.2.2 Continuous Inference Process . 74 5.2.3 Actuation Processes . 77 5.3 Pruning the Search Space . 78 5.4 Tracking Multiple Hypotheses . 81 5.4.1 Extending SAM . 81 5.4.2 Formal Properties and the Markov Assumption . 83 5.4.3 Heuristics and Practical Applicability . 85 5.5 Multiple Hypotheses and Actuation . 86 5.6 SAM and Human-Aware Planning . 87 5.7 Summary . 89 6 Experiments 91 6.1 The PEIS-Ecology Concept and the PEIS-Home . 91 6.2 Human-Aware Planning . 93 6.2.1 Real Domestic Environment . 93 6.2.2 Realistic Factory Environment . 98 6.3 Activity Monitoring . 103 CONTENTS vii 6.3.1 Experimental Setup . 104 6.3.2 Recognizing Activities with SAM . 104 6.3.3 Proactive Service Enactment . 109 6.3.4 A Test Run for Multiple Hypothesis Tracking . 110 6.4 Summary . 113 7 Conclusions 115 7.1 Contributions . 115 7.2 Limitations . 116 7.3 Future Work . 117 List of Figures 1.1 State-of-the-art commercial robots . .2 1.2 Hazardous conditions in a plant in the steel industry . .4 3.1 Example of human-aware planning . 18 3.2 Example of robot and human action transitions . 21 3.3 Example of probabilistic outcomes in action transitions (1) . 21 3.4 Example of probabilistic outcomes in action transitions (2) . 22 3.5 Example of interaction constraints in human-aware planning . 25 3.6 Example of robot policy expansion . 31 3.7 Human-aware planning algorithm . 32 3.8 Full example of robot policy generation . 37 3.9 Performance evaluation: vacuum cleaner scenario, first setup . 40 3.10 Performance evaluation: vacuum cleaner scenario, second setup 42 3.11 Performance evaluation: vacuum cleaner scenario, third setup . 43 3.12 Performance evaluation: moving table scenario . 44 4.1 Example of action transitions with multiple humans . 50 4.2 Example of probabilistic outcomes in action transitions . 52 4.3 Example of robot policy expansion with multiple humans . 54 4.4 Human-aware planning algorithm with multiple humans . 55 4.5 Example of policy generation in a domestic scenario . 59 4.6 Additional morning schedules in a domestic scenario . 61 5.1 Example of knowledge modeling for activity recognition . 69 5.2 Example of knowledge modeling for plan synthesis . 70 5.3 Procedure Sensex(DNt, tnow) .................. 73 5.4 Procedure ActivityRecognition(x,DNt) ........... 76 5.5 Supporting a candidate decision . 76 5.6 Extended ActivityRecognition(x,DNt) procedure . 79 5.7 Timelines relevant to the hypothesis that the user is sleeping . 83 5.8 Evolution in time of multiple hypotheses . 84 ix x LIST OF FIGURES 6.1 The PEIS-Home . 92 6.2 Our framework for human-aware planning .