Human-Robot Interaction and Mapping with a Service Robot: Human Augmented Mapping ELIN ANNA TOPP Doctoral Thesis Stockholm, Sweden 2008 TRITA-CSC-A 2008:12 KTH ISSN-1653-5723 School of Computer Science and Communication ISRN-KTH/CSC/A--08/12--SE SE-100 44 Stockholm ISBN 978–91–7415–118–3 SWEDEN Akademisk avhandling som med tillst˚and av Kungl Tekniska h¨ogskolan framl¨ag- ges till offentlig granskning f¨or avl¨aggande av teknologie doktorsexamen i datalogi m˚andagen den 13 oktober 2008 klockan 10.15 i sal F3, Kungl Tekniska h¨ogskolan, Lindstedtsv¨agen 26, Stockholm. c Elin Anna Topp, September 2008 Tryck: Universitetsservice US AB Abstract An issue widely discussed in robotics research is the ageing society with its conse- quences for care-giving institutions and opportunities for developments in the area of service robots and robot companions. The general idea of using robotic systems in a per- sonal or private context to support an independent way of living not only for the elderly but also for the physically impaired is pursued in different ways, ranging from socially ori- ented robotic pets to mobile assistants. Thus, the idea of the personalised general service robot is not too far fetched. Crucial for such a service robot is the ability to navigate in its working environment, which has to be assumed an arbitrary domestic or office-like environment that is shared with human users and bystanders. With methods developed and investigated in the field of simultaneous localisation and mapping it has become pos- sible for mobile robots to explore and map an unknown environment, while they can stay localised with respect to their starting point and the surroundings. These approaches though do not consider the representation of the environment that is used by humans to refer to particular places. Robotic maps are often metric representations of features that can be obtained from sensory data. Humans have a more topological, in fact partially hierarchical way of representing environments. Especially for the communication between a user and her personal robot it is thus necessary to provide a link between the robotic map and the human understanding of the robot’s workspace. The term Human Augmented Mapping is used for a framework that allows to integrate a robotic map with human concepts. Communication about the environment can thus be facilitated. By assuming an interactive setting for the map acquisition process it is possible for the user to influence the process significantly. Personal preferences can be made part of the environment representation that is acquired by the robot. Advantages become also obvious for the mapping process itself, since in an interactive setting the robot can ask for information and resolve ambiguities with the help of the user. Thus, a scenario of a “guided tour” in which a user can ask a robot to follow and present the surroundings is assumed as the starting point for a system for the integration of robotic mapping, interaction and human environment representations. A central point is the development of a generic, partially hierarchical environment model, that is applied in a topological graph structure as part of an overall experimental Human Augmented Mapping system implementation. Different aspects regarding the rep- resentation of entities of the spatial concepts used in this hierarchical model, particularly considering regions, are investigated. The proposed representation is evaluated both as description of delimited regions and for the detection of transitions between them. In three user studies different aspects of the human-robot interaction issues of Human Augmented Mapping are investigated and discussed. Results from the studies support the proposed model and representation approaches and can serve as basis for further studies in this area. III Preface The present doctoral thesis consolidates results of four years of work with a concep- tual design for an approach to hierarchical, interactively controlled robotic mapping and localisation, conducted at the Centre for Autonomous Systems (CAS) hosted by the Computational Vision and Active Perception (CVAP) group of the School of Computer Science and Communication (CSC) at the Royal Institute of Technology (KTH) in Stockholm. “Human Augmented Mapping” (HAM) is a central term which is introduced and used to subsume the discussed aspects of robotic mapping and human robot interaction. The general concept of HAM is described and dis- cussed together with results that have been achieved with a respective experimental system implementation. Those results also include observations made in three dif- ferent user studies, that were conducted to test the assumptions underlying the models used for the work. The work contributed to a large extent to the integrated EU-project “COGN- IRON - The Cognitive robot companion”1, in particular to the Key Experiment 1 – “The Home Tour”, that served as a demonstrator experiment to document the results achieved in the project’s research activities on “Models of space” and “Multi- modal dialogue”. Additionally the work was to a large extent funded through the project by the European Commission Division FP6-IST Future and Emerging Tech- nologies under Contract FP6-002020. The funding is gratefully acknowledged. Within and related to the project work a number of collaborations inside and outside the Royal Institute of Technology could be established, that contributed to the results presented in this thesis. My own contributions within these collabora- tions are pointed out specifically in the respective chapters and sections. Anders Green and Helge H¨uttenrauch, at the time of writing both affiliated with the Human-Computer Interaction group of CSC at KTH, conducted a user study early in the project that inspired my thoughts about the environment model and the first user study setup described in this thesis. It was also possible for me to use data collected during this first user study to evaluate the tracking method that is part of my approach to a complete system to Human Augmented Mapping. At this point I want to thank Anders and Helge for these opportunities and the inspiring discussions, particularly with Anders on “spatial prompting”, a concept he developed and is about to publish in his doctoral thesis. Further, I designed and conducted all three user studies described in this thesis in cooperation with Helge H¨uttenrauch. Thanks again to Helge, in this case partic- ularly for having the somewhat “different” idea of taking the robot out of the lab and into people’s homes to conduct one of the studies, this was a great experience. One of the studies conducted in the laboratory was actually assigned as a mas- ter’s project to Farah Hassan Ibrahim, jointly supervised by Helge and myself, and at the time of writing a registered student in the computer science programme of CSC/KTH. Other master’s projects and one short-term undergraduate project 1www.cogniron.org (URL verified August 19, 2008) V VI Preface (a German “Studienarbeit”) related to my work and supervised by myself were assigned to and conducted by Alvaro Canivell Garc´ıa de Paredes, Maryamossadat Nematollahi Mahani, and Stephan Platzek. Thanks for working with me and coping with the often vague and exploratory ideas for your tasks! The COGNIRON project’s aim to demonstrate results from different research activities as integrated demonstrators in different Key Experiments built the basis for a very fruitful collaboration with the Applied Computer Science group at the University of Bielefeld, Germany. The efforts put into the transfer of a significant part of my experimental implementation to an integrated interactive framework made by Marc Hanheide, Julia Peltason, Frederic Siepmann, Thorsten Spexard, and Sebastian Wrede (in alphabetical order), and myself led to a prototypical fully integrated interactive system for Human Augmented Mapping, that could be demonstrated successfully in the context of the project. Thanks, for helping to get all that code to work! A joint effort within the research activity on “Models of space” made me travel to the Intelligent Autonomous Systems group at the University of Amsterdam with a SICK laser range finder in my carry-on luggage – also this being an interesting experience in itself – to collect a data set that was made public to give research groups dealing with some form of semantic or interactive mapping a basis to com- pare their approaches. This cooperation with Olaf Booij, Bas Terwijn, and Zoran Zivkovic led to one of the publications listed as related to this thesis. Thanks! The cooperations already indicate that this thesis project involved a lot of trav- elling: A robot travelled through the greater Stockholm area, students travelled back and forth to pursue their international study programmes, a laser range finder got to fly to Amsterdam, and I myself travelled, well ... I was warned right before I travelled back to Germany after my master’s project (which I already travelled to Stockholm for), that I would be travelling alotduring my time as a PhD student, due to the European project. I thought “great, I love travelling”. After more than 30 take-offs and landings – both for professional and for private reasons – during my first year in the graduate program and actually recognising members of the SAS in-flight staff on particular routes after the second I started to revise that statement slightly. I was also warned that my adviser was a person difficult to meet, due to him travelling a lot more than I did anyway. “If you want to talk to him, book the same flight”, I figured – I tried, it did not work, he would end up being seated somewhere completely different on the plane. Nevertheless, I managed to achieve during 48 months of doctoral studies spread over five years what I planned to, actually a bit more, this “bit” now being roughly 1 1 2 years old.