Die approbierte Originalversion dieser Dissertation ist an der Hauptbibliothek der Technischen Universität Wien aufgestellt (http://www.ub.tuwien.ac.at). The approved original version of this thesis is available at the main library of the Vienna University of Technology (http://www.ub.tuwien.ac.at/englweb/). DISSERTATION Ubiquitous Animated Agents for Augmented Reality ausgefuhrt¨ zum Zwecke der Erlangung des akademischen Grades eines Doktors der technischen Wissenschaften unter der Leitung von Univ.-Prof. Dr. Dieter Schmalstieg Institut fur¨ Maschinelles Sehen und Darstellen (ICG) Technische Universitat¨ Graz eingereicht an der Technischen Universitat¨ Wien Fakultat¨ fur¨ Informatik von M.Sc. Istvan´ Barakonyi Rossauer Lande¨ 41/18 1090 Wien Matr.-Nr. 0326849 Wien, im Oktober 2006 Ubiquitous Animated Agents for Augmented Reality Istvan´ Barakonyi – Dissertation Reviewers: Dieter Schmalstieg Andreas Butz Abstract A growing spectrum of Ubiquitous Computing (UbiComp) applications sug- gests that interaction with computers should be as natural and effortless as using pen, paper and language when writing. Unlike current computer environments that require a considerable amount of adaptation from users for smooth interaction, future digital interfaces are envisioned to act unob- trusively and intelligently in our environment. This dissertation describes a novel user interface approach combining Augmented Reality (AR), UbiComp and Autonomous Animated Agents into a single coherent human-computer interface paradigm that makes steps toward this vision. A significant challenge for the UbiComp community is to create efficient, natural and user-friendly interfaces since there are no standards and best practices to follow yet. Typical UbiComp scenarios include numerous mo- bile users roaming a large area while interacting with various stationary and mobile devices. Since the location and behavior of users and devices change rather frequently, an enormous amount of events describing changes gets gen- erated in the environment. Processing such large data sets can be greatly overwhelming for humans, therefore an interface to a UbiComp system is expected to possess certain autonomy in order to filter and interpret relevant events and react proactively without constant user guidance and explicit instructions. By relieving users from dealing with low-level details and al- lowing computers to make decisions by themselves, these interfaces appear to be “smart”. This thesis presents software solutions that employ reactive, autonomous and social digital assistants in UbiComp environments. These systems rely on software agent technology tailored to the needs of AR applications, where system behavior is visualized by virtual animated characters appearing on top of the real world. We discuss how autonomous animated agents can be employed to mediate communication between humans and computers in AR environments while exploiting real world attributes as input and output communication channels. The agents maintain a model of the real world by analyzing data coming from the sensors that measure physical properties such as pose, velocity, sound or light, and autonomously react to changes in the environment in accordance with the users’ perception. Autonomous, emergent behavior is a novel feature in UbiComp, while awareness of real world attributes is yet unexploited by autonomous agents. This dissertation explores the requirements for context-aware animated agents concerning visualization, appearance, and behavior as well as asso- ciated technologies, application areas, and implementation details. Several application scenarios illustrate design and implementation concepts. i Kurzfassung Ubiquitous Computing (UbiComp, ubiquit¨areComputertechnik) zielt darauf ab, daß die Interaktion mit Computers so nat¨urlich und m¨uhelossein soll wie das Schreiben mit einem Stift auf Papier. Im Gegensatz zu derzeitigen Com- putersystemen, die dem Benutzer Anpassung abverlangen, agieren zuk¨unftige digitale Benutzerschnittstellen unauff¨allig und intelligent im Hintergrund. Diese Dissertation beschreibt eine neue Art von Benutzerschnittstellen, die die Vorteile von Augmented Reality (AR, erweiterte Realit¨at),UbiComp und autonomen animierten Agenten vereinigt, um eine verbesserte Mensch- Maschine Interaktion realisieren zu k¨onnen. Eine signifikante Herausforderung dabei ist die Erschaffung effizienter, nat¨urlicher und benutzerfreundlicher Schnittstellen f¨urUbiComp-Systeme, f¨urdie es derzeit keine etablierten Gestaltungsrichtlinien gibt. In typischen UbiComp-Szenarien werden mehrere mobile Benutzer bedient, die sich in einem weitl¨aufigenBereich frei bewegen und dabei mit verschiedenen sta- tion¨arenund mobilen Ger¨ate interagieren. Da der Standort und das Ver- halten der Benutzern sich laufend ¨andert, wird eine enorme Menge von Sta- tusinformation ¨uber den aktuellen Systemstand generiert, die nicht mehr mit manuellen Methoden verwertbar ist. Deshalb wird von zuk¨unftigen UbiComp-Systemen erwartet, daß sie autonom und ohne aktive menschliche Hilfe arbeiten. Diese Dissertation pr¨asentiert eine Softwarel¨osungf¨urdie Implementierung von autonomen und sozialen computergenerierten Assistenten f¨urUbiComp- Umgebungen. Das vorgestellte System benutzt eine Kombination von Metho- den aus den Bereichen Software-Agenten und Augmented Reality, um virtuelle animierte Agenten darzustellen, die mit dem Benutzer sowohl in der virtuellen als auch in der realen Welt interagieren. Die Agenten benutzen ein in- ternes Weltmodell, welches auf der Analyse von Sensordaten f¨urPosition, Geschwindigkeit, Audio, Licht und andere Eigenschaften der realen Umge- bung beruht, und reagieren selbst¨andigauf die Anderungen.¨ Die Neuigkeit des Ansatzes liegt im autonomen, selbst¨andigen Verhalten der Agenten, welche die Attributen der realen Welt bislang noch nicht vollst¨andig ausgenutzt haben. ii Acknowledgements The true winner of this dissertation is my wife, Rita. She was the one who shared my joy at the “ups” and gave me energy when I felt deflated at the “downs”. Without her I would have not succeeded and therefore I dedicate this thesis to her. The image below is for everybody whose partner is working on a PhD. Dieter Schmalstieg, my PhD supervisor provided me with great ideas to improve my research and a stable financial background to let me worry only about scientific results. His continuous attention and guidance gave me steady motivation for my work. I also want to thank him for accepting my stubbornness in my choice of research topics. While working at the Vienna University of Technology, I had a chance and great pleasure to work with Christian Breiteneder, whose professional attitude to scientific work and social competence have made a lasting impact on me. I also would like to thank Mitsuru Ishizuka at the Tokyo University and Helmut Prendinger at the National Institute of Technology, Japan for giving me the opportunity to cooperate with them and for opening up the exciting domain of autonomous agent research for me. One of the most important values gained from my PhD studies was that I was able to share the misery of tight deadlines and the delight of suc- cess with numerous colleagues in the Studierstube team at the Graz and Vi- enna University of Technology. A special honorary mention goes to Joseph Newman (we has it, precious!), thanks, Joe, for being a perfect office and roommate for such a long time! I am grateful to numerous people for their continuous support (in alphabetical order): Alexander Bornik, Tamer Fahmy, Markus Grabner, Denis Kalkofen, Michael Kalkusch, Hannes Kaufmann, iii Karin Kosina, Florian Ledermann, Erick Mendez, Judith M¨uhl,Thomas Pin- taric, Thomas Psik, Bernhard Reitinger, Gerhard Reitmayr, Markus Sareika, Gerhard Schall, Eduardo Veas, Daniel Wagner, and Albert Walzer. I will miss the great group atmosphere! Circled Cube, Ulrich Krispel, Christoph Schinko, and Markus Weilguny contributed precious work to some demo ap- plications of mine. The long way that led me to completing my PhD would have been im- possible without the continuous and unconditional emotional and financial support of my parents. I thank them for always being there for me. iv Contents Abstract i Kurzfassung ii Acknowledgements iii Table of Contents vii List of Figures ix 1 Introduction 1 1.1 The World as User Interface . 1 1.1.1 Augmented Reality . 2 1.1.2 Ubiquitous Computing . 4 1.1.3 Software Agents . 5 1.2 Contribution . 7 2 Related Work 11 2.1 Adaptive User Interfaces . 12 2.1.1 Information Filtering . 12 2.1.2 Adaptive User Interface Components . 13 2.1.3 User Interface Migration . 13 2.2 Software Agents in AR . 15 2.2.1 Animated Characters . 15 2.2.2 Mobile Agents . 18 3 Augmented Reality Agents 21 3.1 Design Requirements for Agents in AR . 21 3.1.1 Agent Representation . 22 3.1.2 Agent Behavior . 25 3.2 The AR Puppet Framework . 28 3.2.1 Puppet . 29 v Contents vi 3.2.2 Puppeteer . 30 3.2.3 Choreographer . 32 3.2.4 Director . 33 3.2.5 Storyteller . 34 3.3 Integration with Applications . 34 3.3.1 Example Application Scenario . 36 3.3.2 Communication Flow between Components . 39 3.4 Interaction between the Real and Virtual . 42 3.4.1 Physical Input Affecting Virtual Output . 43 3.4.2 Virtual Input Affecting Physical Output . 43 3.4.3 Other scenarios . 44 4 Ubiquitous Augmented Reality Agents 45 4.1 Improving AR Puppet . 45 4.1.1 Increasing Mobility . 47 4.1.2 Expect the Unexpected . 48 4.1.3 Multi-user Interface Adaptation . 50 4.1.4 Beliefs, Desires, Intentions . 51 4.1.5 Autonomic and Proactive Behavior . 53 4.2 UbiAgent Components . 54 4.2.1 Shared Agent and Application Memory . 56 4.2.2 Agent Migration . 57 5 Applications 59 5.1 AR Lego . 60 5.1.1 Application Scenario . 61 5.1.2 Agent-Application Communication . 62 5.1.3 LEGO robot agent . 62 5.1.4 Interaction . 65 5.2 Monkeybridge . 66 5.2.1 Motivation of AR Gaming . 66 5.2.2 Application Scenario . 68 5.2.3 Autonomous Game Characters . 70 5.2.4 Domains of Game Experience . 70 5.2.5 Game Setups . 72 5.3 Virtual Tour Guide . 75 5.3.1 Application description . 75 5.3.2 Integration with the APRIL Framework . 75 5.3.3 Hardware Setups . 78 5.4 Character Animation Studio . 78 5.4.1 Application Scenario .