A Decentralized Adaptive Architecture for Ubiquitous Augmented Reality Systems

d d d d d d d d d d d d d d d d d d d d Institut fur¨ Informatik der Technischen Universitat¨ Munchen¨ A Decentralized Adaptive Architecture for Ubiquitous Augmented Reality Systems Asa MacWilliams d d d d d d d d d d d d d d d d d d d d Institut fur¨ Informatik der Technischen Universitat¨ Munchen¨ A Decentralized Adaptive Architecture for Ubiquitous Augmented Reality Systems Asa MacWilliams Vollstandiger¨ Abdruck der von der Fakultat¨ fur¨ Informatik der Technischen Universitat¨ Munchen¨ zur Erlangung des akademischen Grades eines Doktors der Naturwissenschaften (Dr. rer. nat.) genehmigten Dissertation. Vorsitzender: Univ.-Prof. Dr. Uwe Baumgarten Prufer¨ der Dissertation: 1. Univ.-Prof. Bernd Brugge,¨ Ph. D. 2. Univ.-Prof. Dr. Dr. h.c. Manfred Broy Die Dissertation wurde am 9. Dezember 2004 bei der Technischen Universitat¨ Munchen¨ eingereicht und durch die Fakultat¨ fur¨ Informatik am 15. Juni 2005 angenommen. Abstract Ubiquitous augmented reality is an emerging human-computer interaction technology, aris- ing from the convergence of augmented reality and ubiquitous computing. Augmented reality allows interaction with virtual objects spatially registered in the user’s real environment, in order to provide information, facilitate collaboration and control machines. As the computing and interaction devices necessary for augmented reality become ubiquitously available, opportunities arise for improved interaction and new applications. Building ubiquitous augmented reality systems presents three software engineering challenges. First, the system must cope with uncertainty regarding the software components; the users’ mobility changes the availability of distributed devices. Second, during development, the system’s desired behavior is ill-defined, as appropriate interaction metaphors are still be- ing researched and users’ preferences change. Third, the system must maintain near-real-time performance to create a convincing user experience. This dissertation presents a new architectural style, the adaptive service dependency architecture, to address these challenges. The architecture deals with component uncertainty, using middleware for decentralized service management to dynamically adapt the system’s structure. It builds on existing architectural approaches, such as loosely coupled services, service discovery, reflection, and data flow architectures, but additionally considers the interdependencies between distributed services and uses them for system adaption. With the development technique of design at run time, users and developers address ill- defined requirements by changing the system’s behavior while it is running, facilitating the creation of new applications. This is based on existing agile development methods, but takes additional advantage of the architecture’s adaptive and reflective properties. Distributed middleware maintains the required performance by decoupling multimedia data flow from system reconfiguration. It combines existing communication and service discovery technologies, but additionally deals with the distributed coordination of networks of interde- pendent services. Several prototype systems for real-world ubiquitous augmented reality applications have been built using the adaptive service dependency architecture, decentralized middleware, and design at run time. Feedback from users and developers, as well as performance tests, show the concepts’ usefulness for building ubiquitous augmented reality systems. Zusammenfassung Ubiquitous Augmented Reality ist eine neue Form der Mensch-Maschine-Interaktion, die aus der Konvergenz aus Augmented Reality und Ubiquitous Computing entsteht. Augmented Reali- ty erlaubt dem Benutzer die Interaktion mit raumlich¨ registrierten virtuellen Objekten in seiner Umgebung, und stellt damit Informationen bereit, erleichtert Zusammenarbeit und steuert Ma- schinen. Aus der zunehmenden Verbreitung der Gerate,¨ die fur¨ Augmented Reality notig¨ sind, ergeben sich Moglichkeiten¨ fur¨ verbesserte Interaktion und neue Anwendungen. Die Konstruktion von Ubiquitous-Augmented-Reality-Systemen stellt drei softwaretechni- sche Herausforderungen. Erstens besteht zur Laufzeit eine Unsicherheit bezuglich¨ der Kom- ponenten; durch Benutzermobilitat¨ schwankt die Verfugbarkeit¨ verteilter Gerate.¨ Zweitens ist die gewunschte¨ Funktionalitat¨ des Systems wahrend¨ der Entwicklung unvollstandig¨ definiert, da Interaktionsmetaphern sich noch im Forschungsstadium befinden und Benutzervorlieben sich andern.¨ Drittens muss die Performanz den Echtzeitanforderungen fur¨ ein uberzeugendes¨ Benutzererlebnis genugen.¨ Diese Dissertation stellt einen neuen Softwarearchitekturstil namens Adaptive Service Depen- dency Architecture vor, der diesen Herausforderungen begegnet. Die Architektur begegnet der Komponentenunsicherheit mittels Middleware zur dezentra- len Diensteverwaltung, die dynamisch die Systemstruktur anpasst. Sie baut auf bestehenden Architekturansatzen¨ wie loser Kopplung, Dienstsuche, Reflektion und Datenflussarchitekturen auf, berucksichtigt¨ aber zusatzlich¨ die wechselseitigen Abhangigkeiten¨ verteilter Dienste zur Adaption. Mit der Entwicklungstechnik Design at Run Time konnen¨ Benutzer und Entwickler den un- vollstandigen¨ Anforderungen begegnen und das System zur Laufzeit weiterentwickeln. Dies erlaubt das schnelle Erstellen von Prototypen und vereinfacht die Entwicklung neuer Anwen- dungen. Diese Technik baut auf bestehenden agilen Entwicklungsmethoden auf, benutzt aber zusatzlich¨ die Adaptivitat¨ und Reflektivitat¨ der Architektur. Verteilte Middleware erhalt¨ die Systemleistung aufrecht, indem multimediale Datenstrome¨ von der Systemadaption entkoppelt werden. Sie kombiniert bestehende Techniken zur Kom- munikation und zur Dienstsuche, behandelt aber zusatzlich¨ die verteilte Koordination wech- selseitig abhangiger¨ Dienste. Mit dieser Kombination aus Architektur, Middleware und Entwicklungstechnik wurden ei- nige prototypische Systeme fur¨ Ubiquitous-Augmented-Reality-Anwendungen entwickelt. Be- nutzer- und Entwicklerreaktionen sowie Leistungsmessungen zeigen die Nutzlichkeit¨ der Kon- zepte fur¨ die Entwicklung von Ubiquitous-Augmented-Reality-Systemen. Preface In the summer of 1999, I had the lucky opportunity to take part in a summer school, orga- nized by the Technische Universitat¨ Munchen,¨ on the subject of augmented reality and wear- able computers. Since then, I have been fascinated by the subject: a technology offering great potential, but also a multitude of research problems. As a result of this summer school, a group of enthusiastic students, including myself, de- termined to build a software framework for mobile augmented reality. We set out to com- bine what we could learn from our group’s professors on the subjects of software engineering (Bernd Brugge)¨ and augmented reality (Gudrun Klinker). The resulting framework was named DWARF, and in my Diplomarbeit, I reported on the middleware of its first prototype. The domain of ubiquitous augmented reality has proven challenging enough that several members of our group have continued to work in the area, and after four additional years, I am pleased to report the new results. I have been fortunate to receive the support of many people in completing this dissertation. Thank you, Bernd and Gudrun, for your collaboration on the fields of software engineering and augmented reality, and your support for our interdisciplinary research group. Thank you, Thomas, for the vision of a distributed augmented reality framework. Thank you, Martin, Christian, Martin, and Marcus, for much hard work in our group and in the community. Thank you, Franz, Daniel, Otmar, Markus, Lothar, Dagmar, Felix, Christian, Fabian, Vinko, Korbinian, Niko, Peter, Maximilian, Gerrit, Ming-Ju, Marco, Jan-Gregor, Ming, and many more, for helping make DWARF a reality. Thank you, Dieter, Joe, Gerhard, Thomas, Tamer, Andreas, Christoph, Andi and Horst, for productive collaborations. Thank you, Timo, Allen, Hesam, Christoph, Claudia, Christian, Tao, Naoufel, Michael and Oliver, for feedback on my dissertation. Thank you, Helma, Monika and Uta, for excellent organizational support. And thank you, nameless reviewers, for constructive criticism of submitted papers. Special thanks to my parents for much support and last-minute proofreading, and of course to Sabine, for unlimited patience and love. And finally, to you, the reader: thank you for the time you are investing to read this dissertation—I hope it proves worthwhile! For questions or feedback, please do not hesitate to contact me at [email protected]. A Decentralized Adaptive Architecture for Ubiquitous Augmented Reality Systems i Overview 1 Introduction .......................................................................... 1 Ubiquitous augmented reality is the convergence of augmented reality and ubiquitous computing. This dissertation address the challenges of developing software for this emerging field. 2 Developing Software for Ubiquitous Augmented Reality............................... 7 Software for ubiquitous augmented reality must address component uncertainty, ill- defined requirements, and performance constraints, letting developers satisfy user in- terests. 3 Related Work ........................................................................ 25 This work is influenced by research in distributed software architectures, adaptive middleware, context-aware computing, ubiquitous computing, and augmented reality; and builds on previous work of myself and of colleagues. 4 Overview of Proposed

Load more