An Information-Driven Architecture for Cognitive Systems Research Sebastian Wrede Dipl.-Inform. Sebastian Wrede AG Angewandte Informatik Technische Fakultät Universität Bielefeld email: [email protected] Abdruck der genehmigten Dissertation zur Erlangung des akademischen Grades Doktor-Ingenieur (Dr.-Ing.). Der Technischen Fakultät der Universität Bielefeld am 09.07.2008 vorgelegt von Sebastian Wrede, am 27.11.2008 verteidigt und genehmigt. Gutachter: Prof. Dr. Gerhard Sagerer, Universität Bielefeld Assoc. Prof. Bruce A. Draper, Colorado State University Prof. Dr. Christian Bauckhage, Universität Bonn Prüfungsausschuss: Prof. Dr. Helge Ritter, Universität Bielefeld Dr. Robert Haschke, Universität Bielefeld Gedruckt auf alterungsbeständigem Papier nach ISO 9706 An Information-Driven Architecture for Cognitive Systems Research Der Technischen Fakultät der Universität Bielefeld zur Erlangung des Grades Doktor-Ingenieur vorgelegt von Sebastian Wrede Bielefeld – Juli 2008 Acknowledgments Conducting research on a computer science topic and writing a thesis in a collaborative research project where the thesis results are at the same time the basis for scientific innovation in the very same project is a truly challenging experience - both for the author as well as its colleagues. Even though, working in the VAMPIRE EU project was always fun. Consequently, I would first like to thank all the people involved in this particular project for their commitment and cooperation. Among all the people supporting me over my PhD years, I need to single out two persons that were of particular importance. First of all, I would like to thank my advisor Christian Bauckhage for his inspiration and his ongoing support from the early days in the VAMPIRE project up to his final com- ments on this thesis manuscript. Also, I am very grateful to my colleague Marc Hanheide for frequent discussions about our ideas and approaches, for opening insights into image processing problems as well as his patience while serving as a beta tester of the developed software. Not to mention that working with Marc in different projects was always fun and at the same time productive. The development of a fundamental infrastructure approach as described in this thesis is of course not possible in isolation. Hence, I would like to thank the members of the Applied Informatics at Bielefeld University, which are not only great colleagues but also exceptional collaborators. However, as there are too many, I cannot mention everyone. Nevertheless, I would still like to thank Jannik Fritsch for initially applying the concepts in our mobile robotics scenarios and disseminating the ideas to the robotics community as well as Ingo Lütkebohle and Jan Schaefer for their contributions and lots of fruitful discussions which inspired and initiated many extensions of the resulting software framework. Furthermore, I want to say “Dank je wel!” to Lisabeth van Iersel who always assisted in bureaucratic paperwork and was willing to listen to any issue that I came across in these years. Last, but not least, I want to express my gratitude to Gerhard Sagerer and Franz Kummert as they provided me the opportunity to pursue a PhD in the Applied Informatics. In particular, I want to thank Gerhard Sagerer for his ongoing support, his encouraging but thought-provoking impulses and his friendly pressure. Of course, I also want to thank the additional members of the examination board, Helge Ritter, Robert Haschke and Bruce Draper for their time and willingness to review this thesis. In addition, I will not forget to deeply appreciate the help and advice of Thorsten Gröger1 when it came to the graphical illustration of complex circumstances – even when you were on holiday. Finally, I want to express my gratitude to those who should in fact be mentioned first: my family. While my dad left far too early, I will always be grateful for his unquestionable trust in me. I thank my mum for sharing the excitement about my research work until now and crossing fingers whenever necessary as well as my sister for constantly motivating me even in doubtful situations. However, without the everyday assistance and patience of Christina, her negotiation skills with regard to Clara when I was staying up all night in my office, and the emotional support by both of you beloved ones, I would never have managed to come this far. 1http://www.stilwechsel.de i Contents 1 Software Integration in Cognitive Systems - A First Encounter 3 1.1 Cognitive Systems for Human-Machine-Interaction . .............. 5 1.2 Viewpoints on Software Architectures for Integrated Cognitive Systems . 7 1.3 Research Questions, Objectives and Approach . ............ 12 1.4 Outline and Contributions . ....... 13 I A Systems Perspective on a Cognitive Vision Project 15 2 The Project Perspective: The VAMPIRE Endeavour 17 2.1 Cognitive Vision - An Emerging Discipline . .......... 17 2.1.1 Modularity and Multiple Computation . ....... 18 2.1.2 Dynamic Coordination and Adaptation . ...... 19 2.1.3 From Sensorial to Symbolic Information . ....... 20 2.2 TheVAMPIREProject ............................... 21 2.2.1 TheHuman-In-The-Loop. 22 2.2.2 The Visual Active Memory Concept . 24 2.3 Summary ........................................ 29 3 The Collaborative Perspective 31 3.1 The Scenario-Driven Research Process . .......... 31 3.2 Software Development and Scenario-Driven Research . .............. 33 3.2.1 Software Integration as Process . ...... 34 3.3 The Social Complexity of Integration . ......... 36 3.3.1 Collaboration and Usability Aspects . ........ 37 3.3.2 Mutual Understanding and Agreement . ...... 38 3.4 Summary ........................................ 40 4 The Technological Perspective 41 4.1 The Consequences of Parallelism . ........ 41 4.2 Distributed Systems and Software Integration . ............. 43 4.2.1 TheRoleofMiddleware . 46 4.2.2 Requirements of Distributed Systems . ....... 46 4.3 The Relevance of Architecture . ....... 53 4.3.1 Modularity as a Key to Software Quality . ...... 54 4.3.2 Software Coupling and Granularity . ...... 55 4.3.3 Architectural Styles and Software Integration . ............ 56 4.4 Summary ........................................ 57 Bielefeld University ii Contents 5 Requirements and Architectures for Integration of Cognitive Systems 59 5.1 Synopsis of Requirements . ..... 59 5.1.1 FunctionalAspects . ...... ..... ...... ...... ..... 60 5.1.2 Non-Functional Aspects . 64 5.1.3 Implementation-specific and Economic Aspects . .......... 67 5.2 Software Architectures and Middleware for Cognitive Systems............ 67 5.2.1 Domain-specific Architectures . ...... 68 5.2.2 General Middleware Architectures . ....... 72 5.3 Evaluation of Selected Approaches . ......... 75 5.3.1 Object-oriented Middleware . ..... 76 5.3.2 Cognitive Vision Middleware . 79 5.3.3 Cognitive Robotics Middleware . ..... 84 5.4 Conclusion ...................................... 88 II The Information-Driven Integration Approach 89 6 Adopting Event-Based System Models 91 6.1 The Manifesto of Information-Driven Integration . .............. 91 6.1.1 StrategicAims ................................. 92 6.1.2 The Service-Oriented Viewpoint . ...... 93 6.1.3 The Event-Driven Perspective . ..... 94 6.1.4 GuidetotheReader .............................. 96 6.2 DocumentModel ................................... 97 6.2.1 Information-oriented Representation . .......... 98 6.2.2 XML Processing and Extensibility . 101 6.2.3 ExploitingReflection. 102 6.3 EventModel...................................... 103 6.3.1 EventMetadata................................. 104 6.3.2 Optimized Packaging of Binary Data . 105 6.3.3 DomainEvents ................................. 106 6.4 ObservationModel ................................ 107 6.4.1 A Hybrid Subscription Model . 107 6.4.2 Transformation-based Event Filtering . ......... 109 6.5 NotificationModel ................................ 114 6.5.1 Implicit Invocation . 115 6.5.2 Visibility and Scopes . 117 6.5.3 Dynamic Dispatch of Event Notifications . ....... 122 6.5.4 Port-based Optimization . 123 6.6 Summary ........................................ 124 Sebastian Wrede Contents iii 7 From Event-based to Cognitive Systems 125 7.1 ResourceModel................................... 125 7.1.1 Services, Interfaces, and Components . ........ 126 7.1.2 NamingResources ............................... 128 7.1.3 The Federated Naming Service . 130 7.2 InteractionModel................................ 130 7.2.1 Connectors and Service Interfaces . ....... 131 7.2.2 Event-based Realization . 135 7.2.3 Adaptation Patterns . 136 7.3 MemoryModel ..................................... 138 7.3.1 Concepts .................................... 138 7.3.2 The ActiveMemory Architecture . 146 7.4 CoordinationModel............................... 153 7.4.1 Formalizing Coordination with Petri Nets . ......... 154 7.4.2 Development, Analysis, and Execution . ....... 156 7.5 DomainModel ..................................... 158 7.5.1 XMLTypeLibraries .............................. 159 7.5.2 Application Adapters for Computer Vision Tools . .......... 160 7.5.3 Application Specific Libraries . 163 7.6 Summary ........................................ 164 III Experimental Evaluation 167 8 The VAMPIRE System 169 8.1 Augmented-Reality for Context-Aware Assistance . .............. 169 8.1.1 An Augmented-Reality Interface for Human-Machine Interaction . 170 8.1.2 The Assistance Scenario . 171 8.2 An Information-Driven Software Architecture . ............. 173 8.2.1 FunctionalView