Department of Informatics University of Fribourg (Switzerland) FRAMEWORKS, DESCRIPTION LANGUAGES AND FUSION ENGINES FOR MULTIMODAL INTERACTIVE SYSTEMS THESIS Presented to the Faculty of Science, University of Fribourg (Switzerland) in consideration for the award of the academic grade of Doctor Scientiarum Informaticarum by Bruno Dumas from Vuisternens-devant-Romont FR (Switzerland) Thesis N° 1695 UniPrint, Fribourg 2010 Accepted by the Faculty of Science of the University of Fribourg (Switzerland) upon the recommendation of – Prof. Ulrich Ultes-Nitsche, University of Fribourg, Switzerland (Jury President) – Dr. Denis Lalanne, University of Fribourg, Switzerland (Thesis Director) – Prof. Rolf Ingold, University of Fribourg, Switzerland (Expert) – Prof. Laurence Nigay, Université Joseph Fourier, Grenoble, France (External Expert) – Prof. Philippe Palanque, Université Paul Sabatier, Toulouse, France (External Expert) Fribourg, November 16th, 2010 Thesis Director Faculty Dean Dr. Denis Lalanne Prof. Rolf Ingold Table of Contents Table of Contents .................................................................................................................................5 Remerciements .....................................................................................................................................9 Abstract...............................................................................................................................................11 Résumé ................................................................................................................................................13 1 Introduction .................................................................................................................................15 1.1 New Trends in Human-Machine Interaction ..........................................................................15 1.2 Multimodal Interaction: Promises and Challenges.................................................................16 1.3 Specific Challenges Addressed in this Thesis ........................................................................18 1.4 Thesis Context ........................................................................................................................19 1.5 Thesis Structure ......................................................................................................................19 2 Conceptual Background..............................................................................................................23 2.1 Foundations, Aims and Features of Multimodal Interaction..................................................24 2.1.1 Definition of Multimodality ............................................................................................24 2.1.2 Key Features of Multimodal Interaction..........................................................................27 2.1.3 Cognitive Foundations of Multimodal Interaction ..........................................................28 2.1.4 Seminal Works, Findings and Guidelines .......................................................................31 2.2 Multimodal Human Machine Interaction Loop......................................................................33 2.3 Design Spaces and Theoretical Frameworks for Reasoning on Multimodal Interaction.......35 2.3.1 The TYCOON Theoretical Framework...........................................................................36 2.3.2 The CASE Design Space.................................................................................................36 2.3.3 The CARE Properties ......................................................................................................38 2.4 Conclusion ..............................................................................................................................39 6 3 Multimodal Interfaces Prototyping Frameworks & Architectures ........................................41 3.1 Terminology............................................................................................................................42 3.2 Related Work ..........................................................................................................................43 3.3 Multimodal Interaction Creation Tools: Prominent Features Related to Fusion....................47 3.3.1 Multimodal Interaction Creation Tools: Input Management ...........................................47 3.3.2 Multimodal Interaction Creation Tools: Architecture features........................................48 3.3.3 Multimodal Interaction Creation Tools: Dialog Description & Programming...............49 3.3.4 Multimodal Interaction Creation Tools: Fusion Engine Characteristics .........................51 3.3.5 Multimodal Interaction Creation Tools: Miscellaneous Features....................................51 3.3.6 Final Analysis ..................................................................................................................52 3.4 Multimodal Interfaces: a Study of Architectures....................................................................53 3.4.1 Computational Architecture and Key Components .........................................................53 3.4.2 Java Swing MM extension: Usability ..............................................................................55 3.4.3 SCS, the Service Counter System: Expressiveness .........................................................57 3.5 The HephaisTK framework, and its Architecture...................................................................60 3.5.1 Initial Requirements for HephaisTK................................................................................61 3.5.2 General Architecture of HephaisTK ................................................................................62 3.5.3 Usability and Expressiveness: Looking for Balance........................................................68 3.6 Usability, Expressiveness and Prototyping Multimodal Systems: an Outlook.......................68 3.6.1 Summarization and Observations ....................................................................................68 3.6.2 Positioning HephaisTK....................................................................................................69 3.7 Conclusion ..............................................................................................................................69 4 Multimodal Interaction Modeling .............................................................................................73 4.1 Related Work ..........................................................................................................................74 4.2 Spectrum of Multimodal Dialog Description Languages .......................................................78 4.3 Multimodal Dialog Description: the Synchronisation Problem..............................................80 4.4 Guidelines For Languages for Describing Multimodal Dialog ..............................................81 4.5 The SMUIML Description Language.....................................................................................83 4.5.1 Structure of the Language................................................................................................84 7 4.5.2 Recognizers......................................................................................................................86 4.5.3 Triggers............................................................................................................................87 4.5.4 Actions.............................................................................................................................89 4.5.5 Dialog ..............................................................................................................................90 4.5.6 SMUIML language interpretation ...................................................................................91 4.6 Positioning of SMUIML.........................................................................................................92 4.7 Conclusion ..............................................................................................................................93 5 Multimodal Fusion: Dialog Management & Algorithms.........................................................95 5.1 Multimodal Fusion of Input Data: A State of the Art.............................................................96 5.1.1 Levels of Fusion of Input Modalities ..............................................................................96 5.1.2 An Historical View on Fusion Engines ...........................................................................98 5.1.3 Dialogue vs Fusion Management: a Discussion..............................................................98 5.1.4 Dialogue Management...................................................................................................100 5.1.5 Algorithms for Fusion of Input Modalities....................................................................101 5.2 Time Synchronization Issues................................................................................................104 5.3 Fusion Handling in HephaisTK: Affected Components and General Organization ............106 5.4 Fusion Algorithms in HephaisTK.........................................................................................108 5.4.1 Meaning Frame-based Fusion of Input Data .................................................................109