Model-Driven Development of Interactive Multimedia Applications Towards Better Integration of Software Engineering and Creative Design Andreas Pleuß München 2009 Model-Driven Development of Interactive Multimedia Applications Towards Better Integration of Software Engineering and Creative Design Andreas Pleuß Dissertation an der Fakultät für Mathematik, Informatik und Statistik der Ludwig–Maximilians–Universität München vorgelegt von Andreas Pleuß geboren in Würzburg München, den 25. Februar 2009 Erstgutachter: Prof. Dr. Heinrich Hußmann Zweitgutachter: Prof. Dr. Peter Forbrig (Universität Rostock) Prof. Dr. Jean Vanderdonckt (Université Catholique de Louvain, Belgien) Tag der mündlichen Prüfung: 27. Mai 2009 Contents Abstract xv 1 Introduction 1 2 Interactive Multimedia Applications 5 2.1 Basic Terms and Definitions .............................. 5 2.1.1 Multimedia ................................... 5 2.1.2 Multimedia vs. Multimodality ......................... 7 2.1.3 Multimedia Applications ............................ 7 2.1.4 Multimedia Applications vs. Web and Hypermedia Applications ....... 8 2.2 Multimedia Application Development ......................... 9 2.3 Implementation Technologies and Tools ........................ 10 2.3.1 Frameworks and APIs ............................. 11 2.3.2 Declarative Languages ............................. 12 2.3.3 Authoring Tools ................................ 13 2.4 Classification ...................................... 19 2.4.1 Existing Classifications ............................. 19 2.4.2 A Classification for this Thesis ......................... 20 2.5 Conclusions for this Thesis ............................... 22 3 Problem Statement and Proposed Solution 25 3.1 Current Problems in Multimedia Application Development .............. 25 3.1.1 Interdisciplinary Roles ............................. 28 3.1.2 Authoring Tools ................................ 30 3.2 Analyzing the Spectrum of Possible Solutions ..................... 30 3.3 Proposed Solution: A Model-Driven Approach .................... 33 3.4 A Short Introduction into Model-Driven Engineering ................. 34 3.4.1 Models, Metamodels, and Modeling Languages ................ 34 3.4.2 Model-Driven Development .......................... 37 3.4.3 Practical Application and Profiles ....................... 40 3.5 An Illustrating Example Scenario ............................ 43 4 Related Modeling Approaches 47 4.1 User Interface Modeling ................................ 47 4.1.1 General Concepts ................................ 48 4.1.2 Concrete Examples ............................... 51 4.1.3 User Interface Modeling and Software Engineering Concepts . ...... 57 vi CONTENTS 4.2 Web Engineering .................................... 61 4.3 Multimedia Modelling ................................. 63 4.3.1 HyDev ..................................... 64 4.3.2 OMMMA .................................... 68 4.3.3 Code Generation ................................ 74 4.4 Discussion ........................................ 75 5 Modeling Multimedia – MML Core Principles 79 5.1 Basic Decisions and Language Engineering ...................... 79 5.1.1 Scope ...................................... 80 5.1.2 Language Definition .............................. 80 5.1.3 Notation .................................... 81 5.1.4 Custom Icon Design .............................. 82 5.1.5 Basic Technical Metamodeling Principles ................... 84 5.2 Modeling Media Components ............................. 86 5.2.1 Rationale .................................... 86 5.2.2 Media Types and Media Components ..................... 87 5.2.3 Interfaces .................................... 89 5.2.4 Media Representations ............................. 91 5.2.5 Artifacts and Instances ............................. 92 5.2.6 Keywords for Media Representations ..................... 94 5.2.7 Media Parts ................................... 95 5.2.8 Inner Properties ................................. 98 5.2.9 Part Artifacts .................................. 100 5.2.10 Variations .................................... 101 6 MML – A Modeling Language for Interactive Multimedia Applications 103 6.1 MML – Task Model ................................... 103 6.2 MML – Structural Model ................................ 105 6.3 MML – Scene Model .................................. 107 6.4 MML – Presentation Model ............................... 113 6.4.1 Abstract User Interface ............................. 114 6.4.2 UI Realizations ................................. 118 6.4.3 Sensors ..................................... 120 6.5 MML – Interaction Model ............................... 123 6.5.1 Rationale .................................... 123 6.5.2 MML Interaction Model based on UML Activity Diagrams . ....... 126 6.5.3 Temporal Synchronization ........................... 129 6.6 Model Interrelations and Modeling Process ...................... 132 6.6.1 Modeling Process ................................ 132 6.6.2 Conceptual Interrelations ............................ 135 6.7 Modeling Tool Support ................................. 136 7 Integrating Creative Design: From MML to Flash 141 7.1 A Metamodel for Flash and ActionScript ........................ 141 7.1.1 Flash Metamodel Basics ............................ 142 7.1.2 Flash Documents ................................ 144 Contents vii 7.1.3 ActionScript .................................. 147 7.2 A General Structure for Flash Applications ...................... 152 7.2.1 Principles.................................... 152 7.2.2 Scenes ..................................... 153 7.2.3 Complete Structure ............................... 156 7.3 Transformations ..................................... 158 7.3.1 Round-Trip-Engineering ............................ 161 7.3.2 A Tool for Extracting Flash Models from Flash Applications . ....... 164 7.4 Working with the Code Skeletons in the Authoring Tool ................ 164 8 Validation 173 8.1 Platform Independence: Transformations to other Platforms .............. 174 8.1.1 Code Generation for Java ............................ 175 8.1.2 Code Generation for SVG/JavaScript ..................... 176 8.1.3 Code Generation for FlashLite ......................... 176 8.2 External Validation ................................... 178 8.2.1 MML in the Course “Multimedia-Programming” ............... 179 8.2.2 MML in Project Theses ............................ 183 8.2.3 Conclusions ................................... 187 8.3 Internal Validation ................................... 187 8.3.1 Comparison to General Criteria for Modeling Languages ........... 188 8.3.2 Comparison to the Goals of This Thesis .................... 191 8.3.3 Comparison to Other Approaches ....................... 193 9 Outlook: Towards Better Integration of Software Engineering and Creative Design 197 9.1 Creative Design in Software Development ....................... 197 9.2 Vision: Models as Central Hub ............................. 198 9.3 First Steps ........................................ 199 10 Summary and Conclusions 203 10.1 Summary ........................................ 203 10.2 Contributions ...................................... 204 10.3 Future Work ....................................... 204 10.4 Conclusions ....................................... 206 A Multimedia Taxonomy by Hannington and Reed 207 B ActionScript Class MovieClip – Documentation 209 B.1 Properties ........................................ 209 B.2 Properties Inherited from Class Object ......................... 211 B.3 Operations ........................................ 211 B.4 Event Handling Operations ............................... 214 C Transformation from MML to Flash Model 217 D Questionnaire 221 Bibliography 227 viii Contents List of Figures 1.1 Examples for multimedia applications today. (From left to right, starting in the top row: Rich Internet Application [Goo], E-Learning [Jungwirth and Stadler03], Enter- tainment [Hilliges et al.06], Entertainment on mobile devices [Tavares], Infotainment in cars [Inf], Instrumented Environments [Bra]) .................... 2 2.1 Technical media: Classes and examples based on [Hoogeveen97] and MHEG [ISO97b]. 6 2.2 Classification of media according to [Fetterman and Gupta93]. ............ 6 2.3 Multimedia applications in the value chain. ...................... 9 2.4 Structure in the Piccolo framework ........................... 12 2.5 MHEG-5 class hierarchy from [MHE]. ......................... 13 2.6 A screenshot of Adobe Authorware. .......................... 15 2.7 A screenshot of Adobe Director. ............................ 16 3.1 RepresentationOf relationship according to [Favre04c] ................ 35 3.2 ConformsTo relationship according to [Kurtev et al.06] ................ 35 3.3 The basic idea of MDA ................................. 37 3.4 OMG Metadata Architecture according to [Obj05] ................... 39 3.5 Technical Spaces according to [Kurtev et al.06] .................... 40 3.6 Notation options for UML stereotypes. ......................... 41 3.7 Screenshot from the example Racing Game application. ................ 44 3.8 Example: Wheels of a racing car defined as independent inner parts. ......... 45 3.9 The generated skeletons including placeholders can be directly opened and edited in the Flash authoring tool. ................................ 46 4.1 Model-based User Interface Development Environments according to [Szekely96, da Silva00] ....................................... 48 4.2