Automatic Conversion of Natural Language to 3D Animation Minhua Ma B.A., M.A., M.Sc. Faculty of Engineering University of Ulster A thesis submitted in partial fulfillment of the requirements for the degree of Doctor of Philosophy July 2006 ii Table of Contents List of Figures vi List of Tables ix Acknowledgements x Abstract xi Abbreviations xii Note on access to contents xiv 1. INTRODUCTION......................................................................................................................... 1 1.1 Overview of language visualisation ....................................................................................... 2 1.1.1 Multimodal output............................................................................................................ 2 1.1.2 Animation......................................................................................................................... 3 1.1.3 Intelligent ......................................................................................................................... 4 1.2 Problems in language visualisation ........................................................................................ 4 1.3 Objectives of this research...................................................................................................... 5 1.4 Outline of this thesis............................................................................................................... 5 2. APPROACHES TO MULTIMODAL PROCESSING...............................................................8 2.1 Automatic text-to-graphics systems ........................................................................................8 2.1.1 CarSim ..............................................................................................................................9 2.1.2 WordsEye..........................................................................................................................9 2.1.3 Micons and CD-based language animation.....................................................................12 2.1.4 Spoken Image and SONAS.............................................................................................12 2.2 Embodied agents and virtual humans....................................................................................14 2.2.1 Virtual human standards .................................................................................................15 2.2.2 Virtual humans................................................................................................................19 2.2.3 BEAT and other interactive agents .................................................................................20 2.2.4 Divergence on agents’ behaviour production..................................................................21 2.2.5 Gandalf............................................................................................................................22 2.2.6 Humanoid Animation......................................................................................................23 2.3 Multimodal storytelling.........................................................................................................23 2.3.1 Interactive storytelling ....................................................................................................24 2.3.2 AESOPWORLD .............................................................................................................25 2.3.3 Oz....................................................................................................................................26 2.3.4 KidsRoom .......................................................................................................................27 2.3.5 Computer games..............................................................................................................28 2.3.6 Film-inspired computer animations ................................................................................29 2.3.7 Computer graphics films .................................................................................................31 2.3.8 Video-based computer animation generation..................................................................31 2.4 Summary of previous systems...............................................................................................32 2.5 Multimodal allocation ...........................................................................................................32 2.6 Non-speech audio ..................................................................................................................35 2.6.1 Auditory icons.................................................................................................................36 2.6.2 Earcons............................................................................................................................37 2.6.3 Sonification .....................................................................................................................37 2.6.4 Music synthesis ...............................................................................................................37 2.7 Mental imagery in cognitive science.....................................................................................39 iii 2.8 Summary ...............................................................................................................................40 3. NATURAL LANGUAGE SEMANTICS ...................................................................................41 3.1 Natural language semantic representations ...........................................................................41 3.1.1 Semantic networks ..........................................................................................................41 3.1.2 Conceptual Dependency theory and scripts ....................................................................41 3.1.3 Lexical Conceptual Structure (LCS)...............................................................................45 3.1.4 Event-logic truth conditions............................................................................................46 3.1.5 X-schemas and f-structs ..................................................................................................47 3.2 Multimodal semantic representations....................................................................................49 3.2.1 Frame representation and frame-based systems..............................................................49 3.2.2 XML representations.......................................................................................................50 3.2.3 Summary of knowledge representations .........................................................................52 3.3 Temporal relations.................................................................................................................52 3.3.1 Punctual events................................................................................................................55 3.3.2 Verb entailment and troponymy......................................................................................56 3.4 Computational lexicons.........................................................................................................56 3.4.1 WordNet..........................................................................................................................57 3.4.2 FrameNet.........................................................................................................................58 3.4.3 The LCS database and VerbNet......................................................................................59 3.4.4 Comparison of lexicons...................................................................................................61 3.4.5 Generative lexicon ..........................................................................................................62 3.5 Language ontology ................................................................................................................62 3.5.1 Top concepts ...................................................................................................................62 3.5.2 Ontological categories of nouns......................................................................................63 3.5.3 Ontological categories of verbs.......................................................................................65 3.6 Summary ...............................................................................................................................68 4. LEXICAL VISUAL SEMANTIC REPRESENTATION........................................................ 69 4.1 Multimodal representation.................................................................................................... 69 4.2 Ontological categories of concepts (conceptual “parts of speech”) ..................................... 70 4.3 Lexical Visual Semantic Representation (LVSR)................................................................ 71 4.3.1 Finer EVENT predicates ................................................................................................ 74 4.3.2 Under-specification and selection restrictions ............................................................... 76 4.4 Visual semantics of events ................................................................................................... 78 4.4.1