VILNIUS GEDIMINAS TECHNICAL UNIVERSITY Ingrida MAŽONAVIČIŪTĖ LITHUANIAN SPEECH ANIMATION TECHNOLOGY FOR 3D FACIAL MODEL DOCTORAL DISSERTATION TECHNOLOGICAL SCIENCES, INFORMATICS ENGINEERING (07T) Vilnius 2012 Doctoral dissertation was prepared at Vilnius Gediminas Technical University in 2008–2012. Scientific Supervisor Prof Dr Habil Romualdas BAUŠYS (Vilnius Gediminas Technical University, Technological Sciences, Informatics Engineering – 07T). VGTU leidyklos TECHNIKA d080-M mokslo literatūros knyga http://leidykla.vgtu.lt ISBN © VGTU leidykla TECHNIKA, 2012 © Ingrida Mažonavičiūtė, 2012 [email protected] VILNIAUS GEDIMINO TECHNIKOS UNIVERSITETAS Ingrida MAŽONAVIČIŪTĖ LIETUVIŲ KALBOS ANIMAVIMO TECHNOLOGIJA TAIKANT TRIMATĮ VEIDO MODELĮ DAKTARO DISERTACIJA TECHNOLOGIJOS MOKSLAI, INFORMATIKOS INŽINERIJA (07T) Vilnius 2012 Disertacija rengta 2008–2012 metais Vilniaus Gedimino technikos universitete. Mokslinis vadovas prof. habil. dr. Romualdas BAUŠYS (Vilniaus Gedimino technikos universi- tetas, technologijos mokslai, informatikos inžinerija – 07T). Abstract Speech animation is widely used in technical devices to allow the growing num- ber of hearing impaired persons, children, middle-aged and elderly equal partici- pation in communication. Accuracy of the speech recognition or generation en- gine, modelled visemes, phonemes to visemes mapping and coarticulation control model mainly influence speech animation quality, so phonetics of specif- ic language must be analysed to create new „Talking head” (synthetic head model that can produce visual speech). Framework suitable to visualize Lithua- nian language, which includes two new models to improve intelligibility of ani- mated speech is proposed and applied to create Lithuanian „Talking head” „LIT”. The dissertation consists of Introduction, three main chapters and general conclusions. Chapter 1 provides the analysis of the existing speech animation technolo- gies. Different facial modelling techniques are examined to define the most suit- able 3D „Talking head” modelling technique. Viseme classification experiments across different languages are analysed to identify variety of viseme identifica- tion and classification methods. Different coarticulation control models are compared to decide which one should be used to define coarticulation of Lithua- nian speech. Chapter 2 describes theoretical framework for Lithuanian speech animation. Translingual visual speech synthesis methods, specifics of Lithuanian phonetics and phonemes recognition system which is included in the proposed framework are analysed. Chapter 3 presents the proposed Lithuanian speech animation technology and two new models to improve intelligibility of animated Lithuanian speech. The new translingual viseme selection model divides viseme classes of any lan- guage into two parts: constant (translingual) and variable. Technology to create constant (translingual) viseme set for 22 most popular consonants is proposed. Coarticulation control model specified for pure Lithuanian diphthongs is also proposed. The chapter is concluded with practical evaluation of the Lithuanian „Talking head” „LIT”. General conclusions summarise the present study. It is followed by an ex- tensive list of 144 references and a list of 3 publications by the author on the topic of the dissertation. v Reziumė Kalbos animacija naudojama technikos įrenginiuose norint sudaryti vienodas bendravimo galimybes kurtiesiems, vaikams, vidutinio ir vyresnio amžiaus žmonėms. Tai yra sudėtingas procesas, kurio metu žmogaus kalboje atpažinta akustinė informacija (fonemos) yra vizualizuojama naudojant specialiai sumodeliuotas veido išraiškas vadinamas vizemomis. Kuriant naujus kalbai animuoti skirtus modelius „Kalbančias galvas“ būtina atsižvelgti į analizuojamos kalbos fonetikos ypatumus. Kalbos animacijos tikroviškumui didžiausią įtaką turi tinkamas vizemų identifikavimas, modeliavimas ir pasirinktas koartikuliacijos valdymo modelis, naudojamas vizemoms laiko juostoje išrikiuoti. Karkasas, kuriame naudojama Lietuvių kalbai skirta animavimo technologija pasiūlytas ir realizuotas lietuviškai „Kalbančiai galvai“ „LIT“ sukurti. Darbą sudaro įvadas, trys pagrindiniai skyriai, bendrosios išvados, literatūros ir publikacijų sąrašas. Pirmame skyriuje analizuojamos esamos kalbos animavimo technologijos, tiriami skirtingose kalbose naudojami metodai vizemoms identifikuoti ir klasifikuoti, lyginami 4 pagrindiniai kalbos koartikuliacijos valdymo modeliai. Antrajame skyriuje pristatomas karkasas, skirtas lietuvių kalbai animuoti. Išanalizavus esamus sprendimus, skyriuje siūlomas teorinis modelis, kaip atviro kodo anglų kalbos animavimo sistema gali būti panaudota lietuvių kalbos animavimo technologijai realizuoti. Trečiame skyriuje detaliai aprašomi du nauji modeliai, kurie naudojami lietuvių kalbos animavimo technologijai sukurti ir lietuvių kalbos animacijos suprantamumui ir tikroviškumui padidinti. Tarpkalbinis vizemų parinkimo modelis yra naudojamas lietuvių kalbos vizemoms skirstyti į pastoviąją (tarptautinę) ir kintamąją dalis. Atsižvelgiant į lietuvių kalbos fonetikos taisykles pasiūlomas naujas koartikuliacijos valdymo modelis lietuvių kalbos dvibalsiams animuoti. Lietuviškai „Kalbančios galvos“ „LIT“ kokybė įvertinama praktiniais eksperimentais. Darbas baigiamas išvadomis apie tyrimą. Disertacijos tema paskelbti trys straipsniai. vi Notations Abbrevations 2D – two dimensions; 3D – three dimensions; FACS – Facial action system; FAP – Facial animation parameter; FAU – Facial action unit; FDP – Facial definition parameter; HMM – Hidden Markov Model; HTK – Hidden Markov Model toolkit; IPA – International Phonetic Alphabet; MPEG-4 – Moving Picture Experts Group, 4 standard; TTS – Text-To-Speech; VCV – Vowel-Consonant-Vowel; CVC – Consonant-Vowel-Consonant. vii Contents INTRODUCTION ........................................................................................................... 1 Statement of the problem............................................................................................. 1 The importance of the thesis ........................................................................................ 2 The objective of the thesis ........................................................................................... 3 The goal of the thesis ................................................................................................... 3 The tasks of the thesis ................................................................................................. 3 Research methodology ................................................................................................ 4 Scientific novelty ......................................................................................................... 4 Practical significance of achieved results .................................................................... 4 The defended statements ............................................................................................. 5 Approval of the results ................................................................................................ 5 Dissertation structure ................................................................................................... 6 Acknowledgements ..................................................................................................... 6 ANALYSIS OF CURRENT SPEECH ANIMATION TECHNOLOGIES ..................... 7 1.1. General terms ........................................................................................................ 8 1.2. Audiovisual speech synthesis ............................................................................... 8 1.2.1. Audiovisual speech generation methods.................................................. 10 1.2.2. Speech processing for speech animation .................................................. 17 1.2.3. Coarticulation modelling for audiovisual speech ..................................... 21 1.2.4. Facial modelling for speech animation ..................................................... 24 1.3. Facial animation systems .................................................................................... 28 ix 1.3.1. Face parameterization and scripting languages ........................................ 28 1.3.2. Current non-commercial facial animation systems ................................... 30 1.4. Facial animation evaluation methods.................................................................. 31 1.4.1. Advantages and disadvantages of evaluation methods ............................. 31 1.4.2. Challenges in evaluation of different speech animation systems.............. 33 1.5. State of art in Lithuania ...................................................................................... 35 1.6. Conclusions on Chapter 1 ................................................................................... 38 FRAMEWORK FOR LITHUANIAN SPEECH ANIMATION ................................... 41 2.1. iFACE – speech animation system ..................................................................... 42 2.1.1. Geometry of the head model..................................................................... 43 2.1.2. Behavior modelling and Face Modelling Language ................................. 45 2.2. Automatic phonemes recognition for the proposed framework .......................... 46 2.2.1. Specifics of Lithuanian phonetics ............................................................. 46 2.2.2. Framework for Lithuanian