Towards an Interactive Framework for Upper Airway Modeling Integration of Acoustic, Biomechanic, and Parametric Modeling Methods by Florian Vogt Dipl. Ing., Hamburg University of Applied Science, 1998 MSc, Stanford University, 2000 A THESIS SUBMITTED IN PARTIAL FULFILMENT OF THE REQUIREMENTS FOR THE DEGREE OF Doctor of Philosophy in The Faculty of Graduate Studies (Electrical and Computer Engineering) The University Of British Columbia (Vancouver) April 2009 c Florian Vogt 2009 Abstract The human upper airway anatomy consists of the jaw, tongue, pharynx, larynx, palate, nasal cavities, nostrils, lips, and adjacent facial structures. It plays a central role in speaking, mastication, breathing, and swallowing. The interplay and correlated movements between all the anatomical structures are complex and basic physiological functions, such as the muscle activation patterns associated with chewing or swallowing, are not well understood. This work creates a modeling framework as a bridge between such disciplines as linguistics, dentistry, biomechanics, and acoustics to enable the integration of physiological knowledge with interactive simulation methods. This interactive model of the upper airway system allows better understanding of the anatomical structures and their overall function. A three-dimensional computational modeling framework is proposed to mimic the behavior of the upper airway anatomy as a system by combining biomechanic, parametric, and acoustic modeling methods. Graphical user interface components enable the interactive manipulation of models and orchestration of physiological functions. A three-dimensional biomechanical tongue model is modied as a reference model of the modeling framework to demonstrate integration of an existing model and to enable interactivity and validation procedures. Interactivity was achieved by introducing a general-purpose fast linear nite element muscle model. Feasible behavior of the biomechanical tongue model is ensured by comparison with a reference model and matching the model to medical image data. In addition to the presented generic tongue model, individual dierences in shape and function are important for clinical applications. Dierent medical image modalities may jointly enable guidance of the creation on individuals' anatomy models. Automatic methods to extract shape and function are investigated to demonstrate the feasibility of upper airway image-based modeling for this modeling framework. This work may be continued in many other directions to simulate the upper airway for speaking, breathing, and swallowing. For example, progress has already been made to develop a complete vocal tract model whereby the tongue model, jaw model, and acoustic airway are connected. ii Table of Contents Abstract .............................................. ii Table of Contents ........................................ iii List of Tables ........................................... vii List of Figures .......................................... viii Acknowledgments ........................................ x Dedication ............................................. xi Co-Authorship Statement ................................... xii Chapter 1: Introduction .................................... 1 1.1 Motivation . 1 1.2 The Approach . 3 1.3 Contributions and Impact . 5 1.4 Demarcation of the Work . 7 1.5 Thesis Structure . 8 Chapter 2: Background and Related Work ........................ 9 2.1 Physiology of the Upper Airway . 10 2.1.1 Anatomical Structures . 10 2.1.2 Air Cavities . 15 2.1.3 Materials and Couplings . 15 2.2 Principles of Sound Production and Speech Synthesis . 16 2.2.1 Summary . 21 2.3 Underlying Methods for Articulatory Synthesis . 22 2.3.1 Biomechanical Techniques . 22 iii 2.3.2 Measurement-based Techniques . 27 2.3.3 Sound Production Techniques . 28 2.3.4 Modeling Vocal Tracts and Faces . 30 2.4 Imaging and Tracking Background . 34 2.4.1 Comparison of Image Modalities . 34 2.5 Image Extraction Techniques . 37 2.5.1 Image Registration . 37 2.5.2 Image Segmentation . 41 2.5.3 Mesh-based Processing . 42 2.5.4 Summary . 43 2.6 Existing Modeling Solutions . 43 2.6.1 Framework A: Ptolemy . 43 2.6.2 Framework B: Real ow . 44 2.6.3 Framework C: ANSYS and Fluent . 44 2.6.4 Framework D: Software for Interactive Musculoskeletal Modeling (SIMM) . 44 2.6.5 Framework E: Simulation Open Framework Architecture (SOFA) . 44 2.6.6 Discussion . 45 2.7 Summary . 45 Chapter 3: Creation of a Modeling Framework for the Vocal Apparatus . 46 3.1 Framework Requirements . 47 3.1.1 Methodology . 49 3.1.2 Non-functional Requirements . 50 3.1.3 Functional Requirements . 52 3.1.4 Scenarios . 55 3.2 Vocal Tract Simulation Framework . 58 3.2.1 Graphical User interface Design for Model and Simulation Editing and Control 60 3.2.2 Model Components . 62 3.2.3 Validation, Experimentation, and Control Modules . 72 3.2.4 Library of Anatomy Components Default Model . 72 3.2.5 Model Building from Images . 73 3.3 Matching Requirements with Realizations . 74 3.4 Summary . 75 iv Chapter 4: Creation of a Tongue Model for the Complete Vocal Tract . 77 4.1 Building Deformable Anatomical Models . 78 4.1.1 Ecient Anatomical Tongue Model . 79 4.1.2 Other Upper Airway Anatomy . 84 4.2 Validating Deformable Anatomical Models . 87 4.2.1 Comparison with Reference Simulation . 87 4.2.2 Matching Simulation Results to Measurement . 91 4.3 Conclusion and Future Work . 93 Chapter 5: Data Acquisition and Extraction ....................... 95 5.1 Creation and Structure of Vocal Tract Data Sets . 96 5.1.1 Important Existing Data Sets for Vocal Tract Modeling . 96 5.2 Extraction of the Tongue Shapes from MRI . 98 5.2.1 Segmentation Methods . 98 5.2.2 Experiments . 102 5.2.3 MR Segmentation Results . 105 5.2.4 Discussion of MR Segmentation Experiments . 107 5.2.5 Summary of MR Image Segmentation . 108 5.3 Registration of Tongue Shapes Across MRI Images . 109 5.3.1 MR Image Registration Experiments . 111 5.3.2 High Level Designs . 115 5.3.3 Discussion of MR Registration Experiments . 116 5.3.4 Summary of MR Registration Experiments . 118 5.4 Real Time Ultrasound Tongue Tracking . 119 5.4.1 System Design . 120 5.4.2 Tracking Algorithms . 121 5.4.3 Experiment: Vowel Analysis . 123 5.4.4 Experiment: Driving Physics Synthesis Models . 126 5.4.5 Discussion of Tongue and Groove Results . 128 5.4.6 Summary of Tongue and Groove . 129 5.5 Discussion of Data Acquisition and Extraction Results . 129 Chapter 6: Summary & Conclusion ............................ 131 6.1 Summary . 131 v 6.1.1 Modeling Framework . 132 6.1.2 Interactive Biomechanic Tongue Model . 132 6.1.3 Image Modeling . 133 6.1.4 Validation Process . 133 6.2 Contributions and Impact . 133 6.3 Conclusion . 134 6.4 Epilogue . 135 Bibliography ........................................... 136 Appendix A: Authors Publications ............................. 161 Appendix B: Researcher Feedback ............................. 164 B.1 Questionnaire . 165 vi List of Tables 2.1 List of Tongue Muscles . 14 2.2 Comparison of existing speech synthesis . 17 2.3 Finite element formulation . 26 2.4 Summary of Data Types . 36 2.5 List of Simulation Frameworks . 45 3.1 Summary of point-based connection types . 70 3.2 Realization of Nonfunctional Requirements . 75 3.3 Realization of Functional Requirements . 75 4.1 Finite element tongue accuracy results . 89 5.1 Segmentation Parameter . 106 5.2 Segmentation Experiment Summary . 107 5.3 Parameters for FEM based Registration . 112 5.4 Demons Registration Parameters . 112 5.5 Principle Component Analysis of Tongue Tracking Results . 126 B.1 Researcher Feedback . ..