A Study on Reusing Resources of Speech Synthesis for Closely-Related Languages
Total Page:16
File Type:pdf, Size:1020Kb
A Study on Reusing Resources of Speech Synthesis for Closely-Related Languages by Nur Hana Samsudin A thesis submitted to The University of Birmingham for the degree of Doctor of Philosophy School of Computer Science College of Engineering and Physical Sciences The University of Birmingham October 2017 University of Birmingham Research Archive e-theses repository This unpublished thesis/dissertation is copyright of the author and/or third parties. The intellectual property rights of the author or third parties in respect of this work are as defined by The Copyright Designs and Patents Act 1988 or as modified by any successor legislation. Any use made of information contained in this thesis/dissertation must be in accordance with that legislation and must be properly acknowledged. Further distribution or reproduction in any format is prohibited without the permission of the copyright holder. “It does not matter how slowly you go as long as you do not stop." Confucius Thank you, mom Abstract A Study on Reusing Resources of Speech Synthesis for Closely-Related Languages by Nur Hana Samsudin This thesis describes research on building a text-to-speech (TTS) framework that can accommodate the lack of linguistic information of under-resource languages by using existing resources from another language. It describes the adaptation process required when such limited resource is used. The main natural languages involved in this research are Malay and Iban language. Malay represents a language with sufficient speech re- sources while Iban language represents a language with very limited resources. Overall thesis revolves around the two languages. The thesis includes a study on grapheme to phoneme mapping and the substitution of phonemes. A set of substitution matrices will be presented which show the phoneme confusion in term of perception among respondents. The experiments conducted study the intelligibility as well as perception based on context of utterances. The study on the phonetic prosody is then presented and compared to the Klatt duration model. This is to find the similarities of cross language duration model if one exists. Then a comparative study of Iban native speaker with an Iban polyglot TTS (with Malay as focal language) is presented. This is to confirm that the prosody, suprasegmental or the rhythm of Malay can be used to generate Iban synthesised speech. The thesis concludes with the description of Iban polyglot TTS criteria with very mini- mal data using a very closely related language: Malay, as the main resource. The study is concluded with the respondents ratings and feedback. The central hypothesis of this thesis is that by using a closely-related language resource, a natural sounding speech can be produced. The aim of this research was to shaow that by sticking to the indigenous language characteristics, it is possible to build a polyglot synthesised speech system even with insufficient speech resources. Acknowledgements Thank you for my dear supervisor, Mark Lee who has the disadvantage of having a PhD student like me and for sticking together until the end. I am truly grateful for the help he has given me as a PhD supervisor as well as a reliable friend to a foreign land without any clue as to how to survive. Thank you for your patience with my ‘broken English writing skills’. Thank you for not getting angry with me when being kept pushing and asking. Thank you for trying to your best capabilities to help me to come out with this revised thesis. Thank you so very much. This thesis would also not have been possible without the tremendous help from Dr Tan Tien Ping of the School of Computer Sciences, Universiti Sains Malaysia, USM. I am forever grateful for his patience, his guidance, his generous data and equipments sharing as well as his powerful technical skills when I needed it the most. Thank you to the University of Birmingham specifically the School of Computer Science for providing the necessities, the equipment and the tremendous support throughout and post me being there. This thesis would be too detached if not for the wonderful help I received from the whole Sarawak Language and Technology research group (SaLT) of Universiti Malaysia Sarawak, UNIMAS, especially to Sarah Samson Juan and Suhaila Saee. The team compassion and kindness as well as the blooming friendship will be forever cherished. I am genuinely grateful to the faceless warriors of open academic resources which make searching for knowledge less painful even when I was put in a position to be outside of academic environment. Thank you for my sponsor, the Ministry of Higher Education and the people of Malaysia. I am deeply thankful and grateful to be given the privilege. Special thank you to the ex-Dean, of the school of Computer Science, Prof Rosni Abdullah for being the boss, a friend and a supporter, despite having her hands full at all time. Thank you to my beloved friends, in Birmingham, USM and those from neither, for being superbly supportive, for the compassion, for the unconditional friendship, for being the shoulders to cry on, for the constructive idea and for everything you all do. Really. Thank you. Thank you. Thank you. Life is less suffocating because of you all. I sincerely, full-heartedly would like to thank my siblings for trying their best not to involve me too much with the crucial events as well as the never ending drama in the extended family after the passing of two prominent figures in our family. iii This thesis will not be as what it is now if not for the constructive ideas, comments and tremendous guidance by both of my examiners. I am not exaggerate to say this thesis is not even half the quality it is now without them. Thank you to my internal examiner: Dr Peter Hancox and my external examiner: Prof Dr Nick Campbell for the inspiring and motivating hours in the viva room. I will forever be grateful for all their advice. And I sincerely apologise if I missed anyone else. But thank you, is all I can say. Thank you very much. Contents Abstract ii Acknowledgements iii List of Figures ix List of Tables xi Abbreviations xiv 1 Introduction 1 1.1 Short Introduction to Speech Synthesis . .2 1.1.1 History of Multilingual and Polyglot Speech Synthesis . .3 1.2 Research Motivations and Objectives . .5 1.3 Thesis Statement . .6 1.4 Terms Used . .6 1.5 Thesis Organisation . .9 1.6 Summary . 10 2 Language and Melody in Speech 11 2.1 Language Characterisation . 11 2.1.1 Language Classification and Characteristics . 12 2.2 Melody in Speech . 14 2.2.1 Involuntary Aspects of Speech . 16 2.2.1.1 Speaker . 16 2.2.1.2 Manner and Place of articulations (and suprasegmental effects) . 17 2.2.1.3 Context and Emphasis . 17 2.2.2 Perceptual Equality . 18 2.2.3 Perceptual Equivalence . 18 2.2.4 Perceptual Closeness . 19 2.3 Introduction to the Malay Language . 19 2.3.1 The Writing System . 20 2.3.2 Phonemes Variations . 21 2.3.3 Prosody . 22 2.4 Introduction to the Iban Language . 24 2.4.1 The Writing System . 25 v Contents vi 2.5 Malay vs Iban Language Features . 25 2.6 Malay Intonation Pattern in a Sentence . 27 2.7 Summary . 30 3 Multilingual and Polyglot Speech Synthesis Review 31 3.1 Multilingual and Polyglot Speech Synthesis Research and Commercial Product . 31 3.1.1 CHATR . 32 3.1.2 AT&T Bell Labs TTS . 32 3.1.3 CLUSTERGEN . 33 3.1.4 Festival . 34 3.1.5 Verbmobil . 35 3.1.6 Loquendo Text-to-Speech . 35 3.2 Different Approaches in Multilingual and Polyglot Speech Synthesis . 36 3.2.1 Multilingual Speech Synthesis . 36 3.2.2 Polyglot Speech Synthesis . 38 3.3 Literature Review on NLP Manipulation of Multilingual Processing . 39 3.3.1 Text Pre-Processing and Analysis . 39 3.3.2 Phonological Processing in Multilingual TTS . 42 3.3.3 Prosody Modelling . 43 3.3.3.1 Spanish Speech Synthesis using CBR as Prosody Estimator 43 3.4 Rapid Prototyping TTS . 45 3.5 Summary . 47 4 Phoneme Substitution in Letter-to-Phone Processing for Combina- tional Speech Resources 49 4.1 Adapting Phoneme Resources from Resource Languages . 49 4.2 Phoneme Confusion . 50 4.2.1 Studies on Phoneme Confusions . 51 4.2.2 The Study on Phoneme Confusion for Malay . 53 4.2.2.1 Phoneme Confusion Matrix for Consonants in syllable CV 55 4.2.2.2 Phoneme Confusion Matrix for Consonants in syllable VC 58 4.2.2.3 Phonemes Confusion Matrix for Onset in syllable CVC . 62 4.2.2.4 Phonemes Confusion Matrix for Coda in syllable CVC . 63 4.3 Phoneme Substitution . 67 4.3.1 Intelligibility on Substituted Phoneme’s Words . 68 4.3.2 Perception based on Context . 69 4.3.2.1 Onset Evaluation . 69 4.3.2.2 Coda Evaluation . 70 4.4 Summary . 72 5 Prosody Processing for Combinational Speech Synthesis 73 5.1 Study on the Malay Phoneme . 73 5.1.1 About the Data . 74 5.1.2 Klatt Duration Model and Malay Duration Analysis . 76 5.1.2.1 Klatt Duration Model . 77 5.1.2.2 Klatt Segment Duration and Malay Segment Duration Analysis Comparison . 79 Contents vii 5.2 Comparing Malay and Iban Speech Contour . 81 5.3 Study of Similarity . 83 5.3.1 Prosody Comparison Study . 84 5.3.2 Constructing Iban TTS Speech without Speech Data . 86 5.3.2.1 TTS Data and the experiment . 87 5.3.2.2 Measure of Accuracy .