Visual Speech Perception of Arabic Emphatics and Gutturals By Maha Saliba Foster B.A., American University of Beirut, l982 M.A., University of Colorado, 2012 A thesis submitted to the Faculty of the Graduate School of the University of Colorado in partial fulfillment of the requirement for the degree of Doctor of Philosophy Department of Linguistics Department of Cognitive Science 2016 i This thesis entitled: Visual Speech Perception of Arabic Emphatics and Gutturals Written by Maha Saliba Foster has been approved for the Departments of Linguistic and the Institute of Cognitive Science __________________________ Dr. Rebecca Scarborough, Chair _________________________ Dr. Eliana Calunga __________________________ Dr. Bhuvana Narasimhan __________________________ Dr. David Rood __________________________ Dr. Sarel Van Vuuren Date The final copy of this thesis has been examined by the signatories, and we find that both the content and the form meet acceptable presentation standards of scholarly work in the above mentioned discipline. IRB protocol # 14-0394 ii Foster Maha Saliba (Ph.D., Linguistics and Cognitive Science) Visual Speech Perception of Arabic Emphatics and Gutturals Thesis directed by Dr. Rebecca Scarborough ABSTRACT This investigation explores the potential effect on perception of speech visual cues associated with Arabic gutturals (AGs) and Arabic emphatics (AEs); AEs are pharyngealized phonemes characterized by a visually salient primary articulation but a rather invisible secondary articulation produced deep in the pharynx. The corpus consisted of 72 minimal pairs each containing two contrasting consonants of interest (COIs), an emphatic versus a non-emphatic, or a guttural paired with another guttural. In order to assess the potential effect that visual speech information in the lips, chin, cheeks, and neck has on the perception of the COIs, production data elicited from 4 native Lebanese speakers was captured on videos that were edited to allow perceivers to see only certain regions of the face. Fifty three Lebanese perceivers watched the muted movies each presented with a minimal pair containing the word uttered in the video, and selected in a forced identification task the word they thought they saw the speaker say. The speakers’ speech was analyzed to help explore what in their production informed correct identification of the COIs. Perceivers were above chance at correctly identifying AEs and AGs, though AEs were better perceived than AGs. In the emphatic category, the effect on perception of measurement differences between a word and its pair was submitted to automatic speech recognition. The machine learning models were generally successful at correctly classifying COIs as emphatic or non-emphatics across vowel contexts; the models were able to predict the probability of perceivers’ accuracy in identifying certain COIs produced by certain speakers; also, an overlap between the measurements selected by the computer and those selected by human perceivers was found. No difference in perception of AEs according to the part of the face that was visible was observed, suggesting that the lips, present in all of the videos, were most important for perception of emphasis. Conversely, in the perception of AGs, lips were not as informative and perceivers relied more on cheeks and chin. The presence of visible cues associated with the AEs, particularly in the lips, suggests that such visual cues might be informative for non-native learners as well, if they were trained to attend to them. iii Dedication I dedicate this dissertation to my beloved father. Dad, if you weren’t constricted by your aging body, you would be calling me to ask how you could help me get through this trying time of writing a dissertation. Remember how you used to wake me up with a glass of fresh squeezed orange juice when I stayed up late studying for my finals? I know you would be proud of me and how much I wish you could be here to show me once more your love and support. To you my daddy, I dedicate this work. iv Acknowledgements It has been a long and arduous path from start to finish having to deal with a long commute, a full time job, a family and a dissertation. Sitting day in and day out in my little corner, how many times have I cried out to my Lord and Savior Jesus Christ! It is only by putting my faith in Him that I was able to take the next step and then the next, towards the finish line that often seemed more like a mirage than a reality. Thank you, Jesus, for giving me the perseverance to go on, and for surrounding me with a multitude of people who were instrumental in making it possible for me to reach my goal. I apologize for any oversight, for my supporters and encouragers were many! Special thanks to my advisor Dr. Rebecca Scarborough whose guidance was invaluable, and to all four members of my committee, Dr. Bhuvana Narasimhan, Dr. David Rood, Dr. Eliana Colunga, and Dr. Sarel van Vuuren for being there for me and taking time out of their busy schedule to read my dissertation and provide me with valuable comments. I am also very fortunate to be married to a brilliant software engineer who put in countless hours writing code that allowed me to collect data online and capture all the answers. Without you Allan, I would still be formatting my data! That was really a labor of love that you offered me unconditionally. Thank you also to my four boys who witnessed me continuously sitting in the same corner surrounded with papers and with a computer glued to my lap. Writing a dissertation could take a toll on family relations, but I thank God for my loving and supportive family! The completion of this dissertation would not have been possible without the help of many people at the University of Denver. Thank you Dr. Mahoor for providing me with v all I needed to collect my data: a place to run subjects, the equipment, and the needed programs for fitting faces and capturing the relevant measurements; thank you Kris Douglass for all the hard work you put into fitting all the speakers’ faces and Ali Mollahosseini, for lending Kris and I your expertise. Thank you to the staff at the media center who put up with me for months as I was working on editing the videos and provided me with all the much needed technical support; most of all, I owe Dr. Cathy Durso an inexpressible debt of gratitude for the many, many hours she invested in helping me with my statistics and data analysis. I really can’t thank you enough Cathy, you went well over your call of duty! Lastly, none of this would have been possible without the participation of all the three speakers who volunteered their time, and all the perceivers who were willing to put up with a rather lengthy study. I am truly blessed to have had the pleasure to work with such a tremendous team where each member was irreplaceable and played an essential role in contributing to the success of this project. vi Table of Contents Abstract ................................................................................................................ iii Acknowledgement ................................................................................................. v Table of Contents ............................................................................................... vii List of Tables ......................................................................................................... x List of Figures ...................................................................................................... xii CHAPTER 1: Introduction .....................................................................................1 1 Background ...............................................................................................1 2 Arabic Emphatics and Gutturals ................................................................5 2.1 Arabic Emphatics ..................................................................................7 2.2 Arabic Gutturals ....................................................................................9 3 Audio-Visual Speech Perception in Arabic ................................................ 11 4 Goal of this Study ...................................................................................... 12 CHAPTER 2: Production of AEs and AGs ........................................................... 15 1 Preparation of Stimuli ...............................................................................15 2 Experiment 1: Production Data Collection ................................................ 18 3 Measurements .......................................................................................... 20 3.1 Procedure ............................................................................................ 20 3.2 AEs and NEs Arrow Plots .................................................................... 25 CHAPTER 3: Perception of Emphasis ................................................................ 35 1 Experiment 2: Perception Data ................................................................. 35 2 Perception Results ................................................................................... 39 2.1 Descriptive Statistics ............................................................................ 40 vii 2.2 Variables Interactions ........................................................................... 47 2.3 Models by Speaker ............................................................................... 62 CHAPTER 4: Production-Perception