THE PERCEPTION AND PRODUCTION OF STRESS AND INTONATION BY CHILDREN WITH COCHLEAR IMPLANTS ROSEMARY O’HALPIN University College London Department of Phonetics & Linguistics A dissertation submitted to the University of London for the Degree of Doctor of Philosophy 2010 ii ABSTRACT Users of current cochlear implants have limited access to pitch information and hence to intonation in speech. This seems likely to have an important impact on prosodic perception. This thesis examines the perception and production of the prosody of stress in children with cochlear implants. The interdependence of perceptual cues to stress (pitch, timing and loudness) in English is well documented and each of these is considered in analyses of both perception and production. The subject group comprised 17 implanted (CI) children aged 5;7 to 16;11 and using ACE or SPEAK processing strategies. The aims are to establish (i) the extent to which stress and intonation are conveyed to CI children in synthesised bisyllables (BAba vs. baBA) involving controlled changes in F 0, duration and amplitude (Experiment I), and in natural speech involving compound vs. phrase stress and focus (Experiment II). (ii) when pitch cues are missing or are inaudible to the listeners, do other cues such as loudness or timing contribute to the perception of stress and intonation? (iii) whether CI subjects make appropriate use of F 0, duration and amplitude to convey linguistic focus in speech production (Experiment III). Results of Experiment I showed that seven of the subjects were unable to reliably hear pitch differences of 0.84 octaves. Most of the remaining subjects required a large (approx 0.5 octave) difference to reliably hear a pitch change. Performance of the CI children was poorer than that of a normal hearing group of children presented with an acoustic cochlear implant simulation. Some of the CI children who could not discriminate F 0 differences in Experiment I nevertheless scored above chance in tests involving focus in natural speech in Experiment II. Similarly, some CI subjects who were above chance in the production of appropriate F 0 contours in Experiment III could not hear F 0 differences of 0.84 octaves. These results suggest that CI children may not necessarily rely on F 0 cues to stress, and in the absence of F 0 or amplitude cues, duration may provide an alternative cue. iii ACKNOWLEDGEMENTS For guidance and direction I am indebted to my supervisor Dr Andrew Faulkner, who has been accessible and helpful with every aspect of my research, and who has given careful criticism of various drafts of the thesis. I am grateful to Professor Stuart Rosen for constructive comments at different stages of this dissertation; to my specialist adviser Ms Laura Viani, consultant ENT surgeon and director of the Cochlear Implant Programme at Beaumont Hospital, Dublin for her encouragement and support; and to Dr Evelyn Abberton for helpful suggestions at the early stages of this project. I am also grateful to the Health Research Board for a Health Services Research Fellowship which partly funded this research. My thanks are also due to Dr Yi Xu for providing a custom-written PRAAT script for F0 extraction and measurements, and for helpful discussions on prosodic issues; Dr Gary Norman for audiological and mapping details for the children with implants; Steve Nevard for setting up the audio recordings and Dave Cushing for technical assistance at UCL; Jill House for suggestions regarding intonation issues; Dr Michael O’Kelly for help with statistics and comments on a draft of the thesis; Professor Neil Smith and Professor Valerie Hazan for feedback on earlier drafts of some of the chapters. For arranging the use of soundproof facilities in their respective locations I am grateful to Anne Marie Gallagher and her colleagues at Beaumont Hospital, Dr Jesudas Dayalan (Clonmel), and Nick Devery and Bernie Lowry (Tullamore). I must also thank Michael Ashby, Dr Volker Dellwo, Dr Yu-ching Kuo, Anne Parker and Dr Celia Wolf, at UCL for their assistance; my colleagues in the Cochlear Implant Programme and at Beaumont Hospital for all their support; Mary and Billy Kelly, Sian Kelly, Dr. Kate O’Malley and Patrick O’Halpin for their assistance; Julia Boyle at Beaumont Hospital for arranging secondment to carry out this research; Dr Anne Cody and Patricia Cranley at the Health Research Board and Gemma Heath at the Royal College of Surgeons in Ireland for their assistance; Barry O’Halpin for creating iv pictures for the experiments; Nuala Scott and Patricia Vila for formatting the final drafts of the manuscript. I am very grateful to the children, their families, and the talkers who participated enthusiastically in this study, and to the visiting teachers of the children with cochlear implants who were very helpful. Finally, my thanks to my family, relatives and friends, especially to Patrick and Barry for their support and encouragement in the final stages of this project. v TABLE OF CONTENTS PAGE ABSTRACT ii ACKNOWLEDGEMENTS iii TABLE OF CONTENTS v LIST OF FIGURES xiii LIST OF TABLES xvi LIST OF APPENDICES xix CHAPTER ONE – BACKGROUND & REVIEW OF THE LITERATURE 1 1.1. Introduction 2 1.1.1 Limited previous research 4 1.1.2 The hypotheses and framework for the current study 5 1.2. Linguistic aspects of stress and intonation in English 8 1.2.1 The theoretical basis for auditory judgements of stress and intonation in the present study 10 1.3. Developmental issues in the perception and production of stress and intonation 12 1.3.1. The early years 12 1.3.1.1. Perception 12 1.3.1.2. Production 13 1.3.2. The school years 16 1.3.2.1 Perception 16 1.3.2.2 Production 19 1.3.2.3 Developmental issues relating to the production of stress and intonation by deaf children 20 1.3.2.4 The relationship between perception and production 22 1.4 The perceptual and physical correlates of stress 23 1.4.1 Acoustic cues to stress and intonation 23 1.4.2 How important is F 0 in the perception of stress and intonation? 25 1.4.3 Theoretical basis for acoustic analysis of the production data in the current study 27 1.4.4 Acoustic cues in the production in Southern Hiberno English 30 1.4.5 Acoustic cues to stress and intonation in the speech of normal hearing and deaf children 30 1.5 Representation of the correlates of pitch in the acoustic signal 34 vi 1.6 Coding of pitch and loudness in the inner ear: acoustic stimulation in normal hearing 34 1.7 Coding of pitch and loudness in cochlear implants: electrical stimulation 35 1.8 The perception and production of natural tone by children with cochlear implants 37 1.8.1 Perception 37 1.8.2 Production 41 1.8.3 The relationship between perception and production 43 1.9 Experiments with adult cochlear implant users 44 1.10 Cochlear implant simulations with normal hearing adults 47 1.11 Relevance of the literature to the present investigation 50 1.11.1 Higher order acquisition issues 50 1.11.2 Lower order issues 54 1.11.3 Acoustic cues to lexical stress in tone languages: what can we predict for English speaking implanted children from the results of experimental studies of pitch perception and production of Chinese tone? 59 1.11.4 Perception vs. production of tone, stress and intonation 61 1.11.5 Variables which might affect perception (Experiments I and II) and production (Experiment III) performance : stimulation rate, age at implant, duration of implant use 65 1.11.6 CI stimulation studies 69 1.11.7 Methodological considerations 69 1.11.8 The current study 70 CHAPTER TWO – EXPERIMENT 1: SENSITIVITY TO VARIATIONS IN F0, DURATION AND AMPLITUDE IN SYNTHESISED SPEECH SOUNDS 72 2.1 Introduction 73 2.2 Methods 73 2.2.1 Subjects 73 2.2.2 Stimuli 75 2.2.2.1 Syntheses 75 2.2.3 Details of testing 81 2.2.3.1 Adaptive threshold measurement 81 2.2.3.2 Procedure 81 2.3 Results 83 2.3.1 F0 difference thresholds 83 2.3.1.1 Cochlear implant 83 2.3.1.2 Normal hearing simulation condition 85 vii 2.3.1.3 Normal hearing unprocessed condition 85 2.3.1.4 Summary 85 2.3.2 Duration difference thresholds: CI group vs. simulation vs. unprocessed conditions for the NH group 86 2.3.2.1 Cochlear implant 86 2.3.2.2 Normal hearing simulation condition 87 2.3.2.3 Normal hearing unprocessed condition 88 2.3.2.4 Summary 88 2.3.3 Amplitude difference thresholds: CI group vs. simulated and unprocessed conditions for the NH group 88 2.3.3.1 Cochlear implant group 88 2.3.3.2 Normal hearing simulation condition 89 2.3.3.3 Normal hearing unprocessed condition 89 2.3.3.4 Summary 90 2.3.3.5 Learning effect 90 2.3.4 Correlations between F 0 duration and amplitude thresholds 90 2.3.4.1 CI subjects 90 2.3.4.2 NH subjects 92 2.4 Summary and discussion of the results 95 2.4.1 Fundamental frequency (F 0) 95 2.4.1.1 Comparisons between F 0 discrimination by CI group and by the NH group in the unprocessed condition 95 2.4.1.2 Implications of the results for the perception o5 prosodic contrasts 95 2.4.1.3 Are results different from previous findings in studies of implanted adults and children and why might this be? 96 2.4.1.4 Comparisons with the typical acoustic changes in natural speech: F 0 97 2.4.1.5 F0 discrimination by the NH in a CI simulation 97 2.4.2 Discrimination of duration and amplitude cues by NH and CI subjects 98 2.4.2.1 Duration 99 2.4.2.2 Amplitude 100 2.4.3 Were there any correlations between F 0 duration and amplitude thresholds for CI and NH subjects in a stimulation condition? 100 2.4.4 Did factors such as age, duration of implant use, practise, and stimulation rate affect performance in Experiment