Title: Exploring the influence of suprasegmental features of speech on rater judgements of intelligibility Name: Thomas Michael Rogers This is a digitised version of a dissertation submitted to the University of Bedfordshire. It is available to view only. This item is subject to copyright. EXPLORING THE INFLUENCE OF SUPRASEGMENTAL FEATURES OF SPEECH ON RATER JUDGEMENTS OF INTELLIGIBILITY A thesis submitted to the University of Bedfordshire in partial fulfilment of the requirements for the degree of Doctor of Philosophy by Thomas Michael Rogers Centre for Research in English Language Learning and Assessment University of Bedfordshire January 2018 Declaration I, Thomas Rogers, declare that this thesis and the work presented in it are my own and have been generated by me as the result of my own original research. Exploring the Influence of Suprasegmental Features of Speech on Rater Judgements of Intelligibility I confirm that: 1. This work was done wholly or mainly while in candidature for a research degree at this University; 2. Where any part of this thesis has previously been submitted for a degree or any other qualification at this University or any other institution, this has been clearly stated; 3. Where I have cited the published work of others, this is always clearly attributed; 4. Where I have quoted from the work of others, the source is always given. With the exception of such quotations, this thesis is entirely my own work; 5. I have acknowledged all main sources of help; 6. Where the thesis is based on work done by myself jointly with others, I have made clear exactly what was done by others and what I have contributed myself; 7. None of this work has been published before submission. Signed: ………………………………. Date: ……………… iii Abstract The importance of suprasegmental features of speech to pronunciation proficiency is well known, yet limited research has been undertaken to identify how raters attend to suprasegmental features in the English-language speaking test encounter. Currently, such features appear to be underrepresented in language learning frameworks and are not always satisfactorily incorporated into the analytical rating scales that are used by major language testing organisations. This thesis explores the influence of lexical stress, rhythm and intonation on rater decision making in order to provide insight into their proper place in rating scales and frameworks. Data were collected from 30 raters, half of whom were experienced professional raters and half of whom lacked rater training and a background in language learning or teaching. The raters were initially asked to score 12 test taker performances using a 9-point intelligibility scale. The performances were taken from the long turn of Cambridge English Main Suite exams and were selected on the basis of the inclusion of a range of notable suprasegmental features. Following scoring, the raters took part in a stimulated recall procedure to report the features that influenced their decisions. The resulting scores were quantitatively analysed using many-facet Rasch measurement analysis. Transcriptions of the verbal reports were analysed using qualitative methods. Finally, an integrated analysis of the quantitative and qualitative data was undertaken to develop a series of suprasegmental rating scale descriptors. The results showed that experienced raters do appear to attend to specific suprasegmental features in a reliable way, and that their decisions have a great deal in common with the way non- experienced raters regard such features. This indicates that stress, rhythm, and intonation may be somewhat underrepresented on current speaking proficiency scales and frameworks. The study concludes with the presentation of a series of suprasegmental rating scale descriptors. v Acknowledgements Several people were involved in providing access to the test taker responses necessary to carry out this study. I would like to thank Jaime Dunlea, Fiona Barker, Ardeshir Geranpayeh, Bruce Howell, John Slaght, Sarah Brewer, and Veronica Benigo for their generosity during this project. I am also grateful to the panel of phoneticians who helped me make sense of the data: Yuni Kim, Richard Cauldwell, Linda Shockey, and Rachel Smith. Financial support was provided by Pearson, for which I am grateful. Of course this research would have been impossible without the participants who kindly donated their time. I am also indebted to Stefan O’Grady for discussion and comments on drafts of chapters, and Dan Frost and Jane Setter for helpful discussion in the early stages of the project. The staff at CRELLA have been a regular source of support throughout this project, primarily John Field who supervised my doctoral studies. I greatly appreciate the guidance and mentorship that he provided. My sincerest thanks also go to Fumiyo Nakatsuhara who provided insightful feedback at critical junctures. Finally, my most heartfelt gratitude goes to my parents, and to Cara and Masha for their loving support. This thesis reports on research using examination data provided by Cambridge English Language Assessment. vi Table of Contents Abstract....................................................................................................................................... v Acknowledgements ................................................................................................................ vi Table of Contents .................................................................................................................. vii List of Tables .......................................................................................................................... xi List of Figures ........................................................................................................................ xii Additional Material ................................................................................................................ xiii Abbreviations and Acronyms ................................................................................................ xiv The International Phonetic Alphabet ...................................................................................... xv Chapter 1: Introduction ............................................................................................................... 1 1.1 Rationale for the Study ..................................................................................................... 3 1.2 Setting for the Research ................................................................................................... 8 1.2.1 Current Practice in Pronunciation Assessment ........................................................... 8 1.2.2 The Context for this Research ...................................................................................12 1.3 Significance of the Study .................................................................................................13 1.4 Important Terms Defined .................................................................................................14 1.5 Outline of the Thesis ........................................................................................................15 Chapter 2: Literature Review .....................................................................................................21 2.1 Suprasegmental Features of Speech ...............................................................................22 2.1.1 Stress ........................................................................................................................23 2.1.2 Intonation ..................................................................................................................26 2.1.3 Rhythm .....................................................................................................................29 2.2 Defining Intelligibility ........................................................................................................32 2.3 Suprasegmental Correlates of Intelligibility ......................................................................34 2.3.1 Lexical Stress ............................................................................................................35 2.3.2 Intonation ..................................................................................................................36 2.3.3 Rhythm .....................................................................................................................42 2.3.4 Rater Awareness of Suprasegmental Features .........................................................44 2.4 Frameworks and Rating Scales .......................................................................................49 2.4.1 Current Frameworks .................................................................................................50 2.4.2 Rating Scale Development and Validation.................................................................55 2.4.3 Current Rating Scales ...............................................................................................60 2.5 Raters ..............................................................................................................................63 vii 2.5.1 Influence of Rater Experience ...................................................................................64 2.5.2 Linguistic Background ...............................................................................................67 2.6 Summary, Research Questions