DOCSLIB.ORG

Looking Beyond Scores a Study of Rater Orientations and Ratings of Speaking

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Looking Beyond Scores a Study of Rater Orientations and Ratings of Speaking Looking Beyond Scores A Study of Rater Orientations and Ratings of Speaking Linda Borger DEPARTMENT OF EDUCATION AND SPECIAL EDUCATION © LINDA BORGER, 2014 Licentiate thesis in Subject Matter Education at the Department of Education and Special Education, Faculty of Education, University of Gothenburg. The licentiate thesis is available for full text download at Gothenburg University Publications Electronic Archive (GUPEA): http://hdl.handle.net/2077/38158 This licentiate thesis has been carried out within the framework of the Graduate School in Foreign Language Education “De främmande språkens didaktik” (FRAM). The Graduate School, leading to a licentiate degree, is a collaboration between the universities of Gothenburg, Lund, Stockholm and Linnaeus University, and is funded by the Swedish Research Council (project number 729-2011-5277). Abstract Title: Looking Beyond Scores – A Study of Rater Orientations and Ratings of Speaking Author: Linda Borger Language: English with a Swedish summary GUPEA: http://hdl.handle.net/2077/38158 Keywords: Performance assessment, paired speaking test, rater orientations, rater variability, inter-rater reliability, The Common European Framework of Reference for Languages (CEFR), Swedish national tests of English The present study aims to examine rater behaviour and rater orientations across two groups of raters evaluating oral proficiency in a paired speaking test, part of a mandatory Swedish national test of English. Six authentic conversations were rated by (1) a group of Swedish teachers of English (n = 17), using national performance standards, and (2) a group of external raters (n = 14), using scales from the Common European Framework of Reference for Languages (CEFR), the latter to enable a tentative comparison between the Swedish foreign language syllabus for English and the CEFR. Raters provided scores and written comments regarding features of the performances that contributed to their judgement. Statistical analyses of the Swedish raters’ scores show reasonable degrees of variability and, in general, acceptable inter-rater reliabilities, albeit with obvious room for improvement. In addition, the CEFR raters judged the performances of the Swedish students to be, on average, at the intended levels of the test. Analyses of the written comments, using NVivo 10 software, show that raters took a wide array of features into account in their holistic rating decision, however with test-takers’ linguistic and pragmatic competences, and interaction strategies the most salient. Raters also seemed to heed the same features, indicating considerable agreement regarding the construct. Further, a tentative comparison of the written comments and scores shows that the raters noticed fairly similar features across proficiency levels but in some cases evaluated them differently. The findings of the present study have implications for the interpretation of oral test results, and they also provide information that may be useful in the development of tasks and guidelines for different types of oral language assessment in different educational settings. Table of contents ACKNOWLEDGEMENTS CHAPTER ONE: INTRODUCTION ............................................................................ 11 Background ........................................................................................................ 12 The Swedish context ................................................................................... 13 National tests of English ............................................................................. 14 The Common European Framework of Reference for Languages ....... 15 Aim and research questions ............................................................................. 17 CHAPTER TWO: CONCEPTUAL FRAMEWORK ....................................................... 19 Validity and reliability ........................................................................................ 19 Language assessment ......................................................................................... 21 Communicative language assessment ............................................................. 22 Communicative competence ...................................................................... 23 Challenges for communicative language testing ...................................... 28 Performance assessment ................................................................................... 29 Assessment of oral proficiency ........................................................................ 31 The nature of speaking ................................................................................ 32 Speaking test formats .................................................................................. 33 Singleton and paired speaking tests ........................................................... 34 Co-construction and interactional competence as a criterion ................ 35 CHAPTER THREE: PREVIOUS RESEARCH ON SECOND/FOREIGN LANGUAGE PERFORMANCE TESTS OF SPEAKING ...................................................................... 37 Speaking tests ..................................................................................................... 38 Inter-rater reliability ..................................................................................... 38 Rater orientations ......................................................................................... 40 Paired speaking tests ......................................................................................... 43 CHAPTER FOUR: MATERIAL AND METHOD .......................................................... 49 The speaking test ............................................................................................... 50 The test-takers .................................................................................................... 50 The Swedish raters............................................................................................. 51 Rating criteria for Swedish raters ............................................................... 51 The external CEFR raters ................................................................................. 52 Rating criteria for the external CEFR raters ............................................. 53 The rating scales ................................................................................................. 54 Data collection procedure ................................................................................ 55 Data analysis ....................................................................................................... 57 Analysis of quantitative data ....................................................................... 57 Analysis of qualitative data ......................................................................... 58 Use of computer-assisted qualitative data analysis software .................. 67 Methodological considerations ........................................................................ 67 Validity and reliability of the quantitative method ................................... 67 Validity and reliability of the qualitative method ..................................... 68 Closing remarks on validity and reliability ................................................ 69 Ethical concerns ................................................................................................ 70 Informed consent and confidentiality ....................................................... 70 CHAPTER FIVE: RESULTS ........................................................................................ 71 Descriptive statistics for Swedish raters ......................................................... 71 Inter-rater reliability of Swedish raters ............................................................ 76 Descriptive statistics for external CEFR raters .............................................. 78 Analyses of written rater comments ................................................................ 81 Comments per category .................................................................................... 86 Accuracy ........................................................................................................ 86 Coherence ..................................................................................................... 89 Fluency .......................................................................................................... 91 Intelligibility .................................................................................................. 93 Interaction ..................................................................................................... 95 Other ............................................................................................................. 99 Production strategies ................................................................................... 99 Range ........................................................................................................... 101 Sociolinguistic appropriateness ................................................................ 103 Task realisation ........................................................................................... 104 Comments coded as rater reflection ........................................................ 105 Comments coded as inter- or intra-candidate comparison .................. 108 Relationship between rater comments
Load more

Recommended publications
  • Family Language Policy in Bilingual Finnish and Swedish Families in Finland
    FAMILY LANGUAGE POLICY IN BILINGUAL FINNISH AND SWEDISH FAMILIES IN FINLAND Austin Huhta Master’s Thesis Applied Linguistics Department of Language and Communication Studies University of Jyväskylä Fall 2020 UNIVERSITY OF JYVÄSKYLÄ Faculty Department Humanities and Social Sciences Department of Language and Communication Studies Author Austin Huhta Title Family Language Policy in Bilingual Finnish and Swedish Families in Finland Subject Level Applied Language Studies Master’s Thesis Month and year Number of pages December 2020 30 Abstract In Finland families are only allowed to choose one language for their child to be the child’s L1 even if the family is bilingual. With both Finnish and Swedish being national languages of Finland this thesis looked into which language a family chose, why they chose it, and how they helped their child maintain it. Looking at their perspective on this can allow us to get further insight into family language policy in Finland. The research method used here is a case study, with semi-structured interviews for the data collection and interpretive phenomenological analysis for the data analysis. This thesis interviewed a bilingual family with a Finn and a Swedish Swede and their one child. It found that while their initial language choice was Swedish, that their family language policy was dynamic. Over time the child was switched from Swedish medium education to Finnish medium education; however, at home multiple family language policies worked together to help maintain his Swedish language skills. The findings demonstrated that the right combination of family language policies and more formal educational settings can work together to help children grow up to be bilingual even if the minority language is mainly used at home.
    [Show full text]
  • Pdf?Sequence=1
    Journal of Balkan and Black Sea Studies Year 4, Issue 6, June 2021, pp. 155-212. Bulgaria’s Secret Empire: An Ultimatum to North Macedonia Tomasz Kamusella Abstract: In the summer of 2019, 30 years after the end of communism in Europe, Bulgaria began to forward the way of Skopje numerous onerous demands as a salient precondition for opening European Union (EU) accession negotiation talks with North Macedonia. All of these demands are dictated by ethnolinguistic nationalism that underlies the Bulgarian national master narrative. On 9 October 2019, the Bulgarian government officially adopted these demands in the form of an ultimatum, bar the term. Sofia did not have to deploy this ultimatum to stop the talks, since earlier Paris temporarily blocked the pending accession negotiations with Albania and North Macedonia. It appears that in the face of economic and political problems at home, Bulgarian politicians tend to use nationalism (and populism) to divert citizens’ attention. The Bulgarian government seems to emulate the Kremlin’s policy of the “Russian World,” which on the basis of ethnolinguistic nationalism claims for Russia all the territories inhabited by Russian-speakers. After the fall of communism, a similar policy of “Bulgarian World” (Bılgarski sviat) has been pursued by Sofia from Moldova to North Macedonia and Albania, clamouring for recognizing all the Slavic-speakers in this wide area as members of Ph.D., Reader in Modern History, University of St Andrews, Scotland, ORCID ID: 0000-0003-3484-8352; e-mail: [email protected] Submitted: 21.12.2020, Accepted: 16.06.2021 TOMASZ KAMUSELLA the Bulgarian nation.
    [Show full text]
  • Language Perceptions and Practices in Multilingual Universities
    Language Perceptions and Practices in Multilingual Universities Edited by Maria Kuteeva Kathrin Kaufhold Niina Hynninen Language Perceptions and Practices in Multilingual Universities Maria Kuteeva Kathrin Kaufhold • Niina Hynninen Editors Language Perceptions and Practices in Multilingual Universities Editors Maria Kuteeva Kathrin Kaufhold Department of English Department of English Stockholm University Stockholm University Stockholm, Sweden Stockholm, Sweden Niina Hynninen Department of Languages University of Helsinki Helsinki, Finland ISBN 978-3-030-38754-9 ISBN 978-3-030-38755-6 (eBook) https://doi.org/10.1007/978-3-030-38755-6 © The Editor(s) (if applicable) and The Author(s), under exclusive licence to Springer Nature Switzerland AG 2020 This work is subject to copyright. All rights are solely and exclusively licensed by the Publisher, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission or information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now known or hereafter developed. The use of general descriptive names, registered names, trademarks, service marks, etc. in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use. The publisher, the authors and the editors are safe to assume that the advice and information in this book are believed to be true and accurate at the date of publication. Neither the publisher nor the authors or the editors give a warranty, expressed or implied, with respect to the material contained herein or for any errors or omissions that may have been made.
    [Show full text]
  • The Complex Construction of Multilingual Identity: a Comparative Study in Three European Contexts of L3+ Learning
    The Complex Construction of Multilingual Identity: A comparative study in three European contexts of L3+ learning Harper Staples Corpus Christi College Faculty of Education, University of Cambridge June 2020 This thesis is submitted for the degree of Doctor of Philosophy Declaration This thesis is the result of my own work and includes nothing which is the outcome of work done in collaboration except as declared in the preface and specified in the text. It is not substantially the same as any work that has already been submitted before for any degree or other qualification except as declared in the preface and specified in the text. It does not exceed the prescribed word limit for the Education Degree Committee. Harper Staples, 28/05/2020 2 Abstract The commonly-cited phrase that bi- and multilingualism are no longer exceptions but rather the rule in most parts of the world is often-cited. This plurilingual reality has profound effects on the daily lives of many, and thus also represents a common experience of a considerable proportion of today’s students in compulsory education. Despite this wide recognition, there remains still a paucity of studies that examine the impact that the acquisition of multiple languages has upon a learner’s sense of self, and moreover, studies that consider methods though which we might evaluate and model these constructions of complex self-concept. A greater understanding of such processes of identification has potential social and educational implications. This PhD study has explored the phenomenon of adolescent multilingual language learner identity development via a mixed-methods, comparative context approach at three school sites based in Finland, France & England.
    [Show full text]
  • Migrant and Refugee Integration in Stockholm
    MIGRANT AND REFUGEE INTEGRATION IN STOCKHOLM A SCOPING NOTE [Regional Development Series] Migrant and Refugee Integration in Stockholm A Scoping Note About CFE The OECD Centre for Entrepreneurship, SMEs, Regions and Cities provides comparative statistics, analysis and capacity building for local and national actors to work together to unleash the potential of entrepreneurs and small and medium-sized enterprises, promote inclusive and sustainable regions and cities, boost local job creation, and support sound tourism policies. www.oecd.org/cfe/｜@OECD_local © OECD 2019 This paper is published under the responsibility of the Secretary-General of the OECD. The opinions expressed and the arguments employed herein do not necessarily reflect the official views of OECD member countries. This document, as well as any statistical data and map included herein, are without prejudice to the status of or sovereignty over any territory, to the delimitation of international frontiers and boundaries and to the name of any territory, city or area. │ 3 Table of contents Executive Summary .............................................................................................................................. 5 Acknowledgements ................................................................................................................................ 7 Foreword ................................................................................................................................................ 8 Key data.................................................................................................................................................
    [Show full text]
  • Beyond Philology
    BEYOND PHILOLOGY AN INTERNATIONAL JOURNAL OF LINGUISTICS, LITERARY STUDIES AND ENGLISH LANGUAGE TEACHING 15/1 WYDAWNICTWO UNIWERSYTETU GDAŃSKIEGO GDAŃSK 2018 EDITOR-IN-CHIEF Danuta Stanulewicz SECTION EDITORS Olga Sokołowska (Linguistics) Ludmiła Gruszewska-Blaim (Literary Studies, Culture) Wojciech Kubiński (Translation) Magdalena Wawrzyniak-Śliwska (Language Acquisition, Academic Teaching, Education) Jadwiga Węgrodzka (Reviews, Reports, Interviews) ASSISTANT EDITORS Maria Fengler Marcin Fortuna Michał Golubiewski Ewelina Gutowska-Kozielska Karolina Janczukowicz Joanna Redzimska Małgorzata Smentek PROOFREADERS Martin Blaszk Sarah Flamminio Jean Ward Tadeusz Z. Wolański The Editors and Proofreaders are all affiliated with the Institute of English and American Studies, University of Gdańsk, Poland. COVER DESIGN Andrzej Taranek ISSN 1732-1220 eISSN 2451-1498 © Copyright by Uniwersytet Gdański Wydawnictwo Uniwersytetu Gdańskiego Contact address Institute of English and American Studies University of Gdańsk ul. Wita Stwosza 51 80-308 Gdańsk Poland Phone: (+48) 58 523 30 49, (+48) 58 523 30 50 Email: [email protected] ASSOCIATE EDITORIAL BOARD Marta Bogdanowicz (University of Gdańsk, Poland) Joanna Burzyńska-Sylwestrzak (Uczelnia Lingwistyczno-Techniczna, Świecie, Poland) Ewa Dąbrowska (Northumbria University, Newcastle, U.K.) Desmond Graham (University of Newcastle, U.K.) Zoltán Kövecses (Eötvös Loránd University, Budapest, Hungary) Ronald W. Langacker (University of California at San Diego, U.S.A.) Barbara Lewandowska-Tomaszczyk
    [Show full text]
  • Can a True Finn Speak Swedish? the Swedish Language in the Finns Party Discourse
    Journal of Autonomy and Security Studies Vol. 3 Issue 1 Can a True Finn Speak Swedish? The Swedish Language in the Finns Party Discourse Hasan Akintug ´ Journal of Autonomy and Security Studies, 3(1) 2019, 28–39 URL: http://jass.ax/volume-3-issue-1-Akintug/ Abstract This paper aims to analyse the rhetorical utilisation of Swedish language in the discourse of the Finns Party. This contribution will provide an overview of the history of Swedish language in Finland and will attempt to analyse the relationship between the two language groups. This contribution will analyse the rhetorical uses of Swedish language within the discourse of the Finns Party with the intention to highlight the consistently negative portrayals of Swedish language and its status. It will be argued that the historical experience of the inequality of status between the Swedish and Finnish languages have been politicised by the Finns Party as a part of its ethno-nationalist and populist conceptualisation of the Finnish identity. Keywords Rhetoric, Nationalism, Populism, Identity, Discourse About the author: Hasan Akintug holds a Bachelor’s Degree in Political Science and Public Administration from Hacettepe University and is currently a student at the University of Helsinki, European and Nordic Studies Master’s program. He has worked as a research assistant at IKME Sociopolitical Studies Institute, the Åland Islands Peace Institute and the Centre for European Studies at the University of Helsinki. His main research interests include minorities and autonomous regions. 28 Journal of Autonomy and Security Studies Vol. 3 Issue 1 1. Introduction The Swedish language has been one of the eternal issues of Finnish politics.
    [Show full text]
  • UNIVERSITY of CALIFORNIA Los Angeles in a Red Little Cottage
    UNIVERSITY OF CALIFORNIA Los Angeles In a Red Little Cottage: Icons of Identity and Nation in Sweden A dissertation submitted in partial satisfaction of the requirements for the degree Doctor of Philosophy in Germanic Languages by Anna Jenny Katarina Blomster 2016 © Copyright by Anna Jenny Katarina Blomster 2016 ABSTRACT OF THE DISSERTATION In a Red Little Cottage: Icons of Identity and Nation in Sweden by Anna Jenny Katarina Blomster Doctor of Philosophy in Germanic Languages University of California, Los Angeles, 2016 Professor Timothy R. Tangherlini, Chair The image of a red cottage with white corners in a rural idyllic setting holds an iconic status in visual representations of Sweden, and has been produced and reproduced in commercial advertising and tourist industry as well as in the visual rhetoric of contesting political parties for over more than a century. The cottage’s conspicuous position in visual representations of Sweden speaks to the need for an in-depth analysis of the cottage as a trope of national identity. Despite this, it has not received greater attention in research on national symbols and national identity in the field of Scandinavian studies. My dissertation aims to fill this gap by a sustained, close reading of the red cottage as a continuous symbol through which national identity is manifested, and problematized. To fully understand the contemporary position of the red cottage’s position as ”key- ! ii! symbol” in visual representation of Sweden or ”Swedishness”, it is crucial to put it in its socio- historical context. In this dissertation, historical analysis is paired with semiotic visual studies.
    [Show full text]
  • The Fifth Periodic Report by the Government of Finland on the Implementation of the European Charter for Regional Or Minority Languages
    THE FIFTH PERIODIC REPORT BY THE GOVERNMENT OF FINLAND ON THE IMPLEMENTATION OF THE EUROPEAN CHARTER FOR REGIONAL OR MINORITY LANGUAGES NOVEMBER 2017 CONTENTS INTRODUCTION .................................................................................................................................................... 6 PART I ................................................................................................................................................................... 7 1. BASIC INFORMATION ON FINNISH POPULATION AND LANGUAGES ................................................................ 7 1.1. Finnish population according to mother tongue ..................................................................................... 7 1.2. Administration of population data ......................................................................................................... 9 2. SPECIAL STATUS OF THE ÅLAND ISLANDS ....................................................................................................... 9 3. NUMBER OF SPEAKERS OF REGIONAL OR MINORITY LANGUAGES IN FINLAND .............................................. 10 3.1. The numbers of persons speaking a regional or minority language....................................................... 10 3.2. Swedish ............................................................................................................................................... 10 3.3. Sámi languages ..................................................................................................................................
    [Show full text]
  • FINLAND by Päivi Gynther
    FINLAND by Päivi Gynther Gynther, P. (2012). Finland. In C. L. Glenn & J. De Groof (Eds.), Balancing freedom, autonomy and accountability in education: Volume 2 (157-173). Tilburg, NL: Wolf Legal Publishers. Overview The compulsory stage of education in Finland is called comprehensive school (Finnish peruskoulu), preceded by one year of voluntary pre-primary education. Compulsory schooling starts in the year during which the child turns seven and ends when the basic education syllabus has been completed, or ten years after the beginning of compulsory schooling.1 Children permanently residing in Finland have a statutory right and obligation to complete the comprehensive school syllabus. Nonetheless, there is no compulsory school attendance. Secondary education, which is non-compulsory, is divided into academic upper secondary school and vocational school. It is also possible to attend both vocational and upper secondary school simultaneously. Graduates of both academic and vocational track can apply to universities and polytechnics. Major current issues Equal opportunities, high quality and internationalization are outlined as education policy priorities of Finland for the near future.2 Public debate on these issues and their subtopics has been vivid. Distribution of lesson hours is among the most debated topics. An expert group set up by the Ministry of Education and Culture recently made a proposal to diversify language learning and to give the pupils the opportunity to begin language learning earlier than today. At the same time, the
    [Show full text]
  • Sweden Skagerrak, Kattegat and the Norwegian Deep Cold Water Prawn Fishery
    FULL ASSESSMENT REPORT Sweden Skagerrak, Kattegat and the Norwegian Deep cold water prawn fishery Public Certification Report Gothenburg Fish Auction Authors: Bert Keus, Julian Addison, Sigrun Bekkevold Report No.: 2015-001, Rev. 4 Certificate No.: F-DNV-187264 Date: 2015-10-15 Project name: Full assessment report DNV GL - Business Assurance Report title: Sweden Skagerrak, Kattegat and the Norwegian Deep cold water prawn fishery Det Norske Veritas Certification AS Public Certification Report Veritasveien 1 Customer: Gothenburg Fish Auction 1322 HØVIK, Norway Fiskhamnen, Tel: +47 67 57 99 00 414 58 Göteborg http://www.dnvgl.com Contact person: Bengt Gunnarsson Date of issue: 2015-10-15 Project No.: PRJC-494549-2013-MSC-NOR Organisation unit: ZNONO418 Report No.: 2015-001, Rev. 4 Certificate No.: F-DNV-187264 Task and objective: The ojective of this report is the initial assessment of Sweden Skagerrak, Kattegat and the Norwegian Deep cold water prawn fishery against the Marine Stewardship Council Fishery standard: Principles and Criteria for Sustainable Fishing. Prepared by: Verified by: Julian addison Anna Kiseleva Independent P1 & P2 expert DNV GL GSR Bert Keus Independent P3 expert and team leader Sigrun Bekkevold Team member ☒ Unrestricted distribution (internal and external) Keywords: ☐ Unrestricted distribution within DNV GL [Keywords] ☐ Limited distribution within DNV GL after 3 years ☐ No distribution (confidential) ☐ Secret Reference to part of this report which may lead to misinterpretation is not permissible. Rev. No. Date Reason for Issue Prepared by Verified by Approved by Preliminary Draft Report for Client Bert Keus, Julian 0 2015-01-30 review Addison, Sigrun Bekkevold 1 2015-02-03 Peer Review Draft Report Same as above 2 2015-03-09 Public Comment Draft Report Same as above 3 2015-08-27 Final Report Same as above 4 2015-10-15 Public Certification Report Same as above DNV GL – Report No.
    [Show full text]
  • The Churches of the Holy Land in the Twelfth and Thirteenth Centuries 198
    Tracing the Jerusalem Code 1 Tracing the Jerusalem Code Volume 1: The Holy City Christian Cultures in Medieval Scandinavia (ca. 1100–1536) Edited by Kristin B. Aavitsland and Line M. Bonde The research presented in this publication was funded by the Research Council of Norway (RCN), project no. 240448/F10. ISBN 978-3-11-063485-3 e-ISBN (PDF) 978-3-11-063943-8 e-ISBN (EPUB) 978-3-11-063627-7 DOI https://doi.org/10.1515/9783110639438 This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License. For details go to http://creativecommons.org/licenses/by-nc-nd/4.0/. Library of Congress Control Number: 2020950181 Bibliographic information published by the Deutsche Nationalbibliothek The Deutsche Nationalbibliothek lists this publication in the Deutsche Nationalbibliografie; detailed bibliographic data are available on the Internet at http://dnb.dnb.de. © 2021 Kristin B. Aavitsland and Line M. Bonde, published by Walter de Gruyter GmbH, Berlin/Boston. The book is published open access at www.degruyter.com. Cover illustration: Wooden church model, probably the headpiece of a ciborium. Oslo University Museum of Cultural history. Photo: CC BY-SA 4.0 Grete Gundhus. Typesetting: Integra Software Services Pvt. Ltd. Printing and binding: CPI Books GmbH, Leck www.degruyter.com In memory of Erling Sverdrup Sandmo (1963–2020) Contents List of Maps and Illustrations XI List of Abbreviations XVII Editorial comments for all three volumes XIX Kristin B. Aavitsland, Eivor Andersen Oftestad, and Ragnhild Johnsrud
    [Show full text]