Research Notes 55

Research Notes Issue 55 February 2014 ISSN 1756-509X Research Notes Issue 55 / February 2014 A quarterly publication reporting on learning, teaching and assessment Guest Editor Dr Hanan Khalifa, Head of Research and International Development, Cambridge English Language Assessment Senior Editor and Editor Dr Jayanti Banerjee, Research Director, Cambridge Michigan Language Assessments Coreen Docherty, Senior Research and Validation Manager, Cambridge English Language Assessment Editorial Board Cris Betts, Assistant Director, Cambridge English Language Assessment Natalie Nordby Chen, Director of Assessment, Cambridge Michigan Language Assessments Barbara Dobson, Assistant Director, Cambridge Michigan Language Assessments Dr Gad S Lim, Principal Research and Validation Manager, Cambridge English Language Assessment Production Team Rachel Rudge, Marketing Project Co-ordinator, Cambridge English Language Assessment John Savage, Publications Assistant, Cambridge English Language Assessment Printed in the United Kingdom by Canon Business Services CAMBRIDGE ENGLISH : RESEARCH NOTES : ISSUE 55 / FEBRUARY 2014 | 1 Research Notes Issue 55 February 2014 Contents Editorial 2 Safeguarding fairness principles through the test development process: A tale of two organisations 3 Katie Weyant and Amanda Chisholm Investigating grammatical knowledge at the advanced level 7 Fabiana MacMillan, Daniel Walter and Jessica O’Boyle A look into cross-text reading items: Purpose, development and performance 12 Fabiana MacMillan, Mark Chapman and Jill Rachele Stucker The Examination for the Certificate of Competency in English revision project: Maintaining score meaning 16 Natalie Nordby Chen and Jayanti Banerjee A discourse variable approach to measuring prompt effect: Does paired task development lead to comparable writing products? 22 Mark Chapman, Crystal Collins, Barbara Allore Dame and Heather Elliott Nativelike formulaic sequences in office hours: Validating a speaking test for international teaching assistants 28 Ildiko Porter-Szucs and Ummehaany Jameel Dimensionality and factor structure of an English placement test 34 Daniel Walter and Jasmine Hentschel 2 | CAMBRIDGE ENGLISH : RESEARCH NOTES : ISSUE 55 / FEBRUARY 2014 Editorial Welcome to issue 55 of Research Notes, our quarterly to systematically classify C2 level grammar items but also publication reporting on research matters relating to learning, points to the need for more research into the criterial features teaching and assessment within Cambridge English Language that distinguish one level from another, in particular at the C Assessment. This issue draws the attention of the Research levels. Notes readership to the exams and research being conducted The Examination for the Certificate of Competency in at Cambridge Michigan Language Assessments (CaMLA). English (ECCE™), which is aimed at the B2 level of the In September 2010, Cambridge English Language CEFR, was recently revised and the next two articles focus Assessment and the University of Michigan English Language on the changes made to the reading and listening papers Institute Testing and Certification Division formed CaMLA. respectively. First, Fabiana MacMillan, Mark Chapman and This new joint venture brought together two organisations Jill Rachele Stucker describe a new task type introduced into that have long histories in language education creating the reading paper and the rationale for its inclusion. This task opportunities for collaboration. An important benefit of this involves candidates having to process information from more partnership is our ability to offer stakeholders a wider range of than one text in order to answer a series of comprehension exams to meet their specific needs. questions. It requires readers to develop an intertextual The articles in this issue focus on fairness principles, representation, which is a trait associated with high-level frameworks designed to facilitate item writing, and exam readers (Khalifa and Weir 2009). MacMillan, Chapman revision and validation studies. Each article gives the reader a and Stucker present the theoretical basis for including better understanding of CaMLA’s suite of tests as well as the this task type, the challenges associated with developing processes and concerns that are specific to their context of these reading tasks as well as candidate performance. use. Although they found that writing these tasks proved to be The issue begins with an article describing the principles quite challenging, they perform as expected and improve and procedures CaMLA and Cambridge English Language the construct coverage of the paper. The revision process Assessment follow to ensure their tests are fair for all involved in updating the listening paper is then described candidates. Katie Weyant and Amanda Chisholm show how by Natalie Nordby Chen and Jayanti Banerjee. In response fairness is embedded in all stages of the test development to stakeholder consultative exercises, a review of the processes of both organisations even though the specific B2 listening CEFR descriptors and current research into procedures and the way they are operationalised differ slightly second language listening, Part 2 of the listening paper in response to their unique operational contexts. In many was modified from a single long interview to four short ways, both organisations have embraced Spaan’s (2000:35) talks, which allowed the paper to cover more listening proposal that test developers enter into a ‘social contract’ constructs. Both of these articles provide a good example with stakeholders, which entails ensuring their tests are fair, of a rational and explicit approach to test revision and accurate and valid as well as providing users with transparent development. and comprehensive information about all aspects of their The last three articles focus on the validation of different tests (see Cambridge English Language Assessment 2013). CaMLA tests. The first article in this section investigates the This article highlights the link between test fairness and test equivalence of writing prompts in the ECPE test, a C2 level validity in that concerns over fairness will inevitably lead to exam. As this writing test offers candidates a choice between concerns about the validity and reliability of test results. two writing prompts, Mark Chapman, Crystal Collins, Barbara The next article by Fabiana MacMillan, Daniel Walter Allore Dame and Heather Elliott investigate which prompt and Jessica O’Boyle deals with the perennial question of features should be controlled for to ensure that the writing what makes an item more or less difficult. As the authors output is also comparable. This is important as it links to the point out, previous research has shown that item writers are issue of fairness: if the features and characteristics of the not particularly accurate when it comes to predicting item writing produced are shaped by the prompt, then comparable difficulty (see Fortus, Coriat and Fund 1998, Hamp-Lyons prompts will ensure that scripts can be scored consistently. and Mathias 1994 and Sydorenko 2011); however, being able They found that when domain, length and task wording were to identify factors that affect difficulty can facilitate item kept consistent, candidate output did not vary substantially. writing and allow test developers to better manage their Their recommendations could lead to a more systematic item banks. In response to this issue, MacMillan, Walter approach to prompt development and ensure not only and O’Boyle describe the development of a framework for comparability within a test but across test versions. predicting the difficulty of grammatical items being produced Ildiko Porter-Szucs and Ummehaany Jameel then present for the Examination for the Certificate of Proficiency in English research that focuses on the International Teaching Assistant (ECPE™) test, which is targeted at Level C2 of the Common Speaking Assessment (ITASA™) which is used to evaluate European Framework of Reference (CEFR) (Council of Europe the language proficiency of international teaching assistants 2001). The authors make use of corpora and second language (ITAs). As the test is designed to replicate, as much as acquisition research to create a tagging rubric that can be possible, the situations and settings that ITAs operate within, applied to items testing grammatical knowledge and use. This the authors were interested in finding out the extent to rubric was then trialled and the predictive ability of it was which the ‘Office-Hour Role Play’ (Task 3) elicits nativelike tested. This study highlights the challenges involved in trying formulaic sequences and whether the presence of these © UCLES 2014 – The contents of this publication may not be reproduced without the written permission of the copyright holder. CAMBRIDGE ENGLISH : RESEARCH NOTES : ISSUE 55 / FEBRUARY 2014 | 3 features in candidate responses influences their scores. essential test qualities are met and delivered according to Although they didn’t find any relationship between the use of stakeholder expectations. formulaic language and candidate ratings, they were able to demonstrate that the language produced by candidates was Cambridge English Language Assessment (2013) Principles of Good similar to that produced by native speakers performing similar Practice: Quality Management and Validation in Language Assessment, available online: www.cambridgeenglish.org/research-and-validation/ tasks. This study provides further support that this test is

Load more