Linking to the CEFR levels: Research perspectives Linking to the CEFR levels: Neus Figueras & José Noijons (eds.) Research perspectives Neus Figueras & José Noijons (eds.) Noijons José & Figueras Neus Neus Figueras & José Noijons (eds.) ‘The Colloquium �������������������� �������������������� �������������� served very well the Linking to the CEFR levels: levels: CEFR the to Linking Cito, Institute for Educational Measurement Council of Europe mission and aims European Association for Language Testing and Assessment (EALTA) ofperspective Research the Council s 2009 of Europe.’ Linking to the CEFR levels: Research perspectives Neus Figueras & José Noijons (eds.) Cito, Institute for Educational Measurement Council of Europe European Association for Language Testing and Assessment (EALTA) © Cito, EALTA Arnhem, 2009 �������������������� �������������������� �������������� 2 Standard Setting Research and its Relevance to the CEFR Preface Cito, the Institute for Educational Measurement in the Netherlands, is proud to have sponsored the organisation of the colloquium on Standard Setting Research and its Relevance to the CEFR and this publication of its proceedings. Cito has much appreciated the work done by the Council of Europe in developing the CEFR. Cito has been actively engaged in the propagation of the framework and its proper use in the Netherlands and elsewhere in Europe through our participation in DIALANG and the European Survey on Language Competences, among others. One more activity in this context has been the Cito contribution to the development of a Manual on Relating Language Examinations to the CEFR. In the Manual various methods of linking examinations to the CEFR have been described. The standardization phase in the linking process has been of particular concern to Cito. We feel that claims of links to the CEFR have not always been sufficiently substantiated. Even though the Manual gives examples of various methods of setting CEFR standards, Cito has been of the opinion that a discussion among experts could be of much use to the improvement of linking activities that many countries and institutions are now carrying out. Cito is thankful to the European Association for Language Testing and Assessment (EALTA) to have agreed to organising the colloquium, the proceedings of which constitute this volume. Similarly Cito is thankful to the HAU, the Hellenic American Union in Athens, Greece, for having successfully hosted this colloquium. As is borne out by the contributions in this volume, the colloquium has been a most challenging and successful event. We are certain that this publication will contribute to the dissemination of good practices in linking tests and examinations to the CEFR. Marten Roorda, CEO Cito 3 Preface 4 Standard Setting Research and its Relevance to the CEFR Foreword On behalf of the Language Policy Division of the Council of Europe and EALTA we wish to express our gratitude to Cito for taking the initiative and providing generous support for the Research Colloquium held in Athens as a pre-conference event. This is the first time that the two professional bodies and the intergovernmental Organisation have joined forces in this manner. The lively interest in the colloquium and its obvious success encourage us to consider ways of continuing this co-operation. The Colloquium served very well the mission and aims of the Council of Europe. For several decades, the Language Policy Division of the Council of Europe has worked in close co- operation with its 47 member countries and with the language education profession to develop approaches and instruments through co-operative fora and networks in order to assist in the challenging task of promoting plurilingualism in Europe. The Council of Europe has always sought and welcomed the contribution of individuals and institutions willing to share their expertise. The Common European Framework of Reference for Languages : learning, teaching, assessment (CEFR) drew on the main developments of several decades before its publication in 2001. Its aim is to improve the quality, coherence and transparency of all aspects of language learning, teaching and assessment across Europe. In order to provide assistance in linking examinations to the CEFR, the Language Policy Division has developed and piloted a Manual for professionals in the field of assessment. The Committee of Ministers of the Council of Europe considers it very important that linking examinations to the CEFR is carried out with due attention to the quality of the process, and stresses the need for the procedures to be well documented; results should be reported in sufficient detail and made freely available and readily accessible to all the interested parties. (Recommendation 2008 (7) of the Committee of Ministers of the Council of Europe to its member states on the use of the CEFR and the promotion of plurilingualism: www.coe.int/t/cm). The Manual provides guidance on how to do this. Research seminars during which empirical work on linking is reported and discussed provided valuable information and concrete case studies. The Language Policy Division is very grateful to Cambridge ESOL and other ALTE members who organised events for the Council of Europe to discuss case studies on the use of the pilot version of the Manual. This event in Athens had the same goal but used a somewhat different format, and the Council of Europe is very grateful to Cito for, in addition to the organisation of the event, publishing its very interesting deliberations and findings. 5 Foreword EALTA´s mission is broad: it aims to share professional expertise in language testing and assessment, to improve language testing and assessment systems and practice in Europe, and to engage in other activities for the improvement of language testing and assessment in Europe. EALTA considers that, by joining forces with Cito in organising this professional event for the Council of Europe, it made a concrete contribution to the promotion of its mission. It is very gratifying that so many colleagues expressed an interest in presenting and discussing their work on relating examinations to the CEFR, a very topical and important issue for quality, coherence and transparency in language testing and examinations. We have come a long way from the seminar in Helsinki in July 2002, when the process of exploring linkage of examinations to the CEFR was launched. Together we co-operated in planning and organising the event, which launched the development of the Manual and led to the present extensive work with the Manual. EALTA endorses the values, aims and objectives of the Council of Europe and trusts that it will soon join the group of INGOs enjoying participatory status with the Council of Europe. The Language Policy Division of the Council of Europe welcomes EALTA´s contribution and looks forward to continuing its fruitful co-operation with the Association. Johanna Panthier Sauli Takala Administrator President of EALTA Language Policy Division Council of Europe 6 Standard Setting Research and its Relevance to the CEFR Table of contents Preface 3 Foreword 5 Introduction 9 Part I Approaches from theory 11 Views from the expert discussants 1 Standard Setting Theory and Practice: Issues and Difficulties 13 Mark D. Reckase 2 Basket Procedure: The Breadbasket or the Basket Case of Standard Stting Methods? 21 F. Kaftandjieva 3 A comparative approach to constructing a multilingual proficiency framework: 35 constraining the role of standard-setting Neil Jones 4 Linking multilingual survey results to the Common European Framework 45 of Reference Norman Verhelst 5 Standard Setting from a Psychometric point of view 59 Gunter Maris Part II Accounts from practice 67 Reports from the practitioners 6 Relating the Trinity College London International ESOL examinations to the CEFR 69 Cathy Taylor 7 Analyzing the decision-making process of standard setting participants 75 Spiros Papageorgiou 8 Benchmarking of videotaped oral performances in terms of the CEFR 81 Gunter Maris, José Noijons and Evelyn Reichard 7 Table of contents 9 Designing Proficiency Levels for English for Primary and Secondary School Students 87 and the Impact of the CEFR Karmen Pižorn 10 Investigating the Relationship Between the EIKEN Tests and the CEFR 103 Jamie Dunlea and Tomoki Matsudaira 11 Linking SQA’s ESOL Higher to the CEFR 111 Rob van Krieken 12 Bilkent University School of English Language COPE CEFR Linking Project 119 Carole Thomas and Elif Kantarcioglu 13 Standard Setting for Listening, Grammar, Vocabulary and Reading Sections 125 of the Advanced Level Certificate in English (ALCE) N. Downey and C. Kollias 14 Jack of more Trades? 131 Could standard setting serve several functions? Eli Moe 8 Table of contents Introduction The launch of the Preliminary Pilot version of the Manual for relating examinations to the CEFR 2003 initiated many new CEFR related projects and influenced those already in progress. The state of affairs in relation to the uses of the CEFR and associated documents in Europe, including the Manual, was analysed at the Intergovernmental Forum organized by the Council of Europe with the collaboration of Cito and the French Ministry of Education in Strasbourg in February 2007 (report accessible at http://www.coe.int/t/dg4/ linguistic/Forum07_webdocs_EN.asp#TopOfPage) At one of the preparatory meetings for the Forum, EALTA was approached by Cito, in order to jointly organise a research colloquium that would allow professionals in the field of testing to exchange views