Relating Language Examinations to the CEFR: a Manual
Total Page：16
File Type：pdf, Size：1020Kb
January 2009 Relating Language Examinations to the Common European Framework of Reference for Languages: Learning, Teaching, Assessment (CEFR) A Manual Language Policy Division, Strasbourg www.coe.int/lang Contents List of Figures Page iii List of Tables Page v List of Forms Page vii Preface Page ix Chapter 1: The CEFR and the Manual Page 1 Chapter 2: The Linking Process Page 7 Chapter 3: Familiarisation Page 17 Chapter 4: Specification Page 25 Chapter 5: Standardisation Training and Benchmarking Page 35 Chapter 6: Standard Setting Procedures Page 57 Chapter 7: Validation Page 89 References Page 119 Appendix A Forms and Scales for Description and Specification (Ch. 1 & 4) Page 122 A1: Salient Characteristics of CEFR Levels (Ch. 1) Page 123 A2: Forms for Describing the Examination (Ch. 4) Page 126 A3: Specification: Communicative Language Activities (Ch. 4) Page 132 A4: Specification: Communicative Language Competence (Ch. 4) Page 142 A5: Specification: Outcome of the Analysis (Ch. 4) Page 152 Appendix B Content Analysis Grids (Ch.4) B1: CEFR Content Analysis Grid for Listening & Reading Page 153 B2: CEFR Content Analysis Grids for Writing and Speaking Tasks Page 159 Appendix C Forms and Scales for Standardisation & Benchmarking (Ch. 5) Page 181 Reference Supplement: Section A: Summary of the Linking Process Section B: Standard Setting Section C: Classical Test Theory Section D: Qualitative Analysis Methods Section E: Generalisability Theory Section F: Factor Analysis Section G: Item Response Theory Section H: Test Equating i ii List of Figures Figure 2.1: Validity Evidence of Linkage of Examination/Test Results to the CEFR Page 8 Figure 2.2: Visual Representation of Procedures to Relate Examinations to the CEFR Page 15 Figure 6.1: Frequency Distributions of Test Scores in Two Contrasting Groups Page 67 Figure 6.2: Logistic Regression Page 73 Figure 6.3: Panel Member Recording Form for Bookmark Method Page 78 Figure 6.4: Items with Unequal Discrimination Page 82 Figure 6.5: Item Map, Indicating Difficulty and Discrimination Page 83 Figure 7.1: Empirical Item Characteristic Curve for a Problematic Item Page 101 Figure 7.2: A Test Characteristic Curve Page 105 Figure 7.3: Bivariate Decision Table Using Nine Levels Page 112 Figure 7.4: Bivariate Decision Table Using Five Levels Page 113 Figure 7.5: Item Map with Test Items and “Can Do” Statements Page 116 iii iv List of Tables Table 3.1: Time Management for Familiarisation Activities Page 23 Table 3.2: Documents to be Prepared for Familiarisation Activities Page 23 Table 4.1: Forms and Scales for Communicative Language Activities Page 32 Table 4.2: CEFR Scales for Aspects of Communicative Language Competence Page 32 Table 5.1: Time Management for Assessing Oral Performance Samples Page 45 Table 5.2: Time Management for Assessing Written Performance Samples Page 46 Table 5.3: Documents and Tools to be prepared for Rating Writing Page 47 Table 5.4: Reference Sources in the CEFR Page 49 Table 5.5: Standardisation and Benchmarking: Summary Page 55 Table 6.1: Overview of the Methods Discussed Page 61 Table 6.2: Basic Data in the Tucker-Angoff method Page 62 Table 6.3: Computing the Expected Score of 100 Borderline Persons Page 66 Table 6.4: Frequency Distribution Corresponding to Figure 6.1 Page 68 Table 6.5: Decision Tables for Five Cut-off Scores Page 68 Table 6.6: Summary of the Rangefinding Round Page 71 Table 6.7: Results of the Pinpointing Round (partially) Page 73 Table 6.8: Example of an ID Matching Response Form (abridged) Page 75 Table 6.9: Bookmarks and Achievement Levels Page 80 Table 6.10: Estimated Theta Page 81 Table 7.1: Balanced Incomplete Block Design with Three Blocks Page 93 Table 7.2: Balanced Incomplete Block Design with Seven Blocks Page 93 Table 7.3: Example of High Consistency and Total Disagreement Page 98 Table 7.4: Bivariate Frequency Table using Four Levels Page 99 Table 7.5: Frequencies of Allocation of a Single Item to Different CEFR Levels Page 101 Table 7.6: Summary of Disagreement per Item Page 102 Table 7.7: Outcome of a Tucker-Angoff Procedure Page 102 Table 7.8: Variance Decomposition Page 103 v Table 7.9: Decision Accuracy Page 107 Table 7.10: Decision Consistency Page 108 Table 7.11: Marginal Distributions Across Levels (Frequencies) Page 110 Table 7.12: Marginal Distributions Across Levels (Percentages) Page 111 Table 7.13: Design for a Paired Standard Setting Page 115 Table A1: Salient Characteristics: Interaction & Production Page 123 Table A2: Salient Characteristics: Reception Page 124 Table A3: Relevant Qualitative Factors for Reception Page 143 Table A4: Relevant Qualitative Factors for Spoken Interaction Page 148 Table A5: Relevant Qualitative Factors for Production Page 149 Table C1: Global Oral Assessment Scale Page 184 Table C2: Oral Assessment Criteria Grid Page 185 Table C3: Supplementary Criteria Grid: “Plus levels” Page 186 Table C4: Written Assessment Criteria Grid Page 187 vi List of Forms Form A1: General Examination Description Page 126 Form A2: Test Development Page 127 Form A3: Marking Page 129 Form A4: Grading Page 130 Form A5: Reporting Results Page 130 Form A6: Data Analysis Page 131 Form A7: Rationale for Decisions Page 131 Form A8: Initial Estimation of Overall Examination Level Page 28 / 132 Form A9: Listening Comprehension Page 132 Form A10: Reading Comprehension Page 133 Form A11: Spoken Interaction Page 134 Form A12: Written Interaction Page 136 Form A13: Spoken Production Page 137 Form A14: Written Production Page 138 Form A15: Integrated Skills Combinations Page 139 Form A16: Integrated Skills Page 139 Form A17: Spoken Mediation Page 140 Form A18: Written Mediation Page 141 Form A19: Aspects of Language Competence in Reception Page 142 Form A20: Aspects of Language Competence in Interaction Page 145 Form A21: Aspects of Language Competence in Production Page 146 Form A22: Aspects of Language Competence in Mediation Page 150 Form A23: Graphic Profile of the Relationship of the Examination to CEFR Levels Page 33 / 152 Form A24: Confirmed Estimation of Overall Examination Level Page 34 / 152 Form C1: Training Record Form Page 181 Form C2: Analytic Rating Form (Swiss Project) Page 182 Form C3: Holistic Rating Form (DIALANG) Page 182 Form C4: Collation Global Rating Form (DIALANG) Page 183 Form C5: Item Rating Form (DIALANG) Page 183 These forms are also available on the website www.coe.int/lang vii viii Preface The Council of Europe wishes to acknowledge with gratitude all those who have made it possible to develop this Manual, and in particular the contributions by: The Finnish authorities who provided the forum in Helsinki to launch the initiative in July 2002. The “Sounding Board” of consultants for the pilot edition (Prof. Charles Alderson, Dr Gergely A. David, Dr John de Jong, Dr Felianka Kaftandjieva, Dr Michael Makosch, Dr Michael Milanovic, Professor Günther Nold, Professor Mats Oscarson, Prof. Günther Schneider, Dr Claude Springer and also Mr Josef Biro, Ms Erna van Hest, Mr Peter Lenz, Ms Jana Pernicová, Dr Vladimir Kondrat Shleg, Ms Christine Tagliante and Dr John Trim) for their important feedback in the early stage of the project. The Authoring Group, under the leadership of Dr Brian North: Dr Neus Figueras Departament d’Educació, Generalitat de Catalunya, Spain Dr Brian North Eurocentres Foundation, Switzerland Professor Sauli Takala University of Jyväskylä, Finland (emeritus) Dr Piet van Avermaet Centre for Diversity and Learning, University of Ghent, Belgium Association of Language Testers in Europe (ALTE) Dr Norman Verhelst Cito, The Netherlands Dr Jay Banerjee (University of Lancaster) and Dr Felianka Kaftandjieva (University of Sofia) for their contributions to the Reference Supplement to the Manual. The institutions who made available illustrative performance samples and sample test items that have been circulated on DVD/CD ROM and made available on the Council of Europe’s website in order to assist in standardisation training (especially: Eurocentres; Cambridge ESOL; the CIEP; the University for Foreigners, Perugia; the Goethe-Institut; the Finnish authorities; DIALANG; the Generalitat de Catalunya and CAPLE). ALTE (especially Nick Saville) and the members of the “Dutch CEFR project group” (Charles Alderson, Neus Figueras, Günther Nold, Henk Kuijper, Sauli Takala, Claire Tardieu) for contributing to the “Toolkit” related to this Manual with the Content Analysis Grids which they developed for Speaking and Writing, and for Listening and Reading respectively. The many individuals and institutions who gave detailed feedback on the pilot version, especially: the members of ALTE; Asset Languages (Cambridge ESOL); Budapest Business School; Cito; Claudia Harsch; the Goethe-Institut; the Polish Ministry of Education; the Taiwan Ministry of Education; TestDaF; Trinity College London; and the University for Foreigners, Perugia. Language Policy Division Directorate of Education and Languages (DG IV) F – 67075 STRASBOURG Cedex www.coe.int/lang www.coe.int/lang-CEFR ix x Chapter 1 The CEFR and the Manual 1.1. The Aims of the Manual 1.2. The Context of the Manual 1.1. The Aims of the Manual The primary aim of this Manual is to help the providers of examinations to develop, apply and report transparent, practical procedures in a cumulative process of continuing improvement in order to situate their examination(s) in relation to the Common European Framework (CEFR). The Manual is not the sole guide to linking a test to the CEFR and there is no compulsion on any institution to undertake such linking. However, institutions wishing to make claims about the relationship of their examinations to the levels of the CEFR may find the procedures helpful to demonstrate the validity of those claims. The approach developed in the Manual offers guidance to users to: describe the examination coverage, administration and analysis procedures; relate results reported from the examination to the CEFR Common Reference Levels; provide supporting evidence that reports the procedures followed to do so.