The Routledge Handbook of Language Testing The Routledge Handbook of Language Testing offers a critical and comprehensive overview of language testing and assessment within the fields of applied linguistics and language study. An understanding of language testing is essential for applied linguistic research, language education, and a growing range of public policy issues. This handbook is an indispensable introduction and reference to the study of the subject. Specially commissioned chapters by leading academics and researchers of language testing address the most important topics facing researchers and practitioners, including: An overview of the key issues in language testing Key research methods and techniques in language test validation The social and ethical aspects of language testing The philosophical and historical underpinnings of assessment practices The key literature in the field Test design and development practices through use of practical examples The Routledge Handbook of Language Testing is the ideal resource for postgraduate students, language teachers and those working in the field of applied linguistics. Glenn Fulcher is Reader in Education (Applied Linguistics and Language Testing) at the University of Leicester in the United Kingdom. His research interests include validation theory, test and rating scale design, retrofit issues, assessment philosophy, and the politics of testing. Fred Davidson is a Professor of Linguistics at the University of Illinois at Urbana-Champaign. His interests include language test development and the history and philosophy of educational and psychological measurement. This page intentionally left blank The Routledge Handbook of Language Testing Edited by Glenn Fulcher and Fred Davidson First published 2012 by Routledge 2 Park Square, Milton Park, Abingdon, Oxon OX14 4RN Simultaneously published in the USA and Canada by Routledge 711 Third Avenue, New York, NY 10017 Routledge is an imprint of the Taylor & Francis Group, an informa business © 2012 Selection and editorial matter, Glenn Fulcher and Fred Davidson; individual chapters, the contributors. The right of the editors to be identified as the authors of the editorial material, and of the authors for their individual chapters, has been asserted in accordance with sections 77 and 78 of the Copyright, Designs and Patents Act 1988. All rights reserved. No part of this book may be reprinted or reproduced or utilised in any form or by any electronic, mechanical, or other means, now known or hereafter invented, including photocopying and recording, or in any information storage or retrieval system, without permission in writing from the publishers. Trademark notice: Product or corporate names may be trademarks or registered trademarks, and are used only for identification and explanation without intent to infringe. British Library Cataloguing in Publication Data A catalogue record for this book is available from the British Library Library of Congress Cataloging in Publication Data The Routledge handbook of language testing / edited by Glenn Fulcher and Fred Davidson. p. cm. Includes bibliographical references and index. 1. Language and languages–Ability testing. I. Fulcher, Glenn. II. Davidson, Fred. P53.4.R68 2011 418.0028'7–dc23 2011019617 ISBN: 978-0-415-57063-3 (hbk) ISBN: 978-0-203-18128-7 (ebk) Typeset in Times New Roman by Taylor & Francis Books Contents List of illustrations ix List of contributors xi Introduction 1 Glenn Fulcher and Fred Davidson PART I Validity 19 1 Conceptions of validity 21 Carol A. Chapelle 2 Articulating a validity argument 34 Michael Kane 3 Validity issues in designing accommodations for English language learners 48 Jamal Abedi PART II Classroom assessment and washback 63 4 Classroom assessment 65 Carolyn E. Turner 5 Washback 79 Dianne Wall 6 Assessing young learners 93 Angela Hasselgreen v Contents 7 Dynamic assessment 106 Marta Antón 8 Diagnostic assessment in language classrooms 120 Eunice Eunhee Jang PART III The social uses of language testing 135 9 Designing language tests for specific social uses 137 Carol Lynn Moder and Gene B. Halleck 10 Language assessment for communication disorders 150 John W. Oller, Jr. 11 Language assessment for immigration and citizenship 162 Antony John Kunnan 12 Social dimensions of language testing 178 Richard F. Young PART IV Test specifications 195 13 Test specifications and criterion referenced assessment 197 Fred Davidson 14 Evidence-centered design in language testing 208 Robert J. Mislevy and Chengbin Yin 15 Claims, evidence, and inference in performance assessment 223 Steven J. Ross PART V Writing items and tasks 235 16 Item writing and writers 237 Dong-il Shin 17 Writing integrated items 249 Lia Plakans 18 Test-taking strategies and task design 262 Andrew D. Cohen vi Contents PART VI Prototyping and field tests 279 19 Prototyping new item types 281 Susan Nissan and Mary Schedl 20 Pre-operational testing 295 Dorry M. Kenyon and David MacGregor 21 Piloting vocabulary tests 307 John Read PART VII Measurement theory and practice 321 22 Classical test theory 323 James Dean Brown 23 Item response theory 336 Gary J. Ockey 24 Reliability and dependability 350 Neil Jones 25 The generalisability of scores from language tests 363 Rob Schoonen 26 Scoring performance tests 378 Glenn Fulcher PART VIII Administration and training 393 27 Quality management in test production and administration 395 Nick Saville 28 Interlocutor and rater training 413 Annie Brown 29 Technology in language testing 426 Yasuyo Sawaki 30 Validity and the automated scoring of performance tests 438 Xiaoming Xi vii Contents PART IX Ethics and language policy 453 31 Ethical codes and unexpected consequences 455 Alan Davies 32 Fairness 469 F. Scott Walters 33 Standards-based testing 479 Thom Hudson 34 Language testing and language management 495 Bernard Spolsky Index 506 viii Illustrations Tables 7.1 Examiner–student discourse during DA episodes 114 8.1 Incremental granularity in score reporting (for the student JK) 127 14.1 Summary of evidence-centered design layers in the context of language testing 210 14.2 Design pattern attributes and relationships to assessment argument 212 14.3 A design pattern for assessing cause and effect reasoning reading comprehension 213 14.4 Steps taken to redesign TOEFL iBT and TOEIC speaking and writing tests and guided by layers in Evidence-Centered Design 217 23.1 Test taker response on a multiple-choice listening test 339 25.1 Scores for 10 persons on a seven-item test (fictitious data) 366 25.2 Analysis of variance table for the sample data 367 25.3 Scores for 15 persons on two speaking tasks rated twice on a 30-point scale (fictitious data) 369 25.4 Analysis of variance table (pÂtÂr) for the sample data 2 (Table 25.3) 370 25.5 Analysis of variance table (pÂ(r:t)) for the sample data 2 (Table 25.3), with raters nested within task 371 26.1 Clustering scores by levels 380 33.1 Interagency Language Roundtable Levels and selected contexts – speaking 481 33.2 The Interagency Language Roundtable (ILR) to the American Council on the Teaching of Foreign Languages (ACTFL) concordance 482 33.3 Foreign Service Institute descriptor for Level 2 speaking 483 33.4 American Council on the Teaching of Foreign Languages Advanced descriptor 483 33.5 Example standards for foreign language learning 484 33.6 Example descriptors for the intermediate learner range of American Council on the Teaching of Foreign Languages K-12 Guidelines 485 33.7 Canadian Language Benchmarks speaking and listening competencies 486 33.8 Example global performance descriptor and performance conditions from the Canadian Language Benchmarks 486 33.9 Common European Framework—global scale 488 33.10 Example descriptors for the Common European Framework of Reference for Languages 489 ix Illustrations 33.11 California English Language Development Standards Listening and Speaking Strategies and Applications 490 33.12 California English Language Development Test blueprint from grade 2 491 Figures 8.1 Distracter characteristics curves (DCCs) 126 13.1 A sample score report for the internet-based TOEFL, Reading Subsection 198 13.2 The evolutionary role of test specs in test development 203 14.1 Toulmin’s general structure for arguments 209 14.2 Extended Toulmin diagram for assessment 212 19.1 Schematic table 287 19.2 Summary task 288 19.3 Outline completion task 289 21.1 Sample item from the WAT for Dutch Children 315 23.1 Item characteristic curves for three items from the 1PL Rasch model 338 23.2 Item characteristic curves for 1PL, 2PL, and 3PL 342 23.3 Category response curves for a five-point rating scale 344 24.1 Some sources of error in a test 352 25.1 Generalisability coefficients (Eρ2) for different numbers of tasks and raters 370 26.1 Empirically derived, binary-choice, boundary definition scale schematic 385 27.1 Core processes 403 27.2 The assessment cycle 405 27.3 The assessment cycle showing periodic test review 408 32.1 Worksheet grid 476 Box 13.1 A sample test specification 199 x Contributors Jamal Abedi is a Professor at the School of Education of the University of California, Davis, and a research partner at the National Center for Research on Evaluation, Standards, and Student Testing. His research interests include accommodations and classification for English language learners, and comparability of alternate assessments for students with significant cognitive disabilities. Marta Antón is Associate Professor of Spanish at Indiana University–Purdue University Indiana- polis and Research Fellow at the Indiana Center for Intercultural Communication. She has pub- lished on classroom interaction, sociocultural theory, dynamic assessment and Spanish sociolinguistics. Annie Brown is a Principal Research Fellow with the Australian Council for Educational Research. Her research interests include rater and interlocutor behaviour and the assessment of oral proficiency James Dean (J.D.) Brown, Professor and Chair in the Department of Second Language Studies at the University of Hawai’i at Manoa, has worked in places ranging from Brazil to Yugoslavia, and has published numerous articles and books on language testing, curriculum design, and research methods.