Language Testing RECONSIDERED This page intentionally left blank Language Testing RECONSIDERED Edited by Janna Fox Mari Wesche Doreen Bayliss Liying Cheng Carolyn E. Turner Christine Doe University of ACTEXPRESS Ottawa Press © University of Ottawa Press 2007 All rights reserved. No parts of this publication may be reproduced, stored in a retrieval system or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording, or otherwise, without permission in writing from the publisher. Library and Archives Canada Cataloguing in Publication Language testing reconsidered / edited by Janna Fox ... [et al.]. (Actexpress, ISSN 1480-4743) Includes bibliographical references and index. ISBN 978-0-7766-0657-6 1. Language and languages-Ability testing. 2. Language and languages-Examinations. I. Fox, Janna D., 1946- II. Series. P53.4.L353 2007 418.0076 C2007-902093-3 Books in the ACTEXPRESS series are published without the editorial intervention of the University of Ottawa Press. The papers selected for publication in Language Testing Reconsidered were subject to a two-stage blind review process by experts in the field of language testing. Published by the University of Ottawa Press, 2007 542 King Edward Avenue Ottawa, Ontario K2P OZ3 www.uopress.uottawa.ca The University of Ottawa Press acknowledges with gratitude the support extended to its publishing programme by the Government of Canada through its Book Publishing Industry Development Program, by the Canada Council for the Arts, and by the University of Ottawa. The editors gratefully acknowledge the support provided for publication of Language Testing Reconsidered by the Social Sciences and Humanities Research Council of Canada (SSHRC), the University of Ottawa's Second Language Institute, and Carleton University's School of Linguistics and Applied Language Studies. CONTENTS Abbreviations and Acronyms viii About the Editors x Introduction 1 Janna Fox SECTION I WHAT DOES IT MEAN TO KNOW A LANGUAGE? The International Language Testing Association (ILTA) 8 The UCLES/ILTA Lifetime Achievement Award 1. On Second Thoughts 9 Bernard Spolsky SECTION II WHAT ARE WE MEASURING? 2. The Challenge of (Diagnostic) Testing: Do We Know What We Are Measuring? 21 J. Charles Alder son 3. What Is the Construct? The Dialectic of Abilities and Contexts in Defining Constructs in Language Assessment 41 Lyle F. Bachman 4. Assessing Academic English Language Proficiency: 40+ years of U.K. Language Tests 73 Alan Davies v SECTION III LANGUAGE TESTING RESEARCH: POINTS OF DEPARTURE 5. The Coming of Age for Research on Test-Taking Strategies 89 Andrew D. Cohen 6. Qualitative Research Methods in Language Test Development and Validation 113 Anne Lazaraton and Lynda Taylor 7. Language Testing: A Question of Context 131 Tim McNamara SECTION IV ANTECEDENTS AND PROSPECTS 8. Tests as Power Tools: Looking Back, Looking Forward 141 Elana Shohamy The Goals of ILTA 153 Language Testing Research Colloquia (LTRC), 1979-2007 154 Notes on Contributors 155 References 157 Index 175 VI LIST OF TABLES AND FIGURES TABLES 4.1 English Proficiency Test Battery (EPTB), in operation from 1965 to 1980 79 4.2 English Language Testing Service (ELTS) test, in operation from 1980 to 1990 80 4.3 International English Language Testing Service (IELTS) test, in operation since 1989 81 6.1 Coding categories 123 FIGURES 3.1 Approaches to defining the construct in language testing, 1960 to the present 44 8.1 Ninth-grade Hebrew standard grades, according to years of residence 146 8.2 Eleventh-grade math standard grades, according to years of residence 147 8.3 Math grades in monolingual vs. bilingual test versions 148 VII ABBREVIATIONS AND ACRONYMS ACTFL American Council on the Teaching of Foreign Languages AILA Association internationale de linguistique appliquée or International Association of Applied Linguistics ALTE Association of Language Testers in Europe ANOVA Analysis of Variance AUA Assessment Use Argument ATESL Administrators and Teachers of English as a Second Language CA Conversational Analysis CAE Certificate in Advanced English CEF Common European Framework CEFR Common European Framework of Reference CLA Communicative Language Ability CPE Certificate of Proficiency in English DA Discourse Analysis DIALANG Diagnostic + Language EAL English as an Additional Language EFL English as a Foreign Language ELD English Language Development ELP English Language Portfolio ELTS English Language Testing Service EPTB English Proficiency Test Battery ESL English as a Second Language ESOL English for Speakers of Other Languages ETS Educational Testing Service FCE First Certificate in English FSI Foreign Service Institute GPA Grade Point Average IDP International Development Program IELTS International English Language Testing System Vlll ILR Interagency Language Roundtable ILTA International Language Testing Association IMAGE Illinois Measure of Annual Growth in English ISLPR International Second Language Proficiency Ratings KET Key English Test LTRC Language Testing Research Colloquium MANOVA Multivariate Analysis of Variance MELAB Michigan English Language Assessment Battery NAFS A National Association of Foreign Student Advisors NFLC National Foreign Language Center NGO Non-Governmental Organization OE Oral Examiner OET Occupational English Test OPI Oral Proficiency Interview PET Preliminary English Test SLA Second Language Acquisition SLTP Second Language Test Performance SSHRC Social Sciences and Humanities Research Council (of Canada) TESOL Teachers of English to Speakers of Other Languages TLU Target Language Use TOEFL Test of English as a Foreign Language TOEFL CBT Test of English as a Foreign Language Computer-Based Test TOEFL iBT Test of English as a Foreign Language Internet-Based Test TOEFL PBT Test of English as a Foreign Language Paper-Based Test TOEIC Test of English for International Communication TSE Test of Spoken English TTW Test of Test-Wiseness TWE Test of Written English UCLES University of Cambridge Local Examinations Syndicate USIA United States Information Agency VPA Verbal Protocol Analysis IX ABOUT THE EDITORS Janna Fox is an Associate Professor and Director of the Language Assess- ment and Testing Research Unit within the School of Linguistics and Applied Language Studies at Carleton University. She teaches courses in assessment, curriculum, and research methods. Her research interests are language test development and validation, and the interplay between assessment, curricula, and language policy. Mari (Marjorie) Wesche is a recently retired Professor and former Director of the Second Language Institute and still active in the Education Grad- uate Studies program of the University of Ottawa. Her main teaching areas have been second language acquisition, assessment, and content- based second language instruction, with research emphases in second language vocabulary acquisition and assessment, language aptitude, performance-based testing, and immersion instruction. Doreen Bayliss is the former head of Testing and Research Support Services at the Second Language Institute, University of Ottawa. Throughout her career, she has focused on test development and validation. Since her retirement, she has continued to engage in testing projects around the world. Liying Cheng is an Associate Professor and a Director of the Assessment and Evaluation Group (AEG) at the Faculty of Education, Queen's Univer- sity. Her primary research interests are the impact of large-scale testing on instruction, and the relationship between assessment and instruction in EFL/ESL classrooms. Carolyn E. T\irner is an Associate Professor in the Department of Integrated Studies in Education at McGill University. Her main focus and com- mitment are language assessment and testing in educational settings and more recently in healthcare contexts concerning access for linguistic mi- norities. She pursues these through her research, teaching, and service. Christine Doe is in a doctoral program studying language assessment in ed- ucational contexts in the Faculty of Education, Queen's University. Her specific research interests relate to the use of diagnostic assessment to support learning. x INTRODUCTION Janna Fox Carleton University Language Testing Reconsidered1 is a collection of selected papers by several respected colleagues in the field of language testing, who engaged in animated conversation, dialogue, and debate at the Language Testing Research Collo- quium (LTRC) 2005, in Ottawa. Part of that conversation is captured here. As the title of this volume suggests, each of the contributors has reconsidered language testing with the benefit of hindsight—looking back at its history, reflecting on personal experiences that highlighted key challenges or issues of concern, taking stock of both the limits and potential of current practice, and looking ahead to future possibilities. Central themes are evident across the chapters in this volume: • the ongoing challenge of construct definition in language testing and the need for interdisciplinary research, not only within the narrow (and often inward-looking) disciplines of language testing and second lan- guage acquisition but also across the broader fields of applied linguis- tics/language studies and education; • the expanding repertoire of research methods in language test devel- opment and validation, and the recognition of the important role that qualitative approaches can play in increasing our understanding of what tests do and what they measure; • the evolving and increasing