VALIDATING A SET OF JAPANESE EFL PROFICIENCY TESTS: DEMONSTRATING LOCALLY DESIGNED TESTS MEET INTERNATIONAL STANDARDS Jamie Dunlea This is a digitised version of a dissertation submitted to the University of Bedfordshire. It is available to view only. This item is subject to copyright. VALIDATING A SET OF JAPANESE EFL PROFICIENCY TESTS: DEMONSTRATING LOCALLY DESIGNED TESTS MEET INTERNATIONAL STANDARDS JAMIE DUNLEA A thesis submitted to the University of Bedfordshire in fulfillment of the requirements for the degree of Doctor of Philosophy University of Bedfordshire Centre for Research in English Language Learning and Assessment (CRELLA) December 2015 1 VALIDATING A SET OF JAPANESE EFL PROFICIENCY TESTS: DEMONSTRATING LOCALLY DESIGNED TESTS MEET INTERNATIONAL STANDARDS JAMIE DUNLEA ABSTRACT This study applied the latest developments in language testing validation theory to derive a core body of evidence that can contribute to the validation of a large-scale, high-stakes English as a Foreign Language (EFL) testing program in Japan. The testing program consists of a set of seven level-specific tests targeting different levels of proficiency. This core aspect of the program was selected as the main focus of this study. The socio-cognitive model of language test development and validation provided a coherent framework for the collection, analysis and interpretation of evidence. Three research questions targeted core elements of a validity argument identified in the literature on the socio-cognitive model. RQ 1 investigated the criterial contextual and cognitive features of tasks at different levels of proficiency, Expert judgment and automated analysis tools were used to analyze a large bank of items administered in operational tests across multiple years. RQ 2 addressed empirical item difficulty across the seven levels of proficiency. An innovative approach to vertical scaling was used to place previously administered items from all levels onto a single Rasch-based difficulty scale. RQ 3 used multiple standard-setting methods to investigate whether the seven levels could be meaningfully related to an external proficiency framework. In addition, the study identified three subsidiary goals: firstly, to evaluate the efficacy of applying international standards of best practice to a local context: secondly, to critically evaluate the model of validation; and thirdly, to generate insights directly applicable to operational quality assurance. The study provides evidence across all three research questions to support the claim that the seven levels in the program are distinct. At the same time, the results provide insights into how to strengthen explicit task specification to improve consistency across levels. This study is the largest application of the socio-cognitive model in terms of the amount of operational data analyzed, and thus makes a significant contribution to the ongoing study of validity theory in the context of language testing. While the study demonstrates the efficacy of the socio-cognitive model selected to drive the research design, it also provides recommendations for further refining the model, with implications for the theory and practice of language testing validation, i Acknowledgements My deepest gratitude and respect goes to all of my former colleagues who work on the EIKEN tests. Their dedication to all of the details, large and small, is at the heart of the program, and ultimately is what made this study possible. Much of their hard work often goes unheralded behind the senses. I hope this study has contributed to shining a light on the professionalism and high standards they possess. To my supervisors, Professor Cyril Weir, Dr Fumiyo Nakatsuhara, and Dr Chihiro Inoue, I will always be grateful for the opportunity they gave me to start this journey, their professional and academic advice on the way, and their friendship, without any of which the journey may never have been completed. I owe a special debt of appreciation to Professor Barry O’Sullivan, whose support and encouragement has helped me keep sight of the goal and stay the course. And to my wife and my daughter, who have travelled with me, thank you. Your patience has been tested the most, your perseverance has been the longest, but your faith also the strongest, and your support the most important. ii Table of Contents List of Tables ............................................................................................................ vi List of Figures .......................................................................................................... ix Chapter 1 Introduction .............................................................................................. 1 1.1 Aims of the Thesis .................................................................................................. 1 1.2 Rationale for a Validation Study of the EIKEN Testing Program ................................ 3 1.2.1 Overview of the EIKEN Testing program .......................................................... 3 1.2.2 The structure of the EIKEN tests ....................................................................... 4 1.2.3 Changes in the context of use for the EIKEN testing program ............................. 6 1.3 Research Questions .............................................................................................. 14 1.4 Methodology ....................................................................................................... 17 1.5 Overview of the Thesis ......................................................................................... 20 Chapter 2 Literature Review ..................................................................................... 22 2.1 Introduction ......................................................................................................... 22 2.2. Validation Theory and Practice: Selecting a Model for the Study ............................. 22 2.2.1 The acceptance of validity as a unified concept ................................................ 22 2.2.2 Building on a growing consensus .................................................................... 24 2.2.3 Problems with the Messickian legacy .............................................................. 26 2.2.4 Building on Messick: recent trends in the theory of validation ........................... 38 2.2.5 The socio-cognitive model for language test development and validation ...... 45 2.3 Research Question 1: Criterial Features of Reading Test Tasks................................. 54 2.4 Research Question 2: Vertical Scaling and Scoring Validity ..................................... 70 2.4.1 Statistics for evaluating the technical quality of the EIKEN tests ....................... 70 2.4.2 Vertical scaling .............................................................................................. 78 2.5 Research Question 3: Linking Examinations to the CEFR ....................................... 83 Chapter 3 RQ1: Criterial Features of Test Tasks at Each Grade ................................. 93 3.1 Introduction ......................................................................................................... 93 3.2 Methodology ....................................................................................................... 93 3.2.1 Overview ...................................................................................................... 93 3.2.2 Parameters tagged through expert judgment ..................................................... 98 3.2.3 Parameters tagged through automated text analysis ........................................ 101 3.3 Results .............................................................................................................. 107 iii 3.3.1 Expert judgment tags .................................................................................... 107 3.3.2. Automated analysis ...................................................................................... 129 3.4 Conclusions Regarding RQ1 ................................................................................ 148 Chapter 4 RQ2: The Empirical Difficulty of EIKEN Grades .................................... 151 4.1 Introduction ........................................................................................................ 151 4.3 The Use of Vertical Scaling to Investigate RQ3 ..................................................... 159 4.3.1 Rationale for vertical scaling study ................................................................ 159 4.3.2 Methodology for RQ2 ................................................................................... 160 4.3.3 Results ......................................................................................................... 181 4.3.4 Conclusions regarding item difficulty across EIKEN grades ............................ 199 Chapter 5 RQ3: Linking the EIKEN Grades to the CEFR ....................................... 200 5.1 Introduction ........................................................................................................ 200 5.2 Methodology ...................................................................................................... 200 5.2.1 Standardisation ............................................................................................. 201 5.2.2 Validation ..................................................................................................... 206 5.3 Standard setting panel 1 ......................................................................................