EXAMINING SECOND LANGUAGE READING: A CRITICAL REVIEW OF THE SINGAPORE- CAMBRIDGE GENERAL CERTIFICATE OF EDUCATION ORDINARY-LEVEL CHINESE LANGUAGE EXAMINATION by YUN-YEE CHEONG A thesis submitted in fulfillment of the requirements for the degree of Doctor of Philosophy University College London Institute of Education 2018 Supervisors: Professor David Scott, Professor Paul Newton, Ms Katharine Carruthers OBE 1 I hereby declare that, except where explicit attribution is made, the work presented in this thesis is entirely my own. Yun-Yee Cheong 2018 Word count (exclusive of references and appendices): 92,897 words 2 Abstract This mixed methods study critically reviews how the Singapore-Cambridge General Certificate of Education Ordinary-Level Chinese Language Examination (GCE 1162) examines second language reading. The main research question asks, ‘To what degree have the intended measurement objectives of the GCE 1162 reading examination been achieved?’ Four sub research questions address issues of specifications and administration, test-taker characteristics, cognitive parameters and contextual parameters. Resources drawn on include Singapore Ministry of Education and Singapore Examinations and Assessment Board documents, specifically, examination information booklets, syllabuses, committee reports and annual reviews. Subject matter experts were appointed to analyse the reading comprehension passages and test items from 22 sets of GCE 1162 reading examination papers from 2006 to 2016. Semi-structured interviews were carried out with 22 stakeholders involved in coordination, test design, item construction, marking and reviewing. The interviewees included members of an elite policy group with privileged access to test specifications and procedures. Further interviews were carried out with secondary school Chinese language teachers and students, whose perspectives are seldom considered in validation processes. Opinions were also sought from experts in the field of Chinese as a second language, reading and assessment. The study begins with an account of the concepts of validity and reading constructs. Chapter 2 discusses the Singapore education and examination system, foregrounding the history of Chinese language education and the bilingual policy introduced in 1966. A methodology chapter follows. Chapters 4 to 8 address separately each of the four sub research questions in which claims, assumptions, supporting evidence and rebuttals are presented. The final chapter, Chapter 9, addresses a posteriori inferences, including scoring, criterion-related components, and washback and impact. A cautious conclusion is drawn, namely that the measurement quality of the GCE 1162 reading examination is at a moderately unsatisfactory level. 3 Impact statement This study is among the first to provide an in-depth validation analysis of a national examination in the Singaporean context, specifically, the Singapore-Cambridge General Certificate of Education Ordinary-Level Chinese Language Examination (GCE 1162). The primary aim of the study is to evaluate the degree to which the measurement objectives of its reading examination are achieved. In addressing this question, several threats to validity are identified which the Singapore Examinations and Assessment Board (SEAB) and Ministry of Education (MOE) need to help resolve. Rectifying the identified threats to validity will strengthen the measurement quality or validity of the examination. It is argued that SEAB and MOE should not only clarify the purposes of the GCE 1162 reading examination by presenting the rationale underpinning how examination scores are intended to be interpreted and consequently used but also spearhead extensive theoretical and empirical research on Chinese as a second language reading. Evidence suggests that the GCE 1162 reading examination could encompass more varied dimensions of reading assessment such as multiple text reading for problem- solving, and reading volume and interest and that test designers need to increase the proportion and weightage of higher-order thinking items. Further evidence suggests the need for the reading examination to be more authentic and relevant to Singaporean adolescents. In addition, tangible steps should be taken to ensure that mark schemes are improved and made publicly available. Mark schemes should also state explicitly the principles to which markers must abide in order to facilitate consistent scoring. Together, these findings support the need for a culture of transparency and public access to procedures. The implications and impact of this study go beyond uncovering the strengths and limitations of the GCE 1162 reading examination by suggesting ways in which to improve measurement quality at the micro-level. The study illuminates perspectives on and understandings of validity and validation, the reading construct and the Singaporean context. While it might not be feasible for a full-scale validation study 4 such as this to be carried out routinely for all subjects examined by SEAB, the study offers a research foundation and viable frameworks from which smaller-scale and more routine validation studies could be developed. Analysis of the GCE 1162 reading examination reveals a bi-directional relationship at the meso-level between the validation process and the context in which it is carried out. The specific social, political, cultural and educational environments in which validation occurs inevitably influences its feasibility and meaningfulness. At the macro-level, the study, which draws on a unitary view of validity, demonstrates the adequacy of Weir’s (2005) socio-cognitive validity framework and Kane’s (2009, 2006) argument-based approach to validation for amassing validity evidence in ways which are feasible. 5 Acknowledgements Till, with a sudden sharp hot stink of fox, It enters the dark hole of the head. The window is starless still; the clock ticks, The page is printed. The thought fox, Ted Hughes Pursuing a doctoral degree is a test of grit and endurance. I am indebted to my three supervisors at University College London Institute of Education, Professor David Scott, Professor Paul Newton and Confucius Institute Director Katharine Carruthers OBE for their intellectual inspiration and systematic guidance. Their keen sense of academic insight and rigour are qualities that I hope to emulate. My heartfelt appreciation goes to Professor Catherine Snow, too, for her patience and professional advice during my year as a Visiting Fellow with the Harvard Graduate School of Education. Thank you for believing in me. I would also like to thank the Singapore Ministry of Education and the Tan Kah Kee Foundation for funding my research. Their generous support has enabled me to carry out an in-depth validation study of a national examination in the Singaporean context. I would further like to thank my wonderful ex-colleagues from the National Institute of Education, Singapore Centre for Chinese Language and Singapore Ministry of Education, especially Professor Chew Cheng Hai, Professor Chin Chee Kuen, Professor Tan Chee Lay, Professor Aw Guat Poh, Professor Ong Yong Peng, Professor Tan Heng Kiat Kelvin, Mrs Catherine Neubronner-Lim, Principal Benedict Keh Chin Chuan and the Master Teachers who fuelled my enthusiasm and interest in the subject of testing and assessment. I am also grateful to Professor Leong Weng Kee, Professor Zhu Xin Hua, Professor Gordon Stobart, Principal Cheah Chak Mun, Dr Leong See Cheng, Ms Yio Puay Ching and Mr Stuart Shaw who were instrumental in shaping my knowledge of the concepts of validity, the reading construct and the Singaporean context. 6 My appreciation also goes to my course mates and friends who have enlightened me with their unique insights, especially Ai Lian, Chee Wah, Jenifer, Joo Yeon, Judy, Natalia, Qiuyan, Samantha, Weng Fong, Xin Huan, Xiu-Liu, Yuka, Cher Liek, De Cheng, Kwong Tung, Raymond, Ronald, Sheng Yuan and Wael. Finally, I would like to thank the interviewees and subject matter experts for participating in my research. A shout out also goes to all my students in the eight years I was a teacher and lecturer. Thank you for having taught me that not everything that matters can be measured, and not everything that is measured matters. 7 To my family, especially my parents and grandparents for their unwavering love and support 8 Table of contents Abstract ...................................................................................................................................3 Acknowledgments ..................................................................................................................6 List of figures ....................................................................................................................... 14 List of abbreviations ............................................................................................................ 17 Chapter 1: Overview of research .................................................................................... 18 1.1 Introduction ............................................................................................................. 18 1.2 Background to the problem .................................................................................... 19 1.3 Validity ................................................................................................................... 22 1.3.1 A gestational period (mid-1800s-1920) .......................................................... 22 1.3.2 A