Investigating the consequential validity of the Hanyu Shuiping Kaoshi (Chinese proficiency test) by using an Argument-based framework Shujiao Wang Department of Integrated Studies in Education McGill University, Montreal January 2018 A thesis submitted to McGill University in partial fulfilment of the requirements of the degree of doctor of philosophy Copyright © Shujiao Wang, 2018 ii ACKNOWLEDGEMENTS I would like to take some time to thank those without whom this project would never have been possible. Although it is just my name on the cover, many have contributed to the research in ways deserving of special thanks. I am grateful for the financial support I received for my research and funding for my Ph.D. program including but not limited to: • Paragon Testing Grant, 2017 • Herschel & Christine Victor Fellowship, 2014-2015 • SSHRC- Joseph-Armand Bombardier CGS Doctoral Scholarship, 2014-2017 • McGill’s Graduate Entrance Scholarship, 2013 I am also indebted to a long list of individuals who have personally supported me during my studies. First and foremost, I would like to express my deep appreciation and thanks to my supervisor Dr. Carolyn Turner, who has been a tremendous mentor for me. It has been an honor to be her last Ph.D. student. I appreciate all her contributions of time, ideas, and encouragement to make my Ph.D. experience rewarding and to allow me to grow as a researcher. I greatly appreciate the freedom she has given me to find my own path and the guidance and support she offered when needed. I am also thankful for the excellent example she has provided as a successful woman educator and professor. I would also like to express my gratitude to my co-supervisor Dr. Mela Sarkar. I want to thank you for your brilliant comments and suggestions on my dissertation from a different perspective. Our conversations about academia and life have helped shape my personal outlook and professional career. I further acknowledge the contributions of my dissertation advisory committee member Dr. Liying Cheng. Your dedicated work in washback has always inspired iii me. Your critical, insightful and detailed comments based on your expertise have helped me gain deep understanding of the complexity of consequential validity and how to adapt an argument- based framework. My time at McGill was made enjoyable and enriched by dedicated faculty, staff, and friends that have become a privileged part of my life. I am grateful for the time I spent with all of you. Special thanks go to Education and East Asian Studies teachers and students. Thank you all for your help and encouragement along the way. I have benefited from you at various stages of my studies. For example, I gained valuable research experience by assisting Dr. Beverly Baker. Thank you for your insightful and enriching comments on my proposals, presentations, and papers. I want to thank Mengting Pan for her great help in contacting participants during my data collection process. I also want to thank Cindy Chiang for your proofreading of the dissertation. Thanks also to Hui Wang for your help with the statistical analyses of the data. I am grateful as well to all the participants in this study. Your enthusiasm for teaching and learning Chinese always touched and inspired me to conduct research in this area, and even beyond what I ever could have been able to do. It is my hope and persistence that my study will provide useful information to second language teachers and learners, test developers, and other stakeholders. Last but most important of all, I would like to thank my family for all their love, encouragement, and support. To my parents and parents-in-law, thank you for supporting all my pursuits. Words cannot express how grateful I am to you for all of the sacrifices that you have made on my behalf. And for my loving, supportive, encouraging, and patient husband Zhenhuan, your faithful support throughout this PhD experience is deeply appreciated. Most of all, special thanks to my beloved son Zeyu. You are such a good boy who always cheers me up. I love you. iv ABSTRACT In recent years, China’s rising global power has led to an international increase in Chinese language learning. The national standardized test of Chinese language proficiency for non-native speakers, the Hanyu Shuiping Kaoshi (HSK), literally “Chinese Proficiency Test,” has played a vital role in certifying language proficiency for higher education and professional purposes. The multiple uses of the HSK have generated growing concerns about its validity, especially the reformed HSK’s (post-2009) consequential validity. Employing Bachman and Palmer’s (2010) Assessment Use Argument (AUA) framework, this mixed methods sequential exploratory (MMSE) study investigates the HSK’s micro- and macro-level consequences, as well as how and to what extent the test affects Chinese as a second language (CSL) teaching and learning. In Phase I of this MMSE study the official HSK documents were analyzed by content analysis; interviews with 12 test stakeholders were then conducted and analyzed by a two-cycle qualitative coding approach (Saldaña, 2009). In Phase II, 136 CSL/CFL teachers and 512 HSK test-takers participated in a questionnaire, and the data were analyzed by using exploratory factor analysis (EFA) and structural equation modeling (SEM); classroom observations were also conducted and analyzed to contextualize the quantitative results. Phase III involved two exploratory questionnaires and interviews with 35 administrative personnel who use the HSK to inform academic and employment decisions, and the data were analysed through statistical (e.g., descriptive statistics) and qualitative methods (e.g., grounded theory). The results of the MMSE study highlighted the complexity of the HSK’s consequences and washback effects. They indicated that although the HSK had limited effects on teaching, it was somewhat successful in its goal of promoting CSL/CFL learning. In general, HSK scores and other related information (e.g., score report, level interpretation) also provided users with relevant, useful, and meaningful v data for candidate selection. Overall, based on the HSK’s AUA conceptual framework, the findings provided evidence that Claim 1 (Consequences), Claim 2 (Decisions), and Claim 3 (Interpretations) were partially supported, in that the test developers’ intended goals for the HSK were only achieved to a certain degree. This study helped unpack the consequential validity of the HSK in the CSL context, shed light on understanding the washback effects of the HSK, fleshed out the values underlying the multiple interpretations and uses of the test, and pointed to implications for the HSK developers and future consequence/impact/washback research. vi RÉSUMÉ Ces dernières années, la puissance mondiale croissante de la Chine a conduit à une augmentation internationale de l’apprentissage de la langue chinoise. Le test standardisé national de maîtrise de la langue chinoise pour les non-natifs, le Hanyu Shuiping Kaoshi (HSK), littéralement « test de compétence en chinois », a joué un rôle essentiel dans la certification des compétences linguistique et professionnelle. Les multiples utilisations du HSK ont généré de plus en plus d’inquiétudes quant à sa validité, en particulier la validité corrélative du HSK (post- 2009). Utilisant le cadre d’évaluation de l’utilisation [AUA] de Bachman et Palmer (2010), cette étude exploratoire séquentielle à méthodes mixtes (MMSE) examine les conséquences micro et macro du HSK, ainsi que comment et dans quelle mesure le test affecte le chinois en tant que langue seconde de l’enseignement et d’apprentissage (CSL). Dans la phase I de cette étude MMSE, nous avons examiné les contenus des documents officiels de HSK ; des entrevues avec 12 intervenants ont ensuite été réalisées et analysées selon une approche de codage qualitatif en deux cycles [Saldaña, 2009]. Au cours de la phase II, 136 enseignants du programme CSL/CFL et 512 participants au test HSK ont répondu à un questionnaire, ce qui nous ont permis de recueillir des données à l’aide de l’analyse factuelle exploratoire (EFA) et de la modélisation par équation structurelle (SEM) ; des observations en classe ont également été menées et analysées afin de contextualiser les résultats quantitatifs. La phase III comportait deux questionnaires exploratoires et des entretiens avec 35 membres du personnel administratif qui utilisaient la HSK pour éclairer les décisions académiques et professionnelles, et les données étaient analysées en utilisant des méthodes statistiques (statistiques descriptives, par exemple) et des méthodes qualitatives comme la théorisation ancrée. Les résultats de l’étude MMSE ont mis en évidence la complexité des conséquences du HSK et des effets de washback. Ils ont indiqué que bien que le vii HSK ait eu des effets limités sur l’enseignement, il a plutôt réussi à promouvoir l’apprentissage CSL/CFL. En général, les scores HSK et d’autres informations connexes (par exemple, rapport de score, interprétation de niveau) ont également fourni aux utilisateurs des données pertinentes, et significatives pour la sélection des candidats. Dans l’ensemble, selon le cadre conceptuel de l’ASA de HSK, les constatations ont démontré que les revendications 1 (Conséquences), 2 (Décisions) et 3 (Interprétations) étaient partiellement justifiées, car les objectifs des développeurs de tests pour le HSK étaient seulement atteints dans une certaine mesure. Cette étude a permis de comprendre la validité et les impacts de la HSK dans le contexte