Avoiding Ecological Fallacy: Assessing School and Teacher Effectiveness Using HLM and TIMSS Data from British Columbia and Ontario By Yichun Wei B. Eng. University of Hunan M. Sc. University of Hunan A Thesis submitted to the Faculty of Graduate Studies of The University of Manitoba in partial fulfillment of the requirements of the degree of DOCTOR OF PHILOSOPHY Faculty of Education University of Manitoba Winnipeg, MB. Canada Copyright © 2012 by Yichun Wei ABSTRACT There are two serious methodological problems in the research literature on school effectiveness, the ecological problem in the analysis of aggregate data and the problem of not controlling for important confounding variables. This dissertation corrects these errors by using multilevel modeling procedures, specifically Hierarchical Linear Modeling (HLM), and the Canadian Trends in International Mathematics and Science Study (TIMSS) 2007 data, to evaluate the effect of school variables on the students’ academic achievement when a number of theoretically-relevant student variables have been controlled. In this study, I demonstrate that an aggregate analysis gives the most biased results of the schools’ impact on the students’ academic achievement. I also show that a disaggretate analysis gives better results, but HLM gives the most accurate estimates using this nested data set. Using HLM, I show that the physical resources of schools, which have been evaluated by school principals and classroom teachers, actually have no positive impact on the students’ academic achievement. The results imply that the physical resources are important, but an excessive improvement in the physical conditions of schools is unlikely to improve the students’ achievement. Most of the findings in this study are consistent with the best research literature. I conclude the dissertation by suggesting that aggregate analysis should not be used to infer relationships for individual students. Rather, multilevel analysis should be used whenever possible. Keywords: HLM, multilevel, aggregate, ecological fallacy, school, TIMSS, mathematics i ACKNOWLEDGEMENTS Near the end of this most challenging and rewarding endeavor, I want to thank all of the people involved in making this degree a reality. Foremost, I want to express my gratitude to the chair of my committee, Dr. Rodney Clifton, who guided me with such grace and dignity. Dr. Clifton, you have instilled confidence in me ever since our very first meeting. You also gave me the encouragement and gentle pushes I needed to keep me going when my spirit was down. Your demeanor had the calming and encouraging effect that I needed as I journeyed through this process. Thank you for sharing your knowledge and wisdom with me. I also want to thank Dr. Robert Renaud, Dr. Lance Roberts, and Dr. Colleen Metge for their input and guidance. The members of my advisory committee have been very patient and they have been dedicated to insuring that I complete this degree. I also want to thank Dr. Xin Ma, my external examiner, for his valuable suggestions. Special thanks go to Dr. Romulo Magsino, who was my advisor for the first year in the program. Finally, I want to thank Julianna Enns in the Faculty of Education for her patience with my many questions. ii DEDICATION This work is dedicated to my husband, Shuang Hu, who has given continuous support and encouragement throughout the entire process; to my parents, Lanwen Shang and Xin Wei, who have always supported my academic endeavors; and to my two sons, Garry and Owen, who were born during my PhD program and to whom I will hopefully instill a love of learning. iii CONTENTS ABSTRACT ......................................................................................................................... i ACKNOWLEDGEMENTS ................................................................................................ ii DEDICATION ................................................................................................................... iii LIST OF TABLES ............................................................................................................ vii LIST OF FIGURES .............................................................................................................x CHAPTERS 1 INTRODUCTION...........................................................................................................1 The Problem ...................................................................................................................4 Significance....................................................................................................................5 Limitations ...................................................................................................................12 Overview ......................................................................................................................16 2 ESTIMATING THE SCHOOLS’ CROSS-LEVEL EFFECTIVENESS ................19 Cross-level Effectiveness of Schooling .......................................................................19 Student Effects .......................................................................................................24 Teacher, Classroom, and School Effects ...............................................................26 District and Provincial Effects ...............................................................................28 Aggregate Modeling and the Ecological Fallacy .........................................................31 An Ecological Fallacy ............................................................................................32 Causes of An Ecological Fallacy ...........................................................................34 Three Tentative Solutions to Aggregate Inferences...............................................40 iv Multilevel Modeling of School Effectiveness .............................................................45 The General Model and Sub-Models of HLM .......................................................46 Advantages of Multilevel Modeling ......................................................................52 A Typology for Choosing Modeling Techniques ..................................................57 Summary ......................................................................................................................59 3 METHODOLOGY .......................................................................................................61 The Sample ..................................................................................................................62 Sampling Procedures in TIMSS.............................................................................62 Administration of TIMSS Tests .............................................................................64 The Canadian TIMSS Sample ...............................................................................65 Dependent Variable .....................................................................................................69 Independent Variables .................................................................................................71 Provincial Variable ................................................................................................72 School Variables ....................................................................................................72 Teacher/Classroom Variables ................................................................................84 Student Variables .................................................................................................109 Hypotheses .................................................................................................................118 Summary ....................................................................................................................118 4 RESULTS ....................................................................................................................120 Testing Two Assumptions .........................................................................................120 Missing Values.....................................................................................................121 Collinearity ..........................................................................................................127 v Aggregate Analysis ....................................................................................................135 Disaggregate Analysis ..............................................................................................138 HLM Analysis ...........................................................................................................145 Comparing the Three Methods .................................................................................150 The HLM Analysis in More Detail ............................................................................157 Mean Achievement ..............................................................................................159 Variance Components ..........................................................................................164 School Variables ..................................................................................................166 Teacher/Classroom Variables ..............................................................................167 Student Variables .................................................................................................169