DOCSLIB.ORG

Assessing School and Teacher Effectiveness Using HLM and TIMSS Data from British Columbia and Ontario

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Assessing School and Teacher Effectiveness Using HLM and TIMSS Data from British Columbia and Ontario Avoiding Ecological Fallacy: Assessing School and Teacher Effectiveness Using HLM and TIMSS Data from British Columbia and Ontario By Yichun Wei B. Eng. University of Hunan M. Sc. University of Hunan A Thesis submitted to the Faculty of Graduate Studies of The University of Manitoba in partial fulfillment of the requirements of the degree of DOCTOR OF PHILOSOPHY Faculty of Education University of Manitoba Winnipeg, MB. Canada Copyright © 2012 by Yichun Wei ABSTRACT There are two serious methodological problems in the research literature on school effectiveness, the ecological problem in the analysis of aggregate data and the problem of not controlling for important confounding variables. This dissertation corrects these errors by using multilevel modeling procedures, specifically Hierarchical Linear Modeling (HLM), and the Canadian Trends in International Mathematics and Science Study (TIMSS) 2007 data, to evaluate the effect of school variables on the students’ academic achievement when a number of theoretically-relevant student variables have been controlled. In this study, I demonstrate that an aggregate analysis gives the most biased results of the schools’ impact on the students’ academic achievement. I also show that a disaggretate analysis gives better results, but HLM gives the most accurate estimates using this nested data set. Using HLM, I show that the physical resources of schools, which have been evaluated by school principals and classroom teachers, actually have no positive impact on the students’ academic achievement. The results imply that the physical resources are important, but an excessive improvement in the physical conditions of schools is unlikely to improve the students’ achievement. Most of the findings in this study are consistent with the best research literature. I conclude the dissertation by suggesting that aggregate analysis should not be used to infer relationships for individual students. Rather, multilevel analysis should be used whenever possible. Keywords: HLM, multilevel, aggregate, ecological fallacy, school, TIMSS, mathematics i ACKNOWLEDGEMENTS Near the end of this most challenging and rewarding endeavor, I want to thank all of the people involved in making this degree a reality. Foremost, I want to express my gratitude to the chair of my committee, Dr. Rodney Clifton, who guided me with such grace and dignity. Dr. Clifton, you have instilled confidence in me ever since our very first meeting. You also gave me the encouragement and gentle pushes I needed to keep me going when my spirit was down. Your demeanor had the calming and encouraging effect that I needed as I journeyed through this process. Thank you for sharing your knowledge and wisdom with me. I also want to thank Dr. Robert Renaud, Dr. Lance Roberts, and Dr. Colleen Metge for their input and guidance. The members of my advisory committee have been very patient and they have been dedicated to insuring that I complete this degree. I also want to thank Dr. Xin Ma, my external examiner, for his valuable suggestions. Special thanks go to Dr. Romulo Magsino, who was my advisor for the first year in the program. Finally, I want to thank Julianna Enns in the Faculty of Education for her patience with my many questions. ii DEDICATION This work is dedicated to my husband, Shuang Hu, who has given continuous support and encouragement throughout the entire process; to my parents, Lanwen Shang and Xin Wei, who have always supported my academic endeavors; and to my two sons, Garry and Owen, who were born during my PhD program and to whom I will hopefully instill a love of learning. iii CONTENTS ABSTRACT ......................................................................................................................... i ACKNOWLEDGEMENTS ................................................................................................ ii DEDICATION ................................................................................................................... iii LIST OF TABLES ............................................................................................................ vii LIST OF FIGURES .............................................................................................................x CHAPTERS 1 INTRODUCTION...........................................................................................................1 The Problem ...................................................................................................................4 Significance....................................................................................................................5 Limitations ...................................................................................................................12 Overview ......................................................................................................................16 2 ESTIMATING THE SCHOOLS’ CROSS-LEVEL EFFECTIVENESS ................19 Cross-level Effectiveness of Schooling .......................................................................19 Student Effects .......................................................................................................24 Teacher, Classroom, and School Effects ...............................................................26 District and Provincial Effects ...............................................................................28 Aggregate Modeling and the Ecological Fallacy .........................................................31 An Ecological Fallacy ............................................................................................32 Causes of An Ecological Fallacy ...........................................................................34 Three Tentative Solutions to Aggregate Inferences...............................................40 iv Multilevel Modeling of School Effectiveness .............................................................45 The General Model and Sub-Models of HLM .......................................................46 Advantages of Multilevel Modeling ......................................................................52 A Typology for Choosing Modeling Techniques ..................................................57 Summary ......................................................................................................................59 3 METHODOLOGY .......................................................................................................61 The Sample ..................................................................................................................62 Sampling Procedures in TIMSS.............................................................................62 Administration of TIMSS Tests .............................................................................64 The Canadian TIMSS Sample ...............................................................................65 Dependent Variable .....................................................................................................69 Independent Variables .................................................................................................71 Provincial Variable ................................................................................................72 School Variables ....................................................................................................72 Teacher/Classroom Variables ................................................................................84 Student Variables .................................................................................................109 Hypotheses .................................................................................................................118 Summary ....................................................................................................................118 4 RESULTS ....................................................................................................................120 Testing Two Assumptions .........................................................................................120 Missing Values.....................................................................................................121 Collinearity ..........................................................................................................127 v Aggregate Analysis ....................................................................................................135 Disaggregate Analysis ..............................................................................................138 HLM Analysis ...........................................................................................................145 Comparing the Three Methods .................................................................................150 The HLM Analysis in More Detail ............................................................................157 Mean Achievement ..............................................................................................159 Variance Components ..........................................................................................164 School Variables ..................................................................................................166 Teacher/Classroom Variables ..............................................................................167 Student Variables .................................................................................................169
Load more

Recommended publications
  • The Impact Factor Fallacy
    bioRxiv preprint doi: https://doi.org/10.1101/108027; this version posted February 20, 2017. The copyright holder for this preprint (which was not certified by peer review) is the author/funder. All rights reserved. No reuse allowed without permission. 1 Title: 2 The impact factor fallacy 3 Authors: 4 Frieder Michel Paulus 1; Nicole Cruz 2,3; Sören Krach 1 5 Affiliations: 6 1 Department of Psychiatry and Psychotherapy, Social Neuroscience Lab, University of Lübeck, 7 Ratzeburger Allee 160, D-23538 Lübeck, Germany 8 2 Department of Psychological Sciences, Birkbeck, University of London, London, UK 9 3 Laboratoire CHArt, École Pratique des Hautes Études (EPHE), Paris, France 10 Corresponding Authors: 11 Frieder Paulus, phone: ++49-(0)451- 31017527, email: [email protected] 12 Sören Krach, phone: ++49-(0)451-5001717, email: [email protected] 13 Department of Psychiatry and Psychotherapy, Social Neuroscience Lab, University of Lübeck, 14 Ratzeburger Allee 160, D-23538 Lübeck, Germany 15 Author contributions: 16 FMP, NC, and SK wrote the manuscript. 17 Running title: The impact factor fallacy 18 Word count abstract: 135 19 Word count manuscript (including footnotes): 4013 20 Number of tables: 1 21 Number of figures: 0 1 bioRxiv preprint doi: https://doi.org/10.1101/108027; this version posted February 20, 2017. The copyright holder for this preprint (which was not certified by peer review) is the author/funder. All rights reserved. No reuse allowed without permission. 22 Abstract 23 The use of the journal impact factor (JIF) as a measure for the quality of individual 24 manuscripts and the merits of scientists has faced significant criticism in recent years.
    [Show full text]
  • What Can Ecological Studies Tell Us About Death? Yehuda Neumark
    Neumark Israel Journal of Health Policy Research (2017) 6:52 DOI 10.1186/s13584-017-0176-x COMMENTARY Open Access What can ecological studies tell us about death? Yehuda Neumark Abstract: Using an ecological study design, Gordon et al. (Isr J Health Policy Res 6:39, 2017) demonstrate variations in mortality patterns across districts and sub-districts of Israel during 2008–2013. Unlike other epidemiological study designs, the units of analysis in ecological studies are groups of people, often defined geographically, and the exposures and outcomes are aggregated, and often known only at the population-level. The ecologic study has several appealing characteristics (suchasrelianceonpublic-domain anonymous data) alongside a number of important potential limitations including the often mentioned ‘ecological fallacy’.Advantagesand disadvantages of the ecological design are described briefly below. Keywords: Ecological fallacy, Ecological studies, Epidemiology, Mortality, Study design Main text In this journal, Gordon et al. [4] employ an ecological "The aim of epidemiology is to decipher nature with re- design to demonstrate variations in mortality patterns spect to human health and disease, and no one should across districts and sub-districts of Israel during the underestimate the complexities of epidemiological research" five-year period of 2008–2013. Standardized mortality [1]. To achieve this lofty and complex goal, the epide- ratios (SMRs) reveal a 25% excess of kidney disease re- miologic investigative toolbox contains various study lated deaths in the Haifa district, for example, while the methodologies including individual-based experimental risk of death from influenza/pneumonia is 25% lower designs (e.g., randomized controlled trial), individual- there than the national average.
    [Show full text]
  • A Statistical Guide for the Ethically Perplexed Preamble Introduction
    A Statistical Guide for the Ethically Perplexed Lawrence Hubert and Howard Wainer Preamble The meaning of \ethical" adopted here is one of being in accordance with the accepted rules or standards for right conduct that govern the practice of some profession. The profes- sions we have in mind are statistics and the behavioral sciences, and the standards for ethical practice are what we try to instill in our students through the methodology courses we offer, with particular emphasis on the graduate statistics sequence generally required for all the behavioral sciences. Our hope is that the principal general education payoff for competent statistics instruction is an increase in people's ability to be critical and ethical consumers and producers of the statistical reasoning and analyses faced in various applied contexts over the course of their careers. Introduction Generations of graduate students in the behavioral and social sciences have completed mandatory year-long course sequences in statistics, sometimes with difficulty and possibly with less than positive regard for the content and how it was taught. Prior to the 1960's, such a sequence usually emphasized a cookbook approach where formulas were applied un- thinkingly using mechanically operated calculators. The instructional method could be best characterized as \plug and chug," where there was no need to worry about the meaning of what one was doing, only that the numbers could be put in and an answer generated. This process would hopefully lead to numbers that could then be looked up in tables; in turn, p-values were sought that were less than the magical .05, giving some hope of getting an attendant paper published.
    [Show full text]
  • Some Topics in Health Economics That Become Distorted Cliches for Health Policy
    Health Policy Papers Collection 2017 – 09 (bis) SOME TOPICS IN HEALTH ECONOMICS THAT BECOME DISTORTED CLICHES FOR HEALTH POLICY Guillem López Casasnovas Professor at the Department of Economics and Business Universitat Pompeu Fabra Centre for Research in Health and Economics (CRES) Ricard Meneu Fundación Instituto de Investigación en Servicios de Salud (IISS) Centre de Recerca en Economia i Salut (CRES) Barcelona The Policy Papers Collection includes a series form, in Health Economics and Health Policy, carried out and selected by Researchers from the Centre for Research in Health and Economics of the University Pompeu Fabra (CRES - UPF). The Collection Policy Papers is part of an agreement between UPF and Novartis, whose activities provided not conditioned support of Novartis to the dissemination of studies and research of the CRES-UPF. "This is an Open Access article distributed under the terms of the Creative Commons Attribution License 4.0 International, which permits unrestricted use, distribution and reproduction in any medium provided that the original work is properly attributed" https://creativecommons.org/licenses/by/4.0/ Barcelona, October 2017 1 2 Health Policy Papers Collection 2017 – 09(bis) SOME TOPICS IN HEALTH ECONOMICS THAT BECOME DISTORTED CLICHES FOR HEALTH POLICY Guillem López i Casasnovas (1) y Ricard Meneu de Guillerna (2) (1) Department of Economics and Business. Centre for Research in Health and Economics (CRES). Universitat Pompeu Fabra (UPF). (2) Fundación Instituto de Investigación en Servicios de Salud (IISS). Centre for Research in Health and Economics (CRES) Introduction. The purpose of this paper is to pick up some of the best known clichés that surround the translation of the equivalent topics in health economics into health policy.
    [Show full text]
  • Appendix 1 a Great Big List of Fallacies
    Why Brilliant People Believe Nonsense Appendix 1 A Great Big List of Fallacies To avoid falling for the "Intrinsic Value of Senseless Hard Work Fallacy" (see also "Reinventing the Wheel"), I began with Wikipedia's helpful divisions, list, and descriptions as a base (since Wikipedia articles aren't subject to copyright restrictions), but felt free to add new fallacies, and tweak a bit here and there if I felt further explanation was needed. If you don't understand a fallacy from the brief description below, consider Googling the name of the fallacy, or finding an article dedicated to the fallacy in Wikipedia. Consider the list representative rather than exhaustive. Informal fallacies These arguments are fallacious for reasons other than their structure or form (formal = the "form" of the argument). Thus, informal fallacies typically require an examination of the argument's content. • Argument from (personal) incredulity (aka - divine fallacy, appeal to common sense) – I cannot imagine how this could be true, therefore it must be false. • Argument from repetition (argumentum ad nauseam) – signifies that it has been discussed so extensively that nobody cares to discuss it anymore. • Argument from silence (argumentum e silentio) – the conclusion is based on the absence of evidence, rather than the existence of evidence. • Argument to moderation (false compromise, middle ground, fallacy of the mean, argumentum ad temperantiam) – assuming that the compromise between two positions is always correct. • Argumentum verbosium – See proof by verbosity, below. • (Shifting the) burden of proof (see – onus probandi) – I need not prove my claim, you must prove it is false. • Circular reasoning (circulus in demonstrando) – when the reasoner begins with (or assumes) what he or she is trying to end up with; sometimes called assuming the conclusion.
    [Show full text]
  • List of Fallacies
    List of fallacies For specific popular misconceptions, see List of common if>). The following fallacies involve inferences whose misconceptions. correctness is not guaranteed by the behavior of those log- ical connectives, and hence, which are not logically guar- A fallacy is an incorrect argument in logic and rhetoric anteed to yield true conclusions. Types of propositional fallacies: which undermines an argument’s logical validity or more generally an argument’s logical soundness. Fallacies are either formal fallacies or informal fallacies. • Affirming a disjunct – concluding that one disjunct of a logical disjunction must be false because the These are commonly used styles of argument in convinc- other disjunct is true; A or B; A, therefore not B.[8] ing people, where the focus is on communication and re- sults rather than the correctness of the logic, and may be • Affirming the consequent – the antecedent in an in- used whether the point being advanced is correct or not. dicative conditional is claimed to be true because the consequent is true; if A, then B; B, therefore A.[8] 1 Formal fallacies • Denying the antecedent – the consequent in an indicative conditional is claimed to be false because the antecedent is false; if A, then B; not A, therefore Main article: Formal fallacy not B.[8] A formal fallacy is an error in logic that can be seen in the argument’s form.[1] All formal fallacies are specific types 1.2 Quantification fallacies of non sequiturs. A quantification fallacy is an error in logic where the • Appeal to probability – is a statement that takes quantifiers of the premises are in contradiction to the something for granted because it would probably be quantifier of the conclusion.
    [Show full text]
  • Ecological Inference
    P1: FZZ/FZZ P2: FZZ CB658-FMDVR CB654-KING-Sample CB658-KING-Sample.cls June 25, 2004 6:14 Ecological Inference Ecological Inference: New Methodological Strategies brings together a diverse group of scholars to survey the latest strategies for solving ecological inference problems in various fields. The last half decade has witnessed an explosion of research in ecological inference – the attempt to infer individual behavior from aggregate data. The uncertainties and the information lost in aggregation make ecological inference one of the most difficult areas of statistical inference, but such inferences are required in many academic fields, as well as by legislatures and the courts in redistricting, by businesses in marketing research, and by governments in policy analysis. Gary King is the David Florence Professor of Government at Harvard University. He also serves as the director of the Harvard–MIT Data Center and as a member of the steering committee of Harvard’s Center for Basic Research in the Social Sciences. He was elected president of the Society for Political Methodology and Fellow of the American Academy of Arts and Sciences. Professor King received his Ph.D. from the University of Wisconsin. He has won numerous awards for his work, including the Gosnell Prize for his book A Solution to the Ecological Inference Problem (1997), on which the research in this book builds. His home page can be found at http:// GKing.Harvard.edu. Ori Rosen is Assistant Professor of Statistics at the University of Pittsburgh. His research includes work on semiparametric regression models, applications of mixtures-of-experts neural network models in regression, and applications of Markov chain Monte Carlo methods.
    [Show full text]
  • Writing a Good History Paper
    WRITING A GOOD HISTORY PAPER Department of History Hamilton College ©Trustees of Hamilton College Acknowledgements This booklet bears one name, but it is really a communal effort. Iʼd like to thank the Director of the Writing Center, Sharon Williams, who origi- nally had the idea for a History Department writing guide, prodded me gently to get it done, and helped to edit and format it. My colleagues in the History Department read two drafts and made many valuable sugges- tions. Chris Takacs, a writing tutor in the class of 2005, helped me to see writing problems with a studentʼs eye. Yvonne Schick of the Print Shop carefully and professionally oversaw the production. Any limitations result solely from my own carelessness or stupidity. For the second edition, we have expanded the discussion of sources and listed some additional stylistic problems. --Alfred Kelly, for the History Department February, 2005 Cartoon reprinted with special permission of King Features Syndicate. Copyright Dan Piraro. Table of Contents Top Ten Reasons for Negative Comments on History Papers….......……1 Making Sure your History Paper has Substance……..........................…. 2 Common Marginal Remarks on Style, Clarity, Grammar, and Syntax Remarks on Style and Clarity.................................................................. 9 Remarks on Grammar and Syntax.........................................................14 Word and Phrase Usage Problems...........................................................19 Analyzing a Historical Document........................................................... 24 Writing a Book Review........................................................................... 26 Writing a Term Paper or Senior Thesis................................................... 27 Top Ten Signs that you may be Writing a Weak History Paper.............. 29 Final Advice............................................................................................ 29 Welcome to the History Department. You will fi nd that your history professors care a great deal about your writing.
    [Show full text]
  • Fashion Founded on a Flaw
    FASHION FOUNDED ON A FLAW For presentation at the IACCM Annual Conference Rotterdam School of Management, Erasmus University, 20—22 June, 2013 Brendan McSweeney Royal Holloway, University of London and Stockholm University What, I guess, we commonly believe • The world is not ‘flat’ (but how might we partition populations?) • The past matters (but we should not overstate lock-in due to initial conditions) • Culture matters (but what is ‘it’ and how does it operate?) • The idea of ‘culture’ is more easily invoked that defined. • I focus on one notion of culture: ‘national culture’ and as defined as ‘values’ • The ontological status of ‘national culture’; its depiction as bi-polar value ‘dimensions’; the validity of measurements of those dimensions; and the representativeness of samples, have been the object of considerable debate. • Here I addresses a different issue: a reliance on the ecological fallacy (Selvin, 1958). • The fallacious inference that the characteristics (concepts and/or metrics) of an aggregate (historically called ‘ecological’) level also describe those at a lower hierarchical level or levels. The Ecological Fallacy Supposed: What is true at one sub-national space is necessarily true at all other sub-national spaces. • Convent • Brothel (in the same country) • The objection here is not the generation of hypotheses from ecological comparisons. • Some of the recent discoveries of the causes of cancer (e.g. dietary factors) have their origin in the generation of such hypothesis from systematic international comparisons which were then investigated in lower level studies (Pearce, 2000). • The objection is to the doctrinaire (and invalid) transfer of aggregate results to lower levels i.e.
    [Show full text]
  • The Impact Factor Fallacy
    bioRxiv preprint doi: https://doi.org/10.1101/108027; this version posted February 20, 2017. The copyright holder for this preprint (which was not certified by peer review) is the author/funder. All rights reserved. No reuse allowed without permission. 1 Title: 2 The impact factor fallacy 3 Authors: 4 Frieder Michel Paulus 1; Nicole Cruz 2,3; Sören Krach 1 5 Affiliations: 6 1 Department of Psychiatry and Psychotherapy, Social Neuroscience Lab, University of Lübeck, 7 Ratzeburger Allee 160, D-23538 Lübeck, Germany 8 2 Department of Psychological Sciences, Birkbeck, University of London, London, UK 9 3 Laboratoire CHArt, École Pratique des Hautes Études (EPHE), Paris, France 10 Corresponding Authors: 11 Frieder Paulus, phone: ++49-(0)451- 31017527, email: [email protected] 12 Sören Krach, phone: ++49-(0)451-5001717, email: [email protected] 13 Department of Psychiatry and Psychotherapy, Social Neuroscience Lab, University of Lübeck, 14 Ratzeburger Allee 160, D-23538 Lübeck, Germany 15 Author contributions: 16 FMP, NC, and SK wrote the manuscript. 17 Running title: The impact factor fallacy 18 Word count abstract: 2221 19 Word count manuscript: 4014 20 Number of tables: 1 21 Number of figures: 0 1 bioRxiv preprint doi: https://doi.org/10.1101/108027; this version posted February 20, 2017. The copyright holder for this preprint (which was not certified by peer review) is the author/funder. All rights reserved. No reuse allowed without permission. 22 Abstract 23 The use of the journal impact factor (JIF) as a measure for the quality of individual 24 manuscripts and the merits of scientists has faced significant criticism in recent years.
    [Show full text]
  • The Renaissance of Political Culture Or the Renaissance of the Ecological Fallacy? Author(S): Mitchell A
    The Renaissance of Political Culture or the Renaissance of the Ecological Fallacy? Author(s): Mitchell A. Seligson Source: Comparative Politics, Vol. 34, No. 3, (Apr., 2002), pp. 273-292 Published by: Ph.D. Program in Political Science of the City University of New York Stable URL: http://www.jstor.org/stable/4146954 Accessed: 15/07/2008 12:10 Your use of the JSTOR archive indicates your acceptance of JSTOR's Terms and Conditions of Use, available at http://www.jstor.org/page/info/about/policies/terms.jsp. JSTOR's Terms and Conditions of Use provides, in part, that unless you have obtained prior permission, you may not download an entire issue of a journal or multiple copies of articles, and you may use content in the JSTOR archive only for your personal, non-commercial use. Please contact the publisher regarding any further use of this work. Publisher contact information may be obtained at http://www.jstor.org/action/showPublisher?publisherCode=phd. Each copy of any part of a JSTOR transmission must contain the same copyright notice that appears on the screen or printed page of such transmission. JSTOR is a not-for-profit organization founded in 1995 to build trusted digital archives for scholarship. We work with the scholarly community to preserve their work and the materials they rely upon, and to build a common research platform that promotes the discovery and use of these resources. For more information about JSTOR, please contact [email protected]. http://www.jstor.org The Renaissance of Political Cultureor the Renaissance of the Ecological Fallacy? MitchellA.
    [Show full text]
  • Culture As a Consequence: a Multilevel Multivariate Meta-Analysis of the Effects of Individual and Country Characteristics on Work-Related Cultural Values
    Culture as a consequence: A multi-level multivariate meta-analysis of the effects of individual and country characteristics on work-related cultural values By: Piers Steel, and Vasyl Taras Taras, V. & P. Steel. (2010). Culture as a Consequence: A Multilevel Multivariate Meta-Analysis of the Effects of Individual and Country Characteristics on Work-Related Cultural Values. Journal of International Management. 16(3): 211-233. Made available courtesy of Elsevier: http://www.elsevier.com ***Reprinted with permission. No further reproduction is authorized without written permission from Elsevier. This version of the document is not the version of record. Figures and/or pictures may be missing from this format of the document.*** Abstract: Culture as a consequence is a neglected topic. Addressing this, we explore what factors are related to and potentially shape culture, what explains cultural variations within countries, and what the relationship is between cultural values at the individual and national levels. To answer these questions, we use a multi-level multivariate meta-analysis of 508 studies. The findings indicate that national and individual cultural values may be determined by the micro characteristics of age, gender, education, and socio-economic status as well as the macro characteristics of wealth and freedom. This provides a basis for explaining cultural change, both at the individual and national levels. Also, up to 90% of the variance in cultural values is found to reside within countries, stressing that national averages poorly represent specific individuals. Keywords: Culture, Explaining culture, Cultural change, Meta-analysis, HLM Article: Over the last several decades, culture's consequences have received an explosion of interest in fields ranging from psychology and education to accounting and marketing.
    [Show full text]