The University of Southern Mississippi The Aquila Digital Community Dissertations Summer 8-2015 Measuring Student Growth in K–12 Schools Using Item Response Theory Within Structural Equation Models Kenneth Lee Thompson University of Southern Mississippi Follow this and additional works at: https://aquila.usm.edu/dissertations Part of the Educational Assessment, Evaluation, and Research Commons Recommended Citation Thompson, Kenneth Lee, "Measuring Student Growth in K–12 Schools Using Item Response Theory Within Structural Equation Models" (2015). Dissertations. 122. https://aquila.usm.edu/dissertations/122 This Dissertation is brought to you for free and open access by The Aquila Digital Community. It has been accepted for inclusion in Dissertations by an authorized administrator of The Aquila Digital Community. For more information, please contact [email protected]. The University of Southern Mississippi MEASURING STUDENT GROWTH IN K – 12 SCHOOLS USING ITEM RESPONSE THEORY WITHIN STRUCTURAL EQUATION MODELS by Kenneth Lee Thompson Abstract of a Dissertation Submitted to the Graduate School of The University of Southern Mississippi in Partial Fulfillment of the Requirements for the Degree of Doctor of Philosophy August 2015 ABSTRACT MEASURING STUDENT GROWTH IN K – 12 SCHOOLS USING ITEM RESPONSE THEORY WITHIN STRUCTURAL EQUATION MODELING by Kenneth Lee Thompson August 2015 The use of test-based accountability has expanded beyond measurements of school effectiveness to include measurements of teacher effectiveness. However, whereas the use of test-based accountability has expanded, the understanding of the statistical methodologies used in accountability systems has not kept pace. Currently, Student Growth Percentiles and value-added modeling are the most prevalent methodologies for estimating annual student growth. Each of these methodologies is regression-based and relies on scale scores from standardized assessments. Given the prevalence of Item Response Theory in statewide assessment programs, these scale scores often result from Item Response Theory scaling practices. Grounded in earlier work of Brockman (2011), Chiu and Camilli (2013), and Lu, Thomas, and Zumbo (2005), concerning error related to Item Response Theory-based scale scores, this study considers using Item Response Theory as the measurement model in a structural equation model by including simulated item response patterns as indicators of ability. Data were simulated using parameters from the Mississippi Curriculum Test, Second Edition. Separate structural equation models for language arts and mathematics were considered. Upon examining the fit of each model, results indicated a good fit for the measurement ii model in language arts and in mathematics. Results also indicated a good fit for the overall structural equation model, but none of the structural relationships were statistically significant. Additional results and implications of this study are discussed. iii COPYRIGHT BY KENNETH LEE THOMPSON 2015 The University of Southern Mississippi MEASURING STUDENT GROWTH IN K – 12 SCHOOLS USING ITEM RESPONSE THEORY WITHIN STRUCTURAL EQUATION MODELS by Kenneth Lee Thompson A Dissertation Submitted to the Graduate School of The University of Southern Mississippi in Partial Fulfillment of the Requirements for the Degree of Doctor of Philosophy Approved: __________________________________________ Dr. Kyna J. Shelley, Committee Chair Professor, Educational Studies and Research __________________________________________ Dr. Thomas J. Lipscomb, Committee Member Professor, Educational Studies and Research __________________________________________ Dr. Lilian H. Hill, Committee Member Associate Professor, Educational Studies and Research _________________________________________ Dr. Richard S. Mohn Jr., Committee Member Associate Professor, Educational Studies and Research _________________________________________ Dr. Karen S. Coats Dean of the Graduate School August 2015 DEDICATION There is an old adage that as we grow older, our parents get smarter. This dissertation is dedicated to my parents for believing in me when I thought they were the smartest people on earth, for believing in me even when I thought they were far from the smartest people on earth, and for still believing in me when I found out I was right to begin with. Thank you, and I love you for sticking with me to the end; without you, I would not be the person I am today. iv ACKNOWLEDGMENTS I would like to acknowledge the Department of Educational Studies and Research for providing me with opportunities to expand my horizons and to excel as a student. Any successes I may have enjoyed are a direct result of this amazing department. I wish to thank the members of my committee who were more than generous with their vast expertise and limited time. Thank you to Dr. Richard Mohn for joining me on this long, strange trip, to Dr. Thomas Lipscomb for making sure that I say what I mean to say, and to Dr. Lilian Hill for always making sure that I look deeper than the surface. Thanks to each of you for the countless hours reading, commenting, and pushing me beyond my comfort zone. Finally, I offer my sincerest gratitude and deepest thanks to my advisor, my committee chair, and my mentor, Dr. Kyna Shelley. In addition to the countless hours reading and commenting, I want to thank you for always being there when I needed the extra push, or when I needed encouragement. Most of all, I want to thank you for your patience as you guided me from program applicant to writing a dissertation. v TABLE OF CONTENTS ABSTRACT ........................................................................................................................ ii DEDICATION ................................................................................................................... iv ACKNOWLEDGMENTS ...................................................................................................v LIST OF TABLES ........................................................................................................... viii LIST OF ILLUSTRATIONS ............................................................................................. ix LIST OF ABBREVIATIONS ..............................................................................................x CHAPTER I. INTRODUCTION .......................................................................................1 Statement of the Problem Purpose of the Study II. REVIEW OF RELATED LITERATURE ...................................................9 Measuring Student Growth Measurement Practices in Large Scale Assessment Structural Equation Modeling IRT as Measurement Model III. METHODOLOGY ....................................................................................38 Phase 1: Response Data Simulation Phase 2: Dimensionality Analysis Phase 3: Calibration and Scaling Phase 4: Student Growth Percentiles Phase 5: Structural Equation Modeling IV. ANALYSIS OF DATA..............................................................................51 Phase 2: Dimensionality Analysis Phase 3: Calibration and Scaling Phase 5: Structural Equation Modeling V. DISCUSSION ............................................................................................58 Limitations and Suggestions for Future Research APPENDICES ...................................................................................................................65 REFERENCES ................................................................................................................124 LIST OF TABLES Table 1. 2011 MCT2 Mean and Standard Deviation ...........................................................42 2. Assessment Constructs and Number of Items .......................................................43 3. Variables Required for SGPs .................................................................................47 4. Criteria for Factorability of Original Test ..............................................................52 5. Initial Eigenvalues for First Four Factors ..............................................................52 6. Initial Percentage of Variance Explained by First Four Factors ............................53 7. Number of Items per Test after Principle Components Analysis ..........................53 8. Goodness-of-Fit Indices For Constructs Resulting from Confirmatory Factor Analysis..................................................................................................................55 9. Goodness-of-Fit Indices Using Multiple Factors Suggested by PCA ...................55 10. Goodness-of-Fit Indices for Proposed Structural Equation Models ......................56 11. Structural Path Estimates and Statistical Significance Levels ...............................57 12. Estimated Correlation Matrices for Latent Variables ............................................57 viii LIST OF ILLUSTRATIONS Figure 1. Multiple Regression Illustration ............................................................................25 2. Illustration of a Construct ......................................................................................26 3. Illustration of a Structural Equation Model ...........................................................27 4. Typical SEM Implementation ................................................................................27 ix LIST OF ABBREVIATIONS 1PL 1 Parameter Logistic Model 2PL 2 Parameter Logistic Model 2PPC 2 Parameter Partial Credit Model 3PL 3 Parameter