Eastern Michigan University DigitalCommons@EMU Master's Theses, and Doctoral Dissertations, and Master's Theses and Doctoral Dissertations Graduate Capstone Projects 2019 Assessing student success: Predictive validity of the ACT science subscore Meāgan Nichole Treadway Follow this and additional works at: https://commons.emich.edu/theses Part of the Educational Leadership Commons Recommended Citation Treadway, Meāgan Nichole, "Assessing student success: Predictive validity of the ACT science subscore" (2019). Master's Theses and Doctoral Dissertations. 957. https://commons.emich.edu/theses/957 This Open Access Dissertation is brought to you for free and open access by the Master's Theses, and Doctoral Dissertations, and Graduate Capstone Projects at DigitalCommons@EMU. It has been accepted for inclusion in Master's Theses and Doctoral Dissertations by an authorized administrator of DigitalCommons@EMU. For more information, please contact [email protected]. Running head: ASSESSING STUDENT SUCCESS Assessing Student Success: Predictive Validity of the ACT Science Subscore by Meāgan Nichole Treadway Dissertation Submitted to the College of Education Eastern Michigan University in partial fulfillment of the requirements for the degree of DOCTOR OF PHILOSOPHY in Educational Leadership Dissertation Committee: Ronald Williamson, Ed.D., Chair David Anderson, Ed.D. Beth Kubitskey, Ph.D. Jaclynn Tracy, Ph.D. November 29, 2018 Ypsilanti, Michigan ASSESSING STUDENT SUCCESS ii Dedication This dissertation is dedicated to my grandmothers, may they rest in peace, for showing me the power of both patience and perseverance. I continue to practice both daily, reminding myself the benefits of focusing more on the former than the later on many occasions. I also dedicate this dissertation to my father, may he rest in peace, who suddenly passed away during the middle of my writing. His continued challenging of—and belief in— me must be credited, without which I would not have achieved much of what I have in life. “Sometimes, the most brilliant and intelligent minds do not shine in standardized tests because they do not have standardized minds.”―Diane Ravitch "Everything that can be counted does not necessarily count; everything that counts cannot necessarily be counted."―Albert Einstein ASSESSING STUDENT SUCCESS iii Acknowledgements I would like to express my deepest appreciation to my committee chair, Dr. Ronald Williamson, for his support in my pursuit of this research. I thank him for his advice and guidance in how best to navigate the institution that is higher education. His patience is that of a model mentor. Without his counsel and dedication, this dissertation would not have been possible. I would like to thank my committee members, Dr. David Anderson, Dr. Beth Kubitskey, and Dr. Jaclynn Tracy. Their diverse perspective challenged me to think about my work in ways that had not initially occurred to me. This helped me to grow from the microscopic to macroscopic in my focus of educational policy. Finally, I would like to thank Dr. Sango Otieno and his interns in the Statistical Consulting Center at Grand Valley State University for his help in my data analysis. Moreover, I must not forget the support of my family, namely my mother and husband, who made their own sacrifices so that I could dedicate my time here. ASSESSING STUDENT SUCCESS iv Abstract The number of applications to postsecondary institutions continues to increase year over year, and in most cases, the number of applications exceeds the number of students admitted. The use of standardized tests continues to grow to help in these admissions decisions. Due to both high usage rates and the changing demographics of our nation’s student population, the study of test bias is still a relevant conversation today. Aside from larger issues of equity and access, particularly in STEM courses, this has implications for leaders in higher education because universities have a stake in ensuring students are academically fit for the curriculum. The goal of this study was to understand whether differential prediction existed in the ACT science subscore and the extent to which this index was differentially valid for individuals of various demographic subgroups regarding their performance in introductory science courses. Differential prediction and differential validity were investigated through quantitative methods whereby correlation coefficients and regression models were determined for different subgroups based on various demographic characteristics. Results of this study revealed variable differential validity across gender, ethnicity, and student major. Results also revealed differential prediction completely across ethnicity and student major, and variably across gender, Pell-eligibility, and first generation status. In many cases, the results were consistent with current literature in that female student performance was often underpredicted whereas non-White student performance was often overpredicted. In terms of fairness, test bias was highly dependent on particular predictor/criterion combinations for each demographic subgroup investigated. Criterion bias was predominant and presented as a considerable concern as well. ASSESSING STUDENT SUCCESS v Table of Contents Dedication ................................................................................................................................. ii Acknowledgements .................................................................................................................. iii Abstract .................................................................................................................................... iv List of Tables ........................................................................................................................... ix List of Figures ........................................................................................................................ xiv Chapter 1: Introduction and Background .................................................................................. 1 Introduction ........................................................................................................................... 1 Significance of the Study ...................................................................................................... 4 About the researcher. ..........................................................................................................4 Predictive validity and STEM. ...........................................................................................5 Importance for higher education leaders. .........................................................................10 Contributions of the Study .................................................................................................. 11 Conceptual Framework ....................................................................................................... 12 Validity. ............................................................................................................................12 The instrument-based approach to validity. .....................................................................13 The argument-based approach to validity. .......................................................................15 Brief history of validity theory. ........................................................................................17 Types of validity. ..............................................................................................................19 Differential validity versus differential prediction. ..........................................................21 Theories of differential validity and differential prediction. ............................................22 Test bias and fairness. .......................................................................................................23 Assessing differential validity and differential prediction. ..............................................25 ASSESSING STUDENT SUCCESS vi Summary of current research on admission test validity. .................................................27 Overview of Methodology .................................................................................................. 29 Guiding Research Questions ............................................................................................... 30 Limitations and Delimitations ............................................................................................. 31 Limitations. .......................................................................................................................31 Delimitations. ...................................................................................................................32 Definition of Relevant Terms .............................................................................................. 34 Summary ............................................................................................................................. 36 Chapter II: Literature Review ................................................................................................. 37 Introduction ......................................................................................................................... 37 Historical Background of Testing ....................................................................................... 37 History of Admissions Testing ...........................................................................................