The University of Southern Mississippi The Aquila Digital Community Dissertations Spring 5-2016 The Use of Receiver Operating Characteristic Curve Analysis for Academic Progress and Degree Completion Joshua William Schutts University of Southern Mississippi Follow this and additional works at: https://aquila.usm.edu/dissertations Part of the Educational Assessment, Evaluation, and Research Commons, Educational Leadership Commons, Educational Methods Commons, Educational Psychology Commons, Higher Education Commons, Higher Education Administration Commons, and the Social Statistics Commons Recommended Citation Schutts, Joshua William, "The Use of Receiver Operating Characteristic Curve Analysis for Academic Progress and Degree Completion" (2016). Dissertations. 357. https://aquila.usm.edu/dissertations/357 This Dissertation is brought to you for free and open access by The Aquila Digital Community. It has been accepted for inclusion in Dissertations by an authorized administrator of The Aquila Digital Community. For more information, please contact [email protected]. THE USE OF RECEIVER OPERATING CHARACTERISTIC CURVE ANALYSIS FOR ACADEMIC PROGRESS AND DEGREE COMPLETION by Joshua William Schutts A Dissertation Submitted to the Graduate School and the Department of Educational Studies and Research at The University of Southern Mississippi in Partial Fulfillment of the Requirements for the Degree of Doctor of Philosophy Approved: ________________________________________________ Dr. Richard Mohn, Committee Chair Associate Professor, Educational Studies and Research ________________________________________________ Dr. Forrest Lane, Committee Member Assistant Professor, Educational Leadership, Sam Houston State University ________________________________________________ Dr. Thomas Lipscomb, Committee Member Professor, Educational Studies and Research ________________________________________________ Dr. Kyna Shelley, Committee Member Professor, Educational Studies and Research ________________________________________________ Dr. Karen S. Coats Dean of the Graduate School May 2016 ABSTRACT THE USE OF RECEIVER OPERATING CHARACTERISTIC CURVE ANALYSIS FOR ACADEMIC PROGRESS AND DEGREE COMPLETION by Joshua William Schutts May 2016 College student retention and graduation are important to students, institutions, and the community. Institutions must commit to understanding why students persist and depart in order to address student success. As a result, institutions and governmental entities have increased the emphasis they place on using data to improve student success and degree completion. An abundance of research suggests that background factors (such as high school GPA and ACT score) combined with environmental factors (such as one’s major and first semester GPA) are predictive of student success. However, the literature has yet to explore the value of ROC curve analysis as a statistical technique to improve decision making. The purpose of this study was to identify the variables that best predicted satisfactory academic progress and degree completion, and model the use of ROC curve analysis. This study utilized a quantitative approach and secondary data from the institutional research office of a State University System institution in Florida. Logistic regression was successful in identifying factors that were predictive of each outcome. ROC curve analysis successfully discriminated the success/non-success groups based on predicted probability scores and a cumulative risk index. Students with three or more risk factors were less likely to make academic progress or graduate in four years. Students with two or more risk factors were less likely to graduate in six years. ii iii COPYRIGHT BY JOSHUA WILLIAM SCHUTTS 2016 ACKNOWLEDGMENTS Thank you, Dr. Rich Mohn, for being an insightful and supportive mentor to me. You pushed me when I needed it, and likewise gave me space when work and life necessitated it. Most of all, you are a fantastic teacher. You made it fun to learn, and challenged me to be the best student I could be . I am indebted to you, and look forward to pursuing a faculty career in a similar manner to yours. Thank you, Drs. Kyna Shelley, Forrest Lane, and Thomas Lipscomb for your advice, challenge, support, and suggestions along the way. I count you all among my friends and mentors–and most importantly as individuals I hold in the highest regard. Your willingness to serve on my committee is demonstrative of the care and dedication you show to students, and I am extremely honored and grateful for your support Thank you, Dr. Gentry McCreary, Dr. Martha Saunders, Dr. Michael White, Dr. Dorothy Minear, Dr. Kim LeDuff, Dr. Ben Stubbs, Dr. Ruth Davison, Ms. Cindy Cattelier, Ms. Lauren Loeffler, Ms. Robin Zimmern, and all of my coleagues at USM and UWF, for your support, flexibility, and understanding. I am encouraged by all of you, and have felt the warmth of your support throughout all points of this journey. Thank you, Mom, Pop, and Bear, for instilling the value of education in me at such a young age, and encouraging me to learn everything I could. Growing up, I asked “why” endlessly. You always made time to answer me. I cannot possibly quantify the impact you had on me, and love I have for you, in so few words. Thank you, Katie, for being the person most responsible for my perseverance in this effort. You understood when I needed support, listened to me, and pushed me to always make time to write this dissertation. I love you so much. iii TABLE OF CONTENTS ABSTRACT…………………………………………………………..………………… ii ACKNOWLEDGMENTS ……………………………………………………………. iii LIST OF TABLES……………………………………………………………………… vi LIST OF ILLUSTRATIONS…………………………………………………………… vii LIST OF ABBREVIATIONS ………………………………………………………….. xiii CHAPTER I. INTRODUCTION……………………………………………………… 1 Background Statement of the Problem Conceptual Framework Purpose of the Study Research Questions Operational Definitions Delimitations and Assumptions Justification and Significance of the Study Organization of the Dissertation II. REVIEW OF RELATED LITERATURE……………………………… 18 Introduction Overview of Studies on Retention and Degree Completion Major Theoretical Models of Retention and Degree Completion Methods of Studying Retention and Degree Completion Predictors of Retention and Degree Completion Specific Studies Using Logistic Regression on Cohort Data ROC Curves in the Literature Summary III. METHODOLOGY….………………………………………………….. 41 Research Questions Research Design Subjects and Setting Variables Data Collection Procedures Data Analysis iv IV. RESULTS….…………………………………………………………… 61 Data Screening Satisfactory Academic Progress Degree Completion in Four Years Degree Completion in Six Years V. DISCUSSION.………………………………………………………… .133 Satisfactory Academic Progress Degree Completion in Four Years Degree Completion in Six Years Advantages and Disadvantages of Method Limitations Implications for Future Research Implications for Policy and Practice Conclusions APPENDICES…………………………….…………………………………………… 156 REFERENCES…………………………………………………………………....…… 159 v LIST OF TABLES Table 1. Methodological approaches Found in Prior Research……………………….. .. 27 2. General Nature of Relationships between Input and Outcome Variables of this Study…………………………………………………………………… … 30 3. General Nature of Relationships between Environmental and Outcome Variables of this Study……………………………………………………….... 32 4. General Nature of Relationships between Outcome Variables of this Study …. 33 5. Expected Relationship between Predictors and Retention to the Second Year.. 34 6. Expected Relationship between Predictors and Degree Completion in Four Years…………………………………………………………………………… 35 7. Expected Relationship between Predictors and Degree Completion in Six Years…………………………………………………………………………… 36 8. Variables in the Dataset…..…………………….……………………………… 44 9. Statistical Analyses Conducted by Research Question………………………… 48 10. General Guidelines for Sample and Model Evaluation……………………… .. 50 11. Confusion Matrix…..…………………….……………………………..……... 51 12. Comparison of Common Effect Sizes..…………………….……...…………… 53 13. Intercorrelations Among the Predictor Variables……………………………… 65 14. Comparison of Retained and Removed Student Records……………………… 67 15. Group Comparisons, Academic Progress……………………………………… 70 16. Correlations, Academic Progress……………………………………………… 72 17. Classification Statistics for High School GPA and Academic Progress……… 74 18. Classification Statistics for Application Date and Academic Progress……….. 75 19. Classification Statistics for Scholarship Aid and Academic Progress…...……. 76 vi 20. Classification Statistics for First Fall Term GPA and Academic Progress……. 77 21. Classification Statistics for Inputs Model of Academic Progress……………… 79 22. Logistic Regression for Inputs Model of Academic Progress…………………. 79 23. Classification Statistics for Final Model of Academic Progress………………. 82 24. Final Logistic Regression, Academic Progress………………………………… 83 25. Predicted Probabilities, Academic Progress…………………………………… 84 26. Evaluation of Sample and Model, Academic Progress………………………… 85 27. Results by Research Question, Academic Progress…………………………… 89 28. Group Comparisons, Four-Year Degree Completion……………………… …. 91 29. Correlations, Four-Year Degree Completion………………………………….. 93 30. Classification Statistics for High School GPA and Four-Year Degree Completion.……………………………………….………………………….... 95 31. Classification Statistics for Application Date and Four-Year Degree Completion……………………………………………………………………. 96 32. Classification Statistics for Amount of Scholarship