Learning Analytics: Understanding First-Year Engineering Students Through Connected Student-Centered Data
Total Page：16
File Type：pdf, Size：1020Kb
Learning Analytics: Understanding First-Year Engineering Students through Connected Student-Centered Data Stephen Courtland Brozina Dissertation submitted to the faculty of the Virginia Polytechnic Institute and State University in partial fulfillment of the requirements for the degree of Doctor of Philosophy In Engineering Education David B. Knight, Chair Aditya Johri Vinod Lohani Glenda Scales 30 October 2015 Blacksburg, VA Keywords: Engineering Education, Learning Analytics, Educational Data, Learning Management System, Structural Equation Modeling Learning Analytics: Understanding First-Year Engineering Students through Connected Student-Centered Data Stephen Courtland Brozina ABSTRACT This dissertation illuminates patterns across disparate university data sets to identify the insights that may be gained through the analysis of large amounts of disconnected student data on first-year engineering (FYE) students and to understand how FYE instructors use data to inform their teaching practices. Grounded by the Academic Plan Model, which highlights student characteristics as an important consideration in curriculum development, the study brings together seemingly distinct pieces of information related to students’ learning, engagement with class resources, and motivation so that faculty may better understand the characteristics and activities of students enrolled in their classes. In the dissertation’s first manuscript, I analyzed learning management system (LMS) timestamp log-files from 876 students enrolled in the FYE course during Fall 2013. Following a series of quantitative analyses, I discovered that students who use the LMS more frequently are more likely to have higher grades within the course. This finding suggests that LMS usage might be a way to understand how students interact with course materials outside of traditional class time. Additionally, I found differential relationships between LMS usage and course performance across different instructors as well as a relationship between timing of LMS use and students’ course performance. For the second manuscript, I connected three distinct data sets: FYE student’s LMS data, student record data, and FYE program survey data that captured students’ motivation and identity as engineers at two time points. Structural equation modeling results indicate that SAT Math was the largest predictor of success in the FYE course, and that students’ beginning of semester engineering expectancy was the only significant survey construct to predict final course grade. Finally, for the third manuscript I conducted interviews with eight FYE instructors on how they use student data to inform their teaching practices. Ten themes emerged which describe the limited explicit use of formal data, but many instructors use data on an informal basis to understand their students. Findings also point to specific, existing data that the university already collects that could be provided to instructors on an aggregate, class-level basis to help them better understand their students. Dedication To Inna The love of my life who I cherish more with each passing day. iii Acknowledgments To my son, Stephen Jude, you bring great joy to my life. Let this dissertation, ridiculous as it may seem, be an encouragement to you, set a goal and achieve it. I would like to express my greatest appreciation to Dr. David B. Knight, your direction and patience has been invaluable to me on this journey. Even though you attended a school which shall remain nameless, you are truly a great professor and advisor. You have always had a positive attitude through my PhD process, for which I thank you. Continue to march on and do grand things, obviously with a preference towards quantitative analyses. Dr. Aditya Johri, your knowledge and insights are incredible, thank you for pushing me to know more. I look forward to seeing something insane that flips everything upside down in the near future from you because I know you have such ideas. Dr. Vinod Lohani, I am not sure if I would have learned R if it wasn’t for you introducing it to me in your class. Even though it was difficult then, that was just the stepping stone to the wonderful world of R. Dr. Glenda Scales, thank you for asking the tough practical questions. You made sure I did not travel down fruitless paths, which is much appreciated. I would also like to thank Dr. Marie Paretti and Dr. Lisa McNair for helping me improve my writing and introducing me to the great world of engineering education research. To the best engineering education research lab, DEEP, thank you for your continued help throughout our endeavors together. You have all provided valuable insights and encouragement during our time together. Keep pushing boundaries and pursuing your passions! And thank you to all the ENGE community, I look forward to great adventures to come. Thank you to my family and friends, for not thinking I was completely insane for attending Virginia Tech for 25 semesters. iv TABLE OF CONTENTS CHAPTER 1: INTRODUCTION………………………………………………....................... 1 1.1 Introduction……………………………………………………………………..…... 1 1.2 Definitions…………………………………………………………………………... 3 1.3 Purpose of the Study………………………………………………………………. 4 1.4 Scope of Study…………………………………………………………………….. 4 1.5 Research Questions and Research Design …….…………...………………… 5 1.6 Limitations………………………………………………………………………...... 7 1.7 Summary………………………………………………………………………….....8 CHAPTER 2: LEARNING MANAGEMENT SYSTEM USAGE OF FIRST-YEAR ENGINEERING STUDENTS………………………………………………………….………..9 2.1 Introduction……..………………………...………………………………………… 9 2.2 Literature Review……………..…...…………………………….………………...10 2.3 Theoretical Framework……….……………………………….………………..…16 2.4 Data and Methods………...….………….…………………….……………….….18 2.5 Results and Discussion……....………….…………………….……….…………21 2.6 Conclusion………...………...….……….……………………….………..……….48 CHAPTER 3: USING STRUCTURAL EQUATION MODELING TO CONNECT DISPARATE FIRST-YEAR ENGINEERING STUDENT-CENTERED DATA….………..50 3.1 Introduction……..…………………………………………………………………. 50 3.2 Theoretical Framework and Literature Review……..………….……………….51 3.3 Overview of Study…………………………………..…………….…………….....54 3.4 Data Sources……………………………………….....………….………………..56 3.5 Methods…………………………………………………………………………… 57 3.6 Limitations..................................................................................................... 61 3.7 Results of Analyses........................................................................................62 3.8 Conclusions....................................................................................................78 CHAPTER 4: FIRST-YEAR ENGINEERING INSTRUCTORS’ USE AND PERCEPTIONS OF STUDENT DATA: A QUALITATIVE INVESTIGATION.................79 4.1 Introduction.....................................................................................................79 4.2 Literature Review............................................................................................80 4.3 Data and Methods.......................................................................................... 82 4.4 Limitations..................................................................................................... 85 4.5 Findings..........................................................................................................86 4.6 Conclusions....................................................................................................97 CHAPTER 5: SUMMARY.............................................................................................100 5.1 Summary......................................................................................................100 5.2 Significance of the Research........................................................................101 5.3 Limitations................................................................................................... 105 COMPLETE REFERENCE LIST................................................................................. 107 v APPENDIX A: MANUSCRIPT 1 OUTLIER INSPECTION............................................114 APPENDIX B: MANUSCRIPT 1 ROBUST REGRESSION RESULTS.........................115 APPENDIX C: MANUSCRIPT 1 LMS TOOLS INVESTIGATION.................................117 APPENDIX D: MANUSCRIPT 2 CORRELATION MATRIX..........................................123 APPENDIX E: MANUSCRIPT 2 POST HOC MODIFICATIONS...................................124 vi LIST OF TABLES Table 1.1: Overview of Study.............................................................................................6 Table 2.1: LMS Tool Use Frequency...............................................................................20 Table 2.2: LMS Usage by Final Course Grade.................................................................22 Table 2.3: LMS Usage by Gender....................................................................................22 Table 2.4: LMS Usage by Tool.........................................................................................23 Table 2.5: Simulation Model Results...............................................................................25 Table 2.6: ANOVA results for Model 2a and 2b................................................................25 Table 2.7: ANOVA results for Robust Model....................................................................28 Table 2.8: Correlation between Grade & Unique Sessions on WS Site by WSL...............30 Table 2.9: Robust ANOVA with WSL at 3 Levels.............................................................31