Measuring Student Engagement Using Learning Management Systems
Total Page:16
File Type:pdf, Size:1020Kb
Measuring Student Engagement Using Learning Management Systems by Krista Lee Vogt A thesis submitted in conformity with the requirements for the degree of Doctor of Philosophy Ontario Institute for Studies in Education Leadership, Higher & Adult Education University of Toronto © Copyright by Krista Vogt 2016 Measuring Student Engagement Using Learning Management Systems Krista Vogt Doctor of Philosophy Ontario Institute for Studies in Education Leadership, Higher & Adult Education University of Toronto 2016 Abstract In less than two decades, learning management systems (LMS) have gone from being a rarity to having near-ubiquitous presence on Canadian college campuses. This wide-spread adoption of LMS technology has fundamentally changed the learning environment for today’s campus-based college student, yet there is a lack of research that investigates how students engage with the learning environments created by the prevalent use of these systems. The purpose of this study is to explore the relationship between campus-based students’ perceived level of engagement in LMS an online learning environment and their online behaviour as measured by log data for various LMS activities. Additionally, this study examined faculty members’ use of an LMS and the impact of that use on students’ engagement and students’ use of the LMS. The results of the analysis reveal no correlation between students’ online engagement as measured by the Student Engagement Questionnaire (SEQ) and frequency counts of LMS activities. Small correlations were found between students’ estimates of their LMS activity in discussion forums and their overall SEQ score. There was no difference in the mean SEQ score between groups of survey respondents based on faculty use of the LMS, but some differences in student estimates of their ii activity in discussion forums were found between student respondents grouped by age and gender. Keywords: student engagement, learning management systems, learning analytics iii Acknowledgments I am eternally grateful to the community of supporters who were at my side throughout my course work and research. Thank you to Dr. Tricia Seifert for accepting me on to her research team, allowing me to discover qualitative research, and helping me build a network of colleagues. Thank you also to Dr. Ruth Childs who agreed to become my supervisor late in my educational journey and provided me with guidance and encouragement. I am also grateful to Dr. Clare Brett for her insights in online learning and her feedback on my work. Lastly, my utmost appreciation and respect goes to my partner, Geoff, who made it possible for me to engage in this research while working full time and raising two children. iv Table of Contents Acknowledgments .......................................................................................................................... iv Table of Contents ............................................................................................................................ v List of Tables ................................................................................................................................ vii List of Figures .............................................................................................................................. viii Introduction ..................................................................................................................................... 1 Statement of the Problem, Purpose, and Rationale ..................................................................... 6 Research Questions ................................................................................................................... 11 Hypothesis ................................................................................................................................. 11 Terms ........................................................................................................................................ 15 Review of the Literature ............................................................................................................... 17 Changed Campus Environment ................................................................................................ 17 Theoretical Foundations ............................................................................................................ 23 LMS Research ........................................................................................................................... 36 Role of Faculty in Students’ Use of an LMS ............................................................................ 45 Summary ................................................................................................................................... 50 Methods ......................................................................................................................................... 53 Sample ....................................................................................................................................... 53 Procedures ................................................................................................................................. 59 Limitations and Delimitations ................................................................................................... 72 v Summary ................................................................................................................................... 74 Results ........................................................................................................................................... 76 Variables Used in the Analysis ................................................................................................. 77 Research Questions ................................................................................................................... 89 Summary ................................................................................................................................... 99 Discussion and Conclusion ......................................................................................................... 101 Discussion of Results .............................................................................................................. 102 Implications ............................................................................................................................. 128 Limitations .............................................................................................................................. 133 Future research ........................................................................................................................ 135 Conclusions ............................................................................................................................. 139 References ................................................................................................................................... 142 vi List of Tables Table 1. Mapping Student Engagement Types with SEQ Survey Items and LMS Data .............. 13 Table 2. Age of Sample Compared to College Population ........................................................... 57 Table 3. Academic School of Study of Sample Population Compared to College Population .... 58 Table 4. Construct and Variable Descriptions .............................................................................. 61 Table 5. A Comparison of Cronbach’s Alpha Values From the Current Study and Coates (2006) Study ............................................................................................................................................. 63 Table 6. Differences in the Means of Various Variables Between Male Students and Female Students ......................................................................................................................................... 78 Table 7. Differences in the Means of Various Variables Between English Speakers and Others 79 Table 8. Descriptive Statistics for Student Engagement Questionnaire (SEQ) Response Data ... 80 Table 9. Descriptive Statistics for Student LMS Activity From LMS Software .......................... 81 Table 10. Descriptive Statistics for Student LMS Activity as Measured by Student Estimates... 82 Table 11. Descriptive Statistics for Faculty LMS Activity ........................................................... 87 Table 12. Percentage of Faculty and Student Using LMS as Measured by Frequency Counts and Survey Responses ......................................................................................................................... 88 Table 13. Correlation Matrix of LMS Activity Data and SEQ Scale and Overall Scores ............ 92 Table 14. Correlation Matrix of Student Estimates of LMS Activity and SEQ Scales ................ 93 vii List of Figures Figure 1. Model showing influences within the unified theory of acceptance and use of technology (UTAUT). ................................................................................................................... 37 Figure 2. Students' self-reported estimates of login frequency. .................................................... 83 Figure 3. Students’ self-reported estimates of the frequency with which they checked their grades. ........................................................................................................................................... 83 Figure 4.