Measuring Student Engagement Using Learning Management Systems

Total Page:16

File Type:pdf, Size:1020Kb

Measuring Student Engagement Using Learning Management Systems Measuring Student Engagement Using Learning Management Systems by Krista Lee Vogt A thesis submitted in conformity with the requirements for the degree of Doctor of Philosophy Ontario Institute for Studies in Education Leadership, Higher & Adult Education University of Toronto © Copyright by Krista Vogt 2016 Measuring Student Engagement Using Learning Management Systems Krista Vogt Doctor of Philosophy Ontario Institute for Studies in Education Leadership, Higher & Adult Education University of Toronto 2016 Abstract In less than two decades, learning management systems (LMS) have gone from being a rarity to having near-ubiquitous presence on Canadian college campuses. This wide-spread adoption of LMS technology has fundamentally changed the learning environment for today’s campus-based college student, yet there is a lack of research that investigates how students engage with the learning environments created by the prevalent use of these systems. The purpose of this study is to explore the relationship between campus-based students’ perceived level of engagement in LMS an online learning environment and their online behaviour as measured by log data for various LMS activities. Additionally, this study examined faculty members’ use of an LMS and the impact of that use on students’ engagement and students’ use of the LMS. The results of the analysis reveal no correlation between students’ online engagement as measured by the Student Engagement Questionnaire (SEQ) and frequency counts of LMS activities. Small correlations were found between students’ estimates of their LMS activity in discussion forums and their overall SEQ score. There was no difference in the mean SEQ score between groups of survey respondents based on faculty use of the LMS, but some differences in student estimates of their ii activity in discussion forums were found between student respondents grouped by age and gender. Keywords: student engagement, learning management systems, learning analytics iii Acknowledgments I am eternally grateful to the community of supporters who were at my side throughout my course work and research. Thank you to Dr. Tricia Seifert for accepting me on to her research team, allowing me to discover qualitative research, and helping me build a network of colleagues. Thank you also to Dr. Ruth Childs who agreed to become my supervisor late in my educational journey and provided me with guidance and encouragement. I am also grateful to Dr. Clare Brett for her insights in online learning and her feedback on my work. Lastly, my utmost appreciation and respect goes to my partner, Geoff, who made it possible for me to engage in this research while working full time and raising two children. iv Table of Contents Acknowledgments .......................................................................................................................... iv Table of Contents ............................................................................................................................ v List of Tables ................................................................................................................................ vii List of Figures .............................................................................................................................. viii Introduction ..................................................................................................................................... 1 Statement of the Problem, Purpose, and Rationale ..................................................................... 6 Research Questions ................................................................................................................... 11 Hypothesis ................................................................................................................................. 11 Terms ........................................................................................................................................ 15 Review of the Literature ............................................................................................................... 17 Changed Campus Environment ................................................................................................ 17 Theoretical Foundations ............................................................................................................ 23 LMS Research ........................................................................................................................... 36 Role of Faculty in Students’ Use of an LMS ............................................................................ 45 Summary ................................................................................................................................... 50 Methods ......................................................................................................................................... 53 Sample ....................................................................................................................................... 53 Procedures ................................................................................................................................. 59 Limitations and Delimitations ................................................................................................... 72 v Summary ................................................................................................................................... 74 Results ........................................................................................................................................... 76 Variables Used in the Analysis ................................................................................................. 77 Research Questions ................................................................................................................... 89 Summary ................................................................................................................................... 99 Discussion and Conclusion ......................................................................................................... 101 Discussion of Results .............................................................................................................. 102 Implications ............................................................................................................................. 128 Limitations .............................................................................................................................. 133 Future research ........................................................................................................................ 135 Conclusions ............................................................................................................................. 139 References ................................................................................................................................... 142 vi List of Tables Table 1. Mapping Student Engagement Types with SEQ Survey Items and LMS Data .............. 13 Table 2. Age of Sample Compared to College Population ........................................................... 57 Table 3. Academic School of Study of Sample Population Compared to College Population .... 58 Table 4. Construct and Variable Descriptions .............................................................................. 61 Table 5. A Comparison of Cronbach’s Alpha Values From the Current Study and Coates (2006) Study ............................................................................................................................................. 63 Table 6. Differences in the Means of Various Variables Between Male Students and Female Students ......................................................................................................................................... 78 Table 7. Differences in the Means of Various Variables Between English Speakers and Others 79 Table 8. Descriptive Statistics for Student Engagement Questionnaire (SEQ) Response Data ... 80 Table 9. Descriptive Statistics for Student LMS Activity From LMS Software .......................... 81 Table 10. Descriptive Statistics for Student LMS Activity as Measured by Student Estimates... 82 Table 11. Descriptive Statistics for Faculty LMS Activity ........................................................... 87 Table 12. Percentage of Faculty and Student Using LMS as Measured by Frequency Counts and Survey Responses ......................................................................................................................... 88 Table 13. Correlation Matrix of LMS Activity Data and SEQ Scale and Overall Scores ............ 92 Table 14. Correlation Matrix of Student Estimates of LMS Activity and SEQ Scales ................ 93 vii List of Figures Figure 1. Model showing influences within the unified theory of acceptance and use of technology (UTAUT). ................................................................................................................... 37 Figure 2. Students' self-reported estimates of login frequency. .................................................... 83 Figure 3. Students’ self-reported estimates of the frequency with which they checked their grades. ........................................................................................................................................... 83 Figure 4.
Recommended publications
  • Assessment of Student Engagement in Higher Education: a Synthesis of Literature and Assessment Tools
    1 International Journal of Learning, Teaching and Educational Research Vol. 12, No. 2, pp. 1-14, June 2015 Assessment of Student Engagement in Higher Education: A Synthesis of Literature and Assessment Tools B. Jean Mandernach, PhD Grand Canyon University Phoenix, Arizona, United States Abstract. Educational research increasingly highlights the importance of student engagement and its impact on retention, learning and persistence. Despite widespread agreement on the value of student engagement, assessing engagement in higher education remains a challenge. To effectively measure student engagement (and understand its influence on the learning experience), it is essential that each institution defines the scope of engagement within their unique context and selects assessment metrics that align with the target definition. The dynamic nature of engagement mandates a multi-faceted approach to assessment that captures the interactive nature of the behavioral, affective and cognitive dimensions comprising student engagement. The value of various modes and tools for assessing student engagement in higher education are discussed. Keywords: student engagement; assessment of engagement; engagement metrics; cognitive engagement Introduction With increased emphasis on promoting student engagement in postsecondary classrooms (Barkley, 2010; Bowen, 2005; Günüç & Kuzu, 2014; Korobova & Starobin, 2015), it becomes imperative that educators are able to gauge, monitor and assess student engagement as a component of the overall learning experience (Butler,
    [Show full text]
  • Toshalis & Nakkula, "Motivation, Engagement, and Student Voice
    MOTIVATION, ENGAGEMENT, AND APRIL 2012 STUDENT VOICE By Eric Toshalis and Michael J. Nakkula EDITORS’ INTRODUCTION TO THE STUDENTS AT THE CENTER SERIES Students at the Center explores the role that student-centered approaches can play to deepen learning and prepare young people to meet the demands and engage the opportunities of the 21st century. Students at the Center synthesizes existing research on key components of student-centered approaches to learning. The papers that launch this project renew attention to the importance of engaging each student in acquiring the skills, knowledge, and expertise needed for success in college and a career. Student-centered approaches to learning, while recognizing that learning is a social activity, pay particular attention to the importance of customizing education to respond to each student’s needs and interests, making use of new tools for doing so. The broad application of student-centered approaches to learning has much in common with other education reform movements including closing the achievement gaps and providing equitable access to a high-quality education, especially for underserved youth. Student-centered approaches also align with emerging work to attain the promise and meet the demands of the Common Core State Standards. However, critical and distinct elements of student-centered approaches to learning challenge the current schooling and education paradigm: > Embracing the student’s experience and learning theory as the starting point of education; > Harnessing the full range of learning experiences at all times of the day, week, and year; > Expanding and reshaping the role of the educator; and > Determining progression based upon mastery.
    [Show full text]
  • Student Voice: How Young People Can Shape the Future of Education
    WINTER 2019 Student Voice: How Young People Can Shape the Future of Education Student Voice: How Young People Can Shape the Future of Education Overview “It’s all about the students.” How often do we hear this sentiment—or something THE PROJECT similar—in conversations on educational policy and practice? Education leaders The Condition of Education in the take action every day to support and guide students. Dedicated teachers Commonwealth project is one way review and revise their approaches to instruction, while school leaders institute the Rennie Center fulfills its mission of producing non-partisan, high-quality, new strategies to enhance learning inside and outside the school building. independent research that promotes Communities rally around innovative institutions to augment and sustain success. improvement in public education for all Without question, these efforts aim to accomplish a noble goal: helping students Massachusetts children. achieve better outcomes in school and in life. Yet one voice that’s usually missing in discussions about how best to support PROJECT COMPONENTS student outcomes is the one that arguably matters the most: students Data Dashboard: This interactive tool themselves.1 Within the education system, decision-making structures provides an in-depth look at 25 school and practices often do not recognize or encourage students as legitimate performance indicators. Users can delve stakeholders.2 In educational debates dominated by questions of learning inputs deeper by looking at different student groups and monitor progress over time. This (standards, curricula, funding) and outcomes (assessments, college and career data is collected from the Massachusetts success), we often neglect to listen to the students who are most impacted by Department of Early Education and proposed or actual reforms.
    [Show full text]
  • Can Digital Technology Bridge the Classroom Engagement Gap? Findings from a Qualitative Study of K-8 Classrooms in 10 Ontario School Boards
    social sciences $€ £ ¥ Article Can Digital Technology Bridge the Classroom Engagement Gap? Findings from a Qualitative Study of K-8 Classrooms in 10 Ontario School Boards Jessica Rizk 1,* and Scott Davies 2 1 Department of Sociology, University of Waterloo, Waterloo, ON N2L 3G1, Canada 2 Educational Leadership and Policy and Sociology, University of Toronto, Toronto, ON M5S 1V6, Canada; [email protected] * Correspondence: [email protected] Abstract: This study examined impacts of digital technology on a key component of the socioe- conomic gap in education—gaps in student classroom engagement. Whereas print literacy has long been a source of such gaps, newer “digital divide” theories claim classrooms that use digital technology are perpetuating them further. However, these claims are not grounded in close empirical observation and may now already be dated. We aimed to advance understandings of the impact of digital technology on student engagement by examining robotics, tablets, and smart board usage across a range of classrooms, using a conceptual framework that blends theories of interaction ritual chains (IRC) and cultural capital (CC). Data came from observations and interviews with teachers and students in K-8 classrooms across 10 Ontario school boards. We report three major findings. First, almost all students across socioeconomic strata engaged easily and enthusiastically with digital technology. Second, technology spawned new classroom rituals and cultural valuations. Third, digital technology provided connections between school dictates and students’ peer-based and home lives. We argue that digital technology has the potential to narrow classroom engagement gaps that are generated by conventional print media. We end by discussing avenues for future research.
    [Show full text]
  • Improving Accuracy Enhancing Student Engagement with Technology
    International Journal of Computational Intelligence Research ISSN 0973-1873 Volume 13, Number 4 (2017), pp. 517-524 © Research India Publications http://www.ripublication.com Improving Accuracy Enhancing Student Engagement with Technology L. Venkateswara Reddy and Dr A. Rama Mohan Reddy Department of Computer Science and Engineering, Rayalaseema University, Kurnool, Andhra Pradesh, India. Abstract New and exciting applications of technology-enabled learning validate that it has the power to dramatically improve achievement, educational outcomes and retention. Yet the cost of technology, its rapid evolution, and the special knowledge and skills required of its users pose substantial barriers to contextualized learning. Even without educational technology, classrooms are in information overload putting students and instructors on the brink of drowning in data. Immediate innovations in, pedagogy, curriculum, and assessment must be coupled with the usage of instructional technology such as web-based learning, and online teaching and learning in order to produce improvements in the educational outcomes. Without substantial and extended professional development in the innovative models of teaching and learning that instructional technology makes affordable and sustainable, many instructors and students will not use these devices to their full potential. 1. INTRODUCTION social interactions among its subjects, which are heavily influenced by the language and the culture of the people involved8 . The pedagogical advantages of collaborative learning have been extensively analyzed over the last ten years. Its benefits are focused mainly on active learning and on in-depth information processing7 . The teaching models of collaborative learning have been systematically investigated for some time. Those are Knowledge Building, Progressive Inquiry, Knowledge Integration, Knowledge Creation and Social Theory of CSCL8 .
    [Show full text]
  • Measuring Student Engagement in the Online Course: the Online Student Engagement Scale (OSE)
    Measuring Student Engagement in the Online Course: The Online Student Engagement Scale (OSE) Measuring Student Engagement in the Online Course: The Online Student Engagement Scale (OSE) Marcia D. Dixson Indiana University – Purdue University Fort Wayne Abstract Student engagement is critical to student learning, especially in the online environment, where students can often feel isolated and disconnected. Therefore, teachers and researchers need to be able to measure student engagement. This study provides validation of the Online Student Engagement scale (OSE) by correlating student self-reports of engagement (via the OSE) with tracking data of student behaviors from an online course management system. It hypothesized that reported student engagement on the OSE would be significantly correlated with two types of student behaviors: observational learning behaviors (i.e., reading e-mails, reading discussion posts, viewing content lectures and documents) and application learning behaviors (posting to forums, writing e-mails, taking quizzes). The OSE was significantly and positively correlated with application learning behaviors. Results are discussed along with potential uses of the OSE by researchers and online instructors. Introduction Despite the decrease in higher education enrollment overall, online instruction is still growing. A recent report funded by the Online Learning Consortium (Allen & Seaman, 2013) found that 6.7 million students (about 32% of all college students) were taking at least one online course. Indeed, in the analysis of 2,820 institutions (of a potential 4,527 active degree-granting institutions of higher education in the United States), over 69% of chief academic officers felt that online learning is important for the future of their institutions.
    [Show full text]
  • The Effects of Student Engagement, Student Satisfaction, and Perceived Learning in Online Learning Environments
    The Effects of Student Engagement, Student Satisfaction, and Perceived Learning in Online Learning Environments This manuscript has been peer-reviewed, accepted, and endorsed by the National Council of Professors of Educational Administration (NCPEA) as a significant contribution to the scholarship and practice of school administration and K-12 education. Julie A. Gray University of West Florida Melanie DiLoreto University of West Florida Studies have shown that course organization and structure, student engagement, learner interaction, and instructor presence have accounted for considerable variance in student satisfaction and perceived learning in online learning environments through a range of pathways, although no research to date has tested the mediational relationship identified. This study expanded upon the existing literature about online learning and the variables that influence student satisfaction and perceived learning. The researchers investigated the relationships among course structure/organization, learner interaction, student engagement, and instructor presence on student satisfaction and perceived learning. The results of this study were intended to inform practice related to increasing retention and improving the quality of online teaching and learning. NCPEA International Journal of Educational Leadership Preparation, Vol. 11, No. 1– May, 2016 ISSN: 2155-9635 © 2016 National Council of Professors of Educational Administration Introduction “The landscape of distance education is changing” (Eom, Ashill, & Wen, 2006, p. 215). As more universities are offering online courses it is important for faculty to consider the changing aspects of online learning environments, including course structure, learner interaction, and instructor presence (Allen & Seaman, 2015). One study in particular provided a model upon which to develop and build this study (Eom et al., 2006), although our study varied in methodology.
    [Show full text]
  • Effectiveness of Student Engagement Using Learning Management System in the Blended Learning Environment at Saudi Electronic University Yousef Almoslamani
    University of Northern Colorado Scholarship & Creative Works @ Digital UNC Dissertations Student Research 5-2018 Effectiveness of Student Engagement Using Learning Management System in the Blended Learning Environment at Saudi Electronic University Yousef Almoslamani Follow this and additional works at: https://digscholarship.unco.edu/dissertations Recommended Citation Almoslamani, Yousef, "Effectiveness of Student Engagement Using Learning Management System in the Blended Learning Environment at Saudi Electronic University" (2018). Dissertations. 484. https://digscholarship.unco.edu/dissertations/484 This Text is brought to you for free and open access by the Student Research at Scholarship & Creative Works @ Digital UNC. It has been accepted for inclusion in Dissertations by an authorized administrator of Scholarship & Creative Works @ Digital UNC. For more information, please contact [email protected]. UNIVERSITY OF NORTHERN COLORADO Greeley, Colorado The Graduate School EFFECTIVENESS OF STUDENT ENGAGEMENT USING LEARNING MANAGEMENT SYSTEM IN THE BLENDED LEARNING ENVIRONMENT AT SAUDI ELECTRONIC UNIVERSITY A Dissertation Submitted in Partial Fulfillment of the Requirements of the Degree of Doctor of Philosophy Yousef Almoslamani College of Education and Behavioral Sciences Department of Educational Technology May 2018 This Dissertation by: Yousef Almoslamani Entitled: Effectiveness of Student Engagement Using Learning Management System in the Blended Learning Environment at Saudi Electronic University has been approved as meeting the
    [Show full text]
  • MOTIVATION, ENGAGEMENT, and STUDENT VOICE TOOLKIT ACTIVITY #1 MAY 2013 SURVEY and FORCED CHOICE EXERCISE by Eric Toshalis and Michael J
    MOTIVATION, ENGAGEMENT, AND STUDENT VOICE 2013 MAY TOOLKIT By Eric Toshalis and Michael J. Nakkula INTRODUCTION This professional development series is designed to accompany and help put into practice the ideas in Eric Toshalis and Michael J. Nakkula’s Students at the Center paper, Motivation, Engagement, and Student Voice,1 and their chapter, “Prioritizing Motivation and Engagement,” in Anytime, Anywhere: Student-Centered Learning for Schools and Teachers .2 For many teachers, the issues discussed in those pieces are at the heart of the student experience they are trying to provide. The purpose of the activities and materials in this series is to prepare educators to understand, contextualize, and apply the concepts explained in the research so that student-centered approaches are more frequently used in middle and high school classrooms. The activities are designed to facilitate the development of a mindset that encourages a critical analysis of what participants believe, what they do, and what might need to be changed to fully realize the potential of student-centered teaching. Each activity either concludes with or is dedicated to the development of student-centered practices that are anchored in the research findings detailed in the paper. Some activities are best conducted before participants read the paper or book chapter and some are best done after it has already been read ( see table 1 on page 2). Taking seriously the authors’ claim that “to build student-centered classrooms we need to build schools and school cultures that are teacher centered,” the activities are easily personalized for particular audiences and may be adapted to fit time and location constraints.
    [Show full text]
  • Pals: Fostering Student Engagement and Interactive Learning Thomas Hancock, Stella Smith, Candace Timpte, and Jennifer Wunder
    © Journal of Higher Education Outreach and Engagement, Volume 14, Number 4, p. 37, (2010) PALs: Fostering Student Engagement and Interactive Learning Thomas Hancock, Stella Smith, Candace Timpte, and Jennifer Wunder Introduction n 2002, the Association of American Colleges and Universities (AAC&U) issued the report Greater Expectations: A New Vision for Learning as a Nation Goes to College, and called Iupon educators to rethink and reinvigorate education for the 21st century by embracing the concept of a “liberal education” that enables students to become empowered, informed, responsible, and intentional life-long learners (Association of American Colleges and Universities, 2002). Drawing upon extensive research regarding educational practices and in-depth explorations of innovative learning models from almost two dozen campuses, the report consistently stresses that liberal education is most effective “when studies reach beyond the classroom to the larger community, asking students to apply their developing analytical skills and ethical judg- ment to concrete problems in the world around them, and to con- nect theory with the insights gained from practice” (Association of American Colleges and Universities, 2002, pp. 25–26). There is ample evidence to support Greater Expectations’ emphasis on learning strategies that merge intellectual and practical skills, including research indicating that students report increased motivation and display deeper learning and better retention when asked to solve real-world problems and construct knowledge from their experiences both in and beyond the classroom (Ames, 1992; Downing, Kwong, Chan, Lam, & Downing, 2009; Edens, 2000; Hmelo- Silver, 2004). Not surprisingly, increasing numbers of educators in the last few decades also have asserted that active learning, learning by doing, is a rich and engaging strategy (Bransford, Brown, & Cocking, 2000; Herrington & Herrington, 2006; Lombardi, 2007b; Ramsden, 2003).
    [Show full text]
  • The Effects of Technology on Student Engagement and Academic Success
    Northwestern College, Iowa NWCommons Master's Theses & Capstone Projects Education Spring 2021 The Effects of Technology on Student Engagement and Academic Success Austin Reese Follow this and additional works at: https://nwcommons.nwciowa.edu/education_masters Part of the Educational Technology Commons Running Head: The Effects of Technology on Academic Success 1 The Effects of Technology on Student Engagement and Academic Success Austin L. Reese Graduate School of Education, Northwestern College EDU-635 Capstone Dr. Theresa Pedersen April 17, 2020 The Effects of Technology on Academic Success and Student Engagement 2 Table of Contents Introduction………………………………………………………………………………………. 4 Literature Review Educational Technology and Literacy…….……………...……………………….…..…..6 Educational Technology Use In General Education.…....….……….…………………...13 Educational Technology Use In Special Populations.…………………………………...16 Educational Technology Use In Student Motivation………........…...…....………..……20 Gaps/Future Research…………………………………………………………………... 26 Conclusion……………………………………………………………………………………….27 References………………………………………………………………………………………..29 The Effects of Technology on Academic Success and Student Engagement 3 Abstract The prevalence of educational technology has grown exponentially over the last decade. As schools continually increase use of educational technology throughout the various classroom environments and settings, does educational technology actually increase student engagement and academic success? A multitude of research studies regarding educational
    [Show full text]
  • Exploring Students Engagement Towards the Learning Management System (LMS) Using Learning Analytics
    Computer Systems Science & Engineering Tech Science Press DOI:10.32604/csse.2021.015261 Article Exploring Students Engagement Towards the Learning Management System (LMS) Using Learning Analytics Shahrul Nizam Ismail1, Suraya Hamid1,*, Muneer Ahmad1, A. Alaboudi2 and Nz Jhanjhi3 1Department of Information Systems, Faculty of Computer Science & Information Technology, University Malaya, Kuala Lumpur, 50603, Malaysia 2Department of Information systems, College of Computer and Information Sciences, Shaqra University, Saudi Arabia 3School of Computer Science and Engineering, SCE, Taylor’s University, Subang Jaya, 47500, Malaysia ÃCorresponding Author: Suraya Hamid. Email: [email protected] Received: 22 November 2020; Accepted: 17 December 2020 Abstract: Learning analytics is a rapidly evolving research discipline that uses the insights generated from data analysis to support learners as well as optimize both the learning process and environment. This paper studied students’ engagement level of the Learning Management System (LMS) via a learning analytics tool, student’s approach in managing their studies and possible learning analytic meth- ods to analyze student data. Moreover, extensive systematic literature review (SLR) was employed for the selection, sorting and exclusion of articles from diverse renowned sources. The findings show that most of the engagement in LMS are driven by educators. Additionally, we have discussed the factors in LMS, causes of low engagement and ways of increasing engagement factors via the Learning Analytics approach. Nevertheless, apart from recognizing the Learning Analytics approach as being a successful method and technique for ana- lyzing the LMS data, this research further highlighted the possibility of merging the learning analytics technique with the LMS engagement in every institution as being a direction for future research.
    [Show full text]