Engaging with Massive Online Courses
Total Page：16
File Type：pdf, Size：1020Kb
Engaging with Massive Online Courses Ashton Anderson Daniel Huttenlocher Jon Kleinberg Jure Leskovec Stanford University Cornell University Cornell University Stanford University [email protected] {dph, kleinber}@cs.cornell.edu [email protected] ABSTRACT tively little quantitative evidence to support or refute whether such The Web has enabled one of the most visible recent developments intuitions hold in the case of MOOCs. in education—the deployment of massive open online courses. With Understanding how students interact with MOOCs is a crucial is- their global reach and often staggering enrollments, MOOCs have sue, because it affects how we design future online courses and how the potential to become a major new mechanism for learning. De- we evaluate their effectiveness. For students who treat MOOCs like spite this early promise, however, MOOCs are still relatively unex- traditional courses, that run at a fixed pace over a fixed time period, plored and poorly understood. it makes sense to talk about students “falling behind” or “dropping In a MOOC, each student’s complete interaction with the course out.” But for students who might treat MOOCs as online refer- materials takes place on the Web, thus providing a record of learner ences works or textbooks, a completely different set of expectations activity of unprecedented scale and resolution. In this work, we would apply, in which the natural interaction style may consist of use such trace data to develop a conceptual framework for under- bursts of asynchronous engagement and selective sampling of con- standing how users currently engage with MOOCs. We develop a tent. For students who might use MOOCs to sharpen and test their taxonomy of individual behavior, examine the different behavioral skills in an area, it would be reasonable to see them undertaking patterns of high- and low-achieving students, and investigate how portions of the course work without ever viewing lecture content. forum participation relates to other parts of the course. Despite the high level of interest in both the popular press and We also report on a large-scale deployment of badges as incen- recent academic literature, there have been relatively few quantita- tives for engagement in a MOOC, including randomized experi- tive studies of MOOC activity at a large scale, and relatively little ments in which the presentation of badges was varied across sub- understanding of different ways in which students may be engaging populations. We find that making badges more salient produced with MOOCs (in Section 6 we review recent work in the research increases in forum engagement. literature.) Without this understanding it has been hard to rigor- ously evaluate either optimistic claims made about student experi- Categories and Subject Descriptors: H.2.8 [Database manage- ence in MOOCs, or concerns raised about low rates of completion. ment]: Database applications—Data mining. And to the extent that different types of student behaviors have been Keywords: MOOCs; online engagement; badges. anecdotally identified in MOOCs, it has been hard to assess how prevalent they are. In this paper we propose a framework for understanding how stu- 1. INTRODUCTION dents engage with massive online courses, based on quantitative in- Massive open online courses, or MOOCs, have recently gar- vestigations of student behavior in several large Stanford University nered widespread public attention for their potential as a new ed- courses offered on Coursera, one of the major MOOC platforms. ucational vehicle. There are now multiple MOOC platforms, in- Students in these courses can engage in a range of activities— cluding the edX consortium, Coursera and Udacity, offering hun- watching lectures, taking quizzes to test their understanding, work- dreds of courses, some of which have had hundreds of thousands ing on assignments or exams, and engaging in a forum where they of students enrolled. Yet in spite of the high degree of interest can seek help, offer advice, and have discussions. and the rapid development of MOOCs, we still understand remark- We find first of all that each student’s activities in a course can be ably little about how students engage in such courses. In thinking usefully described by one of a small number of engagement styles, about MOOCs we generally apply our intuitions from university- which we formalize in a taxonomy based on the relative frequency level courses in the offline world, thinking of students who enroll of certain activities undertaken by the student. These styles of en- in a course for credit or as an auditor, and who participate on an gagement are quite different from one another, and underscore the ongoing basis over a 10-14 week time period. Yet there is rela- point that students display a small number of recurring but distinct patterns in how they engage with an online course. We also con- sider the issue of performance and grades in the course, and show that thinking about grades in terms of different styles of engage- ment can shed light on how performance is evaluated. Our second main focus in this paper is the course forums, and how to increase engagement in them. They provide interesting Copyright is held by the International World Wide Web Conference Com- patterns of interaction, in which students engage not just with the mittee (IW3C2). IW3C2 reserves the right to provide a hyperlink to the author’s site if the Material is used in electronic media. course material but with each other. To shift the ways in which WWW’14, April 7–11, 2014, Seoul, Korea. students engage in the forums, we designed two large-scale inter- ACM 978-1-4503-2744-2/14/04. ventions. In particular, we report on our deployment of badges 105 105 in the discussion forum of one of the largest MOOCs, with the 104 104 badges serving as incentives to increase forum activity. We find through a randomized experiment that different ways of present- 103 103 ing the badges—to emphasize social signals and progress toward 102 102 milestones—can have an effect on the level of engagement. # of students 101 # of students 101 We now give an overview of these two themes—the styles of 100 100 engagement, and the interventions to modify engagement. 0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0 a a a+l (ML1) a+l (PGM1) A taxonomy of engagement styles 105 105 In characterizing the predominant styles of engagement, we con- 104 104 sider the two fundamental activities of (i) viewing a lecture and (ii) handing in an assignment for credit. (Additional activities includ- 103 103 ing ungraded quizzes and forum participation will be considered 102 102 later in the paper, but won’t be explicitly used in categorizing en- # of students 101 # of students 101 gagement.) Two basic questions to ask about any given student are 100 100 the total volume of lectures and assignments that she completes, 0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0 a a and the extent to which her overall activity is balanced between a+l (ML2) a+l (PGM2) these two modalities. 105 105 A natural way to address this question of balance between ac- 4 4 tivities is to compute a student’s assignment fraction: of the total 10 10 number of lectures and assignments completed by the student, what 103 103 fraction were assignments? Thus, a student with an assignment 102 102 fraction of 0 only viewed lectures, while a student with an assign- # of students # of students 101 101 ment fraction of 1 only handed in assignments, without viewing 100 100 any of the course content. We computed the assignment fraction 0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0 a a for each student in six Coursera classes: three successive offer- a+l (ML3) a+l (PGM3) ings of Machine Learning (which we name ML1, ML2, and ML3) and three successive offerings of Probabilistic Graphical Models Figure 1: Distribution of assignment fraction (ratio of assign- (which we name PGM1, PGM2, and PGM3). Figure 1 shows his- ments, a, to overall activity of assignments and lectures, a + l). tograms of the number of students with each assignment fraction in boundaries between them, we see from the simple plot of relative these six courses. It is striking that the histogram for each course activity levels that there are natural behavioral modes that corre- has three natural peaks: a left mode near the fraction 0, a right spond to these cases. Moreover, we can formalize precise versions mode near the fraction 1, and a central mode in between. This sug- of these styles simply by defining natural thresholds on the x-axis gests three natural styles of engagement, one clustered around each of the histograms in Figure 1 (to define clusters around the three mode. modes), then separating out downloading from watching lectures (to resolve the Collectors from the Viewers) and separating out stu- 1. Viewers, in the left mode of the plot, primarily watch lec- dents with very low total activity levels (to identify the Bystanders). tures, handing in few if any assignments. Already from this simple taxonomy we can see that intuitions Solvers 2. , in the right mode, primarily hand in assignments for about styles of student engagement from the offline world do not a grade, viewing few if any lectures. map precisely into the world of MOOCs. For instance, those just 3. All-rounders, in the middle mode, balance the watching of “viewing” lectures are not necessarily the same as auditors, because lectures with the handing in of assignments.