Penetrating the
FOAnalytics in Learning G and Education By Phil Long and George Siemens
ttempts to imagine the future of education often empha- size new technologies—ubiquitous computing devices, flexible classroom designs, and innovative visual dis- plays. But the most dramatic factor shaping the future of higher education is something that we can’t actu- ally touch or see: big data and analytics. Basing decisions on data and evidence seems stunningly obvious, and Aindeed, research indicates that data-driven decision- making improves organizational output and produc- tivity.1 For many leaders in higher education, however, experience and “gut instinct” have a stronger pull.
ILLUSTRATION BY FRANCESCO BONGIORNI, © 2011 www.educause.edu/er SEPTEMBER/OCTOBER 2011 EDUCAUSEreview 31 Penetrating the Fog: Analytics in Learning and Education
Meanwhile, the move toward using cades, calls have been made for reform vide insight into which students are at data and evidence to make decisions in the efficiency and quality of higher risk of dropping out or need additional is transforming other fields. Notable education. Now, with the Internet, mo- support to increase their success, and is the shift from clinical practice to bile technologies, and open education, confidence, in the learning process. evidence-based medicine in health these calls are gaining a new level of Indeed, some in higher education care. The former relies on individual urgency. Compounding this techno- have recently begun to consider how physicians basing their treatment deci- logical and social change, prominent to apply analytics to better understand sions on their personal experience with investors and businesspeople are ques- the learning process. EDUCAUSE and earlier patient cases.2 The latter is about tioning the time and monetary value of the Next Generation Learning Chal- carefully designed data collection that higher education.5 Unfortunately, the lenge, or NGLC (http://nextgenlearning builds up evidence on which clinical crescendo of calls for higher education .org/), are focusing the educational decisions are based. Medicine is look- reform lacks a foundation for making community on the possibilities that ing even further toward computational decisions on what to do or how to guide can be achieved by modeling learning Higher education, a field that gathers an astonishing array of data about its “customers,” has traditionally been inefficient in its data use, often operating with substantial delays in analyzing readily evident data and feedback.
modeling by using analytics to answer change. It is here—as a framework for interactions based on large-scale data the simple question “who will get sick?” making learning-based reform deci- collection. and then acting on those predictions to sions—that analytics will have the larg- The idea is simple yet potentially assist individuals in making lifestyle or est impact on higher education. transformative: analytics provides a health changes.3 Insurance companies new model for college and university also are turning to predictive model- Data Explosion leaders to improve teaching, learning, ing to determine high-risk customers. A byproduct of the Internet, comput- organizational efficiency, and decision Effective data analysis can produce ers, mobile devices, and enterprise making and, as a consequence, serve insight into how lifestyle choices and learning management systems (LMSs) as a foundation for systemic change. personal health habits affect long-term is the transition from ephemeral to But using analytics requires that we risks.4 Business and governments too captured, explicit data. Listening to a think carefully about what we need to are jumping on the analytics and data- classroom lecture or reading a book know and what data is most likely to tell driven decision-making trends, in the leaves limited trails. A hallway conver- us what we need to know. Continued form of “business intelligence.” sation essentially vaporizes as soon as growth in the amount of data creates Higher education, a field that gath- it is concluded. However, every click, an environment in which new or novel ers an astonishing array of data about every Tweet or Facebook status update, approaches are required to understand its “customers,” has traditionally been every social interaction, and every the patterns of value that exist within inefficient in its data use, often operat- page read online can leave a digital the data. P.W. Anderson stated that ing with substantial delays in analyzing footprint. Additionally, online learn- “more is different,” emphasizing that readily evident data and feedback. Eval- ing, digital student records, student new models of and approaches to data uating student dropouts on an annual cards, sensors, and mobile devices now interaction are desperately needed basis leaves gaping holes of delayed capture rich data trails and activity when we are confronted with abun- action and opportunities for interven- streams. dance. Or, as stated by David Gelernter: tion. Organizational processes—such These learner-produced data trails “If you have three pet dogs, give them as planning and resource allocation— provide valuable insight into what is names. If you have 10,000 head of often fail to utilize large amounts of data actually happening in the learning cattle, don’t bother.”6 Quantity changes on effective learning practices, student process and suggest ways in which the methods and approaches that we profiles, and needed interventions. educators can make improvements. use to interact with and make sense of Something must change. For de- Analysis of learner data may also pro- data.
32 EducausEreview September/October 2011 Penetrating the Fog: Analytics in Learning and Education
Google’s Marissa Mayer7 suggests world in which almost all data interac- contributes to the breadth of meanings that data is today defined by three tions, including scientific research, are attached to it. For our purposes here, a elements: affected: reasonable definition of learning analytics will help to guide discussion and frame 1. Speed—The increasing availabil- This is a world where massive activities. ity of data in real time, making it amounts of data and applied math- According to the 1st International possible to process and act on it ematics replace every other tool that Conference on Learning Analytics instantaneously might be brought to bear. Out with and Knowledge, “learning analytics is 2. Scale—Increase in computing power: every theory of human behavior, the measurement, collection, analysis Moore’s law (stating that the num- from linguistics to sociology. Forget and reporting of data about learners ber of transistors on a circuit board taxonomy, ontology, and psychol- and their contexts, for purposes of un- will double roughly every two years) ogy. Who knows why people do derstanding and optimising learning continues to hold true. what they do? The point is they do and the environments in which it oc- 3. Sensors—New types of data: “Social it, and we can track and measure it curs.”11 Academic analytics, in contrast, data is set to be surpassed in the data with unprecedented fidelity. With is the application of business intel- economy, though, by data published enough data, the numbers speak for ligence in education and emphasizes by physical, real-world objects like themselves.10 analytics at institutional, regional, and sensors, smart grids and connected international levels. John P. Camp- devices”—that is, the “Internet of The key emphasis in big data is that bell, Peter B. DeBlois, and Diana G. Things.”8 the data itself is a point of or a path to Oblinger stated: “Analytics marries value generation in organizations. Data large data sets, statistical techniques, Taken together, these three ele- is not simply the byproduct of interac- and predictive modeling. It could be ments create a situation in which exist- tions and activities within an organiza- thought of as the practice of mining in- ing data-management and decision- tion. Data is a critical value layer for stitutional data to produce ‘actionable making approaches simply are not governments, corporations, and higher intelligence.’ ”12 feasible. Understanding how activities education institutions. Learning analytics is more specific such as research, teaching, and sup- than academic analytics: the focus of port services contribute to learners’ Learning Analytics the former is exclusively on the learn- achievement is not possible in the In colleges and universities, the data ing process, as detailed in Table 1. currently largely linear data-collection focus is increasingly expressed using Academic analytics reflects the role of and data-analysis model. Information the term learning analytics. Though still a data analysis at an institutional level, abundance, and the attendant institu- young concept in education, learning an- whereas learning analytics centers on tional complexity involved in defining alytics already suffers from term sprawl. the learning process (which includes and enacting strategy, suggest rethink- The ubiquity of the term analytics partly analyzing the relationship between ing the role that analytics can play in making sense of data. Table 1: Learning and Academic Analytics Big Data Type of Analytics Level or Object of Analysis Who Benefits? Big data is a term used to describe Course-level: social networks, Learners, faculty the new context of abundance. The conceptual development, McKinsey Global Institute defines big discourse analysis, “intelligent data as “datasets whose size is beyond Learning curriculum” Analytics the ability of typical database software Departmental: predictive Learners, faculty tools to capture, store, manage and ana- modeling, patterns of success/ lyze.”9 In response to the limitations of failure existing data-management techniques, Institutional: learner profiles, Administrators, funders, a new breed of technologies (e.g., performance of academics, marketing Hadoop), databases, and techniques knowledge flow (e.g., data-mining or knowledge discov- Academic Regional (state/provincial): Funders, administrators Analytics ery in databases) has been developed. comparisons between systems As a consequence, theorists have pos- ited that something fundamental has National and International National governments, education authorities changed with the data itself, creating a
34 EducausEreview September/October 2011 Penetrating the Fog: Analytics in Learning and Education
Academic analytics reflects the role of data analysis at an institutional level, whereas learning analytics centers on the learning process (which includes analyzing the relationship between learner, content, institution, and educator).
learner, content, institution, and learners in achieving success. By an- for improvement. Learning-facing educator). alyzing discussion messages posted, analytics, such as the University The distinction of academic analyt- assignments completed, and mes- of Maryland, Baltimore County ics as similar to business intelligence sages read in LMSs such as Moodle (UMBC) Check My Activity tool, al- raises the need for a model or stage of and Desire2Learn, educators can lows learners to “compare their own learning analytics development. We identify students who are at risk of activity . . . against an anonymous propose the following cycle to reflect dropping out.13 summary of their course peers.”15 analytics in learning: 3. They can create, through trans- parent data and analysis, a shared Moving Beyond the LMS 1. Course-level: learning trails, social understanding of the institution’s Analytics from LMSs—or VLEs (Virtual network analysis, discourse analysis successes and challenges. Learning Environments), as they are 2. Educational data-mining: predic- 4. They can innovate and transform known in Europe)—offers one source tive modeling, clustering, pattern the college/university system, as of data for predicting the success of mining well as academic models and peda- learners. Morris, Finnegan, and Wu 3. Intelligent curriculum: the develop- gogical approaches. compared basic activities related to ment of semantically defined cur- 5. They can assist in making sense of LMS participation (e.g., content pages ricular resources complex topics through the combi- viewed, number of posts) and dura- 4. Adaptive content: adaptive sequence nation of social networks and tech- tion of participation (e.g., hours spent of content based on learner behav- nical and information networks: viewing discussion pages and content) ior, recommender systems that is, algorithms can recognize in LMSs and found significant dif- 5. Adaptive learning: the adaptive learn- and provide insight into data and at- ferences between “withdrawers” and ing process (social interactions, risk challenges. “successful completers,” concluding learning activity, learner support, 6. They can help leaders transition to that “time spent on task and frequency not only content) holistic decision-making through of participation are important for suc- analyses of what-if scenarios and cessful online learning.”16 Leah P. Mac- The Value of Analytics experimentation to explore how fadyen and Shane Dawson advocate for Higher Education various elements within a complex for early-warning reporting tools that Analytics spans the full scope and discipline (e.g., retaining students, “can flag at-risk students and allow in- range of activity in higher education, reducing costs) connect and to ex- structors to develop early intervention affecting administration, research, plore the impact of changing core strategies.”17 teaching and learning, and support elements. LMSs have been adopted as learn- resources. The college/university thus 7. They can increase organizational ing analytics tools because the data must become a more intentional, in- productivity and effectiveness by captured is structured and reflects the telligent organization, with data, evi- providing up-to-date information learners’ interaction within a system. dence, and analytics playing the central and allowing rapid response to But distributed networks and physical role in this transition. challenges. world interactions present additional How do big data and analytics gen- 8. They can help institutional leaders challenges for analytics. For example, erate value for higher education? determine the hard (e.g., patents, most LMS analytics models do not cap- research) and soft (e.g., reputation, ture activity by online learners outside 1. They can improve administrative profile, quality of teaching) value of an LMS (i.e., in Facebook, Twitter, or decision-making and organiza- generated by faculty activity.14 blogs). Similarly, most analytics models tional resource allocation. 9. They can provide learners with in- do not capture or utilize physical- 2. They can identify at-risk learners sight into their own learning habits world data, such as library use, access to and provide intervention to assist and can give recommendations learning support, or academic advising.
36 EducausEreview September/October 2011 Penetrating the Fog: Analytics in Learning and Education
figUrE 1: AssEssmEnT ThroUgh AnALyTics of the success that has accompanied the development of the Khan Academy learning modules, even with their sim- Architecture of plistic, mastery-based approach.18 ▲ Learner ▲ Degree/ knowledge knowledge in a discipline Qualifi cation concluding Thoughts Learning analytics is still in the early stages of implementation and experi- ▲ mentation. Numerous questions exist
around how analytics relates to exist- ▲ ing organizational systems. Campbell, Developed through DeBlois, and Oblinger detailed the various concerns that the use of ana-
lytics generates in higher education, ▲ Intelligent data
▲ including privacy, profiling, informa- Formal Informal tion sharing, and data stewardship.19 How can the potential value of the data Mobile devices such as smartphones dicting what they’ll do in the future. be leveraged without succumbing to and tablets/iPads offer the prospect of Analytics in education must be trans- the dangers associated with tracking bridging the divide between the physi- formative, altering existing teaching, students’ learning options based on cal and digital worlds by capturing learning, and assessment processes, deterministic modeling? Additionally, location and activity. Similarly, clickers academic work, and administration. how transparent are the algorithms in classrooms can be integrated with When analytics is applied to cur- and weighting of analytics? How “real data from learners’ activity in online ricular resources, the traditional view time” should analytics be in classroom environments, providing additional of courses is disrupted. The knowledge, settings? Finally, since we risk a return insight into factors that contribute to attitudes, and skills required in any to behaviorism as a learning theory if learners’ success. domain can be rendered as a network we confine analytics to behavioral data, Massive Open Online Courses of relations. The semantic web and how can we account for more than be- (MOOCs), which occur in decentral- linked data are partial instantiations of havioral data? ized, distributed teaching and learning this concept. Knowledge domains can Undoubtedly, analytics and big data networks, offer another challenge. On- be mapped, and learner activity can have a significant role to play in the fu- line social media monitoring tools (e.g., be evaluated in relation to those maps. ture of higher education. The growing Radian6) and reputation or influence Instead of being an “end of course” ac- role of analysis techniques and tech- monitoring tools (e.g., Klout) may pro- tivity, assessment is performed in real nologies in government and business vide educators with a model for analyt- time as learners demonstrate mastery of sectors affirms this trend. In education ics in such networks, in which activity important concepts or ideas (see Figure the value of analytics and big data can is distributed across multiple sites and 1). Learning content is not provided in be found in (1) their role in guiding multiple identities. a packaged textbook but is rendered or reform activities in higher education, computed ”on the fly,” providing each and (2) how they can assist educators in intelligent curriculum learner with resources relevant to his improving teaching and learning. It is not sufficient to treat big data and or her profile, learning goals, and the Yet there are reasons to be cautious analytics as useful only for evaluating knowledge domain the learner is at- as the development of analytical tools what learners have done and for pre- tempting to master. This is the essence for modeling learners’ interactions It is not sufficient to treat big data and analytics as useful only for evaluating what learners have done and for predicting what they’ll do in the future. Analytics in education must be transformative.
38 EducausEreview sEptEmbEr/OctObEr 2011 Penetrating the Fog: Analytics in Learning and Education
gains attention. Like other behavior their peers or about their progress in publications/big_data/pdfs/MGI_big_data_ patterns, models that are deterministic relation to their personal goals can be exec_summary.pdf>. 10. Chris Anderson, “The End of Theory: The Data assume that future conditions can be motivating and encouraging. Finally, Deluge Makes the Scientific Method Obsolete,” completely determined by knowing administrators and decision-makers Wired, June 23, 2008, 40 EducausEreview September/October 2011