Contextualizable Learning Analytics Design: A Generic Model and Writing Analytics Evaluations Antonette Shibani† Simon Knight Simon Buckingham Shum University of Technology Sydney University of Technology Sydney University of Technology Sydney Sydney NSW Australia Sydney NSW Australia Sydney NSW Australia [email protected] [email protected] [email protected] ABSTRACT 1 Introduction A major promise of learning analytics is that through the collection of large amounts of data we can derive insights from With the growing quantity of data generated from the Internet of authentic learning environments, and impact many learners at Things (IoT) and digitization, there is increasing interest in scale. However, the context in which the learning occurs is analytics and big data. In education, high profile initiatives like important for educational innovations to impact student learning. massive online courses (MOOCS), online video-based learning, In particular, for student-facing learning analytics systems like educational apps, and personal and portable computing devices, feedback tools to work effectively, they have to beintegrated with have provided access to increased quantities of learner data [25]. pedagogical approaches and the learning design. This paper Thus, analytics and big data in education holds the potential to proposes a conceptual model to strike a balance between the develop scalable methods that can be employed widely across concepts of generalizable scalable support and contextualized specific support by clarifying key elements that help to large numbers of students and institutions; to obtain big impact contextualize student-facing learning analytics tools. We from big data [25]. While data was previously available in demonstrate an implementation of the model using a writing educational research, learning analytics holds the promise of analytics example, where the features, feedback and learning providing insight for learning research through the longitudinal activities around the automated writing feedback tool are tuned collection of data on various levels of granularity from multiple for the pedagogical context and the assessment regime in hand, sources in authentic learning environments [28]. by co-designing them with the subject experts. The model can be However, an inherent problem in the application of such employed for learning analytics to move from generalized support technologies for education is that educational systems are to meaningful contextualized support for enhancing learning. contextual, meaning different factors like educators, instruction, and task design influence learning in its pedagogical setting. Thus, CCS CONCEPTS generalized solutions cannot cater to all learning contexts in the • Applied computing ~ Computer-assisted instruction same way. For instance, a predictive model trained on data from one discipline may not be generalizable to another discipline if KEYWORDS there are contextual factors that affect the model performance. contextualizable learning analytics, learning design, conceptual The move from big data to meaningful data is important for model, CLAD, writing analytics learning analytics researchers to improve learning [11; 25]. While general productivity and management tools like writing editors, ACM Reference format: and LMSs work well in many cases, the context is particularly Antonette Shibani, Simon Knight, and Simon Buckingham Shum. 2019. important for student-facing tools that provide feedback on Contextualizable Learning Analytics Design: A Generic Model and higher-order constructs of learning. A recognized challenge in the Writing Analytics Evaluations. In Proceedings of the International field of learning analytics is the uncertainty around “LA’s Conference on Learning Analytics and Knowledge (LAK’19). ACM, New pedagogical relevance and value-add in contextualized learning York, NY, USA, 10 pages. https://doi.org/10.1145/3303772.3303785 and teaching settings across different disciplinary domains” [37]. In an empirical study predicting academic success in a blended Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed learning model, it was found that granular course-specific models for profit or commercial advantage and that copies bear this notice and the full citation provided better insights to the instructors compared to the generic on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or models. This prompted a call for predictive models in learning republish, to post on servers or to redistribute to lists, requires prior specific permission analytics to account for instructional conditions so that learning and/or a fee. Request permissions from [email protected]. analytics does not promote a one-size-fits-all approach [10]. That LAK19, March 4–8, 2019, Tempe, AZ, USA is, even in in large scale learning environments this connection © 2019 Copyright is held by the owner/author(s). Publication rights licensed to ACM. between Learning Design (LD) and Learning Analytics (LA) has ACM 978-1-4503-6256-6/19/03…$15.00 https://doi.org/10.1145/3303772.3303785 been emphasized as a means for the pedagogical context to enhance our interpretation of analytics into actionable insights Contextualizable Learning Analytics Design LAK’19, March 2019, Tempe, Arizona USA and meaningful interpretations [10]. While theories connecting Learning analytics pedagogic intervention designs have been LA applications with pedagogy are receiving increasing attention, proposed for integrating LA technologies as part of a larger the challenge of developing discipline-specific and contextualized educational context [41]. This way, Learning Analytics can focus LA systems remains. Most systems are developed to be on ‘augmenting’, rather than revolutionizing existing high quality generalizable and open for wider contexts, and very few learning pedagogy by enhancing classroom practices [20]. By finding new analytics systems can provide contextualized support [22]. methods to solve existing pedagogical issues, LA can contribute Scalability and contextualization seem to contradict each to existing good practices by improving them with insights and other in the sense that they prioritize different objectives. This technical affordances, rather than analytics remaining separated arises from the tension between quantitative approaches that from current classroom teaching and learning practices. often identify generalizable regularities in learning, and Considering classroom-level constraints, and orchestrating LA to qualitative approaches that tend to understand the particularities suit them can better aid the adoption of LA technologies [27]. tied to specific contexts [42]. When structural relations identified To aid this LA-LD alignment, Bakharia et al, proposed a by computational approaches are given more importance than conceptual framework linking learning analytics to learning nuanced processes, it leads to the concern of favoring design. In this framework, the teacher plays a central role by generalizability over contextualization. So how do we strike a bringing context to the analysis, making decisions on the feedback balance between these two? This paper will examine the question to be provided to students and in the adaptation of learning design of how we develop scalable learning analytics applications that [5]. Similarly, Alhadad and Thompson [2] propose the mediation can cater to large number of students, while also catering for of effective teacher inquiry processes, enhancing opportunities specific contexts by considering the nuances that make them for genuinely evidence-informed practice by avoiding the distinctive. The next section will introduce relationships between oversimplification of learning enhancement to a data-driven learning analytics and learning design, before we introduce a process. Despite the importance of aligning LA and LD, few model for identifying four key elements for contextualizing LA for empirical studies demonstrate how this alignment happens in pedagogical contexts. An exemplar implementation is then practice across pedagogical settings. A detailed systematic review provided from writing analytics, in which a tool is customized for of 43 studies in the current landscape connecting LA and LD can specific contexts by embedding it in two different learning be found in [24]. Based on this review, Mangaroska and contexts to meet their respective pedagogical needs. Giannakos suggest that a framework should be developed to capture and systematize learning design grounded in learning analytics, and study how learning design choices influence 2 Learning Analytics and Learning Design learning and performance over time. They also emphasize the Learning analytics promises much more than scaling up using big need for educators to design for learning by making use of LA data, because it is about meaningful data for learning [25]. While affordances as opposed to being providers of knowledge only. much learning analytics work has been conducted with large An example system providing an instructor’s personalized quantities of data in institutional contexts, for senior support to students at scale was E2Coach, which provided management, curriculum designers, and researchers, there is an encouragement, guidance and