Chapter 1: Theory and Learning Analytics Simon Knight, Simon Buckingham Shum Connected Intelligence Centre, University of Technology Sydney, Australia DOI: 10.18608/hla17.001 ABSTRACT 7KHFKDOOHQJHRIXQGHUVWDQGLQJKRZWKHRU\DQGDQDO\WLFVUHODWHLVWRPRYHÜIURPFOLFNVWR FRQVWUXFWVÝLQDSULQFLSOHGZD\/HDUQLQJDQDO\WLFVDUHDVSHFLāFLQFDUQDWLRQRIWKHELJJHU shift to an algorithmically pervaded society, and their wider impact on education needs careful consideration. In this chapter, we argue that by design — or else by accident — the use of a learning analytics tool is always aligned with assessment regimes, which are in turn grounded in epistemological assumptions and pedagogical practices. Fundamentally WKHQZHDUJXHWKDWGHSOR\LQJDJLYHQOHDUQLQJDQDO\WLFVWRROH[SUHVVHVDFRPPLWPHQWWR a particular educational worldview, designed to nurture particular kinds of learners. We outline some key provocations in the development of learning analytic techniques, key questions to draw out the purpose and assumptions built into learning analytics. We suggest WKDWXVLQJÜFODLPVDQDO\VLVÝØDQDO\VLVRIWKHLPSOLFLWRUH[SOLFLWVWDQFHVWDNHQLQWKHGHVLJQ and deploying of technologies — is a productive human-centred method to address these NH\TXHVWLRQVDQGZHRIIHUVRPHH[DPSOHVRIWKHPHWKRGDSSOLHGWRWKRVHSURYRFDWLRQV Keywords: Theory, assessment regime, claims analysis In what has become a well-cited, popular article in ZLOODOZD\VEHVLJQLāFDQW">â@,QVXPZKHQ Wired PDJD]LQHLQWKHQHZHUDRISHWDE\WHVFDOH working with big data, theory is actually more data and analytics, Anderson (2008) envisaged the important, not less, in interpreting results and GHDWKRIWKHRU\PRGHOVDQGWKHVFLHQWLāFPHWKRG1R identifying meaningful, actionable results. longer do we need to create theories about how the )RUWKLVUHDVRQZHKDYHRIIHUHG'DWD*HRORJ\ world works, because the data will tell us directly as 6KDIIHU$UDVWRRSRXUHWDO DQG'DWD we discern, in almost real time, the impacts of probes Archeology (Wise, 2014) as more appropriate and changes we make. PHWDSKRUVWKDQ'DWD0LQLQJIRUWKLQNLQJ 7KLVKLJKSURāOHDUWLFOHDQGVRPHZKDWH[WUHPHFRQFOX - about how we sift through the new masses of VLRQDORQJZLWKRWKHUV VHHIRUH[DPSOH0D\HU6FK·Q - data while attending to underlying conceptual berger & Cukier, 2013), has, not surprisingly, attracted UHODWLRQVKLSVDQGWKHVLWXDWLRQDOFRQWH[W FULWLFLVP ER\G &UDZIRUG3LHWVFK 'DWDLQWHQVLYHPHWKRGVDUHKDYLQJDQGZLOOFRQWLQXH Educational researchers are one community interested WRKDYHDWUDQVIRUPDWLYHLPSDFWRQVFLHQWLāFLQTXLU\ LQWKHDSSOLFDWLRQRIÜELJGDWDÝDSSURDFKHVLQWKHIRUP +H\7DQVOH\ 7ROOH ZLWKIDPLOLDUÜELJVFLHQFHÝ RIOHDUQLQJDQDO\WLFV$FULWLFDOTXHVWLRQWXUQVRQH[ - H[DPSOHVLQFOXGLQJJHQHWLFVDVWURQRP\DQGKLJK BRCA2 gene Red Dwarf actly how theory could, or should shape research in energy physics. The , stars, Higgs bosun this new paradigm. Equally, a critical view is needed and the do not hold strong views on being on how the new tools of the trade enhance/constrain computationally modelled, or who does what with the people WKHRUL]LQJE\YLUWXHRIZKDWWKH\GUDZDWWHQWLRQWR results. However, when become aware that and what they ignore or downplay. Returning to our their behaviour is under surveillance, with potentially opening provocation from Anderson, the opposite important consequences, they may choose to adapt FRQFOXVLRQLVGUDZQE\:LVHDQG6KDIIHU S RUGLVWRUWWKHLUEHKDYLRXUWRFDPRXĂDJHDFWLYLW\RU to game the system. Learning analytics researchers :KDWFRXQWVDVDPHDQLQJIXOāQGLQJZKHQWKH aiming to study learning using such tools must do so number of data points is so large that something aware that they have adopted a particular set of lenses CHAPTER 1 THEORY AND LEARNING ANALYTICS PG 17 RQÜOHDUQLQJÝWKDWDPSOLI\DQGGLVWRUWLQSDUWLFXODU dence, at scale, the kinds of process-intensive learning ways, and that may unintentionally change the system that educators have long argued for, but have to date being tracked. Researchers should stay alert to the SURYHQLPSUDFWLFDO"([DFWO\ZKDWRQHVHHNVWRGRZLWK emerging critical discourse around big data in soci- analytics is at the heart of this chapter. ety, data-intensive science broadly, as well as within To design analytics-based lenses — with our eyes wide education where the debate is at a nascent stage. RSHQWRWKHULVNVRIGLVWRUWLQJRXUGHāQLWLRQRIÜOHDUQLQJÝ Let us turn now to educators and learners . The potential in our desire to track it computationally — we must RIOHDUQLQJDQDO\WLFVLVDUJXDEO\IDUPRUHVLJQLāFDQW XQSLFNZKDWLVDWVWDNHZKHQFODVVLāFDWLRQVFKHPHV than as an enabler of data-intensive educational machine learning, recommendation algorithms, and UHVHDUFKH[FLWLQJDVWKLVLV7KHQHZSRVVLELOLW\LV YLVXDOL]DWLRQVPHGLDWHWKHUHODWLRQVKLSVEHWZHHQ that educators and learners — the stakeholders who educators, learners, policymakers, and researchers. constitute the learning system studied for so long by The challenge of understanding how theory and an- UHVHDUFKHUVØDUHIRUWKHāUVWWLPHDEOHWRVHHWKHLU DO\WLFVUHODWHLVWRPRYHÜIURPFOLFNVWRFRQVWUXFWVÝ own processes and progress rendered in ways that in a principled way. until now were the preserve of researchers outside the /HDUQLQJDQDO\WLFVDUHDVSHFLāFLQFDUQDWLRQRIWKH V\VWHP'DWDJDWKHULQJDQDO\VLVLQWHUSUHWDWLRQDQG bigger shift to an algorithmically pervaded society. The even intervention (in the case of adaptive software) is frame we place around the relationship of theory to no longer the preserve of the researcher, but shifts to learning analytics must therefore be enlarged beyond embedded sociotechnical educational infrastructure. FRQVLGHUDWLRQVRIZKDWLVQRUPDOO\FRQVLGHUHGÜHGX - educators and learners So, for , the interest turns on FDWLRQDOWKHRU\ÝWRHQJDJHZLWKWKHFULWLFDOGLVFRXUVH the ability to gain insight in a timely manner that could around how sociotechnical infrastructures deliver improve outcomes. computational intelligence in society. Thus, with people in the analytic loop, the system The remainder of the chapter argues that by design EHFRPHVUHĂH[LYH SHRSOHFKDQJHLQUHVSRQVHWRWKH — or else by accident — the use of a learning analytics DFWRIREVHUYDWLRQDQGH[SOLFLWIHHGEDFNORRSV DQG tool is always aligned with assessment regimes , which ZHFRQIURQWQHZHWKLFDOGLOHPPDV 3DUGR 6LHPHQV are in turn grounded in epistemological assumptions 3ULQVORR 6ODGH 7KHGHVLJQFKDOOHQJH and pedagogical practices 0RUHRYHUDVZHVKDOOH[ - moves from that of modelling closed, deterministic plain, a long history of design thinking demonstrates V\VWHPVLQWRWKHVSDFHRIÜZLFNHGSUREOHPVÝ 5LWWHO that designed artifacts unavoidably embody implicit DQGFRPSOH[DGDSWLYHV\VWHPV 'HDNLQ&ULFN values and claims. Fundamentally then, we argue that 0DFIDG\HQ'DZVRQ3DUGR *DÇHYLİ $V GHSOR\LQJDJLYHQOHDUQLQJDQDO\WLFVWRROH[SUHVVHVD we hope to clarify, for someone trying to get a robust commitment to a particular educational worldview, PHDVXUHRIÜOHDUQLQJÝIURPGDWDWUDFHVVXFKUHĂH[LYLW\ designed to nurture particular kinds of learners. will be either a curse or a blessing, depending on how important learner agency and creativity are deemed THEORY INTO PRACTICE WREHKRZā[HGWKHLQWHQGHGOHDUQLQJRXWFRPHVDUH whether analytical feedback loops are designed as interventions to shape learner cognition/interaction, In an earlier paper (Knight, Buckingham Shum, & Lit- and so forth. tleton, 2014) we put forward a triadic depiction of the relationship between elements of theory and practice Our view is that it is indeed likely that education, as in the development of learning analytic techniques, ERWKDUHVHDUFKāHOGDQGDVDSURIHVVLRQDOSUDFWLFH as depicted in Figure 1.1 (we refer the reader to this is on the threshold of a data-intensive revolution paper for further discussion of the depicted relation- DQDORJRXVWRWKDWH[SHULHQFHGE\RWKHUāHOGV$VWKH ships). Our intention was to illustrate the tensions and site of political and commercial interests, education inter-relations among the more or less theoretically LVGULYHQE\SROLF\LPSHUDWLYHVIRUÜLPSDFWHYLGHQFHÝ grounded stances we take through our pedagogic and and software products shipping with analytics dash- assessment practices and policies, and their underlying boards. While such drivers are typically viewed with epistemological implications and assumptions. suspicion by educational practitioners and researchers, the opportunity is to be welcomed if we can learn how The use of a triangle highlights these tensions: that to harness and drive the new horsepower offered by assessment can be the driving force in how we un- analytics engines, in order to accelerate innovation and GHUVWDQGZKDWÜNQRZOHGJHÝLVWKDWDVVXPSWLRQVDERXW improve evidence-based decision-making. Systemic SHGDJRJ\ IRUH[DPSOHDNLQGRIIRONSV\FKRORJ\2OVRQ educational shifts are of course tough to effect, but %UXQHU LQĂXHQFHZKRZHDVVHVVDQGKRZWKDW could it be that analytics tools offer new ways to evi- assessment and pedagogy are sometimes in tension, where the desire for summative assessment overrides PG 18 HANDBOOK OF LEARNING ANALYTICS pedagogically motivated formative feedback; and that centred on the triad of epistemology, pedagogy, and drawing alignment between one’s epistemological assessment. view (of the nature of knowledge) and assessment We use these provocations to illustrate how the im- or pedagogy practices is challenging — relationships SOLFLWÜFODLPVÝPDGHE\DOHDUQLQJDQDO\WLFVWRROFDQ between the three may be implied, but they are not EHGHFRQVWUXFWHG7KHDSSURDFKLQYLWHVUHĂHFWLRQ entailed 'DYLV :LOOLDPV 2IFRXUVHRWKHU on the affordances of the tool’s design at different YLVXDOL]DWLRQVPLJKWEHLPDJLQHGDQGWKHWKHRUHWLFDO OHYHOV LQFOXGLQJGDWDPRGHOOHDUQHUH[SHULHQFHDQG and practical purposes for which such heuristics are OHDUQLQJDQDO\WLFVYLVXDOL]DWLRQ GHYLVHGLVLPSRUWDQWWRFRQVLGHU7RJLYHWZRH[DP