Challenges of Realising Scalable Open Education with (Peer) Feedback
Total Page:16
File Type:pdf, Size:1020Kb
Challenges of realising scalable open education with (peer) feedback Open HPI Forum, Potsdam, Germany 28. November 2019 Prof. Dr. Marco Kalz, Professor of Technology Enhanced Learning, Heidelberg University of Education [email protected] mkalz PROJECT BACKGROUND HTTP://WWW.SOONER.NU 4 SOONER PHD PROJECTS MICRO: MESO: MACRO: Learners Course Organizational level level Level STRUKTUR DES VORTRAGS PhD A PhD B PhD C PhD D Self-regulated learning Motivation and Scaling of support, Higher education skill acquisition intentions as key to feedback and institutions: drop-out interaction governance, development and educational innovation Fundamental research Accompanying research PROJECT BACKGROUND MOOCS SPOCS PLATFORMS BLENDED LEARNING SPOCS OOE innovation projects in The Netherlands (2015 - 2018) SCALABILITY CHALLENGE IN EDUCATION Students Year 99 2000 ??? ??? Accreditation Indicators: 153 2009 - Teacher quality - Student – staff ratio - Contact hours 262 2025 414 2030 (Lane, 2014) (UNESCO, 2009; ICDE, 2015) GLOBAL LEARNING AT SCALE 160 000 enrolled THE IRON TRIANGLE Lane, 2014 BREAKING THE IRON TRIANGLE Student-Student student content teacher Content-Content Teacher-Teacher Sufficient levels of deep and meaningful learning can be developed, as long as one of the three forms of interaction (student-teacher; student-student; student-content) is at a very high level. (Anderson, 2002) BREAKING THE IRON TRIANGLE Student-Student student Opening the Iron Triangle: Students: tools & methods to support interactions of: - Student – Student - Student – Content - Student – Teacher content teacher Content-Content Teacher-Teacher Scalable Design Scalable technology Worked out Bots and virtual examples tutors Self-organised Matchmaking SCALABLE groups (Question- Answering) METHODS Peer-Assessment Recommender Systems Self-Assessment Prediction- Algorithms SCALE SCALABILITY THE TEACHER LOAD/BANDWIDTH EDUCATIONAL SCALABILITY IS THE CAPACITY OF AN EDUCATIONAL FORMAT TO MAINTAIN HIGH QUALITY DESPITE INCREASING OR LARGE NUMBERS OF LEARNERS AT A STABLE LEVEL OF TOTAL COSTS WORKING DEFINITION A Framework towards Educational STUDY 1 Scalability of Open Online Course Kasch, J., Van Rosmalen, P., & Kalz, M. (2017). A Framework towards Educational Scalability of Open Online Courses. Journal of Universal Computer Science, 23(9), 845-867. STUDY 1: MODEL OF EDUCATIONAL SCALABILITY • Goal was to develop a model of educational scalability that can be differentiated from technical scalability • Teacher bandwidth as an influencing factor • Differentiation on different cognitive levels • Determine scalable practices in MOOCs regarding support and formative assessment & feedback STUDY 1: MODEL OF EDUCATIONAL SCALABILITY CONSTRUCTIVE COMPLEXITY INTERACTION FORMATIVE ALIGNMENT FEEDBACK Kasch, van Rosmalen & Kalz, 2017 CONSTRUCTIVE ALIGNMENT Biggs, 2003 COMPLEXITY Miller, 1990 INTERACTION QUALITY (Anderson, 2002) FORMATIVE FEEDBACK T&L, 2018 A MODEL OF EDUCATIONAL SCALABILITY Kasch, van Rosmalen & Kalz, 2017 22 ANALYSIS EXAMPLE Kasch, J., Van Rosmalen, P., & Kalz, M. (2017). A Framework towards Educational Scalability of Open Online Courses. Journal of Universal Computer Science, 23(9), 845-867. STUDY 1: MODEL OF EDUCATIONAL SCALABILITY § By using our analysis we found: § concept of scale is not coherently operationalised and an implicit quantitative and technical understanding dominates § MOOC design focuses often on low complexity learning activities with low interactivity and collaboration § There seems to be a challenge in providing complex and interactive learning activities with low teacher cost STUDY 1: MODEL OF EDUCATIONAL SCALABILITY § Course design should focus both on quality as well as quantity: OER ≠ Education and scale ≠ educational scalability § Large scale courses (online & offline) can benefit from design recommendations regarding formative assessment and feedback, e.g. § which feedback types are used and how (feed-up, feedback, feedforward)? § how to provide elaborate feedback? Educational STUDY 2 Scalability in MOOC Design Kasch, J., Van Rosmalen, P., & Kalz, M. (in review). Educational Scalability in MOOCs. Analysing instructional design to find best practices. EDUCATIONAL SCALABILITY IN MOOC DESIGN (GOAL/METHOD) • Goal of the study was to identify best practices in scalable MOOC design • Stratified sample from 50 active MOOCs selected from ClassCentral (including MOOCs on Coursera, edX, FutureLearn, Open2Study) • Analysis of educational design of mid-week of the MOOC Kasch, van Rosmalen, & Kalz, in review EDUCATIONAL SCALABILITY IN MOOC DESIGN (GOAL/METHOD) • Analysis via standardized questionnaire on the basis of the model of educational scalability • Pre-analysis of 5 MOOCs by 2 independent raters with instrument from study 1 lead to an interrater reliability of k = 0.96 • Results stored in Limesurvey plus 4 screenshots per MOOC Kasch, van Rosmalen, & Kalz, in review EDUCATIONAL SCALABILITY IN MOOC DESIGN (STUDY 2) Kasch, van Rosmalen, & Kalz, in review EDUCATIONAL SCALABILITY IN MOOC DESIGN (STUDY 2) Kasch, van Rosmalen, & Kalz, in review EDUCATIONAL SCALABILITY IN MOOC DESIGN (STUDY 2) Examples of scalable best practices (student- content) • Automated elaborated feedback in MCQs • Video-based hints on request • Adding support material to MCQs with feedback • In case of incorrect answers, reference is given to course material/videos (feed forward) Kasch, van Rosmalen, & Kalz, in review EDUCATIONAL SCALABILITY IN MOOC DESIGN (STUDY 2) Examples of scalable best practices (student- student) • Peer-feedback in multiple options • Guided discussions with prompts and guidelines on answering and commenting Kasch, van Rosmalen, & Kalz, in review EDUCATIONAL SCALABILITY IN MOOC DESIGN (STUDY 2) Examples of scalable best practices (student- teacher) • Proxy-feedback for self-comparison • Live-sessions Kasch, van Rosmalen, & Kalz, in review CONCLUSIONS • Majority of activities was of low complexity • Lack of clarity with regards to collaborative learning • Lack of student-teacher interaction • Learning objectives not measurable or not mentioned confirming earlier studies like Margaryan, Bianco, & Littlejohn, (2015) or recently Egloffstein, Kögler & Iffenthaler (2019). • BUT: Goal was to find best practices for educational scalability which could be identified in the sample Kasch, van Rosmalen, & Kalz, in review Impact of Online Peer-Feedback Training and Prior Experience on Student´s Peer STUDY 3 Feedback Perceptions in a Massive Open Online Course Kasch, J., Van Rosmalen, P., & Kalz, M. (in preparation). Impact of Online Peer- Feedback Training and Prior Experience on Student´s Peer Feedback Perceptions in a Massive Open Online Course. RESEARCH QUESTIONS (STUDY 3) • RQ1: To what extent does online peer-feedback training in a MOOC influence students’ perception of peer-feedback and peer- feedback training? • RQ2: What is the relationship between students’ peer-feedback experience and their peer- feedback perceptions? RESEARCH QUESTIONS (METHOD) Kasch, Van Rosmalen, & Kalz (in preparation). VARIABLES AND ANALYSIS (METHOD) • Variables measured: Perception variables (Willingness, Usefulness, Preparedness, General attitude), Demographics, Peer Feedback experience, • T-tests to compare means of treatment and control group (N=45) • Regression analysis to check impact of prior experience on the dependent variables RESULTS (1/2) • No significant differences between treatment and control group for the perception variables • Peer-feedback training did not change the perception of learners • Amount of peer feedback experience has significant impact on perception RESULTS (2/2) • Learners with no peer-feedback experience had significant higher perception than those who experienced it once • Learners with more feedback experience had significantly higher perception than those without experience Rather than talking about learning at scale we should shift the discussion to educational scalability CONCLUSIONS Educational scalability is defined by mix of quantitative and (1/3) qualitative dimensions Educational scalability can be high, but happens often on lower levels of complexity Many MOOCs still lack clarity and ID quality CONCLUSIONS Lack of student-teacher (2/3) interaction BUT: Several good practices of scalable feedback practices could be identified in the sample The usefulness of peer-feedback as scalable practice in MOOCs depends on positive learner perception CONCLUSIONS Perception is more likely to be influenced by repeated positive (3/3) experiences rather than training Scalable digital feedback practices can also influence scalability challenges in campus education RESEARCH TEAM Julia Kasch, Open University of the Netherlands [email protected] Dr. Peter van Rosmalen, Maastricht University [email protected] Prof. Dr. Marco Kalz, Heidelberg University of Education [email protected] Thank you! [email protected] http://twitter.com/mkalz.