Challenges of realising scalable open education with (peer) feedback
Open HPI Forum, Potsdam, Germany 28. November 2019
Prof. Dr. Marco Kalz, Professor of Technology Enhanced Learning, Heidelberg University of Education
mkalz PROJECT BACKGROUND HTTP://WWW.SOONER.NU 4 SOONER PHD PROJECTS
MICRO: MESO: MACRO: Learners Course Organizational level level Level STRUKTUR DES VORTRAGS PhD A PhD B PhD C PhD D
Self-regulated learning Motivation and Scaling of support, Higher education skill acquisition intentions as key to feedback and institutions: drop-out interaction governance, development and educational innovation
Fundamental research Accompanying research PROJECT BACKGROUND
MOOCS
SPOCS
PLATFORMS
BLENDED LEARNING
SPOCS
OOE innovation projects in The Netherlands (2015 - 2018) SCALABILITY CHALLENGE IN EDUCATION
Students Year
99 2000
??? ??? Accreditation Indicators: 153 2009 - Teacher quality - Student – staff ratio - Contact hours 262 2025
414 2030
(Lane, 2014) (UNESCO, 2009; ICDE, 2015) GLOBAL LEARNING AT SCALE
160 000 enrolled THE IRON TRIANGLE
Lane, 2014 BREAKING THE IRON TRIANGLE
Student-Student
student
content teacher
Content-Content Teacher-Teacher
Sufficient levels of deep and meaningful learning can be developed, as long as one of the three forms of interaction (student-teacher; student-student; student-content) is at a very high level.
(Anderson, 2002) BREAKING THE IRON TRIANGLE
Student-Student
student Opening the Iron Triangle: Students: tools & methods to support interactions of: - Student – Student - Student – Content - Student – Teacher
content teacher
Content-Content Teacher-Teacher Scalable Design Scalable technology Worked out Bots and virtual examples tutors Self-organised Matchmaking SCALABLE groups (Question- Answering) METHODS Peer-Assessment Recommender Systems Self-Assessment Prediction- Algorithms
SCALE
SCALABILITY THE TEACHER LOAD/BANDWIDTH EDUCATIONAL SCALABILITY IS THE CAPACITY OF AN EDUCATIONAL FORMAT TO MAINTAIN HIGH QUALITY DESPITE INCREASING OR LARGE NUMBERS OF LEARNERS AT A STABLE LEVEL OF TOTAL COSTS
WORKING DEFINITION A Framework towards Educational STUDY 1 Scalability of Open Online Course
Kasch, J., Van Rosmalen, P., & Kalz, M. (2017). A Framework towards Educational Scalability of Open Online Courses. Journal of Universal Computer Science, 23(9), 845-867. STUDY 1: MODEL OF EDUCATIONAL SCALABILITY
• Goal was to develop a model of educational scalability that can be differentiated from technical scalability • Teacher bandwidth as an influencing factor • Differentiation on different cognitive levels • Determine scalable practices in MOOCs regarding support and formative assessment & feedback STUDY 1: MODEL OF EDUCATIONAL SCALABILITY
CONSTRUCTIVE COMPLEXITY INTERACTION FORMATIVE ALIGNMENT FEEDBACK
Kasch, van Rosmalen & Kalz, 2017 CONSTRUCTIVE ALIGNMENT
Biggs, 2003 COMPLEXITY
Miller, 1990 INTERACTION QUALITY
(Anderson, 2002) FORMATIVE FEEDBACK
T&L, 2018 A MODEL OF EDUCATIONAL SCALABILITY
Kasch, van Rosmalen & Kalz, 2017 22 ANALYSIS EXAMPLE
Kasch, J., Van Rosmalen, P., & Kalz, M. (2017). A Framework towards Educational Scalability of Open Online Courses. Journal of Universal Computer Science, 23(9), 845-867. STUDY 1: MODEL OF EDUCATIONAL SCALABILITY
§ By using our analysis we found:
§ concept of scale is not coherently operationalised and an implicit quantitative and technical understanding dominates
§ MOOC design focuses often on low complexity learning activities with low interactivity and collaboration
§ There seems to be a challenge in providing complex and interactive learning activities with low teacher cost STUDY 1: MODEL OF EDUCATIONAL SCALABILITY
§ Course design should focus both on quality as well as quantity: OER ≠ Education and scale ≠ educational scalability
§ Large scale courses (online & offline) can benefit from design recommendations regarding formative assessment and feedback, e.g. § which feedback types are used and how (feed-up, feedback, feedforward)? § how to provide elaborate feedback? Educational STUDY 2 Scalability in MOOC Design
Kasch, J., Van Rosmalen, P., & Kalz, M. (in review). Educational Scalability in MOOCs. Analysing instructional design to find best practices. EDUCATIONAL SCALABILITY IN MOOC DESIGN (GOAL/METHOD)
• Goal of the study was to identify best practices in scalable MOOC design • Stratified sample from 50 active MOOCs selected from ClassCentral (including MOOCs on Coursera, edX, FutureLearn, Open2Study) • Analysis of educational design of mid-week of the MOOC
Kasch, van Rosmalen, & Kalz, in review EDUCATIONAL SCALABILITY IN MOOC DESIGN (GOAL/METHOD) • Analysis via standardized questionnaire on the basis of the model of educational scalability • Pre-analysis of 5 MOOCs by 2 independent raters with instrument from study 1 lead to an interrater reliability of k = 0.96 • Results stored in Limesurvey plus 4 screenshots per MOOC
Kasch, van Rosmalen, & Kalz, in review EDUCATIONAL SCALABILITY IN MOOC DESIGN (STUDY 2)
Kasch, van Rosmalen, & Kalz, in review EDUCATIONAL SCALABILITY IN MOOC DESIGN (STUDY 2)
Kasch, van Rosmalen, & Kalz, in review EDUCATIONAL SCALABILITY IN MOOC DESIGN (STUDY 2)
Examples of scalable best practices (student- content) • Automated elaborated feedback in MCQs • Video-based hints on request • Adding support material to MCQs with feedback • In case of incorrect answers, reference is given to course material/videos (feed forward)
Kasch, van Rosmalen, & Kalz, in review EDUCATIONAL SCALABILITY IN MOOC DESIGN (STUDY 2)
Examples of scalable best practices (student- student) • Peer-feedback in multiple options • Guided discussions with prompts and guidelines on answering and commenting
Kasch, van Rosmalen, & Kalz, in review EDUCATIONAL SCALABILITY IN MOOC DESIGN (STUDY 2)
Examples of scalable best practices (student- teacher) • Proxy-feedback for self-comparison • Live-sessions
Kasch, van Rosmalen, & Kalz, in review CONCLUSIONS
• Majority of activities was of low complexity • Lack of clarity with regards to collaborative learning • Lack of student-teacher interaction • Learning objectives not measurable or not mentioned confirming earlier studies like Margaryan, Bianco, & Littlejohn, (2015) or recently Egloffstein, Kögler & Iffenthaler (2019). • BUT: Goal was to find best practices for educational scalability which could be identified in the sample
Kasch, van Rosmalen, & Kalz, in review Impact of Online Peer-Feedback Training and Prior Experience on Student´s Peer STUDY 3 Feedback Perceptions in a Massive Open Online Course
Kasch, J., Van Rosmalen, P., & Kalz, M. (in preparation). Impact of Online Peer- Feedback Training and Prior Experience on Student´s Peer Feedback Perceptions in a Massive Open Online Course. RESEARCH QUESTIONS (STUDY 3)
• RQ1: To what extent does online peer-feedback training in a MOOC influence students’ perception of peer-feedback and peer- feedback training?
• RQ2: What is the relationship between students’ peer-feedback experience and their peer- feedback perceptions? RESEARCH QUESTIONS (METHOD)
Kasch, Van Rosmalen, & Kalz (in preparation). VARIABLES AND ANALYSIS (METHOD)
• Variables measured: Perception variables (Willingness, Usefulness, Preparedness, General attitude), Demographics, Peer Feedback experience, • T-tests to compare means of treatment and control group (N=45) • Regression analysis to check impact of prior experience on the dependent variables RESULTS (1/2)
• No significant differences between treatment and control group for the perception variables • Peer-feedback training did not change the perception of learners • Amount of peer feedback experience has significant impact on perception RESULTS (2/2)
• Learners with no peer-feedback experience had significant higher perception than those who experienced it once • Learners with more feedback experience had significantly higher perception than those without experience Rather than talking about learning at scale we should shift the discussion to educational scalability
CONCLUSIONS Educational scalability is defined by mix of quantitative and (1/3) qualitative dimensions
Educational scalability can be high, but happens often on lower levels of complexity Many MOOCs still lack clarity and ID quality
CONCLUSIONS Lack of student-teacher (2/3) interaction
BUT: Several good practices of scalable feedback practices could be identified in the sample The usefulness of peer-feedback as scalable practice in MOOCs depends on positive learner perception
CONCLUSIONS Perception is more likely to be influenced by repeated positive (3/3) experiences rather than training
Scalable digital feedback practices can also influence scalability challenges in campus education RESEARCH TEAM
Julia Kasch, Open University of the Netherlands
Dr. Peter van Rosmalen, Maastricht University
Prof. Dr. Marco Kalz, Heidelberg University of Education
Thank you!
http://twitter.com/mkalz