Adaptive Learning Meets Crowdsourcing: Towards Development of Cost-Effective Adaptive Educational Systems

Companion Proceedings 10th International Conference on Learning Analytics & Knowledge (LAK20)

Hassan Khosravi The University of Queensland [email protected]

ABSTRACT: A growing body of evidence demonstrates that adaptive educational systems (AESs) can provide an efficient, effective, and customised learning experience for students. Despite their success in enhancing learning, AESs may encounter barriers to adoption as they are generally expensive to develop, challenging to scale across disciplines, and face limitation in their ability to engage students in higher-order thinking. The use of crowdsourcing to support learning at scale and personalisation has recently received significant attention in the Artificial Intelligence in Education (AIED) and Educational Data Mining (EDM) communities. Building on this momentum, this short article considers the viability of using crowdsourcing as a way of addressing the abovementioned common AES challenges. We first discuss the viability and benefits of using crowdsourcing in adaptive educational systems. We then present a system called RiPPLE to demonstrate one approach for implementing a discipline-agnostic, cost-effective crowdsourced adaptive educational system that holds potential for promoting higher-order learning. We share initial results and lessons learned from piloting RiPPLE in 20 courses from 8 different disciplines and conclude by offering general implications and challenges for employing crowdsourcing within AESs.

Keywords: Adaptive educational systems, learner sourcing, crowdsourcing

1 INTRODUCTION

An adaptive educational system (AES) uses data about students, learning processes, and learning products to provide an efficient, effective, and customised learning experience for students. The system achieves this by dynamically adapting instruction, learning content, and activities to suit students' individual abilities or preferences (Aleven, McLaughlin, Glenn & Koedinger, 2016).

A consistent and growing body of knowledge over the past three decades has provided evidence about the effectiveness of AESs relative to traditional educational systems that offer instructions and learning activities that are not adaptive (Anderson, Boyle & Reiser, 1985; VanLehn, 2011; Ma, Adescope, Nesbit & Liu, 2014). Despite their ability to enhance learning, however, AESs have been embraced slowly by higher education, with adoption restricted mostly to research projects (Aleven et al., 2016; Essa, 2016).

To effectively adapt to the learning needs of individual students, an AES requires access to a large repository of learning resources. These resources are commonly created by domain experts. The development time for earlier versions of AESs is estimated at more than 50 hours of an expert's time for each hour of instruction (Aleven, McLaren, Sewall & Koedinger, 2006). Smart tools for authoring an AES, such as Cognitive Tutor Authoring Tools (Aleven et al., 2006; Aleven et al., 2016), have reduced the development time to roughly 25 hours of a domain expert's time per instructional hour.

Creative Commons License, Attribution - NonCommercial-NoDerivs 3.0 Unported (CC BY-NC-ND 3.0) 1

Companion Proceedings 10th International Conference on Learning Analytics & Knowledge (LAK20)

Nevertheless, an AES is still very expensive to develop and challenging to scale across different domains.

How can institutions provide cost-effective AESs across many domains? One potential solution is to adopt a crowdsourcing approach, engaging students in the creation, moderation, and evaluation of learning resources (Heffernan et al., 2016). A crowdsourcing approach can significantly reduce development costs and has the potential to foster higher-order learning for students across many domains. But is this vision theoretically viable? Can students create high-quality resources? Are students able to effectively evaluate the quality of their peer-created resources? How does creating and evaluating resources impact learning? Following is an attempt to answer these critical questions.

2 CREATING AND MODERATING LEARNING RESOURCES IN PARTNERSHIP WITH STUDENTS

There seems to be adequate evidence suggesting that students can create high-quality learning resources that meet rigorous qualitative and statistical criteria (Walsh, Harris, Denny, Smith, 2018; Tacket et al., 2018; Galloway & Burns 2015; Bates, Galloway, Riise & Homer, 2014; Denny, Hamer & Luxton-Reilly, 2009). In fact, resources developed by students may have a lower chance of suffering from an expert blind spot (Nathan, Koedinger & Alibali, 2001). However, it seems likely that some learning resources developed by students may be ineffective, inappropriate, or incorrect (Bates, et al., 2014). Therefore, in order to effectively utilise resources developed by students, a selection and moderation process is needed to ensure the quality of each resource. This can also be done via a crowdsourcing approach. Research suggests that students as experts-in-training can accurately determine the quality of a learning resource and that the use of crowd-consensus algorithms in combination with optimal spot-checking by experts can increase the accuracy of assessment results (Whitehill, Aguerrebere & Hylak, 2019).

Not only can students create and evaluative resources effectively, but these activities also might enhance learning in and of itself. Classical and contemporary models of learning have emphasised the benefits of engaging students in activities across many higher-level objectives of the cognitive domain in Bloom's Taxonomy (Anderson & Krathwohl, 2001). In particular, students' development of creativity and evaluative judgment—"the capability to make decisions about the quality of work of self and others"—has been recognised as essential for student learning (Sadler, 2010). Honing these skills enables students to develop expertise in their field and to extend their understanding beyond their current work to future endeavors, including lifelong learning.

But can the vision of developing a cost-effective, discipline-agnostic AES via crowdsourcing be operationalised? An example of such a system follows.

3 RIPPLE: A CROWDSOURCED ADAPTIVE EDUCATIONAL SYSTEM

RiPPLE1 is an adaptive learning system that recommends personalised learning activities to students, based on their knowledge state, from a pool of crowdsourced learning activities that are generated

1 https://ripplelearning.org/ Creative Commons License, Attribution - NonCommercial-NoDerivs 3.0 Unported (CC BY-NC-ND 3.0) 2

Companion Proceedings 10th International Conference on Learning Analytics & Knowledge (LAK20) and evaluated by educators and the students themselves (Khosravi, Kitto, & Williams, 2019). RiPPLE integrates insights from crowdsourcing, learning sciences and adaptive learning, aiming to narrow the gap between these large bodies of research, and practical implementation into a platform that instructors can easily use in their courses. Figure 1 demonstrates an overview of one of the main pages of RiPPLE.

Figure 1: Overview of one of the main pages of RiPPLE

RiPPLE has the following three interconnected functions.

Student Modelling and Recommendation. The upper part of Figure 1 contains an interactive visualisation widget allowing students to view an abstract representation of their knowledge state based on a set of topics associated with a course offering. The colour of the bars, determined by the underlying algorithm modelling the student, categorises competence into three levels for a particular unit of knowledge red, yellow, and blue which signify, respectively, inadequate competence, adequate competence with room for improvement, and mastery. Currently, RiPPLE employs the Elo rating system for approximating the knowledge state of users (Abdi, Khosravi, Sadiq, Gasevic, 2019). The lower part of Figure 1 displays learning resources recommended to a student based on their learning needs.

Content creation. RiPPLE enables students to create a wide range of learning resources, including MCQs, worked examples, and general notes, incorporating text, tables, images, videos and scientific formulas. Given that students are developing as domain experts, it is likely that some of these learning resources may be ineffective, inappropriate or incorrect (Bates, 2014). Hence, there is a need for a moderation process to identify the quality of each resource. Here again, RiPPLE relies on the wisdom of the crowd and seeks help from students as moderators.

Content moderation: RiPPLE provides two “formal” moderation options that enable instructors to partner with students to review the quality of the student-created exercises before they are added

Creative Commons License, Attribution - NonCommercial-NoDerivs 3.0 Unported (CC BY-NC-ND 3.0) 3

Companion Proceedings 10th International Conference on Learning Analytics & Knowledge (LAK20) to a course's repository of learning resources. In both, (1) instructors determine the minimum number of moderations required per resource (e.g., 3 or 5) and (2) students review resources and provide a simple judgement, alongside a rationale for their decision. The two moderation options differ as to how the outcome of the process is determined. The two possibilities are (1) instructor’s make the final call based on students’ moderations or (2) the system automatically makes the final call based on students’ moderation ratings and their reliability as computed by the system itself.

RiPPLE has the following three interconnected functions.

Content moderation: RiPPLE provides two “formal” moderation options that enable instructors to partner with students to review the quality of the student-created exercises before they are added to a course's repository of learning resources. In both, (1) instructors determine the minimum number of moderations required per resource (e.g., 3 or 5) and (2) students review resources and provide a simple judgement, alongside a rationale for their decision. The two moderation options differ as to how the outcome of the process is determined. The two possibilities are (1) instructor’s make the final call based on students’ moderations or (2) the system automatically makes the final call based on students’ moderation ratings and their reliability as computed by the system itself.

To date, more than 5,000 registered users from 20 courses from 8 different disciplines have used RiPPLE to create over 8,000 learning resources and either attempt or review over 450,000 learning resources. In alignment with the literature, our findings suggest the following:

• Using RiPPLE as an AES that engages students in the creation and evaluation of resources led to measurable learning gains and, importantly, was perceived by students as beneficially supporting their learning (Khosravi, Kitto, & Williams, 2019).

Creative Commons License, Attribution - NonCommercial-NoDerivs 3.0 Unported (CC BY-NC-ND 3.0) 4

Companion Proceedings 10th International Conference on Learning Analytics & Knowledge (LAK20)

• Providing open and transparent learner models as part of an AES can help students better understand their own learning needs and improve self-regulation (Abdi, Khosravi, Sadiq, & Gasevic, 2020).

• Using RiPPLE allows for the provision of personalised recommendations based on students' knowledge gaps and interests (Khosravi, Cooper & Kitto, 2017; Abdi, Khosravi, & Sadiq, 2019).

• Providing guides, exemplars, and rubrics supports students in developing their capacity for creating and evaluating resources—leading to an increase in the quality of the content repository. (Khosravi, Gyamfi, Hanna & Lodge, 2020)

• Utilising mechanisms such as gamification in education motivates students to be actively engaged, which can improve learning (Borges, Durelli, Reis & Isotani, 2014).

• Considering learning theories and pedagogical approaches is important for developing educational technologies; however, other factors such as usability, flexibility, and scalability are also critical (Khosravi, Sadiq & Gasevic, 2020).

4 PRACTICAL AND INTELLECTUAL CHALLENGES

While there exists adequate theoretical evidence regarding the potential of using crowdsourcing for development of AESs, there are many practical and intellectual challenges that still need to be addressed before this vision can be effectively operationalised at scale. A few of these challenges are listed below.

Quality Control. What mechanisms can crowdsourced AESs use to accurately judge the quality of a learning resource that is created by a student? Is the resource correct? Does it effectively help other students learn? Is it too similar to other resources that might have already been included in the resource repository?

Reliability Systems. How can crowdsourced AESs transparently, fairly, and accurately rate the reliability of each of the students?

Optimal Spot Checking. How can crowdsourced AESs optimally utilise the minimal availability of instructors in moderating resources to maximise the accuracy of the moderation process and reliability of student ratings?

Incentives. Despite students’ personal beliefs and strong evidence from the learning science literature about the benefits of engaging in resource creation and moderation, based on our experience, students often require additional incentive mechanisms to engage with these activities. How can crowdsourced AESs incentivise students to engage with content creation and moderation?

Training and Support. Despite the recognition of the value of evaluative judgement and creativity in higher education, little attention has been paid to the development of tools and strategies to support their growth. How can crowdsouced AESs help students actively develop their creativity and evaluative judgment skills while creating learning resources?

Benchmark and Metrics. What benchmarking metrics can be used to measure the effectiveness of a crowdsourced AES?

Creative Commons License, Attribution - NonCommercial-NoDerivs 3.0 Unported (CC BY-NC-ND 3.0) 5

Companion Proceedings 10th International Conference on Learning Analytics & Knowledge (LAK20)

Ethics: Ethical considerations should drive the design and implementation of crowdsourced AES. How can ensure that crowdsourced AESs comply with the ethical guidelines, protocols and principles which have been proposed by the learning analytics community?

REFERENCES

Abdi, S., Khosravi, H., Sadiq, S., & Gasevic, D. (2020). A Multivariate ELO-based Learner Model for Adaptive Educational Systems. In the proceedings of the 12th International Conference on Educational Data Mining 12, 228 – 233 Abdi, S., Khosravi, H., Sadiq, S., & Gasevic, D. (2020). Complementing Educational Recommender Systems with Open Learner Models. To appear in the proceedings of the 10th International Conference in Learning Analytics and Knowledge. Aleven, V., McLaren, B. M., Sewall, J., & Koedinger, K. R. (2006, June). The cognitive tutor authoring tools (CTAT): preliminary evaluation of efficiency gains. In International Conference on Intelligent Tutoring Systems (pp. 61-70). Springer, Berlin, Heidelberg. Aleven, V., McLaughlin, E. A., Glenn, R. A., & Koedinger, K. R. (2016). Instruction based on adaptive learning technologies. Handbook of research on learning and instruction. Routledge. Anderson, J. R., Boyle, C. F., & Reiser, B. J. (1985). Intelligent tutoring systems. Science, 228(4698), 456-462. Anderson, L. W., Krathwohl, D. R. (Eds.). (2001). A taxonomy for learning, teaching and assessing: A revision of Bloom’s taxonomy of educational objectives: Complete edition. New York, NY: Longman. Bates, S. P., Galloway, R. K., Riise, J., & Homer, D. (2014). Assessing the quality of a student-generated question repository. Physical Review Special Topics-Physics Education Research, 10(2), 020105. Denny, P., Hamer, J., & Luxton-Reilly, A. (2009). Students sharing and evaluating MCQs in a large first year engineering course. In 20th Annual Conference for the Australasian Association for Engineering Education (AAEE ’09), 6–9 December 2009, The University of Adelaide, Adelaide, Australia (pp. 575– 580). Barton, Australia: Engineers Australia. De Sousa Borges, S., Durelli, V. H., Reis, H. M., & Isotani, S. (2014, March). A systematic mapping on gamification applied to education. In Proceedings of the 29th annual ACM symposium on applied computing (pp. 216-222). ACM. Essa, A. (2016). A possible future for next generation adaptive learning systems. Smart Learning Environments, 3(1), 16. https://dx.doi.org/10.1186/s40561-016-0038-y Galloway, K. W., & Burns, S. (2015). Doing it for themselves: students creating a high quality peer-learning environment. Chemistry Education Research and Practice, 16(1), 82-92. Khosravi, H., & Cooper, K, Kitto, K. (2017). RiPLE: Recommendation in Peer-Learning Environments Based on Knowledge Gaps and Interests. Journal of Educational Data Mining, 9(1), 42–67. Khosravi, H., Kitto, K., & Williams, J. J. (2019). RiPPLE: A Crowdsourced Adaptive Platform for Recommendation of Learning Activities. Journal of Learning Analytics, 6(3), 83–97. http://dx.doi.org/10.18608/jla.2019.63.12 Khosravi, H., Gyamfi, G., Hanna, B. & Lodge, J. (2010). Fostering and Supporting Empirical Research on EvaluativeJudgement via a Crowdsourced Adaptive Learning System. To appear in the Proceedings of the 10th International Conference on Learning Anaytics and Knowledge. Khosravi, H., Sadiq, S., Gasevic, D., (2020) Development and Adoption of an Adaptive Learning System: Reflections and Lessons Learned. To appear in the proceedings of 51 Special Interest Group on Computer Science Education

Creative Commons License, Attribution - NonCommercial-NoDerivs 3.0 Unported (CC BY-NC-ND 3.0) 6

Companion Proceedings 10th International Conference on Learning Analytics & Knowledge (LAK20)

Ma, W., Adesope, O. O., Nesbit, J. C., & Liu, Q. (2014). Intelligent tutoring systems and learning outcomes: A meta-analysis. Journal of educational psychology, 106(4), 901. Nathan, M. J., Koedinger, K. R., & Alibali, M. W. (2001). Expert blind spot: When content knowledge eclipses pedagogical content knowledge. In Proceedings of the third international conference on cognitive science (Vol. 644648). Neil Heffernan, Korinn Ostrow, Kim Kelly, Douglas Selent, Eric Van Inwegen, Xiaolu Xiong, and Joseph Jay Williams. (2016) The Future of Adaptive Learning: Does the Crowd Hold the Key?, International Journal of Artificial Intelligence in Education 26 (2), 615-644. Sadler, D. R. (2010). Beyond feedback: Developing student capability in complex appraisal. Assessment & Evaluation in Higher Education, 35(5), 535-550. Tackett, S., Raymond, M., Desai, R., Haist, S. A., Morales, A., Gaglani, S., & Clyman, S. G. (2018). Crowdsourcing for assessment items to support adaptive learning. Medical teacher, 40(8), 838-841. VanLehn, K. (2011). The relative effectiveness of human tutoring, intelligent tutoring systems, and other tutoring systems. Educational Psychologist, 46(4), 197-221. Walsh, J. L., Harris, B. H., Denny, P., & Smith, P. (2018). Formative student-authored question bank: Perceptions, question quality and association with summative performance. Postgraduate Medical Journal, 94(1108), 97–103. https://dx.doi.org/10.1136/postgradmedj-2017-135018 Whitehill, J., Aguerrebere, C., & Hylak, B. (2019). Do learners know what’s good for them? Crowdsourcing subjective ratings of OERs to predict learning gains. Educational Data Mining, 2019, 12th.

Creative Commons License, Attribution - NonCommercial-NoDerivs 3.0 Unported (CC BY-NC-ND 3.0) 7