Programmeren? Educational Data Mining: Hoe Leren Studenten
Total Page：16
File Type：pdf, Size：1020Kb
Educational data mining: hoe leren studenten programmeren? Louise Deconinck Studentennummer: 01406820 Promotoren: prof. dr. Peter Dawyndt, Bart Mesuere Masterproef ingediend tot het behalen van de academische graad van Master of Science in de informatica Academiejaar 2018-2019 ii Educational data mining: hoe leren studenten pro- grammeren? In deze thesis bekijken we hoe we de data uit het online leerplatform Dodona kunnen gebruiken om lesgevers en studenten meer inzicht te geven in het leerpro- ces. We willen ten eerste bekijken of het gedrag van studenten binnen Dodona een voorspellende waarde heeft: kan het gedrag in Dodona iets zeggen over de voortgang van een student in de cursus? Ten tweede willen we ook oefeningen aanraden aan de student. Om dit te doen moeten we in kaart brengen hoe goed de student bepaalde programmeerconcepten reeds onder de knie heeft. Dodona Dodona is een online leerplatform ontwikkeld aan de UGent waar studenten al oefenend leren programmeren. Het platform wordt voor 2 vakken reeds gebruikt sinds 26 september 2016, waardoor we voor die vakken reeds 3 jaar historische data hebben. Educational Data Mining en Learning Analytics Educational Data Mining en Learning Analytics zijn twee onderzoeksgebieden die door gebruik te maken van machine learning technieken proberen om meer inzicht te krijgen in educationele processen. Dit is zeer breed: aan de ene kant kan men studenten meer inzicht geven in hun leerproces: hen een inschatting gven of ze goed bezig zijn in een cursus of niet. Aan de andere kant is het ook mogelijk om lesgevers inzicht te geven in hoe studenten werken of welke stukken lesmateriaal voor een beter begrip bij de studenten zorgen. Voorspelling voortgang van studenten in de cursus Op basis van de gedragsdata van studenten in Dodona en de punten van hun tussentijdse evaluaties proberen we de voortgang van de student in de cursus te voorspellen. De gedragsdata houdt het volgende in: het tijdstip van de ingediende oplossingen, de reeks en de oefening waartoe deze ingediende oplossingen behoren en de status van de ingediende oplossingen. iii De vakken die we hier beschouwen zijn het vak Programmeren aan de faculteit Wetenschappen aan de UGent en het vak Informatica, aan de faculteit Ingenieurs- wetenschappen en Architectuur, ook aan de UGent. Op basis van die data is het mogelijk om reeds vroeg in het semester (voor de eerste tussentijdse evaluatie) met meer dan 70% accuraatheid te bepalen of een student zal slagen of falen voor dit vak. Conceptextractie uit oefeningen We splitsen eerst de overkoepelende programmervaardigheid op in kleinere concep- ten. Dit doen we op basis van een reeds bestaande lijst van concepten die gebruikt wordt om de oefeningen voor het vak Programmeren op te stellen. Om de vaardigheid van een student met een specifiek programmeerconcept te bepalen, moeten we een manier vinden om deze concepten uit de broncode van de ingediende oplossingen te halen. We defini¨eren voor ieder van de 62 concepten een methode die op basis van Abstract Syntax Trees (AST) de juiste concepten herkent. Door het gebruik van AST's wordt enkel de structuur van het programma behouden en niet de syntax. Deze conceptextractie is onder andere nuttig om te achterhalen of bepaalde con- cepten waarvan gewenst is dat de studenten deze onder de knie hebben ook effectief aan bod komen in de oefeningen. Educational data mining: how do students learn to code? Louise Deconinck Supervisor(s): Prof. Dr. P. Dawyndt, Dr. B. Mesuere, D. Mourisse Abstract— Dodona is an online learning platform tus of that submission. We specifically do not where students can learn to program by solving exer- take any personal data (such as gender, previous cises on which they receive automated feedback. In this paper we predict the performance of students in an intro- education or location) into account for privacy ductory programming course using historical behavioural reasons. This also means that we do not need to data from this platform. This shows that the behaviour question the students during the course and we of students is a good predictor for their performance in can use the historical data already collected as an introductory programming course. such. Keywords— educational data mining, learning analyt- ics, performance prediction, Dodona II. Related work I. Introduction Educational Data Mining and Learning Analyt- The Dodona project (dodona.ugent.be) was ics both want to solve educational problems using first launched on the 26th of September 2016. It data collected by online learning platforms. These is an online learning platform with which students research areas are very broad and contain a lot of can learn to program. A course within Dodona different aspects: this ranges from better support consists of several series of exercises. Students for students to plagiarism detection [2]. can submit solutions to these exercises. The plat- We look specifically at performance prediction: form automatically provides feedback to the stu- how to predict the probability that a student will dents regarding the correctness of their submis- succeed or fail at a course. In Liao et al. [1], sions, which the students can use to improve their clicker data from students was used to measure submissions. their progress in the course. This data consisted Two courses have used Dodona since its of answers from students during in-class ques- launch: a course on programming from the fac- tions. In week 3 (the course lasted 12 weeks) an ulty of Sciences and one from the faculty of Engi- accuracy of 70% was reached to determine which neering and Architecture. Both of these courses students were going to fail or succeed. are introductory computer sciences courses that It is also possible to use data from an instru- teach Python 3 to the students. The course in mented IDE to determine the eventual perfor- the Sciences faculty comprises of 160 to 400 stu- mance of a student with 75% accuracy [4]. Each dents per year. The combined number of submis- time a student compiled their code, an event con- sions lies between 130 000 and 200 000 per year. taining the compilation status and the possible The course of the Engineering faculty comprises error was logged. This data was then scored and of over 400 students per year, with about 480 000 used to perform Linear Regression. However, this - 500 000 submissions per year. Until now, the research was only performed on one cohort of stu- data collected through Dodona has never been dents. systematically analysed. We want to predict the Other research which specifically only uses data performance of students through the course and that is already generated by the students dur- see whether a student is on track to successfully ing their programming exercises was used in Vi- finish the course or not. havainen [3], where a an accuracy of 78% was We choose to only use behavioural data: this reached, using data from only one cohort of stu- includes the time of a submission and the sta- dents. III. Performance Prediction Abbreviation Explanation subm Total nr of submissions We want to use the behavioural data of stu- nosubm Total nr of exercises without dents, supplemented with the results of evalua- submissions tions held during the semester, to predict if the correct Total nr of correct submissions students will successfully finish the course. We after Total nr of submissions after also want to be able to detect failing students as correct early as possible. the first correct submission per exercise, summated over all exercises of a series A. Extracted Features before Total nr of submissions after correct We want to detect failing students early on in the before correct submission the semester. Because we work with 3 years of per exercise, summated over data, we need to consider certain points in the all exercises of a series semester where we want to predict the perfor- time series Difference in time between the mance of a student. We simply choose these last and first submission in a points as the deadline for each series in the series course. We can do this because the courses have time correct Difference in time between the not really changed in those 3 years: the same first correct submission and concepts are taught in the same series. If this is the first submission in a series. not the case, the prediction might suffer. wrong Nr of submissions with status For each year of data, we extract subdatasets. wrong These subdatasets contain only the submissions comp error Nr of submissions with status up to one of the deadlines. In the case of the compilation error course at the faculty of Sciences, there’s 10 dead- runtime Nr of submissions with status lines. This results in 10 subdatasets. error runtime error The raw data consists of per student a number correct Nr of exercises where the series containing exercises. These exercises then after 2h contain a number of submissions. We want to use first correct submission hap- these submissions to extract useful features for pens less than 2 hours after the machine learning model. Per series present in the first submission correct the subdataset we extract the features outlined Nr of exercises where the after 24h in Table 1. first correct submission hap- pens less than 24 hours after B. Machine Learning Models the first submission correct Nr of exercises where the We want to interpret the results of the machine after 5m learning models. This means that we want to first correct submission hap- know which features contributed to the decision pens less than 5 minutes after to classify the student as failing or succeeding. the first submission correct We consider four possible machine learning Nr of exercises where the after 15m models: Stochastic Gradient Descent, Logistic first correct submission hap- Regression, a Support Vector Machine and a Ran- pens less than 15 minutes after dom Forest.