Automated Feedback for Learning Code Refactoring Hieke Keuning Automated Feedback for Learning Code Refactoring Druk: Ridderprint | www.ridderprint.nl Ontwerp omslag: Robbert Menko ISBN: 978-94-6416-127-4 Dit proefschrift komt voort uit een Promotiebeurs voor Leraren (projectnum- mer 023.005.063), genancierd door de Nederlandse Organisatie voor Weten- schappelijk Onderzoek (NWO). Copyright © Hieke Keuning, 2020 Automated Feedback for Learning Code Refactoring PROEFSCHRIFT ter verkrijging van de graad van doctor aan de Open Universiteit op gezag van de rector magnicus prof. dr. Th.J. Bastiaens ten overstaan van een door het College voor promoties ingestelde commissie in het openbaar te verdedigen op vrijdag 9 oktober 2020 te Heerlen om 13:30 uur precies door Hebeltje Wijtske Keuning geboren op 14 augustus 1981 te Hardenberg Promotor Prof. dr. J.T. Jeuring Open Universiteit, Universiteit Utrecht Co-promotor Dr. B.J. Heeren Open Universiteit Leden beoordelingscommissie Prof. dr. J. Börstler Blekinge Institute of Technology Prof. dr. ir. J.M.W. Visser Universiteit Leiden Prof. dr. J. Voigtländer Universität Duisburg-Essen Prof. dr. E. Barendsen Open Universiteit, Radboud Universiteit Dr. A. Fehnker Universiteit Twente Dr. ir. F.F.J. Hermans Universiteit Leiden v Contents 1 Introduction 1 1.1 A short history of programming education research . .2 1.2 The struggles of novice programmers . .4 1.3 Tools supporting the learning of programming . .6 1.4 Teaching programming style and code quality . .8 1.5 Research questions and thesis structure . 10 1.5.1 Other work . 13 2 A Systematic Literature Review of Automated Feedback Generation for Programming Exercises 15 2.1 Introduction . 16 2.2 Related work . 19 2.3 Method . 19 2.3.1 Research questions . 19 2.3.2 Criteria . 20 2.3.3 Search process . 22 2.3.4 Coding . 23 2.4 Labelling . 24 2.4.1 Feedback types (RQ1) . 24 2.4.2 Technique (RQ2) . 27 2.4.3 Adaptability (RQ3) . 29 2.4.4 Quality (RQ4) . 29 2.5 General tool characteristics . 30 2.6 Feedback types (RQ1) . 34 2.6.1 Knowledge about task constraints (KTC) . 34 2.6.2 Knowledge about concepts (KC) . 37 2.6.3 Knowledge about mistakes (KM) . 37 2.6.4 Knowledge about how to proceed (KH) . 42 2.6.5 Knowledge about meta-cognition (KMC) . 44 vi 2.6.6 Trends . 44 2.7 Technique (RQ2) . 46 2.7.1 General ITS techniques . 47 2.7.2 Domain-specic techniques for programming . 49 2.7.3 Other techniques . 52 2.7.4 Combining techniques . 55 2.7.5 Trends . 55 2.8 Adaptability (RQ3) . 56 2.8.1 Solution templates (ST)................... 57 2.8.2 Model solutions (MS).................... 58 2.8.3 Test data (TD)........................ 58 2.8.4 Error data (ED)........................ 59 2.8.5 Other . 59 2.9 Quality (RQ4) . 60 2.9.1 Analytical (ANL)....................... 61 2.9.2 Empirical assessment . 61 2.9.3 Trends . 64 2.10 Discussion . 64 2.10.1 Feedback types . 64 2.10.2 Feedback generation techniques . 66 2.10.3 Tool adjustability . 67 2.10.4 Tool evaluation . 67 2.10.5 Classifying feedback . 68 2.10.6 Threats to validity . 69 2.11 Conclusion . 69 3 Code Quality Issues in Student Programs 71 3.1 Introduction . 72 3.2 Related work . 73 3.3 Method . 75 3.3.1 Blackbox database . 75 3.3.2 Data analysis . 75 3.4 Results . 80 3.4.1 All issues (RQ1) . 80 3.4.2 Selected issues (RQ1) . 81 3.4.3 Fixing (RQ2) . 84 3.4.4 Extensions (RQ3) . 84 3.5 Discussion . 86 vii 3.5.1 Threats to validity . 87 3.6 Conclusion and future work . 88 4 How Teachers Would Help Students to Improve Their Code 89 4.1 Introduction . 90 4.2 Background and related work . 90 4.2.1 Code quality terms and denitions . 91 4.2.2 Code quality in education . 91 4.3 Method . 92 4.3.1 Study design . 92 4.3.2 Data analysis . 93 4.4 Results . 94 4.4.1 Background of teachers . 94 4.4.2 Role of code quality (RQ1) . 95 4.4.3 Program hints and steps (RQ2 and RQ3) . 96 4.5 Discussion . 103 4.5.1 Threats to validity . 105 4.6 Conclusion and future work . 105 5 A Tutoring System to Learn Code Refactoring 107 5.1 Introduction . 108 5.2 Background and related work . 109 5.2.1 Code quality and refactoring . 109 5.2.2 Professional tools . 110 5.2.3 Tutoring systems . 110 5.2.4 Teachers’ perspective and conclusion . 111 5.3 Method . 112 5.4 A tutoring session . 113 5.4.1 Example 1: Sum of values . 114 5.4.2 Example 2: Odd sum . 116 5.5 Design . 118 5.5.1 Implementation . 118 5.5.2 Rules and strategies . 119 5.5.3 Feedback services . 120 5.6 Evaluation and discussion . 121 5.6.1 Evaluation . 121 5.6.2 Discussion . 123 5.6.3 Threats to validity . 123 viii 5.7 Conclusion and future work . 124 6 Student Refactoring Behaviour in a Programming Tutor 125 6.1 Introduction . 126 6.2 Background and related work . 127 6.2.1 Code quality in education . 127 6.2.2 Tutoring systems for programming . 128 6.2.3 Automated feedback on code quality . 129 6.3 The Refactor Tutor . 130 6.4 Method . 134 6.4.1 Study design . 134 6.4.2 Analysis . 136 6.5 Results . 136 6.5.1 Solving exercises (RQ1) . 137 6.5.2 Hint seeking (RQ1 and RQ2) . 142 6.5.3 Student evaluation (RQ3) . 154 6.6 Discussion . 157 6.6.1 Student refactoring behaviour . 157 6.6.2 Teaching code refactoring . 159 6.6.3 Quality of the rule set . 159 6.6.4 Threats to validity . 160 6.7 Conclusion and future work . 161 7 Epilogue 163 7.1 Conclusions . 163 7.2 Recent trends . 165 7.3 Future work and nal thoughts . 166 Samenvatting 169 Dankwoord 175 CV 177 Appendix A Code Refactoring Questionnaire 179 Appendix B Stepwise improvement sequences with hints 187 Bibliography 197 1 Chapter 1 Introduction Learning to program is hard, or is it? Many papers in the eld of novice programming begin with stating that it is indeed ‘hard’, ‘challenging’, and a ‘struggle’. Guzdial contemplates that ‘maybe the task of programming is innately one of the most complex cognitive tasks that humans have ever cre- ated’.1 While there is plenty of evidence that students indeed struggle, there is also nuance: Luxton-Reilly believes our expectations and demands of novices are too high [172], reasoning that if children can learn how to program, it should not be that hard to learn at least the basics. He argues that demanding too much from beginners in too little time, which inevitably leads to unsatis- factory results, does not imply that programming itself is hard. To write a good program, novices need knowledge of programming lan- guages and tools, as well as the skills to adequately use these resources to solve actual problems [228]. Du Boulay identies ve main problem areas [71]: ori- entation (the goal of programming), the notional machine (an abstraction of how the computer executes a program), notation (language syntax and se- mantics), structures (such as plans or schemas to perform small tasks), and pragmatics (planning, developing, testing, debugging, etc.). Programming re- quires handling all of these aspects almost simultaneously, making things even harder. Consequently, it is not surprising that this high cognitive load being placed on novices, combined with often awed mental models, leads to strug- gling students. Research into programming education has been focussing mainly on stu- dent diculties, and the mistakes that they make aecting functional correct- ness. Style,.