EXAMENSARBETE INOM TEKNIK, GRUNDNIVÅ, 15 HP STOCKHOLM, SVERIGE 2021 HOW CAN COMPUTER-BASED PROGRAMMING EXAMS BE IMPLEMENTED FOR ENGINEERING STUDENTS? RICHARD FARSHCHI ALVAREZ FREDRIK GÖLMAN KTH SKOLAN FÖR ELEKTROTEKNIK OCH DATAVETENSKAP Abstract Programmering inkluderar vanligtvis användningen av en dator och en textredigerare, men ändå avslutas programmeringskurser vid högskolor och universitet ofta med traditionella skriftliga tentor för att bedöma studenternas förvärvade praktiska kunskaper. Denna traditionella examineringsmetod begränsar komplexiteten i programmeringsproblem och kan leda till oavsiktliga subjektiva bedömningar. Tidigare studier om datorbaserade programmeringstentor som efterliknar verkliga programmeringsförhållanden tyder på att bedömningen av studenternas förvärvade kunskaper kan effektiviseras samtidigt som både betygs- och administrationsprocesser förenklas. Vi använder Bunges generella vetenskapliga metod och fallstudiemetodik för att utveckla ett system för datorbaserade programmeringstentor som kan implementeras på KTH Kista. Våra resultat tyder på att programmeringstentor kan utföras säkert på studenternas egna datorer genom att starta datorn i en begränsad förkonfigurerad systemmiljö med blockerad internetåtkomst från ett USB- minne. Parat med den öppna lärplattformen Moodle kan tentorna administreras elektroniskt och utformas med automatiska rättningssprocesser. För att hjälpa tentavakter utvecklade vi även ett observationsverktyg som kan upptäcka om den begränsade systemmiljön kringgås. Trots uteblivna verkliga tester av systemet på grund av omständigheterna orsakade av den pågående pandemin, drar vi, med stöd av tidigare studier, slutsatsen att vår föreslagna lösning för datorbaserade programmeringstentor kan förbättra kvaliteten och effektiviteten av programmeringskurser och deras examinationsprocesser. Nyckelord Tentamen; Programmering; Lärplattform; Moodle; Datorbaserad; Digital; Fuskförebyggande KTH ROYAL INSTITUTE OF TECHNOLOGY ELEKTROTEKNIK OCH DATAVETENSKAP i ii Abstract Programming usually includes the use of a computer and a text editor, yet programming courses given at institutions of higher education often conclude with traditional written exams to assess the students’ acquired practical knowledge. This traditional examination method restricts the complexity of programming problems and may result in unintentional subjective assessments. Previous research on computer-based programming exams that mimic real programming conditions suggest that the assessment of the students’ acquired knowledge can be made more efficient while also simplifying both grading and administration processes. We use Bunge’s general scientific method and case study methodology to develop a system for computer-based programming exams that can be implemented at KTH Kista. Our results suggest that programming exams can be safely performed on the students’ own computers by booting the computer into a restricted preconfigured system environment with blocked Internet access from a usb flash drive. Paired with the open-source learning management system Moodle, the exams can be administered electronically and designed with automatic grading processes. To help exam invigilators we also developed an observation tool that can detect if the restricted system environment is circumvented. Despite the lack of real tests of the system due to the circumstances caused by the ongoing pandemic, we conclude, with the support of previous studies, that our proposed solution for computer-based programming exams can improve the quality and efficiency of programming courses and their examination processes. Keywords Examination; Programming; Learning management system; Moodle; Computer-based; Digital; Cheat-prevention iii iv Acknowledgements We would like to express our deepest gratitute and appreciation to our supervisor Anders Sjögren, the Program Director for the Bachelor’s Degree Program in Computer Engineering at KTH. Thank you for all the insightful guidance and feedback that made this thesis possible and thank you for all the valuable knowledge you have imparted during this entire education. A special thanks also goes out to our thesis examiner Johan Montelius, Associate Professor in Communication Systems at KTH. Thank you for all the feedback and suggestions during this thesis and for your courses that got us here. v vi Table of Contents 1 Introduction ................................................................................................................. 1 1.1 Background ..................................................................................................................... 1 1.2 Problem............................................................................................................................. 2 1.3 Purpose ............................................................................................................................. 2 1.4 Goal ..................................................................................................................................... 2 1.5 Societal benefits, Ethics and Sustainability .......................................................... 2 1.6 Methodology / Methods .............................................................................................. 3 1.7 Stakeholders ................................................................................................................... 3 1.8 Delimitations .................................................................................................................. 3 1.9 Disposition ....................................................................................................................... 4 2 Theoretical background and literature review .............................................. 5 2.1 Related studies ............................................................................................................... 5 2.2 Existing solutions .......................................................................................................... 7 3 Methodology and methods .................................................................................. 15 3.1 Research methodology and methods ................................................................... 15 3.2 Project methodology and methods ....................................................................... 18 4 Results ......................................................................................................................... 21 4.1 Case study design ........................................................................................................ 21 4.2 Solution requirements .............................................................................................. 22 4.3 Existing solutions as viable alternatives ............................................................. 23 4.4 Proposed solution ....................................................................................................... 24 4.5 Case study questionnaires ....................................................................................... 29 5 Discussion .................................................................................................................. 31 5.1 Existing solutions ........................................................................................................ 31 5.2 Proposed solution ....................................................................................................... 33 5.3 Project assignments in lieu of written exams ................................................... 37 5.4 Validity and reliability ............................................................................................... 39 6 Conclusion ................................................................................................................. 41 6.1 Limitations ..................................................................................................................... 41 6.2 Future work ................................................................................................................... 42 References .......................................................................................................................... 43 Appendix A ......................................................................................................................... 47 vii viii 1 Introduction This thesis investigates the benefits of computer-based exams for programming courses and evaluates the systems for computer-based exams currently implemented at various institutions of higher education to find a suitable solution that could be implemented for programming courses at KTH Kista. 1.1 Background Written exams using pen and paper is a common examination method for courses given at institutions of higher education [1]. They are simple to perform and the rules that students must adhere to are easy to enforce, however, they have their shortcomings, especially in the examination of practical programming skills. This thesis has its roots in the limitations that a written exam using pen and paper often imposes on programming problems. One limitation is that they must be simplified for multiple reasons. While students would normally be expected to be familiar with the basic syntax of the programming language in question, the unavailability of application programming interface (API) reference materials, an integrated development environment (IDE) or at the very least a