Labback: an Extendible Platform for Secure and Robust In-The-Cloud Automatic Assessment of Student Programs
Total Page:16
File Type:pdf, Size:1020Kb
LabBack: an extendible platform for secure and robust in-the-cloud automatic assessment of student programs Master’s Thesis Vlad A. Vergu LabBack: an extendible platform for secure and robust in-the-cloud automatic assessment of student programs THESIS submitted in partial fulfillment of the requirements for the degree of MASTER OF SCIENCE in COMPUTER SCIENCE by Vlad A. Vergu born in Bucharest, Romania Software Engineering Research Group Department of Software Technology Faculty EEMCS, Delft University of Technology Delft, the Netherlands www.ewi.tudelft.nl c 2012 Vlad A. Vergu. Cover picture: The last straw, drawing by Maria I. Ionescu, 2012 LabBack: an extendible platform for secure and robust in-the-cloud automatic assessment of student programs Author: Vlad A. Vergu Student id: 1195549 Email: [email protected] Abstract In software engineering education manual assessment of students’ programs is a time consuming endeavour that does not scale to high numbers of pupils or to large amounts of practice material. Numerous automatic grading programs, aim- ing to alleviate this problem, have been developed since 1960. These, however, only support fixed programming languages and fixed assessment methods, thus prohibiting their reuse throughout different programming curricula. Educators investigating new grading methods either have to accept the engineering burden of creating a complete grading system or revert to manual grading. This thesis presents LabBack - a reusable automatic grading platform that is extendible with language- and assessment-specific functionality by means of plugins. LabBack provides the necessary infrastructure for building and hosting automatic grading functionality, eliminating the need to consider the issues of scalability and security against malicious programs. LabBack can be hosted in the cloud and provides immediate student feedback. Plugins providing automatic assessment for Scala, JavaScript and C have been implemented and LabBack has been validated in a university-level course with over 100 students. Thesis Committee: Chair: Dr. E. Visser, Faculty EEMCS, TU Delft University supervisor: Dr. E. Visser, Faculty EEMCS, TU Delft Committee member: Dr. A.E. Zaidman, Faculty EEMCS, TU Delft Committee member: Dr. ir. J.F.M. Tonino, Faculty EEMCS, TU Delft Preface About this thesis This thesis essentially consists of two parts. The first part (Chap- ter 1 to Chapter 5) presents the architecture of LabBack’s proof of concept - Lab- Back 2:0 - and its validation in a case study, serving as motivation for the second part. The second part (Chapter 6 to Chapter 8) describes the architecture, implementation and evaluation of LabBack 2:0 - an extendible and reusable platform for automatic grading that builds on LabBack 2:0’s main security concepts. Acknowledgements I am grateful to my supervisor Eelco Visser for the inspira- tional discussions and feedback sessions and for his development of WebLab, without whom this project would not have been possible. I would like to thank Karl Trygve Kalleberg for our interesting discussions during his visit to The Netherlands and Danny Groenewegen for his support with using WebDSL. Last, but certainly not least, I am grateful to my family for supporting me during my time at the university. Vlad A. Vergu Delft, The Netherlands October 30, 2012 iii Contents Preface iii Contents v List of Figures vii 1 Introduction 1 1.1 Research Questions . 3 1.2 Contributions . 4 1.3 Outline of this Thesis . 5 2 Background 7 2.1 Massive Open Online Course Platforms . 7 2.2 Case study: COPL . 9 2.3 WebLab . 10 2.4 Summary . 19 3 Analysis of LabBack Requirements 21 3.1 Functional Requirements . 21 3.2 Non-functional Requirements . 25 3.3 Summary . 32 4 Satisfying the Non-Functional Requirements of LabBack 33 4.1 Native Tooling . 33 4.2 Virtualised Solutions . 36 4.3 Build Tools . 38 4.4 Continuous Integration Systems . 39 4.5 Custom Solution . 40 4.6 Conclusion and Recommendation . 41 5 Proof of Concept: LabBack 1:0 43 5.1 Architecture . 43 5.2 Implementation Details . 56 5.3 Evaluation . 64 v CONTENTS 5.4 Conclusion . 67 6 LabBack 2:0 Architecture 69 6.1 Java Plugin Frameworks . 69 6.2 LabBack 2:0 Context . 71 6.3 System Decomposition . 72 6.4 Information Architecture . 77 6.5 Concurrency . 82 6.6 Business processes . 85 6.7 Services . 94 6.8 Security . 103 6.9 Deployment Requirements . 106 6.10 Conclusion . 107 7 LabBack 2:0 Implementation and Integration Details 109 7.1 WebDSL and Java code . 109 7.2 Plugin System . 111 7.3 WebLab as Client: Contributions to WebLab . 117 7.4 Summary . 119 8 Evaluation 121 8.1 Goals and Structure . 121 8.2 Evaluation of Correctness, Robustness and Security . 122 8.3 Evaluation of Scalability and Load Distribution . 127 9 Related work 141 9.1 Automatic Grading . 141 9.2 Web IDEs and Containment of Third-party Programs . 142 9.3 Load Balancing and Distribution . 144 10 Conclusion 145 10.1 Summary . 145 10.2 Conclusions . 147 10.3 Future Work . 150 Bibliography 153 A LabBack 1:0 Scheduler Implementation 159 B LabBack 1:0 Classloader Implementation 163 C CounterLogic Execution Implementation 165 D Plugin Loader Implementation of LabBack 2:0 167 vi List of Figures 2.1 Course edition dashboard in WebLab as seen by an instructor . 11 2.2 WebLab course information page for the COPL 2012 course. 12 2.3 WebLab course information page showing correlation plots. 13 2.4 WebLab page for editing a programming question. 13 2.5 Specification test template for Javascript question . 14 2.6 WebLab page for answering a programming question. 15 2.7 Editing of question’s grading options in WebLab . 16 2.8 Page for submission grading in WebLab. 17 2.9 WebLab architectural components. 18 2.10 Example unit tests using the Scalatest framework. 20 3.1 Summarised classification of LabBack requirements. 22 3.2 Classification of potential LabBack security attack vectors and their in- tended purposes. 26 3.3 Example of JVM memory starvation implemented in Java. 29 3.4 Example fork-bomb in C. 29 3.5 Example fork-bomb in Java. 29 4.1 Classification of ecosystem alternatives . 41 5.1 LabBack 1:0 context diagram. Arrows indicate usage. 44 5.2 LabBack 1:0 system decomposition. 45 5.3 Job data model. 48 5.4 Example XML representation of completed job. 50 5.5 LabBack 1:0 processes and threads. 51 5.6 Job state diagram. 51 5.7 BPMN blueprint of LabBack’s main process. 53 5.8 LabBack 1:0 security policy. 54 5.9 LabBack 1:0’s scheduler maintenance loop algorithm. 57 5.10 Custom class loader algorithm. 61 5.11 Usage example of LabBack’s custom class loader. 62 5.12 Example stoppable thread in Java. 64 vii LIST OF FIGURES 6.1 Comparison of Java plugin frameworks. Highlighted cells indicate critical limitations. 70 6.2 LabBack 2:0 multi-instance deployment tree. 71 6.3 LabBack 2:0 system decomposition. 72 6.4 Plugin manifest for the JavaScript plugin. 73 6.5 Class diagram of LabBack 2:0’s core data model. 78 6.6 Data model of LabBack 2:0’s job. 78 6.7 Data model of LabBack 2:0’s main plugin entity. 81 6.8 Process and thread model of LabBack 2:0 . 82 6.9 State model of a single job or a chains of jobs in LabBack 2:0. 84 6.10 Transition function of job chain state machine. 85 6.11 Bluepring of the Plugin Deployment (PD) process. 86 6.12 Blueprint of the Execution Loading (EL) process. 87 6.13 Partial blueprint of the Local Job Scheduling (LJS) process. 89 6.14 Schematic blueprint of Remote Job Scheduling (RJS) process. 91 6.15 Summary of job-related services offered by LabBack 2:0. 95 6.16 Meaning of flag combinations in response of queryJobStatus service. 98 6.17 Summary of plugin-related services offered by LabBack 2:0. 100 6.18 Comparison of LabBack 1:0 and LabBack 2:0 functional groups. 104 6.19 Software and hardware requirements of LabBack 2:0. 106 7.1 Definition of a WebDSL page with inter-thread message passing. 110 7.2 Processing steps of plug.sc.counter plugin. 111 7.3 Manifest of plug.sc.counter plugin. 112 7.4 Compulsory interface of all plugin entry points in LabBack 2:0. 112 7.5 Execution logic in CounterLogic. 114 7.6 Pseudocode specification of plugin loading mechanism. 115 7.7 Illustration of plugin detachment upon upgrade . 116 7.8 Summary UI contributions to WebLab. 117 7.9 Monitoring and configuration UI for LabBack 2:0 instances in WebLab. 118 7.10 WebLab’s Language configuration UI. 119 8.1 Distribution of job states and languages in evaluation dataset . 123 8.2 Differences between original and new grades in the failure cases. 124 8.3 Clustering of grade validation failures into questions. 125 8.4 Summary of variables for simulation control. 129 8.5 User interface of LabBack’s deployment simulator. 130 8.6 Simulation results for control experiment. 132 8.7 Simulation results for 10 node star topology. 133 8.8 Simulation results for 10 node tree of depth 6. 134 8.9 Simulation results for 10 node chain topology. 135 8.10 Simulation results for 30 node tree of degree 3. 136 8.11 Simulation results for 30 node tree of degree 5. 137 8.12 Simulation results for 100 node tree of degree 8. 138 8.13 Simulation results for 100 node tree of degree 8 and varying job durations. 139 viii Chapter 1 Introduction Covering all areas from smartphones, portable music players, cooking robots to In- ternet core-infrastructure, certificate registrars and nuclear power plants software con- tinues to increase its infiltration in all aspects of our modern societies.