MAURICE YAP MAURICE DR ANANDHA GOPALAN AUTHOR // AUTHOR COMPUTING INDIVIDUAL PROJECT INDIVIDUAL COMPUTING NG SUPERVISOR // ME A portable web-based platform for testing distributed algorithms Submitted in partial fulfilment of the requirements for the degree of Master of Engineering in Computing of Imperial College London Abstract The study of fault-tolerating distributed algorithms — designed to run on networked computers (nodes) which communicate with each other only by sending messages — is common on university computer science courses. Students of distributed algorithms oen implement such distributed algorithms in order to simulate them in a network of nodes in order to better understand them and to analyse their behaviour and performance. Many tools exist which allow students to do this, and there are also more involved other ways to test distributed algorithms such as implementing and running nodes from scratch in some programming language or setting up networks of Docker containers. However, such methods can be less than ideal since they may require users to use one particular language to implement algorithms, require root access to computers which may not be possible in a classroom or they may require users to spend lots of time learn to use other technologies which takes time away from learning about the algorithms themselves. We present the design of Diorama, a browser-based application which facilitates: the writing of distributed algorithms in the user’s choice of a range of available languages; the defining of a network topology which can include automatically-generated nodes; and the simulation of this distributed network. We create a proof-of-concept of Diorama to demonstrate its value and technical feasibility and publish a deployment tool which can be used to install and run Diorama from a cloud- or hypervisor-based virtual machine. iii Acknowledgements I would like to thank: • my supervisor for this project, Dr Anandha Gopalan, for his excellent advice, guid- ance and support over the past academic year; • my parents for enabling me to come to Imperial College London to study and to develop, and for their support during this time; • those working both on the front line and behind the scenes to continually improve the teaching, facilities, academic support and student welfare provision of the Department of Computing; • and finally, God, who is exclusively responsible for my academic gis and abilities. I also want to express gratitude to the vital financial support I received during my four years of study from the university’s Imperial Bursary scheme. This funding has been vital for me and many of my peers to ensure that we minimise our financial burden on our already-squeezed families. It will continue to be essential in making sure future students from more economically-deprived backgrounds are not discouraged, for financial reasons, from coming to this institution to both study and contribute to student life. The Imperial Bursary scheme is something of which this university should be immensely proud. v Contents 1 Introduction1 1.1 Objectives . 2 1.2 Contributions . 3 2 Distributed Algorithms5 2.1 Network topology . 5 3 Related Work9 3.1 A Simulator for Self-Stabilizing Distributed Algorithms ........... 9 3.2 JBotSim .................................... 13 3.3 DSLabs .................................... 16 3.4 Container orchestration: Docker ....................... 18 3.5 Network simulators: TETCOS NetSim .................... 20 3.6 Online integrated development environments (IDEs) . 22 3.6.1 Codeanywhere ............................ 23 3.6.2 Repl.it ................................. 23 3.7 Serverless/FaaS: AWS Lambda ........................ 24 3.8 Summary of related work . 27 4 Soware Design 29 4.1 Overview . 29 4.2 Goals of the soware package . 29 4.2.1 Use by an educator: Leroy’s user journey . 30 4.2.2 Use by a learner: Stacey’s user journey . 32 4.3 Technical representation of concepts in distributed algorithms . 34 4.4 Programming interface . 35 4.4.1 Node program API . 35 4.4.2 Network topology schema . 37 4.4.3 User events interface . 40 4.5 Web user interface . 43 4.5.1 Main elements . 43 4.5.2 Programs explorer . 44 4.5.3 Program editor . 44 4.5.4 Network topology editor . 46 vii 4.5.5 Advanced configuration . 47 4.5.6 Simulation . 47 4.6 Concluding remarks . 49 5 Proof-of-concept: Diorama 51 5.1 Aims for the proof-of-concept . 51 5.2 Overview of product . 51 5.2.1 Gallery . 52 5.3 Web service architecture . 59 5.4 Back-end: implementing programs, nodes and networks with Docker . 61 5.4.1 Node program images and containers . 61 5.4.2 Networking nodes . 65 5.5 Back-end: Main WebSocket server . 65 5.5.1 Using WebSocket messages . 66 5.5.2 Storing data persistently . 67 5.5.3 Interacting with Docker Engine ................... 68 5.5.4 Fetching node program code . 69 5.5.5 Handling Docker container messages: node logger server . 69 5.6 Front-end:React web application . 71 5.6.1 State management . 72 5.6.2 Page routing . 73 5.6.3 User event scheduling . 74 5.6.4 Use of selected third-party libraries . 76 5.7 Installing and running Diorama ....................... 81 6 Evaluation 83 6.1 Supported functionality . 83 6.2 Usability . 85 6.2.1 User testing . 86 6.2.2 Comparison to directly using Docker ................ 93 6.2.3 Conclusion . 94 6.3 Portability . 95 6.3.1 The web interface . 95 6.3.2 The server . 95 6.3.3 The installation tool . 96 6.3.4 Conclusion . 96 6.4 Extensibility . 97 6.4.1 Node program runtimes . 97 6.4.2 Network topology definition mechanisms . 98 6.4.3 Front end localisation . 99 7 Conclusion 101 7.1 Future work . 101 viii A Example network topology code in JSON 105 B Installation guide 107 B.1 Pre-made images . 107 B.2 For local VMs . 107 B.2.1 Pre-requisites . 107 B.2.2 Deploying . 108 C Guide for test users 109 D User testing node programs 113 E User testing notes 115 E.1 “Matt” . 115 E.2 “Nick" . 116 E.3 “Alan" . 117 E.4 “Gordon" . 117 E.5 “Amy" . 117 Bibliography 119 ix List of Figures 2.1 A pseudo-code representation of Eager Reliable Broadcast. 5 2.2 A network of two connected nodes. 6 2.3 A ring network . 6 2.4 A line network . 6 2.5 A fully-connected network . 6 2.6 A star network . 6 2.7 A tree network . 7 3.1 Screenshot from A Simulator for Self-Stabilizing Distributed Algorithms . 10 3.2 Add connection dialog in A Simulator for Self-Stabilizing Distributed Algorithms 11 3.3 Screenshot of JBotSim in action [16]. 13 3.4 Example code to run JBotSim.......................... 14 3.5 Creating links in JBotSim. ........................... 15 3.6 Screenshot of the function designer for AWS Lambda [6]. 25 3.7 Screenshot of the Amazon Cloudwatch logs for a Lambda function [7]. 25 3.8 Screenshot of AWS Lambda’s function code editor [6]. 26 4.1 State diagram for a simulated node. 34 4.2 The pseudo-code for our example node program. 36 4.3 The Python code for our example node program, using our interface. 37 4.4 An example network topology. 41 4.5 The YAML example network topology. 42 4.6 The timeline of node states generated by our example user events. 43 4.7 Wireframe of the main navigational and other fixed elements of the web application. 43 4.8 Wireframe of the programs viewer. 44 4.9 Wireframe of the program editor. 45 4.10 Wireframe of the network topology editor. 46 4.11 Wireframes of the graphical topology viewer and the connection parameters editor. 47 4.12 Wireframe of the advanced configuration page. 48 4.13 Wireframe of the node manager view within the simulation page and the user event scheduler modal. 49 4.14 Wireframe of the node output logs view within the simulation page. 50 x 5.1 The home page. 52 5.2 The documentation page. 53 5.3 The user interface preferences modal, where users can select their language and colour scheme. 53 5.4 The node programs viewer. 54 5.5 The node program editor, showing raw code input. 54 5.6 The network topology editor. 55 5.7 The network topology editor with the API Documentation accordion expanded 55 5.8 The network connection editor modal. 56 5.9 The simulation page, before a simulation has been started. 56 5.10 The node manager view on the simulation page. 57 5.11 The user events scheduler modal. ..