DEGREE PROJECT IN COMPUTER SCIENCE AND ENGINEERING, SECOND CYCLE, 30 CREDITS STOCKHOLM, SWEDEN 2020 Improving Software Testing in an Agile Environment JÉRÔME DE CHAUVERON KTH ROYAL INSTITUTE OF TECHNOLOGY SCHOOL OF ELECTRICAL ENGINEERING AND COMPUTER SCIENCE 1 Abstract Software development has evolved at an ever-increasing pace over the past years, one of the forces behind this acceleration is the move from on-premise application to cloud- based software: Software as a service (SaaS). Cloud computing has changed the way applications are deployed used and tested. Time-to-market for software has decreased from the order of month to the order of day. Furthermore, Source Code Management based on Git (introduced in 2005), changed the way software is developed allowing for collaborative work thanks to automatic merge and versioning. Additionally, continu- ous integration (CI) tools developed on top of Git facilitate regular testing, build and deployment.Nevertheless, despite of being necessary Integration tools, they require an extensive amount of cloud resources which may stretch the duration of code integra- tion. The goal of this thesis is to optimize the speed of the CI pipeline, improve the software tests while optimizing the load on the cloud resources. As a result of the work of this thesis the completion time of the software test has been decreased by 21%, and the total continuous integration completion time decreased by 18%. Furthermore, new bugs and anomalies were detected thanks to improved soft- ware test and new approaches for the emulation of extreme scenarios. The bugs were corrected making the system more resilient and improving the user experience. 1 Sammanfattning Programvaruutveckling har accelererat i en allt snabbare takt under de senaste åren. En av krafterna bakom denna acceleration är övergången från lokal applikation till molnbaserad programvara: Programvara som en tjänst (SaaS). Molntjänster (Cloud computing) har förändrat sättet applikationer distribueras, används och testas. Tiden till att mjukvaran når marknaden har minskat från månader till dagligen. Dessutom har källkodshantering baserad på Git (introducerades 2005), ändrat hur programvaran utvecklas, vilket möjliggör samarbete, detta tack vare automatisk sam- manslagning och versionshantering. Dessutom underlättar integrationsverktyg (CI) utvecklade ovanpå Git, regelbunden testning, byggnad och utveckling. Trots att de är nödvändiga integrationsverktyg, kräver de en omfattande mängd molnresurser som kan tänja på varaktigheten för integration. Målet med denna avhandling är att optimera hastigheten på CI-pipeline, förbättra programvarutester medan belastningen på molnresurserna optimeras. Som ett resultat av arbetet med denna avhandling har mjukvarutesternas slutföringstid reducerats med 21 %, och den totala kompletteringstiden för kontinuerlig integration reducerats med 18 %. Dessutom upptäcktes nya buggar och avvikelser, tack vare förbättrade mjukvarutester och nya tillvägagångssätt för emulering av extrema scenarier. Buggarna korrigerades, vilket gör systemet mer motståndskraftigt med en förbättrad användarupplevelse. 2 Acknowledgements I would like to thank my supervisor Shatha Jaradat for her hard work, time, and guidance throughout this thesis. I would also like to thank my manager and colleagues at Dassault system for their cooperation, advice and friendship. It has been a very challenging experience. I would like to thank my examiner Mihhail Matskin. Febuary, 9 2020 J´er^omede Chauveron 3 Contents 1 Introduction 8 1.1 Motivation . .8 1.2 Background . .8 1.3 Problem . .9 1.4 Purpose . .9 1.5 Goal . 10 1.6 Research Methodology . 10 1.7 Delimitation . 10 1.8 Ethics and Sustainability . 11 1.9 Outline . 11 2 Background and Related Work 12 2.1 Background on Software testing . 12 2.2 Pipeline Description . 14 2.3 Code and Testing Metrics . 16 2.4 Integration Testing & Functional testing . 17 2.5 Fuzzy testing Background . 19 2.6 Load testing Background . 21 2.7 Literature review and related work . 22 3 Methods 25 3.1 Code coverage for End-to-End testing . 25 3.2 Automating Coverage for End-to-End tests . 26 3.3 Background and Set-up description . 27 3.4 Implementation description . 28 3.5 Load Testing Tools Comparison . 28 4 Experiments 30 4.1 Functional test traces analysis . 30 4.2 Parameters influencing failure rate . 32 4.3 Gitlab Runner usage optimization . 34 4.4 Troubleshooting and Fuzz testing Results and Analysis . 36 5 Conclusion and future work 39 4 List of Figures 1 Software testing Pyramid [1] . 13 2 Git practices description [3] . 14 3 CI/CD Pipeline description . 15 4 Representation of process arrival in Gitlab [6] . 15 5 Example of Cyclomatic number computation . 16 6 Code Example for Halstead complexity computation . 17 7 Representation of an End to End Test scenario [5] . 17 8 Code example of an unreachable path via blackbox testing . 19 9 SAGE algorithm description [14] . 20 10 Flow Chart of the AFL algorithm . 21 11 GAN architecture [21] . 23 12 Instrumenting code an example . 25 13 HTML Code Coverage Report . 26 14 Troubleshooting setup description . 28 15 Evolution of Katalon test duration . 30 16 Pipeline execution time distribution . 31 17 Job triggering pipeline failure distribution . 33 18 Pipeline success function of the number of .ts files modified . 34 19 Merge Pipeline total execution time evolution . 35 20 Example of an asynchronous request failure . 37 21 Example of Linear size increase of factor 2 on a JSON of depth 2 . 37 22 Example of Recursive size increase of factor 2 on a JSON of depth 2 . 37 5 List of Tables 1 Comparison of different End-to-End testing tools . 18 2 Comparison of the different tool used for troubleshooting . 27 3 Comparison of the different Inter-process communication tools . 28 4 Comparison of the different tool for load testing . 29 5 Correlation Between code metrics and Pipeline success rate . 32 6 List of acronyms API Application Programming Interface AWS Amazon Web Services CI/CD Continuous Integration / Continuous Deployment HTTPS Hypertext Transfer Protocol Secure JSON JavaScript Object Notation IDE Integrated Development Environment OS Operating System QA Quality Assurance REST REpresentational State Transfer SSL Secure Socket Layer TCP Transmission Control Protocol UI User Interface VM Virtual Machine 7 1 Introduction 1.1 Motivation The DevOps approach and the Agile methodology for software development had a profound impact on the way software is being developed, shipped, and deployed. The era of main- frames (1970-1980) was defined by technologies such as Cobol and Multiple Virtual Storage (MVS) with 1-5 years release cycle and extremely high risks in terms of meeting the cus- tomer needs. Then came the era of client/server (1990) with still high risk and relatively slow release cycle: 3-12 month, the current era of cloud is pushing down the release cycle to the order of days. This new approach pushing for faster release cycle and thus driving quality analysis and software deployment towards more automation as manual testing is tedious, lengthy, and expensive. This thesis was conducted during the development of the UI for a web application referred to as the web-app with the purpose of improving software testing and the CI pipeline efficiency. 1.2 Background Software quality is a leading concern for companies as a bug might be extremely expensive in terms of time spent correcting it and impact on the client. To maintain a certain level of quality, tests are used to certify that the system can handle a typical user scenario (with different network configurations). Moreover the system needs to remain functional despite having given servers unreachable, high network congestion or unexpected API response. In fact those conditions may occur in a real-world scenario despite being very unlikely to arise in a testing environment. The objective is to have the most relevant tests possible and to have them being automated while using the minimum resources in terms of VM's. Relevant test meaning that the tests need to go through as much as possible of the system and remain close to real-world usage. Moreover, the test should access that the system is functional in various scenarios The studied system contains multiple types of tests the one detailed in this thesis are the following: • Functional test • Load Test • Unit Test • Chaos testing To evaluate and improve the relevancy of those tests we are using: • Code coverage a metric indicating how much of the code is being executed through a test (can be applied to any type of test but in our case is designed for functional testing). It is the percentage of the line being executed during the test as well as the execution detail line-per-line. The objective is to maximize the percentage of executed code and complete the functional test by unit tests for the part of the code not executed during the functional tests. 8 • Mitmproxy a proxy to modify on the fly response given by the server to test the resilience of the UI • Netem a network emulation tool: adding delay, loss, packet re-ordering, and breaking connection 1 The Code management platform used is Gitlab which enables us to extract traces of the activity and pipeline history and will be used to optimize the continuous Integration pipeline. 1.3 Problem Continuous Integration pipeline enables software to be built, tested and integrated in the master directory continuously. Enabling more collaborative work and faster release cycle. However, a CI pipeline is expensive to set-up and maintain due to the different jobs re- quired: deployment of the code, End-to-End test, Unit test and Build. All those jobs require a VMs, and maintaining them is a tedious process. Moreover the CI pipeline con- tains E2E test checking the entire system. But the E2E test is lengthy in terms of execution time and hard to maintain. On top of this, those tests need to remain relevant and evolve with the system. Otherwise a bug won't be detected when code is pushed, making the bug harder to locate and correct.