FACULDADE DE ENGENHARIA DA UNIVERSIDADE DO PORTO A Container-based architecture for accelerating software tests via setup state caching and parallelization Nuno Miguel Ladeira Neto Mestrado Integrado em Engenharia Informática e Computação Supervisor: João Miguel Rocha da Silva, PhD July 30, 2019 A Container-based architecture for accelerating software tests via setup state caching and parallelization Nuno Miguel Ladeira Neto Mestrado Integrado em Engenharia Informática e Computação Approved in oral examination by the committee: Chair: Prof. João Correia Lopes External Examiner: Prof. José Paulo Leal Supervisor: Prof. João Miguel Rocha da Silva July 30, 2019 Abstract In software testing, the scope of the tests can be represented as a pyramid, often called the “Test Pyramid” [31], which categorizes tests along 3 levels: Unit, Service, and End-to-end. The latter traverses the entire system attempting to simulate real user interaction and are the focus of this dissertation. The referred problem in this project will target the long execution times typically associated with end-to-end tests. This challenging and interesting problem was encountered during the de- velopment of Dendro [7], a data management platform for researchers, developed at FEUP Info- Lab [13]. With over 2000 end-to-end tests and 4 hours to complete the pipeline, the developers found it hard to get quick feedback on their work and to integrate with CI tools like Travis.CI which typically have a timeout period of 1 hour. Several tools have been developed in an attempt to optimize the execution times of tests and builds. Like: “CUT” [12]; “A Service Framework for Parallel Test Execution on a Developer’s Local Development Workstation” [35]; “Cloudbuild" [11]; and “Bazel” [2]. Although the first two target unit testing and the last two were designed to optimize builds, they all provide interest- ing solutions regarding time optimization. Some of these design solutions can be considered and adapted for optimizing local end-to-end execution. The approach implemented will: 1. accelerate end-to-end tests by using a setup caching mech- anism. This will prevent repetition patterns at the beginning of every end-to-end test by creating and saving the state previously; 2. parallelize the tests by instantiating multiple namespaced en- vironments. This would take advantage of the currently under-utilized CPU and I/O resources, which can only run a single build at a time. This dissertation aims to: develop or deploy a framework to improve the execution times of end-to-end tests; successfully completing the test stage under 1 hour so that common CI tools do not report timeout failures; publish a research paper on container-driven testing. The evaluation will be carried out through experiments with the large set of end-to-end tests for Dendro, running on the same hardware, and comparing the execution times with and without the developed solution. After a successful implementation, the tests proved that it is possible to reduce execution times by 75% only by implementing setup-caching. By introducing parallelization with 4 instances, the times can be further reduced by 66%. This solution converts a conventional run that takes 4 hours and 23 minutes to a run that takes 22 minutes with setup-caching and 4 parallel instances. Keywords: DevOps, software testing, pipelines, end-to-end tests, setup caching, parallelization, containers, Docker i ii Resumo Nos testes de software o escopo dos testes pode ser representado por uma pirâmide, também vulgarmente conhecia como a “Pirâmida de Testes” [31], a qual categoriza os testes em de 3 níveis: Unitários, Serviço e End-to-End. Estes últimos percorrem todo o sistema, procurando simular interacções reais do utilizador. Serão o foco desta dissertação. O problema referido neste projecto tem como objetivo o problema do longo tempo de exe- cução tipicamente associada aos testes end-to-end. Este desafiante e interessante problema surgiu aquando do desenvolvimento do Dendro [7], uma plataforma de gestão para investigadores, de- senvolvida no Infolab [13] FEUP. Com cerca de 2000 testes end-to-end e 4 horas para concluir a pipeline, os programadores começaram a ter dificuldades para ter rápido feedback no trabalho desenvolvido. Para além disso, a integração com plataformas de CI ficaram cada vez mais difíceis visto que tipicamente têm um intervalo de timeout de cerca de 1 hora. Várias ferramentas foram desenvolvidas numa tentativa de otimizar os tempos de execução de testes e builds. Por exemplo: “CUT” [12]; “A Service Framework for Parallel Test Execution on a Developer’s Local Development Workstation” [35]; “Cloudbuild" [11]; e “Bazel” [2]. Apesar de as duas primeiras terem como alvo testes unitários e as duas últimas terem sido desenhadas para otimizar builds, todas propõem soluções interessantes para otimizar tempos de execução. Algumas delas podem ser consideradas e adaptadas para otimizar execução local de testes end-to-end. Espera-se que a abordagem consiga: 1. acelerar os testes end-to-end através do uso do mecan- ismo de setup caching. Isto irá prevenir a repetição de padrões no início de cada teste através da persistência do estado numa fase prévia. E 2. paralelizar os testes através do instanciamento de containers nomeados. Isto tirará vantagem de possíveis recursos de CPU e I/O não utilizados, os quais, neste momento, só executam uma build de cada vez. Esta dissertação procura: desenvolver e criar uma framework para melhor a execução local dos testes end-to-end; Completar com sucesso a fase de teste em menos de 1 hora de modo a que as típicas ferramentas de CI não reportem falha por timeout; publicar um artigo relacionado com execução de testes orientado a containers. A avaliação vai ser executada através de várias experiências tendo como base a grande quanti- dade de testes end-to-end do Dendro. As experiências irão correr na mesma máquina e comparar os tempos de execução com e sem a solução desenvolvida. Depois de uma implementação bem sucedida, os testes provaram que é possivel reduzir os tempos de execução em cerca de 75% apenas por implementar setup-caching. Ao introduzir par- alelismo com 4 instâncias, os tempos são reduzidos em mais 66%. A solução converte uma ex- ecução convencial que demora 4 horas e 23 minutos numa que demora apenas 22 minutos com setup-caching e 4 instâncias paralelas. Palavras-chave: DevOps, teste de software, pipelines, testes end-to-end, caching de setup, par- alelização, containers, Docker iii iv Acknowledgements In this section, I will give my acknowledgements to everyone that helped during the course of the last five years and this dissertation. First I would like to give a special thanks to my supervisor, João Rocha da Silva. He is an example of what a true supervisor should be. Always available to help, charismatic and organized. Throughout the dissertation process, he provided multiple enlightening meetings that were full of creative ideas and solutions. The results of this dissertation would not be the same without his guidance. I would also like to thank my family. Even though they do not fully understand the topics of the course and my struggles, they were always supportive and never stopped believing in me. Without their inexhaustible support, I wouldn’t be here in the first place. Finally, I would like to thank all my former colleagues. Those colleagues, which are now close friends, provided a pleasant environment—both academic and personal—where we shared both our problems and happiness. It was key to look forward and finish this cycle. Nuno Neto Funding Acknowledgments This work is financed by the ERDF – European Regional Development Fund through the Operational Programme for Competitiveness and Internationalisation - COMPETE 2020 Programme, and by National Funds through the Portuguese funding agency, FCT - Fundação para a Ciência e a Tecnologia, within project POCI-01-0145-FEDER-016736. v vi “It is hard to fail, but it is worse never to have tried to succeed” Theodore Roosevelt vii viii Contents 1 Introduction1 1.1 Context . .1 1.2 Motivation . .1 1.3 Objectives . .2 2 Background and State of the Art5 2.1 Software quality assurance . .5 2.1.1 Levels of tests . .6 2.1.2 Black box testing vs. white box testing . .7 2.1.3 The test pyramid . .8 2.1.3.1 Unit tests . .8 2.1.3.2 Service tests . .8 2.1.3.3 End-to-end tests . .9 2.1.3.4 Test environment virtualization . .9 2.1.4 Structure of a test . .9 2.1.5 TDD and BDD . 10 2.1.6 Continuous integration . 10 2.1.7 Continuous delivery . 12 2.1.8 Continuous deployment . 12 2.1.9 Pipelines . 12 2.2 Virtual environments for automated testing and deployment . 12 2.2.1 Virtual machine architecture . 13 2.2.2 Container-based architecture . 15 2.2.2.1 Orchestration . 16 2.2.2.2 Applications . 16 2.2.3 Virtual machines vs. containers . 18 2.3 State of the Art . 19 2.3.1 Discussion . 23 3 Approach 25 3.1 Setup caching . 25 3.1.1 Test execution behaviour with setup caching with 3 distinct tests . 31 3.2 Test parallelization . 35 3.3 Integration tool . 39 4 Implementation 41 4.1 Goal, tools and frameworks . 41 4.2 Docker-mocha . 43 ix x CONTENTS 4.2.1 Tests and setups file . 44 4.2.2 Compose file . 45 4.2.3 Execution and options . 45 4.3 Architecture . 47 4.3.1 Runner . 47 4.3.2 Manager . 49 4.3.3 DockerMocha . 49 4.3.4 NoDocker . 50 4.3.5 Other . 50 4.4 Class execution architecture . 50 4.5 Graph extraction . 51 4.6 Issues . 51 4.6.1 Docker for Windows and macOS .