Design and Implementation of a Continuous Integration System

Prakash Acharya

DESIGN AND IMPLEMENTATION OF A CONTINUOUS INTEGRATION SYSTEM

Prakash Acharya Bachelor’s Thesis Spring 2019 Information Technology Oulu University of Applied Sciences

ABSTRACT

Oulu University of Applied Sciences Information Technology, Internet Services

Author: Prakash Acharya Title of the bachelor’s thesis: Design and Implementation of a Continuous Integration System Supervisor: Lasse Haverinen Term and year of completion: Spring 2019 Number of pages: 43

Not having an automated test system in the software development process leaves space for errors to go unnoticed in code review, which can break a codebase if integrated. The aim of the Bachelor’s thesis was to design and implement a Continuous Integration system where tests and build tasks could be automated.

The Buildbot framework was used as a Continuous Integration and automation framework. The Ansible playbook was used for automating the deployment of Buildbot configuration. The Buildbot configuration was created in such a way that it allows Buildbot instances to be created easily and in a configurable manner.

In addition, automation of the Python static code analysis tool, Prospector, was added to the Continuous Integration system.

The result of having a configurable Buildbot system was that the developers could start adding automation tasks and test and verify their change without affecting the production instance. Having an automated static code analysis run on changes brings into attention potential problems and error before they are integrated.

Even though the Buildbot system is not taken into use, it was tested in a test environment and it is shown to work. It will be taken into use once the configuration for an integration builder is added. The main benefit of the system is that tests and build tasks can be added and automated in Buildbot.

Keywords: Ansible, Buildbot, Continuous Integration, Gitlab

PREFACE

The thesis work was done for GE Healthcare Finland Oy, Helsinki, during the Spring of 2019.

I would like to thank my thesis instructor, Lasse Haverinen, and my language teacher, Kaija Posio, for their feedback on my writing. I would also like to thank my thesis supervisor, Riku Hämäläinen, for his help in clarifying ideas and tasks during the work.

Prakash Acharya

02.05.2019, Helsinki

CONTENTS

ABSTRACT 3 PREFACE 4 CONTENTS 5 1 INTRODUCTION 6 2 SOFTWARE DEVELOPMENT 8 2.1 Waterfall development 8 2.2 Spiral development 10 2.3 Agile development 12 3 CONTINUOUS INTEGRATION 13 3.1 Solutions 13 3.1.1 Buildbot 13 3.1.2 Jenkins 15 3.1.3 Gitlab CI 16 3.2 Continuous Delivery 17 3.3 Continuous Deployment 18 4 SOFTWARE TESTING 20 4.1 Functional testing 20 4.2 Non-functional testing 22 5 CONFIGURATION MANAGEMENT 24 Ansible 24 6 CONTINUOUS INTEGRATION DESIGN 28 6.1 Design choices 29 6.2 Components and design 30 7 IMPLEMENTATION 33 8 CONCLUSION 39 9 ANALYSIS AND FURTHER PLANS 40 REFERENCES 41

1 INTRODUCTION

Software development is a process encompassing requirements definition, planning, designing, writing code, testing, releasing, and maintenance of a software product. As software projects grow bigger, in terms of both developers and complexity, the development workflow in the team also becomes complex. To reduce the complexity involved in the integration and release cycle of development, the concept of Continuous Integration (CI) and Continuous Delivery (CD) were introduced in the software development process. In Continuous Integration, the process of testing software changes and integration are automated. Continuous Delivery takes this further by automating building and production of software and related artefacts regularly. CI and CD together ensure a regular incremental change to the software product.

Testing is an integral part of a software development process. Tests are important to ensure that changes do not introduce bugs, to ensure that changes do not break the existing functionality, and to ensure that the software works as it is required to. Software changes can be tested manually, or they can be automated depending in which phase of the development process they are carried out. Test automation is preferred to manual testing because it helps organizations to save resources and team time.

The thesis work was carried out for GE Healthcare Finland Oy, Helsinki. GE Healthcare produces patient monitoring devices. The work was done in a tool development team responsible for development and maintenance of tools and infrastructure used for automating build, integration, and release of software. The thesis project had two objectives.

The primary objective was to design and implement a Continuous Integration system for the team. To improve the current workflow, where changes to the tools developed by the team are integrated after code review, the requirement was to have a Continuous Integration system where tests and build tasks could be automated. When designing the CI system, another requirement was to design it 6

in such a way that allows the CI system to be easily reproduced. This requirement originates from the fact that verifying a change in the CI system itself, unlike in the tools which the CI system automates tasks for, requires a replication of the production instance of the CI system and making changes on top of it. This involves changing production environment specific parameters, such as database connections and secrets from the CI configuration. As the CI system is expanded with more automation tasks, identifying the variable components and changing them becomes difficult. In order to make verifying changes to the CI system itself easy, the requirement was to design the CI system to have a way to easily handle environment specific parameters. The other benefit of this approach would be that multiple developers could work on making changes to the CI system at the same time and verifying their changes independently without affecting the production instance of the CI system.

The secondary objective was to automate the static code analysis for Python tools in the newly developed Continuous Integration system so that the system would have a value for taking into use. Static code analysis tools analyse source code, without compiling or running them, to detect errors and complexity in the code. Most of the tools developed by the team are Python tools. Automation of the static code analysis on changes would give an immediate feedback to a developer if errors, coding standard violations, complexity or potential problems were detected in the change. Currently, developers manually run a static code analysis before submitting a change for a code review. This objective of the project shifts running of the static code analysis from manual to automated. Having the static code analysis automated also eliminates the possibility of a human error where a developer forgets to run the static code analysis. In addition, it also paves a way for expanding the CI to add more rigorous tests, e.g. unit tests and system tests, and automation tasks based on the needs of the team.

2 SOFTWARE DEVELOPMENT

Since software development involves people with different skillsets and their interaction for achieving a common goal, it is elementary that they co-operate and have transparency in their work. Different software development process models have been developed with an aim to optimise the process. While some process models, such as the Waterfall model, prescribe certain development steps to be executed in a certain order, other rather broader mindsets, such as the Agile, emphasise values and principles that should drive the development process. The size and requirements of a project play a critical role in determining the suitability of a model for the project.

2.1 Waterfall development

The Waterfall development model was one of the first to be introduced in the software development process. According to Hamilton (1999, ch1), the Waterfall model added a formal structure to the overall software development process for the first time and identified key steps, such as requirements definition, system design, coding, and testing. The main feature of this model is linearity: development is carried out in phases and each phase accomplishes a goal.

“-- it was created in 1970 by Winston Royce, and addresses all of the standard life cycle phases. It progresses nicely through requirements gathering and analysis, to architectural design, detailed design, coding, debugging, system testing, release, and maintenance. It requires detailed documentation at each stage, along with reviews, archiving of the documents, sign-offs at each process phase, configuration management, and close management of the entire project. It's a model of the plan-driven process.” (Dooley 2011, ch2.)

The figure 1 below shows how development is carried out in phases, each working as an input for the next. For instance, a software architectural design phase starts only when all the requirements for the project have been defined. This requires that all the requirements for a software are well-known and

exhaustively covered during a requirement gathering phase. The same pattern is seen throughout the process. This model is heavily criticised.

FIGURE 1. Unmodified Waterfall model (Wikipedia 2019c. Date of retrieval 24.03.2019)

The Waterfall model works well for projects where the requirements for the project are well understood at the beginning of the process. Since it makes a clear distinction between the phases, planning and allocation of resources as well as progress tracking can be more effective. For instance, it works when developing software for a digital clock whose design or requirements are unlikely to change during the development. However, this model does not work for most systems.

Having no iteration is one of its major drawbacks. Dooley (2011, ch2) argues that the model is fundamentally based on an assembly-line mentality for developing software. The waterfall diagram shows no way to go back and rework on design if a problem during the implementation is found. Another reason for its 9

disapproval is that it does not address a requirements-change. It is very difficult to know all the requirements at the beginning of the work. In most cases, requirements change somewhere during development. Thus, finishing one phase before the other begins is difficult. (Dooley 2011, ch2.) The requirements can change, for instance, because a customer’s business needs change, or a customer requires different features than the ones initially defined for a competitive advantage.

2.2 Spiral development

The Spiral development model addresses the shortcomings of waterfall model and emphasises heavily on risk management. Barry Boehm, who introduced this model, describes the Spiral development as “a risk-driven process model generator.” (2000, 3). It not only goes one step further from the Waterfall model to introduce iteration, it generates process models based on risk assessment for each iteration.

A process model answers what should be done after a phase and for how long. Risks are defined as situations that can cause a project to fail to meet its goals. Under the spiral model, a process model is chosen based on risk considerations and it can vary even from one spiral cycle to the next. (ibid., 4.)

A redrawing of the original spiral model (ibid., 2) showing four phases in an iteration is shown in the figure 2. Boehm argues that the spiral model is not universally understood. The figure 2 contains oversimplifications that cause the model to be poorly understood. He emphasises the need to note that not everything on the project follows a single spiral sequence and not every element in the diagram needs to be visited in the order indicated. It is also possible to backtrack to revisit previous decisions. Hence, the spiral model is a risk-driven process model generator, in which different risk patterns can lead to choosing incremental, waterfall, evolutionary prototyping, or other subsets of the process elements in the spiral model diagram. (ibid., 3)

FIGURE 2. Spiral development model (Boehm 2000, 2)

The Spiral model has its advantages and disadvantages. It is suitable for large projects where a risk analysis and its early mitigation is essential. This model maximises the confidence in chosen methods and technologies since alternatives and the risks associated with them are investigated early and thoroughly. Since iterative development is promoted, customer feedback can be considered when planning another iteration.

However, the Spiral model is not suited for small projects. It requires complex management to ensure the suitability of each iteration’s plan and it can make low- risk projects unnecessarily complex. Another downside of this model is the production of extensive documents. Documents produced during several

intermediate phases of development may not necessarily be used in the final product.

2.3 Agile development

While previous development models focused on planning, Agile focuses on adaptation to change. Agile software development is an umbrella term for a set of frameworks and practices based on the values and principles expressed in the Manifesto for Agile Software Development and the 12 principles behind it. Agile development values motivated individuals and interactions over processes and tools, working software over comprehensive documentation, customer collaboration over contract negotiation, and responding to change over following a plan. (Agile Alliance 2001a. Date of retrieval 31.03.2019.)

Agile development does not define methodologies or conventions to follow. Rather, development teams are encouraged to find the most appropriate set of conventions and follow them. If the conventions conform to the values and principles of Agile Manifesto, then the teams are agile. Some of the principles behind Agile Manifesto include a continuous delivery of valuable software to customers, welcoming a change even late in development, building projects around motivated individuals giving them the right environment, paying a continuous attention to a technical excellence, and reflecting on how to become more effective at regular intervals. (Agile Alliance 2001b. Date of retrieval 31.03.2019.)

In order to deliver working software frequently to the customers, a traditional approach of building software only before delivery and often late in development is insufficient. To overcome this insufficiency, development teams started to automate the software build process as well as running tests. The concept known as Continuous Integration refers to the practice of integrating code changes often and having automated tests run on changes before integrating. Continuous Delivery extends this to also building the software regularly, e.g. daily, so that a functional shippable software is available at any time.

3 CONTINUOUS INTEGRATION

The term continuous in Continuous Integration (CI) implies automation. Automation ensures that a change can proceed, without human intervention, through the stages of testing and validation as well as reporting of failures during testing to a responsible developer. (Laster 2017. ch1.) This automation enables developers to see early if their changes pass tests and do not result in a broken code base. Developers are automatically notified of the test results, thus they can work on a fix if one is needed.

3.1 Solutions

Because CI helps developers to maximize their work, many CI tools have been developed with an aim to provide developers the features they need and maintainers the possibility to maintain and scale the tool easily.

3.1.1 Buildbot

Buildbot is an open-source Continuous Integration framework written in Python. Buildbot focuses on being customisable for projects so that projects can take a full advantage of Buildbot features and extend them according to project needs. “At its core, Buildbot is a job scheduling system: it queues jobs, executes the jobs when the required resources are available, and reports the results.” (Buildbot 2019a. Date of retrieval 31.03.2019).

Buildbot has two major components that interact to perform all the work: a master and workers. A typical Buildbot setup consists of one master, also called a build master, to receive a change and to decide how the change should be processed, and multiple workers, often installed on different machines, to execute the tasks sent by a master. A Buildbot master is configured using a configuration file called master configuration. Buildbot master configuration must contain all the information that Buildbot needs, such as worker names and credentials for authentication, a repository address to process changes from, and an instruction

on how different kinds of changes should be processed. Buildbot workers also have configuration files with information, such as the worker’s name, password to use for authentication to the master, and contact information of the maintainer of the worker. Buildbot worker configuration does not change often because once workers are connected to the master, Buildbot master configuration determines what the worker executes and when.

FIGURE 3. Buildbot in Action (Buildbot 2019a. Date of retrieval 31.03.2019)

As shown in the figure 3, Buildbot can poll repositories for changes or repositories can notify Buildbot of changes. A software change often implies a commit or a group of commits that change the code in a version-controlled codebase. Polling means inquiring a source repository periodically to find if any changes have been made. When a change is detected, the Buildbot master sends commands to workers to execute tasks depending on how the master is configured. The Build master can also be configured to send a build status report to multiple targets, such as email messages and IRC (Internet Relay Chat) chats.

However, the process of determining how and when to process a change involves Buildbot components called schedulers and builders. Schedulers are components that receive every change notification from the build master and determine whether to start a build and when. If schedulers decide to process a

change, they schedule a build for one or more builders. A builder is composed of build steps that are executed in a worker in the order defined in the Buildbot master configuration. A step can even be configured to trigger a different scheduler that is configured to start a builder with entirely different build steps. For example, a builder can have steps to fetch source code, build an application and trigger a scheduler that schedules builds for another builder configured to build documentation. This allows a conditional execution of certain types of builds. (Buildbot 2019b. Date of retrieval 02.05.2019)

One major advantage of Buildbot is its customisability and extensibility. Since the Buildbot master configuration file is essentially a Python script, the full power of the Python programming language can be leveraged when configuring a Buildbot instance. For example, tools can be written and used in the master configuration file as well as the libraries offered by the Python programming language. (G2 crowd 2019a. Date of retrieval 02.05.2019.)

Buildbot is also popular among its users for its waterfall view of the builds. The view shows all the builds of one kind in a waterfall-like vertical view so that it is easier to see how each build is performing over time. A waterfall view gives an overview of all the builds on a single page.

One major weakness of Buildbot is its documentation. Buildbot’s documentation is well-versioned but it is incomplete and outdated in some cases. Buildbot also has bugs which take a very long time to be fixed by the Buildbot developers. (G2 crowd 2019a. Date of retrieval 02.05.2019.)

3.1.2 Jenkins

Jenkins is a self-contained, open source automation server which can be used to automate tasks related to building, testing, and delivering or deploying software (Jenkins. Date of retrieval 01.04.2019). Jenkins integrates with many systems and has more than 1000 community-contributed plugins to support automation. It is a Java-based CI system.

Since Jenkins is open-source, it is popular and has a large userbase making it easy to get help from an online community. One of Jenkins’s major strengths is its plugins. Jenkins plugins have support for build notification, integration with cloud service providers, plugins for different tests, and integration with authentication systems, among others. Plugins are not just developed by the developers of Jenkins program, users can also submit their plugins so that other users can use and collaborate on enhancing them.

Although having support for extensive plugins makes it easier to integrate services to Jenkins, it also makes it difficult to maintain the plugins. Plugin-centric design makes it a heavy program compared to other CI systems. Compatibility of plugins is another issue with Jenkins. Similar plugins providing the same feature may not be compatible with one another. Updating Jenkins can also result in an unstable system if the plugins have incompatibility with a newer version of Jenkins. (G2 crowd 2019b. Date of retrieval 28.04.2019.)

3.1.3 Gitlab CI

GitLab is a single application for the entire software development lifecycle. Gitlab has support from project planning and source code management to CI/CD, monitoring and security. (Gitlab 2019b. Date of retrieval 02.04.2019). Gitlab is a free and open-source CI system which has features for repository hosting, code review, software development planning boards, information pages for projects, and CI/CD features.

As the figure 4 shows, Gitlab CI can build and automatically run tests on changes which can be reviewed and deployed to a production environment for consumption by users.

FIGURE 4. Gitlab CI/CD Pipeline (Gitlab 2019a. Date of retrieval 02.04.2019)

Gitlab is based on the Git version control system which is a popular version control system. A version control system keeps track of changes made to files that are added to the version control system. Git is useful for version-controlling text files.

Gitlab’s CI system is based on a configuration file where CI jobs are defined. Since the configuration file is version-controlled and is stored in a repository, it gives freedom and visibility to developers to see how a CI system is being run.

Since Gitlab is a comprehensive package providing all the features needed for a software development project, its requirements for resource are high. Gitlab is offered as a community and an enterprise edition. The enterprise edition is not free and offers support and more features than the community edition.

3.2 Continuous Delivery

Continuous Delivery is an extension of Continuous Integration. In the Continuous Delivery cycle, the isolated changes merged and validated during CI can be combined with the remaining product code. Then, the set is put through progressively more encompassing types of testing. The goal of Continuous Delivery is not necessarily to deploy the end result, but to prove that the end result is deployable. (Laster 2017. ch1.)

A CD system can have more rigorous tests implemented than CI. While CI tests whether individual changes work separately, CD takes all the changes made during a period, e.g. a day, and produces the software. This production process is automated and repeatable. The software produced in each cycle in this manner should be potentially releasable. The major benefit of this approach is an automated and repeatable release of builds. This also reduces the software release related work near a software release compared to the traditional development practices.

What is not done in CD is the deployment of software to a production environment. A production environment is where the software users can access and use the software. For example, users of a web application access the application using a web address. The production environment is the server that serves the web content under the web address that users use. CD can make available a new version of software which can be tested further by potential users before deploying to the production.

3.3 Continuous Deployment

A Continuous Delivery combined with an automated deployment is a Continuous Deployment. While a Continuous Delivery ensures that a potentially releasable software is made available in short intervals, a Continuous Deployment deploys the latest software so that no human intervention is involved in the release process. The implication is that a new version of application is available for users to use in short cycles. A new version of software is made available for downloading in case of compiled applications.

To minimize the risks associated with an automatic deployment, several deployment techniques can be employed. For example, a new deployment can be made available to a group of customers who wants to try new features and provide feedback, or the application can be deployed to users of a certain region. The advantage is that if any unexpected behaviour emerges in the release, the

change can be rolled back from that group of users. In this way, not all users are affected if a change proves to be faulty.

The figure 5 shows the major difference between a Continuous Delivery and a Continuous Deployment. Unlike in a Continuous Delivery, a change goes through the release pipeline without human intervention in a Continuous Deployment.

FIGURE 5. Continuous Deployment: Beyond Continuous Delivery (Slideshare. Date of retrieval 10.04.2019)

4 SOFTWARE TESTING

Software testing is an activity aimed at verifying a software’s functionality as well as finding defects. Software testing is an integral part of software development. Software’s functionality can be verified by testing whether the software performs as expected when operated in a way it was designed to be operated. This can include using the input the program expects and seeing whether the desired output is received. On the other hand, testing also aims at finding whether a program does something that it is not expected to do. Furthermore, tests also aim to find how software performs when it is supplied data that it does not expect.

For example, a program that finds the day of week from a given date, expects a date to be provided by its user. Tests can be implemented to find whether the application returns a right day for a given date and how it behaves if the date is given in a different format, e.g. month/day/year instead of day/month/year, or some random letters are given instead. A reliable program should work as intended under normal conditions and should be able to handle unexpected situations without breaking the program.

Tests can majorly be categorized into two broad groups: functional tests and non- functional tests. Tests that verify a system’s functionality are functional tests whereas tests that verify a system’s qualitative attributes, such as ease of use, performance and load-handling are non-functional tests. An example of a functional test is a unit test where a single unit of a software, a function for example, is tested. An example of a non-functional test is a usability test where a software is tested to find out how easily users can find what they look for in an interface.

4.1 Functional testing

Functional testing is directly or indirectly tied to functional requirements of a software. Functional requirements specify what functions a system should be able to do. For example, a functional requirement could be that a user needs to

be able to create a user account on a web application. Functional testing then tests whether the requirement is fulfilled. There are various types of functional tests carried out at different phases of software development.

Unit testing Unit tests are functional tests which are run in a unit component of a software. The unit can be a function or method, or it can also be a class in an object- oriented program. A unit test ensures a software unit works as expected. For example, a function may accept two numbers and return their sum. A unit test can test this function with positive and negative integers, letters and other characters and ensure that invalid data raises an exception which should be handled in the function. Unit tests are run independently for each unit. These tests are written by developers as they write software source code. This process helps to produce better code that goes through other tests afterwards.

System testing System testing tests a software system as a whole after components are integrated based on the requirements of the system. System testing is aimed at finding defects based on a built software’s behaviour. System tests are performed after unit tests and integration tests. Integration tests test how well software modules, which need to work together, perform when integrated. System testing aims to find out if the system works as expected, and it meets the requirements set by the users.

Acceptance testing Acceptance testing in software testing is done when software has been developed. Acceptance testing is done in order to ensure that the product meets the requirements that were defined at the beginning of the project. If all the tests in the acceptance test suite pass and the project stakeholders are satisfied with the product, the customers can accept or reject the product. In the acceptance test execution, test cases are executed with predetermined data as input, results are recorded as outputs, actual and expected results are compared and test results are determined. The objective is to provide confidence that the developed 21

product meets both functional and non-functional requirements. (Wikipedia 2019a. Date of retrieval 23.04.2019).

FIGURE 6. Software testing pyramid. (CSQE. Date of retrieval 14.04.2019)

As shown in the figure 6, the cost of maintenance and complexity involved in running tests increase as the software development process nears its end.

4.2 Non-functional testing

In contrast to functional tests, non-functional testing tests qualitative aspects of a program. Qualitative aspects are equally important, but they are not readily visible. Tests to check a software’s performance under load, software’s ease of use, software’s security, and software’s compliance to standards fall under a non- functional test category. These tests are mostly performed after functional tests.

Usability testing Tests conducted to find out how easily users of a software can perform tasks is known as usability testing. The easier it is for users to navigate and find items in a user interface, the more efficient and productive the user experience is likely to

be. Usability tests may be conducted during or after product development. Participants in the test are real users or people from outside the development team. This provides a metric on how real users use the product and whether they can find an item of interest easily. For example, an online web store can ask participants to find a product, add it to cart, and proceed to buy the item in their website. Participants’ activities are monitored to find how they interact with the system and where they search for items. If participants have difficulty finding the products, it might suggest that search functionality needs an improvement.

Load testing A load test is usually conducted to understand the behaviour of a system under both normal and anticipated peak load conditions. This load can be the expected concurrent number of users on the application performing a specific number of transactions within the set duration. (Wikipedia 2019b. Date of retrieval 14.04.2019.)

Software may perform inadequately. A long response time may be observed, for instance, under a load for various reasons, such as database transaction limitations, memory insufficiency, and CPU load. Load testing shows areas that can be improved to better handle heavy-load situations.

Static analysis Static code analysis tools analyse the source code without running them. They aim to find defects, potential programming mistakes, and non-conformance to coding conventions and standards. Static code analyses give an overview of the quality of the code. The importance of static code analysis is that it indicates at an early phase of development if code starts to violate design principles. When accumulated, these violations make the maintenance of programs difficult. If engineering design principles are followed and adhered to throughout the development process, software programs become more maintainable. For example, a Python static code analysis tool, Prospector, defines its function as “to analyse Python code and output information about errors, potential problems, convention violations and complexity”. (Prospector. Date of retrieval 23.04.2019.) 23

5 CONFIGURATION MANAGEMENT

Configuration management encompasses the practices and tooling to automate the delivery and operation of infrastructure. Configuration management solutions model infrastructure, continually monitor and enforce desired configurations, and automatically remediate any unexpected changes or configuration drift. (Puppet. Date of retrieval 07.04.2019.) Server configuration management tools keep track of server’s desired configuration state. It is the responsibility of the configuration management tool to enforce the desired state on the servers. For example, if it must be ensured that a web server is always running, the configuration management tool is updated to indicate the same. The configuration management tool is responsible for ensuring that the desired state is always achieved, i.e. a web server is always running. The same configuration management tool is responsible for starting a web service if needed.

Configuration management tools make server management easier. A new server with identical configuration to an existing one can be created with a minimal manual work. This is useful when a server goes down and an identical replacement server is needed. When the configuration management tool learns that a server needs to be in a certain configuration state, it does all the necessary tasks to bring it to that state. In addition, since the desired state of the server is written to files, version controlling can be used on those files to keep track of the configuration change over time. This also enables an easy rollback of configuration change if some service breaks upon a new configuration deployment.

Different tools have their own mechanisms for enforcing the desired state. Some of the popular configuration management tools are Ansible, Puppet, and Chef.

Ansible

Ansible is a simple IT automation engine which automates cloud provisioning, configuration management, application deployment, intra-service orchestration,

and many other IT needs. (Ansible. Date of retrieval 14.04.2019). Ansible is only installed in a machine that controls the infrastructure. Ansible accesses target servers using SSH (Secure Shell). Tasks to be performed in order to bring the host in a desired state are defined in the YAML (YAML Ain’t Markup Language) language. YAML is a human-readable language. Ansible has components, such as playbooks, roles, and Ansible Vault, which make the working of Ansible possible.

Playbooks Ansible playbooks are used to manage a configuration deployment or to execute a series of commands on the target host. Playbooks can contain one or more plays to be run on defined hosts. A play is a list of tasks. For example, a playbook can have two plays: one running in a Windows server and the other in a Linux server. Each play can have an operating system dependent list of tasks, e.g. installing and starting an IIS web server on Windows and installing and starting an Apache web server in Linux.

The figure 7 shows a playbook containing one play with two tasks. The first task is to install an Apache web server and the next is to keep the server in a running state.

The tasks defined in a playbook are idempotent. Idempotency means that the result of running a playbook once and running it more than once has the same effect. For example, running the playbook in the figure 7 should for the first time install an Apache server and start it. The second run, however, will have no change because the desired state mentioned in the play has already been achieved, i.e. Apache has already been installed and the server is already running from the first run. Ansible playbooks are manually run whenever a new configuration needs to be pushed to controlled hosts.

FIGURE 7. Ansible playbook example. (Slides. Date of retrieval 14.04.2019)

Roles Ansible roles allow a more advanced configuration management. Ansible roles are useful when a series of steps need to be executed in order to enforce a desired state. In addition to having a list of tasks, roles allow executing tasks based on certain events, making use of template files and having dependent roles that are run before running the current role. For example, a role for managing a web server can have a task that is triggered only if a web configuration changes, a task that updates a web server configuration in the target host by using a template file. The template file can be filled automatically using the variables passed to the role. Ansible roles have a defined directory structure, which determines the tasks and their order to be executed in the target host. Ansible roles are typically written for achieving a single defined goal.

Vault Ansible Vault is an Ansible feature to store secrets, such as passwords and security keys. Managing servers and services requires using secrets; and storing them in playbooks or roles is insecure. Vault files are encrypted and require a password for decryption, but they can be shared and version controlled. Secrets stored in Ansible Vault can be passed to the playbook being run at the time of execution by specifying the location of Ansible Vault file and its password.

6 CONTINUOUS INTEGRATION DESIGN

As shown in the figure 8 below, the current workflow in the team does not involve running tests of any kind on code changes. Making a change to a tool starts by cloning the repository, updating a local copy if already cloned, from an internally hosted repository server. A change is made in a developer’s work environment and pushed to Gerrit for a code review. Gerrit is a code review tool where developers can review other developers’ code change and give feedback. Code Change is integrated once code reviewers approve the change in Gerrit. This workflow has an inherent weakness of mistakes being overlooked during the code review.

FIGURE 8. Current workflow for making a change to tools 28

In addition to mistakes being overlooked, the current workflow suffers from having no mechanism to examine a code change for conformance to coding standards and conventions. Code not conforming to standards and conventions quickly makes maintenance of the codebase a burden. The current mode of working also requires someone to always examine the code after each change. This is very inefficient because reviewers spend time examining the code and still run the risk of not noticing errors.

In order to address the lack of an automation system, the primary objective of the project was to design and implement a Continuous Integration system. The design requirement was that the CI system should be easily reproducible for verifying changes in the CI system itself. This would serve two purposes: tests and automation tasks could be added to the CI system, and the addition of tasks could be verified easily before incorporating the change to the production system. The CI system would act as a bridge enabling the team to add automation tasks for code changes – mainly automation of tests.

The secondary objective of the project was to automate a static code analysis for Python tools in the newly developed Continuous Integration system so that it would be valuable to be taken into use.

6.1 Design choices

To address the first objective of designing a CI system, different Continuous Integration systems were studied and evaluated. The Buildbot framework was chosen as the CI framework for the project. The main reason was its support for extension and customisation. Buildbot is also lightweight compared to Jenkins. Gitlab CI would have been a good alternative for its integration with the Gitlab repository. But since the team was developing and maintaining other Buildbot instances for software development teams, it would be more consistent with the existing setup to have a Buildbot CI. In addition, the team had expertise in maintaining Buildbot systems.

Ansible was chosen as the configuration management tool for its ease of use and Ansible was already being used for maintaining other Buildbot instances. This would allow the reuse of existing Ansible roles and dependencies. Ansible was chosen to automate the tasks involved in creating a new Buildbot instance. Ansible roles would be written for automating the installation of the Buildbot master and worker, Ansible Vault would be used for storing secrets, and Ansible playbooks would be written for executing the roles passing the secrets from Ansible Vault.

To achieve the secondary objective of adding a value to the system, the idea at the very beginning was to write and automate unit tests for an existing Python tool. But later, it was decided that running a static code analysis on Python tools would be more valuable. This would have an immediate and broader benefit of covering a large number of tools compared to having unit tests for a single tool. Prospector was chosen as the static code analysis tool because it brings together the functionality of several other static code analysis tools. It combines tools, such as Pylint, a static analysis tool; pep8, a tool to analyse conformance to style guides; and McCabe, a code complexity checking tool.

In addition to the implementation of a Continuous Integration system, a few major changes to the current workflow were planned. It was planned that the source repository server would be changed from Gerrit to Gitlab. Migration to Gitlab for repository hosting was mostly because of an organisational change. The code review would be done in Gitlab. The Continuous Integration system would send its build status report to Gitlab so that code reviewers could see whether tests were passing from the Gitlab code review interface.

6.2 Components and design

The figure 9 shows the design of the workflow after implementing the new CI system.

FIGURE 9. Workflow after CI system implementation

The new workflow was designed so that Buildbot would poll the Gitlab repository for changes. Upon detecting a change, Buildbot would automatically trigger a static code analysis build if Python tools were changed. Buildbot would send a build status back to Gitlab. The build status would be visible to code reviewers so that they would not have to review code that did not pass tests. A developer would submit a new patch set if CI failed or if reviewers suggested something to be done differently. Furthermore, if merge conflicts occurred, i.e. other developers changed the same part of the code as the developer in question did, a conflict would have to be resolved first. When the change would be approved, it would trigger an integration build which would integrate the change.

To address the requirement of having an easily reproducible CI setup, it was planned to store environment specific variables of the Buildbot instance in a file. The Buildbot master would be configured to read the variables file and create a Buildbot instance using the supplied variables. This would ensure that every other

detail in the master configuration remains the same while only environment specific parameters are changed. The variables file would be copied from the Ansible role to the Buildbot master’s directory as a templated file. The variables would be passed as parameters to the Ansible role in the playbook so that it would be visible while running the playbook. In addition, the Ansible role would be created for installing Buildbot, for copying the Buildbot master configuration file to the Buildbot’s installation directory, and for copying variables file and secret files.

Ansible role •Use role for creating •Buildbot master master configuration •Pass variables to role •Install Buildbot •Use variables file for •Use role for creating •Copy master creating Buildbot worker configuration file instance •Copy variables as a file Ansible to Buildbot's Buildbot playbook installation directory

FIGURE 10. Interplay of Buildbot and Ansible

As shown in the figure 10, it would be possible to create new instances of Buildbot by passing different variables in the Ansible playbook which would be used by the Buildbot master configuration.

It was planned that the Buildbot master configuration would be created by a function. The function would read the variables file copied by the Ansible playbook and create a Buildbot instance accordingly. The Buildbot master configuration would also be updated for adding a builder configuration for running Prospector. The builder configuration would be conditional to run only if a change would introduce a Python tool change. 32

7 IMPLEMENTATION

Buildbot configuration After the system had been designed, it was implemented in steps. First, the latest version of Buildbot, v2.0.1 at the time, was decided to be used for the project. Buildbot’s default configuration file was used as a template for creating the desired configuration. Initially, only the configuration necessary for running a Buildbot instance was added. This configuration included, for example, the name and URL (Uniform Resource Locator) of the Buildbot, a repository URL to poll for changes, Buildbot’s web server port, and the database URL. Buildbot was configured to poll the Gitlab repository every 30 seconds. A scheduler was added to process changes in any branch of the repository. The scheduler was configured to schedule a build for a generic builder. The builder was configured, at this stage, with only one step to fetch the new change from Gitlab to Buildbot.

Since fetching the change was configured as a build step, reporting a build status to Gitlab on whether fetching was succeeded or not was possible. A personal access token required for sending the build status to Gitlab was generated and saved in Ansible Vault. The configuration required for sending the build status to Gitlab was added to the master configuration.

Configuration was added in the Buildbot master configuration to allow a worker to connect to the Buildbot instance. A password required by the worker to connect to the master was also saved in Ansible Vault.

The Buildbot configuration was moved inside a function. The function was defined to take parameters passed to a variables file. A functionality to read the variables file and call the function with its content was added. An overview of the Buildbot master configuration is shown in the figure 11. The parameters passed to function would be substituted in the main configuration to create a similar but a configurable Buildbot instance.

Function to create Buildbot instance (takes parameters): Set Buildbot’s database and access URL Polling source repository for changes every 30 seconds Scheduler listening for any change Builder that defines how to fetch new change Worker definition where change is fetched Send build status to Gitlab Show builds in web User Interface

Create Buildbot instance by passing variable file’s contents

FIGURE 11. Buildbot’s master configuration overview

When Buildbot configuration was in place, Ansible roles and playbooks were added.

Ansible roles

Ansible roles were needed for installing and configuring both the Buildbot master and workers. Roles for installing the Buildbot master and workers were created by another team member. The author added tasks to the role to template and copy the variables file and tasks for copying secrets from Ansible Vault to the Buildbot master.

The variables needed for creating a Buildbot instance were planned to be passed from the Ansible playbook as role parameters. A variables template file was added to the role to configure the Buildbot master. The Ansible role would substitute the values passed as role parameters in the variables template. The templated file would be copied to the Buildbot master’s directory. Tasks to copy secrets, such as worker authentication secret, SSH key, and Gitlab personal access token, were added to the role. The secrets were configured to be copied from Ansible Vault when the Ansible playbook would be run. 34

Tasks: - Install Buildbot in virtual environment - Copy Buildbot master configuration

- Copy SSH key

- Copy worker secrets

- Copy Gitlab personal access token

- Template and copy variables file

FIGURE 12. Overview of Ansible role for configuring Buildbot master

As shown in the figure 12, tasks were added to the master role to copy secrets and configuration files to the Buildbot master’s directory.

Ansible Vault secrets

Secrets, such as a Gitlab personal access token and worker connection secrets, were saved in Ansible Vault. As shown in the figure 13, these secrets were defined as key value pairs so that the secret values could be accessed from Ansible roles by using corresponding keys. The name of the secrets as saved in Ansible Vault would be passed as role parameters. The passed name would be looked for in Vault and a corresponding secret would be transferred to the Buildbot master. Ansible Vault was password protected.

SSH keys: Key name: SSH key Gitlab tokens: personal access token name: token Workers:

worker name: worker password

FIGURE 13. Overview of secrets stored in Ansible Vault 35

Ansible playbook

A Playbook for deploying both the Buildbot master and worker was created. Two plays were defined in the playbook: one for deploying the Buildbot master and one for the worker. Corresponding roles were added to the plays and required parameters were passed to the roles.

Play to setup Buildbot master: Run on defined hosts Run tasks defined in master role: With these parameters: Buildbot Configuration file: file name SSH key name: key name Gitlab personal access token name: token name Repository URL: repository URL

Database URL: database URL

FIGURE 14. Overview of Ansible playbook for setting up Buildbot master

As shown in the figure 14, Ansible playbook was created for setting up the Buildbot master where variables could be passed to the Buildbot master role. A play for configuring the Buildbot worker was also added in the playbook.

When the Ansible Vault, playbook, and role configurations were in place, a documentation explaining how to setup a new Buildbot instance was created. The documentation was followed to create a Buildbot instance in a test environment. A new Buildbot master and worker instances were created by running the playbook. The master polled a test repository in Gitlab every 30 seconds. If any changes were found in the repository, it was fetched. The status of the fetching operation was visible in the Gitlab web user interface.

When the current configuration was proven to work, one of the requirements of the project to have an easily reproducible Buildbot instance was achieved. It was now possible to create new instances of Buildbot by passing different parameters from the Ansible playbook. The process was now easy and automated because of Ansible roles and playbook.

Static code analysis

The second requirement of the project was to automate the static code analysis on Python tools. A new task was added in the playbook for installing Prospector in a virtual environment in the worker. An existing Ansible role for installing Python packages in a virtual environment was used for installing Prospector. The installation of Prospector would be run for Python3 tools changes. The Buildbot configuration was updated to add builders and schedulers for running Prospector.

A builder for running Prospector was added. A corresponding scheduler was added. It could be triggered by a Python3 tool change. The scheduler was configured to start the Prospector builder. The builder for running Prospector was updated with steps necessary to run a static code analysis. If any issues were found by Prospector, the build status would be sent as a failure to Gitlab.

The builder previously used only for fetching a change was expanded to extract the name of changed files in a change. A list of Python3 tools was also constructed. The builder was configured to compare the changed file names and file names in the Python3 tools list. It was then configured to trigger the Prospector build if a Python3 tool change was detected. The steps that are run in the main builder and the Prospector builder and how Prospector is run only if the Python3 tool is changed can be seen in the figure 15 below.

FIGURE 15. Steps executed in main builder and Prospector builder

When the play for installing Prospector was added to the playbook and a builder configuration to run Prospector was added in the Buildbot master configuration, the playbook was run again. The updated Buildbot configuration was in place, and Prospector was installed in the worker.

The configuration change was tested using a test repository and dummy Python3 tools. Triggering of Prospector from the main builder and Prospector’s code analysis on changed tools worked.

8 CONCLUSION

The aim of the project was to design an easily reproducible Continuous Integration system and to implement automation of static code analysis for Python tools. The tool development team had no setup to automate tasks and tests for the tools they developed. A Continuous Integration system was designed and developed in the project. The static code analysis for Python3 tools was automated and tested in the testing environment.

Both the objectives of having an easily configurable CI system as well as an automated static code analysis for Python tools were achieved. However, the system needs one more builder before it can be taken into use. The next task is to configure an integration builder which integrates a change after the change has been approved in Gitlab. When deploying the system to the production environment, the variables and secrets used when testing the configuration needs to be changed as well.

Since the project had two objectives and Ansible was new to me, sometimes the concepts were not immediately clear, and I misunderstood some ideas. Several meetings were held to plan and discuss the tasks in the team which proved to be very effective. I also have now a better understanding of Continuous Integration systems in general and also of the Scrum framework.

In conclusion, even though the result of the project is not taken into use yet, it is very close. The work that has been done so far has been tested and works as expected. It will be ready to be taken into use once the configuration for the integration builder is added. When the new system is taken into use, it will enable the team to add further tests and build automation tasks. This addition of tasks can then be easily tested and verified before taking into use thanks to the configurable design.

9 ANALYSIS AND FURTHER PLANS

Overall, the project went well. As the aim of the project was to have a system in place where automation tasks could be added, there remains a lot to be added now. Most importantly, tests can be added and automated. Documentation building can be automated in the new system as well.

REFERENCES

Ansible. How Ansible works. Date of retrieval 14.04.2019. https://www.ansible.com/overview/how-ansible-works

Agile Alliance. 2001a. Manifesto for Agile Software Development. Date of retrieval 31.03.2019. https://www.agilealliance.org/agile101/the-agile-manifesto/

Agile Alliance. 2001b. 12 Principles Behind the Agile Manifesto. Date of retrieval 31.03.2019. https://www.agilealliance.org/agile101/12-principles-behind-the-agile-manifesto/

Boehm, B. 2000. Spiral Development: Experience, Principles, and Refinements. Spiral Development Workshop, February 9, 2000. CMU/SEI-2000-SR-008.

Buildbot. 2019a. Buildbot in Action. Date of retrieval 31.03.2019. https://buildbot.net/index.html#basics

Buildbot. 2019b. Introduction. Date of retrieval 02.05.2019. https://docs.buildbot.net/2.0.1/manual/introduction.html

CSQE. Software Quality Blog. Date of retrieval 14.04.2019. https://www.cqse.eu/en/blog/junit3-migration/

Dooley, J. 2011. Software Development and Professional Practice. United States of America: Apress.

G2 crowd. 2019a. Buildbot reviews. Date of retrieval 02.05.2019. https://www.g2.com/products/buildbot/reviews

G2 crowd. 2019b. Jenkins reviews and product details. Date of retrieval 28.04.2019. https://www.g2.com/products/jenkins/reviews#survey-response-447090

Gitlab. 2019a. Gitlab Continuous Integration. Date of retrieval 02.04.2019. https://about.gitlab.com/product/continuous-integration/

Gitlab. 2019b. The first single application for the entire DevOps lifecycle – Gitlab Date of retrieval 02.04.2019. https://about.gitlab.com/

Hamilton, M. 1999. Software Development: Building Reliable Systems. Upper Saddle River, NJ 07458: Prentice Hall

Jenkins. Jenkins User Documentation. Date of retrieval 01.04.2019. https://jenkins.io/doc/#what-is-jenkins

Laster, B. 2017. Continuous Integration vs. Continuous Delivery vs. Continuous Deployment. United States of America. O’Reilly Media, Inc.

Prospector. Prospector - Python Static Analysis. Date of retrieval 23.04.2019. https://prospector.readthedocs.io/en/master/#about

Puppet. What is configuration management. Date of retrieval 07.04.2019. https://puppet.com/solutions/configuration-management

Slides. Ansible Fundamentals. Date of retrieval 14.04.2019. http://slides.com/racku/ansible-fundamentals#/5/14

Slideshare. Continuous Deployment: Beyond Continuous Delivery. Date of retrieval 10.04.2019. https://www.slideshare.net/TimothyFitz/continuous- deployment-beyond-continuous-delivery

Wikipedia. 2019a. Acceptance testing. Date of retrieval 23.04.2019. https://en.wikipedia.org/wiki/Acceptance_testing

Wikipedia. 2019b. Software performance testing. Date of retrieval 14.04.2019. https://en.wikipedia.org/wiki/Software_performance_testing

Wikipedia. 2019c. Waterfall model. Date of retrieval 24.03.2019. https://en.wikipedia.org/wiki/Waterfall_model#/media/File:Waterfall_model.svg