Technical University of Munich Department of Informatics
Bachelor’s Thesis in Informatics
Robust Development with Continuous Integration in TIDOWA
Robuste Entwicklung mit Kontinuierlicher Integration in TIDOWA
Author: Hoang-Trieu Tong Supervisor: Prof. Dr. rer. nat. Martin Schulz Advisor: Dr. rer. nat. Martin Schreiber Submission Date: 15.04.2021
I confirm that this bachelor’s thesis is my own work and I have documented all sources and material used.
Place, Date Hoang-Trieu Tong
Abstract
As software projects grow, the respective source code will become more complex and the number of developers that work simultaneously on the project will increase. Unfortunately, the same applies for the probability of errors occurring during the period of developing such a software. Thus, some countermeasures are required to minimize the probability of major errors and to detect software bugs in the project. Continuous integration (CI) and CI environments represent possible countermeasures. This thesis reports on CI and various implementations of CI applied to TIDOWA, a recently developed software program that is able to compute the approximate solution of initial value problems by means of time integration methods. In addition, a test framework is developed and implemented to verify the validity of TIDOWA’s implementations especially the validity regarding the convergence behavior of the implemented time integration methods. The CI environment in combination with the test framework represents a step towards creating a software development environment that allows better cooperation of developers and improves the quality of the software.
i Contents Contents
1 Motivation 1
2 Ordinary Differential Equations 3 2.1 Introduction ...... 3 2.2 Initial Value Problems ...... 4 2.3 Examples ...... 7 2.3.1 System of Linear Oscillating Equations ...... 7 2.3.2 Model of the Circadian Clock ...... 7 2.4 Numerical Solution of IVPs ...... 8 2.4.1 Introduction ...... 8 2.4.2 Explicit Euler Method ...... 9 2.4.3 Heun Method ...... 9 2.4.4 Runge-Kutta Method of Order 4 ...... 10 2.4.5 Implicit Time Integration Method ...... 12 2.5 Concept of Convergence ...... 12
3 TIDOWA 15
4 Continuous Integration (CI) 17 4.1 Definition and Examination of CI ...... 17 4.2 Common Practices ...... 19 4.3 Overview of CI Implementations ...... 20
5 CI-Oriented Changes and Extensions in TIDOWA 27 5.1 Preliminaries ...... 27 5.2 Restructuring of TIDOWA ...... 27 5.3 Integration of Python Setuptools and Virtual Environment ...... 30 5.4 Test Framework ...... 31 5.5 Further Extensions ...... 31
ii 6 Continuous Integration in TIDOWA 32 6.1 Introduction ...... 32 6.2 Implementation of the Test Framework ...... 32 6.2.1 System Design: Essential Requirements and Design of the Test Framework 32 6.2.2 Implementation: Introduction ...... 38 6.2.3 Implementation: Preliminaries ...... 40 6.2.4 Implementation: Modules - The Core Four ...... 42 6.2.5 Implementation: Connectors - Data Transfer between Modules ...... 47 6.2.6 Implementation: Administrative Structure - Unification into One Program 49 6.3 Integration of CI Environment ...... 54 6.3.1 System Design: Selection and Essential Requirements of the CI Environment 54 6.3.2 Implementation: Preliminaries ...... 55 6.3.3 Implementation: Integration of Test Framework ...... 56
7 Case Study: Usage of Test Framework 59 7.1 Structure of Tests ...... 59 7.2 Interaction with Tests ...... 60 7.3 Test Execution in Local Environment ...... 60
8 Conclusion and Outlook 61
Bibliography 63
iii
1 Motivation
Ordinary differential equations are often used in the field of physics, biology, medicine or even warfare to model complex correlations, e.g. the behavior of a pendulum or the ratio of two species over a certain period of time. What is more interesting than creating a model to certain problems is to find a way to solve them and thus predict events that may occur. One possible approach with which one may solve ordinary differential equations is the numeric approach that applies numerical methods onto these equations to derive an approximation of the solution. As some of the methods were developed by mathematicians who lived back then before the digital age, it is possible to use these procedures manually without any computational assistance. However, using this approach for models and problems that contain hundreds or thousands of equations takes a fair amount of time. Fortunately, with the digital revolution it is possible to hand such tasks over to computers with suitable programs.
A program that is capable of this is TIDOWA. TIDOWA, developed at the Technical University Munich (TUM), provides implementations of different numerical methods that can be used to approximate the solution of ordinary differential equations. The software already offers a variety of numerical methods, nonetheless the project still is in its early stages.
As the program grows and new functionalities are added, it is essential that the parts implemented so far are correct. This is important as future extensions might be based on old functionality. If older parts are erroneous, future developers will encounter lots of trouble and require much effort to fix parts of the software for which they are not responsible for and with which they are not familiar with. As this should not be the fate of every successive developer, it is time to take TIDOWA to the next level and introduce an environment that continuously tests the software and detects incorrect implementations, thus allowing developers to fix them in a timely manner.
This thesis reports on the introduction of an environment to TIDOWA that continuously reviews the state of the software to detect possible errors as early as possible. In this context, it also covers the implementation of a test framework for TIDOWA that allows developers to
1 1 Motivation verify certain attributes concerning the behavior of implemented numerical methods.
Chapter 2 discusses the fundamental concepts of ordinary differential equations that are used in TIDOWA. It introduces the theory and terminology necessary to further develop the software.
Chapter 3 summarizes the history and the current state of the program. It also depicts the way how TIDOWA works and the interaction of the individual modules and stages within the software.
Chapter 4 examines the subject of continuous integration from a theoretical standpoint. Additionally the chapter presents and compares various existing implementations of continu- ous integration environments.
Chapter 5 presents the changes that were introduced to the software to integrate it into a continuous integration environment and to simplify its development for future developers.
Chapter 6 discusses the two components that were newly introduced with the work of this thesis, them being the continuous integration environment and the test framework. First, the theory behind both is examined, which includes the selection process of the continuous integration environment and the design of the test framework. Afterwards, the details con- cerning both implementations are described.
Chapter 7 describes the usage of the test framework. The chapter displays how tests are structured, added, removed and further executed inside a local environment.
2 2 Ordinary Differential Equations
This chapter discusses the fundamentals regarding ordinary differential equations (ODEs), initial value problems (IVPs), time integration (TI) methods and convergence. It briefly introduces important concepts and terms that are included in TIDOWA. The following explanations and descriptions in Chapter 2.1 are based on [1, 2, 3, 4].
2.1 Introduction
Ordinary differential equations (ODEs) are equations that connect:
• y(t), an unknown function with t being a variable, often used to identify the varying aspect time and • derivatives of this unknown function that are displayed with e.g. y0(t), y0, y(1)(t) or y(1) for the first derivative of y(t).
In this context, y is referred to as dependent variable and t as independent variable.
There are two important terms regarding ODEs: order and degree. The order of an ODE is equal to the “order of the highest derivative that appears in the”[1] ODE. Furthermore, the degree of an ODE is identical to the exponent of the highest derivative. If we consider the following exemplary ODE, it has an order of 4 and a degree of 1.
y0(t)6 + 5y00(t)2 − 3y000(t)5 − 2y0000(t) = 0
One way of presenting ODEs is the so called standard form or normal form of the ODE. The standard form or normal form of a general ODE of order n is as follows:
y(n)(t) = f(t, y(t), y(2)(t), ..., y(n−1)(t)) where f depicts a given functional relationship of y and t.
The function s(t) is a solution of the ODE, defined in the interval I ⊆ R, if the follow- ing conditions apply:
3 2 Ordinary Differential Equations
• the function s(t) is differentiable in I • functional identity: If y(t) and its derivatives y(i)(t) can be replaced by s(t) and its derivatives s(i)(t), with ∀t ∈ I, so that the equation is still satisfied, it is called functional identity
We consider the following ODE as an example:
0 + y (t) = 4y(t) t ∈ R
The solution of this example is the function s(t) = e4t. When y(t) and y0(t) are replaced with s(t) and s0(t) = 4e4t, the equation is still satisfied with: 4e4t = 4e4t. Thus, the functional + identity is fulfilled. In addition, s(t) is differentiable in R . It should be noted that in general an ODE has not only one but infinite solutions. The whole set of solutions is referred to as general solution or complete solution.
In general we deal with systems of ODEs that consist of multiple ordinary differential equa- tions rather than individual equations. Examples for such systems are examined in Chapter 2.3.
For the sake of completeness it should be mentioned that other types of differential equations such as partial differential equations exist as well. In contrast, partial differential equations contain multiple independent variables. Note that a further discussion of this type of differ- ential equation would go beyond the scope of this thesis as a different numerical approach, not implemented in TIDOWA, is required. For interested readers the given references so far provide further information on this topic.
2.2 Initial Value Problems
This chapter focuses on one type of problem regarding ODEs called initial value problems. It is based on [1, 2, 3, 4, 5, 6].
In most cases one is not interested in the complete set of solutions of an ODE but rather a specific solution that fulfills certain conditions expressed through specific additional informa- tion. These conditions limit the number of possible solutions to the ones of interest. Initial
4 2.2 Initial Value Problems value problems describe one approach on how to use such constraints to obtain the specific solution of an ODE.
An initial value problem (IVP) consists of:
• an ODE or a system of ODEs as defined in Chapter 2.1 • initial conditions that limit the set of solutions to only one possible solution. They are
described with the use of y0 and t0 in the case of a single ODE. For both the following
relation applies: y(t0) = y0.
An example for an IVP is given below:
0 2 + y (t) = 4y(t) t ∈ Z y0 = y(0) = 1
For the sake of completeness it should be mentioned that there is a similar type of problem called boundary value problem that uses multiple conditions y(ti) = yi to determine the solution to an ODE given.
As already mentioned, initial conditions limit the number of possible solutions. In the following, two ways of solving IVPs are shown. To find the solution to an IVP means that first the general solution to the ODE is needed. This set of solutions is then filtered until the ones that comply with the initial conditions given remain. According to Chapter 2.1, a function that fulfills the condition of functional identity and that is differentiable solves the ODE. The solution may be found with the help of antiderivatives as shown below with an example. The following problem is given:
0 2 y (t) = 4t y0 = y(0) = 1 Z Z 4t3 ⇒ y(t) = y0(t) = 4t2 = + C 3
As C can represent any constant, y(t) is the general solution of the ODE given. To obtain the solution to the IVP, the initial condition y0 = y(0) = 1 is inserted: 4(0)3 1 = + C ↔ C = 1 3 The solution of the IVP given is: 4t3 4t3 + 3 y(t) = + 1 = 3 3 5 2 Ordinary Differential Equations
Another approach to solve IVPs is referred to as separation of variables and is described in the following with an example broken down into the main steps. Again an IVP is given as follows: 0 2 + y (t) = 4y(t) t ∈ Z y0 = y(0) = 1 1. Transformation of the equation into the form: dy = 4y2 dt 2. Rearrangement of the variables to separate them: dy = 4dt y2 3. Integration on both sides of the equation: Z y 1 Z t 2 dγ = 4dτ y0 γ t0 4. Determination of the solution of the integrals: 1 1 − + = 4t − 4t0 y y0 5. Solving for y and retransformation, yielding:
1 1 1 4y0(t − t0) − 1 −y0 − = 4(t − t0) − ⇔ − = ⇔ y = y y0 y y0 4y0(t − t0) − 1 −y ⇒ y(t) = 0 4y0(t − t0) − 1 Here, y(t) again denotes the general solution. By inserting the initial condition, the solution of this IVP is: −1 −1 y(t) = = 4(t − 0) − 1 4t − 1 As may be seen for the two examples above, determining the solution of simple IVPs is work-intensive and requires complex concepts. One has to imagine the effort needed to find solutions for complex systems of ODEs that may consist of a fair amount of equations in the order of hundreds or thousands or even larger. Even with the use of computers the task of identifying the solution by means of the approaches described above still remains difficult. Therefore, alternatives are discussed later in Chapter 2.4.
6 2.3 Examples
2.3 Examples
This chapter presents two systems of ordinary differential equations. Chapter 2.3.1 is based on the knowledge, acquired through the lecture Numerical Programming in [4], discussions with the advisor prior to this thesis and [5, 7]. Chapter 2.3.2 is based on [8, 9].
2.3.1 System of Linear Oscillating Equations
The first system of ODEs introduced here, is the system of linear oscillating equations that is defined as follows: 0 u (t) = −c1v(t) 0 v (t) = c2u(t) u(t) and v(t) and are unknown functions. Thus, u0(t) and v0(t) are the derivatives of these functions. c1 and c2 are real valued parameters that can be chosen arbitrarily. This system is a prominent example used especially in physics to describe several experiments, e.g. the movement of an object that is attached to a spring, known as harmonic oscillator or the behavior of the oscillating pendulum.
A system of ODEs may also be portrayed using matrices and vectors. Thus, the system of linear oscillating equations also has the following mathematical display: 0 −c u(t) 1 L U~ = c2 0 v(t)
By means of matrix multiplication, the above can be derived.
2.3.2 Model of the Circadian Clock
The next system of ordinary differential equations deals with a subject from biology known as circadian clock or more commonly the biological clock. To be more precisely, the system of ODEs deals with the biological clock of the plant Arabidopsis thaliana. Similar to humans or other living beings, this plant possesses a system named circadian system that is responsible for creating biological rhythms and thus, concerning the plant, is in charge of important processes like leaf movement or photosynthesis. It is assumed that so called feedback loops, derived by “interactions of rhythmically expressed genes”[8], trigger such biological rhythms.
7 2 Ordinary Differential Equations
The system of ODEs that tries to model such feedback loops consists of 13 equations. A detailed description of this model would exceed the scope of this thesis. Interested readers may consult [8, 9] for additional information. More importantly, this example again highlights why the approaches that were discussed so far do not always lead to a solution of reasonable workload.
2.4 Numerical Solution of IVPs
In Chapter 2.2 two basic methods of solving simple ODEs and IVPs were presented. The conclusion of that chapter was that different, less work-intensive and complex approaches are needed to solve ODEs and IVPs. There are several numerical approaches that allow one to determine the solution in a less demanding manner compared to the analytical approach. However, the usage of these approaches might raise the disadvantage that precision is lost up to a certain degree due to the fact that these only determine approximations of the solution in contrast to the two methods discussed before that compute precise analytical solutions. To what extent this might become a problem will be examined in Chapter 2.5. This chapter is based on [2, 4, 5, 6].
2.4.1 Introduction
The following sections will introduce some well known approaches to solve IVPs numerically, altogether, called time integrator (TI) methods. These TI methods will only be covered briefly as they have been examined before by former developers and in addition are already integrated in TIDOWA. Throughout the chapter, the following IVP will be used as an example to demonstrate the different TI methods:
0 y (t) = 0.25y(t) t ∈ R ≥ 0 y0 = y(0) = 1
By means of the separation the variables, the specific solution of this IVP is as follows:
y(t) = e0.25t
The general task of the following TI methods is to determine the approximation y(ti+1) of the general first-order IVP’s solution at the point ti+1. The IVP is given as follows:
8 2.4 Numerical Solution of IVPs
0 y (t) = f(t, y(t)) y(t0) = y0 t ∈ I ⊆ R
The basic concept of the following methods is that in order to determine an approximation y(ti+1), previous approximations y(ti), y(ti−1), ..., y(t0) have to be identified recursively. These are then used along with the approximations of their derivatives obtained through the
IVP given to compute the estimation y(ti+1). It is comparable to a recursive function, which starts at the initial condition y(t0) and consecutively computes the next steps until y(ti+1). The difference between the individual methods presented in the following is the extent to what the previous approximations and their derivatives are used.
2.4.2 Explicit Euler Method
The explicit Euler method determines the approximation y(ti+1) through the following relations: y(ti+1) = y(ti) + δt f(ti, y(ti))
ti = t0 + i δt δt = ti+1 − ti
The method uses the approximation y(ti), or if i = 0 the initial condition, along with the 0 estimation of its derivative y (ti) = f(ti, y(ti)) and δt to compute the approximation y(ti+1). In general, δt, often referred to as step size, displays the invariable difference between two consecutive computational steps ti and ti+1. Hence, ti is the sum of the starting point t0 and δt multiplied with the factor i. As an example the first computational step of this method, y(t1) ≈ y(1), for the IVP given and a step size of 1 is:
y(t1) = y(t0) + δt f(t0, y(t0)) = 1 + 1 × (0.25 × 1) = 1.25
Analogously, the consecutive approximations are computed.
2.4.3 Heun Method
Considering the general first-order IVP, the Heun method defines the approximation y(ti+1) as follows: δt y(t ) = y(t ) + ( f(t , y(t )) + f(t , y(t ) + δt f(t , y(t ))) ) i+1 i 2 i i i+1 i i i
ti = t0 + i δt δt = ti+1 − ti
9 2 Ordinary Differential Equations
The variables δt, ti, f(ti, y(ti)) and y(ti) are defined as in Chapter 2.4.2. In contrast to the 0 explicit Euler method that only uses the estimation of y (ti) = f(ti, y(ti)), the Heun method 0 0 uses the average of two estimations: y (ti) = f(ti, y(ti)) and y (ti+1) = f(ti+1, y(ti+1)). As y(ti+1) is the desired variable, the second estimation might become problematic. To avoid this problem the explicit Euler method is used to determine y(ti+1) on the right side of the relation above, thus it may be replaced by y(ti) + δt f(ti, y(ti)). The approximation of y(ti+1) on the left side of the relation may then be computed using the Heun method. In this context, the step in which y(ti+1) is obtained by means of the explicit Euler method, is denoted as predictor. The computation of y(ti+1) on the left side of the equation is referred to as corrector. Again, the first computational step of the Heun method for the IVP given and a step size of 1 is shown as an example: 1 y(t ) = y(t ) + ( f(t , y(t )) + f(t , y(t ) + f(t , y(t ))) ) 1 0 2 0 0 1 0 0 0 1 = 1 + ((0.25 × 1) + (0.25 × 1.25)) = 1.28125 2 Analogously, the consecutive approximations are computed.
2.4.4 Runge-Kutta Method of Order 4
The Runge-Kutta method of order 4 (RK4) describes one of the most common approaches to numerically solve an IVP. Given the general first-order IVP, RK4 is defined with the following set of equations: δt y(t ) = y(t ) + ( R + 2R + 2R + R ) i+1 i 6 1 2 3 4 δt δt R = f(t , y(t )) R = f(t + , y(t ) + R ) 1 i i 2 i 2 i 2 1 δt δt R = f(t + , y(t ) + R ) R = f(t + δt, y(t ) + δtR ) 3 i 2 i 2 2 4 i i 3 ti = t0 + i δt δt = ti+1 − ti
The variables δt, ti, f(ti, y(ti)) and y(ti) are defined as in Chapter 2.4.2. This method is similar to the Heun method as both use multiple estimations but RK4 considers four weighted estimations R1, R2, R3, R4 to determine the approximation y(ti+1) instead of just two. Exemplary, the computation of the first step of RK4 for the IVP given and a step size of 1 is shown: δt y(t ) = y(t ) + ( R + 2R + 2R + R ) 1 0 6 1 2 3 4 1 1 R = f(t , y(t )) = × 1 = 1 0 0 4 4
10 2.4 Numerical Solution of IVPs
δt δt 1 1 1 9 R = f(t + , y(t ) + R ) = × (1 + × ) = 2 0 2 0 2 1 4 2 4 32 δt δt 1 1 9 73 R = f(t + , y(t ) + R ) = × (1 + × ) = 3 0 2 0 2 2 4 2 32 256 1 73 329 R = f(t + δt, y(t ) + δtR ) = × (1 + ) = 4 0 0 3 4 256 1024 1 1 9 73 329 y(t ) = 1 + ( + 2 × + 2 × + ) = 1.2840169 1 6 4 32 32 1024
3.0
2.5
2.0
1.5 y ( t)
1.0
y(t) = e0.25t 0.5 Explicit Euler approximations Heun Method approximations RK4 approximations
0.0 0 1 2 3 4 t
Figure 2.1: Plot of the function y(t) = e0.25t (orange curve) that represents the analytical solution along with the explicit Euler method approximations of y(t) (pink circles), the Heun approximations of y(t) (blue hexagons) and the RK4 approximations of y(t) (wine red diamonds).
Now let us further examine Figure 2.1 that contains the analytical solution of the IVP along with the TI methods’ approximations of the solution. Considering the explicit Euler approxi- mations, for the first two steps they agree well with the analytical solution, but with increasing t the discrepancy between analytical solution and explicit Euler approximation becomes quite noticeable. It is different for the Heun approximations. Each Heun approximation is closer to the analytical solution than the respective explicit Euler approximation. But similar to the explicit Euler approximation, the accuracy of the Heun approximation deteriorates with increasing t. The RK4 approximations, however, agree well with the analytical solution even with increasing t.
11 2 Ordinary Differential Equations
2.4.5 Implicit Time Integration Method
All methods presented in the previous chapters have in common that they are referred to as explicit TI methods. This is due to the fact that it is possible to derive the desired
approximation y(ti+1) without any rearrangement of the equation. TI methods that require such rearrangements are called implicit TI methods. Implicit TI methods can be identified
easily as y(ti+1) appears on both sides of the equation, which makes it necessary to rearrange the equation prior to determining the approximate solution of an IVP. One example for an implicit time integration method is the implicit Euler method. Considering a general first-order IVP, the implicit Euler method is defined as follows:
y(ti+1) = y(ti) + δt f(ti+1, y(ti+1))
ti = t0 + i δt δt = ti+1 − ti Compared to the previous explicit methods, the implicit method is more complicated as an additional step is necessary to obtain the approximation of the IVP’s solution. That step is the rearrangement of the equation. However, implicit TI methods still remain valuable to numerically solve IVPs. The reason to this will be examined in the next chapter.
2.5 Concept of Convergence
This following chapter discusses important theoretical terms and concepts that are part of the test framework. This chapter is based on [2, 4, 5, 6]. As concluded in Chapter 2.4.4, the individual TI methods yield different results for the approximation of the IVP’s solution, thus leading to different discrepancies between analytical solution and TI method approximation. There are two expressions that describe the difference between the analytical solution of the IVP and its approximation. The first one is called local discretization error and represents the discrepancy between analytical solution and its estimation that emerges in one step. The second term is the global discretization error that displays the entire discrepancy between analytical solution and estimation throughout all steps.
Both of these errors show two important properties of individual TI methods. These are consistency and convergence. The term consistency describes the phenomenon that the local
12 2.5 Concept of Convergence discretization error approaches zero when δt tends to zero. Similarly, convergence holds, if the global discretization error approaches zero, when δt is tends to zero. It holds that “convergence implies consistency”[5]. The reversed only holds, if another criteria is met, namely stability. Stability describes the behavior of a method’s result when inaccurate input data is given, more precisely the discrepancy between a method’s solution using more accurate and less accurate input data.
The importance of implicit TI methods had been mentioned in Chapter 2.4.5. The rea- son for its importance may be explained with the term stiffness. ODEs are denoted as stiff when convergence or consistency, respectively, only apply to the ODE, if δt is infinitesimally small. In this cases, the usage of implicit methods avoids this problem.
Another circumstance concerning the local and global discretization error as well as the step size δt, which appears for different TI methods, is displayed in Table 2.1.
t 0 1 2 3 4
EE approximation with δt = 1 1 1.250000 1.562500 1.953125 2.441406
global disc. error 0 0.034025 0.086221 0.163875 0.276876
EE approximation with δt = 0.5 1 1.265625 1.601807 2.027287 2.565785
global disc. error 0 0.018400 0.046914 0.089713 0.152497
EE approximation with δt = 0.25 1 1.274429 1.624170 2.069890 2.637928
global disc. error 0 0.009596 0.024551 0.047110 0.080354
Table 2.1: Explicit Euler (EE) approximations of the IVP given for values t ∈ {0, 1, 2, 3, 4} with various step sizes δt and respective global discretization errors. Note that for t = 0 the explicit Euler approximations for each δt are equal to 1 as y(0) = 1 is the initial condition for each explicit Euler approximation. Thus, the difference to the analytical solution is 0.
Comparing the values contained in the table, it is clear that there is a correlation between the choice of δt and the value of the global discretization error, thus the accuracy of the result. The smaller δt, the more accurate the approximation is, i.e. the lower the discretization error
13 2 Ordinary Differential Equations is. This relation is referred to as order p. The order of a TI method determines to what extent a decrease of δt has a positive impact towards the asymptotic behavior of the estimation. That means increasing the accuracy of the approximation and therefore decreasing the error. It holds that the order of a TI method scales exponentially with the error, meaning that if a TI method has the order p, the error is of order O(δt p). Exemplary for the explicit Euler method, which has an order of 1, the effect of dividing δt in half will reduce the error 1 by a factor of almost 2 . Continuing with even smaller δt, this reduction will further occur 1 with factors closer or equal to 2 . This effect can be seen in Table 2.1. Note that there are two types of orders, namely the order of the local discretization error and the order of the global discretization error, however in this thesis and in the test framework only the latter is considered. The following Table 2.2 summarizes the order of all discussed TI methods.
TI method explicit Euler method Heun method RK4 implicit Euler method order 1 2 4 1
Table 2.2: Summary of the TI methods’ orders.
14 3 TIDOWA
After Chapter 2 provided an overview of the necessary fundamentals regarding ordinary differ- ential equations, time integration methods and convergence, Chapter 3 introduces TIDOWA, which stands for “Time Integration DOmain-specific Wicked Awesome stuff”[10]. This chapter is based on previous projects and research done by Severin Bals, Anna Mittermair and Taylor Lei. Their work gives an insight to the parts of this program that are essential for this thesis. Further information on the project may be found in [10, 11].
TIDOWA is a software program that is able to calculate the solution of an IVP or to be more accurate an approximation of this solution. The program originally used the pro- gramming languages Python and C. Further extensions include the support of other languages and frameworks such as CUDA, OpenCL and OpenMP as well as an optimizer. The program itself was first developed by Severin Bals in 2019. Anna Mittermair enhanced it with several programming languages and frameworks. The latest addition was the optimizer implemented by Taylor Lei in 2020. The entire development took place under the supervision and support of Dr. rer. nat. Martin Schreiber. This thesis will concentrate on TIDOWA itself excluding the optimizer. In the following, the current state of the program is described.
The program receives a specific ODE configuration with parameters and initial values along with a chosen time integration method as input and returns an approximation of the solution of the IVP, calculated by using the time integration method given. The process of TIDOWA can be distinguished into three phases, namely Discretization stage, Code generation stage and Execution stage. Figure 3.1 visualizes the general program flow of TIDOWA, which is described in the following.
15 3 TIDOWA
discretized source ODE code input 3. Execution stage
ODE 2. Code generation 1. Discretization stage compilation configuration stage
parameters program & execution initial conditions
Figure 3.1: Simplified flow diagram displaying the program flow of TIDOWA in three stages. The blue rounded rectangles represent the stages, the yellow components represent the input or the preliminary results, respectively.
The Discretization stage is the first of the three stages. It transforms the input, namely the ODE configuration into a discrete version of itself with regard to the chosen time integration method. For this stage, TIDOWA already contains a selection of time integration methods implemented, more specifically the ones that were discussed in Chapter 2.4. Additionally, the program offers pre-implemented ODEs to test.
Next, the Code generation stage uses the discrete version of the ODE to derive a source code that is used to calculate the approximation of the ODE’s solution. For this stage, TIDOWA supports the previously mentioned programming languages and frameworks.
Lastly, the Execution stage generates an executable program that uses the parameters and the initial values to calculate an approximation of the solution of the ODE given the specific input.
Note that in the specific implementation, TIDOWA originally uses ODES and transforms them into IVPs using additional descriptions provided by a file containing the parameters and initial conditions. This transformation process takes place during the program execution, thus the parameters and initial conditions point towards the Execution stage in Figure 3.1.
16 4 Continuous Integration (CI)
4.1 Definition and Examination of CI
Within this chapter, the term Continuous Integration (CI) is introduced and defined. In addition to that, the requirements and benefits of CI are examined, the importance of CI is stressed and the term itself is distinguished from other similar terms and concepts. This chapter is based on [12, 13, 14].
In the following, some definitions of continuous integration are listed to provide a better understanding and a common basis of knowledge. Paul Duvall et al. for example think that continuous integration is “the centerpiece of software development, as it ensures the health of software”[13] and deals with “the fundamental activities in software development”[13]. Brent Laster takes another approach and divides the process of developing a software into individual stages or rather processes. In this structure the stage concerning CI is referred to as “phase, [in which] changes from a developer are merged and validated”[14]. A definition to which Paul Duvall et al. also refers to is the one from Martin Fowler. Martin Fowler interprets CI to be a “software development practice where members of a team integrate their work frequently”[12] and where the respective state of the work then “is verified by an automated build (including test) to detect integration errors”[12].
To unify all three definitions this thesis will try to capture the essence of these definitions. All definitions state that CI is some kind of process that
(a) is related to software development and its concepts, (b) is used to check additions to the existing source code, thus improving its quality and (c) is something that is initiated by the developer and therefore inevitably requires human involvement.
Having established a common basis of knowledge, the requirements for CI are discussed in the following. Paul Duvall et al. identify four requirements for CI to which they refer to as features of CI. These four necessities are a version control system (VCS) where the source
17 4 Continuous Integration (CI)
code is stored, a build script that contains tests and further instructions that have to be passed and executed successfully before the state of the software can be labeled as correct, a CI server, also called CI tool that executes such script and a feedback mechanism that notifies the developer about the outcome of this execution. In this context one also refers to running an integration build or in general build when talking about this process of verifying the state of the software prior to labelling it as correct.
As already stated in the beginning of this thesis, in the case of TIDOWA, CI helps de- tecting errors and an erroneous source code early. This main benefit is also mentioned by both Duvall et al. and Fowler as both refer to it as reduced risks. Similar to the case of TIDOWA the general usage of CI helps to detect erroneous source code and gives one the possibility of correct- ing them as early as possible. In addition to that, Duvall et al. name some additional benefits like “better project visibility”[13] or “deployable software at any time and at any place”[13]. For more information on the extensive impact of CI, this thesis suggests reading Duvall et al., as they are more experienced with large software projects. They also describe a large variety of concrete use cases, in which CI is used to solve problems that occur during software projects.
Finally, this chapter stresses the need to distinguish between multiple terms that are quite similar, them being CI, continuous deployment and continuous delivery. Again, the approach of Brent Laster is used. As mentioned before, he separates the process of developing a software into individual stages or processes, respectively. Laster refers to this whole process as continuous delivery. Continuous integration represents an individual stage of continuous delivery. Furthermore, Laster states that continuous deployment also is an individual stage and describes it as “the process that makes the end product available to users”[14], comparable to the final meters of a software development process.
The next chapter discusses common practices that should be put into practice when ap- plying continuous integration.
18 4.2 Common Practices
4.2 Common Practices
Both, Fowler and Duvall et al. describe various practices that come along with the usage of continuous integration and that should be applied. This chapter summarizes the most important practices and additionally briefly discusses how TIDOWA can apply them or rather provides a preview on how the practices are already implemented in TIDOWA. This chapter is based on [12, 13].
One of Fowler’s practices describes the usage of a version control system (VCS), show- ing again the variety of definitions as this is set as a requirement by Duvall et al.. In addition to the usage of a VCS, Fowler demands a clear structure and documentation of the source code along with the provision of every crucial component, necessary to run a build. TIDOWA uses git as VCS in combination with GitHub (prior to that GitLab), which hosts the repository. With the reorganization of TIDOWA’s directory structure a clear and comprehensible structure was introduced. The last aspect of his demand is realized with the usage of scripts that handle all steps required, i.e. the installation of third-party modules or the setup of environment variables. This allows the use of the program after the exe- cution of the scripts. The realization of this practice is further described in Chapter 5.5 and 5.3.
Several points mentioned by Fowler such as “Automate the Build”[12] or “Test in a Clone of the Production Environment”[12] are combined in this thesis as a simplification of the usage for developers. Fowler stresses the requirement of complete automated builds, i.e. scripts to simplify the creation of builds and thus the usage of the software. Furthermore, Fowler states “Test in a Clone of the Production Environment”[12] to demand equivalent execution environments. This means that the builds that verify the source code are executed under the same circumstances, under which the source code was written, e.g. the usage of the exact same version of the programming language or the exact same version of a third-party component. These practices are already implemented in TIDOWA as it uses scripts that set up the execution environment, install specific versions of third-party dependencies and in addition are easy to use.
The last aspect examined unites two practices demanded, with them being “Make Your
19 4 Continuous Integration (CI)
Build Self-Testing”[12] and “Keep the Build Fast”[12]. The latter insists on low execution times for tests. Concerning the former, according to Fowler, self-testing means that a build should fail if not all tests pass. In the context of TIDOWA the latter is considered by supporting parallelism of the test execution (see Chapter 6.2.1 and 6.3.1) and the former is achieved by letting the build fail when tests do not pass. This is considered in the imple- mentation of the test framework but not discussed in detail in this thesis. The majority of the remaining practices, e.g. “Everyone Commits To the Mainline Every Day”[12], “Don’t commit broken code”[13] or “Fix broken builds immediately”[13] include, as stated before, a human involvement and can not be forced by a CI implementation.
Up to now, this thesis only considered theoretical aspects of continuous integration. With the following chapter, we will examine the concrete implementation of several CI environments from which one will be selected for TIDOWA’s CI implementation.
4.3 Overview of CI Implementations
This chapter presents a selection of common CI environments. It will examine these imple- mentations and summarize the main aspects. Additionally, this chapter will be used as a basis for the selection process of the CI environment for TIDOWA. Note that the sources are stated for each CI environment and interested readers can refer to them for more in-depth information. The selected criteria for the examination are:
• Sources: relevant sources from where the information is derived • Version: current version of the CI environment, if stated • Developer: current developer of the CI environment • Host: host possibilities of CI tool, (e.g. self-hosted or hosted by provider) • VCS compatibility: version control systems (e.g. Git, Subversion, Perforce) or hosts of version control systems (e.g. GitHub, GitLab, Bitbucket) that are compatible with the respective CI environment • OS compatibility: operating systems that are compatible with the respective CI environment (only for self-hosted CI tools as CI environments that are not self-hosted, generally do not have any specific demands regarding the operating system)
20 4.3 Overview of CI Implementations
• Supported programming languages and frameworks: list of programming lan- guages and frameworks that are supported by the respective CI environment • Software requirements: software that is required for the respective CI environment (only for self-hosted CI tools as CI environments that are not self-hosted generally only require a registration on the website) • Hardware requirements: required hardware specifications for the respective CI environment (only for self-hosted CI tools as CI environments that are not self-hosted, in general do not have any specific demands regarding the hardware) • Pricing: costs resulting from the usage of the service. If not stated otherwise, these are specified as monthly fees given in USD
Bitbucket Pipelines: Bitbucket Pipelines is a CI tool that is integrated within Atlassian Bitbucket, thus is only offered for Bitbucket. It makes use of a YAML file to configure the program flow of the tests. The file can be seen as a build script with a specific description language which is used by various CI environments. Similar to other tools offered by Atlassian, Bitbucket Pipelines provides the possibility of including other tools like Docker or Slack into the program flow. As Bitbucket Pipelines is integrated within Bitbucket, the monthly fee for the usage of Bitbucket is considered to be the price of using the CI tool. Atlassian offers three pricing options for using Bitbucket. The first is named Free. As the name suggests the usage is free of costs. Thus limitations apply as builds are only allowed to run for 50 minutes in total per month and the number of users is restricted to five. The other options, named Standard and Premium are charged monthly with 3$ or 6$ per user on average apart from discounts, respectively. Both do not restrict the number of users and in addition offer monthly up to 2500 respectively 3500 minutes for the builds to run. Note that both pricing options also have further benefits not related to CI. Table 4.1 contains additional information to this CI environment. This section is based on [15, 16, 17, 18].
Travis CI: Travis CI offers a variety of functionality for developers. It is compatible with several common VCS hosts like GitHub or Bitbucket and does not require any additional registration (for the service itself; a registered account at the VCS host is still necessary). The usage of Travis CI has the benefit that several execution environments may be used. Next to
21 4 Continuous Integration (CI)
Linux, Travis CI also provides several distributions of Windows and macOS. In addition to that, if Linux is the system of choice, one is able to select from multiple CPU architectures offered. Similar to Bitbucket Pipelines, YAML files that are saved in the repository are used to configure the program flow of the tests. Travis CI publicly offers five pricing options. The first two, Open Source and Free Plan, are free of charge. The former is only available for open source projects, the latter only allows temporary usage. The remaining three options only differ in the aspect of parallelism, i.e. how many jobs can run at the same time. Table 4.1 contains further information to the pricing and the CI environment in general. This section is based on [19, 20, 21, 22, 23].
CI Bitbucket Pipelines Travis CI environment
Sources [15, 16, 17, 18] [19, 20, 21, 22, 23]
Developer Atlassian Travis CI GmbH
Host CI environment that CI environment that is hosted by Atlassian is hosted by Travis CI GmbH
VCS • Bitbucket • GitHub compatibility • Bitbucket • GitLab • Assembla
Supported Languages that may be 30 different languages including programming integrated into Linux by the Python, Java, C and Haskell languages and means of Docker images amongst others frameworks
• Free: 0$ • Open Source: 0$ Pricing • Standard: ≤3$ per user • Free Plan: 0$ • Premium: ≤6$ per user • 1 concurrent job Plan: 69$ • 2 concurrent job Plan: 129$ • 5 concurrent job Plan: 249$
Table 4.1: Overview of the CI environments Bitbucket Pipelines and Travis CI structured by means of the list of criteria.
22 4.3 Overview of CI Implementations
AppVeyor: AppVeyor, a CI tool hosted by AppVeyor Systems Inc., is similar to Travis CI as both do not require an additional registration. Furthermore, AppVeyor is compatible with a variety of VCS hosts, even more than Travis CI. AppVeyor also offers multiple execution environments like Windows, macOS or Linux. Alike the two previously mentioned CI tools, AppVeyor also uses YAML. Further, the CI environment also provides an user interface to configure builds. There are four pricing options from which one is reserved for open source purposes. The other three essentially only differ in the aspect of parallelism. Table 4.2 contains additional information to this CI environment. This section is based on [24, 25, 26, 27, 28, 29].
Buddy: Similar to the other CI tools, Buddy is a CI environment that is compatible with the common VCS hosts. Buddy supports over 20 languages and frameworks, like Python, Haskell or Maven and like the previous CI tools uses YAML. Uniquely from the first two CI tools, Buddy also offers a graphical user interface to configure the builds. Similar to the previously mentioned CI environments, the monthly costs are determined by the number of projects, the availability of parallelism and the minutes that are required to run the builds. Table 4.2 contains further information to the CI environment in general. This section is based on [30, 31, 32, 33, 34, 35].
CI AppVeyor Buddy environment
Sources [24, 25, 26, 27, 28, 29] [30, 31, 32, 33, 34, 35]
Developer AppVeyor Systems Inc. Buddy
Host CI environment that is hosted CI environment that is hosted by AppVeyor Systems Inc. by Buddy
VCS • GitHub • GitHub compatibility • GitLab • GitLab • Bitbucket • Bitbucket • Azure Devops • Buddy Git Hosting • in general: Git, Mercurial • own Git repositories and Subversion source code repositories
23 4 Continuous Integration (CI)
Supported Explicit support for Python, programming ≥ 20 different programming Java, Ruby, C amongst others; languages and languages and frameworks Also support for Docker frameworks
Pricing • Open Source: 0$ • Free: 0$ • Basic: 29$ • Pro: 75$ • Pro: 59$ • Hyper: ≥99$ • Premium: 99$
Table 4.2: Overview the CI environments AppVeyor and Buddy structured by means of the list of criteria.
Jenkins: Jenkins is the first discussed self-hosted CI environment here. As Jenkins is an open source project that is maintained by the Jenkins Project, there are no costs for the usage of this CI environment. Jenkins is compatible with Windows, Linux and macOS and requires either a Java Runtime Environment (JRE) or a Java Development Kit (JDK) to run. Additionally, the usage of Docker is recommended. As it is a self-hosted CI environment, Jenkins supports the languages and frameworks that are supported by the host. If Docker is used, compatible Docker images can also be used. If interested, further information is provided in Table 4.3. This section is based on [36, 37, 38, 39, 40, 41, 42].
Bamboo: Bamboo is an implementation of a self-hosted CI environment offered by At- lassian. The implementation is fully compatible with the operating systems Windows and Linux, whereas macOS and Solaris are not fully supported as Atlassian plans to discontinue the support for both. Furthermore, Bamboo is compatible with the VCSs Git and Subversion. Depending on one’s own requirements, there is a choice between two pricing options: small teams and larger teams. The former allows the unrestricted usage of local agents that are “service[s] that provide[...] capabilities to run job builds”[43] locally. The latter permits the usage of remote agents that run remotely from the server additionally. Depending on the amount of desired remote agents (between 1 and 1000 agents), the price varies. Note that the fee is charged once, but this only accounts for 12 months of support. Further information on Bamboo may be found in Table 4.3. This section is based on [43, 44, 45, 46, 47, 48, 49].
24 4.3 Overview of CI Implementations
TeamCity: Lastly, the CI environment TeamCity is discussed. This self-hosted CI tool is offered by JetBrains and is compatible to several VCS hosts, such as GitHub or GitLab. It is compatible with the common operating systems and similar to the previous two, requires Java to run. Analogously to Bamboo the prices are charged only once with the following 12 months being the time period in which one remains eligible to system updates, and are dependent on the number of agents (between 3 and 100 agents) amongst others. Table 4.3 provides additional information. This section is based on [50, 51, 52, 53].
CI Jenkins Bamboo TeamCity environment
[36, 37, 38, 39, 40, 41, [43, 44, 45, 46, 47, 48, Sources [50, 51, 52, 53] 42] 49]
Version 2.277.1 LTS 7.2 2020.2
Developer Jenkins Project Atlassian JetBrains
Host Self-hosted CI Self-hosted CI Self-hosted CI environment environment environment
VCS Not stated explicitly • Git • GitHub compatibility but at least Git as it • Subversion • GitLab was used for this • Bitbucket thesis project • Azure Devops • Git, Subversion, Perforce, Mercurial
OS • Windows • Windows • Windows compatibility • Linux • Linux • Linux • macOS • macOS (partial) • macOS • Solaris (partial)
25 4 Continuous Integration (CI)
Supported Languages and Support of all • Java programming frameworks, languages • .NET languages and supported by the • Python frameworks host machine or • Rake Docker images • Shell scripts
Software • JRE/JDK • JDK • JRE requirements • Docker
Hardware • CPU: - • CPU: 4 - 16 cores • CPU: 4 cores requirements • RAM: 256 MB • RAM: 4 - 16 GB recommended minimum • Storage: ∼20 GB • RAM: 4 GB • Storage: 1GB recommended • Storage: - minimum
Pricing Open Source: 0$ (one-time payment) (one-time payment) • Small teams: 10$ • Professional • Larger teams: • server license: 0€ 1,500$ - 167,300$ • Enterprise server license: 1,999€ - 21,999€
Table 4.3: Overview of the CI environments Jenkins, Bamboo and TeamCity structured by means of the list of criteria.
26 5 CI-Oriented Changes and Extensions in TIDOWA
5.1 Preliminaries
The changes and extensions introduced to TIDOWA were made for various reasons. First, the adjustments aim at simplifying the usage and development of the program for end users and successive developers. Therefore, a common basis is desirable. Further, common practices of software engineering will be established, e.g. the separation of source code and execution. Consequently, the development of the test framework becomes more robust. In the following sections the modifications are presented.
5.2 Restructuring of TIDOWA
During the course of this thesis project, it has become apparent that the structure of TIDOWA had some drawbacks. One problem had been that the structure became more complex and consequently difficult to work with. Additionally, the execution of the program had been static, which had the consequence that an end user of the program needed to be in specific working directories to be able to execute it correctly. Besides that, there also had not been a distinct separation between the execution of the program and its source code. Due to this, some adjustments were made that resulted in a new software structure, which is described in the following.
27 5 CI-Oriented Changes and Extensions in TIDOWA
bin tidowa_ tidowa_ TIDOWA codegen.py run.py
_archive benchmarks bin build data tidowa_ tidowa_setup_ compile.py example_input.py
tidowa_ docs examples libtidowa_src tests disc.py
libtidowa_src/tidowa/timeintegration
libtidowa_src/ model_ codegen configuration convergence odedsl tidowa/ evalution optimization
plotter program
Figure 5.1: Overview of the structure of TIDOWA. The directory symbols and yellow rounded rectangles represent (sub-)directories, the blue rounded rectangles represent source code.
Figure 5.1 visualizes the directory structure of TIDOWA. The structure does not directly distinguish between the different projects, i.e. the original TIDOWA and the optimizer anymore and instead merges these segments into one combined project. The combination mainly consists of two predominant parts, the source code and the binaries or rather the executables. The source code may be found in the directory libtidowa_src. In this directory, the division between the original TIDOWA and the optimizer still exists for clarity. The directory libtidowa_src contains the part of the program that is designated to be read-only, e.g. the procedure instructions which transform an ODE into a discrete version of it. An additional directory libtidowa_src/timeintegration/program has been added. It contains the static part of the program execution. The executables may be found in the directory bin. These new introduced modules represents the interface between the program and the end user.
28 5.2 Restructuring of TIDOWA
TIDOWA bin tidowa_ tidowa_ _archive benchmarks bin build data codegen.py run.py
tidowa_ tidowa_setup_ compile.py example_input.py docs examples libtidowa_src tests tidowa_ disc.py
user input libtidowa_src/tidowa/timeintegration 1. user model_ codegen configuration convergence odedsl invokes method evalution
2. tidowa_disc calls plotter program main_disc in program
program
main_codegen.py main_disc.py
main_compile.py main_run.py 3. results are saved temporarily in /tmp
/tmp (results)
Figure 5.2: Event flow of the invocation of TIDOWA. Directory symbols and yellow rounded rectangles represent (sub-)directories, the blue rounded rectangles represent source code. The program flow is depicted with orange arrows.
Figure 5.2 visualizes the process that takes place, when the program is invoked. When invoked, the program execution starts with the modules in the directory bin, e.g. tidowa_disc.py. This module then invokes main_disc.py and transfers the user input, which then is processed regularly conforming with the TIDOWA structure as known. This holds for all stages of the program.
Further restructuring include the introduction of an archive, a directory for the tests, new examples on how to use the software and some additional minor changes. In addition to that, the results of each stage are now saved temporarily and locally in a clear defined structure outside of the source code and relative to the user’s current work directory. The results are ordered by the individual stages that take place during the execution of the program. This is one of two steps towards allowing the end user to invoke TIDOWA from any desired working directory instead of requiring specifically defined working directories.
29 5 CI-Oriented Changes and Extensions in TIDOWA
5.3 Integration of Python Setuptools and Virtual Envi- ronment
Another extension made towards the software is the introduction of Python Virtual Environ- ments and the integration of Python Setuptools. This section is based on [54, 55, 56] for the Python Virtual Environment as well as [57, 58, 59] for Python Setuptools.
Python Virtual Environment is a tool within Python that provides the option to run a program in an isolated location without interfering with the end user’s system, especially concerning the utilization of additional necessary Python packages, e.g. SymPy or NumPy in the case of TIDOWA. In addition, it lays the foundation to establish an uniform working environment for end users and developers to use and work on TIDOWA. This may be a benefit when releasing the software, since it is less challenging to support one defined environment and verify the correct functionality of the program within this environment, i.e. the correct functionality of the program using particular versions of imported Python packages, instead of supporting a variety of versions and then create the necessity of verifying the correctness of the software for each possible combination of package versions. Further, it may become less demanding and complicated to integrate newer versions of third-party Python packages into the software and release it while always supporting the latest versions available.
Python Setuptools is a tool that enables “developers to more easily build and distribute Python packages, especially ones that have dependencies on other packages”[59]. It replaces Python distutils, which was “the original Python packaging system”[57].
TIDOWA as a program, in large amounts at least, relies on dependencies inbetween the program’s modules as they are intertwined. Additionally, the program uses a considerable amount of imports as the results of some stages are saved in further Python modules, which have to be imported when arriving at the succeeding stage. Therefore, Python Setuptools is used to simplify the import of modules and thus the software structure and development. Moreover, Python Setuptools standardizes the usage of imports as all imports are absolute, hence not depending on the working directory of the end user any longer, and begin with the
30 5.4 Test Framework prefix tidowa.. This is the second step of Section 5.2 to facilitate flexibility in the usage of the program.
Both, Python Setuptools and Python Virtual Environments are introduced to the soft- ware with slightly-adjusted scripts that may be found in [60]. These scripts take full care of the setup needed for the program. They also make use of system variables to further simplify the usage of the software.
5.4 Test Framework
For the test framework, new functionality has to be implemented. This includes instructions belonging to the setup of the test environment, the execution, as well as the evaluation of the results and the teardown. As these changes play a considerable role in the concrete implementation, they will be discussed later in Chapter 6.
5.5 Further Extensions
Additional to the changes mentioned before, some further improvements within TIDOWA were made. One of them being the addition of descriptions in the input file containing the initial values along with the parameters, which simplifies the usage of the program for end users. Another one is that the model of the circadian clock is joining the list of pre-implemented ODEs. This may be used as an example for users or as additional testing instance for the test framework. Finally, from the developer’s perspective, the platform, where the program was located and developed changed from GitLab to GitHub, which initially had its reasons in the usage of Travis CI.
31 6 Continuous Integration in TIDOWA 6 Continuous Integration in TIDOWA
6.1 Introduction
After main aspects concerning the theory of ODEs, CI and TIDOWA were discussed and necessary changes regarding TIDOWA were introduced, this chapter describes in detail the integration of CI to the project. As mentioned earlier, this process consists of two parts, namely the CI environment itself and the test framework used to verify the correct behavior of implemented TI methods with respect to convergence. Therefore, this chapter is divided into two parts, one part for the test framework and one for the CI environment. Both sections will discuss the theoretical background (Chapters 6.2.1 and 6.3.1) and the concrete implementation (Chapters 6.2.2 - 6.2.6 for the test framework along with Chapters 6.3.2 and 6.3.3 for the CI environment) of the respective systems.
6.2 Implementation of the Test Framework
6.2.1 System Design: Essential Requirements and Design of the Test Framework
There are requirements that the implementation of the test framework should meet, which makes the correct design of such test framework crucial. This chapter summarizes these requirements and examines the design of the test framework afterwards.
Extensibility: One requirement concerning the design of the test framework is the continual option and support of future extensions. As TIDOWA grows, it will provide more function- alities and becomes more complex with them. Hence, the correctness of these extensions has to be verified with additional tests and test cases. Therefore, the test framework has to be extensible. It should be mentioned that the extensions can be of any nature, e.g. novel pre-implemented ODEs or IVPs, respectively, new TI methods or even more complex additions like the support of boundary value problems.
32 6.2 Implementation of the Test Framework
Usage: Another requirement is the usage of the test framework. It should be simple to use the system, otherwise no other developer will use it, even if the advantages outweigh the effort required to learn the functionality provided by the test framework. Thus, adding, changing, removing and executing test cases should be straightforward.
Parallelism: The time required to execute the test cases is the next important aspect. As TIDOWA grows, it is inevitable that the number of test cases will increase as well. To reduce the execution time, the framework should be designed in a way that permits and supports parallelism.
ODEs: This next requirement is different to the former ones, but as significant. There is the general problem that not every ODE and consequently not every IVP has an analytical solution, e.g. the Lorenz system that consists of three ordinary differential equations. In such cases, another approach has to be considered to yield values close to the behavior of the ODE or the system of ODEs, respectively.
In order to meet these requirements, a structure consisting of individual modules is se- lected for the test framework. Figure 6.1 visualizes such a modular structure. A module can be interpreted as a small box that receives some input, processes this input in some way and returns an output, which then may be used by other modules as their respective input. In general, a module handles just one specific task. The program, in this case the test framework, would then consist of multiple interrelated modules that altogether form a program flow. In other words, it would be a program that consists of smaller subprograms, each performing their own specific task and cooperating with the other subprograms.
33 6 Continuous Integration in TIDOWA
module A module D
input module E output
module B module C
Figure 6.1: Exemplary modular structure consisting of input and output (both yellow), modules (blue rounded rectangles) that accomplish their respective task, connectors (red arrows) that connect individual modules as well as an underlying administrative structure that controls the program flow (green rectangle).
As visualized in Figure 6.1, besides the input and output, three additional components are required to realize a modular design. These three components identified are:
• the individual modules that perform their task individually, • connectors to transfer data (i.e. results of individual modules) between modules, • an administrating structure that handles the program flow, e.g. determines the execution order of the individual modules.
For our test framework, the selection of this modular structure has some advantages. A system that consists of individual but interrelated modules, which handle their tasks separately and share their results with the other modules is easier to extend. In addition, such systems are more compact in a way that when a specific task is needed multiple times during the execution, it can be performed by one module instead of for example placing the needed instructions separately multiple times in the source code. Furthermore, it is less difficult to verify the correctness of these modules as the source code is more compact and if errors occur, they may be traced back to individual modules instead of multiple locations in the source code. Additionally, a stepwise verification is easier due to this structure. Another advantage is that a modular structure enhances parallelism as the separate tasks handled by these modules can be distributed to different processes.
For now, the test framework itself is considered to be a large, abstract machine with in- put parameters being processed in a yet still unknown way and some output being returned
34 6.2 Implementation of the Test Framework as a result of this process, comparable to the exemplary modular structure in Figure 6.1. In the following, this structure will be extended and transformed into the specific structure of the test framework, which has the same form as depicted in Figure 6.2. The following sections will examine all parts of this structure using the previous classification of components of the modular structure as basis. Prior to that, a precise job description for the framework has to be formulated.
(A) program flow
(A) output (C) Difference Generator underlying structure (M) Difference Generation input test case directory: test case directory: module approximations differences derived with of APPROX X various step sizes and ANA test case directory: input analytical solution
APPROX APPROX APPROX APPROX DIFF DIFF DIFF DIFF A B C D A B C D ANA
(A) (A) (A) output output pairs as input
(C) Analytical Generator (C) Approximation Generator
(M) Analytical (M) Approximation Generation Generation (M) Evaluation module module module
number list of IVP of TI method step sizes final steps output {True, False} input
Figure 6.2: System design of the test framework. Components marked with (A) are related to the administrative structure, these marked with (C) to connectors and these marked with (M) to modules. This figure serves as overview and reference for the following chapters that explain each component.
The task of this framework is to examine the convergence of a given TI method for a given IVP at a point t. As known from Chapter 2.5, in order to check the convergence of a TI
35 6 Continuous Integration in TIDOWA method for a specific IVP, the analytical solution of the IVP at point t, the TI method’s approximations of the solution at t and the discretization error at t are necessary. Looking back to Chapter 2.5, multiple approximations of the TI method have to be computed using various step sizes. From this description, a lot of information can be derived for the framework.
Considering the analogy of our unknown machine again, the gap between input and output can be filled and some processes may be defined now while using the job description as basis. The abstract input of the test framework is the tuple (IVP, TI method, list of step sizes, number of steps). Here, the point t is equal to the product of the largest record in the list of step sizes and the number of steps. The output (both abstract and specific as they do not differ) of the test framework is a logical value {True, False} as the convergence of a TI method and consequently the correctness of the underlying implementation can either be verified or disproved. The processes that were retrieved from the job description are listed below:
• Identify the analytical solution of the IVP at point t. • Determine the approximations of the analytical solution at point t using the TI method and various step sizes- • After these values are computed, the discretization error for each approximation has to be determined (i.e. the difference of analytical solution and approximation). • Compare the discretization errors of two approximations with different step sizes, check the convergence criteria and return the result.
These four bullet points are the four steps needed to check the convergence for a TI method plus IVP in general. As a modular structure is to be implemented, each step is translated or transformed into one module, respectively. Following the order of the list above, the four modules are denoted as: Analytical Generation module, Approximation Generation module, Difference Generation module and Evaluation module. These modules rep- resent the main core of the test framework. Therefore, all four stages together are referred to as the Core Four in the following. Using the idea of Figure 6.1, the Core Four are the modules or the subprograms, respectively. In Figure 6.2, the Core Four are marked with the letter (M). The connectors and the underlying, administrative structure are two components that are still missing.
36 6.2 Implementation of the Test Framework
As known from the job description and the definition of the modules, the results of the Core Four are the analytical solution of the IVP at point t, the various approximations of the IVP’s solution at t, their differences, derived from subtracting the analytical value with the respective approximation, and a logical value. Because of the modular structure, the imple- mentation needs to provide a solution how to store these values and make them available to the modules that require them. In addition to that, the framework should ensure that the results are not falsified due to rounding or conversions as they are passed around between the modules.
One possible solution is to use the functionality of a third-party module that is presented in Chapter 6.2.3, hence the problem of connecting the modules to transfer results in between is solved. For now, it is only important to know that the results are saved in files. In Figure 6.2, the connectors are marked with the letter (C). After discussing the modules and the connectors, the design of only one component remains unknown, this being the administrative structure.
The administrative structure should unite the individual modules and the connectors into one single program that accomplishes the task described in the job description above. Therefore, a structure that arranges these individual modules into one program flow and in addition handles further possible necessities has to be formed. The procedure concerning the Core Four is relatively clear. First, the Analytical Generation module and the Approximation Generation module have to accomplish their tasks. As both results are required at the same time in the following, the order of both is irrelevant in this case. Using connectors (i.e. by saving the results of both modules in the form of files), the results are passed over to the Difference Generation module, where the difference of both is computed. As multiple approximations exist, there will also be multiple differences. Again using connectors, the differences are handed over to the Evaluation module, where the convergence criteria is evaluated. Finally, this result is then given to the user.
As parallelism should be enabled, the test framework should make sure that each approxi- mation and the analytical solution is computed using a separate script or a separate set of instructions, respectively. This is necessary as this part of the program requires most of the resources, i.e. the most amount of time. In addition to that, the results and the individual
37 6 Continuous Integration in TIDOWA instructions of each computation process have to be saved in such a way that they can not disturb other parts of the program or interfere with them.
Thus, a further task, which may be located at the start of the program flow emerges, namely the task of preventing such events. To avoid interference and disturbances between individual computation processes, a separate directory for each computation process is required. The task will be handled by an additional module, introduced with the administrative structure, which is used to set up the program by creating the necessary instructions and directories. In summary, the administrative structure fulfills the task of setting up the program and determining as well as administering its program flow. In Figure 6.2, the administrative structure is marked with the letter (A).
After the final design of the system was described, the concrete implementation of this design is examined in the following chapters.
6.2.2 Implementation: Introduction
As the final structure is quite large and complex, the concrete implementation will be examined in separate parts. First, Implementation: Preliminaries will introduce two important third-party and built-in modules that are used for the implementation. Then, the structure of the following chapters is analogous to the order introduced with the system design. In other words, first the Core Four, the modules that implement the main functionality and afterwards the connectors, the components that should link the modules are discussed. Then both components are unified during the examination of the administrative structure. Before discussing the preliminaries and examining the implementation, the directory structure of the test framework, which is illustrated in Figure 6.3 has to be discussed first.
38 6.2 Implementation of the Test Framework
convergence directory (libtidowa_src/tidowa/timeintegration/convergence)
analyticalsol_ approximation_ difference_ convergence_ generator.py generator.py generator.py evaluator.py source code
analyticalsol_ approximation_ difference_ structure_ module.py module.py module.py generator.py
tests directory (tests/convergence_tests)
directories, 10_ 10_ representing conv_DATA conv_COMMON test cases
execution bin test.sh scripts for program flow
bin 10_conv_COMMON tidowa_ct_ tidowa_ct_ tidowa_ct_ approximation_ analyticalsol_ convergence_ generate_ post_ generator.py generator.py evaluator.py scripts.sh processing.sh tidowa_ct_ tidowa_ct_ difference_ structure_ post_ generator.py generator.py processing_ run_scripts.sh cleanup.sh
Figure 6.3: Structure of convergence directory and tests directory. (Sub-)directories are illustrated by yellow rounded rectangles and directory symbols. Python modules (.py) and Shell scripts (.sh) are illustrated by blue rounded rectangles.
Given the adapted program structure illustrated in Figure 5.1, all components introduced with the test framework are located in two directories, namely /tests/convergence_tests (omitted in Figure 5.1 and abbreviated with tests directory hereafter) and /libtidowa_- src/tidowa/timeintegration/convergence (abbreviated with convergence directory in the following). Similar to the CI-oriented changes of TIDOWA, the test framework introduces a directory structure that separates source code and execution. Thus, /tests/convergence_- tests is used to execute the program and the other directory to store the source code. Similar to the general changes made concerning TIDOWA’s structure from Chapter 5.2, the directory that handles the execution contains a subdirectory bin. Here, tests directory has an additional subdirectory 10_conv_COMMON that itself contains several scripts. This aspect and the structure of tests directory will be reiterated during the unification of the individual components in Chapter 6.2.6. In addition, the following chapters will closely examine each
39 6 Continuous Integration in TIDOWA
Python module and all other components located in both directories.
For now, the given structure is sufficient to follow and understand the upcoming expla- nations. Note that in the following chapters less relevant parts of the program are occasionally omitted due to reasons of visibility, compactness and clarity, e.g. method parameters. Also remember that when ODEs are referred to, especially in the context of the specific implemen- tation that this is no error. It is important to understand that TIDOWA receives the ODEs and the initial conditions separately as input and only applies the initial conditions to the ODE in the last step of its program flow. For this reason, the new implemented methods also only receive the ODEs as input. The initial conditions are separately set up by the method tidowa_setup_example_input.py, which saves them at a known location prior to any other computation, thus are not handed over as parameters generally. As the location is known TIDOWA may use the initial conditions for its last step of the program flow. Because of this, the result of any computation of TIDOWA is an approximation of the IVP and as such is referred to as approximation of an IVP. The same circumstance also applies to other parameters concerning the ODE configuration.
6.2.3 Implementation: Preliminaries
The test framework uses some functionality of third-party modules along with specific built-in Python modules. Note that modules in the test framework are different to these modules. Before the implementation is examined, the most relevant third-party modules used are presented. The first sections concerning SymPy are based on [61, 62, 63, 64]. The second part regarding pickle is based on [65].
SymPy is used to hand over the ODE given and its configuration to the program using symbolic mathematics, e.g. the variables in the configuration file use this symbolic form, which then can be replaced by numeric values afterwards. In addition to that, it is used by later stages of TIDOWA to describe the results of the individual stages.
Regarding the test framework, SymPy offers further useful functionality concerning the solution of less complex IVPs. The third-party module provides methods, called dsolve, subs
40 6.2 Implementation of the Test Framework
and evalf, with which one is able to compute the analytical solution of simpler IVPs, as shown in the following code extract used to compute the solution of the IVP from Chapter 2.4.1.
1 # source code of example.py 2 import sympy as sp 3 4 # specify the parameters and variables of the ODE 5 t= sp.Symbol("t") 6 variablenames= "u" 7 u= sp.symbols(variablenames, cls=sp.Function) 8 9 # specify the ODE 10 ode= sp.Eq(u(t).diff(t), 0.25*u(t)) 11 12 if __name__ == '__main__': 13 # IVP: ODE given and initial condition u(0) = 1 and solve the IVP with dsolve 14 analytical_solution= sp.dsolve(ode, u(t), ics={u(0):1}) 15 print(analytical_solution) 16 # compute the analytical solution for t = 4 with subs and evalf 17 value_at_4=x.rhs.subs(t,4).evalf() 18 print(value_at_4) 19 20 # invocation of example.py 21 # > Eq(u(t), exp(0.25*t)) 22 # > 2.71828182845905
The method dsolve is used to obtain the analytical solution of a given IVP in form of an equa- tion u(t) = rhs. The methods subs and evalf are then applied to the equation to determine its solution for a specific point by substituting t with a numeric value and evaluating the equation thereafter. As stated earlier, the analytical solution can not be computed for each IVP. Therefore, the presented methods are only used to a small extent in the test framework.
The second essential Python module is called pickle. This module is used to transfer the results, i.e. the analytical solution, the approximations and the differences, inbetween the individual modules without modifying them. To achieve this, two functions are provided, called dump and load, which are shown in the following code extract.
1 # exemplary usage of the functionality of pickle 2 result= 42 3 4 # define a .pickle-file in which the result should be saved 5 path=/path/to/desired/file.pickle 6 output_file= open(path, "wb") 7 8 # generate a byte-representation of result with dump 9 # and save it in output_file
41 6 Continuous Integration in TIDOWA
10 pickle.dump(result, output_file) 11 output_file.close() 12 13 # read the content of the '.pickle'-file 14 input_file= open(path, "rb") 15 16 # transform the byte-representation back to the result 17 pickle_result= pickle.load(input_file) 18 file.close()
By applying a dump to a Python object, a byte-representation of this object is saved to a file. The method load is able to transform this byte-representation back into a Python object without changing its former structure, e.g. its data type. Pickle enables the possibility of saving results accurately for further computations and solves the problem of perturbed data inbetween modules.
6.2.4 Implementation: Modules - The Core Four
As discussed during system design, the test framework contains four main modules, namely the Core Four. This chapter examines the specific implementation of them, isolated from the other components of the final system design.
Approximation Generation module:
Approximation Generation module
output input task: computation of the TI methods' approximation of the analytical solution of TI methods' - ODE the IVP at point t - TI method approximation of the analytical - list of step sizes implemented methods: solution of the - number of steps procedure_ODE_TI(dt, steps, language, approx) IVP at t invokes procedure(ode, ti_methods, dt, steps, language, approx, implicit) invokes procedure_file(directory, config, ti_methods, dt, steps, language, approx, implicit)
Figure 6.4: Overview of the Approximation Generation module. The yellow components represent the input and the output, the blue rounded rectangle represents the module with its tasks and the implemented methods it contains. The black arrows inside the module represent dependencies and the red arrows represent the connectors. This serves as a summarized overview of this chapter for further reference.
As defined during the system design, the Approximation Generation module handles the computation of the TI methods’ approximations of a given IVP at a point t. Figure 6.4 briefly visualizes this module with the input that is required and the output that is produced. To
42 6.2 Implementation of the Test Framework
accomplish the task, the entire abstract input tuple (IVP, TI method, list of step sizes, number of steps) that was defined during the system design is assumed to be the required input. The result of this module is a list of approximations of the IVP’s solution at t. Notice that the result is a list as the TI method computes one approximation for each record in the list of δts.
This module is realized with the Python module approximation_module.py. approximation_- module.py defines functions for pairs of pre-implemented TI methods and pre-configured ODEs (remember that ODEs are later transformed into IVPs) using the following naming scheme: procedure_ODE_TI(). Exemplary, the function for the pair (TI method: RK4, ODE: oscillatory ODE) is defined as:
1 def procedure_oscillating_rk4(dt, steps, language, approx): 2 result_rk4= procedure("oscillating",("RK4", "RK4",), dt, steps, 3 language, approx, implicit=False) 4 return result_rk4
These methods invoke the original functionality of TIDOWA to compute the approximation of the IVP’s solution by invoking procedure_file() (in most cases invoked through a wrapper procedure()) with the necessary parameters. procedure_file() represents the core of this Python module as this is the function that implements the invocation of TIDOWA and returns the approximation with a slightly modified format to conform with future requirements.
As one may have seen, the implemented methods procedure() and in this case consequently also procedure_file() possess additional parameters language, approx and implicit not being part of the abstract input tuple. language determines the language or the framework, respectively that TIDOWA will use for its computations and implicit denotes implicit TI methods. approx is related to Analytical Generation module, which is examined in the next section.
43 6 Continuous Integration in TIDOWA
Analytical Generation module:
Analytical Generation module input output task: computation of the analytical solution of the IVP at point t - ODE analytical - list of step sizes solution implemented methods: - number of steps of the IVP at t solve_at(t, ode, language) solve_int(dt, steps, ode, language)
invokes functionality of Approximation Generation module
Figure 6.5: Overview of the Analytical Generation module. The yellow components represent the input and the output, the blue rounded rectangle represents the module with its tasks and the implemented methods it contains. The black arrows inside the module represent dependencies and the red arrows represent the connectors. This serves as a summarized overview of this chapter for further reference.
The Analytical Generation module handles the computation of the analytical solution of an IVP at point t. Figure 6.5 briefly visualizes this module with the input data and resulting output data. To accomplish this task, the module needs the tuple (IVP, t) as input, thus, only two of four parameters from the abstract input tuple. The output of the module is the analytical solution of the IVP at point t.
This functionality is implemented by the Python module analyticalsol_module.py. For each pre-implemented ODE, the Python module contains the methods solve_at(t, ODE) and solve_int(dt, steps, ODE). solve_at(t, ODE) computes the analytical solution of the IVP at point t and solve_int(dt, steps, ODE) returns a list that contains the analytical solution of the IVP for the point t being {δt × 0, δt × 1, ..., δt × (steps − 1)}. The difference between both methods is the former returns the analytical solution for one specific point, whereas the latter computes the analytical solution of multiple equidistant points and returns them in form of a list.
For IVPs that have an analytical solution this is derived using functionality provided by SymPy. IVPs without an analytical solution can not use this approach. Therefore another technique is used to approximate the behavior of such IVPs. The test framework solves this problem with precise RK4 approximations of the IVP’s solution to estimate the IVP’s
44 6.2 Implementation of the Test Framework
behavior. The important aspect here is that these RK4 approximations are computed using a significantly small step size δt = 10−6. As RK4 possesses an order of four, the usage of a step size that small will derive an approximation, which is very close to the expected behavior of the IVP. Nevertheless, for reasons of clarity such values will still be denoted as analytical solution in this thesis, even though in reality, these are approximations.
In the implementation of analyticalsol_module.py, the RK4 approximations are obtained by the usage of methods defined in approximation_module.py. Note that as functionality of the Approximation Generation module and as a consequence TIDOWA is used, the parame- ters of solve_at() and solve_int() are ODEs. An exemplary definition to compute the analytical solution of the IVP with the Lorenz system is shown the following:
1 def solve_at(t, ODE): 2 ... 3 if ODE == lorenz: 4 approximation= convergence.procedure_lorenz_rk4( 5 dt= 0.000001, steps= int(t/0.000001), language, approx= True) 6 ... 7 ...
The parameter steps is chosen this way due to a concept of the test framework called balancing. If for example a TI method is instructed to compute an approximation at t = 1 with δt = 1, one step is necessary. When a smaller step size, e.g. δt = 0.5 is considered, in order to reach t = 1, two steps are necessary. In case of the test framework, if the approximation at t is to t be computed with a step size of δt = 0.000001, 0.000001 steps are required. The conversion to int is important as partial steps do not exist. The concept of balancing is used multiple times throughout the test framework. Also, here the reasoning behind the parameter approx becomes clear. The parameter is used as a logical value denoting approximations computed for analyticalsol_module.py.
45 6 Continuous Integration in TIDOWA
Difference Generation module:
Difference Generation module input output task: - analytical difference of computation the difference between analytical solution and solution at t analytical approximation solution and - approximation approximation implemented methods: at point t at point t generate_diffs(analytical_solution, approximation)
Figure 6.6: Overview of the Difference Generation module. The yellow components represent the input and the output, the blue rounded rectangle represents the module with its tasks and the implemented methods it contains. The red arrows represent the connectors. This serves as a summarized overview of this chapter for further reference.
From the final system design it is known that this module should compute the discretization error, i.e. the difference between the analytical solution of an IVP at point t and the approxi- mation of this solution at t. Figure 6.6 briefly illustrates this module. To accomplish this task the module requires the analytical solution and the approximation as input and returns the difference as output.
The Python module difference_module.py, more precisely the method generate_diffs( analytical_result_path, approximation_path) in the Python module accomplishes this task. The method receives the analytical result and the approximation, then computes the absolute difference |analytical_result - approximation| and returns it along with additional information, i.e. the step size. The reason why the method receives two paths instead of just two results will be examined in a section later on.
46 6.2 Implementation of the Test Framework
Evaluation module:
Evaluation module input output task: - differences of logical value check the convergence criteria with the differences of various various step either step sizes sizes confirming or - order of disproving the implemented methods: TI method convergence evaluate(difference_dt1, difference_dt2)
Figure 6.7: Overview of the Evaluation module. The yellow components represent the input and the output, the blue rounded rectangle represents the module with its tasks and the implemented methods it contains. The red components represent the connectors. This serves as a summarized overview of this chapter for further reference.
The Evaluation module shown in Figure 6.7 has the assignment to evaluate the convergence criteria, which leads to the final result, either True or False. Thus, the module requires several discretization errors, more precisely prior computed differences with varying step sizes, along with the order of the TI method as input. The output is a logical value.
The Python module convergence_evaluator.py implements this evaluation process with the method evaluate(difference_dt1, difference_dt2, order). Remember that the differences also contain additional information, e.g. the step size δt. When given two difference values with their respective step sizes (it holds: δt1 ≤ δt2), the following inequation has to be fulfilled: δt1 difference_dt1 ≤ difference_dt2 × ( )order δt2
The mechanism to check this inequality is implemented within the method and returns a logical value either verifying or disproving the convergence behavior of the TI method’s implementation.
6.2.5 Implementation: Connectors - Data Transfer between Mod- ules
So far, all Python modules implementing the Core Four were examined. Now a mechanism or system that links up the Core Four is required, since up to now it remains unknown how the implementation realizes the interaction between individual modules. This motivates the examination of the connectors in the following section.
47 6 Continuous Integration in TIDOWA
Due to the modular structure of the final system design, the test framework needs to handle the transfer of the results between the individual modules without modifying them. As previ- ously stated, the implementation needs to transfer three results, them being the analytical solution yielded from the Analytical Generation module, its approximation yielded from the Approximation Generation module and the difference of both results computed in the Difference Generation module. The result of the Evaluation module is excluded as it is the output displayed to the user.
The respective tasks are all handled by the Python modules that contain the suffix generator.py, except the structure_generator.py. The task of transferring the analytical solution is han- dled by the Python module analyticalsol_generator.py. Equally, the transfer of the approximation is performed by approximation_generator.py and the transfer of the differ- ence is administered by difference_generator.py. All three act to some extent as interface to the respective Core Four module that performs the specific computation. Furthermore, all three handle the transfer process with the usage of pickle, introduced in Chapter 6.2.3. Hence, they all possess a similar structure and program flow. The general flow is illustrated in Figure 6.8.
input: path and 2. prepare necessary necessary parameters 1. check, if a result no parameters and 3. save the result in already exists at the invoke the respective afile using pickle given path yes module
respective result module
file: result.pickle
Figure 6.8: General program flow of connectors. The program flow is illustrated with stages (red rectangles), connected by black arrows. The black diamond represents the two possible results of the connector’s if-query, the dotted arrows represent the invocation of the respective module (blue) and the yellow component represents the file in which the result is stored.
As one may see, the general connector receives the necessary input parameters depending on the respective module along with a path as input. The connector inspects, if the path passed is correct and furthermore checks, if a computation yielding the same result was already
48 6.2 Implementation of the Test Framework executed. If this is the case, the connector is finished. Otherwise, the connector invokes the desired method from the respective module and expects the result from the module in return. As a last step, the result is sometimes slightly modified to conform with desired formats, and then is saved in a file using the functionality of pickle. Consecutive modules then have access to these files, if they know the location where the file has been saved. For this reason, the former examined Difference Generation module receives paths that refer to the location of required values as input. The idea of skipping computations when their results already exist is implemented in order to reduce the execution time as some computations are very time consuming.
6.2.6 Implementation: Administrative Structure - Unification into One Program
After the implementation of the Core Four modules as well as the implementation of the connectors were examined, the administrative structure that unifies all components into one program is discussed. As this is the part of the program, where everything concerning the program flow of the individual modules and connectors along with the execution of the program is handled and as a clear separation of source code and execution was demanded in Chapter 6.2.2, prior to discussing the program flow, tests directory has to be examined further, because this is the directory that contains the required components regarding the execution of the program. tests directory consists of several subdirectories, which are examined in the following. /10_conv_COMMON is a directory, which contains shell-scripts that are responsible for the program flow of the test framework. Further, tests directory has a subdirectory /10_- conv_DATA used to store relevant data required for program execution, i.e. the list of step sizes, and a subdirectory /bin, which contains the interfaces used to invoke the modules in convergence directory, i.e. the Core Four through the connectors. The labelling of these interfaces is equal to the labelling of the modules in convergence directory, prepended with the prefix tidowa_ct_. In general, convergence directory also contains the test cases, for example 10_test_osc_conv_RK1 is a test case for the tuple (TI method: RK1, IVP: IVP with oscillatory ODE).
49 6 Continuous Integration in TIDOWA
As the structure of both directories is mostly known now (the Python modules from convergence directory were examined throughout the previous chapters), apart from the Python module structure_generator.py in convergence directory and its interface in the subdirectory /bin, as those are explained later due to them being part of the administrative structure, the program flow will be further examined. To accomplish this, a single iteration of the test framework is considered. Therefore, a general test case 10_test_ODE_conv_TI for the tuple (TI method: TI, IVP: IVP with ODE ODE) is used. A test case is implemented with a directory and script test.sh included. The script contains the parameters required that are handed over to the program by an user and instructions that refer to other scripts and determine the program flow. The order of these instructions is the following:
1. generate_scripts.sh 2. run_scripts.sh 3. post_processing.sh 4. post_processing_cleanup.sh
With the assumption that all parameters required were handed over to the program correctly and the program will execute these instruction in that specific order, similar to earlier each instruction is now examined separately.
1. generate_scripts.sh: This script has two tasks, namely the setup of the IVP and the creation of folders and scripts needed to verify the convergence. The former is accom- plished by invoking tidowa_setup_example_input.py to set up the IVP according to the specification of TIDOWA. The latter uses the yet unknown Python module tidowa_ct_- structure_generator.py and thus structure_generator.py.
This Python module should create the required folders and subscripts needed for further steps in the program flow, with other words, it sets up the program flow of the Core Four. The result of this script is shown in Figure 6.9.
50 6.2 Implementation of the Test Framework
10_test_ODE_conv_TI
job_ODE_TI_dt_dt1 job_ODE_TI_dt_dt2 job_ODE_TI_dt_dt3 job_ODE_TI_dt_dt4
run.sh run.sh run.sh run.sh
job_ODE_TI_dt_dt5 ref_ODE_TI
run.sh run.sh test.sh
Figure 6.9: Exemplary directory structure created by invoking generate_scripts.sh. The directory symbol and the yellow rounded rectangles represent (sub-)directories, the blue rounded rectangles illustrate scripts.
As the program flow regarding the Core Four starts with determining the analytical solution and the TI methods’ approximations, both have to be set up. As parallelism is desired, the computation of the analytical solution and the computation of the TI methods’ approximations have to be separated. Furthermore, the computation of the individual approximations also has to be separated. For each step size given, the Python module structure_generator.py creates a directory that contains another script run.sh. Moreover, it creates a similar structure for the computation of the analytical solution.
The individual scripts run.sh contain instructions that use functionality of the Core Four to compute the respective values. The following code extracts denote three exemplary run.sh scripts, two for the computation of approximations and one for the computation of the analytical solution.
1 # calculate approximation 2 python $TIDOWA_TEST/bin/tidowa_ct_approximation_generator.py -dt0.01 -steps 161 -ode ODE -language C ,→ -ti TI -directory 10_test_ODE_conv_TI
1 # calculate approximation 2 python $TIDOWA_TEST/bin/tidowa_ct_approximation_generator.py -dt0.16 -steps 11 -ode ODE -language C ,→ -ti TI -directory 10_test_ODE_conv_TI
1 # calculate analytical solution 2 python $TIDOWA_TEST/bin/tidowa_ct_analytical_generator.py -steps 10 -ode ODE -language C -ref0.16 ,→ -directory 10_test_circa_conv_RK1 || exit -1
51 6 Continuous Integration in TIDOWA
The first two would determine approximations of the analytical solution of the IVP with the ODE ODE at a point t = 1.6, using the TI method TI, if invoked. The last one would compute the analytical solution at this point. Additionally, with the option -language TIDOWA is forced to use C for its computations. Notice that as different step sizes are in use, structure_generator.py also implements balancing to balance the number of steps for each step size. After the structure is set up, the script is finished.
2. run_scripts.sh: This script has the task of invoking the previously generated scripts run.sh one after the other or in parallel, thus computing the analytical solution along with the approximations and saving the results in the respective directories in a file output.pickle. The exemplary structure would evolve into the following depicted in Figure 6.10:
10_test_ODE_conv_TI
job_ODE_TI_dt_dt1 job_ODE_TI_dt_dt2 job_ODE_TI_dt_dt3 job_ODE_TI_dt_dt4
run.sh output.pickle run.sh output.pickle run.sh output.pickle run.sh output.pickle
job_ODE_TI_dt_dt5 ref_ODE_TI
run.sh output.pickle run.sh output.pickle test.sh
Figure 6.10: Directory structure after run_scripts.sh. The directory symbol and the yellow rounded rectangles represent (sub-)directories, the blue rounded rectangles illus- trate scripts. The orange rounded rectangles represent the analytical solution or the approximation, respectively.
After invoking each script run.sh, every directory contains a file output.pickle, which is a byte-representation of the approximation of the IVP’s solution using the TI method given and the respective step size or a byte-representation of the IVP’s analytical solution, respectively.
3. post_processing.sh: This script determines the absolute differences of the analytical solution and the various approximations, yielding the structure shown in Figure 6.11. For said task, tidowa_ct_difference_generator.py is invoked for each instance of output.pickle that represents an approximation and the output.pickle of the directory ref_ODE_TI. Hence, the absolute difference is computed for each approximation and saved in their respective
52 6.2 Implementation of the Test Framework directory in a file postprocess.pickle.
10_test_ODE_conv_TI
job_ODE_TI_dt_dt1 job_ODE_TI_dt_dt2 job_ODE_TI_dt_dt3 job_ODE_TI_dt_dt4
run.sh output.pickle run.sh output.pickle run.sh output.pickle run.sh output.pickle
postprocess.pickle postprocess.pickle postprocess.pickle postprocess.pickle
job_ODE_TI_dt_dt5 ref_ODE_TI
run.sh output.pickle run.sh output.pickle test.sh
postprocess.pickle
Figure 6.11: Directory structure after post_process.sh. The directory symbol and the yellow rounded rectangles represent (sub-)directories, the blue rounded rectangles illus- trate scripts. The orange rounded rectangles represent the analytical solution/the approximation (output.pickle) or the difference of both (postprocess.pickle), respectively.
Besides that, the script also checks for the criteria of convergence after the differences are ob- tained. This is accomplished by invoking methods of tidowa_ct_convergence_evaluator.py multiple times with the file postprocess.pickle of two arbitrary approximations with differ- ent step sizes as input. After post_processing.sh has terminated, the convergence is either verified or disproved.
4. post_processing_cleanup.sh: This script is invoked to remove the directories created by TIDOWA during the calculation of the approximations, especially the directory /tmp. This step is required as otherwise multiple executions of TIDOWA will interfere with each other because the intermediate results of their respective computations are saved all in the directory /tmp by default.
53 6 Continuous Integration in TIDOWA
6.3 Integration of CI Environment
6.3.1 System Design: Selection and Essential Requirements of the CI Environment
After Chapter 4.3 presented some commonly used CI environments, this chapter concentrates on the question, which CI environment is to be selected for TIDOWA. There are some re- quirements regarding the CI tool that play an important role in the selection process. These will be described in the following.
Support: The first constraint is that the CI system has to support the languages and frameworks that are used by TIDOWA. Desirably, there is a native support for the languages and frameworks, but the use of additional plugins or installations is also sufficient.
Parallelism: The next requirement is that the CI tool supports parallelism. This is especially necessary as the program grows and therefore the number of test cases, as does the amount of time needed for test execution increases. Parallelism can be used to execute multiple tests simultaneously, hence making it possible to execute the same amount of tests in a less amount of time.
Additional support of hardware: An optional but desired requirement is the possibility of choosing or integrating various CPU architectures or even GPUs as TIDOWA supports some frameworks.
Costs: As this is still a small project it is also necessary that the usage of the CI envi- ronment does not have any costs. Therefore, implementations that are free of charge are preferred.
After analyzing and discussing the requirements and applying them on the CI environ- ments, there were two remaining options to choose from, namely Travis CI and Jenkins, as both meet the majority of the requirements. Initially, Travis CI was chosen to be the CI environment. Unfortunately, this decision had to be reversed as the provider changed its
54 6.3 Integration of CI Environment pricing policy, which contradicts with the requirement of no costs. Therefore, the decision had to be changed in the midst of the project and Jenkins became the CI environment of choice. As Jenkins is a CI environment that has to be hosted by TIDOWA itself, a server had to be set up and hardware had to be provided. Chapters 6.3.2 and 6.3.3 describe various aspects of this process.
6.3.2 Implementation: Preliminaries
Prior to installing and using Jenkins, the server infrastructure has to be examined and the requirements of Jenkins have to be met. Both of these aspects will be addressed in this chapter that itself is mostly based on discussions with the advisor and also [42, 66, 67] for the sections concerning Docker.
As Jenkins needs to be hosted on our own devices, a virtual server was set up for this purpose. Figure 6.12 displays the server structure as well as portrays how the virtual server is accessed.
configuration of virtual server (e.g. start-up, teardown)
Internet ssh ssh
Jenkins
any PC
login server virtual server
configuration and usage of Jenkins (e.g. configuration of tests)
Internet Jenkins browser
any PC virtual server
Figure 6.12: Server infrastructure that is used for the integration of Jenkins and ways of accessing Jenkins and the virtual server depending on the purpose.
The virtual server itself uses an Linux distribution and is accessible through another instance
55 6 Continuous Integration in TIDOWA comparable to a login server using ssh. Jenkins should be accessible through the Internet with the usage of a browser. Later, a developer or the VCS host GitHub should be able to conveniently access or reach the CI server, respectively, without the necessity of using ssh. This is necessary as the user interface of Jenkins is only accessible through a web browser. The concrete implementation of this is examined further in the next chapter.
After the virtual server was made accessible, Jenkins had to be set up. Prior to that, the requirements were installed. This step includes a JDK distribution necessary to run Jenk- ins, git, a VCS that allows the virtual server to set up a connection to the repository, packages and dependencies necessary to execute TIDOWA, e.g. Python, and more interestingly Docker. Docker is a software that allows one to execute applications inside of a Docker container, this being “a standard unit of software that packages up code and all its dependencies so [that] the application runs quickly and reliably”[66]. With the help of Dockerfiles, used to generate Docker containers, Jenkins is able to execute its tests inside of various different environments. The usage of Dockerfiles in combination with Jenkins can vary from for example testing software against different versions of third-party modules to testing it against various Linux distributions that differ to the host’s distribution. Interested readers may be referred to the references for additional information on the functionality of Docker.
6.3.3 Implementation: Integration of Test Framework
After the virtual server is set up and the requirements concerning Jenkins are fulfilled, the setup of the CI server itself is examined. This chapter describes the most important aspects of Jenkins’ setup. The documentation of Jenkins [37] was the basis for the majority of the configuration done and thus, it is the basis for this section.
The installation process itself followed the instructions of [37]. After the installation was done, Jenkins was set up to provide two possible ways on how to use the system. Firstly, the builds should run on the host machine. Secondly, the builds should also be able to use Docker containers to run builds. To implement the former, Jenkins provides Freestyle projects (term derived from Jenkins), for the latter the functionality of Pipelines is offered.
56 6.3 Integration of CI Environment
Freestyle projects allow us to run the builds of TIDOWA on the host machine. By specifying the location of the source code repository by means of its URL and a build script, one is able to check and verify the correctness of the current source code. This is done by firstly using a functionality of git to copy the repository into the local storage of the virtual server and then executing the build script, which in case of TIDOWA consists of the following instructions:
1 bash python_setup.sh 2 source activate_env_vars.sh 3 bash $TIDOWA_CONV_TEST
python_setup.sh and activate_env_vars.sh are used to set up the environment required. TIDOWA_CONV_TEST is an environment variable that points to the test script. Thus, one is able to invoke the test script by using bash $TIDOWA_CONV_TEST. As for TIDOWA it is assumed that if the tests pass, the build is considered to be successful otherwise the build is considered to be a failure. This assumption holds because of the fact that TIDOWA’s tests automatically terminate when they fail.
To allow the use of Docker within tests, the basic functionality of Pipelines has to be used by means of a Jenkinsfile, which is comparable to the build script above. To include Docker an additional Dockerfile is used. The specific Jenkinsfile used for TIDOWA has the following structure:
1 pipeline { 2 agent { dockerfile { filename 'Dockerfile' }} 3 stages { stage('Test') { steps { sh "ls" } } } 4 }
The Jenkinsfile uses the construct agent{} to specify the Docker container that should be used as execution environment for the build. In addition, a construct stage{} is defined to verify that the computation in Docker was successful, if tests had been executed. For the former construct, the Docker container is defined in the file Dockerfile. The following extract represents an exemplary configuration of a Dockerfile that can be used in combination with TIDOWA.
1 FROM ubuntu:18.04 2 RUN apt-get update -y && apt-get install -y python3.8 && apt install -y python3-pip && mkdir tidowa 3 COPY . tidowa
57 6 Continuous Integration in TIDOWA
4 5 RUN pip3 install -U pip numpy scipy sympy 6 7 WORKDIR /tidowa/libtidowa_src 8 RUN python3 ./setup.py develop && chmod -v 755 "/tidowa/bin/"* && chmod -v 755 ,→ "/tidowa/python_cleanup.sh" 9 10 WORKDIR /tidowa 11 ENV TIDOWA_ROOT="/tidowa" TIDOWA_DATA="/tidowa/data" TIDOWA_TEST="/tidowa/tests/convergence_tests" ,→ TIDOWA_CONV_TEST="/tidowa/tests/convergence_tests/test.sh" PATH="/tidowa/bin:${PATH}" 12 13 RUN bash $TIDOWA_CONV_TEST
The following explanations are based on [68]. This Dockerfile is used to set up an execution environment that is based on the Linux distribution Ubuntu 18.04. First, Python and pip is installed. After that the current status of TIDOWA is copied into the execution environment followed by installing the third-party modules required. Beginning at line 7, instructions concerning the specific setup of TIDOWA are executed. These can also be found in python_setup.sh and activate_env_vars.sh. For the future it is planned to offer additional execution environments, therefore allowing tests for the other languages and frameworks that are offered by TIDOWA.
58 7 Case Study: Usage of Test Frame- work
This case study examines the usage of the test framework. It describes the basic functionality offered by the system with the use of concrete examples. This chapter addresses future developers that want to use the framework or want to extend it with own tests.
7.1 Structure of Tests
As described in Chapter 6.2.6, all test cases concerning convergence are saved in the directory tests directory. Every individual test case, e.g. 10_test_ODE_conv_TI for the tuple (TI method: TI, IVP: IVP with ODE ODE), needs a script test.sh. This script contains parameters and common instructions that specify and define the test. An exemplary configuration of test.sh is depicted in the following code extract for 10_test_osc_conv_RK4:
1 #! /bin/bash 2 FOLDER=10_test_osc_conv_RK4 ODE=osc STEPS=10TI=RK4 3 LANGUAGE=CDT=10_test_osc_conv_RK4 4 5 bash $TIDOWA_TEST/10_conv_COMMON/generate_scripts .sh $FOLDER $ODE $STEPS $TI $LANGUAGE $DT || exit -1 6 bash $TIDOWA_TEST/10_conv_COMMON/run_scripts .sh $FOLDER || exit -1 7 bash $TIDOWA_TEST/10_conv_COMMON/post_processing .sh $FOLDER $TI || exit -1 8 bash $TIDOWA_TEST/10_conv_COMMON/post_processing_cleanup.sh || exit -1
The parameter FOLDER denotes the name of the directory, i.e. the name of the test case. ODE specifies the ODE and STEPS the number of steps performed by the TI method for the highest δt given. TI identifies the time integration method and DT refers to the name of the file in the directory 10_conv_DATA that contains the step sizes, which are used in the test framework. In the future, LANGUAGE should denote the programming language and framework, respectively, which TIDOWA should use to perform the computations. The following four instructions invoked with bash are mandatory for each test and represent the program flow of the test.
59 7 Case Study: Usage of Test Framework
7.2 Interaction with Tests
When an additional test is desired, one has to create an additional directory in tests directory. The name of the directory has to be conform with the required format, which has the following structure: 10_test_ODE_conv_TI. The preceding number is used to denote the priority of the test case. The directory has to contain a script test.sh that has to comply with previous definitions. As the selection of the tests is accomplished by using the names of the directories, one is able to temporarily deactivate the execution of a test by renaming it to 10__ODE_conv_TI. To remove tests permanently, one can delete the directory.
Notice that after an iteration of the test framework, a test directory 10_test_ODE_conv_TI contains multiple subdirectories, which themselves contain the result of previous computations. This approach is in place to avoid redundant computations. If a re-computation of this test is desired, e.g. with alternative values, one has to remove the entire content of the test directory except the test script.
One last aspect, which is described here is how exactly the step sizes are handed over to the test framework, more specifically its format. As already known, a file in the directory 10_conv_DATA can be used to accomplish this task. This file has to comply with two require- ments, namely a) one step size δt per line and b) one additional line at the end of the file. The name of the file is then announced using DT in test.sh.
7.3 Test Execution in Local Environment
This chapter demonstrates the usage of the test framework in a local environment while not using a CI tool to for example test own extensions before integrating them into the project. The following steps are required to run the test cases in a local environment (assuming that the required software is already installed):
• Setup of the required virtual environment: bash python_setup.sh • Activation of the virtual environment: source activate_env_vars.sh • Invocation of the test framework: bash $TIDOWA_CONV_TEST
60 8 Conclusion and Outlook
This thesis project took another step further in the development of TIDOWA. It examined the subject of continuous integration and compared a variety of CI implementations to select a CI implementation most suitable to be used for the project. Additionally, a test framework was implemented to allow future developers to test the correctness of the software concerning the behavior of convergence of the individual time integration methods. For that, first the mathematical basis concerning ordinary differential equations, time integration methods, errors and convergence were discussed. Prior to the implementation of the test framework required changes in TIDOWA itself were introduced in this thesis and the design of the test framework was presented along with the requirements that emerged during the work of the thesis. Furthermore, this thesis briefly demonstrated the usage of the test framework in a case study.
Nevertheless, this project still has not reached its full potential and provides enough pos- sibilities to further enhance and extend the software. Concerning the CI environment, an interesting aspect could be the further enhancement and usage of parallelism. Also integrating additional hardware like GPUs to the CI server and use them in the test framework could be an interesting project. Regarding the test framework, it is possible to further extend the test cases to examine and verify other characteristics of given time integration methods. Additionally, further extensions can be made to automate and improve the test selection and allow more variability in the tests.
The test framework and the CI environment form a first basis for further developers to verify the correctness of the program and expand its possibilities. This thesis project took a small step towards an environment comparable to those in large software development projects to support the future development of the software program TIDOWA.
61 8 Conclusion and Outlook Acknowledgments
With this section I want to acknowledge and thank special people who accompanied and inspired me while writing the thesis but also throughout my years of study and life.
Dr. Martin Schreiber for the help, the guidance and the constant support. Also in general for being a great advisor who always gave me helpful advice throughout the thesis.
Prof. Dr. Martin Schulz for giving me the opportunity to write my thesis at the Chair of Computer Architecture and Parallel Systems at the Technical University of Munich.
Taylor Lei for the support regarding TIDOWA at the beginning of this thesis.
My parents, my sister Phuong Anh & my brother Hoang Long for the support and motivation they provided throughout my life, especially while writing the thesis.
Andreas Dachsberger, Leonard Endriß & Simon Borowski for the time we shared and the fun we had during our years of studies.
Elizabeth (Lizzie) Olsen for being a true inspiration, almost a role model to me by showing that with dedication, hard work and kindness everything is possible.
Friends & People with whom I share special moments and who supported/helped me through- out my life, in particular Mohammad Soubeity for always supporting me and looking after me. Volkan Yildirim for being one of the nicest and kindest people I have ever met, for your support and for the fun we had. Jonas Hübner for the time we shared and the fun we always have (esp. together at Botlane in LoL). Tamara Srdic, also one of the nicest and kindest people I have ever met, for inspiring me to always do better and become a better person, to help other people plus to stay kind and humble. F.Göttler & S.Adelhardt for teaching me informatics and math exceptionally well, thus inspiring me to study it. S.Neubauer for her immense help and support back then during school, no matter the extra work for her.
62 Bibliography Bibliography
[1] Williams A. Adkins and Mark G. Davidson. Ordinary Differential Equations. Springer, 2012. isbn: 978-1-4614-3617-1.
[2] J. David Logan. A First Course in Differential Equations. 3. ed. Springer, 2015. isbn: 978-3-319-17851-6.
[3] Donal O’Regan and Ravi P. Agarwal. An Introduction to Ordinary Differential Equations. Springer, 2008. isbn: 978-0-387-71275-8.
[4] Hans-Joachim Bungartz. Numerical Programming (IN0019). 2019.
[5] Hans-Joachim Bungartz. Modeling and Simulation: An application-oriented introduction. Springer, 2014. isbn: 978-3-642-39523-9.
[6] Claus-Dieter Munz. Numerische Behandlung gewöhnlicher und partieller Differenzial- gleichungen: Ein anwendungsorientiertes Lehrbuch für Ingenieure. 4., verbesserte und überarbeitete Auflage. Springer Vieweg, 2019. isbn: 978-3-662-55885-0.
[7] Steven L. Garrett. Understanding Acoustics - An Experimentalist’s View of Sound and Vibration. Second edition. Springer, 2020. isbn: 978-3-030-44786-1.
[8] J. C. W. Locke et al. “Extension of a Genetic Network Model by Iterative Experimenta- tion and Mathematical Analysis”. In: Molecular Systems Biology 1 (2005), p. 2005.0013. doi: 10.1038/msb4100018.
[9] J. C. W. Locke, A. J. Millar, and M. S. Turner. “Modelling Genetic Networks with Noisy and Varied Experimental Data: The Circadian Clock in Arabidopsis Thaliana”. In: Journal of Theoretical Biology 234 (2005), p. 383. doi: 10.1016/j.jtbi.2004.11.038.
[10] Taylor Lei. Optimization of Parametrized High-Dimensional ODE Simulations (Bache- lor’s Thesis in Informatics). Technical University Munich. 2020.
[11] Severin Bals. Development of a Domain-Specific Language for the Efficient Time In- tegration of ODEs (Bachelor’s Thesis in Informatics). Technical University Munich. 2019.
[12] Martin Fowler - Continuous Integration. https://martinfowler.com/articles/continuousIntegration.html. (last accessed: 11.04.2021).
63 Bibliography
[13] Paul Duvall, Steve Matyas, and Andrew Glover. Continuous Integration: Improving Software Quality and Reducing Risk. 1st edition. Addison-Wesley Professional, 2007.
[14] Brent Laster. Continuous Integration vs. Continuous Delivery vs. Continuous Deploy- ment. 2nd edition. O’Reilly Media, Inc., 2020.
[15] Atlassian Bitbucket Pipelines - Continuous Delivery. (2021) [Software]: Atlassian. https://bitbucket.org/product/features/pipelines. (last accessed: 11.04.2021).
[16] Atlassian Bitbucket - Pricing. https://bitbucket.org/product/pricing. (last accessed: 11.04.2021).
[17] Atlassian Bitbucket Documentation - Get Started with Bitbucket Pipelines. https://support.atlassian.com/bitbucket-cloud/docs/get-started-with-bitbucket-pipelines. (last accessed: 11.04.2021).
[18] Atlassian Bitbucket Documentation - Use Pipelines in Different Software Languages. https://support.atlassian.com/bitbucket-cloud/docs/use-pipelines-in-different-software- languages. (last accessed: 11.04.2021).
[19] Travis CI - Test and Deploy with Confidence. (2021) [Software]: Travis CI GmbH. https://www.travis-ci.com. (last accessed: 11.04.2021).
[20] Travis CI - Pricing. https://www.travis-ci.com/plans. (last accessed: 11.04.2021).
[21] Travis CI Documentation - Overview. https://docs.travis-ci.com. (last accessed: 11.04.2021).
[22] Travis CI Documentation - Build Environment OS. https://docs.travis-ci.com/user/reference/overview. (last accessed: 11.04.2021).
[23] Travis CI Documentation - Build Environment CPU. https://docs.travis-ci.com/user/multi-cpu-architectures. (last accessed: 11.04.2021).
[24] AppVeyor - CI/CD service for Windows, Linux and macOS. (2021) [Software]: AppVeyor Systems Inc. https://www.appveyor.com. (last accessed: 11.04.2021).
[25] AppVeyor - Pricing. https://www.appveyor.com/pricing. (last accessed: 11.04.2021).
64 Bibliography
[26] AppVeyor Documentation - Overview. https://www.appveyor.com/docs. (last accessed: 11.04.2021).
[27] AppVeyor Documentation - Linux Starting Guide. https://www.appveyor.com/docs/getting-started-with-appveyor-for-linux. (last accessed: 11.04.2021).
[28] AppVeyor Documentation - Build Environment. https://www.appveyor.com/docs/build-environment. (last accessed: 11.04.2021).
[29] AppVeyor Documentation - Build Configuration. https://www.appveyor.com/docs/build-configuration. (last accessed: 11.04.2021).
[30] Buddy - The DevOps Automation Platform. (2021) [Software]: Buddy. https://buddy.works. (last accessed: 11.04.2021).
[31] Buddy - Pricing and Plans. https://buddy.works/pricing. (last accessed: 11.04.2021).
[32] Buddy Documentation - Overview. https://buddy.works/docs. (last accessed: 11.04.2021).
[33] Buddy Documentation - CI/CD Guide for Python. https://buddy.works/docs/quickstart/python. (last accessed: 11.04.2021).
[34] Buddy Documentation - Builds and Testing. https://buddy.works/docs/pipelines/builds-and-testing. (last accessed: 11.04.2021).
[35] Buddy Documentation - YAML / GUI Introduction. https://buddy.works/docs/yaml/yaml-gui. (last accessed: 11.04.2021).
[36] Jenkins. (Version 2.277.1 LTS) [Software]: Jenkins Project. https://www.jenkins.io. (last accessed: 11.04.2021).
[37] Jenkins Documentation - Overview. https://www.jenkins.io/doc/book. (last accessed: 11.04.2021).
[38] Jenkins Documentation - Windows. https://www.jenkins.io/doc/book/installing/windows. (last accessed: 11.04.2021).
[39] Jenkins Documentation - macOS. https://www.jenkins.io/doc/book/installing/macos. (last accessed: 11.04.2021).
65 Bibliography
[40] Jenkins Documentation - Linux. https://www.jenkins.io/doc/book/installing/linux. (last accessed: 11.04.2021).
[41] Jenkins Documentation - Java Requirements. https://www.jenkins.io/doc/administration/requirements/java. (last accessed: 11.04.2021).
[42] Jenkins Documentation - Docker. https://www.jenkins.io/doc/book/installing/docker. (last accessed: 11.04.2021).
[43] Bamboo - Pricing FAQ. https://www.atlassian.com/licensing/bamboo. (last accessed: 11.04.2021).
[44] Bamboo - Continuous Integration and Deployment Build Server. (Version 7.2) [Software]: Atlassian. https://www.atlassian.com/software/bamboo. (last accessed: 11.04.2021).
[45] Bamboo - Pricing. https://www.atlassian.com/software/bamboo/pricing. (last accessed: 11.04.2021).
[46] Bamboo - Features. https://www.atlassian.com/software/bamboo/features. (last accessed: 11.04.2021).
[47] Bamboo Documentation - Introduction. https://confluence.atlassian.com/bamboo/getting-started-with-bamboo. (last accessed: 11.04.2021).
[48] Bamboo Documentation - Supported Platforms. https://confluence.atlassian.com/bamboo/supported-platforms-289276764.html. (last accessed: 11.04.2021).
[49] Bamboo Documentation - System Requirements. https://confluence.atlassian.com/bamboo/bamboo-best-practice- system-requirements-388401170.html. (last accessed: 11.04.2021).
[50] TeamCity - the Hassle-Free CI and CD Server. (Version 2020.2) [Software]: JetBrains. https://www.jetbrains.com/teamcity. (last accessed: 11.04.2021).
[51] TeamCity - Pricing. https://www.jetbrains.com/teamcity/buy. (last accessed: 11.04.2021).
[52] TeamCity Documentation - Overview. https://www.jetbrains.com/teamcity/learn. (last accessed: 11.04.2021).
66 Bibliography
[53] TeamCity Documentation - Supported Platforms and Environments. https://www.jetbrains.com/help/teamcity/supported-platforms-and-environments.html. (last accessed: 11.04.2021).
[54] Python Packaging Authority - Python Packaging User Guide: Glossary. https://packaging.python.org/glossary. (last accessed: 11.04.2021).
[55] Python Packaging Authority - Python Packaging User Guide: Installing Packages Using Pip and Virtual Environments. https://packaging.python.org/guides/installing-using-pip-and-virtual-environments. (last accessed: 11.04.2021).
[56] Python Packaging Authority - Python Packaging User Guide: Installing Packages. https://packaging.python.org/tutorials/installing-packages. (last accessed: 11.04.2021).
[57] Python Packaging Authority - Python Packaging User Guide: Project Summaries. https://packaging.python.org/key_projects/#setuptools. (last accessed: 11.04.2021).
[58] Python 3.9.4 Documentation - Distutils. https://docs.python.org/3/library/distutils. (last accessed: 11.04.2021).
[59] Setuptools 56.0.0 Documentation - User Guide. https://setuptools.readthedocs.io/en/latest/userguide. (last accessed: 11.04.2021).
[60] Martin Schreiber et al. SWEET - Shallow Water Equation Environment for Tests, Awesome! https://github.com/schreiberx/sweet. (last accessed: 11.04.2021).
[61] SymPy - Overview. https://www.sympy.org. (last accessed: 11.04.2021).
[62] SymPy 1.8 Documentation - Introduction. https://docs.sympy.org/latest/tutorial/intro. (last accessed: 11.04.2021).
[63] SymPy 1.8 Documentation - ODE. https://docs.sympy.org/latest/modules/solvers/ode. (last accessed: 11.04.2021).
[64] SymPy 1.8 Documentation - Core. https://docs.sympy.org/latest/modules/core. (last accessed: 11.04.2021).
[65] Python 3.9.4 Documentation - Pickle: Python Object Serialization. https://docs.python.org/3/library/pickle. (last accessed: 11.04.2021).
67 Bibliography
[66] Docker - App Containerization. https://www.docker.com/resources/what-container. (last accessed: 11.04.2021).
[67] Docker Documentation - Language-Specific Guides: Python. https://docs.docker.com/language/python/build-images. (last accessed: 11.04.2021).
[68] Docker Documentation - Dockerfile Reference. https://docs.docker.com/engine/reference/builder. (last accessed: 11.04.2021).
68