PERFORMANCE EVALUATION AND OPTIMIZATION OF CONTINUOUS INTEGRATION BASED AUTOMATED TOOLCHAIN FOR SAFETY RELATED EMBEDDED APPLICATIONS SOFTWARE Master Thesis Submitted in Fulfilment of the Requirements for the Academic Degree M.Sc. Dept. of Computer Science Chair of Computer Engineering Submitted by: Zain Ullah Student ID: 395153 Date: 25.10.2016 Supervising tutor: Prof. Dr. W. Hardt Donny Masril [Baumer Hübner GmbH] Abstract Continues Integration has been a vital part of software development process in order to make the development process fast and reliable. There are number of actors which play an important role with support of third party tools that helps the development process to be effective and productive in nature. The CI- toolchain is capable of doing much more than the compilation of the software project which covers the daily life tasks of the developers like testing, documentation etc. The important part of automated toolchain is the conversion of source code artifacts into executables with the help of the build system. The selection of proper build system is a matter of subjective in nature and it depends upon the number of factors that should be analyzed before proceeding forward towards the selection mechanism. This thesis focuses on software rebuilding and proves practically with experiments that could help developers and managers to decide between two important software build systems SCons and CMake. It has been experimentally proved that what are the conditions and situations where SCons performs better and what are the moments where it is wise to select CMake as a build tool. At first, individual build tools are evaluated in terms of scalability, conveniency, consistency, correctness, performance (in terms of speed and targets) and later, the build systems are experimented by automating the workflow by increasing the source code artifacts to evaluate the performance when there is limited user interaction. The behavior of the build systems are also tried with other third party tools like Tessy for testing purposes, Jenkins as CI server, and Polarion as requirement engineering tool to show how much effort is required to integrate third party tools with the build system in order to increase the functionality. The evaluation of the build systems is important because that will highlights the areas where potential candidates are better and where there is lack of functional specifications. Generally speaking, SCons has an advantage of being Pythonic in nature and provides the developer ease of use to specify the build configurations using programmatic skills. CMake on other hand are on top of shelves where there is no need to understanding and caring about the underlying platform and where developers want to generate the native build tool solutions which are readily available for exporting them into IDEs. Though both of the build systems has different goals, for example SCons is ready to sacrifices the performance while providing user correctness of the build while CMake focuses on generating native build tools by understanding the underlying platform. All of these types of situations are discussed with experiments in this thesis and serves as the practical guides for high level managers to decide the build tools among others. After evaluation, this thesis firstly suggests the general techniques where the bottlenecks could be covered and then build tool specific optimizations and recommendations are discussed to speed-up the development process. Keywords: Build systems, SCons, CMake, Performance Evaluation, Optimization 1 Acknowledgments With the utmost respect, I would like to thank my immediate supervisor and mentor, Mr. Donny Masril who have each left an indelible mark on my life, and for that I am humbled and eternally grateful. Donny, you have motivated me not only to set big goals, but to put into motion a plan of action to achieve them with your enthusiasm, talent, and dedication which are truly awe-inspiring. I would also like to thank my University supervisor Prof. Dr. Wolfram Hardt, at TU Chemnitz who has become personal role model of mine, exemplify the type of strong work ethic and commitment to quality that I can only hope to emulate. I would like to dedicate this work to my family and friends. Without your support, this thesis would not have been possible. 2 Table of Contents 1. Introduction ..................................................................................................... 9 1.1 Motivation ................................................................................................................................................ 9 1.2 Contributions ......................................................................................................................................... 10 1.3 Organization .......................................................................................................................................... 10 2. Background .................................................................................................... 11 2.1 Safety Critical Systems ........................................................................................................................... 12 2.2 Continues Integration ............................................................................................................................. 12 2.3 Build Automation ................................................................................................................................... 13 2.4 Toolchain & Components ....................................................................................................................... 15 2.4.1 Version Control System ......................................................................................................... 16 2.4.2 Continues Integration Server (Jenkins) .................................................................................. 17 2.4.3 Build and test Server ............................................................................................................... 18 2.4.4 System Test Platform ............................................................................................................. 19 2.4.4.1 Architecture ............................................................................................................... 20 2.4.4.2 Testing framework .................................................................................................... 20 2.4.4.3 Output files ................................................................................................................ 21 2.4.5 Polarion ALM ......................................................................................................................... 23 2.4.5.1 Requirement Engineering ......................................................................................... 23 2.4.5.2 Test Management ...................................................................................................... 24 2.4.5.3 Bug and Issue tracker ................................................................................................ 25 2.5 Design Goals ......................................................................................................................................... 26 2.5.1 Convenience ........................................................................................................................... 26 2.5.2 Consistency .............................................................................................................................. 26 2.5.3 Correctness .............................................................................................................................. 27 2.5.4 Scalability ................................................................................................................................ 27 2.5.5 Flexibility................................................................................................................................. 27 2.5.6 Performance ............................................................................................................................ 27 2.6 Build System Features .......................................................................................................................... 28 3 2.6.1 Partial compilation .................................................................................................................. 28 2.6.2 Variant Builds ......................................................................................................................... 28 2.6.3 Configuration .......................................................................................................................... 29 2.6.4 Parallel building ...................................................................................................................... 29 2.6.5 Distributed builds .................................................................................................................... 30 2.7 Summary ............................................................................................................................................... 30 3. Build Systems ................................................................................................. 31 3.1 Build System Philosophy ...................................................................................................................... 31 3.2 Build Machines ....................................................................................................................................