An Introduction Into Open Build Service (OBS)

» The Linux Foundation An Introduction into Open Build Service (OBS) .................. By Rudolf Streif, The Linux Foundation June 2011 A Publication By The Linux Foundation Characters from the MeeGo Project (http://www.meego.com) http://www.linuxfoundation.org Background Any serious software organization sooner or later will have to solve the problem on how to reliably build their software for quality assurance, deployment and delivery. While local build environments using IDEs, scripts, etc. are suitable for single developers and possibly for small engineering workgroups demands quickly extend beyond the capabilities of such solutions when integration with other parts of the organization and defined interfaces for handing off deliverables are required. The organization then faces the task of adopting and implementing a build system. As frequently the case with any other decision, the approaches range from building your own solution to integrating a readily available system, either commercially or open-source. In this whitepaper, we will provide a definition of a build system, an overview over common requirements a build system has to fulfill and a walk through the typical build and packaging process for Linux distributions and their software packages. We will then introduce the Open Build Service (OBS) and its architecture. This paper is accompanied by two other papers, the first one demonstrating the usage of OBS with the MeeGo Build Service [1] and the second one providing step-by-step instructions on how to setup an OBS instance [2]. What is a Build System? A build system seeks to automate the typical steps of creating a software package from source code to the executable potentially including other files such as binary libraries, resource files, etc. Historically, software developers would use build scripts to call compilers and linkers. Within these script files they would pass command-line options to the tools to influence the process. It is simple and straightforward to pass one or a few source code modules to the compilers and linker for processing and create a deployable executable but as software systems grow and include outside binary libraries, resource files, etc. a more flexible way of building is required. Using make and make scripting e.g. a makefile offers a better alternative and also provides additional features such as checking and automatically building dependencies, makedepend, and incremental building. Developers quickly started to extend the make process with pre- and post-processing such as checking-out source files from repositories, calling external tools for packaging, deploying objects to targets etc. What started as automation tasks eventually evolved into workflow management and build automation and management solutions, whether commercial or open-source, are now providing developers, workgroups and entire organizations with the necessary power and flexibility to manage even the most complex software build processes. Build System Requirements While every organization has their own specific requirements for a build system to support their software engineering, quality assurance and deployment process, there is a core set of requirements that virtually everybody shares: • Repeatable – The software build process must yield the exact same results every time independent of who executes it. • Reproducible – It must be possible to take the entire build system and copy/move it from one An Introduction into Open Build Service (OBS) 1 1796 18th Street, Suite C . San Francisco, CA 94107 . +1 415 723 9709 . http://www.linuxfoundation.org . computer/server to another and the new instance will again yield the exact same results. Repeatable and reproducible, while intrinsically different, are complementary. • Standardized – All of the projects follow the what either the organization and/or the industry has defined as their best practice. The core set of requirements is typically complemented by specific requirements to support a particular process or set of best practices: • Continuous Integration – Frequent builds either scheduled for instance nightly builds or triggered on every commit to a version control system. • Dependency Management – Resolution of build-time as well as run-time dependencies on other source and/or binary modules. • Incremental Build Processing – Only modified objects are built. • Build Acceleration – Parallel build with dependency management on build farms or computers with multiple processors or processor cores. • Documentation Generation – Automatically build accompanying documentation such as release notes, help packages, API descriptions, etc. • Test Integration – Execute test suites after the build has finished. • Deployment – Copy build results to deployment and/or download servers and/or perform installation on target systems. • Reporting – Generate detailed reports of the build process. • Scalability – Scale the build system as the requirements of the organization grow. The ultimate goal of a build system is to improve product quality by acceleration of the build process, elimination of redundant tasks, reducing dependency on key personnel all of which essentially leads to savings of time and money. From Source to Image Building an entire operating system such as a Linux distribution may possibly one of the most demanding tasks that a build system has to execute. It typically requires a 3-step process of Compiling, Packaging, and Imaging to create a final image from source that can be copied to a USB stick or a CD for booting by a computer (Figure 1). Each of these steps can further be decomposed into a series of more granular operations which is dependent on the particular entity processed by the step. An Introduction into Open Build Service (OBS) 2 1796 18th Street, Suite C . San Francisco, CA 94107 . +1 415 723 9709 . http://www.linuxfoundation.org . Figure 1. From Source to Image Each of the three steps compile, package and image requires three “ingredients” that must be provided by the developer. The ingredients vary by step but are somewhat similar: tools for processing, input to be processed, and a recipe for processing. The following paragraphs describe the three steps in a greatly simplified way to outline the process. It takes skill and care to correctly build binaries from sources, assemble packages from binaries and layout images from packages. However, once this is done OBS greatly automates the steps and creates consistent results. Compile Step For the compile step tools is a toolchain typically comprised of a compiler, assembler, linker, make utility, autoconf etc. Linux distributions in general use the GNU toolchain provided by the Free Software Foundation (FSF) but other tools may be used or required dependent on what the project is. Of course, the developer must provide the source files, in general C/C++ source An Introduction into Open Build Service (OBS) 3 1796 18th Street, Suite C . San Francisco, CA 94107 . +1 415 723 9709 . http://www.linuxfoundation.org . and header files, as the input. The recipe typically consists of autoconf scripts to automatically configure the source packages and other files telling the build system and the tools on how to process the input. The output of the compile step is a series of files in general statically and/or dynamically linkable libraries, executables, scripts etc. Package Step After a project is compiled the output files need to be packaged using a packaging format suitable for the target distribution. For Linux distributions packages are typically created in RedHat’s RPM or Debian’s DEB formats. OBS supports both and other formats such as simple tarballs are possible too. Again, the developer must specify the packaging tools and the recipe on how the package is build including a manifest that contains meta information about the package. Image Step A complete and operational Linux distribution consists of at least a couple of hundreds if not thousands of packages that are laid out in a particular fashion in one or more file systems. At a minimum the obligatory Linux kernel is required and some additional packages providing core functionality after the kernel is initialized. The image step combines these items and sets them up in a file system image that can be directly used by the target system for booting. Images can be built in various ways for different target systems and it is the developer’s responsibility to specify tools and recipe to create the image. OBS is integrated with the openSUSE KIWI Image System which provides a complete operating system image solution for hardware platforms supported by Linux as well as for virtualization systems like Xen, Qemu or VMWare. Introduction to OBS OBS is an open distribution development and deployment platform supporting use cases from the perspectives of users, software developers, distributors and independent software vendors (ISVs). For Users openSUSE releases of packages and entire distributions are transparently built in the openSUSE Factory Project which includes automatic image creation. Users can find the latest distributions and packages for their distribution on http://software.opensuse.org/113/en. Mirrors around the globe ensure high availability for everyone. For Developers OBS is an efficient platform for collaboration. Working together comes at ease with OBS’s project model and a complete infrastructure that relieves developers from setting up their own “compiler farms” for building for different architectures and multiple Linux distributions. Automatic

Load more