» The Linux Foundation
An Introduction into Open Build Service (OBS)
...... By Rudolf Streif, The Linux Foundation June 2011
A Publication By The Linux Foundation Characters from the MeeGo Project (http://www.meego.com) http://www.linuxfoundation.org Background Any serious software organization sooner or later will have to solve the problem on how to reliably build their software for quality assurance, deployment and delivery. While local build environments using IDEs, scripts, etc. are suitable for single developers and possibly for small engineering workgroups demands quickly extend beyond the capabilities of such solutions when integration with other parts of the organization and defined interfaces for handing off deliverables are required. The organization then faces the task of adopting and implementing a build system. As frequently the case with any other decision, the approaches range from building your own solution to integrating a readily available system, either commercially or open-source. In this whitepaper, we will provide a definition of a build system, an overview over common requirements a build system has to fulfill and a walk through the typical build and packaging process for Linux distributions and their software packages. We will then introduce the Open Build Service (OBS) and its architecture. This paper is accompanied by two other papers, the first one demonstrating the usage of OBS with the MeeGo Build Service [1] and the second one providing step-by-step instructions on how to setup an OBS instance [2]. What is a Build System? A build system seeks to automate the typical steps of creating a software package from source code to the executable potentially including other files such as binary libraries, resource files, etc. Historically, software developers would use build scripts to call compilers and linkers. Within these script files they would pass command-line options to the tools to influence the process. It is simple and straightforward to pass one or a few source code modules to the compilers and linker for processing and create a deployable executable but as software systems grow and include outside binary libraries, resource files, etc. a more flexible way of building is required. Using make and make scripting e.g. a makefile offers a better alternative and also provides additional features such as checking and automatically building dependencies, makedepend, and incremental building. Developers quickly started to extend the make process with pre- and post-processing such as checking-out source files from repositories, calling external tools for packaging, deploying objects to targets etc. What started as automation tasks eventually evolved into workflow management and build automation and management solutions, whether commercial or open-source, are now providing developers, workgroups and entire organizations with the necessary power and flexibility to manage even the most complex software build processes. Build System Requirements While every organization has their own specific requirements for a build system to support their software engineering, quality assurance and deployment process, there is a core set of requirements that virtually everybody shares: • Repeatable – The software build process must yield the exact same results every time independent of who executes it. • Reproducible – It must be possible to take the entire build system and copy/move it from one
An Introduction into Open Build Service (OBS) 1
1796 18th Street, Suite C . San Francisco, CA 94107 . +1 415 723 9709 . http://www.linuxfoundation.org . computer/server to another and the new instance will again yield the exact same results. Repeatable and reproducible, while intrinsically different, are complementary. • Standardized – All of the projects follow the what either the organization and/or the industry has defined as their best practice. The core set of requirements is typically complemented by specific requirements to support a particular process or set of best practices: • Continuous Integration – Frequent builds either scheduled for instance nightly builds or triggered on every commit to a version control system. • Dependency Management – Resolution of build-time as well as run-time dependencies on other source and/or binary modules. • Incremental Build Processing – Only modified objects are built. • Build Acceleration – Parallel build with dependency management on build farms or computers with multiple processors or processor cores. • Documentation Generation – Automatically build accompanying documentation such as release notes, help packages, API descriptions, etc. • Test Integration – Execute test suites after the build has finished. • Deployment – Copy build results to deployment and/or download servers and/or perform installation on target systems. • Reporting – Generate detailed reports of the build process. • Scalability – Scale the build system as the requirements of the organization grow. The ultimate goal of a build system is to improve product quality by acceleration of the build process, elimination of redundant tasks, reducing dependency on key personnel all of which essentially leads to savings of time and money. From Source to Image Building an entire operating system such as a Linux distribution may possibly one of the most demanding tasks that a build system has to execute. It typically requires a 3-step process of Compiling, Packaging, and Imaging to create a final image from source that can be copied to a USB stick or a CD for booting by a computer (Figure 1). Each of these steps can further be decomposed into a series of more granular operations which is dependent on the particular entity processed by the step.
An Introduction into Open Build Service (OBS) 2
1796 18th Street, Suite C . San Francisco, CA 94107 . +1 415 723 9709 . http://www.linuxfoundation.org . Figure 1. From Source to Image
Each of the three steps compile, package and image requires three “ingredients” that must be provided by the developer. The ingredients vary by step but are somewhat similar: tools for processing, input to be processed, and a recipe for processing. The following paragraphs describe the three steps in a greatly simplified way to outline the process. It takes skill and care to correctly build binaries from sources, assemble packages from binaries and layout images from packages. However, once this is done OBS greatly automates the steps and creates consistent results.
Compile Step For the compile step tools is a toolchain typically comprised of a compiler, assembler, linker, make utility, autoconf etc. Linux distributions in general use the GNU toolchain provided by the Free Software Foundation (FSF) but other tools may be used or required dependent on what the project is. Of course, the developer must provide the source files, in general C/C++ source
An Introduction into Open Build Service (OBS) 3
1796 18th Street, Suite C . San Francisco, CA 94107 . +1 415 723 9709 . http://www.linuxfoundation.org . and header files, as the input. The recipe typically consists of autoconf scripts to automatically configure the source packages and other files telling the build system and the tools on how to process the input. The output of the compile step is a series of files in general statically and/or dynamically linkable libraries, executables, scripts etc.
Package Step After a project is compiled the output files need to be packaged using a packaging format suitable for the target distribution. For Linux distributions packages are typically created in RedHat’s RPM or Debian’s DEB formats. OBS supports both and other formats such as simple tarballs are possible too. Again, the developer must specify the packaging tools and the recipe on how the package is build including a manifest that contains meta information about the package.
Image Step A complete and operational Linux distribution consists of at least a couple of hundreds if not thousands of packages that are laid out in a particular fashion in one or more file systems. At a minimum the obligatory Linux kernel is required and some additional packages providing core functionality after the kernel is initialized. The image step combines these items and sets them up in a file system image that can be directly used by the target system for booting. Images can be built in various ways for different target systems and it is the developer’s responsibility to specify tools and recipe to create the image. OBS is integrated with the openSUSE KIWI Image System which provides a complete operating system image solution for hardware platforms supported by Linux as well as for virtualization systems like Xen, Qemu or VMWare. Introduction to OBS OBS is an open distribution development and deployment platform supporting use cases from the perspectives of users, software developers, distributors and independent software vendors (ISVs).
For Users openSUSE releases of packages and entire distributions are transparently built in the openSUSE Factory Project which includes automatic image creation. Users can find the latest distributions and packages for their distribution on http://software.opensuse.org/113/en. Mirrors around the globe ensure high availability for everyone.
For Developers OBS is an efficient platform for collaboration. Working together comes at ease with OBS’s project model and a complete infrastructure that relieves developers from setting up their own “compiler farms” for building for different architectures and multiple Linux distributions. Automatic resolution of dependencies avoids inconsistencies by triggering rebuilds of a package if one of its dependencies changes. Furthermore, open interfaces allow easy integration with different clients and external services. OBS itself provides Web-based client based on Ruby/Rails, accessible through https://build.opensuse.org, and an extensible command line client osc written in Python which is similar to the Subversion client.
An Introduction into Open Build Service (OBS) 4
1796 18th Street, Suite C . San Francisco, CA 94107 . +1 415 723 9709 . http://www.linuxfoundation.org . For Distributors and ISVs They can run their own build service based on OBS. The software is licensed under GPL and can be installed on any hardware meeting the minimum requirements. The easiest way of running an individual instance is to start with an OBS appliance. The appliance contains a fully functional all-in-one build service and can directly be installed on target hardware or run within a virtual machine. When the requirements grow, the appliance can be extended following a systematic approach into a fully distributed build service with database and repository backends running on different systems and a multitude of workers building packages for different architectures and distributions. More than a Build System OBS integrates a lot of functionality that is not typically found in a more traditional build system. However, for any serious software engineering organization that extends beyond pure code development that functionality is required to successfully validate, package and ship software products. That holds particularly true for embedded systems where the software, commonly called firmware, is integrated and delivered with the hardware. • Repository »» OBS provides integrated repository servers for sources and binaries. Since OBS’s atomic entity for managing sources is the Package which contains the individual files the source repository server is not a replacement for typical source control systems such as GIT, CVS, SVN, and so on which track changes on an individual file level. The OBS repositories track changes on the Package level only. However, source control systems for development can easily be integrated with the OBS workflow. »» Toolchains used for development are treated equally to any other Package. That is typical for Linux where the toolchain used to build the system is considered part of the build dependencies. It is somewhat different from the traditional embedded approach where toolchains to build the software for a particular target are treated as external. That obviously does not imply that a target distribution built with OBS for an embedded system must include the toolchain as it is the case for any other software components that are required to build the system versus the components required to run the system. OBS distinguishes, like any good build system, between build dependencies and runtime dependencies. • Project Management »» OBS utilizes Projects to organize Packages into logical units. A Project may include all Packages necessary to build an entire distribution or it may simply be the workspace for a developer only containing the Packages the developer is working on. »» Projects can depend on each other, so-called Subprojects, or Packages can depend on Packages contained in other Projects. OBS keeps track of the dependencies and automatically triggers rebuilds of depending entities, may it be Subprojects or Packages, if a dependency has changed hence maintaining consistency. »» Projects can be collaboration entities for multiple users and every OBS user has his/her own Home Project. The Home Project is a sandbox for the developers in which they can build their own Packages against other Projects in the OBS repositories without the risk of affecting those Projects.
An Introduction into Open Build Service (OBS) 5
1796 18th Street, Suite C . San Francisco, CA 94107 . +1 415 723 9709 . http://www.linuxfoundation.org . • Distributed Build Service »» OBS includes a build service that can scale from a single instance running on the same physical system together with all other OBS components to a distributed build farm with many instances distributed across multiple physical systems that do not necessarily have to be co-located. »» Build service instances can build natively for the architecture of their host system or use cross build toolchains to build for other architectures. »» Developers can schedule the Packages they are working on for building on the build service. • Local Build Environment »» OBS users can build Packages and entire Projects locally on their own systems. OBS will automatically download and install all required components for building including the toolchain in a chroot environment on the user’s system. There is no need for the developers to obtain, install and maintain the toolchain directly locally on their system or in a virtual machine. »» Local builds use the exact same build environment and toolchain as the build service. This ensures consistency between local and server builds avoiding typical build environment configuration discrepancies that may cause hard to identify issues. »» Local builds are supported on most common Linux distributions such as Debian, Fedora, MeeGo, openSUSE, and Ubuntu. • Packager »» OBS packages binaries into the package format used by the target distribution. It currently supports RPM and DEB package formats. • Collaboration Platform »» OBS supports many users to collaborate effectively and work concurrently on the same Projects. »» OBS monitors dependencies and continuously integrates changes whenever Packages are checked into the source repository. Every check-in of a Package triggers and automatic rebuild of the Package and all Packages depending on it. If the build of the Package or a depending package fails then the developer is notified. This continuous integration workflow avoids inconsistencies and diversion from the beginning. OBS combines many features that make it a universal tool for building and working with Linux distributions. • Scalability »» OBS scales from an all-in-one deployment installed from a ready-to-go appliance for individual developers and small workgroups to a fully distributed and redundant system supporting large organizations. • Continuous Integration »» Automatic builds are triggered on every check-in of a modified source Package and on every change of a dependency. »» Continuous integration and build also helps load balancing of the build farm and early detection of integration issues since builds happen on request and not during scheduled times.
An Introduction into Open Build Service (OBS) 6
1796 18th Street, Suite C . San Francisco, CA 94107 . +1 415 723 9709 . http://www.linuxfoundation.org . • Toolchain Management »» Toolchains are maintained within the OBS repository like any other Package. Consequently all Packages depend on the toolchain for build and a change in the toolchain will automatically trigger the rebuild of all other Packages. »» Build service builds as well as local developer builds utilize the exact same toolchain. Build service builds and local builds are always performed within chroot environment avoiding any issues with different system configurations. • Multi-platform Support »» OBS can build Projects and Packages for multiple platforms and architectures at the same time. »» Compiler flags and other settings can be individually defined per architecture. • Multi-distribution Support »» OBS can build Projects and Packages for multiple platforms and architectures at the same time. »» Compiler flags and other settings can be individually defined per architecture. »» OBS builds and integrates packages for multiple distributions and versions of distributions at the same time. »» Patches can be built and tested against Packages from multiple Projects. • Source-to-Image Workflow »» OBS supports the entire workflow from building binaries from source, over packaging binaries, configuration files, documentation, etc. to creating images for entire distributions. »» Diffs and patches are created automatically even when the developer uses sources that are checked-out locally to his/her development system. • Open Interfaces »» OBS provides a set of open and well-documented APIs allowing clients, IDE integrations and external services to interface with OBS components and utilize their resources. • Quality Control »» Automatic quality checks for instance rpmlint, etc. can be integrated with the OBS workflow for instant validation after building. • Publishing »» Built Packages together with their metadata for automatic installation, images, etc. can automatically be distributed to download servers. • Bottom-up or Top-Down Distribution Creation »» Build a distribution bottom-up starting with a toolchain and the Linux kernel. »» Use a stable distribution as a starting point either through interlinking an OBS instance with another OBS providing the distribution e.g. build.meego.com or build.opensuse. org, download on demand from another OBS using RPM or DEB packages, or manually importing packages. Behind the Scenes Conceptually, OBS is composed of a frontend and a backend. Figure 2 illustrates the OBS
An Introduction into Open Build Service (OBS) 7
1796 18th Street, Suite C . San Francisco, CA 94107 . +1 415 723 9709 . http://www.linuxfoundation.org . architecture. In the following subsections we examine the role of each of the elements in the OBS architecture.
Metadata Database
Workers Warden
Browser Repository WebUI Servers
Schedulers Dispatcher
Source API Server
Publisher
OSC
Signer
Figure 2. OBS Architecture
The Frontend The frontend connects OBS to clients via its WebUI and API. The WebUI is a human-to-machine interface based on Ruby/Rails. Users can connect to it with a web browser. Authentication and access control lists (ACL) provide the necessary security. The API offers machine-to-machine interface for the OSC command line client or for integration with other systems such as IDEs (for instance a plugin for Eclipse is available). The API can be accessed via two routes: 1. Using the base URL (e.g. https://api.opensuse.org) which requires proper authentication, or 2. Using the public interface (e.g. https://api.opensuse.org/public) which allows access as an anonymous user. The public interface is limited to certain operations such as read access only to publicly available repositories and projects.
The Backend
An Introduction into Open Build Service (OBS) 8
1796 18th Street, Suite C . San Francisco, CA 94107 . +1 415 723 9709 . http://www.linuxfoundation.org . The backend comprises several components of which some of them can be deployed in multiple instances to scale the system with increasing requirements.
The Metadata Database The metadata database stores information about projects, packages, users, user groups, access control and so forth. For performance and consistency reasons, the data is organized in a relational database schema stored in a MySQL database server. There is only one central metadata databse in an OBS deployment.
The Source Server There is one source server per OBS managing the sources, revisions, project/package metadata, package search, etc. The source server is the interface point of the backend for the frontend through its HTTP interface.
The Repository Server The repository server serves binary packages to the workers and the frontend API. It also receives built packages back from the workers and maintains the build history. If signing is enabled, it hands built packages to the signing server to calculate hashes and signatures. An OBS deployment can have more than one repository servers that can be distributed across multiple physical systems. A common deployment scheme utilizes one repository server per architecture. The repository server provides an HTTP interface through which it accepts requests.
The Scheduler Service For each architecture, OBS provides one scheduler service. The scheduler calculates the build dependencies across all packages in all projects and repositories and creates the build job descriptions. If sources have changed or build jobs have completed the scheduler receives an event that triggers the recalculation of the dependencies. Through this mechanism the scheduler takes care that all repositories and projects remain in a consistent state at all times.
The Dispatcher Job descriptions created by the schedulers are received by the dispatcher. There is one dispatcher per OBS which keeps track of the states of all worker in the system. When the dispatcher receives a job description it identifies a suitable worker to execute the job and notifies the worker about the job.
The Worker Once notified by the dispatcher aworker accesses the source server to retrieve the source files for a project or package and the binaries it depends on from the repository server. It then builds the package according to the build configuration. After the build has completed successfully it transfers the binary package to the repository server. The build capacity of an OBS deployment
An Introduction into Open Build Service (OBS) 9
1796 18th Street, Suite C . San Francisco, CA 94107 . +1 415 723 9709 . http://www.linuxfoundation.org . scales with the number of its workers. Obviously, workers can be deployed on multiple different physical systems. These systems then potentially define a worker’s capabilities. For instance, a worker deployed on an ARM-based system can build packages for ARM natively. On the other hand, a worker deployed on an x86-based system can build packages for x86 but may also be able to build packages for ARM if a cross-toolchain or emulation is available.
The Publisher The publisher’s responsibility is to make binary packages available to users for download and installation on their computers. It generates the necessary metadata for publishing and pushes the packages to download servers.
The Warden Process Workers can be monitored by an optional warden process. The warden process is a watchdog that monitors the activity of the workers. If a worker does not shut down cleanly after building or is otherwise stuck building a package the warden can terminate it and return it to the worker pool.
The Signer Process Signing binary packages is a recommended procedure to attest the integrity of the package and its source. This gives end users the confidence that a binary package has not accidentally or intentionally been corrupted or manipulated after it was built. OBS therefore provides a signer service to compute hashes and signatures for the binary packages. Conclusion Build systems capable of meeting the demands of increasingly complex software systems and expanding organizations require an initial as well as continuous investment of the organization looking to deploy them. However, the advantages are undeniable and have been proven many times hence the question is not whether an organization can afford a deploying a build system but whether it cannot. Even for small software engineering organizations it makes very much sense to use a build system early on. Migration at a later point in time is always possible but typically more disruptive. Planning a software organization with a build system as part of the process from the beginning is simplified by a build system such as OBS which easily scales over time. While there are other solutions available, OBS is an excellent solution for organizations developing software for Linux. The fact that it is open-source and freely available is an added bonus lowering the barrier of entrance.
An Introduction into Open Build Service (OBS) 10
1796 18th Street, Suite C . San Francisco, CA 94107 . +1 415 723 9709 . http://www.linuxfoundation.org . MeeGo and OBS Training The Linux Foundation offers many training courses around Linux: http://training.linuxfoundation.org/. For MeeGo and OBS the following courses provide in-depth information and hands-on experience: • LF363 Portable Application Development for MeeGo Devices • LF361 Hands-on Approach to MeeGo Internals and OBS Build System • LF273 MeeGo Training For Non-Developers Training courses are offered in various formats including Classroom Training, Online Training, On-Site Corporate Training and Custom Corporate Training. Please contact the Linux Foundation (http://training.linuxfoundation.org/contact/solutions) for more information.
References [1] Building MeeGo with OBS – An introduction to the MeeGo Build Infrastructure, Rudolf Streif, The Linux Foundation [2] OBS Setup and Administration – Step-by-step Instructions to your own OBS Instance, Jan-Simon Moeller and Rudolf Streif, The Linux Foundation [3] Open Build Service Wiki: http://en.opensuse.org/Portal:Build_Service [4] Open Build Service Instance: https://build.opensuse.org/ [5] MeeGo Build Infrastructure: http://wiki.meego.com/Build_Infrastructure [6] MeeGo Build Service Instance: http://build.meego.com/ [7] MeeGo Community Build Service Instance: https://build.pub.meego.com/ About the Author Rudolf Streif manages The Linux Foundation’s initiatives for embedded solutions working with the community to provide environments and platforms for embedded Linux systems. About MeeGo The MeeGo project is the open source software platform for the next generation of computing devices. MeeGo combines Intel’s Moblin™ and Nokia’s Maemo projects into one Linux-based platform. MeeGo will be deployed across many computing device types - including pocketable mobile computers, netbooks, tablets, mediaphones, connected TVs and in-vehicle infotainment systems, and brings together the leaders in computing and mobile communications as the project’s backers. MeeGo is designed for cross-device, cross-architecture computing and is built from the ground up, for a new class of powerful computing devices. The Linux Foundation’s MeeGo workgroup welcomes contributors and encourages developers and others to get involved at the site or find out more about joining the Linux Foundation at http://www.linuxfoundation.org/meego. Moblin and Maemo contributors are encouraged to participate at MeeGo.com.
An Introduction into Open Build Service (OBS) 11
1796 18th Street, Suite C . San Francisco, CA 94107 . +1 415 723 9709 . http://www.linuxfoundation.org . The Linux Foundation promotes, protects, and advances Linux by providing unified resources and services needed for open source to successfully compete with closed platforms.
To learn more about The Linux Foundation, the MeeGo project or our other initiatives, please visit us at http://www.linuxfoundation.org/.