Version + Patches = Buildout
Total Page:16
File Type:pdf, Size:1020Kb
Understanding Buildout In SlapOS Details Buildout is a Python-based system for creating, assembling and deploying applications from multiple parts, which may or not be Python-based. It allows to create Buildout configuration files from which the exact same software can (later) be reproduced. Buildout originated from the Zope/Plone community to automate the deployment of customized instances of their own software. Lead by Jim Fulton (CTO of Zope Corporation), Buildout has become a stable and mature software over the years (source code). Buildout is an integral part of SlapOS. The following design document explains what Buildout is and how Buildout files are used in SlapOS before discussing advantages and alleged shortcomings of Buildout. For more information on the way Buildout is integrated in the SlapOS ecosystem, refer to the SlapOS architecture introduction. Table of Content Introducing Buildout Simple Buildout Example Buildout.cfg Walkthrough Advantages of Buildout Details Introducing Buildout Details Buildout is a tool for automating software assembly including running build tools to build software and applying software and templates to generate configuration files and scripts Buildout is applicable to all phases of software development and in production deployment. It is based on three core principles: Repeatability, Componentization and Automation. slapos.buildout (Gitlab repository) is a fork of zc.buildout maintained by SlapOS which includes various fixes that may or may not have been merged back into Buildout upstream (see Buildout developer guidelines for how to contribute to Buildout). Why Buildout? Details Buildout is used in SlapOS to define what software to install and instantiate on a node and is therefore key to the architecture of SlapOS. Buildout is one answer to the fact that there will never be a possible software standard in the cloud: every system gets patched. From large scale production systems operating in the cloud or privately to single components like relational databases. Patching is required to meet performance requirements of given applications as soon as data volume grows. If a cloud operating system does not provide the possibility to patch any of its software components, it is simply unusable for large scale production environments. SlapOS is usable because its definition of what it considers a software is based on the possibility of patching any dependent software component - which is done using Buildout. Version + Patches = Buildout Details It is often assumed giving a name to a software (like "KVM" or "MySQL") is sufficient. While it actually is for many cases (and SlapOS provides aliases "KVM" and "MySQL" which link to explicit Buildout configurations), the reality is not as straightforward. For example, some releases of KVM support NBD protocol over IPv6 while some don't. Some support Sheepdog distributed block storage while others don't. Some support CEPH distributed block storage and so on. Most users using KVM do not care about IPv6, Sheepdog or CEPH, but users of SlapOS need IPv6 support to access NBD which at the time of writing is only available as a patch. A resilient storage based on Sheepdog was only made available from version 0.13 and CEPH support requires another patch. For MySQL, the above possibilities multiply with multiple sources from the official one M( ySQL), MariaDB, Percona InnoDB or Cubrid, which is not MySQL but claims to be 90% compatible. Each source has different versions and default compilation options. Maintainers of large scale applications know very well that the performance of applications can be seriously impacted by subtle changes to the SQL optimizer so the possibility to choose which source of MySQL to use and which patches to apply are crucial when running enterprise applications. Details As the number of possibilities to patch and the specific user requirements can be close to infinite, the easiest way to know which software is actually being installed on a SlapOS node is simply to list where the original source code was obtained from and which patches were applied. This is exactly what Buildout does with a few lines of configuration. It also eliminates the process of distributing binary packages on a wide range of hardware architectures thanks to a trusted, distributed caching mechanism which does not even use a centralized signature. Simple Buildout Example Details A Buildout configuration is based on a .cfg file called the buildout profile. This profile may contain multiple sections, called parts, starting with the [part name] syntax. The first part is always called [buildout] and allows to define the list of other parts to execute. Each other part is executed by a python script called a recipe. If the recipe is not available in the system, it will automatically be downloaded by Buildout. Bootstrapping Buildout > mkdir buildoutenv && cd buildoutenv > vi buildout.cfg > curl -s htp://svn.zope.org/*checkout*/zc.buildout/trunk/bootstrap/bootstra p.py | python -S - Generated script 'bin/buildout'. > ./bin/buildout > ls eggs ipython-0.10.1-py2.6.egg zc.recipe.egg-1.3.2-py2.6.egg Details Buildout has to be bootstrapped in order to allow it to initialize the deployment environment. Once done, ab in/buildout binary becomes available and has to be run to execute the respective profile. The executable should be available to run this program in the bin directory (bin/ipython). Buildout.cfg // buildout.cfg [buildout] parts = demo [demo] recipe = zc.recipe.egg eggs = ipython Details In this example, the recipe zc.recipe.egg is used to install python packages (called eggs). When executed, this recipe will download the egg called 'ipython' (along with its dependencies) and install it. Compiling Binaries // buildout.cfg [buildout] parts = demo [demo] recipe = hexagonit.recipe.cmmi url = htp://gondor.apana.org.au/~herbert/dash/fles/dash-0.5.6.1.tar.gz Details Buildout can be used to compile C/C++ code using gcc. To do this the recipeh exagonit.recipe.cmmi can be used. It takes the URL of the source package as parameter. When executed, the source package will be automatically downloaded, extracted, configured, compiled and installed in the parts directory. [...] > ./bin/buildout > ls parts/demo/bin dash Running Multiple Parts // buildout.cfg [buildout] parts = ipython dash [dash] recipe = hexagonit.recipe.cmmi url = htp://gondor.apana.org.au/~herbert/dash/fles/dash-0.5.6.1.tar.gz [ipython] recipe = zc.recipe.egg eggs = ipython Details Of course multiple parts can configured in the profile. Their execution order follows the parts parameter (ipython is executed before dash). Buildout.cfg Walkthrough Details The following section will walk through a more complex Buildout.cfg file explaining how the different sections will be used to build a specific software release. For more information on Buildout, please refer to the Buildout getting started page. You can find additional examples of SlapOS component Buildout configuration files in theS lapOS repository component folder. These components are used to build more complex software (Buildout files can use other Buildout files)N. ote, that while the following file may be outdated, it still explains many concepts and thus serves as a good example of describing how SlapOS uses Buildout. The file being used is an old buildout.cfg file for SlapOS (version 0.69). Once you have read through this section, you could compare this file to the latest version (version 1.0.63) to see whether you can apply the principles to a different but related file. What is Buildout (1) [buildout] extends = ../../stack/shacache-client.cfg ../lxml-python/buildout.cfg ../python-2.7/buildout.cfg parts = slapos find-links = http://www.nexedi.org/static/packages/source/slapos.buildout/ versions = versions Details The first section of a Buildout is named [buildout]. It defines which software is going to be built and how. In the case of SlapOS Buildout we can first see that SlapOS Buildout extends existing Buildout definitions, namely shacache-client, lxml-python and python2.7 itself. It then defines the parts variable which lists the sections which are going to be required as part of the build process. A section in a Buildout file is defined by a name inside square brackets. [buildout], [lxml-python], [slapos] and [versions] are the four sections of this Buildout file. The find-links variable defines a repository of eggs. Eggs is a standard distribution mechanism for python packages. Buildout itself is written in python and can be extended using python language and eggs. Since not all eggs are published in Python Packing Index (PyPI), it is sometimes necessary to provide additional repositories of eggs. One should note here that the find-links variable is a generic variable used by python setuptools, the python module in charge of managing python distributions. Details In our case we use this find-links variable to override the default Buildout distribution with SlapOS's own Buildout. SlapOS extends Buildout solving minor issues (support of distributed network caching and small bug fixes), which are not yet integrated in the default Buildout. However, since Buildout profiles are self contained, it is possible to specify which version of Buildout to use (in the [versions] section discussed below) and where to find that version through find-links, as long as this version of Buildout made by SlapOS is published somewhere. The versions variable specifies which Buildout section is used to define specific versions of python distributions (eggs). Being able to specify the version of each egg is required in order to make sure that the exact same software is installed on every node of a distributed cloud and that no uncontrolled upgrade will happen. The allow-hosts variable is used to specify explicitly which sources of python distributions (eggs) are accepted. It is a useful variable to make sure that the Buildout process does not get interrupted by lost connectivity to unreliable sites containing python distributions.