Building Linux Software with Conary

Building Linux Software with Conary Michael K. Johnson rpath, Inc. [email protected] Abstract of Conary terminology and design, you may want to read the paper Repository-Based Sys- tem Management Using Conary, in Proceed- This paper describes best practices in Conary ings of the Linux Symposium, Volume Two, packaging: writing recipes that take advan- 2004, kept updated at http://www.rpath. tage of Conary features; avoiding redundancy com/technology/techoverview/, which with recipe inheritance and design; implement- introduces Conary’s design and vocabulary in ing release management using branches, shad- greater detail. Terms called out in boldface in ows, labels, redirects, and flavors; and design- this paper without an explicit definition are de- ing and writing dynamic tag handlers. It defined in that overview. scribes how Conary policy prevents common packaging errors. It provides examples from our rpath Linux distribution, illustrating the design principles of the Conary build process. It 1 Conary Source Management then describes the steps needed to create a new distribution based on the rpath Linux distribu- Unlike legacy package management tools, tion, using the distributed branch and shadow Conary has integral management for source features of Conary. code and binaries, and the binaries are directly associated with the source code from which Conary is a distributed software management they have been built. system for Linux distributions. Based on exten- sive experience developing Linux distributions Conary stores source files in source compo- and package management tools, it replaces tra- nents, and then uses a recipe (described later) ditional package management solutions (such to build binary components that it can in- as RPM and dpkg) with one designed to enable stall on a system. While most of the code loose collaboration across the Internet. It en- that handles the two kinds of components is ables sets of distributed and loosely connected actually the same, the interface is different. repositories to define the components which are The source components are managed using a installed on a Linux system. Rather than having Software Configuration Management (SCM) a full distribution come from a single vendor, it model, and the binary components are managed allows administrators and developers to branch using a system management model. a distribution, keeping the pieces which fit their environment while grabbing components from The SCM model for managing source com- other repositories across the Internet. ponents is sufficiently familiar to experienced Concurrent Versioning System (cvs) or Subver- If you do not have a basic working knowledge sion (svn) users; you can create a new source • 249 • 250 • Building Linux Software with Conary component, check out an existing source com- Conary also has two features that support con- ponent, add files to a source component, re- verging source code bases: shadows that act move files from a source component (this is a like branches but support convergent instead of single action in Conary, unlike cvs’s two-step divergent development by providing intentional operation), rename files in a source component tracking semantics, and redirects that allow (like svn, but unlike cvs), and commit the cur- redirecting from the current head of the branch rent set of changes since the last commit (like (or shadow) to any other branch (or shadow), svn, and like cvs except that the commit is including but not limited to that branch’s par- atomic). ent. The redirect is not necessarily permanent; the branch can be revived. A redirect can even Conary has not been optimized as a complete point to a different name package entirely, and SCM system. For example, we do not use it so is useful when upstream names change, or to manage subdirectories within a source com- when obsoleting one package in favor of an- ponent, and instead of importing source code other. into vendor branches, we import archives and apply patches to them. Conary may some- 1.1 Cooking with Conary day support a richer SCM model, since there are no significant structural or design barriers within Conary. It was a choice made for the Using a recipe to turn source into binary is sake of simplicity and convenience, and to fo- called cooking. cus attention on Conary as a system management tool rather than as an SCM tool—to en- The exact output produced by building source courage using Conary to track upstream devel- code into a binary is defined by several factors, opment, rather than encourage using it to create among them the instruction set (or sets) that the forks. compiler emits, and the set of features selected to be built. Conary encodes each combination In addition, Conary stores in the repository of configuration and instruction set as a flavor. (without cluttering the directory in which the The configuration items can be system-wide or other files are stored) all files that are refer- package-local. When cooking, Conary builds enced by URL. When they are needed, they are a changeset file that represents the entire con- downloaded (from the repository if they have tents of the cooked package. been added to the repository; otherwise, via the URL) and stored in a separate directory. Then, There are three ways to cook: when committing, Conary stores those auto- matically added source files (auto-source files) • A local cook builds a changeset on the in the repository so that the exact same source special local@local:COOK branch. It is always available, enabling repeatable builds. loads the recipe from the local filesystem, and can cook with recipes and sources that Like cvs, Conary can create branches to sup- are not checked into the repository. It will port divergent development (forks); unlike cvs, download any automatic sources required those branches can span repositories, and the to build. repository being branched from is not modified, so the user only needs write privileges in the • A repository cook builds a transient repository in which the branch is created. Un- changeset on the same branch as the like cvs (and any other SCM we are aware of), source component, and then commits it to 2005 Linux Symposium • 251 the repository. It loads the source compo- be copied from recipe to recipe. This example nent (including the recipe and all sources) is truly representative of the the most common from the repository, not the local filesys- class of recipes; the great majority of packag- tem. It finds all automatic sources in the ing tasks do not require any further knowledge repository. The same version can be built of how recipes work. In other words, this exam- into the repository multiple times with dif- ple is representative, not simplistic. New pack- ferent flavors, allowing users to receive the agers tend to find it easy to learn to write new build that best matches their system flavor Conary packages. when they request a trove. However, some programs are not designed for • An emerge builds a transient change- such easy packaging, and many packagers have set on the special local@local:EMERGE become used to the extreme complexity re- branch, and then commits it to (that is, quired by some common packaging systems. installs it on) the local system. Like a This experience can lead to writing needlessly repository cook, it takes the recipe and complex and thereby hard-to-maintain recipes. all sources from the repository, not the This, in turn, means that while reading the filesystem. (This is the only kind of cook RPM spec file or Debian rules for building a that Conary allows to be done as the root package can be an easy way to find a resolution user.) to a general packaging problem when you are writing a Conary recipe, trying to translate either of them word-by-word is likely to lead to a 2 The Conary Recipe poor Conary package. The internal structure of objects that underlie All software built in Conary is controlled Conary recipes makes them scale gracefully through a recipe, which is essentially a Python from simple recipes (as in the example) to com- module with several characteristics. Here is an plex ones (the kernel recipe includes several example recipe, which has a typical complexity independent Python classes that make it eas- level:1 ier to manage the kernel configuration process). Some of the more complex recipe possibilities require a deeper structural understanding. class MyProgram(PackageRecipe): name = ’myprogram’ version = ’1.0’ def setup(r): r.addArchive( ’http://example.com/%(name)s-%(version)s.tar.gz’) • The recipe module contains a class r.Configure() r.Make() that is instantiated as the recipe object r.MakeInstall() (MyProgram in the example above). This class declares, as class data, a name string The goal of Conary’s recipe structure is not to that matches the name of the module, a make all packaging trivial, but to make it possi- version string, and a setup() method. ble to write readable and maintainable complex This class must be a subclass of one of a recipes where necessary, while still keeping the small family of abstract superclasses (such great majority of recipes extremely simple— as PackageRecipe). and above all, avoiding boilerplate that needs to • Conary calls the recipe object’s setup() 1No, I do not like two-column formatting for techni- method, which populates lists of things cal papers, either. to do; each to-do item is represented by 252 • Building Linux Software with Conary an object. There are source objects, string; the substitution is done within the func- which represent adding an archive, patch, tions you call. It also means that macros can or source file; and build objects, which reference each other. Be aware that changes to represent actions to take while building the macros object all take place before any list and installing the software.

Load more