BitKeeper for Kernel Developers Val Henson Jeff Garzik [email protected] [email protected] Abstract per project, normally checked in changes di- rectly to the central repository, and could eas- BitKeeper1 is a revolutionary new distributed ily communicate with other developers work- source control management suite which is ideal ing on the same part of the code. Unsurpris- for Linux kernel development. BitKeeper pro- ingly, the source control software written un- vides tools which automate and simplify many der these assumptions was not very useful for common kernel development tasks. In this pa- thousands of loosely connected developers dis- per, we describe basic BitKeeper concepts and tributed world-wide. operations, BitKeeper solutions for common The BitKeeper distributed source control sys- kernel development problems, and a work- tem was designed for, written for, and tested flow for interacting with other Linux develop- by Linux kernel developers. Linux kernel de- ers using BitKeeper. We also discuss some of velopment provided the perfect test case for BitKeeper’s shortcomings and what is being a truly distributed source control system, and done to correct them. We conclude that Bit- BitKeeper has been and continues to be shaped Keeper can dramatically improve the efficiency by input from kernel developers. As a result, of Linux kernel developers. it is preeminently useful for kernel develop- ment. The purpose of this paper is to famil- 1 Introduction iarize kernel developers with the most useful and time saving features of BitKeeper, so that developers can spend less time on mechani- A new source control system is available - why cal make-work and more time on development. should Linux kernel developers care? Because After reading this paper, developers new to Bit- this particular source control system was de- Keeper may consider trying BitKeeper for the signed from the ground up to solve exactly first time, and developers already using Bit- the problems inherent in Linux kernel devel- Keeper may learn a few new tricks. opment. Kernel developers need to manage thousands of files, live and work all over the First, we’ll briefly review basic BitKeeper con- world, often have limited bandwidth and con- cepts and operations (experienced BitKeeper nectivity, and frequently merge large numbers users should skip this section). We’ll then of changes. Older source control systems were examine a variety of problems frequently en- designed for a development model where most countered during kernel development and show developers worked in the same physical build- how BitKeeper solves these problems. Next, ing and had 24-hour access to a central reposi- we’ll review the workflow involved in using tory over high bandwidth local networks. The BitKeeper for Linux kernel development. Fi- developers, rarely numbering more than 100 nally, we’ll discuss some of the shortcomings of BitKeeper and what is being done to correct 1BitKeeper is a trademark of BitMover, Inc. Ottawa Linux Symposium 2002 198 them. 2 Basic BitKeeper Concepts This section presumes knowledge of basic source control concepts such as “check in” and Figure 1: Parent pointers after cloning. “check out.” We will instead concentrate on the ways in which BitKeeper is different from tra- ditional source control systems. Some of the major differences between the architecture of traditional source control systems and the ar- chitecture of BitKeeper exist in order to satisfy one of its key design requirements: Developers should be able to commit work locally, without accessing a remote repository, until the devel- oper is ready to merge with the remote reposi- Figure 2: Parent pointers after changing with tory. Some other key design goals were repro- “bk parent”. ducibility, data integrity, and performance. 2.1 Running BitKeeper 2.2 Clones, parents, and children First, let’s go over the nuts and bolts of using A BitKeeper repository is a collection of BitKeeper: How do you get it, and how do you source controlled files. To create a working run it? Download BitKeeper by going to: copy or the equivalent of a CVS sandbox, a developer “clones” a repository. The word http://www.bitkeeper.com “clone” was chosen because cloning a reposi- tory creates an exact copy of the original repos- And clicking on “Downloads.” All BitKeeper itory. All of the information and administra- commands are of the form “bk <command>” tive files in the original repository are included to avoid namespace clashes. BitKeeper has in the new repository, making it possible to built-in help, just run “bk helptool” (for work in any repository completely indepen- the GUI tool) or “bk help” (for the com- dent of any other repository. After the clone mand line tool). While BitKeeper has many is completed, the new repository regards the useful graphical tools, a developer can work original repository as its “parent.” The parent with BitKeeper using only the command line of a tree can be changed at any time, to any tools - BitKeeper does not require a windowing other related tree, or to no tree at all (see Fig- environment. We also recommend that first- ures 1, 2). Because each repository is identi- time users run the demo, which is at: cal, any repository can be cloned, and the child of one repository can also be the parent of an- other repository (see Figure 3). Note, how- ever, that despite all the parent-child terminol- http://www.bitkeeper.com/Test.html ogy, BitKeeper repositories interact on a peer- Ottawa Linux Symposium 2002 199 Figure 3: Example BitKeeper repository structure. to-peer basis, since the relationship between 2.4 Push and pull any two trees can be changed at any time. Changesets are exchanged between reposi- 2.3 Changesets tories using “bk push” and “bk pull”. (Note that commits modify only the local In BitKeeper, changes to individual files are repository, and do not affect the parent reposi- grouped together into changesets. A changeset tory.) Push will send changesets from the child is a grouping of one or more deltas to one or to the parent, and pull will retrieve change- more files representing a single logical change. sets from the parent to the child. Each push Each changeset can contain multiple deltas to or pull only sends the changesets which are the same file. Each revision to each file in the present in one tree but not in the remote tree changeset is commented, as is the changeset (see Figure 4). A push will only send changes as a whole. Logically related changes to sep- which are already merged with the changes in arate files can now be explicitly grouped to- the remote tree, so merging with another tree is gether. For example, if one bug fix requires done by first pulling the remote tree’s changes, changes to three different files, all three files’ merging them in the local tree, and then push- changes can be grouped into one changeset. ing the merged changes back. Push and pull Being able to explicitly group changes together will by default push to or pull from the parent rather than guessing at their relationships (from of the local tree, but these commands can also last modified date, or location in the same di- take an argument specifying a different tree to rectory) is very useful. Even more useful is that push to or pull from. each changeset is an automatic synchronization point, similar to a CVS tag. Users can repro- Clones can be thought of as creating a per- duce the exact state of the repository as of the sonal, private, unnamed “branch,” and pulls as point that any changeset was committed. a convenient way of merging with the “trunk” Ottawa Linux Symposium 2002 200 Figure 4: Example of a push: Initial clone, commit a change, push it back. Ottawa Linux Symposium 2002 201 without pushing the “branch’s” changes to the changeset created by the merge, along with “trunk.” The push-pull model gives developers your comments. For developers who don’t control over whether or not local changes are trust auto-merging, “bk pull” has an option pushed to other repositories without sacrific- to disable the auto-merge feature. Each conflict ing ease of synchronization with other reposi- can then be individually hand-merged or auto- tories. CVS users will enjoy the freedom of be- merged and the results approved before be- ing able to commit half-finished changes with- ing committed. We recommend that develop- out breaking the main tree. ers try BitKeeper’s auto-merge algorithm even if they have had bad experiences with auto- 2.5 Conflict resolution merging in the past; the new algorithm is an immense improvement over all previous algo- Usually, a push or a pull that changes a lo- rithms and, in the authors’ experience, always cally modified file will be auto-merged by merges changes correctly. BitKeeper. In typical use, BitKeeper auto- What we just described is only the most com- merges approximately 95% of conflicts that mon kind of conflict, a conflict in the data of would not have been merged by CVS or diff the file itself, or a content conflict. BitKeeper and patch. The percentage of successful also resolves conflicts in many other file at- merges relative to CVS actually increases with tributes: permissions, ownership, type, path- the number of developers working on the same name, and more. Viewing the pathname of a repository. BitKeeper improves the auto-merge file as just one more file attribute makes it easy rate in two ways. First, each merge is only to move files around within a BitKeeper repos- done once - CVS remerges from the point itory. where the trunk and the branch first separated every time a branch’s changes are pushed back to the trunk.