sign in sign up
<<
Home , Linus Torvalds, Linux Kernel, Mm tree

Cover Intro: Kernel Tricks COVER STORY KERNELPromising projects from the kernelTRICKS community

Developers are constantly looking for new ways to interact with the versatile . This month we study some innovative projects leading deep into kernel space.

BY JON MASTERS, JAN RÄHM, AND JOE CASAD

ost Linux users know they can tion. At the heart of that distribution is is really doing, take a look at Figure 1. rely on the kernel without ever the Linux kernel itself – a complex col- The kernel appears in the center of the Mgiving it much thought. But if lection of software routines that takes diagram, surrounded by a sample of the you do look a little deeper, you’ll find control whenever a machine first boots operations it assists in performing. some evolving technologies that extend and periodically (many times each sec- On the left, hardware from the kernel in interesting ways. This ond) handles the resource requirements devices such as disks, network cards, month we cover some innovations at of applications, manages system hard- sound devices, graphics cards, and so on the edges of the Linux kernel. ware devices, and generally does a lot arrive at unpredictable times (data has Our first article describes the new of the low level dirty work. perhaps been requested, but there is no userspace driver model in upcoming ker- The Linux kernel is regarded as a fully way to know exactly when the device nel versions. In our second story, Klaus functional, complex, and well-docu- will actually deliver that data). In a simi- Knopper looks at block device compres- mented role model for source soft- lar way, the system timer (on most sys- sion with the Cloop module, which lets ware. Everyone has access to the source Live CD developers put up to 2GBs on a code, which is updated daily; and ev- 700MB CD. We’ll also examine kernel- eryone can join in with dis- based with KVM, and cussions on the mailing we’ll end with a look at the Flash Trans- lists or submit lation Layer and the LogFS flash filesys- patches. tem. To under- We hope you enjoy this month’s stand what Kernel Tricks cover story collection. But the kernel first, on for a brief introduction to the brain of Linux. The Linux Kernel The term Linux, in the strictest sense, does not refer to the shiny collection of software components and applications that is often known as a Linux distribu-

COVER STORY

Userspace Drivers ...... 28

Cloop ...... 32

KVM ...... 37

LogFS ...... 40

JANUARY 2008 ISSUE 86 21

021-026_coverintro.indd 21 16.11.2007 13:32:04 Uhr COVER STORY Cover Intro: Kernel Tricks

opment, Linux 2.4 was released in the late . Linux 2.4 was significant because it was the first kernel release widely used by the masses – not just computer enthusiasts, but also those try- ing Linux for the first time on their desk- top computers, servers, and even embed- ded gadgets. Version 2.4 was also used in the new generation of “Enterprise” distributions from major Linux vendors. Once Linux 2.4 was safely out of the way, work began in earnest on a 2.5 de- velopment series, leading up to a major 2.6 kernel release a few years ago. Version 2.6 was a complete revolution for Linux. It had a rewritten scheduler (that can scale to many thousands of CPUs by now), a full device manage- Figure 1: The Linux kernel manages processes, hardware interrupts, and housekeeping tasks. ment subsystem with greatly improved and re-written dynamic device support, tems) fires regularly to allow Linux to kernel itself should be thought of more and literally countless other improve- perform periodic housekeeping opera- as a collection of software routines in a ments that came as a result of the grow- tions many times per second. privileged library. These specially privi- ing acceptance of Linux as a viable com- A kernel would not be particularly leged routines are always executed in mercial offering. By this point, features useful if it were not able to also service response to specific events, and at that weren’t coming in from enthusiasts at the user in the provision of support for time, whatever else was running is tem- nearly the same rate that they were running programs (called tasks, within porarily saved as the system switches being offered by well-known Linux the world of the Linux kernel). On the into kernel code. Most of the time, ker- hackers employed by Linux vendors and right side of the diagram are three tasks. nel code runs either as a result of timers companies like IBM. Each represents a different kind of appli- and hardware interrupts, or as the result cation. The first, at the bottom, is a regu- of a request from an application. It also Linux Is Not Just lar user program – in this case, a runs at startup and in various error con- Although it should be obvious by now, shell running on a user’s desktop. Most ditions. Linux is not the day-to-day product of programs don’t communicate with the one man’s efforts. is kernel directly. Instead, they use the A Little History widely recognized as the inventor and GNU Library functions, which in turn The Linux dates back head of the Linux kernel project, but call standard kernel interfaces in order to Linus Torvalds’ first experiments in Linus serves more as a highly skilled to provide required services. 1991. At that time, Linux was merely a project manager at this point (Figure 2). Sometimes, an application does not toy operating system alternative. It ran From a qualitative point of view, Linus use the C library but talks to the kernel only on more directly instead. This is the case 386-compatible with certain low-level, statically linked processors; how- tools – for example, busybox (commonly ever, it was very used as a recovery tool, as well as in em- hackable, and it bedded gadgets such as the N810 wasn’t long be- Internet Tablet). In the diagram, you can fore Linux was see one of these special tools providing available for a the services of init. Finally, at times the wide variety of kernel runs special kernel code as if it different ma- were almost a regular program (but with chines – first privileges). came the more The key point of Figure 1 is that a ker- generic PC sup- nel is far from magical. Its behaviors and port, but thereaf- processes can be explained. Many peo- ter, developers ple believe that a kernel is somehow began to port “running” all the time and constantly Linux to other overseeing system operations. Although architectures. Figure 2: Linus Torvalds at the Kernel 2007: the founding it is true that Linux does perform a vari- After almost a father of the Linux kernel still makes a major contribution to its devel- ety of system monitoring functions, the decade of devel- opment.

22 ISSUE 86 JANUARY 2008

021-026_coverintro.indd 22 16.11.2007 13:32:47 Uhr Anzeige Nokia

021-026_coverintro.indd 23 16.11.2007 13:33:07 Uhr COVER STORY Cover Intro: Kernel Tricks

When you read the document, you may Linus’ repository. Git is the Linux notice some gaps and that the column kernel source management utility writ- contains an s for orphaned status. These ten by Linus Torvalds. orphaned projects are projects that no- body is currently working on. Of course, Tree Structure the developer community is always The kernel tree structure consists of happy for somebody to step in and as- four main trees above the branches or sume responsibility for an orphaned de- subtrees: main 2.6.x, 2.6.x.y-stable, velopment sector, but if you are new to 2.6.x-git, and 2.6.x-mm. Linus Torvalds kernel development, you should proba- maintains the main 2.6.x tree. When a bly consider sticking to bug fixes and new kernel version is imminent, - patches before attempting the main- valds opens a two-week window in Figure 3: Andrew Morton is the number two tainer role. which developers can send him their in kernel development. His -mm branch diffs. provides the impetus for new features in Linux Day-to-Day In nearly every case, changes destined the Linux kernel. Development for the main tree have been tested in the Linux kernel development takes place on -mm kernel over a period of days or may not be the main code developer; a minute-by-minute basis, around the weeks. Andrew Morton’s 2.6.x-mm tree however, he still coordinates releases clock, every day of the year. Central differs from Torvald’s Vanilla kernel in and work on the kernel, contributes to co-ordination happens using the Linux that it contains untested changes by mailing lists, and takes an active part Kernel Mailing List (LKML), an archive Morton himself or resulting from patches in discussions on new functions. of which you can find at lkml. org. reviewed by Morton. Morton integrates Below Linus Torvalds in the kernel Although general Linux kernel develop- changes to all subsystems and patches hierarchy are a number of maintainers ment discussion happens on the LKML, from the mailing list with his version. supervising work on different parts of literally dozens of other popular email- The -mm branch is regarded as a play- the kernel. A maintainer decides on the ing lists cover each of the many features ground for new developments and fea- development roadmap, typically in col- within Linux. tures. Once a has proved its reli- laboration with other developers. An- The central repository of Linux kernel ability, it is very likely to it into drew Morton (Figure 3) is often consid- source is held at kernel. org, to which Torvalds' kernel tree. ered the second in command of the ker- only a limited number of people have Under normal circumstances, Linus nel development team. Morton main- direct access, but from which ev- publishes the first release candidate at tains the -mm kernel branch, as well as eryone can freely download the Linux the end of the two-week window. From various subprojects, such as the develop- source itself. Most developers, however, then, major changes are typically ruled ment of the netdev driver and the don’t download source archives from out, although Linus Torvalds might and filesystems. kernel. org. They instead “pull” from make an exception in case of critical The MAINTAINERS document in the main branch of the kernel source pack- Constructive Contributions age lists the various maintainers associ- Before you send bug reports or patches In addition to an unequivocal de- ated with Linux kernel development. to the kernel community, it makes sense scription, submitters should note the to keep to a couple of rules: Patches following: Large patches that solve Which Linux Kernel? must be available as diffs. If you have complex problems are easier for Many variants of the Linux kernel are clearly identified and removed the bug, the maintainer to investigate if you currently in use. For many reasons – the maintainer of the subsystem affected split them into several smaller sub- including timing of individual product by the patch is the right person to con- patches. Maintainers are unlikely to releases – few distributions ship exactly tact. An undocumented patch is more have much time to investigate work the same release of the kernel, and each or less useless. Add a description of the by a newbie near the end of a merge may apply different patches that modify bug, the patch itself, and the effect of the window. the kernel in some way. For example, patch to make it easier for the maintainer Because kernel developers always check last-minute updates might need to be to investigate your proposal and, at the patches before accepting them, you made to various sound drivers just be- same time, approve your chances of it should keep your patches as simple and fore release, and these may not yet be in being accepted. transparent as possible. Unfinished the official Linux kernel from kernel. org. Claims such as “This patch makes 2000 patches accompanied with a promise to In some cases, Linux vendors apply lines of code superfluous …” or “I have fix or complete later are typically ignored their own optimizations as a means of tested this patch on five different architec- or dropped into a black hole. For more adding value. Whatever the reason for tures…” will grab the maintainer's atten- information, read the documents apply- the differences, you should know that it tion. But mails saying things like: “I’ve ing-patches.txt, SubmittingPatches, is unlikely that your Linux system is run- been doing it this way for 20 years, so pay SubmittingDrivers ,and SubmitChecklist ning the current “official” Linux kernel attention …” are likely to get on the main- in the kernel source Documentation posted at kernel. org. tainer's nerves. directory.

24 ISSUE 86 JANUARY 2008

021-026_coverintro.indd 24 16.11.2007 13:33:07 Uhr Cover Intro: Kernel Tricks COVER STORY

drivers or security patches. If everything works as planned, a second release can- didate appears after another week. The differences between the two are typically no more than cosmetic code modifica- tions. After the sixth release candidate, the is normally completed, and the next stable version of the Linux ker- nel is ready. The 2.6.x.y-stable tree, the kernel ver- sions with four-digit release numbers, see only minor changes, with the focus on security fixes. This branch is consid- ered particularly reliable. If no version has a four-digit number, the highest 2.6.x version is typically the most stable. The 2.6.x-git tree is a daily snapshot of Torvald’s kernel tree on the Git ver- sion control system initiated by Linus himself. This tree is far more experimen- tal than a release candidate. The - Figure 4: Linux is maintained through the git code management tool, which was shots are taken automatically without designed and developed by Linus himself. developer interaction. list of all of the public git repositories at To obtain a copy of this development Pulling from the git Tree git. kernel. org. Browse Linus’ git tree at: tree for your own uses, you first need If you want to experiment with living http:// git. kernel. org/ ?p=linux/ kernel/ to make sure you have the git tools in- like a kernel developer, you can obtain a git/ torvalds/ linux-2. 6. git;a=summary. stalled. Linux distributions typically 16 PROCESSOR CORES • Quad AMD Opteron 8000 • 2U rackmount • Serial ATA or SAS RAID 16 cores from £3,758 + VAT • Up to 64GB of memory 8 cores from £2,573 + VAT • Linux OS Prices correct as of 16-10-2007

DNUK is one of the UK’s leading suppliers of workstations, servers and storage systems designed and optimised for the GNU/Linux based operating systems. From scientific to e-commerce applications, our products can be used as building blocks to create complete solutions. We’ve been building Linux computers since September 1998. www.dnuk.com [email protected] 0161 343 5333

021-026_coverintro.indd 25 16.11.2007 13:33:16 Uhr COVER STORY Cover Intro: Kernel Tricks

include the git tools, which you can install using:

$ yum install git U # Fedora and OpenSUSE systems $ apt-get install git U # and systems

You can also install these tools from source, which is available directly from git. kernel. org. Once you have installed git, you can use it to obtain the latest release of Linus’ code tree:

$ git cloneU git://git.kernel.org/ pub/ U scm/ linux/kernel/ git/ torvalds/ U linux-2.6.git linus-2.6

This command will place a copy of Linus’ kernel in a local directory called linus-2.6. To update this copy to the lat- Figure 5: The Linux Kernel Archive at kernel.org is the epicenter for Linux kernel development. est version, at any time, simply go to that directory and type: tion. Don’t forget to experiment with the can just study the code to learn more gitk utility if you would like a graphical about the art of kernel programming. $ git pull source browser. Once you have copied the code to the linus-2.6 directory, you Further Reading You can find a complete set of git com- can build the kernel as usual (see the If you are getting started with Linux ker- mands in the git reference documenta- box titled “Building the Kernel”), or you nel development, try joining the Linux Kernel Newbies project http://www. Building the Kernel kernelnewbies. org/ . Here, you can meet The steps for building the Linux kernel $ make with like-minded people who help each may vary from one distribution to an- After saving the kernel, you can build it other learn about the Linux kernel. You other (your distribution will supply an using: might also want to try the Kernel Jani- example of how to do this for modern $ make tors Project – an ongoing effort at mak- 2.6 Linux kernels). This brief summary ing various trivial cleanups to the Linux and install the updated kernel modules is intended for illustration purposes. using: kernel source tree (a great way to get Consult your vendor documentation. started). You’ll even find a kernelnew- $ make modules_install In the case of a Fedora-like system, you’d bies Facebook group. Finally, you will need to rebuild any first want to install the standard Fedora Although there is no substitute for kernel config file from your /boot directory: (initrd/initramfs) that you are using. Your distribution will reading the Linux Kernel Mailing List U $ cp /boot/config- have more detail, but in the case of (LKML) if you want to keep on top of `uname -r` .config a Fedora system, try a command similar individual patches as they are posted, most people don’t have nearly enough $ make oldconfig to: U time in their day for such things. Espe- Linux supports a variety of make targets, $ mkinird /boot/initrd= cially those people who are not paid to including both oldconfig, menuconfig, initrd-2.6.x.y 2.6.x.y follow the development so closely. This and many others. The oldconfig com- where x.y reflects whatever version is why online resources such as Linux mand will pull in existing selections from of the 2.6 Linux kernel you just built. the .config file that was supplied by your You will then need to update your boot- Weekly News (LWN) were created. For a Linux vendor, though there will likely be loader (such as grub) by editing /boot/ small contribution of less than US$ 10 some differences in configuration options grub/menu.lst to reference your new per month, you can read the latest (and (you will be prompted about these, and kernel before you can try test booting highly detailed) summaries of kernel de- offered some help in deciding appropriate it. Most major Linux vendor distributions velopment from Jonathan Corbet (au- options), so don’t get too flustered when include tools to help with this. Fedora thor of Linux Device Drivers and a long- these come up. Once you have imported provides the new-kernel-pkg command, time Linux kernel hacker). Even if you an existing configuration, you can also which automates the last two steps don’t feel like subscribing, you can al- run the menuconfig target for a visual (rebuilding the initrd and installing the ways read postings from previous weeks menu of available configuration options: modified grub bootloader entries). for free online. I

26 ISSUE 86 JANUARY 2008

021-026_coverintro.indd 26 16.11.2007 13:33:23 Uhr