Proceedings of the Linux Symposium Volume Two July 23rd–26th, 2008 Ottawa, Ontario Canada Contents Where Linux Kernel Documentation Hides 7 Rob Landley Tux meets Radar O’Reilly—Linux in military telecom 19 Grant Likely & Shawn Bienert A Symphony of Flavours: Using the device tree to describe embedded hardware 27 Grant Likely & Josh Boyer Tux on the Air: The State of Linux Wireless Networking 39 John W. Linville AUGEAS—a configuration API 47 David Lutterkort ‘Real Time’ vs. ‘Real Fast’: How to Choose? 57 Paul E. McKenney If I turn this knob. what happens? 67 Arnaldo Carvalho de Melo Performance Inspector Tools with Instruction Tracing and Per-Thread / Function Profiling 75 M. Milenkovic, S.T. Jones, F. Levine, & E. Pineda Containers checkpointing and live migration 85 A. Mirkin, A. Kuznetsov, & K. Kolyshkin Building a Robust Linux kernel piggybacking The Linux Test Project 91 Subrata Modak Have You Driven an SELinux Lately? 101 James Morris Coding Eye-Candy for Portable Linux Devices 115 Bob Murphy SELinux for Consumer Electronics Devices 125 Yuichi Nakamura PATting Linux 135 Venkatesh Pallipadi & Suresh Siddha Pathfinder—A new approach to Trust Management 145 Patrick Patterson & Dave Coombs Linux Data Integrity Extensions 151 Martin K. Petersen Red Hat Linux 5.1 vs. CentOS 5.1: ten years of change 157 D. Hugh Redelmeier Measuring DCCP for Linux against TCP and UDP With Wireless Mobile Devices 163 Leandro Melo Sales Smack in Embedded Computing 179 Casey Schaufler Energy-aware task and interrupt management in Linux 187 V. Srinivasan, G. Shenoy, D. Sarma, S. Vaddagiri, & V. Pallipadi Choosing an application framework for your Linux mobile device 199 Shreyas Srinivasan & Phaneendra Kumar SCSI Fault Injection Test 205 Kenichi Tanaka A Survey of Virtualization Workloads 215 A. Theurer, K. Rister, & S. Dobbelstein Thermal Management in User Space 227 Sujith Thomas A Model for Sustainable Student Involvement in Community Open Source 235 Chris Tyler A Runtime Code Modification Method for Application Programs 245 Kazuhiro Yamato SynergyFS: A Stackable File System Creating Synergies between Heterogeneous Storage Devices 255 Keun Soo Yim Live Migration with Pass-through Device for Linux VM 261 Edwin Zhai Conference Organizers Andrew J. Hutton, Steamballoon, Inc., Linux Symposium, Thin Lines Mountaineering C. Craig Ross, Linux Symposium Review Committee Andrew J. Hutton, Steamballoon, Inc., Linux Symposium, Thin Lines Mountaineering Dirk Hohndel, Intel Gerrit Huizenga, IBM Dave Jones, Red Hat, Inc. Matthew Wilson, rPath C. Craig Ross, Linux Symposium Proceedings Formatting Team John W. Lockhart, Red Hat, Inc. Gurhan Ozen, Red Hat, Inc. Eugene Teo, Red Hat, Inc. Kyle McMartin, Red Hat, Inc. Jake Edge, LWN.net Robyn Bergeron Dave Boutcher, IBM Mats Wichmann, Intel Authors retain copyright to all submitted papers, but have granted unlimited redistribution rights to all as a condition of submission. Where Linux Kernel Documentation Hides Rob Landley Impact Linux [email protected] Abstract Coping with such an enormous slush pile is fundamen- tally an editorial task. Only after finding and indexing Last year my day job was documenting the Linux kernel. the existing mass of documentation can anyone figure It seemed like a simple task: read through the kernel’s out whether what’s out there for a given topic is com- Documentation directory, fill in the holes, and make sure plete and up-to-date. You can’t fill in the holes without it all stays up to date. In reality, I found most kernel doc- first knowing where they are. umentation lives in web pages, magazine articles, online This paper is not an attempt to summarize seven months books, blog entires, wikis, conference papers, videos, of reading about why UTF-8 is a good internationaliza- standards, man pages, list archives, commit messages, tion format or how to load firmware out of initramfs for and more. The real problem is editorial: only after find- a statically linked device driver. It’s about turning the ing and indexing the existing documentation can we fig- dark matter of kernel documentation into something you ure out whether it’s up to date and what’s missing. can browse. In this talk I’ll list the documentation sources I found, show my attempts at organizing them (on http:// 2 The editorial task kernel.org/doc), and explain what would be nec- essary to really get this issue under control. Google is great at finding things, but it doesn’t tell you what to search for. Google is not a reference work that allows one to see available topics and home in on an 1 Introduction area of interest, moving from more general to more spe- cific. But there’s more to teaching someone English than In 2007 the Linux Foundation awarded me a fellowship handing them a dictionary: a good reference work is not to deal with the ongoing lack of good Linux kernel doc- necessarily a good tutorial. Adding a reference index to umentation. Unfortunately, a good problem statement a tutorial is fairly easy, turning a reference into a tutorial isn’t necessarily a good plan of attack. I spent most of is harder. But creating a reference from scratch is also the next seven months just trying to figure out what “fix- easier than creating a tutorial from scratch, a reference ing it” actually meant, and how to go about it. The re- can be little more than a collection of sorted links, while sults may be found at http://kernel.org/doc. a tutorial has to make sense. Indexing the web’s Linux kernel documentation just to My first big surprise was that the kernel’s Documenta- provide a comprehensive reference is a huge undertak- tion/ directory is just the tip of the iceberg. Most ker- ing. Even keeping an index up to date after its creation nel documentation lives in web pages, magazine arti- would be a project on par with any other major kernel cles, online books, blog entries, wikis, conference pa- subsystem. But without knowing here the holes are, pers, audio and video recordings of talks, standards, writing new documentation to try to fill in those holes man pages, list archives, commit messages, and more. tends to reinvent the wheel. The real problem isn’t a LACK of documentation, it’s that no human being can ever hope to read more than a In the absence of an obvious place to go on the web to tiny fraction of what’s said, written, and recorded about find the most up-to-date existing documentation, newly the kernel on a daily basis. Merging all this raw data created documentation tends to be repetitive and over- into the kernel tarball would be like trying to burn the lapping. Half-hearted attempts to collate what’s avail- internet to CD. able into a single new comprehensive version often just • 7 • 8 • Where Linux Kernel Documentation Hides add one more variant to the pile. A well-meaning editor is the central repository for all kernel documentation, or who isn’t already an expert on a given topic probably could easily be turned into such. This mistaken assump- won’t create a new category killer document attracting tion cost me about 3 months. patches instead of competition. If they didn’t feel like First of all, the Documentation directory isn’t even contributing to someone else’s existing document, why the only significant source of documentation within would the next well-meaning editor be different? the kernel tarball itself. The kerneldoc entries in More to the point, author and editor are different jobs. the source code (used by make htmldocs) and Researching and writing new documentation isn’t really the kconfig help entries (used by the help option of an editorial task. An editor collects and organizes the make menuconfig) are each significant and com- submissions of others. A real editor spends most of pletely separate sources of documentation. The files their time wading through a slush pile and saying “no” in Documentation/ seldom if ever refer to htmldocs or to most of it, or saying “no” to things their assistant ed- menuconfig help, and those seldom refer back to it (or itors pass up to them. to each other). If this sounds familiar, it’s because fighting off stur- This other information cannot easily be migrated to the geon’s law is what editors do. Open source project Documentation directory. The other sources of docu- maintainers perform an editorial job, publishing regu- mentation in the kernel source are usually located near lar editions of source code anthologies. Putting together the things they document, to benefit from locality of ref- Linux distributions is another layer of editorial filter- erence. There’s a reason they live where they do, as do ing. Open source developers are already familiar with over two dozen README files in the source code, the this process. Applying it to documentation, providing a output of make help, references to IETF RFC docu- brief summary and a pile of links to existing documents ments in source comments, and so on. is more effective than trying to become an expert on ev- In addition, the data formats are different. Documenta- ery single topic, and has the advantage of not making tion/ consists primarily of flat text files, htmldocs uses the clutter any worse. structured source code comments to generate docbook The other big editorial problem is keeping documenta- (and from that HTML or PDF output), and kconfig is tion up to date. When code is a living thing, its docu- in its own format which has dedicated viewer programs mentation must also be. The best documentation about (such as menuconfig). the innards of the 2.4 kernel is only passably accurate None of these is really an obvious choice for indexing about early 2.6, and in many ways the first 2.6 release the others.