Proceedings of the Linux Symposium Volume Two July 19th–22nd, 2006 Ottawa, Ontario Canada Contents Evolution in Kernel Debugging using Hardware Virtualization With Xen 1 Nitin A. Kamble Improving Linux Startup Time Using Software Resume (and other techniques) 17 Hiroki Kaminaga Automated Regression Hunting 27 A. Bowen, P. Fox, J. Kenefick, A. Romney, J. Ruesch, J. Wilde, & J. Wilson Hacking the Linux Automounter—Current Limitations and Future Directions 37 Ian Maxwell Kent & Jeff Moyer Why NFS Sucks 51 Olaf Kirch Efficient Use of the Page Cache with 64 KB Pages 65 Dave Kleikamp and Badari Pulavarty Startup Time in the 21st Century: Filesystem Hacks and Assorted Tweaks 71 Benjamin C.R. LaHaise Using Hugetlbfs for Mapping Application Text Regions 75 H.J. Lu, K. Doshi, R. Seth, & J. Tran Towards a Better SCM: Revlog and Mercurial 83 Matt Mackall Roadmap to a GL-based composited desktop for Linux 91 K.E. Martin and K. Packard Probing the Guts of Kprobes 101 A. Mavinakayanahalli, P. Panchamukhi, J. Keniston, A. Keshavamurthy, & M. Hiramatsu Shared Page Tables Redux 117 Dave McCracken Extending RCU for Realtime and Embedded Workloads 123 Paul E. McKenney OSTRA: Experiments With on-the-fly Source Patching 139 Arnaldo Carvalho de Melo Design and Implementation to Support Multiple Key Exchange Protocols for IPsec 143 K. Miyazawa, S. Sakane, K. Kamada, M. Kanda, & A. Fukumoto The State of Linux Power Management 2006 151 Patrick Mochel I/O Workload Fingerprinting in the Genetic-Library 165 Jake Moilanen X86-64 XenLinux: Architecture, Implementation, and Optimizations 173 Jun Nakajima, Asit Mallick GCC—An Architectural Overview, Current Status, and Future Directions 185 Diego Novillo Shared-Subtree Concept, Implementation, and Applications in Linux 201 Al Viro & Ram Pai The Ondemand Governor 215 Venkatesh Pallipadi & Alexey Starikovskiy Linux Bootup Time Reduction for Digital Still Camera 231 Chan-Ju Park A Lockless Pagecache in Linux—Introduction, Progress, Performance 241 Nick Piggin The Ongoing Evolution of Xen 255 I. Pratt, D. Magenheimer, H. Blanchard, J. Xenidis, J. Nakajima, & A. Liguori NFSv4 Test Project 267 Tony Reix Measuring Resource Demand on Linux 287 Rik van Riel Improving the Approach to Linux Performance Analysis 295 Jose Santos & Guanglei Li Resizing Memory With Balloons and Hotplug 305 J.H. Schopp, K. Fraser, & M.J. Silbermann Collaborative Memory Management in Hosted Linux Environments 313 M. Schwidefsky, H. Franke, R. Mansell, H. Raj, D. Osisek, & J.H. Choi Chip Multi Processing aware Linux Kernel Scheduler 329 Siddha, Pallipadi and Mallick Dynamic Device Handling on the Modern Desktop 341 K. Sievers & D. Zeuthen Unionfs: User- and Community-Oriented Development of a Unification File System 349 D. Quigley, J. Sipek, C.P. Wright, & E. Zadok VMI: An Interface for Paravirtualization 363 Z. Amsden, D. Arai, D. Hecht, A. Holler, & P. Subrahmanyam HTTP-FUSE Xenoppix 379 K. Suzaki, T. Yagi, K. Iijima, K. Kitagawa, S. Tashiro Virtual Scalability: Charting the Performance of Linux in a Virtual World 393 A. Theurer, K. Rister, O. Krieger, R. Harper, & S. Dobbelstein Automatic System for Linux Kernel Performance Testing 403 A. Ufimtsev & L. Murphy MD RAID Acceleration 409 Dan J. Williams Catalyzing Hardware Driver Development 415 D.J. Wong, A. Bruemmer, D. Fry, & M. Salyzyn Conference Organizers Andrew J. Hutton, Steamballoon, Inc. C. Craig Ross, Linux Symposium Review Committee Jeff Garzik, Red Hat Software Gerrit Huizenga, IBM Dave Jones, Red Hat Software Ben LaHaise, Intel Corporation Matt Mackall, Selenic Consulting Patrick Mochel, Intel Corporation C. Craig Ross, Linux Symposium Andrew Hutton, Steamballoon, Inc. Proceedings Formatting Team John W. Lockhart, Red Hat, Inc. David M. Fellows, Fellows and Carr, Inc. Kyle McMartin Authors retain copyright to all submitted papers, but have granted unlimited redistribution rights to all as a condition of submission. Evolution in Kernel Debugging using Hardware Virtualization With Xen Nitin A. Kamble [email protected] Jun Nakajima [email protected] Asit K. Mallick [email protected] Open Source Technology Center, Intel Corporation Abstract guests. This Xen capability to run unmodified Linux OS or any other unmodified OS also pro- vides a new opportunity to debug an unmodi- Xen’s ability to run unmodified guests with the fied OS using the Xen VMM. virtualization available in hardware opens new doors of possibilities in the kernel debugging. Now it’s possible to debug the Linux kernel or any other PC operating system similar to de- bugging a user process in Linux. Since hard- ware virtualization in-processor enables Xen to implement full virtualization of a guest OS, With this guest debug capability, it is possible there is no need to change the kernel in any way to trap into an unmodified guest such as any to debug it. Linux, Windows, DOS, or any other PC OS; and check the register state, modify registers, This paper demonstrates the new evolutionary set debug breakpoints anywhere including in debug techniques using examples. It also ex- the kernel, read and write memory, or inspect or plains how the new technique actually works. modify the code currently being executed. This new method uses gdb[3] as the front end for debugging. With gdb also comes the source- 1 Introduction level debugging of an unmodified Linux kernel. There are some advantages of using this debug approach compared to other kernel debug op- The Xen[1] open source virtual machine moni- tions, such as the Linux kernel stays unmodi- tor initially started with software virtualization fied, and ability of setting of breakpoints any- by modifying the guest OS kernel. Since Xen where in the code. In fact it is also possible to 3.0, it also supports the Intel R Virtualization set breakpoints in the boot loader such as grub Technology R [2] to create and run unmodified [4] or inside the guest BIOS code. 2 • Evolution in Kernel Debugging using Hardware Virtualization With Xen 2 The Architecture and Design of Guest 0 Guest 1 debugging of an unmodified guest VM Exit VM Entry VM Entry VM Exit The virtualization technology in the processor, and Xen’s ability to take advantage of it, let an VMXON VM Monitor VMXOFF unmodified OS run inside a virtual machine. The following sections first briefly describe the Figure 1: Interaction of Virtual-Machine Mon- virtualization technology in the Intel IA32 pro- itor and Guests cessors, and how Xen[5] hypervisor utilizes this hardware virtualization to create virtual machines (domain) for unmodified guests. its. Figure 1 depicts the interactions between the VMX root and VMX nonroot operations. 2.1 Intel Virtualization Technology for Processor behavior in VMX root operation is IA32 Architecture very much as it is outside VMX operation or without the VT feature in the processor. The Virtualiztion Techinology in the Intel proces- principal differences are that a set of new in- sors augment the IA32 architecture by pro- structions (the VMX instructions) is available viding new processor operation modes called and that the values that can be loaded into cer- VMX operations. And the Virtual-Machine tain control registers are limited. Processor be- Control Structure controls the operation of the havior in VMX non-root operation is restricted virtual machine running in the VMX operation. and modified to facilitate virtualization. Instead of their ordinary operation, certain instructions The following subsections introduce the VMX (including the new VMCALL instruction) and Operation and the Virtual-Machine Control events cause VM exits to the VMM. Because Structure briefly. these VM exits replace ordinary behavior, the functionality of software in VMX non-root op- eration is limited. It is this limitation that al- 2.1.1 Introduction to VMX operation lows the VMM to retain control of processor resources. VT processor support for virtualization is pro- Because VMX operation places these restric- vided by a new form of processor operation tions even on software running with current called VMX operation. There are two kinds privilege level (CPL) 0, guest software can run of VMX operations: VMX root operation and at the privilege level for which it was originally VMX nonroot operation. The Xen VMM runs designed. in VMX root operation and guest software runs in VMX non-root operation. Transitions be- tween VMX root operation and VMX non-root operation are called VMX transitions. There 2.1.2 Virtual-Machine Control Structure are two kinds of VMX transitions. Transitions into VMX non-root operation are called VM en- VMX non-root operation and VMX transitions tries. Transitions from VMX non-root opera- are controlled by a data structure called a vir- tion to VMX root operation are called VM ex- tual machine control structure (VMCS). Ac- 2006 Linux Symposium, Volume Two • 3 cess to the VMCS is managed through a com- 2.3 Internals of debugging an unmodified ponent of processor state called the VMCS guest on Xen pointer (one per logical processor). The value of the VMCS pointer is the 64-bit ad- dress of the VMCS. The VMCS pointer can Figure 3 shows the interactions happening in be read and written using the instructions various Xen components when an unmodi- gdb VMPTRST and VMPTRLD. The VMM con- fied guest is being debugged. Both gdbserver-xen figures a VMCS using other instructions: VM- and are processes running READ, VMWRITE, and VMCLEAR. in the Xen-paravirtualized service OS, also known as domain-0. gdb is totally unmodi- fied. gdbserver is a gdb tool used for re- Please refer to the latest IA-32 SDM[6] for mote debug. gdbserver-xen is a modified more details on the Virtual Machine Extensions gdbserver for utilizing Xen hyper call based in- (VMX) in the Intel Processor. terfaces available in domain-0. The following sections describe interactions 2.2 Xen support for unmodified Guest us- and implementation details for the the Xen ing the Hardware Virtualization components exercised while debugging a un- modified guest OS.