Kdump, A Kexec-based Kernel Crash Dumping Mechanism Vivek Goyal Eric W. Biederman Hariprasad Nellitheertha IBM Linux NetworkX IBM [email protected] [email protected] [email protected] Abstract important consideration for the success of a so- lution has been the reliability and ease of use. Kdump is a crash dumping solution that pro- Kdump is a kexec based kernel crash dump- vides a very reliable dump generation and cap- ing mechanism, which is being perceived as turing mechanism [01]. It is simple, easy to a reliable crash dumping solution for Linux R . configure and provides a great deal of flexibility This paper begins with brief description of what in terms of dump device selection, dump saving kexec is and what it can do in general case, and mechanism, and plugging-in filtering mecha- then details how kexec has been modified to nism. boot a new kernel even in a system crash event. The idea of kdump has been around for Kexec enables booting into a new kernel while quite some time now, and initial patches for preserving the memory contents in a crash sce- kdump implementation were posted to the nario, and kdump uses this feature to capture Linux kernel mailing list last year [03]. Since the kernel crash dump. Physical memory lay- then, kdump has undergone significant design out and processor state are encoded in ELF core changes to ensure improved reliability, en- format, and these headers are stored in a re- hanced ease of use and cleaner interfaces. This served section of memory. Upon a crash, new paper starts with an overview of the kdump de- kernel boots up from reserved memory and pro- sign and development history. Then the limi- vides a platform to retrieve stored ELF headers tations of existing designs are highlighted and and capture the crash dump. Also detailed are this paper goes on to detail the new design and ELF core header creation, dump capture mech- enhancements. anism, and how to configure and use the kdump feature. Section 2 provides background of kexec and kdump development. Details regarding how kexec has been enhanced to boot-into a new 1 Introduction kernel in panic events are covered in section 3. Section 4 details the new kdump design. De- tails about how to configure and use this mech- Various crash dumping solutions have been anism are captured in Section 5. Briefly dis- evolving over a period of time for Linux and cussed are advantages and limitations of this other UNIX R like operating systems. All so- approach in section 6. A concise description lutions have their pros and cons, but the most of current status of project and TODOs are in- • 169 • 170 • Kdump, A Kexec-based Kernel Crash Dumping Mechanism cluded in Section 7. 2.2 A Brief History of Kdump Develop- ment The core design principle behind this approach 2 Background is that dump is captured with the help of a cus- tom built kernel that runs with a small amount of memory. This custom built kernel is called capture kernel and is booted into upon a sys- This section provides an overview of the kexec tem crash event without clearing crashed ker- and original kdump design philosophy and im- nel’s memory. Here onwards, for discussion plementation approach. It also brings forward purposes, crashing kernel is referred to as first the design deficiencies of kdump approach so kernel and the kernel which captures the dump far, and highlights the requirements that justi- after a system crash is called capture kernel. fied kexec and kdump design enhancements. While capture kernel boots, first kernel’s mem- ory is not overwritten except for the small 2.1 Kexec amount of memory used by new kernel for its execution. Kdump used this feature of kexec and added hooks in kexec code to boot into a capture kernel in a panic event without stomp- Kexec is a kernel-to-kernel boot-loader [07], ing crashed kernel’s memory. which provides the functionality to boot into a new kernel, over a reboot, without going Capture kernel used the first 16 MB of memory through the BIOS. Essentially, kexec pre-loads for booting and this region of memory needed the new kernel and stores the kernel image in to be preserved before booting into capture ker- RAM. Memory required to store the new kernel nel. Kdump added the functionality to copy the image need not be contiguous and kexec keeps contents of the first 16 MB to a reserved mem- a track of pages where new kernel image has ory area called backup region. Memory for the been stored. When a reboot is initiated, kexec backup region was reserved during the first ker- copies the new kernel image to destination lo- nel’s boot time, and location and size of the cation from where the new kernel is supposed backup region was specified using kernel con- to run, and after executing some setup code, fig options. Kdump also copied over the CPU kexec transfers the control to the new kernel. register states to an area immediately after the backup region during a crash event [03]. Kexec functionality is constituted of mainly two components; kernel space [08] and user After the crash event, the system is unstable space [02]. Kernel space component imple- and usual device shutdown methods can not be ments a new system call kexec_load() relied upon, hence, devices are not shutdown which facilitates pre-loading of new kernel. after a crash. This essentially means that any User space component, here onwards called ongoing DMAs at the time of crash are not kexec tools, parses the new kernel image, pre- stopped. In the above approach, the capture pares the appropriate parameter segment, and kernel was booting from the same memory lo- setup code segment and passes the this data to cation as the first kernel (1 MB) and used first the running kernel through newly implemented 16 MB to boot, hence, it was prone to cor- system call for further processing. ruption due to any on-going DMA in that re- 2005 Linux Symposium • 171 gion. An idea was proposed and a prototype 1. In the design above, kexec pre-loads the patch was provided for booting the capture ker- capture kernel wherever it can manage to nel from a reserved region of memory instead grab a page frame. At the time of crash, of a default location. This reduced the chances the capture kernel image is copied to the of corruption of the capture kernel due to on- destination location and control is trans- going DMA [04] [05]. Kdump’s design was up- ferred to the new kernel. Given the fact dated to accommodate this change and now the that the capture kernel runs from a re- capture kernel booted from reserved location. served area of memory, it can be loaded This reserved region was still being determined there directly and extra copying of kernel by kernel config options [06]. can be avoided. In general terms, kexec can be enhanced to provide a fast reboot Despite the fact that the capture kernel was path to handle booting into a new kernel booting from a reserved region of memory, it in crash events also. needed first 640 KB of memory to boot for SMP configurations. This memory was re- 2. Capture kernel and the associated data is quired to retrieve configuration data like the pre-loaded and stored in the kernel mem- MP configuration table saved by BIOS while ory, but there is no way to detect any data booting the first kernel. It was also required to corruption due to faulty kernel program- place the trampoline code needed to kick-start ming. application processors in the system. Kdump reserved 640 KB of memory (backup region) 3. During the first kernel boot, kdump re- immediately after the reserved region, and pre- serves a chunk of memory for booting the served the first 640 KB of memory contents by capture kernel. The location of this region copying it to a backup region just before trans- is determined during kernel compilation ferring control to capture kernel. CPU register time with the help of config options. De- states were being stored immediately after the termining the location of reserved region backup region [06]. through config options is a little cumber- some. It brings in hard-coding in many After booting, capture kernel retrieved the places, at the same time it is static in na- saved register states and backup region con- ture and a user has to compile the kernel tents, and made available the old kernel’s dump again if he decides to change the location image through two kernel interfaces. The of reserved region. first one was through the /proc/vmcore in- terface, which exported the dump image in 4. Capture kernel has to boot into a lim- ELF core format, and other one being the ited amount of memory, and to achieve /dev/oldmem, which provided a linear raw this, the capture kernel is booted with view of memory. user defined memory map with the help of memmap=exactmap command line op- tions. User has to provide this user de- 2.3 Need for Design Enhancement fined memory map while pre-loading the capture kernel and need to be explicitly aware of memory region reserved for cap- Following are some of the key limitations of ture kernel. This process can be automated the above approach that triggered the design en- by kexec tools and these details can be hancement of kdump. made opaque to the user.