Optimizing Crash Dump in Virtualized Environments Yijian Huang Haibo Chen Binyu Zang Parallel Processing Institute Parallel Processing Institute Parallel Processing Institute Fudan University Fudan University Fudan University [email protected] [email protected] [email protected] Abstract Crash dump, or core dump [7], is a typical way to save the Crash dump, or core dump is the typical way to save memory image crashed or hang context to persistent storage for future offline de- on system crash for future offline debugging and analysis. However, bugging and analysis [5, 6] before restarting the system. However, for typical server machines with likely abundant memory, the time for traditional server machines with likely abundant memory, it is of core dump can significantly increase the mean time to repair usually time-consuming to save the failure context (usually includ- (MTTR) by delaying the reboot-based recovery, while not dumping ing the whole memory), which delays the recovery of the system the failure context for analysis would risk recurring crashes on the and increases the mean time to repair (MTTR). For instance, core- same problems. dumping 32 GB memory into a commodity SCSI disk with 70 In this paper, we propose several optimization techniques for MB/sec write rate could take more than 400 seconds. Hence, some core dump in virtualized environments, in order to shorten the administrators may choose to directly restart the system without a MTTR of consolidated virtual machines during crashes. First, we crash dump, risking recurring crashes on the same problems. parallelize the process of crash dump and the process of rebooting In this paper, we analyze the characteristics of crash dump the crashed VM, by dynamically reclaiming and allocating memory and propose several techniques to optimize the process of crash between the crashed VM and the newly spawned VM. Second, we dump of virtual machines in consolidated virtualized environments. use the virtual machine management layer to introspect the critical Unlike that in a native environment, the crash or hang of a virtual data structures of the crashed VM to filter out the dump of unused machine will not cause an end of the system, as the hypervisor memory. Finally, we implement disk I/O rate control between core and VM management tools remain fully operational. This opens dump and the newly spawned VM according to user-tuned rate opportunities to optimize crash dump to minimize the downtime control policy to balance the time of crash dump and quality of caused by the system crash. services in the recovery VM. We propose to parallelize the core dump of the crashed virtual We have implemented a working prototype, Vicover, that opti- machine and the booting of another recovery virtual machine to mizes core dump on system crash of a virtual machine in Xen, to continue the same application services. In order to retain the per- minimize the MTTR of core dump and recovery as a whole. In our sistent states, the recovery VM uses the same file system as the experiment on a virtualized TPC-W server, Vicover shortens the crashed VM. As a result, persistent states updated by the crashed downtime caused by crash dump by around 5X. VM remain accessible by the recovery VM after the recovery VM is started. The sharing is safe, because the two VMs do not access Categories and Subject Descriptors K.6.3 [Management of Com- the file system simultaneously: the recovery VM accesses the file puting and Information Systems]: Software Management—Software system only after the other VM crashes and stops the access to the maintenance; D.4.5 [Operating Systems]: Reliability file system. Here, we assume that the crash of a virtual machine does not cause irreversible system damages and can be repaired by General Terms Management, Performance, Reliability file system consistency check (e.g., fsck) when booting the recov- ery VM. Keywords Core Dump, Parallel Core Dump, Virtual Machines When a VM crashes, its CPU and I/O resources are immediately released. However, due to the long-term memory reservation by 1. Introduction the core dump tool, using the memory owned by the crashed VM for recovery is not allowed until the core dump ends. Hence, we Reliability has been one of the major concerns of modern com- propose to concurrently reclaim core-dumped memory and allocate puter systems, which may be frequently disturbed by software and it to the recovery VM. To this end, the pseudo-physical memory of hardware errors, unsafe software/hardware interactions, as well as a VM is divided into several chunks of the same size. If the VM operation errors. While restarting the frozen or crashed system is crashes, its pseudo-physical memory is core-dumped to disk chunk the usual way to recover from failures, it is demanding to diagnose by chunk, instead of all at once. For each chunk, once it is core- the root cause of the failure for future software and hardware fixes. dumped, it is immediately reallocated to another VM which boots the system from the shared storage to recover application services. As core dump continues, the hypervisor eventually releases all memory owned by the crashed VM, and the recovery VM gradually Permission to make digital or hard copies of all or part of this work for personal or reaches its presumed memory capacity. The recovery VM not only classroom use is granted without fee provided that copies are not made or distributed gains memory from the crashed VM as early as possible, but also for profit or commercial advantage and that copies bear this notice and the full citation utilizes the CPU resource which core dump, as an I/O-intensive on the first page. To copy otherwise, to republish, to post on servers or to redistribute process, does not fully utilize. to lists, requires prior specific permission and/or a fee. To minimize the time of core dump, we also implement a se- VEE’10, March 17–19, 2010, Pittsburgh, Pennsylvania, USA. Copyright c 2010 ACM 978-1-60558-910-7/10/03. $10.00 lective dump strategy that only dumps memory that is in use by the crashed VM, instead of dumping the whole memory of a vir- Core dump tools are available for commodity OSes as well as tual machine. To identify which pages are in use, we introspect the for virtualized environments. Microsoft Windows can be set up to crashed VM, extract key data structures of the crashed guest OS save a 64 KB minidump of basic states, a kernel dump of memory and recognize free memory pages not used on system crash, so that in the Windows kernel, or a full dump of the whole RAM on system only pages used by the guest OS or applications of the VM at the crash. A debugger tool like WinDbg can be used to analyze the moment of crash are core-dumped. The trustworthiness depends on failure data saved in the core dump image. that the introspected part of memory is not corrupted by system In Linux, Kdump [7] provides a reliable core dump mechanism. crash, which is a trade-off between the speed and correctness of Kdump [7] improves Kexec by fast and automatically booting into a core dump. new kernel on system crashes and executing core dump in the con- Both the core dump VM and the recovery VM consume the text of the new kernel. Kexec is a boot loader which fast boots into a I/O bandwidth. To balance the time of crash dump and quality of new kernel from the old one without clearing the memory of the old services in the recovery VM, we implement a disk I/O rate control kernel. It pre-loads a new kernel and associated data into a reserved mechanism to balance the I/O bandwidth between the two VMs. area of memory by the kexec load() system call. The reserved area The bandwidth allocation policy is tuned by user as a trade-off is configured not to be used by the old kernel. To quickly load the between the speed of reclaiming memory by core dump and the new kernel, Kexec boots into the new kernel without hardware re- disk I/O bandwidth for recovery. For a none I/O-intensive workload set, firmware operations or the bootloader stage. The new kernel is of booting the recovery VM, core dump should be given higher directly loaded from the reserved area so that the memory of the old priority of disk I/O. Otherwise, if both core dump and booting the kernel is not cleared. Kdump modifies Kexec to automatically boot recovery VM are I/O-intensive, the policy should be tuned by user into a new kernel on kernel panic. After the new kernel boots, the to minimize downtime. memory of the old one is accessible through special UNIX device We have implemented a working prototype, called Vicover, files in the context of the new kernel and can be core-dumped into based on Xen and Linux. Vicover has a utility program running persistent storage. The output core dump image is analyzable by at the VM management layer. It automatically detects system crash debugging or analysis tools (e.g. gdb [4] and crash analysis utility of a VM, and initiates core dump of the VM. It also concurrently [6]). boots the recovery VM that shares the same file system with the In virtualized environments, core dump tools save the memory crashed VM. of a VM to persistent storages, whose reliability is guaranteed Experimental results of core dump and recovery from system by the isolation mechanism from the virtualization layer.