Ksplice: An automatic system for rebootless kernel security updates Jeffrey Brian Arnold Massachusetts Institute of Technology [email protected] Abstract of many kernels), they create and distribute a patch, Ksplice∗ allows system administrators to apply security which consists of a set of changes to the source code. In patches to their operating system kernels without hav- the case of a kernel change, software vendors or system ing to reboot. Based on a source code patch and the administrators apply the patch to their copy of the source kernel source code to be patched, Ksplice applies the code, build a new kernel, and then distribute that new patch to the corresponding running kernel, without re- binary kernel to servers and end-user machines, which quiring work from a programmer. To be fully automatic, must be rebooted in order to run the new kernel. Ksplice’s design is limited to patches that do not intro- Safely performing a hot update, instead of a traditional duce semantic changes to data structures, but a study of update, on an unmodified C program currently requires all significant x86-32 Linux security patches from May significant programmer involvement. Existing hot up- 2005 to May 2008 finds that only nine patches of 65 date practices, described in Section 5, rely on a program- make semantic changes. An evaluation with Debian and mer to write source code files with certain properties kernel.org Linux kernels shows that Ksplice can auto- (e.g., [7, 20]) or require manual inspection of the running matically apply the remaining 56 patches, which means binary to achieve safety guarantees and resolve ambigu- that 86% of the Linux kernel vulnerabilities from this in- ous symbols (e.g., [1]). terval can be corrected by Ksplice without the need for Requiring significant programmer involvement in cre- rebooting. ating a hot update increases both the cost and the risk of the update, which discourages the adoption of hot up- dates. Ideally, we would like to be able to safely con- 1 Introduction struct hot updates, with no human involvement, from ex- isting information, such as the patch used to correct the Several contemporary operating systems release kernel security vulnerability. security patches many times per year. Some of these kernel patches repair vulnerabilities which would other- Although performing an arbitrary source code change wise potentially allow an attacker to gain full adminis- as a hot update is not feasible without a programmer trator privileges on the operating system. Applying these writing new code to perform the transition, some patches security patches typically requires rebooting the kernel, can safely be performed as a hot update based solely on which results in downtime and loss of state (e.g., all ac- the source code differences. Specifically, we have suf- tive network connections). Since rebooting can cause ficient information for a hot update when a patch does disruption, system administrators often delay perform- not make semantic changes to data structures—that is, ing security updates, despite the risk of compromises. changes that would require existing instances of kernel This paper describes and evaluates Ksplice, a system for data structures to be transformed (e.g., a patch that adds performing hot updates, which change a running kernel a field to a global data structure would require the ex- without rebooting it. isting data structures to change). Since kernel security When software developers correct a security problem patches tend to make as few software modifications as in the source code of a C program (which is the language possible, we can expect many of these patches to make no semantic changes to data structures. ∗ Ksplice is free software, and its documentation is free. This paper focuses on developing techniques for safely You can redistribute and/or modify this document under the terms of the GNU General Public License, version 2. See performing hot updates without programmer involve- <http://web.mit.edu/ksplice> for more information. ment, in the common case that the security patch does 1 not make semantic changes to data structures. This prob- approach for constructing hot updates and an evalua- lem contains several challenges that have not been solved tion of this system against Linux security patches. This by previous hot update systems, and effectively solving Ksplice evaluation is, to our knowledge, the first eval- these problems helps many hot update systems (not just uation of any kernel hot update system against a com- systems that focus on patches without semantic changes prehensive list of the significant security vulnerabilities to data structures). The techniques that we describe are, within a commodity operating system over a period of compared to previous work, oriented much more towards time. We believe that this evaluation clearly demon- analysis at the object code level, which offers various ad- strates that hot updates currently have the potential to vantages, as described in Section 2 and Section 5. eliminate most kernel security reboots, while requiring Ksplice can, without restarting the kernel, apply any little additional work from any programmer. source code patch that only needs to modify the kernel The rest of this paper is organized as follows: The next text (including patches to kernel modules and assembly section presents Ksplice’s design for performing hot ker- files). Unlike other hot update systems, Ksplice takes as nel updates. Section 3 describes Linux-specific imple- input only a source code patch and the original kernel mentation considerations. Section 4 tests Ksplice against source code, and it updates the running kernel correctly, security patches from May 2005 to May 2008. Section 5 with no further human assistance required. and Section 6 relate Ksplice to previous work and ker- Ksplice does not require any preparation before the nel developer community discussions. Section 7 sum- system is originally booted. The running kernel does marizes our conclusions and directions for future work. not need to have been specially compiled, for example. Other hot update systems require a special kernel de- sign [3, 4], require a customized compiler [1, 21], or re- 2 Design quire a virtual machine monitor [7]. The performance impact of inserting a Ksplice update To apply a patch, Ksplice replaces all of the functions into a kernel is minimal. A small amount of additional that the patch changes. If any code within a function kernel memory, proportional to the size of the replace- is patched, then Ksplice will replace the entire function. ment functions, will be expended, and function calls to Ksplice replaces a function by linking the patched ver- the replaced functions will take a few cycles longer be- sion of the function into the kernel’s address space and cause of inserted jump instructions. On modern systems, by causing all callers of the original function to invoke this overhead should be negligible in most situations. the patched version. Ksplice replaces entire functions Although generating a Ksplice hot update from a com- since they tend to have a well-defined entry point, at the patible patch requires no human effort, Ksplice requires beginning of the function, which is convenient for redi- a person to perform a single check before invoking recting execution flow away from an obsolete function to Ksplice: a person is expected to confirm that the target a replacement function. security patch does not make any semantic changes to Although replacing entire functions is relatively con- data structures. Performing this check requires only sec- venient, this replacement must be done with care. It onds or a few minutes for most security patches. This involves generating the object code for the patched check is much simpler than the programmer work re- function, resolving symbols in the object code of the quired by other hot update systems. patched function, stopping the kernel temporarily, rewrit- We implemented Ksplice for Linux, but the techniques ing the initial instructions in the obsolete function to that Ksplice employs apply to other operating systems. point callers at the patched function, and starting the ker- To evaluate Ksplice’s approach, we applied Ksplice to nel again. The rest of the section discusses how Ksplice 65 Linux patches for kernel security vulnerabilities from performs these operations. May 2005 to May 2008. The 65 include all documented x86-32 Linux kernel vulnerabilities from this time inter- 2.1 Overview val with greater consequences than denial of service. We applied the patches to five different running Debian ker- Ksplice accepts as input the original kernel source code nels and six different running kernel.org kernels. Of the and a source code patch. Through a multi-stage process, 65 patches, nine make semantic changes and cannot be Ksplice uses this input to create a kernel module that con- applied by Ksplice. The remaining 56 of the 65 patches tains the patched functions (see Figure 1). Most of the can be applied using Ksplice without the need for re- stages of this process make few assumptions about the booting, which is a significant advance over the current underlying operating system. The design assumes a rea- state in which system administrators always have to re- sonable binary format for object code and a basic facility boot their systems. for kernel modules. In order to make our examples spe- The contributions of this paper are a new binary-level cific, we assume in our discussion that the object code 2 function signature changes, changes to functions with static local variables, and changes to assembly files, fea- tures which are notably missing from other hot update work [1]. Third, Ksplice’s binary comparison techniques avoid a subtle problem with inline functions. This problem af- fects other hot update systems that, like Ksplice, are de- signed to update programs that were compiled without foresight of the update system.