Instant OS Updates via Userspace Checkpoint-and-Restart Sanidhya Kashyap, Changwoo Min, Byoungyoung Lee, and Taesoo Kim, Georgia Institute of Technology; Pavel Emelyanov, CRIU and Odin, Inc. https://www.usenix.org/conference/atc16/technical-sessions/presentation/kashyap This paper is included in the Proceedings of the 2016 USENIX Annual Technical Conference (USENIX ATC ’16). June 22–24, 2016 • Denver, CO, USA 978-1-931971-30-0 Open access to the Proceedings of the 2016 USENIX Annual Technical Conference (USENIX ATC ’16) is sponsored by USENIX. Instant OS Updates via Userspace Checkpoint-and-Restart Sanidhya Kashyap Changwoo Min Byoungyoung Lee Taesoo Kim Pavel Emelyanov† Georgia Institute of Technology †CRIU & Odin, Inc. # errors # lines Abstract 50 1000K 40 100K In recent years, operating systems have become increas- 10K 30 1K 20 ingly complex and thus more prone to security and per- 100 formance issues. Accordingly, system updates to address 10 10 these issues have become more frequently available and 0 1 increasingly important. To complete such updates, users 3.13.0-x 3.16.0-x 3.19.0-x May 2014 must reboot their systems, resulting in unavoidable down- build/diff errors #layout errors Jun 2015 time and further loss of the states of running applications. #static local errors #num lines++ We present KUP, a practical OS update mechanism that Figure 1: Limitation of dynamic kernel hot-patching using employs a userspace checkpoint-and-restart mechanism, kpatch. Only two successful updates (3.13.0.32 34 and → which uses an optimized data structure for checkpoint- 3.19.0.20 21) out of 23 Ubuntu kernel package releases. X- → ing on disk as well as a memory persistence mechanism axis represents before-version and after-version, and dotted bars across the update, coupled with a fast in-place kernel represent failures in executing kpatch. Each failure case rep- switch. This allows for instant kernel updates spanning resents the following legends: build/diff errors—kpatch failed across major kernel versions without any kernel modifica- during a build and diff processes; # static local—a patch mod- tions. ifies a value of static local variables; # layout error—a patch Our evaluation shows that KUP can support any type changes a layout of a data structure. of real kernel patches (e.g., security, minor or even ma- jor releases) with large-scale applications that include 73] directly applies patches to the running kernel. As a memcached, mysql, or in the middle of the Linux kernel result, system updates can be performed without incurring compilation, unlike well-known dynamic hot-patching application downtime. However, dynamic hot-patching techniques (e.g., ksplice). Not only that, KUP can update is inherently limited to patches that do not semantically a running Linux kernel in 3 seconds (overall downtime) modify data structures. Therefore, it is commonly used without losing 32 GB of memcached data from kernel ver- to apply simple security patches that contain minor code sion v3.17-rc7 to v4.1. changes. For instance, kpatch [67], a popular dynamic update tool developed by Red Hat, was able to support 2 1 Introduction out of 23 minor updates over a year of Ubuntu’s kernel Today, computer users routinely update their operating releases (see Figure 1). systems either to patch security vulnerabilities, fix bugs, Rolling updates [20, 60] are another viable technique or add a new set of features to the existing system. Unfor- for large-scale systems. In rolling updates, system admin- tunately, to fully incorporate a new update, users have no istrators apply an update to a small group of machines; choice but to restart their systems, which results in the loss if there is no failure, they apply the same update to the of the running states of applications. This unavoidable rest of the machines in the data center. This helps admin- disruption not only brings inconvenience to end users, but istrators minimize the risk of update failures and system also causes an adverse financial impact on business.For downtime. However, this process requires careful plan- example, updating a memcached [57] server at Facebook ning to minimize the disruption of running services due that caches around 120 GB data in 144 GB RAM [24] re- to the inevitable downtime of applications. quires a prolonged warm-up phase of 90-100 minutes [30]. There are several research projects that attempt to solve The more critical problem is when a system update fails. the OS update problem either by proposing new OS con- Such failure often leads to additional downtime or main- struction [5, 7, 23, 65, 80], or by providing a transfer tenance costs thereafter, resulting in an immediate loss of function for system updates [28]. In order to keep their in- active customers [20, 21]. Still, even with the unavoidable ternal states switchable, such OSes require radical design disruption and risks involved in update failures, system changes, thereby making these modifications impractical updates are necessary to promptly mitigate the known for the commodity OSes [19, 69]. security issues and resolve critical bugs that might hurt To address these issues, we present KUP—a new up- the correctness of operations [8, 25, 44, 68, 71]. date mechanism that allows for prompt kernel updates To solve these problems, two large sets of techniques without modifying a commodity OS. KUP incorporates are used in practice: dynamic hot-patching and rolling application checkpoint-and-restart (C/R) for saving and updates. Dynamic hot-patching (or live update) [4, 67, restoring the application states before and after the update, 1 USENIX Association 2016 USENIX Annual Technical Conference 605 and an in-place kernel switch to quickly boot into a new 2 Related work kernel. This allows KUP to minimize the downtime of In this section, we compare KUP’s approach to previous running applications during the system update. Unlike studies in four areas: dynamic hot-patching, new OS dynamic hot-patching mechanisms, KUP can perform a designs, live migration, and application C/R for updates. full system update regardless of the complexity of up- dates, and thus can support a broader range of system Dynamic hot-patching. Industries rely on solutions updates, including security and bug fixes, the addition of such as Ksplice [4], kpatch [67], and KGraft [73] to new features, and even major kernel upgrades. To address promptly mitigate the security vulnerabilities without update failures, KUP provides a safe fallback mechanism any significant outage of their production systems. How- that provides a mechanism to restore the original system, ever, as shown in Figure 1, the hot-patching techniques checkpointed right before the update. have critical limitations, especially in handling data lay- out changes, thus making it impossible to guarantee the KUP works by leveraging the OS provided infrastruc- safety of live updates. Previous studies have proposed ture such as procfs, ptrace system call [37] for obtain- dynamic software update schemes on event-driven sys- ing information during the checkpoint of an application, tems [1, 27, 28, 36], object oriented languages [39, 63] and process forking (clone, fork) mechanism along with and even C language [4, 35, 53, 54, 56] including formal other system calls [15, 37] and netlink sockets to re- proofs [34, 55, 72]. store the application after update. To enable an effi- However, unlike previous update techniques, KUP re- cient, userspace-based C/R of an application during an lies on a whole kernel switch to apply any kind of patch, update, we design an efficient checkpointing storage for- regardless of its complexity. KUP allows for system-wide mat that improves both the checkpoint size and KUP’s live updates without modifying programs or tracking their performance. However, the aforementioned solution of state changes, unlike previous works [19, 35]. KUP still ends up storing RAM data on the disk, which is not suitable for disk-less systems [30, 57].1 We further ex- New operating system designs for live update. An- tend KUP and make it generic enough to support disk-less other approach is to design a new OS with well-defined systems by proposing a memory persistence mechanism, abstractions between interfaces and implementations so which we implement via binary patching and dynamic that each component can be replaced online without dis- OS instrumentation. This approach is complementary to ruption. Representative examples include Microkernel our userspace solution. (e.g., exokernel [23], K42 [5, 7, 70], Barrelfish [6, 80]) and LibOS (e.g., Drawbridge [65]). In particular, Pro- This paper makes the following three contributions: teos [28] based on MINIX 3, and a Linux variant [69], are designed to update their components online and also • We design a simple, yet robust update mechanism transform internal data structures to adopt the updated ker- by using application C/R, and implement an open nel. Unfortunately, with a large amount of code changes source prototype of KUP. (e.g., new features), constructing the state transfer func- • We show KUP’s effectiveness by providing an in- tions either requires manual effort or is infeasible in many depth analysis of the proposed techniques with real cases [5, 7, 11, 28, 69, 70]. applications, micro-benchmarks and full software Autopod [66] and the work by Siniavine et al. [69] are stack representing generic data center application. closest to KUP. Autopod uses a namespace mechanism • We devise, implement and evaluate a safe fall- for process decoupling and migrating applications across back mechanism for KUP that can be easily real- machines, whereas Siniavine et al. provides a kernel space ized through application C/R and an in-place kernel C/R mechanism coupled with an in-place kernel switch to switch. update the OS, and employs techniques similar to those of Otherworld [19].