X-Containers: Breaking Down Barriers to Improve Performance and Isolation of Cloud-Native Containers Zhiming Shen Zhen Sun Gur-Eyal Sela∗ Cornell University Cornell University University of California, Berkeley Eugene Bagdasaryan Christina Delimitrou Robbert Van Renesse Cornell University Cornell University Cornell University Hakim Weatherspoon Cornell University Abstract CCS Concepts • Security and privacy → Virtualiza- “Cloud-native” container platforms, such as Kubernetes, have tion and security; • Software and its engineering → become an integral part of production cloud environments. Operating systems. One of the principles in designing cloud-native applica- Keywords Containers; X-Containers; Cloud-Native; Li- tions is called Single Concern Principle, which suggests that brary OS; exokernel each container should handle a single responsibility well. In this paper, we propose X-Containers as a new security ACM Reference Format: paradigm for isolating single-concerned cloud-native con- Zhiming Shen, Zhen Sun, Gur-Eyal Sela, Eugene Bagdasaryan, tainers. Each container is run with a Library OS (LibOS) Christina Delimitrou, Robbert Van Renesse, and Hakim Weath- that supports multi-processing for concurrency and compat- erspoon. 2019. X-Containers: Breaking Down Barriers to Improve Performance and Isolation of Cloud-Native Containers. In 2019 Ar- ibility. A minimal exokernel ensures strong isolation with chitectural Support for Programming Languages and Operating Sys- small kernel attack surface. We show an implementation tems (ASPLOS ’19), April 13–17, 2019, Providence, RI, USA. ACM, New of the X-Containers architecture that leverages Xen para- York, NY, USA, 15 pages. https://doi.org/10.1145/3297858.3304016 virtualization (PV) to turn Linux kernel into a LibOS. Do- ing so results in a highly efficient LibOS platform that does 1 Introduction not require hardware-assisted virtualization, improves inter- container isolation, and supports binary compatibility and An important recent trend in cloud computing is the rise of multi-processing. By eliminating some security barriers such “cloud-native” container platforms, such as Kubernetes [38], as seccomp and Meltdown patch, X-Containers have up to which have become an integral part of production envi- 27× higher raw system call throughput compared to Docker ronments. Such platforms support applications designed containers, while also achieving competitive or superior per- specifically for cloud infrastructures that consist of loosely- formance on various benchmarks compared to recent con- coupled microservices [62] running in containers, enabling tainer platforms such as Google’s gVisor and Intel’s Clear automatic orchestration and agile DevOps practices [33]. In Containers. cloud-native platforms, container design is similar to object design in object-oriented (OO) software systems: each con- tainer should have a single responsibility and handle that ∗Work conducted at Cornell University. responsibility well [39]. By focusing on a single concern, cloud-native containers are easier to scale horizontally, and Permission to make digital or hard copies of all or part of this work for replace, reuse, and upgrade transparently. Similar to the Sin- personal or classroom use is granted without fee provided that copies gle Responsibility Principle in OO-languages, this has been are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights termed the “Single Concern Principle” [51], and is recom- for components of this work owned by others than the author(s) must mended by Docker [8]. be honored. Abstracting with credit is permitted. To copy otherwise, or Running multiple containers on the same host does not republish, to post on servers or to redistribute to lists, requires prior specific come without problems. From a security perspective, if one permission and/or a fee. Request permissions from [email protected]. container is compromised, all containers on the same Op- ASPLOS ’19, April 13–17, 2019, Providence, RI, USA erating System (OS) kernel are put under risk. Due to the © 2019 Copyright held by the owner/author(s). Publication rights licensed to ACM. concern of application isolation, containers are generally ACM ISBN 978-1-4503-6240-5/19/04...$15.00 not allowed to install their own kernel modules, a limitation https://doi.org/10.1145/3297858.3304016 for applications that require kernel customization. Nor can the OS kernel be easily tuned and optimized for a particular seccomp filters and Meltdown [57] patch, X-Containers have container since it is shared by other containers. up to 27× higher raw system call throughput compared to There have been several proposals to address the is- native Docker containers running in the cloud. X-Containers sue of container isolation. Hypervisor-based container run- also achieve competitive or superior performance compared times [17], such as Clear Containers [15], Kata Contain- to recent container platforms such as gVisor and Clear Con- ers [16], and Hyper Containers [13], wrap containers with tainers, as well as other LibOSes like Unikernel and Graphene a dedicated OS kernel running in a virtual machine (VM). on various benchmarks. These platforms require hardware-assisted virtualization sup- The X-Container architecture, however, also imposes sev- port to reduce the overhead of adding another layer of indi- eral limitations. For example, the change in the threat model rection. However, many public and private clouds, including makes it unsuitable for running some containers that still Amazon EC2, do not support nested hardware virtualization. require process and kernel isolation. Due to the require- Even in clouds like Google Compute Engine where nested ment of running a LibOS with each container, X-Containers hardware virtualization is enabled, its performance overhead take longer time to boot and have bigger memory footprint. is high (Section 5 and [5]). LightVM [60] wraps a container X-Containers also face challenges of page table operation in a paravirtualized Xen instance without hardware virtu- efficiency and dynamic memory management. We discuss alization support. Unfortunately, it introduces a significant these limitations in the paper. performance penalty in x86-64 platforms (Section 4.1 and This paper makes the following contributions: 5). Finally, Google gVisor [12] is a user-space kernel written • We present X-Containers, a new exokernel-inspired in Go that supports container runtime sandboxing, but it container architecture that is designed specifically for only offers limited system call compatibility [55] and incurs single-concerned cloud-native applications. We dis- significant performance overheads (Section 5). cuss the new threat model and the trade-offs it intro- The trend of running a single application in its own VM duces, the advantages and limitations of the proposed for enhanced security has led to a renewed interest in Lib- design, including those related to running unmodified OSes, as suggested by the Unikernel [58] model. LibOSes applications in X-Containers. avoid the overhead of security isolation between the appli- • We demonstrate how the Xen paravirtualization ar- cation and the OS, and allow each LibOS to be carefully chitecture and the Linux kernel can be turned into a optimized for the application at hand. Designing a container secure and efficient LibOS platform that supports both architecture inspired by the exokernel+LibOS [43] model can binary compatibility and multi-processing. improve both container isolation and performance. However, • We present a technology for automatically changing existing LibOSes, such as MirageOS [58], Graphene [69], system calls into function calls to optimize applications and OSv [53], lack features like full binary compatibility or running on a LibOS. multi-processing support. This makes porting containerized • We evaluate the efficacy of X-Containers against applications very challenging. Docker, gVisor, Clear Container, and other LibOSes In this paper, we propose a new LibOS platform called (Unikernel and Graphene), and demonstrate competi- X-Containers that improves container isolation without re- tive or superior performance. quiring hardware virtualization support. An X-Container can support one or more user processes that all run at the same privilege level as the LibOS. Different processes inside an 2 X-Containers as a New Security X-Container still have their own address spaces for resource Paradigm management and compatibility, but they no longer provide secure isolation from one another; in this new security para- 2.1 Single-Concerned Containers digm processes are used for concurrency, while X-Containers Cloud-native applications are designed to fully exploit the provide isolation between containers. We show an implemen- potential of cloud infrastructures. Although legacy applica- tation of the X-Containers architecture that leverages Xen’s tions can be packaged in containers and run in a cloud, these paravirtualization (PV) architecture [32] and turns the Linux applications cannot take full advantage of the automated kernel into a LibOS that supports both binary compatibility deployment, scaling, and orchestration offered by systems and multi-processing. like Kubernetes, which are designed for single-concerned Without hardware virtualization support, executing sys- containers [30, 31]. The shift to single-concerned containers tem call instructions is expensive, as they are