X-Containers: Breaking Down Barriers to Improve Performance and Isolation of Cloud-Native Containers Zhiming Shen, Zhen Sun, Gur-Eyal Sela, Eugene Bagdasaryan Christina Delimitrou, Robbert Van Renesse, Hakim Weatherspoon Cornell University Abstract operating systems, becomes redundant in most single- “Cloud-native” container platforms, such as Kubernetes, concerned containers. On the other hand, inter-container have become an integral part of production cloud isolation becomes increasingly important, and raises many environments. One of the principles in designing cloud- concerns because containers share a monolithic OS kernel native applications is called “Single Concern Principle”, with a large Trusted Computing Base (TCB), as well as a large which suggests that each container should handle a single attack surface in terms of the number of kernel interfaces. responsibility well. Due to the resulting change in the There have been several proposals to address the issue threat model, process isolation within the container becomes of container isolation. Virtualization-based solutions, such redundant in most single-concerned containers, and inter- as Clear Containers [8], Kata Containers [9], and Hyper container isolation becomes increasingly important. In this Containers [6], wrap containers with a dedicated OS kernel paper, we propose a new exokernel-inspired architecture running in a virtual machine (VM). These platforms require called X-Containers that improves both the security and native hardware virtualization support to reduce the overhead the performance of cloud-native containers. We show that, of adding another layer of indirection. However, most through relatively minor modifications, the Xen hypervisor public and private clouds, including Amazon EC2, do not can serve as an exokernel, and Linux can be turned into support nested hardware virtualization. Even in clouds a LibOS. Doing so results in a highly secure and efficient such as Google Compute Engine where nested hardware LibOS platform that, unlike other available LibOSes, supports virtualization is enabled, its performance overhead is high binary compatibility and multicore processing. X-Containers (see Section5 and [ 15]). LightVM [49] wraps a container in a have up to 27× higher raw system call throughput compared paravirtualized Xen instance without hardware virtualization to Docker containers, while also significantly outperforming support. Unfortunately, it introduces a significant performance recent container platforms such as Google’s gVisor, Intel’s penalty in x86-64 platforms (see Sections 4.1 and5). Finally, Clear Containers, as well as Library OSes like Unikernel and Google gVisor [5] is a user-space kernel written in Go that Graphene on web benchmarks. supports container runtime sandboxing, but it only offers 1. Introduction limited system call compatibility [13] and incurs significant performance overheads (see Section5). An important recent trend in cloud computing is the rise of “cloud-native” container platforms, such as Kubernetes [30], The trend of running a single application in its own which have become an integral part of production VM for enhanced security has led to a renewed interest environments. Such platforms support applications designed in Library Operating Systems (LibOSes), as suggested by specifically for cloud infrastructures that consist of loosely- the Unikernel [46] model. LibOSes avoid the overhead of coupled microservices [51] running in containers, with security isolation between the application and the OS, and distributed management and orchestration. Cloud-native allow each LibOS to be carefully optimized for the application platforms offer support for automated elastic scaling and agile at hand. Designing a container architecture inspired by the DevOps practices [26], which allow companies to bring new exokernel+LibOS [34] model can improve both container ideas to market faster, respond quickly to customer demands, isolation and performance. However, existing LibOSes, such v and handle failures more transparently. as MirageOS [46], Graphene [58], and OS [41], lack features, In cloud-native platforms, container design is similar to such as full binary compatibility or multicore support. This object design in object-oriented (OO) software systems: each makes porting containerized applications very challenging. container should have a single responsibility and handle that In this paper, we propose a new LibOS platform called X- responsibility well [31]. By focusing on a single concern, Containers that improves both the security and performance of cloud-native containers are easier to scale horizontally, and containers without requiring hardware virtualization support. replace, reuse, and upgrade transparently. Similar to the We demonstrate that Xen’s paravirtualization architecture [25] Single Responsibility Principle in OO-languages, this has can be modified to serve as a highly secure and efficient LibOS been termed the “Single Concern Principle” [39], and is platform that supports both binary compatibility and multicore recommended by Docker [3]. processing. An X-Container can support one or more user From a security perspective, single-concerned containers processes that all run at the same privilege level as the LibOS. change the threat model of the container architecture, Different processes inside an X-Container still have their own presenting both a challenge and an opportunity. Process address spaces for resource management and compatibility, isolation, which separates different applications in traditional but they no longer provide secure isolation from one another; in this new model processes are used for concurrency, while mechanisms for grouping containers that need to be tightly X-Containers provide isolation between containers. coupled, e.g., using a “pod” in Google Kubernetes [16], and Without hardware virtualization support, system calls are a “task” in Amazon ECS [2]. It is important to note that expensive, as they are first handled by the exokernel and single-concerned containers are not necessarily single-process. then redirected to the LibOS. The X-Container platform Some applications might spawn multiple worker processes for automatically optimizes the binary of an application during concurrency [3], such as NGINX, or Apache webserver. The runtime to improve performance by rewriting costly system key is that all processes within a single-concerned container calls into much cheaper function calls in the LibOS. As a belong to the same service, thus they are tightly coupled and result, X-Containers have up to 27× higher raw system call mutually trusting. throughput compared to native Docker containers running in 2.2. Application Isolation the cloud, and are competitive to or even outperform native containers for other benchmarks. In traditional operating systems, processes are used for both The X-Container platform also outperforms other LibOS multiprocessing and security isolation between applications. architectures for specialized services, such as serverless Due to the coarse granularity of the process model, compute, which are gaining traction for short-running, user- applications often implement their own security properties driven online services with intermittent behavior. We compare in the application logic, isolating different users within the X-Containers, Unikernel, and Graphene, using NGINX [12], same application. In single-concerned cloud-native containers, a stateless front-end webserver driven by the wrk workload process isolation within the container is unnecessary. generator. We show that X-Containers have comparable Indeed, an informal survey of popular containers shows performance to Unikernel, and twice the throughput compared that multi-client applications rarely rely on processes for to Graphene. Moreover, when running PHP and MySQL, isolating mutually-untrusting clients by dedicating a process X-Container achieves approximately 3× the performance of to each client—many do not even use multiple processes at Unikernel. all. All the top 10 most popular containerized applications [1] This paper includes the following contributions: (NGINX, Redis, ElasticSearch, Registry, Postgres, MySQL, • We demonstrate how the Xen paravirtualization etcd, Fluentd, MongoDB, and RabbitMQ) use either a single- architecture and the Linux kernel can be turned into a threaded event-driven model or multi-threading instead of secure and efficient LibOS platform that supports both multiple processes. NGINX and Fluentd can be configured binary compatibility and multicore processing. to use a process pool to improving concurrency, similar to • We present X-Containers, a new exokernel-based Apache webserver—each process has multiple threads that container architecture that is designed specifically for can be used to serve different clients. These applications cloud-native applications. X-Containers are compatible implement client isolation inside the application logic through with Linux containers, and to the best of our knowledge, mechanisms such as a strongly-typed language runtime, role- they are the first architecture to support secure isolation of based access control, authentication, and encryption. containers in the cloud, without sacrificing compatibility On the other hand, inter-container isolation remains or performance. important, and depends on enforcement by the OS kernel. • We present a technology for automatically changing Unfortunately, modern monolithic OS kernels, such as Linux, system calls into function