"(c) 2015 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other users, including reprinting/ republishing this material for advertising or promotional purposes, creating new collective works for resale or redistribution to servers or lists, or reuse of any copyrighted components of this work in other works." 2015 IEEE 7th International Conference on Cloud Computing Technology and Science IncludeOS: A minimal, resource efficient unikernel for cloud services Alfred Bratterud, Alf-Andre Walla, Harek˚ Haugerud, Paal E. Engelstad, Kyrre Begnum Dept. of Computer Science Oslo and Akershus University College of Applied Sciences Oslo, Norway [email protected] Abstract—The emergence of cloud computing as devices ranging from Gigabit network interfaces a ubiquitous platform for elastically scaling services to sound cards and last century printers. As a has generated need and opportunity for new types consequence, modern operating systems require a of operating systems. A service that needs to be both elastic and resource efficient needs A) highly lot of disk- and memory for features the service it specialized components, and B) to run with minimal runs might not need. They also produce a steady resource overhead. Classical general purpose operat- flow of CPU and I/O usage. For example, for each ing systems designed for extensive hardware support Linux- or Windows VM, the hypervisor has to are by design far from meeting these requirements. emulate the timer interrupt, causing the virtual In this paper we present IncludeOS, a single tasking library operating system for cloud services, machine to spend energy while doing nothing. written from scratch in C++. Key features include: This might seem negligible at first, but for elastic extremely small disk- and memory footprint, effi- cloud services at scale, requiring a large number cient asynchronous I/O, OS-library where only what of virtual machines, it amounts to a significant your service needs gets included, and only one device resource waste and a serious limitation to the driver by default (virtio). As a test case a bootable disk image consisting capacity of hypervisors [3]. of a simple DNS server with OS included is shown In this paper we present IncludeOS, a single- to require only 158 kb of disk space and to require tasking operating system designed for virtualized 5-20% less CPU-time, depending on hardware, com- environments. IncludeOS provides a novel way for pared to the same binary running on Linux. developers to build their C++-based code directly Index Terms—unikernel, library OS, full virtual- ization, virtio into a virtual machine at compile-time. The key contributions of IncludeOS are: I. INTRODUCTION • Extreme resource efficiency and footprint While cloud computing is rapidly becoming a IncludeOS will use minimal resources com- preferred platform for running services, a major pared to standard operating systems. When concern is the increased energy consumption of idle, it uses no CPU at all. Only the parts of cloud deployments worldwide. In reports by Green IncludeOS required for its single service gets Peace from 2010 and 2014, the combined energy included, which reduces waste and results in consumption of cloud computing surpassed coun- better network and memory performance. tries such as Germany, Canada and Brazil [1], [2], • Efficient deployment process making it the 6th larges energy consumer in the Using a custom GCC-based toolchain, writ- world. For many such deployments, much of the ing #include <os> will literally Include computing power is allocated to running virtual the operating system. During link time the machines, which in turn are running general- build-system will extract whatever the service purpose operating systems. needs from the pre-compiled OS-library and Todays major operating systems were designed form a single binary. A boot sector is attached to run a large number of programs in parallel, and it is all written to an image-file, resulting supporting a huge spectrum of different hardware in a bootable disk image. 978-1-4673-9560-1/15 $31.00 © 2015 IEEE 250 DOI 10.1109/CloudCom.2015.89 • Virtualization platform independence III. INCLUDEOS ARCHITECTURE IncludeOS is written to run on virtualized x86 A. The Zero-overhead principle hardware. The resulting disk-image can be For any service designed to scale by deploying uploaded directly to openStack via the API or more virtual machines, it is crucial that each the web interface, or automatically formatted such machine incurs minimal resource overhead. to fit most virtualization environments, such In contrast to classical operating systems, which as VirtualBox. include as many features as possible, IncludeOS The rest of the paper is organized as follows: aims for true minimality in the sense that nothing Related work is described in Section 2, and the should be included by default that the service does design and architecture of IncludeOS in section 3. not explicitly need. This corresponds to the zero Section 4 shows the resource usage and memory overhead principle of e.g. C++; ”what you don’t performance of IncludeOS compared to a stan- use you don’t pay for” [10, p. 10]. dard operating system, and section 5 demonstrates the performance of a DNS-service running on B. Statically linked libraries and GCC-toolchain IncludeOS, compared to the same binary running The mechanism used for extracting only what on Linux. Our results are discussed in Section 6 is needed from the operating system, is the one with conclusion and future work. provided by default by modern linkers. Each part of the OS is compiled into an object-file, such II. RELATED WORK as ip4.o, udp.o, pci_device.o etc., which The idea of a ”library operating system” dates are then combined using ar to form a static at least back to the exokernel from 1995 [4], library os.a. When a program links with this were the idea was that the applications in a multi library, only what’s necessary will automatically process OS each provide their own abstractions be extracted by the linker and end up in the final over hardware. With the proliferation of virtu- binary. To facilitate this build process a custom alization technologies the idea of single tasking GCC-toolchain has been created. operating systems have regained relevance, and IncludeOS does not have a program loader, so there has been a surge of activity to find the best there is no classical main-function, with param- adaptation for the context of cloud computing. A eters and return value. Instead, a Service-class good overview of is provided by Madhavapeddy is provided, and the user is expected to implement et. al. in [5], where they also present ”Mirage Service::start which will be called after the OS”, and dub this and other recent single-tasking OS has completed initialization. OS’es unikernels. IncludeOS has many features in common with Mirage; it is single tasking, it C. Standard libraries draws only the required OS functionality from an RedHat’s newlib has been chosen as C stan- OS library, and links these parts together with a dard library implementation primarily because it service, to form a bootable virtual machine image, is small, and designed to rely on only a handful also called an appliance. of system calls, and because it compiles into Common for some of the existing solutions is a statically linked library. This way the linker that they are designed for supporting a certain will again naturally include only the parts of the high-level language, such as OCaml in the case of standard library actually used by either the OS Mirage, or Java in the case of OSv [6]. Other so- components or the service itself, leaving out the lutions are limited to partial virtualization relying rest. on the host to provide abstractions, such as threads The C++ standard library is larger and trickier. and drivers, in the case of BSD rump kernels [7], Since the STL containers rely heavily on excep- or directly sharing its kernel, such as with Docker tions we have chosen not to use these inside the [8]. Others again are highly specialized, such as kernel, but instead an exception-free variant by ClickOS, which is designed specifically to run Electronic Arts, EASTL [11]. While this imple- ”Click-based middleboxes”, i.e. the Click modular mentation contains the most important parts of router [9]. STL such as string, streams, vector and map, 251 it is not complete so several features have been 2) Delegates as connections between modules: implemented, and some are still in the pipeline. We use ”The fastest possible C++ delegates”1 as Our version of this library will be available to the connections between modules. The delegates were service (i.e. userspace), but future work is likely tested and found to yield identical performance to to include a port of a full-featured implementation plain C function pointers. Connecting two network such as GCC’s libstdc++. modules via delegates only, essentially means that each object only knows about the function D. Virtio network driver signature of one relevant member function (i.e. IncludeOS currently has only one device driver, method) in the class it connects to. For example, namely a VirtioNet Device driver. The key benefit the UDP class only knows that it has a pointer to a of virtio is that the hypervisor does not need class called ”network layer”, on which it can call to emulate a certain physical device, but instead the ”transmit” function to pass outbound packets can insert data directly into a queue in mem- down stream. In the current implementation, each ory shared by the guest. While Virtio 1.0 has module is compiled separately, and are instantiated recently emerged as an OASIS standard [12], and connected to each other at run-time by the none of the hypervisors used during development Inet-object.