A Binary-Compatible Unikernel Pierre Olivier Daniel Chiba∗ Stefan Lankes Virginia Tech, USA Qualcomm Technologies Inc., USA RWTH Aachen University, Germany [email protected] [email protected] [email protected] Changwoo Min Binoy Ravindran Virginia Tech, USA Virginia Tech, USA [email protected] [email protected] Abstract on Virtual Execution Environments (VEE ’19), April 13–14, 2019, Unikernels are minimal single-purpose virtual machines. Providence, RI, USA. ACM, New York, NY, USA, 15 pages. https: They are highly popular in the research domain due to the //doi.org/10.1145/3313808.3313817 benefits they provide. A barrier to their widespread adoption is the difficulty/impossibility to port existing applications 1 Introduction to current unikernels. HermiTux is the first unikernel pro- Unikernels have become popular in academic research, in the viding binary-compatibility with Linux applications. It is form of a virtualized LibOS model bringing numerous bene- composed of a hypervisor and lightweight kernel layer emu- fits: increased security, performance improvements, isolation, lating OS interfaces at load- and runtime in accordance with cost reduction, ease of deployment, etc. Their potential ap- the Linux ABI. HermiTux relieves application developers plication domains are plentiful: cloud- and edge-deployed from the burden of porting software, while providing uniker- micro-services/SaaS/FaaS-based software [8, 29, 30, 41, 63], nel benefits such as security through hardware-assisted vir- server applications [30, 39, 40, 63, 80], NFV [14, 40–42], tualized isolation, swift boot time, and low disk/memory IoT [14, 16], HPC [31], efficient VM introspection/malware footprint. Fast system calls and kernel modularity are en- analysis [79], and regular desktop applications [56, 68]. While abled through binary rewriting and analysis techniques, they are presented as a secure and attractive alternative to as well as shared library substitution. Compared to other containers [41, 55, 78], unikernels still struggle to gain sig- unikernels, HermiTux boots faster and has a lower mem- nificant traction in industry and their adoption rate is quite ory/disk footprint. We demonstrate that over a range of slow [41, 71]. One of the major reasons is the difficulty, and native C/C++/Fortran/Python Linux applications, HermiTux sometimes impossibility, of porting legacy/existing applica- performs similarly to Linux in most cases: its performance tions to current unikernel models [7, 21, 40, 41, 51, 59, 61, 71]. overhead averages 3% in memory- and compute-bound sce- In situations such as the use of compiled proprietary code, narios. the unavailability of an application’s sources makes it im- CCS Concepts • Software and its engineering → Vir- possible for a user to port and run it using any of the exist- tual machines; Operating systems. ing unikernel models. Such binaries are generally stripped and obfuscated, thus disassembling and re-linking with a Keywords Unikernels, Linux Kernel, Binary Compatibility. unikernel layer is not suitable. Even when sources are avail- Virtualization, Operating Systems able, considering unikernel models supporting legacy pro- ACM Reference Format: gramming languages (C/C++) [8, 24, 28, 31, 42], porting a Pierre Olivier, Daniel Chiba, Stefan Lankes, Changwoo Min, and Bi- medium/large-sized or complex codebase can still be diffi- noy Ravindran. 2019. A Binary-Compatible Unikernel. In Proceed- cult [7, 21, 40, 41, 51, 59, 61, 71]. This is due to factors such ings of the 15th ACM SIGPLAN/SIGOPS International Conference as incompatible/missing libraries/features, complex build in- ∗This work was done while Daniel Chiba was at Virginia Tech. frastructures, lack of developer tools (debuggers/profilers), and unsupported languages. Porting complexity is further Permission to make digital or hard copies of all or part of this work for increased as that process requires expertise in both the appli- personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear cation and the considered unikernel model [71]. Because it is this notice and the full citation on the first page. Copyrights for components currently the burden of the application programmer [59], we of this work owned by others than ACM must be honored. Abstracting with believe that this significant porting effort is one of the biggest credit is permitted. To copy otherwise, or republish, to post on servers or to roadblocks preventing wide-spread adoption of unikernels. redistribute to lists, requires prior specific permission and/or a fee. Request The solution we propose is a unikernel that offers binary permissions from [email protected]. compatibility for regular (i.e. Linux) applications, while keep- VEE ’19, April 13–14, 2019, Providence, RI, USA © 2019 Association for Computing Machinery. ing classical unikernel benefits. It allows the development ACM ISBN 978-1-4503-6020-3/19/04...$15.00 effort to be focused on the unikernel layer. In this context, we https://doi.org/10.1145/3313808.3313817 present a prototype named HermiTux, an extension of the VEE ’19, April 13–14, 2019, Providence, RI, USA P. Olivier, D. Chiba, S. Lankes, C. Min and B. Ravindran HermitCore [31] unikernel, which is able to run native (no 2 Background and Motivation recompilation/relinking) Linux executables as unikernels. By Unikernels & Applications Port. A unikernel [40] is an providing this infrastructure, HermiTux transforms the port- application statically compiled with the necessary libraries ing effort from the application programmer into a supporting and a thin OS layer into a binary able to be executed as effort from the unikernel layer developer. In this model, not a virtualized guest on top of a hypervisor. Unikernels are only can unikernel benefits be obtained transparently for qualified as: (A) single purpose: a unikernel contains only native Linux applications, but furthermore it is now possible one application; and (B) single address space: because of (A), to run previously un-portable applications such as propri- there is no need for memory protection within the unikernel, etary software. With HermiTux, the effort to port and run a consequently the application and the kernel share a single legacy application as a unikernel is non-existent, even when address space and all the code executes with the highest its sources are unavailable. HermiTux supports statically and privilege level. dynamically linked executables, is compatible with multi- Such a model provides significant benefits. In terms of se- ple languages (C/C++/Fortran/Python), compilers (GCC and curity, the strong isolation between unikernels provided by LLVM), full optimizations (-O3), and stripped/obfuscated the hypervisor makes them good candidates for cloud deploy- binaries. It supports multithreading and Symmetric Multi- ments. Moreover, a unikernel contains only the necessary Processors (SMP), checkpoint/restart and migration. Finally, software needed to run a given application. Combined with HermiTux offers intuitive debugging and profiling tools. We the very small size of the kernel, this leads to a significant demonstrate HermiTux on a set of native Linux applications reduction in the application attack surface compared to regu- on the x86-64 architecture. Their performance running in lar VMs [39]. Some unikernels are also written in languages HermiTux is mostly similar to a Linux execution. providing memory-safety guarantees [9, 40, 74]. Concerning The first challenge HermiTux tackles is how to provide performance, unikernel system calls are fast because they binary compatibility? To that end, HermiTux sets up the exe- are common function calls: there is no costly world switch cution environment and emulates OS interfaces at runtime in between privilege levels [31]. Context switches are also swift accordance with Linux’s Application Binary Interface (ABI). as there is no page table switch or TLB flush. In addition to A custom hypervisor-based ELF loader is used to run a Linux the codebase reduction due to small kernels, unikernel OS binary alongside a minimal kernel in a single address space layers are generally modular: it is possible to configure them Virtual Machine (VM). System calls made by the program are to include only the necessary features for a given application. redirected to the implementations the unikernel provides. A Small size and modularity lead to a reduction in resource second challenge HermiTux faces is how to maintain uniker- usage (RAM, disk), which translates into cost reduction for nel benefits while providing such binary compatibility? Some the cloud user, and high per-host VM density for the cloud come naturally (small disk/memory footprints, virtualization- provider [37, 41]. enforced isolation), while others (fast system calls and kernel Porting existing software to run as a unikernel in order modularity) pose technical challenges when assuming no to reap these benefits can be difficult or even impossible. access to sources. To enable such benefits, HermiTux uses First, in some situations, the unavailability of an applica- binary rewriting and analysis techniques for static executa- tion’s sources (proprietary software) makes porting it to bles, and substitutes at runtime a unikernel-aware C library any existing unikernel impossible, as all require recompila- for dynamically linked executables. Finally, HermiTux is op- tion/relinking. Second, porting legacy software to a uniker- timized