DOI:10.1145/2541883.2541895 Article development led by queue.acm.org What if all the software layers in a virtual appliance were compiled within the same safe, high-level language framework? BY ANIL MADHAVAPEddy AND DAVID J. SCOTT Unikernels: The Rise of the Virtual Library Operating System CLOUD COMPUTING HAS been pioneering the business of renting computing resources in large data centers to multiple (and possibly competing) tenants. The basic enabling technology for the cloud is operating-system virtualization such as Xen1 or VMWare, which allows customers to multiplex virtual machines (VMs) on a shared cluster of physical machines. actions. They can be centrally backed Each VM presents as a self-contained up and migrated across different phys- computer, booting a standard operating- ical hosts without interrupting service. system kernel and running unmodified Today commercial providers such as applications just as if it were executing Amazon and Rackspace maintain vast on a physical machine. data centers that host millions of VMs. A key driver to the growth of cloud These cloud providers relieve their computing in the early days was server customers of the burden of managing consolidation. Existing applications data centers and achieve economies of were often installed on physical hosts scale, thereby lowering costs. that were individually underutilized, While operating-system virtual- and virtualization made it feasible to ization is undeniably useful, it adds pack them onto fewer hosts without yet another layer to an already highly requiring any modifications or code layered software stack now including: recompilation. VMs are also managed support for old physical protocols (for via software APIs rather than physical example, disk standards developed JANUary 2014 | VOL. 57 | NO. 1 | COMMUNICATIONS OF THE ACM 61 practice in the 1980s such as IDE); irrelevant This problem has received a lot modular components that are flexible, optimizations (for example, disk el- of thought at the University of Cam- secure, and reusable in the style of a evator algorithms on SSD drives); bridge, both at the Computer Labora- library operating system. What would backward-compatible interfaces (for tory (where the Xen hypervisor originat- the benefits be if all the software lay- example, Posix); user-space processes ed in 2003) and within the Xen Project ers in an appliance could be compiled and threads (in addition to VMs on a (custodian of the hypervisor that now within the same high-level language hypervisor); and managed-code run- powers the public cloud via companies framework instead of dynamically as- times (for example, OCaml, .NET, or such as Amazon and Rackspace). The sembling them on every boot? First, Java). All of these layers sit beneath the solution—dubbed MirageOS—has its some background information about application code. Are we really doomed ideas rooted in research concepts that appliances, library operating systems, to adding new layers of indirection have been around for decades but are and type-safe programming languages. and abstraction every few years, leav- only now viable to deploy at scale since The shift to single-purpose appli- ing future generations of program- the availability of cloud-computing re- ances. A typical VM running on the mers to become virtual archaeologists sources has become more widespread. cloud today contains a full operating- as they dig through hundreds of layers The goal of MirageOS is to restruc- system image: a kernel such as Linux or of software emulation to debug even ture entire VMs—including all ker- Windows hosting a primary application the simplest applications?5,18 nel and user-space code—into more running in user space (for example, MySQL or Apache), along with second- Figure 1. Software layers and a stand-alone kernel compilation. ary services (for example, syslog or NTP) running concurrently. The generic soft- ware within each VM is initialized every Configuration Files Mirage Compiler time the VM is booted by reading con- application source code Application Binary figuration files from storage. configuration files Language Runtime hardware architecture Despite containing many flexible whole-system optimization layers of software, most deployed VMs Parallel Threads ultimately perform a single function User Processes Application Code such as acting as a database or Web OS Kernel Mirage Runtime specialized server. The shift toward single-purpose unikernel Hypervisor Hypervisor VMs is a reflection of just how easy it has become to deploy a new virtual Hardware Hardware } computer on demand. Even a decade ago, it would have taken more time and money to deploy a single (physical) ma- chine instance, so the single machine Figure 2. Logical workflow in MirageOS. would need to run multiple end-user applications and therefore be carefully configured to isolate the constituent OCaml code git device Programmer resolve services and users from each other. repositories drivers dependencies The software layers that form a VM have not yet caught up to this trend, dependency SAT git protocols analysis solver algorithms and this represents a real opportunity for optimization—not only in terms application full static type source source Branch-consistent of performance by adapting the appli- checking module source ance to its task, but also for improv- config application ing security by eliminating redundant compiler files Compiler functionality and reducing the attack surface of services running on the pub- config whole program tree http lic cloud. Doing so statically is a chal- optimization lenge, however, because of the struc- http ture of existing operating systems. linker Limitations of current operating systems. The modern hypervisor pro- xen service dns unikernel monitor dhcp vides a resource abstraction that can outputs be scaled dynamically—both verti- cally by adding memory and cores, openstack Compute binary and horizontally by spawning more Cloud repository Branch-consistent VMs. Many applications and operating binaries systems cannot fully utilize this capa- bility since they were designed before modern hypervisors came about (and 62 COMMUNICATIONS OF THE ACM | JANUary 2014 | VOL. 57 | NO. 1 practice the physical analogues such as mem- existential questions about operating ory hotplug were never ubiquitous in systems. Several research groups have commodity hardware). Often, exter- proposed operating-system designs nal application-level load balancers based on an architecture known as a are added to traditional applications library operating system (or libOS). The running in VMs in order to make the The goal of first such systems were Exokernel6 and service respond elastically by spawn- MirageOS is to Nemesis10 in the late 1990s. In a libOS, ing new VMs when load increases. protection boundaries are pushed to Traditional systems, however, are not restructure entire the lowest hardware layers, resulting optimized for size or boot time (Win- VMs—including in: a set of libraries that implement dows might apply a number of patches mechanisms, such as those needed to at boot time, for example), so the load all kernel and drive hardware or talk network proto- balancer must compensate by keep- cols; and a set of policies that enforce ing idle VMs around to deal with load user-space code— access control and isolation in the ap- spikes, wasting resources and money. into more modular plication layer. Why couldn’t these problems with The libOS architecture has several operating systems simply be fixed? components that advantages over more conventional Modern operating systems are in- are flexible, secure, designs. For applications where per- tended to remain resolutely general formance—and especially predictable purpose to solve problems for a wide and reusable in performance—is required, a libOS audience. For example, Linux runs on the style of a library wins by allowing applications to ac- an incredibly diverse set of platforms, cess hardware resources directly with- from low-power mobile devices to operating system. out having to make repeated privilege high-end servers powering vast data transitions to move data between user centers. Compromising this flexibility space and kernel space. The libOS simply to help one class of users im- does not have a central networking prove application performance would service into which both high-priority not be acceptable. network packets (such as those from a On the other hand, a specialized videoconference call) and low-priority server appliance no longer requires packets (such as from a background an OS to act as a resource multiplexer file download) are forced to mix and since the hypervisor can do this at a interfere. Instead, libOS applications lower level. One obvious problem with have entirely separate queues, and this approach is that most existing packets mix together only when they code presumes the existence of large arrive at the network device. but rather calcified interfaces such as The libOS architecture has two big POSIX or the Win32 API. Another po- drawbacks. First, running multiple tential problem is that conventional applications side by side with strong operating systems provide services resource isolation is tricky (although such as a TCP/IP stack for communica- Nemesis did an admirable job of min- tion and a file-system interface for stor- imizing crosstalk between interactive ing persistent data: in our brave new applications). Second, device drivers world, where would these come from? must be rewritten to fit the new mod- The MirageOS architecture— el. The fast-moving world of commod- dubbed unikernels—is outlined in Fig- ity PC hardware meant that, no mat- ure 1. Unikernels are specialized OS ter how many graduate students were kernels that are written in a high-level tasked to write drivers, any research language and act as individual soft- libOS prototype was doomed to be- ware components. A full application come obsolete in a few short years. (or appliance) consists of a set of run- This approach worked only in the ning unikernels working together as a real-time operating-system space (for distributed system.