RedLeaf: Isolation and Communication in a Safe Operating System Vikram Narayanan Tianjiao Huang University of California, Irvine University of California, Irvine David Detweiler Dan Appel University of California, Irvine University of California, Irvine Zhaofeng Li Gerd Zellweger Anton Burtsev University of California, Irvine VMware Research University of California, Irvine Abstract remained prohibitive for low-level operating system code. RedLeaf is a new operating system developed from scratch Traditionally, safe languages require a managed runtime, and in Rust to explore the impact of language safety on operat- specifically, garbage collection, to implement safety. Despite ing system organization. In contrast to commodity systems, many advances in garbage collection, its overhead is high for RedLeaf does not rely on hardware address spaces for isola- systems designed to process millions of requests per second tion and instead uses only type and memory safety of the Rust per core (the fastest garbage collected languages experience language. Departure from costly hardware isolation mecha- 20-50% slowdown compared to C on a typical device driver nisms allows us to explore the design space of systems that workload [28]). embrace lightweight fine-grained isolation. We develop a For decades, breaking the design choice of a monolithic ker- new abstraction of a lightweight language-based isolation nel remained impractical. As a result, modern kernels suffer domain that provides a unit of information hiding and fault from lack of isolation and its benefits: clean modularity, infor- isolation. Domains can be dynamically loaded and cleanly mation hiding, fault isolation, transparent subsystem recovery, terminated, i.e., errors in one domain do not affect the ex- and fine-grained access control. ecution of other domains. Building on RedLeaf isolation The historical balance of isolation and performance is mechanisms, we demonstrate the possibility to implement changing with the development of Rust, arguably, the first end-to-end zero-copy, fault isolation, and transparent recov- practical language that achieves safety without garbage col- ery of device drivers. To evaluate the practicality of RedLeaf lection [45]. Rust combines an old idea of linear types [86] abstractions, we implement Rv6, a POSIX-subset operating with pragmatic language design. Rust enforces type and mem- system as a collection of RedLeaf domains. Finally, to demon- ory safety through a restricted ownership model allowing strate that Rust and fine-grained isolation are practical—we only one unique reference to each live object in memory. This develop efficient versions of a 10Gbps Intel ixgbe network allows statically tracking the lifetime of the object and deallo- and NVMe solid-state disk device drivers that match the per- cating it without a garbage collector. The runtime overhead formance of the fastest DPDK and SPDK equivalents. of the language is limited to bounds checking, which in many cases can be concealed by modern superscalar out-of-order 1 Introduction CPUs that can predict and execute the correct path around Four decades ago, early operating system designs identified the check [28]. To enable practical non-linear data structures, the ability to isolate kernel subsystems as a critical mecha- Rust provides a small set of carefully chosen primitives that nism for increasing the reliability and security of the entire allow escaping strict limitations of the linear type system. system [12,32]. Unfortunately, despite many attempts to in- Rust is quickly gaining popularity as a tool for development troduce fine-grained isolation to the kernel, modern systems of low-level systems that traditionally were done in C [4, remain monolithic. Historically, software and hardware mech- 24, 40, 47, 50, 65]. Low-overhead safety brings a range of anisms remain prohibitively expensive for isolation of subsys- immediate security benefits—it is expected, that two-thirds tems with tightest performance budgets. Multiple hardware of vulnerabilities caused by low-level programming idioms projects explored the ability to implement fine-grained, low- typical for unsafe languages can be eliminated through the overhead isolation mechanisms in hardware [84,89,90]. How- use of a safe language alone [20, 22, 67, 69, 77]. ever, focusing on performance, modern commodity CPUs Unfortunately, recent projects mostly use Rust as a drop- provide only basic support for coarse-grained isolation of in replacement for C. We, however, argue that true benefits user applications. Similarly, for decades, overheads of safe of language safety lie in the possibility to enable practical, languages that can provide fine-grained isolation in software lightweight, fine-grained isolation and a range of mechanisms that remained in the focus of systems research but remained on a tight co-design of the language and operating system to impractical for decades: fault isolation [79], transparent de- implement its isolation mechanisms. Nevertheless, several re- vice driver recovery [78], safe kernel extensions [13,75], fine- cent systems suggesting the idea of using Rust for lightweight grained capability-based access control [76], and more. isolation, e.g., Netbricks [68] and Splinter [47], struggled to RedLeaf1 is a new operating system aimed at exploring the articulate the principles of implementing isolation, instead impact of language safety on operating system organization, falling back to substituting fault isolation for information and specifically the ability to utilize fine-grained isolation hiding already provided by Rust. Similar, Tock, a recent oper- and its benefits in the kernel. RedLeaf is implemented from ating system in Rust, supports fault isolation of user processes scratch in Rust. It does not rely on hardware mechanisms for through traditional hardware mechanisms and a restricted sys- isolation and instead uses only type and memory safety of the tem call interface, but fails to provide fault isolation of its Rust language. device drivers (capsules) implemented in safe Rust [50]. Despite multiple projects exploring isolation in language- Our work develops principles and mechanisms of fault iso- based systems [6, 35, 39, 85] articulating principles of iso- lation in a safe language. We introduce an abstraction of lation and providing a practical implementation in Rust re- a language-based isolation domain that serves as a unit of mains challenging. In general, safe languages provide mech- information hiding, loading, and fault isolation. To encapsu- anisms to control access to the fields of individual objects late domain’s state and implement fault isolation at domain (e.g., through pub access modifier in Rust) and protect point- boundary, we develop the following principles: ers, i.e., restrict access to the state of the program transi- • Heap isolation We enforce heap isolation as an invari- tively reachable through visible global variables and explicitly ant across domains, i.e., domains never hold pointers into passed arguments. Control over references and communica- private heaps of other domains. Heap isolation is key for tion channels allows isolating the state of the program on termination and unloading of crashing domains, since function and module boundaries enforcing confidentiality and no other domains hold pointers into the private heap of a integrity, and, more generally, constructing a broad range of crashing domain, it’s safe to deallocate the entire heap. least-privilege systems through a collection of techniques To enable cross-domain communication, we introduce a explored by object-capability languages [59]. special shared heap that allows allocation of objects that Unfortunately, built-in language mechanisms alone are not can be exchanged between domains. sufficient for implementing a system that isolates mutually • Exchangeable types To enforce heap isolation, we in- distrusting computations, e.g., an operating system kernel that troduce the idea of exchangeable types, i.e., types that relies on language safety for isolating applications and kernel can be safely exchanged across domains without leaking subsystems. To protect the execution of the entire system, the pointers to private heaps. Exchangeable types allow us to kernel needs a mechanism that isolates faults, i.e., provides statically enforce the invariant that objects allocated on a way to terminate a faulting or misbehaving computation in the shared heap cannot have pointers into private domain such a way that it leaves the system in a clean state. Specif- heaps, but can have references to other objects on the ically, after the subsystem is terminated the isolation mech- shared heap. anisms should provide a way to 1) deallocate all resources • Ownership tracking To deallocate resources owned by that were in use by the subsystem, 2) preserve the objects a crashing domain on the shared heap, we track owner- that were allocated by the subsystem but then were passed ship of all objects on the shared heap. When an object to other subsystems through communication channels, and is passed between domains we update its ownership 3) ensure that all future invocations of the interfaces exposed depending on whether it’s moved between domains or by the terminated subsystem do not violate safety or block borrowed in a read-only access. We rely on Rust’s own- the caller, but instead return an error. Fault isolation is chal- ership discipline to enforce that domains lose ownership lenging in the face of semantically-rich interfaces encouraged when