Twizzler: an Operating System for Next-Generation Memory Hierarchies

Twizzler: An Operating System for Next-Generation Memory Hierarchies Technical Report UCSC-CRSS-17-01 UCSC-SSRC-17-01 December 5th, 2017 Daniel Bittman Matt Bryson Yuanjiang Ni [email protected] [email protected] [email protected] Arjun Govindjee Isaak Cherdak [email protected] [email protected] Pankaj Mehra Darrell Long Ethan Miller [email protected] [email protected] [email protected] Center for Research in Storage Systems Storage Systems Research Center Baskin School of Engineering University of California, Santa Cruz Santa Cruz, CA 95064 http://crss.ucsc.edu/ http://ssrc.ucsc.edu/ Abstract The introduction of NVDIMMs (truly non-volatile and directly accessible) requires us to rethink all lev- els of the system stack, from processor features to applications. Operating systems, too, must evolve to support new I/O models for applications access- ing persistent data. We are developing Twizzler, an operating system for next-generation memory hierarchies. Twizzler is designed to provide applications with direct access to persistent storage while providing mechanisms for cross-object pointers, removing Figure 1: Expanded form of the memory hierarchy. itself from the common access path to persistent We expect non-volatile memory to have asymmet- data, and providing fine-grained security and recov- ric access time with latency similar to DRAM [17]. erability. The design of Twizzler is a better fit for We expect NVM presence to range from NVM-only low-latency and byte-addressable persistent storage systems (IoT devices) to a mix between DRAM and than prior I/O models of contemporary operating NVM. systems. We are in the process of building a prototype inside FreeBSD, allowing us to leverage and explore the ideas presented in this report without in- Given a new memory hierarchy including byte- vesting the time to build an operating system from addressable non-volatile memory (shown in Fig- scratch first. ure 1), we have a chance to revisit operating sys- The purpose of this report is to provide an tem design. Existing systems maintain separate overview of Twizzler and the design behind it while management domains for volatile, high-speed mem- giving a direction for our upcoming work. We in- ory, on which computation operates, and persistent tend to further develop our ideas by extending our high-latency storage, which cannot be operated on FreeBSD prototype, followed by designing a kernel directly. Operating systems designed for such sys- and hardware extensions to make better use of some tems reflect this structure and provide applications of our designs (resulting in improved security, sim- with interfaces to interact with the memory hierar- plicity, and improved support for persistent kernel chy. System calls such as write() and read() re- state). flect the basic structure of using DRAM as a cache for disk, while mmap() uses virtual memory to pro- 1 Introduction vide the illusion of direct persistent object access. Both cases are fundamentally the same, with the We are on the cusp of a fundamental shift in operating system interposing itself between the ap- the way that we program computers. For seven plication and its desired actions of persistent data decades, programmers have written programs that access, with the need to manage ephemeral copies are loaded, initialized, executed, and terminated. of persistent data in volatile memory. Moreover, In the very near future, multiple vendors will in- the existing approach to persistence via block stor- troduce persistent memory that will attach to the age requires data serialization, exacting a penalty memory bus. Not since the days of small magnetic that is becoming increasingly larger as NVM perfor- core memories fifty years ago have we had mutable mance increases [62]. Applications that use NVM- memory from which our programs execute directly optimized techniques for persistence [10, 34, 64] and still survive power cycling. Unlike flash and avoid this overhead, but the problem of managing PCIe based memories, these new memories will be persistent pointers in a very large address space re- byte addressable. Consequently, we are entering a mains. new era, and it is essential that the operating sys- To address these challenges, our lab has been tem and other system software make the best pos- developing Twizzler, an operating system designed sible use of this new technology. However, leaving to support persistent, byte-addressable memory by the design choices solely to operating system ven- “getting out of the way” of applications as much as dors risks incremental change and the inability to possible as they operate on persistent data. We are benefit from the dramatic improvements in perfor- focusing on the following items that we believe to be mance, programming models, and security that this key requirements and challenges of an NVM-aware may facilitate. system: 1 Persistent Object Access The I/O model that dress space size and reintroduce segmentation sup- the operating system provides must allow applica- port. While existing processors require us to emit tions direct access to persistent data with mini- additional pointer swizzling and dereference instruc- mal kernel involvement while also providing a rich tions, hardware support to enable direct support for enough model for cross-object data references and our cross-object pointer design and security features a framework for object naming and late-binding. such as protection lookaside buffers could improve Twizzler will maintain a namespace of data objects, performance and simplicity of Twizzler as well. each with a unique 128-bit ID. These objects may contain code, data, or both. Much like the Unix While our eventual goal is to provide this sup- and Plan-9 “everything is a file” approach, we plan port by creating a new operating system (including to have every piece of data on which applications a new kernel), such an “all-or-nothing” approach operate be one of these objects. Since these persis- would require too much initial effort before being tent objects are directly accessible from the proces- able to demonstrate the efficacy of the approach. sor, and because we can uniquely identify objects Thus, we are currently implementing these tech- by ID, we can have “cross-object pointers”. Such niques inside FreeBSD, allowing us to evaluate them pointers refer to data in other objects, and an ap- without the need to build an entire operating sys- plication may use these pointers to read and oper- tem from scratch. This incremental approach has ate on data in other objects. Cross-object pointers a second advantage: it allows system designers to allow us to support more natural programming se- immediately leverage the techniques, which we will mantics for non-volatile memory without the man- open-source, without the need to adopt a new op- ual pointer-loading and swizzling (pointer transla- erating system, thus increasing the impact of the tion) [39] required by applications on current oper- research. Our prototype Twizzler system imple- ating systems [14]. mented in FreeBSD will provide applications with a full Twizzler framework and interface, allowing Security of Persistent Objects Direct access to us to later implement a kernel ourselves to better persistent objects increases the need for richer secu- demonstrate some of our ideas. rity models. Applications must be able to access sensitive or protected data while protecting that 2 Persistent Objects data from untrusted libraries or unverified subcom- ponents. In Twizzler, we separate components of Twizzler’s I/O model follows from our expectations applications into security contexts in order to pro- of upcoming memory hierarchies designs. Cur- vide isolation and access control. Security contexts rent I/O models that use calls such as read() and allow us to protect applications at finer granularity write() are designed for indirect, high-latency ac- than current systems, thereby reducing the risk of cess to persistent storage. These archaic models allowing direct access to persistent data. involve significant operating system overhead that, while acceptable in a two-tier memory hierarchy model, is unacceptable in a memory hierarchy with Persistent Kernel State The kernel must be de- low-latency access to persistent storage. Since per- signed with persistent kernel state in mind. The sistent data is directly accessible, copying data into kernel uses persistent objects accessible from user- temporary buffers, as is common when using read() space to determine the internal kernel state (such as and write(), makes little sense. Instead, we can thread state and address spaces), thereby allowing directly map the same persistent data into multiple the operating system state to be recovered easily af- address spaces without wasteful copying. Zero-copy ter a power failure. The kernel state is then a cache I/O [45, 52, 61] alleviates these problems to some of kernel state objects which can be reconstructed extent, but still has heavy kernel involvement and after a power failure. Kernel state objects have the is still designed for a model with an explicit device added benefit of reducing system calls and simpli- I/O operation to persist and acquire data. While fying the application-kernel boundary. memory mapping (mmap()) provides some relief, it places a heavy burden on application programmers Hardware Support for Persistent Objects who wish to share mapped regions across processes While we can implement our design on existing because the address space layouts differ across pro- hardware, extending processors with new primitives grams and so shared data must be translated in will allow applications to be simpler and have higher some way or be location-dependent. performance. We propose to increase the virtual ad- 2 Reducing copies means that each address space that maps the same data will see the same con- tents. If that data contains pointers, they must have a form which allows any process to dereference the pointer regardless of the mapped location.

Load more