Exokernel: an operating system architecture for application-level resource management Dawson R. Engler, M. Frans Kaashoek and James O'Toole Jr. M.I.T. Laboratory for Computer Science Cambridge, MA 02139 engler,kaashoek,james ¡ @lcs.mit.edu March 24, 1995 Abstract management that is strongly centralized. Centralized man- agement can con¯ict with application needs, limiting both We describe an operating system architecture that securely performance and ¯exibility. We believe these problems can multiplexes machine resources while permitting an unprece- be solved through distributed, application-level, resource dented degree of application-speci®c customization of tradi- management. To this end, we have designed a kernel that se- tional operating system abstractions. By abstracting physical curely multiplexes machine resources and permits traditional hardware resources, traditional operating systems have sig- operation system abstractions to be implemented ef®ciently ni®cantly limited the performance, ¯exibility, and function- at application-level, so that they can easily be extended, spe- ality of applications. The exokernel architecture removes cialized, or even replaced. these limitations by allowing untrusted software to imple- Traditionally, operating systems hide information about ment traditional operating system abstractions entirely at machine resources behind high-level core abstractions, application-level. choosing particular implementations of abstractions such as We have implemented a prototype exokernel-based sys- processes, ®le system storage, address spaces, inter-process tem that includes Aegis, an exokernel, and ExOS, an un- communication, exception handling, etc. Core abstractions trusted application-level operating system. Aegis de®nes de®ne a virtual machine on which applications execute, and the low-level interface to machine resources. Applications their implementation cannot be replaced by untrusted appli- can allocate and use machine resources, ef®ciently handle cations. We believe that ®xing the implementations of these events, and participate in resource revocation. Measure- traditional operating system abstractions is unacceptable be- ments show that most primitive Aegis operations are 10±100 cause this denies applications the advantages of domain- times faster than Ultrix,a mature monolithic UNIX operating speci®c optimizations. More important, it restricts the ¯ex- system. ExOS implements processes, virtual memory, and ibility of application builders in adding new resource ab- inter-process communication abstractions entirely within a stractions to the operating system because they must resort library. Measurements show that ExOS's application-level to emulating the new abstraction on top of high-level core virtual memory and IPC primitives are 5±50 times faster abstractions. than Ultrix's primitives. These results demonstrate that the Substantial evidence exists that applications can bene®t exokernel operating system design is practical and offers an greatly from having more control over how machine re- excellent combination of performance and ¯exibility. sources are used to implement higher-level abstractions. Ap- pel et al. [4] reported that the high cost of general-purpose virtual memory primitives reduces the performance of persis- 1 Introduction tent stores, garbage collectors, and distributed shared mem- ory systems. Cao et al. demonstrated that application-level Operating systems de®ne the interface between applications control over ®le caching can reduce the number of I/O op- and physical resources. Unfortunately, this interface can erations by up to 80% [9]. Cheriton et al. [21] and Krueger signi®cantly limit the performance and implementation free- et al. [25] showed how application-speci®c virtual mem- dom of applications. This problem arises because the op- ory policies can increase application performance. Stone- erating system abstracts the details of hardware resources braker [43] demonstrated that inappropriate ®le-system im- to provide a more portable and more full-featured interface plementation decisions can have a dramatic impact on the than is directly implemented by the hardware. The end result performance of databases. Thekkath et al. [45] showed that of such a full-featured interface is an approach to resource by deferring signal handling to applications the cost of ex- ceptions can be reduced by an order of magnitude. This work was supported in part by the Advanced Research Projects Agency under contracts N00014-94-1-0985 and by a NSF National Young We have designed a new operating system architecture in Investigator Award. which traditional operating system abstractions are imple- 1 DRAFT COPY Ð Do not distribute or cite. 2 mented entirely at application level by untrusted software. In summarize performance measurements of Aegis and ExOS this architecture, an exokernel securely multiplexes available (Sections 4 and 5), discuss global optimizations (Section 6), hardware resources. Using the exokernel, applications can summarize related work (Section 7), and report our conclu- securely bind to machine resources, ef®ciently handle events, sions (Section 8). and participate in a resource revocation protocol. The ex- okernel interface is very low-level and can be implemented extremely ef®ciently. Library operating systems, working 2 Motivation for Exokernels above the exokernel interface, implement higher-level ab- stractions and can de®ne special-purpose implementations Traditionally, operating systems have centralized resource that best meet the performance and functionality goals of management in a set of core abstractions that cannot be spe- applications. cialized, extended, or replaced. Whether provided by the We have implemented a prototype exokernel-based system kernel or by trusted user-level servers, these core abstrac- that includes an exokernel (Aegis) and an untrusted library tions are implemented by privileged software that must be operating system (ExOS). This system demonstrates several used by all applications, and therefore cannot be changed important properties of the exokernel architecture: by untrusted software. Typically, the core abstractions de- ®ned by the operating system include processes, ®le storage, Low-level secure multiplexing of hardware resources address spaces, and inter-process communication. can be implemented ef®ciently. In this section, we argue that ®xing the implementation of these high-level abstractions can reduce the performance, Traditional core abstractions can be implemented ef®- increase the complexity, and limit the functionality of appli- ciently at application-level. cation programs. We then give an end-to-end argument for Applications can create special-purpose implementa- the exokernel architecture and discuss the role of application- tions of core abstractions. level library operating systems. In practice, our implementation provides applications with 2.1 The Cost of Core Abstractions greater ¯exibility and better performance than in a mono- lithic system. Aegis's low-level interface allows application- Application performance suffers because there is no sin- level software, such as ExOS, to manipulate resources very gle way to abstract physical resources or to implement a ef®ciently. Aegis's protected control transfer is three times core abstraction that is best for all applications. In imple- faster than the best reported implementation [29]. Aegis's menting a core abstraction, the operating system is forced exception forwarding and control transfers are close to to make trade-offs between support for sparse or dense ad- 100 times faster than in Ultrix 4.2, a mature monolithic dress spaces, read-intensive or write-intensive workloads, system using identical hardware. Because of this ef®ciency, etc. Any such trade-off penalizes some applications, and of- ExOS is able to implement virtual memory entirely at appli- ten the applications that suffer most are those whose behav- cation level. ior is the most predictable. Relational databases and garbage Aegis also permits ExOS (or other application-level soft- collectors sometimes have very predictable data access pat- ware) ¯exibility that is not available in microkernel-based terns, and their performance suffers when a general-purpose systems. Aegis's ef®cient protected control transfer allows page replacement strategy such as LRU is imposed by the applications to trade between a wide array of IPC semantics operating system. that differ in performance by a factor of 10. In contrast, High-level core abstractions hide information from microkernel systems such as Amoeba [44], Chorus [39], application-level (untrusted) software. For example, most Mach [1], and V [13], do not allow untrusted application current systems do not make low-level exceptions, timer software to de®ne specialized IPC primitive because vir- interrupts, or raw device I/O directly available to applica- tual memory and message passing services are implemented tions. Unfortunately, hiding this information makes it dif®- by the kernel and trusted servers. Similarly, many other cult or impossible for applications to implement their own microkernel abstractions, such as page-table structures and resource management abstractions. For example, database process abstractions, are ®xed. Finally, many of the hard- implementations must struggle to emulate random-access ware resources in microkernel systems, such as the network, record storage on top of ®le systems [43]. Implementing screen, and disk, are encapsulated