The Architecture of a Reliable Operating System Jorrit N. Herder, Herbert Bos, Ben Gras, Philip Homburg, and Andrew S. Tanenbaum Dept. of Computer Science, Vrije Universiteit Amsterdam, The Netherlands {jnherder, herbertb, beng, philip, ast}@cs.vu.nl Keywords: Operating System Reliability, Design and Implementation, Multiserver Architecture, MINIX 3 Abstract from crashes and other failures, results in a highly re- liable, completely multiserver operating system that In this paper, we discuss the architecture of a fully still looks and feels like UNIX. modular, self-healing operating system, which ex- While some of the mechanisms are well-known, ploits the principle of least authority to provide re- and multiserver operating systems have been first pro- liability beyond that of most other operating systems. posed years ago, to the best of our knowledge, we are The system can be characterized as a minimal kernel the first to explore such an extreme decomposition of with the entire operating system running as a set of the operating system that is designed for reliability, compartmentalized user-mode servers and drivers. while providing reasonable performance. Quite a few By moving most of the code to unprivileged user- ideas and technologies have been around for a long mode processes and restricting the powers of each time, but were often abandoned for performance rea- one, we gain proper fault isolation and limit the dam- sons. We believe that the time has come to reconsider age bugs can do. Moreover, the system has been de- the choices that were made in common operating sys- signed to survive and automatically recover from fail- tem design. ures in critical modules, such as device drivers, trans- parent to applications and without user intervention. 1.1 Contribution We used this design to develop a highly reli- The contribution of this work is the design and able, open-source, POSIX-conformant member of the implementation of an operating system that takes the UNIX family, which is freely available and has been concept of multiserver to an extreme in order to pro- downloaded 50,000 times in the past 3 months. vide a dependable computing platform. The concrete goal of this research is to build a UNIX-like operat- 1 INTRODUCTION ing system that can transparently survive crashes of critical components, such as device drivers. Operating systems are expected to function flaw- As we mentioned earlier, the answer that we came lessly, but, unfortunately, most of today’s operating up with is to break the system into manageable units systems frequently fail. As discussed in Sec. 2, many and rigidly control the power of each unit. The ulti- problems stem from the monolithic design that un- mate goal is that a fatal bug in, say, a device driver derlies most common systems. All operating system should not crash the operating system; instead, the functionality, for example, runs in kernel mode with- failed component should be automatically and trans- out proper fault isolation, so that any bug can poten- parently replaced by a fresh copy, and running user tially trash the entire system. processes should not be affected. Like other groups, we believe that reducing the To achieve this goal, our system provides: simple, operating system kernel is a first important step in yet efficient and reliable IPC; disentangling of inter- the direction of designing for reliability. In partic- rupt handling from user-mode device drivers; sepa- ular, running drivers and other core components in ration of policies and mechanisms; flexible, run-time user mode helps to minimize the damage that may be operating system configuration; decoupling of servers caused by bugs in such code. However, our system and drivers through a publish-subscribe system; and explores an extreme in the design space of UNIX-like error detection and transparent recovery for common operating systems where the entire operating system drivers failures. We will discuss these features in is run as a collection of independent, tightly restricted, more detail in the rest of the paper. user-mode processes. This structure, combined with While microkernels, user-mode device drivers, several explicit mechanisms for transparent recovery multiserver operating systems, fault tolerance, etc. are not new, no one has put all pieces together. We User User User User User User User User User believe that we are the first to realize a fully mod- ular, open-source, POSIX-conformant operating sys- tem that is designed to be highly reliable. The system VFS Inet UNIX server is called MINIX 3. It has been released (with all the Drivers Paging Microkernel source code) and 50,000 people have downloaded it Kernel space so far, as discussed later. Linux Mac OS X (a) (b) 1.2 Paper Outline Figure 1: Two typical monolithic systems discussed in Sec. 2.1: (a) Vanilla Linux and (b) Mac OS X. We first introduce how operating system structures have evolved over time (Sec. 2). Then we proceed with a discussion of the kernel and the organization of 2.2 Single-Server Systems the user-mode servers on top of it (Sec. 3). We review A single-server system has a reduced kernel, and some implementation issues (Sec. 4) and briefly dis- runs a large fraction of the operating system as a sin- cuss the system’s reliability (Sec. 5) and performance gle, monolithic user-mode server. In terms of reliabil- (Sec. 6). Finally, we draw conclusions (Sec. 7). ity, this setup adds little over monolithic systems, as there still is a single point of failure. The only gain in 2 RELATED WORK case of an operating system crash is a faster reboot. An advantage of this setup is that it preserves a This section illustrates the operating system design UNIX environment while one may experiment with a space with three typical structures and some variants microkernel approach. The combination of legacy ap- thereof. While most structures are probably familiar plications and real-time or secure modules allows for to the reader, we introduce them explicitly to show a smooth transition to a new computing environment. an overview of the design space that has monolithic Mach-UX [1] was one of the first systems to run systems at one extreme and ours at the other. BSD UNIX in user-mode on top of the Mach 3 mi- It is sometimes said that virtual machines and ex- crokernel, as shown in Fig. 2(a). Another example, okernels provide sufficient isolation and modularity shown in Fig. 2(b), is Perseus [9], running Linux and for making a system safe. However, these technolo- some specialized components for secure digital signa- gies provide an interface to an operating system, but tures on top of the L4 microkernel. do not represent a complete system by themselves. The operating system on top of a virtual machine or User User User User User User User User exokernel can have any of the following structures. User space BSD Unix L4 Linux GUI 2.1 Monolithic Systems Drivers Sign Monolithic kernels provide rich and powerful ab- Drivers Paging stractions. All operating system services are provided Kernel space Mach 3 L4 by a single, monolithic program that runs in kernel (a) (b) mode; applications run in user mode and can request services directly from the kernel. Figure 2: Two typical single-server systems discussed in Monolithic designs have some inherent problems Sec. 2.2: (a) Mach-UX and (b) Perseus. that affect their reliability. All operating system code, for example, runs at the highest privilege level with- 2.3 Multiserver Systems out proper fault isolation, so that any bug can poten- tially trash the entire system. With millions of lines In a multiserver design, the operating system en- of code (LoC) and 1-16 bugs per 1000 LOC [13, 14], vironment is formed by a set of cooperating servers. monolithic systems are likely to contain many bugs. Untrusted, third-party code such as device drivers Running untrusted, third-party code in the kernel also can be run in separate, user-mode modules to pre- diminishes the system’s reliability, as evidenced by vent faults from spreading. High reliability can be the fact that 70% to 85% of all operating system achieved by applying the principle of least author- crashes are caused by device drivers [2, 11] ity [10], and tightly controlling the powers of each. From a high-level reliability perspective, a mono- A multiserver design also has other advantages. lithic kernel is unstructured. The kernel may be parti- The modular structure, for example, makes system tioned into domains but there are no protection barri- administration easier and provides a convenient pro- ers enforced between the components. Two simplified gramming environment. A detailed discussion is out examples, Linux and MacOS X, are given in Fig. 1. of the scope of this paper. Several multiserver operating systems exist. An Before we continue with the discussion of the com- early system is MINIX [12], which distributed operat- ponents of MINIX 3, we give some examples to illus- ing system functionality over two user-mode servers, trate how our multiserver operating system actually but still ran the device drivers in the kernel, as shown works. Fig. 4 also shows some typical IPC interac- in Fig. 3(a). More recently, IBM Research designed tions initiated by user processes. Although the POSIX SawMill Linux [3], a multiserver environment on top operating system interface is implemented by multi- of the L4 microkernel, as illustrated in Fig. 3(b). ple servers, system calls are transparently targeted to While the goal was a full multiserver variant of Linux, the right server by the system libraries. Four exam- the project never passed the stage of a rudimentary ples are given below: prototype, and was then abandoned when the people (1) The application that wants to create a child pro- working on it left IBM.