MSR-TR-2006-51 This is a draft paper that is under submission. Please contact Galen Hunt for citation information. Sealing OS Processes to Improve Dependability and Security Galen Hunt, Mark Aiken, Paul Barham, Manuel Fähndrich, Chris Hawblitzel, Orion Hodson, James Larus, Steven Levi, Nick Murphy, Bjarne Steensgaard, David Tarditi, Ted Wobber, Brian Zill Microsoft Research Abstract On most modern operating systems, a process is a hardware-protected abstraction for executing potentially mutable code and data. Common features of processes include: dynamic code loading, dynamic code generation, access to cross-process shared memory, and a universal API. This paper argues that many of the dependability and security weaknesses of modern systems are exacerbated by this open process architecture. Moreover, this architecture impairs the ability of tools to analyze code statically, to improve its performance or dependability. By contrast, a sealed process architecture prohibits dynamic code loading, prohibits self-modifying code, prohibits shared memory, and replaces a universal API with a process-limited API. This paper describes an implementation of a sealed process architecture in the Singularity operating system, discusses its merits, and evaluates its impact. Among the benefits are: improved static program analysis, strong security guarantees, elimination of OS redundancies found in language runtimes such as the JVM and CLR, and better software engineering. 1. Introduction 1.1. Disadvantages of Open Processes The process, as a recognized operating system Although common, open processes are not “free.” abstraction, debuted in Multics [52] in the 1960s. This architecture has negative consequences for Multics processes included support for dynamic code dependability, correctness, security, and performance. loading, dynamic code generation, access to cross- Almost all open process systems use hardware process shared memory, and a universal API capable protection [41] to ensure process isolation. All of the of directly modifying data in another process. systems named above, and many others, rely on paged This open process architecture is now nearly memory management and differentiated user and universal. Aspects of this architecture, such as kernel instructions to ensure process isolation. dynamic code loading and cross-process shared Hardware-based process isolation is so common that it memory, were not in the early versions of UNIX [38] is generally assumed to be a “free” processor feature. or early PC operating systems. However, their Unfortunately, hardware-based isolation is not free. In absences reflected limited address spaces and a lack of Aiken et al. [3], we demonstrated performance costs dynamic linking technology, and this “weakness” was from 2.5% (in a compute-bound task with no paging) quickly remedied. Today, for example, all four aspects to 33% (in an IPC bound task). Page table of open processes are found in FreeBSD, Linux, management and cache and TLB misses are Solaris, and Windows. responsible for this increase in execution time. In the beginning, open processes gained popularity Moreover, the extension mechanism commonly because they improved memory utilization by sharing associated with open processes is a major cause of library code across processes. Today, open process software reliability, security, and backward architecture’s most characteristic feature is the ease compatibility problems. Although extension code is with which the OS or an application can be extended rarely trusted, verified, or even fully correct, it is with plug-ins. New features and functionality are loaded directly into a host’s process with no hard routinely loaded directly into these systems’ kernels interface or boundary between host and extension. The and processes. For example, Microsoft Windows outcome is often unpleasant. For example, Swift supports over 100,000 in-kernel device drivers, which reports that faulty device drivers cause 85% of enable it to control almost any hardware device. Other diagnosed Windows system crashes [47]. Moreover, examples of widely used in-process extensions include because an extension lacks a hard interface, it can use dynamic content extensions in web servers (like private aspects of its host’s implementation, which can ISAPI extensions to Microsoft’s IIS), extended stored constrain evolution of a program and require extensive procedures in databases, and web browser plug-ins. testing to avoid future incompatibilities. 1 MSR-TR-2006-51 This is a draft paper that is under submission. Please contact Galen Hunt for citation information. Dynamic code loading imposes a less obvious tax on behavior is easier to verify, and run-time failures can performance and correctness. Software that can load be contained. code is an open environment, in which it is impossible The rest of the paper is organized as follows. Section to make sound assumptions about the system’s states, 2 describes the sealed process architecture and invariants, or valid transitions. Consider the Java discusses its qualitative merits. Section 3 describes the Virtual Machine (JVM). An interrupt, exception, or implementation of sealed processes in Singularity. thread switch can invoke code that loads a new file, Section 4 contains a quantitative evaluation of the overwrites class and method bodies, and modifies sealed process architecture in Singularity. Section 5 global state [45]. In general, the only feasible way to describes related work and Section 6 contains analyze a program running under such conditions is to conclusions and a discussion of future work. start with the unsound assumption that the environment will not change arbitrarily between any 2. Sealed Process Architecture two operations. A sealed process architecture is an architecture in Open processes complicate security. If the code which the operating system guarantees that the code of running in a process is not known a priori, it is a process cannot be altered once the process starts difficult to rationalize subsequent access control executing and in which the state of a process cannot decisions based on code identity. Perhaps for this be directly modified by another process. reason it is rare for operating systems to provide any A sealed kernel is an OS kernel whose code cannot be strong guarantees about application identity, relying altered after system boot and whose state cannot be instead on the identity of an authenticated user or an modified by any process. An operating system with an application masquerading as a user. Software open kernel could provide sealed processes. However, attestation solutions like Terra [20] and NGSCB [35] in this paper, we shall assume that a sealed process must use hardware and virtual machines to prevent architecture also implies a sealed kernel. modifications to the code of running processes. 1.2. Contributions We can draw an instructive analogy between sealed processes and sealed classes in object-oriented This paper describes a new sealed process languages. Practice has shown that designers and architecture in which the OS guarantees that the code implementers of classes need a mechanism to limit in a process cannot be altered once the process starts class extension and force extensions to use a declared executing. The architecture prohibits dynamic code interface [8] in order to avoiding errors that typically loading, prohibits self-modifying code, prohibits appear only at runtime. Similarly, by running dynamic cross-process sharing of memory, and replaces a extensions in a separate process, the sealed process universal API with a process-limited API. Of architecture increases system dependability by forcing particular import, the sealed architecture incorporates application and system extensions to use declared a single extension mechanism for both OS and interfaces. applications: extension code executes in a new 2.1. Sealed Process Invariants process, distinct from its host. The sealed process architecture maintains four Sealed processes offer many advantages. They invariants: increase the potential of static program analysis to improve performance and reliability. They enable The fixed code invariant: Code within a process stronger security guarantees. They allow the cannot be altered once the process starts elimination of duplication of OS-level features in execution. virtual execution environments such as the Sun’s JVM The state isolation invariant: Data within a and Microsoft’s Common Language Runtime (CLR). process cannot be directly accessed by another Finally, they encourage better software engineering. process. The explicit communication invariant: All We have implemented sealed process in the communication between processes must occur Singularity operating system. Singularity is a new through explicit mechanisms, with explicit system being developed as a basis for more identification of the sender, and explicit receiver dependable system and application software [28, 29]. admission control over incoming communication. Singularity exploits advances in programming The closed API invariant: The API between a languages and tools to create an environment in which process and the system must maintain the fixed software is more likely to be built correctly, program 2 MSR-TR-2006-51 This is a draft paper that is under submission. Please contact Galen Hunt for citation information. code, state isolation, and explicit communication mechanism that could subvert the fixed code, state invariants. isolation, or explicit communication