Towards Application Security on Untrusted Operating Systems Dan R. K. Ports Tal Garfinkel MIT CSAIL & VMware, Inc.∗ VMware, Inc. [email protected] [email protected] Abstract OS compromise, the implications of continuing to rely on OS services if they turn malicious are poorly understood. Complexity in commodity operating systems makes We explore this problem and potential solutions in the compromises inevitable. Consequently, a great deal of context of Overshadow, a virtualization-based system we work has examined how to protect security-critical por- developed that protects applications in a VM from the tions of applications from the OS through mechanisms guest operating system in that VM. Overshadow attempts such as microkernels, virtual machine monitors, and new to maintain the secrecy and integrity of an application’s processor architectures. Unfortunately, most work has data even if the OS is completely compromised. For each focused on CPU and memory isolation and neglected major OS component, we examine how malicious behav- OS semantics. Thus, while much is known about how ior could undermine application secrecy and integrity, to prevent OS and application processes from modifying and suggest potential mitigations. While we present our each other, far less is understood about how different OS analysis and solutions in the context of Linux and Over- components can undermine application security if they shadow, they are more generally applicable to any system turn malicious. attempting to secure application execution in the face of We consider this problem in the context of our work a compromised OS. on Overshadow, a virtual-machine-based system for We begin with a review of systems that enforce iso- retrofitting protection in commodity operating systems. lation between protected applications and untrusted op- We explore how malicious behavior in each major OS sub- erating systems in Section 2. Next, we explain how a system can undermine application security, and present malicious OS can subvert them, using false system call potential mitigations. While our discussion is presented return values to trick an application into revealing its in terms of Overshadow and Linux, many of the prob- secrets, and argue for a solution based on a verifiable lems and solutions are applicable to other systems where system call interface in Section 3. Finally, we examine trusted applications rely on untrusted, potentially mali- the implications of making specific OS components un- cious OS components. trusted, and propose defenses against possible attacks, in Section 4. 1 Introduction 2 Isolation Architectures Commodity operating systems are tasked with storing and processing our most sensitive information, from man- Traditional commodity operating systems are monolithic aging financial and medical records to carrying out online programs that directly manage all hardware, so any OS purchases. The generality and rich functionality these exploit results in total system compromise. A variety systems provide leads to complexity in their implementa- of architectures have been proposed to mitigate the im- tion, and hence their assurance often falls short. pact of an OS compromise by providing memory and Many solutions have been proposed for enhancing the CPU isolation using a separate layer below the commod- assurance of these systems. Some use microkernels to ity OS, either in the form of a virtual machine monitor, contain potentially malicious behavior by placing major microkernel or architectural extensions — taking sole OS and application components into separate isolated responsibility for protection out of the hands of the OS. processes [11]. Others, such as NGSCB (formerly Palla- These architectures prevent a compromised OS from dium) [3], Proxos [12], XOM [7], and Overshadow [2], reading or modifying application CPU and memory state. attempt to retrofit orthogonal, higher assurance execution We are primarily interested not in these attacks but rather environments alongside a commodity OS, allowing part what comes next, i.e. once we have prevented direct at- or all of an application to run in a protected environment, tacks on application state, how we can prevent higher- but still use OS services. Unfortunately, while these sys- level attacks based on modifying the semantics of system tems provide CPU and memory isolation in the face of calls. However, CPU and memory isolation is a prereq- uisite for our work, so we begin with a survey of these ∗Work done during an internship at VMware, Inc. isolation architectures. Microkernel-based architectures, such as those based on L4 [5], divide both the application and OS into mul- tiple components, placing each in its own process. In Proxos, a virtual-machine-based system, trusted compo- nents including the protected applications and a trusted “private OS” are placed in a single VM, that runs along- side another VM running a commodity OS [12]. A proxy forwards the application’s system calls to either the pri- vate OS or commodity OS, depending on an application- specific security policy. NGSCB takes a similar, but less backwards-compatible approach, by requiring the appli- cation to be refactored into a trusted part which runs in its own secure compartment and an untrusted part which runs on the commodity OS [3]. XOM is a processor ar- chitecture that isolates protected applications from each other and the operating system using a combination of register-level tagging and cryptographic techniques [7]. While seemingly different, all these architectures pro- vide the same basic property: some or all of the applica- Figure 1: Overshadow architecture. Trusted components tion is run in a separate protection domain from the OS. are indicated with a shaded background. In addition This protects the secrecy and integrity of resources like to the standard virtualization barrier that isolates entire registers and memory, and ensures control flow integrity, VMs from each other and the monitor, Overshadow adds i.e. control flow cannot be altered except via well-defined a second virtualization barrier that isolates protected ap- protocols such as signal delivery and system calls. plications from other components inside the same VM. Overshadow. Overshadow [2] is a system we devel- oped for protecting applications running inside a VM implement system calls securely. from the operating system in that VM. Overshadow works Figure 1 shows the components required to protect an by leveraging the virtual machine monitor’s control over application with Overshadow. In addition to the applica- memory mappings to present different views of memory tion itself, the shim, VMM, and hardware are trusted. The pages depending on the execution context. When an ap- OS kernel and all other applications, including privileged plication accesses its own memory, access proceeds as daemons and other system services, are untrusted. normal; if the OS or another application accesses it, the Overshadow uses a simple protection model based on VMM transparently encrypts the memory page and saves whole-application protection. Each application runs in its a secure hash for integrity verification. This allows all re- own secure compartment identified by a unique security source management functionality to remain in the hands ID (SID). Each process or thread thus has an associated of the OS, without compromising secrecy and integrity: SID. Every resource, such as a memory region or file, has for example, the OS can swap memory pages to disk and an owner SID. Access to a resource is permitted only if its restore them, but cannot read or modify their contents. SID is the same as that of the currently-executing thread. To adapt applications to this new execution environ- Though simple, this model is sufficient for many interest- ment, without requiring any modifications to the applica- ing applications. It would be relatively straightforward to tion or OS, we add a shim to each application at load time. extend this model to support trust relationships between The shim is user-level code (akin to a shared library), but applications and more sophisticated access control for communicates directly with the VMM via a hypercall shared resources. interface. The shim and VMM together provide secure control transfer by saving execution state in protected 3 Implications of a Malicious OS memory when an application is interrupted, and restoring The protection architectures in the previous section guar- it when the application resumes execution. The VMM antee that applications that do not interact explicitly with also redirects system calls to the shim, which adapts their the OS execute correctly, because the integrity of code semantics as necessary for compatibility; for example, and data is protected, and control flow proceeds as nor- data being passed to the OS is automatically copied to mal. Assuming the basic protection architecture is sound, unprotected memory so that the OS can access it. In no loss of integrity or data leakage is possible, although the extensions to Overshadow we propose in Section 4, availability is not guaranteed because the OS could sim- we make use of this system call interposition to help ply stop scheduling the application. However, any non-trivial program makes system calls, system calls are so intimately related to the OS im- and this presents an opportunity for a malicious OS to plementation that their correctness cannot be guaran- influence the program’s data and control flow by manipu- teed — for example, those related to kernel modules lating system call results. To take a simple example, if a or scheduling policies. multithreaded program relies on the OS to implement a • We can emulate the system call in trusted code. mutual-exclusion lock, the OS could grant the lock to two This option should also be used sparingly, since threads at once. Since the OS also handles scheduling, reimplementing system calls adds to the size of the this has the potential to create a race condition even in TCB. Nevertheless, it may be the best option for correctly written code, with arbitrarily bad consequences.