Speculative Execution as an Operating System Service Michael E. Locasto Angelos D. Keromytis Dept. of Computer Science Dept. of Computer Science Columbia University Columbia University [email protected] [email protected] Abstract tion be inside the sandbox or virtual machine. It is our hypothesis that supervising the entire execution of a pro- Software faults and vulnerabilities continue to present cess is not necessary. Instead, we advocate a virtualiza- significant obstacles to achieving reliable and secure tion approach in which only portions of an application's software. In an effort to overcome these obstacles, sys- execution are supervised. Reducing the amount of su- tems often incorporate self-monitoring and self-healing pervision seems like it would result in a significant per- functionality. Our hypothesis is that internal monitoring formance increase for most popular applications. This is not an effective long-term strategy. However, moni- paper examines the use of virtualization to abstract ac- toring mechanisms that are completely external lose the cess to the execution of a machine language from within advantage of application-specific knowledge available to the operating system kernel. an inline monitor. To balance these tradeoffs, we present the design of VxF, an environment where both supervi- sion and automatic remediation can take place by specu- 1.1 Virtualization Within an OS latively executing “slices” of an application. VxF intro- Virtualization is a layer of abstraction interposed be- duces the concept of an endolithic kernel by providing tween an underlying resource (often a physical device execution as an operating system service: execution of or component) and “clients” of that resource. Virtualiza- a process slice takes place inside a kernel thread rather tion provides a construct that looks and behaves the same than directly on the system microprocessor. as the real or physical component but is typically imple- mented by a software substitute. This virtual component 1 Introduction enables three critical capabilities: isolation, inspection, and enforcement. A key problem in computer security is the inability of Virtualization is not a new idea; it was first popularly systems to automatically protect themselves from attack. realized in the IBM System/360. Recently, the use of In order survive or deflect current attacks, systems need virtual machines has come back into fashion in both re- an environment where defensive operations, including search and industry to leverage underutilized hardware, remediation, can take place. Recent research on “self reduce management complexity, and provide isolation. healing” systems attempts to address this problem. Most approaches to OS virtualization place the virtual However, it is unlikely that applications can incorpo- layer either above (e.g., UML, VMWare, JVM, etc.) or rate effective self-supervision mechanisms. First, any below (e.g., Xen) the operating system. We propose a such introspective security mechanism will be subject to layer of virtualization within the kernel (VxF) that can attack or subversion along with the rest of the applica- be selectively invoked for arbitrarily fine “slices” of a tion proper. Second, there is no guarantee that the ad process. In this approach, the entire “guest OS” is re- hoc collection of security mechanisms developed for in- duced to a kernel thread that is occasionally invoked, as dividual applications will be implemented correctly or shown in Figure 1. VxF is complimentary to and not a provide complete coverage. An independent, compre- replacement for current VM implementations. hensive, and general supervision mechanism would be VxF introduces the notion of execution as an operat- much more coherent and maintainable. ing system service. It provides support for a set of virtual Most current technologies for supervising or sandbox- executors (virX's) within the kernel. We call this partic- ing application execution require that the entire applica- ular organization an endolithic kernel (“endo-” meaning 1 within and “-lithic” referring to tight integration with the sition [7, 25, 19], which is the basis of many sandboxing rest of the kernel). An endolithic kernel virtualizes the techniques. These approaches differ from VxF primarily CPU within a kernel thread for a portion of a process's because they only seek to detect or contain the damage execution. Although the main motivation for creating rather than provide any way to fix the underlying fault or VxF is to provide an environment in which self-healing vulnerability. In addition, VxF's main operation does not and automatic repair can take place, the framework can perform system call interposition. Instead, virX's super- be leveraged for more than just security; we discuss some vise the execution of a process's instruction stream2. other applications in Section 3. This paper presents a feasibility study; we introduce the notion of an endolithic kernel, illustrate the basic de- 1.2 Motivation and Goals sign concept, and report on our prototype implementa- tion of VxF for the 2.6.15.6 kernel in Section 1 and Sec- Our motivation originates from our work on constructing tion 3. We focus on the mechanism of VxF – discussion an emulator (STEM) [23] to supervise program execu- of the design and construction of the policy layer is de- tion in response to exploits and errors. Unfortunately, the ferred to future work. In order to provide context for use of an emulator imposes a considerable performance VxF's design decisions, we next consider related work overhead since every program instruction is executed in on virtualization and self-healing software systems. software. One way to ease this burden is to limit the scope of emulation to portions of the program suspected of being vulnerable, or to distribute the monitoring task 2 Related Work among a large collection of machines [13]. In addition, our current emulator, STEM, does not follow execution Virtual machine emulation of operating systems or pro- into the kernel; when a system call is invoked, STEM re- cessor architectures to provide a sandboxed environment linquishes control to the kernel, temporarily ending su- is an active area of research [1, 10, 8]. As an interest- pervision and protection until the system call returns. ing twist, King et al. [9] have recently proposed using VxF can be used to help address these shortcomings. VMMs to implement rootkits. Our micro-speculation Most self-healing and automatic reaction mechanisms techinque is akin to approaches [21, 18] that utilize a sec- follow what we term the ROAR (Recognize, Orient, ondary host machine as a sandbox or instrumented hon- Adapt, Respond) workflow. These systems (a) Recog- eypot: work is offloaded to this host, thus minimizing nize a threat or attack has occurred, (b) Orient the sys- exposure to the primary host. tem to this threat by analyzing it, (c) Adapt to the threat by constructing appropriate fixes or changes in state, and 2.1 Speculative Execution finally (d) Respond to the threat by verifying and deploy- ing those adaptations. Speculative execution is a technique used in micropro- One way in which to gain enough time to execute the cessors to execute the instructions in a code branch be- ROAR workflow is to “delegate and wait” by combin- fore the evaluation of the branch conditional is finished. ing micro-sandboxing with speculative execution of po- Micro-speculation introduces an additional layer of spec- tentially vulnerable slices of a process. If this micro- ulative execution in which the acceptance of a particular speculation succeeds, then the results are committed. If execution path is not based on the evaluation of a branch not, then the temporary results are ignored or replaced conditional, but rather a higher-order constraint. according to the particular response strategy being em- Several recent efforts make use of speculation in a ployed. Of course, knowing how long to wait is not a de- number of interesting ways. Work that is closely re- cidable problem1. We are performing related work [14] lated to ours is Oplinger and Lam's proposal [17] for us- on a survey of the length of this window size for various ing thread-level speculation (TLS) to improve software applications. reliability. The key idea is to execute an application's monitoring code in parallel with the primary computa- 1.3 Contributions and Organization tion and roll back the computation “transaction” depend- ing on the results of the monitoring code. Chang and The major contribution of VxF is to add a policy-driven Gibson [5] speculatively execute an application's code layer of indirection to the operating system to intercept during otherwise idle cycles in order to discover targets and examine the actions of a process before they become of future read operations. Similarly, Nightingale et al. “committed” or visible at the architectural level. This [16] discuss ways for performing speculative execution mechanism is accomplished by performing virtualization at the file system level in order to overcome delays in of a process's execution within a kernel thread. An ana- network-mounted file systems. Finally, the Pulse system log to this approach at the OS level is system call interpo- uses speculation to detect and break deadlocks [12]. 2 2.2 Recovery and Repair take advantage of VxF, an application should not have to be recompiled. However, VxF may provide a means Effective remediation strategies remain a challenge. The for exporting control and information to any applications traditional response of protection mechanisms has been that explicitly want to take advantage of it. to terminate the attacked process. This approach is un- VxF adopts an endolithic kernel. Whereas VMM's appealing for a variety of reasons; to wit, the loss of ac- host multiple guest OS's on a single hypervisor, an en- cumlated state is an overarching concern. Furthermore, dolithic kernel “hosts” multiple processes on multiple crashing leaves systems susceptible to the original fault virtual execution engines within the operating system it- upon restart.