Code-Centric Domain Isolation: A Hardware/Software Co-Design for Efficient Program Isolation Llu´ıs Vilanova <[email protected]> Barcelona, 2015 ADVISORS: Nacho Navarro Universitat Politecnica` de Catalunya Barcelona Supercomputing Center Yoav Etsion Technion — Israel Institute of Technology Technion Computer Engineering Center A thesis submitted in fulfillment of the requirements for the degree of DOCTOR OF PHILOSOPHY /DOCTOR PER LA UPC International Doctorate Mention Departament d’Arquitectura de Computadors Universitat Politecnica` de Catalunya Thanks to all the lights along the journey, without whom it would be too dark to travel. Abstract Current Operating Systems (OSs) employ a process-centric isolation model. This is partly attributed to existing processors providing memory isolation across page tables. In this prevailing model, threads are bound to their creating process, and invoking functionality across processes requires costly OS kernel medi- ation and application developer involvement to synchronize and exchange information through Inter-Process Communication (IPC) channels. Therefore, using processes as an isolation unit imposes performance and programmability overheads. Nonetheless, processes also serve two other necessary roles. First, they act as resource containers; OSs track resources like memory and open files at the process granularity. Second, pro- cesses provide in-memory persistence; using a process ensures that its state is consistent across the coming and going of other processes that communicate with it. The architectural foundations used for building pro- cesses impose performance overheads in the excess of 10 and 1000 compared to a function call; that is, privilege level and page table switches, respectively. Even× more, part× of these overheads are not attributable to the hardware itself, but to the inherent overheads imposed by current OS designs. This thesis proposes a hardware and software co-design to eliminate the overheads of process isolation, while providing a path for gradual adoption for more aggressive optimizations. On the hardware side, this thesis proposes the CODOMs protection architecture. It provides memory and privilege protection across software components in a way that is at the same time very efficient and very flexible. This hardware substrate is then used to propose DomOS, which provides changes to the OS at the runtime and kernel layers to allow threads to efficiently and securely cross process boundaries using regular function calls. That is, a thread in one process is allowed to call into a function residing in another process without involving the OS in the critical communication path. This is achieved by mapping processes into a shared address space and eliminating IPC overheads through a combination of new hardware primitives and compile-time and run-time optimizations. IPC in DomOS is up to 24 times faster than Linux pipes, and up to 14 times faster than IPC in L4 Fiasco.OC. When applied to a multi-tier× web server, DomOS performs up to 2.18× better than an unmodified Linux system, and 1.32 on average. On all configurations, DomOS provides more× than 85% of the ideal system efficiency. × i Contents 1 Introduction 1 1.1 The Granularity, Performance and Programmability Dilemma.................1 1.2 Conjoining Isolation, Performance and Programmability...................2 1.3 Document Organization.....................................4 2 A Comparison of Related System Organizations and Isolation Primitives7 2.1 A Note on Nomenclature....................................7 2.2 Hardware Protection Mechanisms...............................7 2.2.1 Privilege Levels.....................................8 2.2.2 Address Spaces.....................................9 2.2.3 Machine Virtualization................................. 11 2.2.4 Capability Architectures................................ 12 2.3 Operating System Organization Models............................ 16 2.4 Software Isolation Mechanisms................................. 17 2.4.1 Processes........................................ 17 2.4.2 Address Spaces..................................... 17 2.4.3 Machine Virtualization................................. 18 2.4.4 Software Fault Isolation................................ 18 3 The Interplay Between Isolation and System Design 19 3.1 The Inadequacies of Process-Based Isolation.......................... 19 3.2 Mismatch Between Procedure Call and IPC Semantics.................... 23 3.3 False Concurrency....................................... 25 3.4 A Case for Configurable Isolation Policies........................... 26 4 Efficient and Composable Isolation Primitives 27 4.1 Security Model......................................... 28 4.1.1 Asymmetric Isolation Policies............................. 28 4.2 System Design Overview.................................... 29 4.2.1 CODOMs........................................ 30 4.2.2 Compiler Support.................................... 33 4.2.3 DomOS......................................... 33 4.3 Isolation Scenarios....................................... 34 iii 5 Hardware Support 37 5.1 Code-Centric Protection.................................... 37 5.1.1 Page Table Capabilities................................. 37 5.1.2 Domain Access Permissions.............................. 40 5.1.3 Access Protection List................................. 41 5.1.4 Example of Code-Centric Protection.......................... 41 5.2 Capability Protection...................................... 42 5.2.1 Example of Capability Protection........................... 44 5.2.2 Capability Confidentiality............................... 45 5.2.3 Capability Unforgeability and Integrity........................ 45 5.2.4 Domain Capability Stack................................ 45 5.2.5 Capability Revocation................................. 46 5.3 Enforcing Domain Entry and Exit Points............................ 47 5.4 Implementation of Access Protection Checks......................... 48 5.5 Efficient Execution in Out-of-Order Pipelines......................... 50 5.6 Revisiting Isolation Scenarios................................. 50 6 Compiler Support 53 6.1 Language Interface....................................... 53 6.2 Implementation......................................... 55 6.3 Example............................................. 56 7 Operating System Support 59 7.1 Processes and Threads..................................... 60 7.2 Low-Level Isolation Interface.................................. 61 7.2.1 Domain Management.................................. 61 7.2.2 Entry Point and Cross-Domain Proxy Management.................. 61 7.2.3 Capability Management................................ 64 7.3 Runtime Support........................................ 65 7.3.1 Program Loading.................................... 65 7.3.2 Entry Point Resolution................................. 66 7.4 Implementation Details..................................... 67 7.4.1 Unified Virtual Address Space............................. 67 7.4.2 Thread Management.................................. 68 7.4.3 Thread-Local Storage.................................. 68 7.4.4 Cross-Process Proxies................................. 69 7.4.5 Fault Notification.................................... 69 7.5 Capabilities as Opaque Handles to User-Defined Objects................... 69 7.6 Unified Resource Access Controls............................... 70 8 Evaluation 71 8.1 Hardware Mechanisms (CODOMs).............................. 71 8.1.1 Comparison of Different Cross-Domain Call Mechanisms.............. 71 8.1.2 Effect of Isolation Policies............................... 74 8.1.3 Area and Energy Overheads.............................. 75 8.1.4 Linux Kernel Module Isolation............................. 76 8.2 System Performance (DomOS)................................. 77 8.2.1 Comparison of Different Domain Communication Primitives............. 78 8.2.2 Thread Isolation.................................... 81 8.2.3 High-Performance Device Driver Isolation...................... 83 8.2.4 Multi-Tier Web Server Isolation............................ 84 9 Conclusions 87 A FlowTAS: Making Data-Centric Mandatory Access Control Practical 89 A.1 Introduction........................................... 89 A.2 Motivation............................................ 93 A.2.1 Example Setting and Privacy Expectations....................... 93 A.2.2 Data Leaks with Untrusted Applications........................ 93 A.2.3 Plugging Untrusted-Application Data Leaks...................... 93 A.2.4 Limitations of Prior MAC systems........................... 94 A.2.5 Problem Definition................................... 96 A.2.6 FlowTAS Overview................................... 97 A.3 Design of the FlowTAS System................................. 98 A.3.1 Trusted User Interface................................. 98 A.3.2 Web Proxy....................................... 98 A.3.3 Container Manager................................... 99 A.3.4 Template Processor................................... 99 A.3.5 Storage Declassifier.................................. 100 A.4 FlowTAS Implementation.................................... 100 A.4.1 Object Categories for Data-Centric MAC....................... 100 A.4.2 Web Proxy and Container Manager.......................... 102 A.4.3 Container and Application Instance Management................... 102 A.4.4 Template Processor................................... 104 A.5 Security Properties......................................