CheriOS: Designing an untrusted single-address-space capability operating system utilising capability hardware and a minimal hypervisor Lawrence G. Esswood University of Cambridge Computer Laboratory Churchill College July 2020 This thesis is submitted for the degree of Doctor of Philosophy Declaration This thesis is the result of my own work and includes nothing which is the outcome of work done in collaboration except as declared in the Preface and specified in the text. I further state that no substantial part of my thesis has already been submitted, or, is being concurrently submitted for any such degree, diploma or other qualification at the University of Cambridge or any other University or similar institution except as declared in the Preface and specified in the text. It does not exceed the prescribed word limit for the relevant Degree Committee. Lawrence G. Esswood July 2020 iii Abstract CheriOS: Designing an untrusted single-address-space capability operating system utilising capability hardware and a minimal hypervisor Lawrence G. Esswood This thesis presents the design, implementation, and evaluation of a novel capability operating system: CheriOS. The guiding motivation behind CheriOS is to provide strong security guarantees to programmers, even allowing them to continue to program in fast, but typically unsafe, languages such as C. Furthermore, it does this in the presence of an extremely strong adversarial model: in CheriOS, every compartment – and even the operating system itself – is considered actively malicious. Building on top of the architecturally enforced capabilities offered by the CHERI microprocessor, I show that only a few more capability types and enforcement checks are required to provide a strong compartmentalisation model that can facilitate mutual distrust. I implement these new primitives in software, in a new abstraction layer I dub the nanokernel. Among the new OS primitives I introduce are one for integrity and confidentiality called a Reservation (which allows allocating private memory without trusting the allocator), as well as another that can provide attestation about the state of the system, a Foundation (which provides a key to sign and protect capabilities based on a signature of the starting state of a program). I show that, using these new facilities, it is possible to design an operating system without having to trust the implementation is correct. CheriOS is fundamentally fail-safe; there are no assumptions about the behaviour of the system, apart from the CHERI processor and the nanokernel, to be broken. Using CHERI and the new nanokernel primitives, programmers can expect full isolation at scopes ranging from a whole program to a single function, and not just with respect to other programs but the system itself. Programs compiled for and run on CheriOS offer full memory safety, both spatial and temporal, enforced control flow integrity between compartments and protection against common vulnerabilities such as buffer overflows, code injection and Return-Oriented- Programming attacks. I achieve this by designing a new CHERI-based ABI (Application Binary Interface) which includes a novel stack structure that offers temporal safety. I evaluate how practical the new designs are by prototyping them and offering a detailed performance evaluation. I also contrast with existing offerings from both industry and academia. CHERI capabilities can be used to restrict access to system resources, such as memory, with the required dynamic checks being performed by hardware in parallel with normal operation. Using the accelerating features of CHERI, I show that many of the security guarantees that CheriOS offers can come at little to no cost. I present a novel and secure IO/IPC layer that allows secure marshalling of multiple data streams through mutually distrusting compartments, with fine-grained authenticated access control for end-points, and without either copying or encryption. For example, CheriOS can restrict its TCP stack from having access to packet contents, or restrict an open socket to ensure data sent on it to arrives at an endpoint signed as a TLS implementation. Even with added security requirements, CheriOS can perform well on real workloads. I showcase this by running a state-of-the-art webserver, NGINX, atop both CheriOS and FreeBSD and show improvements in performance ranging from 3x to 6x when running on a small-scale low-power FPGA implementation of CHERI-MIPS. v Acknowledgements I would like to fill this space by thanking all the people whose talents and support made this possible. First, my supervisor, Robert Watson, for the opportunity and freedom to pursue such an interesting project, and work with CHERI at all. Without your ideas and guidance, this work would look nothing like it does. You have been very patient. Thanks to both of my examiners, Timothy Roscoe and Tim Harris, for all their help fine-tuning this thesis. I am thankful to Arm for partially funding my time at the lab. I am extremely thankful to Nathaniel Filardo, who has read and re-read this document. You have been an invaluable source of feedback, ideas, and interesting anecdotes. Many thanks to David Chisnall, who taught me the basics of working with LLVM, and who I constantly interrupted seeking answers. I have learnt many things from you during my time at the lab, not least of which the importance of teatime. Thank you to Peter Rugg, who scoured this document with me for errors. The fact we rarely find the same ones leaves me with a great deal of concern. We truly do stand on the shoulders of giants, and none of this would have been possible without all the people who work on the larger CHERI project. You are too numerous to mention each by name, but I would like to give specific thanks to Alexander Richardson and Jonathan Woodruff, who always helped me with my compiler- and hardware-related difficulties. I give my deepest thanks to all my friends (you know who you are) during these PhD years, new and old. I would not have got through it without you. Last, but not least, to my parents, Aim´eeand Paul. Thank you for your years of compassionate and unwavering support. Although you may not understand it all, I hope this small success makes you proud. vii Contents List of Figures xv List of Tables xvii 1 Introduction 19 1.1 Security in contemporary computing . 19 1.2 Breaking the hierarchy . 20 1.3 Contributions . 22 1.4 Thesis overview . 24 2 Background 25 2.1 Software protection . 25 Safe languages . 25 Formal verification . 26 Compiler techniques . 27 2.2 Hardware protection . 28 Segmentation . 29 Paged virtual memory . 29 Rings . 30 Intel SGX . 30 Arm TrustZone . 31 Arm MTE . 32 Intel MPX . 32 2.3 Access control . 32 Ambient authority . 32 Capabilities . 33 Capability machines . 34 Capability operating systems . 35 2.4 CHERI . 36 CHERI capabilities . 36 Architectural capabilities . 38 Sealing . 38 Bearer tokens . 39 Object capabilities & CCall . 39 ix 2.5 LLVM . 40 3 Designing CheriOS 43 3.1 System model and security guarantees . 45 System model . 45 Mapping into languages . 46 3.2 The nanokernel . 47 Primitives . 48 Reservations . 48 Foundations . 49 CPU contexts . 50 Physical & virtual pages . 50 3.3 The microkernel . 50 3.4 The system . 52 3.5 Userspace . 52 4 Memory Safety 55 4.1 Memory allocation . 55 Static storage . 58 Heap memory . 59 Stack memory . 60 Slinky stacks . 60 CHERI-aware escape analysis . 66 4.2 Revocation . 66 MMU-based revocation . 67 Sweep-based revocation . 67 4.3 Cross-process capability exchange . 70 The memory manager . 71 Malloc . 73 Guarded instructions . 73 User exceptions . 74 Comparison of access techniques . 74 4.4 Evaluation . 75 Revocation . 75 Cost of a sweep . 75 Concurrent sweeping . 77 Slinky stacks . 80 Static analysis . 80 Dynamic cost . 82 x 4.5 Related work . 83 Protecting stacks . 83 Local/Global capabilities . 84 Linear capabilities . 85 Spatial safety on the stack . 86 Revocation . ..