Formal Reasoning About Capability Machines
Total Page:16
File Type:pdf, Size:1020Kb
Formal Reasoning about Capability Machines Lau Skorstengaard PhD Dissertation Department of Computer Science Aarhus University Denmark Formal Reasoning about Capability Machines A Dissertation Presented to the Faculty of Science and Technology of Aarhus University in Partial Fulfillment of the Requirements for the PhD Degree by Lau Skorstengaard Friday 30th August, 2019 Abstract Today, computer security is often based on mitigations that make exploita- tion cumbersome or unlikely. In other words, mitigation techniques provide no security guarantee, and time and time again mitigations have been circum- vented. To stop the ongoing exploit-mitigation arms-race, we need security enforcement mechanisms that can be proven impossible to circumvent. Un- fortunately, mainstream computers have the necessary security primitives to support such enforcement mechanisms. Capability machines are computers that provide additional security primitives by replacing pointers with capa- bilities. Capabilities are unforgeable tokens of authority that must be used to access memory and other system resources. In this dissertation, we present foundational research in the field of se- cure compilation that targets capability machines. In particular, we present novel and provably secure enforcement mechanisms for control-flow and encapsulation properties of high-level programming languages – properties that all programmers rely on. The security proof demonstrates for the first time how state-of-the-art reasoning techniques for high-level programming languages can be used to reason about capability safety and secure compi- lation at the very lowest level. We also contribute new proof techniques for secure compilation proofs. In Chapter1, we introduce the concept of capabilities, show various in- stances of capabilities throughout computer science, and present and moti- vate the contents of this dissertation. In Chapter2, we present a formalisation of a capability machine with lo- cal capabilities and a calling convention that provably enforces well-bracketed control-flow (WBCF) and local-state encapsulation (LSE). We also present a notion of capability safety in terms of a logical relation. Using the logical relation, we prove correctness of program examples that interact with un- known code and rely on non-trivial control-flow for correctness. In Chapter3, we present a formalisation of a capability machine with lin- ear capabilities along with StkTokens: another calling convention provably enforcing WBCF and LSE. Via a novel proof technique called fully-abstract overlay semantics, we prove that StkTokens enforces these properties. The overlay semantics simplifies the full-abstraction proof significantly by retain- ing the language syntax and adding the desired property semantically. i Resume´ De fleste programmer er skrevet i højniveausprogrammeringssprog med ab- straktioner, der gør, at programmører ikke behøver at bekymre sig om hardwa- redetaljer. Computere kan dog ikke køre sadanne˚ programmer, hvilket løses ved at oversætte dem til et lavniveausprogrammeringssprog, som compute- ren forstar.˚ Umiddelbart lyder dette uskyldigt, men hvad sker der egentlig med højniveausabstraktionerne efter oversættelsen? Er de bibeholdt, og hvis ikke betyder det noget for sikkerheden? Svaret er, groft sagt, at abstraktioner- ne ikke er bibeholdt, og det giver anledning til sikkerhedsbrister. Eksempel- vis er buffer overflow angreb og return oriented-programming angreb baseret pa,˚ at forventede højniveauabstraktioner ikke bibeholdes efter oversættelse. Sikker oversættelse (i forskningsverdenen kendt som secure compilation) omhandler oversættelser, der bevarer sikkerhed efter oversættelse. Der er mange former for sikkerhed og ligesa˚ mange former for sikker oversættel- se. Full abstraction er en form for sikker oversættelse, der beviseligt bevarer abstraktioner - selv hvis programmet interagerer med ukendte ondsindede programmer. Dette er en meget stærk egenskab, som kun er opfyldt, hvis et oversat program handhæve˚ abstraktionerne. Der eksisterer i dag ikke nogen rigtig oversætter, som er fully abstract1. Et problem, i forhold til at lave en sikker oversættelse, er, at vores compu- tere ikke har tilstrækkelige sikkerhedsprimitiver til at handhæve˚ højniveaus- abstraktionerne. Hvis vi ønsker at have sikre computere, sa˚ bliver vi nødt til at ga˚ væk fra den traditionelle computer over til en computer med flere sikkerhedsprimitiver, men hvilken? Et bud kunne være capability machines, hvilket er computere, der udskifter pointers med capabilities. En capability er essentielt set en polet, der ikke kan blive forfalsket og som giver adgang til noget, eksempelvis at læse fra en del af hukommelsen. I denne afhandling tager vi et stort skridt mod at lave en rigtig sikker oversætter ved at vise, at capability machines kan udgøre grundlaget for sik- ker oversættelse. Vi viser, hvordan en capability machines ekstra sikkerheds- primitiver kan handhæve˚ en handfuld˚ højniveausabstraktioner. Vi giver til- med matematiske beviser for, at abstraktionerne er handhævet.˚ 1Der eksisterer oversættelser mellem legetøjsprogrammeringsprog, der beviseligt opfyl- der egenskaben. iii Acknowledgements First and foremost, I want to thank my supervisor Lars Birkedal for his guid- ance throughout my PhD studies. I am especially grateful for Lars’ words of encouragement when I was needlessly hard on my own work. I also want to extend a big thanks to Dominique Devriese for a long and fruitful collabora- tion as well as for hosting me at KU Leuven on a number of occasions. Al- though Dominique has not officially supervised me, he has provided plenty of advice and guidance. I also want to extend my gratitude to the people that helped and encour- aged me during my early studies. In particular, Jens Studsgaard deserves recognition for inspiring me to study computer science, and Olivier Danvy deserves thanks for guiding me during my bachelor and master studies and encouraging me to apply for the PhD studies. Finally, a big thanks goes to my friends and family who have supported me during my studies. I thank you all for taking an interest in my PhD project by asking questions about it whether or not you believed the answer would make sense to you. No PhD student is an island; in my case that is all thanks to you. Lau Skorstengaard, Aarhus, Friday 30th August, 2019. v Contents Abstracti Resume´ iii Acknowledgementsv Contents vii I Overview1 1 Introduction3 1.1 Capabilities.............................. 4 1.2 Capabilities in This Dissertation.................. 19 1.3 Dissertation Outline......................... 23 1.4 Future Work............................. 26 II Publications 29 2 Reasoning About a Machine with Local Capabilities 31 2.1 Introduction............................. 32 2.2 A Capability Machine with Local Capabilities.......... 35 2.3 Stack and Return Pointer Management Using Local Capabilities 39 2.4 Logical Relation........................... 46 2.5 Malloc................................. 60 2.6 Reusable macro instructions.................... 63 2.7 Reasoning about programs on a capability machine . 66 2.8 Examples............................... 71 2.9 Discussion .............................. 78 2.10 Related Work............................. 85 2.A Appendix............................... 88 vii viii Contents 3 StkTokens: Enforcing Well-Bracketed Control Flow and Stack Encapsulation Using . 101 3.1 Introduction.............................101 3.2 A Capability Machine with Sealing and Linear Capabilities . 105 3.3 Linear Stack and Return Capabilities . 113 3.4 Formulating Security with a Fully Abstract Overlay Semantics 119 3.5 Proving full abstraction.......................138 3.6 Discussion ..............................183 3.7 Related Work.............................187 3.A Appendix...............................190 Bibliography 195 Part I Overview 1 Chapter 1 Introduction Capability machines are low-level machines with built-in fine-grained ac- cess control. Capabilities are unforgeable tokens of authority that replace pointers on a capability machine; every operation on a capability machine is subject to a dynamic capability authority check. Capability machines have been around for decades, but recently they have seen an increased interest with the increased focus on security on computers. Security issues are often subtle, so to ensure security the enforcement must be 100% watertight. The only way to be certain that security enforcement cannot be circumvented is to formally prove it. This dissertation shows that security properties can be enforced on capability machines in a provably secure way. This dissertation is comprised of two papers that formalises two capabil- ity machines and show how they can provably enforce security properties of high-level languages. In this introduction, we provide intuition about the general notion of capabilities. The introduction is somewhat informal in its presentation whereas the papers present capability machines formally. The remainder of the introduction consists of three sections: an introduc- tion to capabilities, two ways to motivate the dissertation, and an outline of the dissertation. Capabilities exist as a concept on all abstraction layers in a computer. The introduction to capabilities, Section 1.1, contains a general capability characterisation that fits the concept of capability found on every layer. Section 1.1 also presents capabilities on different abstraction layers. Specifically, the section presents capability machines, operating system ca- pabilities,