Improving System Security Through TCB Reduction
Total Page:16
File Type:pdf, Size:1020Kb
Improving System Security Through TCB Reduction Bernhard Kauer March 31, 2015 Dissertation vorgelegt an der Technischen Universität Dresden Fakultät Informatik zur Erlangung des akademischen Grades Doktoringenieur (Dr.-Ing.) Erstgutachter Prof. Dr. rer. nat. Hermann Härtig Technische Universität Dresden Zweitgutachter Prof. Dr. Paulo Esteves Veríssimo University of Luxembourg Verteidigung 15.12.2014 Abstract The OS (operating system) is the primary target of todays attacks. A single exploitable defect can be sufficient to break the security of the system and give fully control over all the software on the machine. Because current operating systems are too large to be defect free, the best approach to improve the system security is to reduce their code to more manageable levels. This work shows how the security-critical part of theOS, the so called TCB (Trusted Computing Base), can be reduced from millions to less than hundred thousand lines of code to achieve these security goals. Shrinking the software stack by more than an order of magnitude is an open challenge since no single technique can currently achieve this. We therefore followed a holistic approach and improved the design as well as implementation of several system layers starting with a newOS called NOVA. NOVA provides a small TCB for both newly written applications but also for legacy code running inside virtual machines. Virtualization is thereby the key technique to ensure that compatibility requirements will not increase the minimal TCB of our system. The main contribution of this work is to show how the virtual machine monitor for NOVA was implemented with significantly less lines of code without affecting the per- formance of its guest OS. To reduce the overall TCB of our system, other parts had to be improved as well. Additional contributions are the simplification of theOS debugging interface, the reduction of the boot stack and a new programming language called B1 that can be more easily compiled. 3 Acknowledgements First, I would like to thank my supervisor Hermann Härtig for his support as well as the freedom in choosing my research directions. Thanks are also due to the members of the OS group for uncountable discussions during my time at TU Dresden. I am particular indebted to Udo Steinberg, which I had the pleasure to share a room with. His NOVA Hypervisor proved to be an excellent base for my virtualization research. I am also grateful to Alexander Böttcher and Julian Stecklina for extending the software I have developed. During my thesis I had the privilege to get to know industrial and academic research in the US and Portugal. Thanks to Marcus Peinado and Sebastian Schoenberg for the summers at Microsoft and Intel as well as to Paulo Veríssimo and the navigators for the rewarding time and excellent coffee in Lisbon. I am grateful to all who have read and commented on drafts of this thesis. Monika Scholz, Anne Kauer, and Norman Feske deserve particular thanks for spending hours of their live to improve what I had written. The errors still remaining in this work are mine alone. Finally, I want to thank my family for always believing in me and supporting this endeavor for so a long time. Without Anne and my parents I would not have made it. Thank you all! 5 6 Contents List of Figures 11 1 Introduction 13 1.1 Motivation................................... 13 1.2 Approach.................................... 14 1.2.1 Minimizing the OS........................... 14 1.2.2 Ensuring Compatibility........................ 15 1.2.3 Additional Layers............................ 15 1.3 Contributions.................................. 16 2 A Smaller Virtualization Stack 19 2.1 Background and Related Work........................ 20 2.1.1 A short History of Virtualization................... 20 2.1.2 Virtualization Today.......................... 23 2.1.3 Securing the Virtualization Layer................... 26 2.1.4 Advanced Use Cases for Virtualization................ 28 2.2 Design...................................... 29 2.2.1 NOVA Architecture.......................... 29 2.2.2 The NOVA Microhypervisor...................... 30 2.2.3 NUL: The NOVA User-Level Environment.............. 32 2.2.4 Vancouver: A Small VMM for NOVA................ 38 2.3 Device Models................................. 42 2.3.1 Approaches to Reduce the TCB................... 42 2.3.2 Reusing an Existing VMM...................... 43 2.3.3 Implementation............................. 46 2.3.4 Implementation Details........................ 51 2.3.5 Evaluation: Code Size and Density.................. 54 2.3.6 Future Work.............................. 56 2.4 Instruction Emulator.............................. 57 2.4.1 Background............................... 57 2.4.2 First Generation: Hand-written Tables................ 58 2.4.3 Automating the Knowledge Extraction................ 58 2.4.4 Current Generation: Reuse the Assembler.............. 60 2.4.5 Summary and Future Work...................... 65 2.5 Virtual BIOS.................................. 67 2.5.1 Design.................................. 67 2.5.2 Implementation............................. 69 2.5.3 Enabling Emulators.......................... 70 7 CONTENTS 2.5.4 Summary................................ 71 2.6 Performance Evaluation............................ 71 2.6.1 Setting up the Benchmark....................... 72 2.6.2 Results................................. 74 2.6.3 Measurement Error........................... 78 2.6.4 Detailing the Overhead........................ 80 2.6.5 Performance Outlook......................... 84 2.7 Conclusions................................... 85 2.7.1 Size Outlook.............................. 86 3 TCB-aware Debugging 87 3.1 Requirements.................................. 87 3.1.1 Virtual Machines and Emulators................... 87 3.1.2 On-target versus Remote Debugger.................. 88 3.1.3 Tracing and Interactive Debugging.................. 89 3.1.4 Special Requirements for NOVA................... 90 3.2 The Vertical Debugger VDB.......................... 90 3.2.1 Reusing GDB?............................. 90 3.2.2 Design.................................. 91 3.2.3 Implementation............................. 93 3.3 Debugging without a Target Driver...................... 95 3.3.1 Choosing the Hardware........................ 96 3.3.2 Remote Access Without Runtime Code on the Target....... 97 3.3.3 Surviving a Bus-Reset......................... 98 3.3.4 Implementation............................. 100 3.3.5 Firewire Performance......................... 100 3.3.6 Related and Future Work....................... 103 3.4 Minimizing the Debug Stub.......................... 103 3.4.1 Design of a Halt and Resume Stub.................. 104 3.4.2 Debugging a NOVA system...................... 106 3.5 Summary.................................... 108 4 Simplifying the Compiler 109 4.1 Design...................................... 109 4.1.1 The Syntax............................... 110 4.1.2 Variables and Constants........................ 111 4.1.3 Operators, Functions and Control Structures............ 113 4.1.4 Discussion................................ 115 4.2 Implementation................................. 117 4.2.1 Implementation Language....................... 117 4.2.2 Let Python Parse............................ 118 4.2.3 Compiling to an Intermediate Representation............ 119 4.2.4 Optimizing the Compiler Output................... 119 4.2.5 Generating Machine Code....................... 120 4.2.6 Linking the Binary........................... 121 4.2.7 Implementing the Standard Library................. 121 4.2.8 Summary................................ 122 4.3 Evaluation.................................... 123 4.3.1 The Influence of the Optimizer.................... 123 4.3.2 Exception Handling.......................... 125 8 CONTENTS 4.3.3 System Calls: dd ............................ 127 4.3.4 Simple Calculation: wc ......................... 127 4.3.5 Complex Calculation: gunzip ..................... 130 4.3.6 Number Crunching: sha1sum ..................... 131 4.3.7 Summary................................ 133 4.4 Conclusions and Future Work......................... 133 5 Shrinking the Boot Stack 135 5.1 Background: Booting an OS on a PC..................... 135 5.1.1 The Firmware.............................. 136 5.1.2 The Bootloader............................. 136 5.1.3 The OS................................. 137 5.2 The Bootloader................................. 137 5.2.1 Features................................. 138 5.2.2 Design.................................. 138 5.2.3 Implementation............................. 140 5.2.4 Evaluation............................... 145 5.2.5 Summary................................ 146 5.3 Trusted Computing............................... 147 5.3.1 Secure Boot............................... 147 5.3.2 Trusted Computing with a Static Root of Trust for Measurement. 148 5.3.3 Trusted Computing with a Dynamic RTM.............. 149 5.3.4 Security Challenges.......................... 150 5.3.5 Summary................................ 152 5.4 ATARE: Parsing ACPI Tables with Regular Expressions.......... 152 5.4.1 Background............................... 153 5.4.2 Pattern Matching on AML...................... 154 5.4.3 The Search Algorithm......................... 156 5.4.4 Evaluation............................... 156 5.4.5 Summary................................ 159 5.5 Conclusions................................... 159 6 Conclusions 161 6.1 Techniques..................................