DISS. ETH NO. 26619 On Memory Addressing A thesis submitted to attain the degree of Doctor of Sciences of ETH Zurich (Dr. sc. ETH Zurich) presented by Reto Achermann Master of Science ETH in Computer Science, ETH Zurich born on 04. June 1989 citizen of Ennetbürgen, Switzerland accepted on the recommendation of Prof. Dr. Timothy Roscoe (ETH Zurich), examiner Prof. Dr. David Basin (ETH Zurich), co-examiner Prof. Dr. Gernot Heiser (UNSW Sydney), co-examiner Dr. David Cock (ETH Zurich), co-examiner 2020 On Memory Addressing. Copyright © 2020, Reto Achermann. Permission to print for personal and academic use, as well as permission for electronic reproduction and dissemination in unaltered and complete form are granted. All other rights reserved. DOI: 10.3929/ethz-b-000400029 Abstract Operating systems manage and configure a machine’s physical resources such as memory and translation hardware. This task is mission critical: the operating system must always correctly configure memory address translations and unambiguously name the physical resources of a system. However, operating systems today use abstractions and assumptions which unfaithfully represent the actual topology of the hardware they are man- aging. This mismatch leads to bugs and security vulnerabilities in system software. This is a problem. This dissertation presents a new abstraction model to faithfully represent the memory subsystem of a hardware platform as seen by software. The core abstraction of the new model is the address space, which defines the context for address decoding. An address space either translates addresses or terminates address resolution within its context. The Decoding Net formally specifies the semantics of address decoding behavior of address spaces in the Isabelle/HOL theorem prover. This provides a sound basis for reasoning about the current hardware configuration of a platform. Address spaces are inherently dynamic in two ways: i) new devices are discovered, powered on or off, or hot-plugged introducing new address spaces in the system, and ii) a memory allocation request requires an update of the translation configuration of an address space. Changing the configuration of an address space is a privileged operation and requires a certain authority. This is expressed as an extension to the Decoding Net with a layer adding a notion of configurability and fine-grained authority following the principle of least-privilege. Guided by an executable specification of the dynamic Decoding Net model, the implementation in Barrelfish/MAS is driven following the principle of least-privilege. The resulting implementation demonstrates that it is possible to implement the detailed address space model and least-privilege memory management in an operating system efficiently and with little overhead and matching performance to the Linux operating system. iii Zusammenfassung Betriebssysteme verwalten und konfigurieren die physikalischen Ressour- cen wie Hauptspeicher und Adressierungsübersetzungshardware einer Re- chenmaschine. Dieser Arbeitsschritt ist missionskritisch: das Betriebs- system muss die Hardware stets korrekt konfigurieren wie auch die phy- sikalischen Ressourcen des Systems eindeutig benennen können. Die Betriebssysteme von heute benutzen jedoch Abstraktionen und Annahmen welche gerade die eigentliche Topologie der verwalteten Hardware inak- kurat repräsentieren. Diese Diskrepanz führt zu verschiedensten Defekten und Sicherheitslücken in System Software. Dies ist ein Problem. Diese Dissertation präsentiert einen neues Abstraktionsmodell welches die Hardwarekonfiguration einer Rechenmaschine, wie sie von der Software gesehen wird, akkurat repräsentiert. Die Zentrale Abstraktion dieses neuen Modells ist der Adressraum, welcher einen Kontext für Adressdekodierung definiert. Ein Adressraum übersetzt oder schliesst die Adressauflösung für eine Adresse innerhalb seines Kontextes ab. Die Semantik der Adressauf- lösung und der Adressräume ist dann formalisiert im “Decoding Net”, eine Spezifikation der Adressraumabstraktion in Isabelle/HOL. Dies bildet eine wohldefinierte Grundlage, um über die gegenwärtige Hardwarekonfigura- tion einer Plattform zu argumentieren. Die Adressräume sind von Natur aus dynamisch in zwei Arten: i) das Auffinden, Anschliessen oder Entfernen von neuen Hardwarekomponenten verändert die Anzahl der Adressräume im System, und ii) die Konfiguration dieser Adressräume kann verändert werden. Diese privilegierte Aktion benötigt die entsprechenden Befugnisse. Das “Decoding Net” Modell wird erweitert mit einer Auffassung von Konfigurierbarkeit und detailgenauen Befugnissen im Sinne von Prinzip des minimalen Rechts. Gelenkt von einer ausführbaren Spezifikation des dynamischen “Decoding Net” Modells wird die Implementierung in Barrelfish/MAS durchgeführt unter dem Prinzip des minimalen Rechts. Die resultierende Implementie- rung zeigt, dass es möglich ist das detaillierte Adressraummodell und eine Speicherverwaltung mit minimalen Rechten effizient in einem Betriebssys- tem zu realisieren. v Acknowledgments I am exceptionally thankful to my many amazing and wonderful friends and colleagues, who contributed, in some way or another, to the research presented in this thesis. To my adviser, Prof. Timothy “Mothy” Roscoe. Thank you for giving me the opportunity to do my masters and doctoral studies in the Systems Group. I am truly grateful for your openness, support, and the feedback and advice I have received from you during my time at ETH Zurich. It was a great pleasure to work with you. To Dr. David Cock. I am exceptionally grateful for the opportunity to work with you. Thank you for your inputs to the formal modeling, constructive feedback, and Isabelle/HOL expertise, which enabled and propelled many aspects of the work presented in this thesis. To Prof. Gernot Heiser. Thank you for agreeing to be on my committee, and your outstanding dedication in providing valuable feedback and insightful comments, which greatly improved my dissertation. To Prof. David Basin. I appreciate that you have agreed to be on my committee, and your valuable and assuring feedback on the dissertation. To my outstanding and truly exceptional collaborators at ETH Zurich, Hewlett-Packard Labs, VMware Research and elsewhere. I would like to express my greatest gratitude for giving me the opportunity to work and collaborate with you. Your dedication, comments and feedback during many meetings and interactions, and contributions to papers, reports and projects either directly or indirectly contributed to my dissertation: David Cock, Lukas Humbel, Roni Häcki, Simon Gerber, Jayneel Gandhi, Dejan Milojicic, Kornilios Kourtis, Stefan Kästle, Michael Giardino, Nora Hossle, Daniel Schwyn, Gerd Zellweger, Moritz Hoffmann, Ashish Panwar and Abhishek Bhattacharjee. To my friends and colleagues at ETH Zurich. Roni, Pravin, Renato, Simon, Stefan, Frances, Michael, Claude, Moritz, Gerd, Daniel, David, Nora, vii Anastasiia, Andrea, Lukas, Melissa, Michael, Monica, Raphael, Daniel and Barbara. Thank you for making my time in the Systems Group such a great experience. A special thank you to the amazing admins of the Systems Group: Simonetta, Nadia, and Jena. Finally, I would like to thank my parents, Irene and Beppi, for their unconditional support. Zurich, February 2020. viii Contents 1 Introduction1 1.1 Motivation.........................2 1.2 Problem Statement....................5 1.3 Structure of the Dissertation...............7 1.4 Related publications....................9 2 Problem Statement 11 2.1 Motivation......................... 11 2.2 Survey of Memory Address Translation......... 12 2.2.1 Address Spaces and Address Definitions..... 13 2.2.2 Translation Schemes in Real Hardware..... 18 2.2.3 Proposed Translation Schemes.......... 35 2.2.4 Summary..................... 43 2.3 Current Physical Address Space Model.......... 43 2.4 Problems with the Current Model............ 45 2.4.1 Observations................... 45 2.4.2 Resulting Problems in Operating Systems.... 56 2.5 Implications for Operating System Architectures..... 58 2.5.1 Operating System Design Implications...... 59 2.5.2 Virtualization as a Solution?........... 60 2.5.3 Operating System Design Challenges...... 61 2.6 Conclusion........................ 63 3 Related Work 65 3.1 System Topology Descriptions.............. 66 3.1.1 Self-Describing Hardware and Firmware.... 66 3.1.2 Domain Specific Languages........... 67 ix Contents 3.1.3 System Topologies Summary........... 69 3.2 Behavioral System Descriptions............. 69 3.2.1 Micro Architecture Specifications........ 69 3.2.2 Processor Models................. 71 3.2.3 Behavioral Models Summary.......... 72 3.3 Memory Management in Operating Systems....... 72 3.3.1 Monolithic Operating Systems.......... 73 3.3.2 Single-Address-Space Operating Systems.... 74 3.3.3 Verified Operating Systems........... 75 3.3.4 Microkernel Operating Systems......... 77 3.3.5 Hypervisors and Simulators........... 80 3.3.6 Early Capability-Based Operating Systems... 80 3.3.7 Other Operating Systems............. 82 3.3.8 Operating Systems Summary........... 83 3.4 Runtime Systems and Programming Languages..... 83 3.4.1 Memory Topology Models............ 84 3.4.2 Cache Topology Models............. 86 3.4.3 Co-Processor Offloading............. 87 3.4.4 Programming Languages............. 88 3.4.5 Runtimes and Programming Languages Summary 89 3.5 Summary......................... 89 4 A Formal Model for Memory Addressing 91 4.1 Motivation......................... 92 4.2 An Accurate Model for Memory Address Translation.