Formal models and verification of memory management in a hypervisor Pauline Bolignano To cite this version: Pauline Bolignano. Formal models and verification of memory management in a hypervisor. Cryp- tography and Security [cs.CR]. Université Rennes 1, 2017. English. NNT : 2017REN1S026. tel- 01637937 HAL Id: tel-01637937 https://tel.archives-ouvertes.fr/tel-01637937 Submitted on 18 Nov 2017 HAL is a multi-disciplinary open access L’archive ouverte pluridisciplinaire HAL, est archive for the deposit and dissemination of sci- destinée au dépôt et à la diffusion de documents entific research documents, whether they are pub- scientifiques de niveau recherche, publiés ou non, lished or not. The documents may come from émanant des établissements d’enseignement et de teaching and research institutions in France or recherche français ou étrangers, des laboratoires abroad, or from public or private research centers. publics ou privés. ANNÉE 2017 THÈSE / UNIVERSITÉ DE RENNES 1 sous le sceau de l’Université Bretagne Loire pour le grade de DOCTEUR DE L’UNIVERSITÉ DE RENNES 1 Mention : Informatique Ecole doctorale Matisse présentée par Pauline Bolignano préparée à l’unité de recherche 6074 - IRISA Institut de Recherche en Informatique et Systèmes Aléatoires Thèse soutenue à Rennes le 24 mai 2017 Formal Models and devant le jury composé de : Mads DAM Verification of Professeur, KTH Royal Technical University/Rapporteur Marie-Laure POTET Memory Professeure, Ensimag/Rapporteuse Delphine DEMANGE Maître de Conférence, Université de Rennes 1 / Examina- Management trice Mario SUDHOLT in a Hypervisor Professeur, Institut Mines Télécom/Examinateur Thomas JENSEN Directeur de Recherche, Inria/Directeur de thèse Vincent SILES Ingénieur Docteur, Prove & Run/Co-directeur de thèse i Contents Remerciements vii Résumé en Francais ix Introduction 1 1 Context 3 1.1 Hypervisors .................................... 4 1.1.1 Operating System Kernels ........................ 4 1.1.2 Different Types of Hypervisors ..................... 4 1.1.3 Memory Virtualization .......................... 6 Memory Management in an OS ..................... 6 Memory Management in a Hypervisor ................ 7 1.2 Security Properties ................................ 9 1.2.1 Non-Interference ............................. 9 1.2.2 Variants of Non-Interference ...................... 9 1.3 Formal Methods .................................. 10 1.3.1 Tools for Theorem Proving ....................... 10 1.3.2 Methods for Theorem Proving ..................... 11 Annotations ................................ 11 Modeling and Interactive Proving ................... 11 1.3.3 Proof by Abstraction ........................... 12 1.3.4 Prove & Run Tools ............................ 12 1.4 Certification .................................... 14 1.5 Key Points ..................................... 15 2 State of the Art 17 2.1 Early System Verification Projects ........................ 18 2.2 Recent OS Verification Projects ......................... 19 2.2.1 SeL4 .................................... 19 2.3 Hypervisor Verification .............................. 20 2.3.1 Prosper ................................... 21 2.3.2 Verisoft XT ................................. 21 2.4 The Methodology of Proof by Abstraction ................... 22 2.4.1 Commutation ............................... 23 2.4.2 Transferring Properties to the Concrete Model . 23 2.4.3 Comparison of our Abstraction to State of the Art . 25 2.5 Contributions ................................... 26 2.6 Overview of the Chapters ............................ 27 2.7 Key Points ..................................... 29 ii Contents 3 Concrete Model of the Hypervisor 31 3.1 Basic Types and Notations ............................ 32 3.2 Modeling of the Page Tables ........................... 32 3.2.1 Decomposition of the Function pt ................... 33 3.2.2 Virtual Page Table Walk ......................... 36 3.2.3 Set of Addresses Mapped by a Page Table . 37 3.3 Static Structures .................................. 37 3.3.1 Memory Layout .............................. 37 Static Permissions ............................ 38 Hypervisor Space ............................. 38 3.3.2 Host Page Table .............................. 38 3.4 Low-Level State of the Hypervisor ....................... 38 3.4.1 Hardware State .............................. 39 Memory .................................. 39 Modes ................................... 40 Application Program Status Register . 41 Core Registers ............................... 41 Coprocessor 15 .............................. 41 Generic Interrupt Controller ...................... 42 Caches ................................... 42 3.4.2 Hypervisor State ............................. 42 Virtual Mode ............................... 44 Virtual Core and Banked Registers ................... 44 MMU Registers .............................. 45 Generic Interruption Controller Registers . 45 3.5 Low-Level Transitions .............................. 45 3.5.1 Guest Transition ............................. 46 3.5.2 Save State Transition ........................... 48 3.5.3 Hypervisor Transitions .......................... 49 Memory Management Transitions ................... 50 Schedule Transition ............................ 53 GIC Transitions .............................. 53 Modify Registers Transitions ...................... 54 3.5.4 Restore Transition ............................ 55 3.6 Key Points ..................................... 56 4 Invariant Properties of the System 57 4.1 Invariants on Page Tables ............................ 59 4.1.1 Page Tables Well-formedness ...................... 59 4.1.2 Translation of Hypervisor Virtual Space . 61 4.2 Invariants Specific to some Transitions ..................... 64 4.2.1 Guest Transition ............................. 65 Exception Handlers ........................... 65 4.2.2 Map a Page ................................ 66 4.2.3 Unmap a Page ............................... 68 4.2.4 Unmap all ................................. 69 4.2.5 Well-formed Registers .......................... 71 4.2.6 Interdependencies ............................ 72 4.3 Specifications of the Effects of some Transitions . 75 Contents iii 4.3.1 Map .................................... 75 4.3.2 Unmap................................... 78 4.3.3 Unmap All................................. 79 4.3.4 Guest Transition ............................. 80 4.4 Conclusion..................................... 81 4.5 Key Points ..................................... 82 5 Abstract Model of the Hypervisor 83 5.1 Abstract State ................................... 85 5.1.1 Memory Cells ............................... 85 5.1.2 Guest State ................................ 86 5.1.3 Whole State ................................ 87 5.2 Abstraction ..................................... 87 5.2.1 Registers .................................. 87 5.2.2 Segments ................................. 88 Private Segment .............................. 89 Shared Segments ............................. 89 5.2.3 Abstraction Function ........................... 90 5.3 Abstract Transitions ................................ 90 5.3.1 Oracle ................................... 91 5.3.2 Guest Transition ............................. 94 Guest Run ................................. 94 Guest Synchronize ............................ 97 Whole Transition ............................. 97 5.3.3 Hypervisor Transition .......................... 98 Memory Management .......................... 98 Schedule .................................. 98 Nop ..................................... 98 Registers Modification .......................... 99 5.3.4 Restore Transition ............................ 99 5.3.5 Abstract Transition ............................ 99 5.4 Security properties ................................ 99 5.4.1 Integrity .................................. 100 5.4.2 Confidentiality .............................. 100 5.5 Refinement ..................................... 102 5.5.1 Guest Transition ............................. 102 5.5.2 Memory Transitions . 104 Map .................................... 104 Unmap ................................... 109 Unmap All ................................. 111 5.6 Impact of Optimizations on the Abstract Model . 112 5.6.1 Several SPTs per Guest . 112 5.6.2 Allocator .................................. 112 5.6.3 Dynamic Configuration . 113 5.7 Key Points ..................................... 113 iv Contents 6 Benchmarks and Measurements 115 6.1 Benchmarks .................................... 115 6.2 Proofs ........................................ 116 6.2.1 Example: Proof of Unmap Commutation . 117 6.2.2 Quantification of the Proof Effort . 118 6.2.3 Hints to Time Spent on Proofs . 121 6.3 Proof Maintenance ................................ 121 6.4 Conclusion ..................................... 121 Conclusion 123 6.5 Summary ...................................... 123 6.6 Contributions ................................... 123 6.7 Perspectives .................................... 124 Glossary 127 Bibliography 129 v List of Figures 1 Shadow Page Tables ............................... x 1.1 Monolithic versus Micro-Kernel based OS ................... 5 1.2 Two-Level Page Table ............................... 7 1.3 Shadow Page Tables ............................... 8 1.4 Commutation Diagram .............................. 12 1.5 Example of the Signature of a Predicate in Smart . 13 1.6 Example of Lemma Written in Smart .....................