UCAM-CL-TR-954 Technical Report ISSN 1476-2986 Number 954 Computer Laboratory Inline and sideline approaches for low-cost memory safety in C Myoung Jin Nam February 2021 15 JJ Thomson Avenue Cambridge CB3 0FD United Kingdom phone +44 1223 763500 https://www.cl.cam.ac.uk/ c 2021 Myoung Jin Nam This technical report is based on a dissertation submitted November 2020 by the author for the degree of Doctor of Philosophy to the University of Cambridge, Selwyn College. Technical reports published by the University of Cambridge Computer Laboratory are freely available via the Internet: https://www.cl.cam.ac.uk/techreports/ ISSN 1476-2986 Inline and Sideline Approaches for Low-cost Memory Safety in C Myoung Jin Nam System languages such as C or C++ are widely used for their high performance, however the allowance of arbitrary pointer arithmetic and typecast introduces a risk of memory corruptions. These memory errors cause unexpected termination of pro- grams, or even worse, attackers can exploit them to alter the behavior of programs or leak crucial data. Despite advances in memory safety solutions, high and unpredictable overhead remains a major challenge. Accepting that it is extremely difficult to achieve com- plete memory safety with the performance level suitable for production deployment, researchers attempt to strike a balance between performance, detection coverage, in- teroperability, precision, and detection timing. Some properties are much more desir- able, e.g. the interoperability with pre-compiled libraries. Comparatively less critical properties are sacrificed for performance, for example, tolerating longer detection de- lay or narrowing down detection coverage by performing approximate or probabilistic checking or detecting only certain errors. Modern solutions compete for performance. The performance matrix of memory safety solutions has two major assessment crite- ria – run-time and memory overheads. Researchers trade-off and balance performance metrics depending on its purpose or placement. Many of them tolerate the increase in memory use for better speed, since memory safety enforcement is more desirable for troubleshooting or testing during development, where a memory resource is not the main issue. Run-time overhead, considered more critical, is impacted by cache misses, dynamic instructions, DRAM row activations, branch predictions and other factors. This research proposes, implements, and evaluates MIU: Memory Integrity Utilities containing three solutions – MemPatrol, FRAMER and spaceMiu. MIU suggests new techniques for practical deployment of memory safety by exploiting free resources with the following focuses: (1) achieving memory safety with overhead < 1% by using concurrency and trading off prompt detection and coverage; but yet providing eventual detection by a monitor isolation design of an in-register monitor process and the use of AES instructions (2) complete memory safety with near-zero false negatives focusing on eliminating overhead, that hardware support cannot resolve, by using a new tagged-pointer representation utilising the top unused bits of a pointer. Acknowledgments First of all, the completion of my PhD would not have been possible without all the support of my supervisor Dr. David J. Greaves. I would like to express my gratitude for his support, advice, and patience. I would like to extend my gratitude to my examiners, Prof. Jon Crowcroft and Prof. Alastair F. Donaldson at Imperial College London, for sharing their valuable ideas and filling the viva with kindness. My sincere thanks also go to Prof. Lawrence C. Paulson, Prof. Byron Cook at University College London, and Dr. Sean B. Holden for their constant encouragement and help. I am indebted to Dr. Periklis Akritidis at Niometrics for his insightful guidance, support, and profound belief in my work and me, along with constructive criticism that pushed me to become a better version of myself. My gratitude extends to Dr. Stephen Kell at University of Kent, Prof. Guy Lemieux at University of British Columbia, Prof. Jin-Young Choi at Korea University, Prof. Wonhong Nam at KonKuk University, Christos Rikoudis at Niometrics, and Dr. Matt Staats at Google for their support and helpful advice that improved my work. I gratefully acknowledge the funding from the Research Foundation of Korea. I would like to thank my former lab mates from the Networks and Operating Sys- tems Group and Automated Reasoning Group and friends for their company and care that made my life in Cambridge a wonderful time, especially (I will omit titles of ’Dr’ from this point. Sorry, Doctors!) – Julien Quintard, Martin A. Ruehl, Ce- cile Ritte, Bence Pasztor, Shu Yan Chan, Atif Alvi, Aisha Elsafty, Jordan Auge, Jean E. Martina, Damien Fay, Ilias Leontiadis, Charalampos Rotsos, Thomas Tuerk, Joon Woong Kim, Maria Rita Massaro, Foula Vagena, Mina Brimpari, Daniel Holland, Tony Coleby, Nicole Coleby, and Vassilis Laganakos. I am also thankful to dear friends for their company during my time in Seoul and Singapore – Hyunyoung Kil, Jeongeun Suh, So Jin Ahn, Nickos Ventouras, Miyoung Kang, Chong Yi, and many more. My deepest gratitude goes to my parents and two brothers for their support and everything. And finally, this thesis is dedicated to my dearest Miu – my little girl, my fur angel – in heaven. 5 Contents Abstract 3 Acknowledgements5 Table of contents9 List of figures 11 List of tables 13 1 Introduction 15 1.1 Contributions.................................. 17 2 Background 19 2.1 Memory Safety................................. 21 2.1.1 Spatial Memory Safety......................... 21 2.1.2 Temporal Memory Safety....................... 25 2.1.3 Architectural Support and Capability Model............ 26 2.2 Type Safety.................................... 29 2.3 Control-Flow Protection............................ 32 2.4 Concurrent Monitoring............................. 33 2.4.1 Monitor Isolation............................ 34 2.4.2 Kernel Integrity Monitors....................... 35 2.4.3 Cryptographic Key Protection.................... 35 3 Overview 37 3.1 FRAMER..................................... 37 3.2 spaceMiu..................................... 37 3.3 MemPatrol.................................... 38 4 FRAMER: A Tagged-Pointer Capability System with Memory Safety Appli- cations 39 4.1 Overview..................................... 39 4.2 FRAMER Approach.............................. 40 4.2.1 Frame Definitions........................... 41 4.2.2 Frame Selection............................. 42 4.2.3 Metadata Storage Management.................... 42 4.3 FRAMER Implementation........................... 46 4.3.1 Overview................................ 46 7 8 Contents 4.3.2 Memory Allocation Transformations................ 47 4.3.3 Memory Access............................. 48 4.3.4 Interoperability............................. 49 4.4 FRAMER Applications............................. 49 4.4.1 Spatial Memory Safety......................... 49 4.4.2 Temporal Memory Safety....................... 52 4.5 Optimisations.................................. 52 4.6 Evaluation.................................... 53 4.6.1 Memory Overhead........................... 54 4.6.2 Slowdown................................ 55 4.6.3 Data Cache Misses........................... 57 4.6.4 Instructions Executed......................... 58 4.6.5 Branch Misses.............................. 59 4.7 Discussion.................................... 60 4.7.1 Comparison with Other Approaches................ 60 4.7.2 Hardware Implementation of FRAMER............... 60 4.7.3 Additional Optimisations....................... 61 4.8 Conclusion.................................... 61 5 SpaceMiu: Practical Type Safety for C 63 5.1 Overview..................................... 63 5.2 Types and Type Relations........................... 64 5.2.1 Type Representation.......................... 64 5.2.2 Physical Equality............................ 65 5.2.3 Physical Sub-typing and Type Relation............... 66 5.3 Run-Time Typecast Checking......................... 66 5.3.1 Object-to-type Mapping........................ 68 5.3.2 Per-object Metadata Management and Pointer-to-Type Mapping 69 5.3.3 Type Confusion Checking....................... 70 5.4 Union Type................................... 71 5.5 spaceMiu Implementation........................... 73 5.5.1 Creation of Type Descriptors..................... 74 5.5.2 Program Initialisation......................... 74 5.5.3 Memory Allocation........................... 74 5.5.4 Type Cast................................ 76 5.5.5 Memory Access............................. 76 5.5.6 String Functions............................ 76 5.6 Evaluation.................................... 76 5.6.1 Memory Overheads.......................... 77 5.6.2 Slowdown................................ 78 5.6.3 Executed Instructions......................... 79 5.6.4 L1 D-cache Misses........................... 80 5.6.5 Branch Prediction............................ 80 5.7 Discussion.................................... 81 5.7.1 Effective Type Detection for Heap Objects............. 81 5.7.2 String Functions and Effective Type................. 81 Contents 9 5.7.3 Aliasing Rules.............................. 82 5.7.4 Per-object Metadata Placement.................... 82 5.8 Conclusion.................................... 83 6 MemPatrol: Reliable Sideline Integrity Monitoring for High-Performance Systems 85 6.1 Overview..................................... 85 6.2 Threat Model.................................