Efficient Subobject-Granularity Spatial Memory Safety Enforcement with In-Fat Pointer by Shengjie Xu A thesis submitted in conformity with the requirements for the degree of Master of Applied Science Graduate Department of Electrical & Computer Engineering University of Toronto c Copyright 2021 by Shengjie Xu Abstract Efficient Subobject-Granularity Spatial Memory Safety Enforcement with In-Fat Pointer Shengjie Xu Master of Applied Science Graduate Department of Electrical & Computer Engineering University of Toronto 2021 Lack of memory safety in programming languages like C and C++ can leave programs written in these languages with exploitable memory corruption vulnerabilities. Spatial memory safety defense can catch memory corruptions from out-of-bounds pointer arithmetic. However, none of the existing works can achieve low overhead, high compatibility, and fine-grained protection at the same time. This thesis presents In-Fat Pointer, a hardware-assisted spatial memory safety defense that improves the protection granularity of existing tagged-pointer schemes using object metadata to subobject-bound granularity while maintaining their high compatibility and low overhead. In-Fat Pointer introduces multiple object metadata schemes to spare pointer tag bits from object metadata lookup, and use the spared bits with in-memory type metadata for subobject bound computation. The hardware prototype is implemented on an FGPA board, and In-Fat Pointer is evaluated in functionality, runtime and memory performance, and estimated hardware cost. ii Acknowledgements I would like to thank my supervisor, Professor David Lie, for his support and guidance throughout my research. I greatly appreciate his effort and advice on all aspects of my academic development. I also want to thank Wei Huang for his valuable support and feedback on this project, and I'm grateful to He (Shawn) Shuang, Lianying (Viau) Zhao, Rongzhen (Gavin) Cui, and Weicheng Cao for their feedback and discussion. I'm also grateful for the financial support from University of Toronto and Department of Electrical and Computer Engineering. Lastly, I would like to thank my family and friends for their support throughout my studies. iii Contents 1 Introduction 1 1.1 Contributions . .2 1.2 Thesis Structure . .2 2 Background and Related Works 4 2.1 Memory Errors and Attack Methods . .4 2.2 Tagged-pointers . .5 2.3 Related work . .5 2.3.1 Fat-pointer scheme . .7 2.3.2 Object-based scheme . .8 2.3.3 Memory-based scheme . .9 2.3.4 Tagged-pointer scheme for spatial memory safety . .9 2.4 Implementation background . 10 2.4.1 FPGA and HDL . 10 2.4.2 RISC-V Instruction Set . 10 2.4.3 Message Authentication Code . 11 3 Design of In-Fat Pointer 12 3.1 Overview . 13 3.1.1 Promote Instruction and Metadata Organization . 15 3.1.2 Poison bits . 16 3.1.3 In-Fat Pointer Registers . 17 3.2 Object Metadata Schemes . 19 3.2.1 Local Offset Scheme . 20 3.2.2 Subheap Scheme . 20 3.2.3 Global Table Scheme . 21 3.3 Layout Table and Subobject Bound Narrowing . 22 3.3.1 Layout Table . 22 3.3.2 Subobject Index . 26 3.4 Compiler Instrumentation . 27 4 Implementation Details 31 4.1 Metadata Format . 31 4.1.1 Global Table Scheme . 32 iv 4.1.2 Local Offset Scheme . 33 4.1.3 Subheap Scheme . 33 4.1.4 Layout table . 34 4.2 Instruction Set Extension . 35 4.2.1 IFPR and calling convention . 35 4.2.2 New Instructions . 38 4.2.3 New CSRs . 41 4.3 Hardware Modification . 42 4.3.1 Load-store unit . 43 4.3.2 IFP Unit . 44 4.3.3 Debug facility . 46 4.4 Compiler Modification . 47 4.4.1 IR Analysis and Instrumentation Pass . 47 4.4.2 Backend Modification . 57 4.5 Software Modification . 57 5 Evaluation 59 5.1 Functional Evaluation . 59 5.2 Microbenchmark . 60 5.3 Performance Evaluation . 63 5.3.1 Runtime Overhead . 65 5.3.2 Memory Overhead . 69 5.4 Hardware Overhead . 70 6 Limitations 72 6.1 Tagged Pointers . 72 6.2 Object Metadata Schemes . 72 6.3 Subobject Protection and Layout Tables . 73 7 Conclusion 74 7.1 Future works . 74 Bibliography 76 v List of Tables 2.1 Related Works Comparison . .6 3.1 Object Metadata Schemes Comparison . 19 3.2 Layout Table Generation Rules . 27 3.3 Handling of Pointer Operations . 27 4.1 Layout Table Formats . 34 4.2 List of New Instructions . 39 5.1 Average Runtime of Evaluated Applications . 66 5.2 Dynamic Instruction Count Composition of Evaluated Applications . 66 5.3 Average Memory Consumption of Evaluated Applications . 69 vi List of Figures 2.1 Category of Related Work . .7 3.1 In-Fat Pointer Workflow . 12 3.2 In-Fat Pointer Bound Retrieval . 13 3.3 In-Fat Pointer Instrumented Operations (High-level) . 14 3.4 In-Fat Pointer Instrumentation Overview (Code Example) . 15 3.5 Metadata Organization and Promote Operation Overview . 16 3.6 Pointer Form Conversions and Operations . 18 3.7 Local Offset Scheme . 20 3.8 Subheap Scheme . 21 3.9 Global Table scheme . 22 3.10 Layout Table Example . 23 3.11 Compiler Instrumentation for Pointer Operations . 28 4.1 Pointer Tag Formats . 32 4.2 Processor Micro-architecture Changes . 43 4.3 Pointer Dataflow Graph Example . 51 5.1 Promotion Performance Microbenchmark . 61 5.2 Runtime Overhead on Evaluated Applications . 65 5.3 Dynamic Instruction Count Composition on Evaluated Applications . 67 5.4 IFP Instruction Dynamic Count Composition on Evaluated Applications . 68 5.5 Memory Overhead on Evaluated Applications . 69 5.6 LUT Usage Increase in the Modified Processor . 71 5.7 LUT Usage Breakdown for IFP Unit . 71 vii Chapter 1 Introduction Memory corruption vulnerabilities in programs written in unsafe programming languages like C and C++ have been a research focus for more than a decade [41]. These languages do not strictly enforce the memory safety requirement that a pointer should point to the object from which it is derived from, therefore when such a.