DYNAMIC POINTER TRACKING AND ITS APPLICATIONS A Dissertation Presented to The Academic Faculty by Kun Zhang In Partial Fulfillment of the Requirements for the Degree Doctor of Philosophy in the College of Computing Georgia Institute of Technology May 2010 COPYWRITE 2010 BY KUN ZHANG DYNAMIC POINTER TRACKING AND ITS APPLICATIONS Approved by: Dr. Santosh Pande, Advisor Dr. Nathan Clark College of Computing College of Computing Georgia Institute of Technology Georgia Institute of Technology Dr. Jonathon Giffin Dr. Hyesoon Kim College of Computing College of Computing Georgia Institute of Technology Georgia Institute of Technology Dr. Hsien-Hsin S. Lee School of Electrical and Computer Engineering Georgia Institute of Technology Date Approved: December 10, 2009 ACKNOWLEDGEMENTS I wish to thank all of the people who support me during the time of pursuing my PhD. First of all, I would like to give my extraordinary gratitude to my advisor Santosh Pande for his good supervision and valuable advice through my PhD study. He provided me encouragement and support in various ways. His passions for research and his novel ideas nourish my growth as a PhD student. Thanks to all of my committee members, Dr. Nathan Clark, Dr. Jonathon Giffin, Dr. Hyesson Kim and Dr. Hsien-Hsin S. Lee for advising me in my dissertation work. Their suggestions and comments help me to improve the quality of this dissertation. I would like to acknowledge my colleagues, Tao Zhang, Xiaotong Zhuang, Tushark Kumar, Romain Cledat, and Jaswanth Sreeram. It is my pleasure to work with them in our compiler team. Special thanks to my husband and my parents for their constant understanding, unselfish support and persistent confidence in me. Finally, I would like to thank all my friends at different places who make my graduate study enjoyable. iii TABLE OF CONTENT Page ACKNOWLEDGEMENTS.............................................................................................. III LIST OF FIGURES ........................................................................................................... X SUMMARY ................................................................................................................... XIII 1. INTRODUCTION .......................................................................................................... 1 1.1 Motivation............................................................................................................... 1 1.2 Traditional Pointer Analysis Approaches................................................................ 3 1.3 Our Approach.......................................................................................................... 5 1.4 Contribution Statement ........................................................................................... 6 1.5 Thesis Organization ................................................................................................ 7 2. DYNAMIC POITER TRACKING MOTIVATION EXAMPLES................................. 9 2.1 Memory Protection ................................................................................................. 9 2.2 Program Parallelism.............................................................................................. 10 2.3 Path Prediction...................................................................................................... 11 2.4 Dynamic Invariant Detection................................................................................ 12 2.5 Garbage Collection ............................................................................................... 14 2.6 Summary............................................................................................................... 15 3. BASIC DYNAMIC POINTER TRACKING ALGORITHMS .................................... 16 3.1 Data Structures...................................................................................................... 17 3.2 Special Issues on Data Structure Design .............................................................. 17 3.2.1 Pointer on the Heap...................................................................................... 18 3.2.2 Recursive Function ...................................................................................... 20 3.2.3 Explicit and Implicit Pointer Array.............................................................. 20 iv 3.2.4 Pointer Parameter and Pointer Return.......................................................... 22 3.2.5 Pointer Type Casting.................................................................................... 23 3.2.6 Function Pointer........................................................................................... 24 3.2.7 Unions.......................................................................................................... 24 3.3 Data Structure Management ................................................................................. 25 3.4 Update Instructions ............................................................................................... 28 3.4.1 Function Entry and Exit............................................................................... 28 3.4.2 Malloc and Free ........................................................................................... 28 3.4.3 Pointer Assignments..................................................................................... 29 3.5 Limitations of the Work ........................................................................................ 31 3.5.1 Compatibility with Legacy Code................................................................. 31 3.5.2 Separation of Space...................................................................................... 32 3.5.3 Visibility of Pointers and Pointer Assignments ........................................... 32 3.5.4 Setjmp/Longjmp .......................................................................................... 33 3.6 User Support ......................................................................................................... 33 3.7 Possible Optimizations.......................................................................................... 34 3.7.1 Pointer Definition and Use........................................................................... 34 3.7.2 Remove Redundant Pointer Chasing Update............................................... 36 3.7.3 Delay Update out of Loop............................................................................ 39 3.7.4 Delay Update for Special Patterns ............................................................... 41 3.7.5 Remove Update for Pointer Initialization.................................................... 44 3.7.6 Propagate Update......................................................................................... 46 3.7.7 Invariant Removal........................................................................................ 48 3.7.8 On Demand Dynamic Pointer Tracking....................................................... 49 3.8 Dynamic Pointer Tracking Experimental Results................................................. 50 v 3.8.1 Static Pointer Analysis ................................................................................. 51 3.8.2 Dynamic Pointer Analysis............................................................................ 53 3.8.3 Recursive Function Analysis........................................................................ 55 3.8.4 Performance Evaluation............................................................................... 56 3.8.5 Dynamic Load/Store .................................................................................... 57 3.8.6 Space Cost.................................................................................................... 58 3.9 Summary............................................................................................................... 58 4. MEMORY PROTECTION THROUGH DYNAMIC POINTER TRACKING........... 59 4.1 Introduction of the Problem.................................................................................. 59 4.2 Previous Approaches............................................................................................. 61 4.2.1 Control Flow Monitoring............................................................................. 62 4.2.2 Non-control Data Attack Detection ............................................................. 64 4.2.3 Bounds Checking......................................................................................... 68 4.2.4 Other Approaches......................................................................................... 70 4.2.5 Previous Work against Attacks .................................................................... 71 4.2.6 Our Approach............................................................................................... 74 4.3 Attack Model ........................................................................................................ 74 4.4 Compiler Analysis and Optimizations .................................................................. 75 4.4.1 Baseline Scheme .......................................................................................... 75 4.4.2 Compiler Framework Overview .................................................................. 76 4.4.3 Write Range Identification........................................................................... 78 4.4.4 Protection Points Hoisting