DETECTING BUFFER OVERFLOWS USING TESTCASE SYNTHESIS AND CODE INSTRUMENTATION by MICHAEL A. ZHIVICH S.B., Electrical Engineering and Computer Science (2004) Massachusetts Institute of Technology SUBMITTED TO THE DEPARTMENT OF ELECTRICAL ENGINEERING AND COMPUTER SCIENCE IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF MASTER OF ENGINEERING IN ELECTRICAL ENGINEERING AND COMPUTER SCIENCE at the MASSACHUSETTS INSTITUTE OF TECHNOLOGY June 2005 c Massachusetts Institute of Technology 2005. All rights reserved. Author............................................................................ Department of Electrical Engineering and Computer Science May 19, 2005 Certified by........................................................................ Richard Lippmann Senior Scientist, MIT Lincoln Laboratory Thesis Supervisor Accepted by....................................................................... Arthur C. Smith Chairman, Department Committee on Graduate Students 2 Detecting Buffer Overflows Using Testcase Synthesis and Code Instrumentation by Michael A. Zhivich Submitted to the Department of Electrical Engineering and Computer Science on May 19, 2005, in partial fulfillment of the requirements for the degree of Master of Engineering in Electrical Engineering and Computer Science Abstract The research presented in this thesis aims to improve existing approaches to dynamic buffer overflow detection by developing a system that utilizes code instrumentation and adap- tive test case synthesis to find buffer overflows and corresponding failure-inducing inputs automatically. An evaluation of seven modern dynamic buffer overflow detection tools de- termined that C Range Error Detector (CRED) is capable of providing fine-grained buffer access information necessary for the creation of this system. CRED was also selected be- cause of its ability to provide comprehensive error reports and compile complex programs with reasonable performance overhead. CRED was extended to provide appropriate code instrumentation for the adaptive testing system, which also includes a test case synthe- sizer that uses data perturbation techniques on legal inputs to produce new test cases, and an analytical module that evaluates the effectiveness of these test cases. Using informa- tion provided by code instrumentation in further test case generation creates a feedback loop that enables a focused exploration of the input space and faster buffer overflow detec- tion. Applying the adaptive testing system to jabberd, a Jabber Instant Messaging server, demonstrates its effectiveness in finding buffer overflows and its advantages over existing dynamic testing systems. Adaptive test case synthesis using CRED to provide buffer access information for feedback discovered 6 buffer overflows in jabberd using only 53 messages, while dynamic testing using random messages generated from a protocol description found only 4 overflows after sending 10,000 messages. Thesis Supervisor: Richard Lippmann Title: Senior Scientist, MIT Lincoln Laboratory Acknowledgments I am indebted to many people without whom this work would not be possible. First and foremost, I would like to thank my advisor, Richard Lippmann, for his guidance, suggestions and constructive criticism throughout this project. I would also like to thank Tim Leek for all his contributions to this work, including the real exploit evaluation of dynamic buffer overflow detection tools and modifications to the PCFG random message generator and CRED-HW compiler. Without Tim’s contributions and technical expertise, building an adaptive testing system would have been very difficult. I would like to thank Misha Zitser for developing a thorough classification of buffer overflows and providing the test corpus from his thesis, which was an invaluable part of the evaluation of dynamic buffer overflow detection tools. I would also like to thank Rob Cunningham, Graham Baker, Kendra Kratkiewicz, Chris Scott and all the folks in Group 62 at the Lincoln Laboratory, with whom I have had many interesting discussions, and who have made my experience at Lincoln so rewarding. Last, but certainly not least, I would like to thank my family and friends for their encouragement and support throughout this time — I would not have made it here without you, and I give you my heartfelt thanks. Contents 1 Introduction 13 1.1 Cost of Software Failures . 13 1.2 Types of Vulnerabilities in Software . 15 1.3 Buffer Overflows — A Persistent Vulnerability . 16 1.4 Approaches to Buffer Overflow Prevention . 16 1.5 Goals of this Thesis . 19 2 Exploiting Buffer Overflows 21 2.1 Memory Layout of a Process . 21 2.2 Arbitrary Code Execution . 23 2.2.1 Injecting Code . 23 2.2.2 Changing Code Pointer Directly . 24 2.2.3 Changing Code Pointer Indirectly . 24 2.3 Logic-based Attacks . 25 2.4 Unauthorized Access to Information . 26 2.5 Classifying Buffer Overflows . 26 2.5.1 Access Pattern . 27 2.5.2 Access Type . 27 2.5.3 Size of Overflow . 27 2.5.4 Location . 28 3 Approaches to Buffer Overflow Detection 29 3.1 Security Policies and Code Reviews . 29 3.2 Language Approach . 30 3.3 Safe C Libraries . 31 5 3.4 Operating System Extensions . 32 3.5 Static Analysis . 32 3.6 Runtime Buffer Overflow Detection . 33 3.6.1 Compiler-based Code Instrumentation . 33 3.6.2 Executable Monitoring . 35 3.7 Software Fault Injection . 37 3.8 Mini-simulation with Random Messages . 37 3.9 An Adaptive Testing System . 38 4 Evaluating Dynamic Buffer Overflow Detection Tools 39 4.1 Dynamic Buffer Overflow Detection Tools . 40 4.1.1 Executable Monitoring Tools . 40 4.1.2 Compiler-based Tools . 41 4.1.3 Common Limitations of Compiler-based Tools . 44 4.2 Variable-overflow Testsuite Evaluation . 45 4.2.1 Test Procedure . 45 4.2.2 Sample Analysis . 46 4.2.3 Variable-overflow Testsuite Results . 50 4.3 Artificial Exploit Evaluation . 52 4.3.1 Test Procedure . 53 4.3.2 Artificial Exploit Results . 53 4.4 Real Exploit Evaluation . 53 4.4.1 Test Procedure . 54 4.4.2 Real Exploit Results . 55 4.5 Performance Overhead . 55 4.6 Discussion . 57 5 Dynamic Testing with Code Instrumentation 59 5.1 Dynamic Testing Architecture . 60 5.1.1 Code Instrumentation . 61 5.1.2 Tester and Random Message Generator . 61 5.1.3 Process Monitor . 62 5.2 Random Message Generation . 63 6 5.2.1 HMM Random Message Generator . 63 5.2.2 PCFG Random Message Generator . 64 5.3 Evaluating Dynamic Testing . 66 5.3.1 Jabber Instant Messaging Server . 66 5.3.2 Test Environment . 68 5.3.3 Test Procedure . 69 5.3.4 Test Results . 71 5.4 Discussion . 73 6 Adaptive Test Case Synthesis 75 6.1 Dynamic Testing with Feedback . 75 6.2 Message Database . 76 6.3 Test Case Generator . 78 6.3.1 Tokenizing Messages . 79 6.3.2 Mutating Tokens . 80 6.4 Feedback Loop . 81 6.4.1 CRED-HW Instrumentation . 81 6.4.2 Buffer Access Signature Analysis . 82 6.5 Static Analysis Module . 84 6.5.1 CIL: C Intermediate Language . 86 6.5.2 Source Analysis with CIL . 86 6.5.3 Literals in jabberd ............................ 87 6.6 Evaluating Adaptive Testing System . 87 6.6.1 Test Environment . 87 6.6.2 Test Procedure . 88 6.6.3 Sample Test Run . 90 6.6.4 Test Results . 93 6.7 Discussion . 96 7 Future Work 99 7.1 Tokenizing Messages . 99 7.2 Token Mutations . 100 7.3 Buffer Instances . 100 7 8 Conclusions 103 8.1 Required Instrumentation . 103 8.2 Generating Test Cases . 104 8.3 Advantages of the Adaptive Testing Approach . 105 A HMM Description 107 B PCFG Description 123 C CRED Error Messages 127 D PCFG Messages 129 E Feedback Messages 135 F Literals Found in jabberd 141 8 List of Figures 1-1 Cumulative Exploits in Commonly Used Server Software . 17 2-1 Memory Layout of a Process . 22 4-1 Example of Discrete Overflow Test Case . 46 4-2 Stack Layout for Discrete Buffer Overflow Example . 48 4-3 Results for Discrete Buffer Overflow Example . 48 4-4 Example of Continuous Overflow Test Case . 49 4-5 Stack Layout for Continuous Buffer Overflow Example . 51 4-6 Results for Continuous Buffer Overflow Example . 51 4-7 Summary of Results for Variable-overflow Testsuite . 52 5-1 System Architecture for Dynamic Testing with Code Instrumentation . 60 5-2 Typical Jabber Server Deployment . 67 5-3 Architecture of jabberd ............................. 69 5-4 A Typical Jabber IM Message . 70 6-1 System Architecture for Adaptive Testing . 77 6-2 Test Case Generator Architecture . 78 6-3 A Legal Jabber IM Message . ..