This document is downloaded from DR‑NTU (https://dr.ntu.edu.sg) Nanyang Technological University, Singapore. Auditing buffer overflow vulnerabilities using program analysis and data mining techniques Bindu Madhavi Padmanabhuni 2016 Padmanabhuni, B. M. (2016). Auditing buffer overflow vulnerabilities using program analysis and data mining techniques. Doctoral thesis, Nanyang Technological University, Singapore. https://hdl.handle.net/10356/68915 https://doi.org/10.32657/10356/68915 Downloaded on 03 Oct 2021 20:09:18 SGT AUDITING BUFFER OVERFLOW VULNERABILITIES USING PROGRAM ANALYSIS AND DATA MINING TECHNIQUES BINDU MADHAVI PADMANABHUNI SCHOOL OF ELECTRICAL AND ELECTRONIC ENGINEERING 2016 AUDITING BUFFER OVERFLOW VULNERABILITIES USING PROGRAM ANALYSIS AND DATA MINING TECHNIQUES BINDU MADHAVI PADMANABHUNI BINDU MADHAVI PADMANABHUNI School of Electrical and Electronic Engineering A thesis submitted to the Nanyang Technological University in partial fulfillment of the requirement for the degree of Doctor of Philosophy 2016 Acknowledgment Life as a research student was filled with myriad of enriching experiences. The skills learnt, the challenges faced and the conversations held have significantly contributed towards enhancing my critical-thinking and problem-solving skills. I have many cherished moments that are fondly etched in the memory and am indebted to all those who made them possible. I would like to express my sincere and profound gratitude to my supervisor, Prof. Tan Hee Beng Kuan for initiating me into the field of research and nurturing me throughout the course of study. I am extremely grateful for his invaluable guidance, advice and continuous support. Under his tutelage, I have become a fine researcher and better writer. I also take this opportunity to thank Dr. Chia Tee Kiah for sharing his expertise and giving insightful suggestions on various research ideas. Special mention of thanks goes to Vivek, Le Ha Thanh, Deepak and Lei Lei Win for the wonderful tête-à-têtes, debates and camaraderie during my time in the Computer Security Lab. I would also like to thank my fellow students Kaiping, Shar, Ding Sun, Mahinthan and Charlie for the rigorous and fruitful discussions pertaining research ideas and issues at hand. I am greatly indebted to my parents, especially my mother who has always been my pillar of strength, parents-in-law and sisters for their care, encouragement and love. Special thanks to my husband, Sridhar, for his practical and emotional support and my children Shyam and Saket for brightening up my days with their love and innocence. ii TABLE OF CONTENTS Acknowledgment ............................................................................................................................... ii Table of Contents ............................................................................................................................. iii Summary….. .................................................................................................................................... vi List of Figures .................................................................................................................................. ix List of Tables .................................................................................................................................... xi Chapter 1 Introduction ................................................................................................................... 1 1.1 Motivation ......................................................................................................................... 1 1.2 Objectives .......................................................................................................................... 4 1.3 Overview of Our Research ................................................................................................ 4 1.4 Major Contributions .......................................................................................................... 6 1.5 Thesis Organization ........................................................................................................... 7 Chapter 2 Background ................................................................................................................. 10 2.1 Buffer Overflow Vulnerability ........................................................................................ 11 2.1.1 Function Activation Record Exploits ...................................................................... 11 2.1.2 Pointer Subterfuge Exploits ..................................................................................... 12 2.1.3 Heap-based Exploits ................................................................................................ 14 2.2 Program Analysis ............................................................................................................ 15 2.2.1 Static Program Analysis .......................................................................................... 15 2.2.2 Dynamic Program Analysis ..................................................................................... 18 2.3 Data Mining ..................................................................................................................... 20 2.3.1 Data Pre-processing ................................................................................................. 20 2.3.2 Data Analysis ........................................................................................................... 21 2.3.3 Classifiers ................................................................................................................ 22 2.3.4 Evaluation ................................................................................................................ 26 Chapter 3 Related Work ............................................................................................................... 28 3.1 Defensive Coding Practices ............................................................................................. 28 3.2 Vulnerability Testing ....................................................................................................... 32 3.3 Data Mining Techniques ................................................................................................. 36 3.4 Vulnerability Detection ................................................................................................... 40 3.4.1 Static Analysis-based Vulnerability Detection ........................................................ 40 3.4.2 Hybrid Analysis-based Vulnerability Detection ...................................................... 43 3.5 Runtime Attack Prevention ............................................................................................. 44 3.6 Summary ......................................................................................................................... 51 iii Chapter 4 Detecting Buffer Overflow Vulnerabilities through Light-Weight Rule-Based Test Case Generation 52 4.1 Research Hypothesis ....................................................................................................... 54 4.2 Rule-Based Test Case Generation ................................................................................... 55 4.2.1 Input Length (IL) Rule ............................................................................................ 56 4.2.2 Character Check (CC) Rule .................................................................................... 56 4.2.3 String Check (SC) Rule ........................................................................................... 58 4.2.4 Pattern Check (PC) Rule ......................................................................................... 59 4.2.5 Specific Character at Specific Index Check (SCASIC) .......................................... 59 4.3 Test Case Generation Framework ................................................................................... 62 4.4 Prototype Tool and Benchmarks ..................................................................................... 62 4.5 Evaluation ....................................................................................................................... 63 4.5.1 Comparison with Symbolic Evaluation Solution .................................................... 65 4.5.2 Comparison with Evolutionary Algorithm .............................................................. 66 4.6 Conclusion ...................................................................................................................... 67 Chapter 5 Hybrid Vulnerability Auditing from Static-Dynamic Analysis and Machine Learning….. ................................................................................................................................... 69 5.1 Research Hypothesis ....................................................................................................... 71 5.2 Static Code Attributes ..................................................................................................... 72 5.2.1 Sink and Input Classification .................................................................................. 73 5.2.2 Input Validation and Buffer Size Check Predicate Classification .......................... 74 5.2.3 Data Buffer Declaration Statement Classification .................................................. 77 5.2.4 Sink Characteristics Classification .........................................................................