Improved Kernel Security Through Code Validation, Diversification, and Minimization Dannie Michael Stanley Purdue University
Total Page：16
File Type：pdf, Size：1020Kb
Purdue University Purdue e-Pubs Open Access Dissertations Theses and Dissertations Fall 2013 Improved Kernel Security Through Code Validation, Diversification, and Minimization Dannie Michael Stanley Purdue University Follow this and additional works at: https://docs.lib.purdue.edu/open_access_dissertations Part of the Computer Sciences Commons Recommended Citation Stanley, Dannie Michael, "Improved Kernel Security Through Code Validation, Diversification, and Minimization" (2013). Open Access Dissertations. 44. https://docs.lib.purdue.edu/open_access_dissertations/44 This document has been made available through Purdue e-Pubs, a service of the Purdue University Libraries. Please contact [email protected] for additional information. Graduate School ETD Form 9 (Revised 12/07) PURDUE UNIVERSITY GRADUATE SCHOOL Thesis/Dissertation Acceptance This is to certify that the thesis/dissertation prepared By Dannie Michael Stanley Entitled Improved Kernel Security Through Code Validation, Diversification, and Minimization Doctor of Philosophy For the degree of Is approved by the final examining committee: Eugene H. Spafford Chair Dongyan Xu Samuel Liles Sonia Fahmy To the best of my knowledge and as understood by the student in the Research Integrity and Copyright Disclaimer (Graduate School Form 20), this thesis/dissertation adheres to the provisions of Purdue University’s “Policy on Integrity in Research” and the use of copyrighted material. Approved by Major Professor(s): ____________________________________Eugene H. Spafford _______________Dongyan Xu _____________________ Approved by: Sunil Prabhakar / William J. Gorman 12/02/2013 Head of the Graduate Program Date IMPROVED KERNEL SECURITY THROUGH CODE VALIDATION, DIVERSIFICATION, AND MINIMIZATION A Dissertation Submitted to the Faculty of Purdue University by Dannie M. Stanley In Partial Fulfillment of the Requirements for the Degree of Doctor of Philosophy December 2013 Purdue University West Lafayette, Indiana ii To my wife, my parents, and my children. iii ACKNOWLEDGMENTS I would like to express my deep appreciation and gratitude to my advisors Dr. Eugene H. Spafford and Dr. Dongyan Xu for their patience, guidance, encouragement and the many wonderful opportunities that they afforded me. I am fortunate to have the opportunity to learn from such highly respected and accomplished mentors. I would also like to thank my additional committee members Dr. Sonia Fahmy and Dr. Samuel Liles for their guidance and helpful feedback. I would also like to thank Dr. Ryan Riley. Ryan has been a peer-mentor to me. He helped guide me through the early years of graduate school. I am forever indebted to him and aspire to follow in his footsteps. Finally, I would like to thank my wife. She is my champion. She provided light when the rigors of graduate school were casting a shadow. She has sacrificed beyond what I wished for her and has done so with grace. iv TABLE OF CONTENTS Page LIST OF TABLES :::::::::::::::::::::::::::::::: vii LIST OF FIGURES ::::::::::::::::::::::::::::::: viii ABSTRACT ::::::::::::::::::::::::::::::::::: ix 1 Introduction :::::::::::::::::::::::::::::::::: 1 1.1 Background ::::::::::::::::::::::::::::::: 1 1.2 Contributions :::::::::::::::::::::::::::::: 2 1.2.1 Diversification :::::::::::::::::::::::::: 2 1.2.2 Validation :::::::::::::::::::::::::::: 3 1.2.3 Minimization :::::::::::::::::::::::::: 4 1.3 Terminology ::::::::::::::::::::::::::::::: 4 1.4 Models :::::::::::::::::::::::::::::::::: 4 1.4.1 Computation Model :::::::::::::::::::::: 4 1.4.2 Execution Model :::::::::::::::::::::::: 6 1.4.3 Process Memory Model ::::::::::::::::::::: 7 1.4.4 Threat Model :::::::::::::::::::::::::: 10 2 Related Work ::::::::::::::::::::::::::::::::: 11 2.1 Attack Techniques ::::::::::::::::::::::::::: 11 2.1.1 Buffer Overflow Attack ::::::::::::::::::::: 11 2.1.2 Format String Attack :::::::::::::::::::::: 14 2.1.3 Integer Overflow Attack :::::::::::::::::::: 14 2.1.4 Return and Jump-Oriented Programming :::::::::: 15 2.2 Mitigation Techniques ::::::::::::::::::::::::: 19 2.2.1 Hardware Fault Isolation :::::::::::::::::::: 19 2.2.2 Reference Monitors ::::::::::::::::::::::: 21 2.2.3 Control Flow Integrity ::::::::::::::::::::: 22 2.2.4 NX :::::::::::::::::::::::::::::::: 22 2.2.5 Stack Canaries ::::::::::::::::::::::::: 25 2.2.6 System Randomization ::::::::::::::::::::: 26 2.2.7 ROP Defenses :::::::::::::::::::::::::: 28 2.3 Kernel Specialization and Minimization :::::::::::::::: 29 2.3.1 Kernel Specialization :::::::::::::::::::::: 29 2.3.2 Operating System Minimization :::::::::::::::: 30 3 Validating the Integrity of Included Kernel Components :::::::::: 32 v Page 3.1 Introduction ::::::::::::::::::::::::::::::: 32 3.2 Chapter Organization :::::::::::::::::::::::::: 34 3.3 Design :::::::::::::::::::::::::::::::::: 34 3.3.1 Problem ::::::::::::::::::::::::::::: 34 3.3.2 Approach :::::::::::::::::::::::::::: 37 3.4 Implementation ::::::::::::::::::::::::::::: 41 3.4.1 Experimental Setup ::::::::::::::::::::::: 43 3.4.2 Patch Validation :::::::::::::::::::::::: 43 3.5 Evaluation :::::::::::::::::::::::::::::::: 47 3.6 Discussion :::::::::::::::::::::::::::::::: 48 3.7 Summary :::::::::::::::::::::::::::::::: 49 4 Increasing the Diversity of Included Kernel Components ::::::::: 50 4.1 Introduction ::::::::::::::::::::::::::::::: 50 4.2 Chapter Organization :::::::::::::::::::::::::: 53 4.3 Design :::::::::::::::::::::::::::::::::: 53 4.3.1 Record Field Order Randomization :::::::::::::: 53 4.3.2 Suitability of a Record for Field Reordering ::::::::: 56 4.3.3 Subroutine Argument Order Randomization ::::::::: 59 4.4 Implementation ::::::::::::::::::::::::::::: 62 4.4.1 Record Field Order Randomization :::::::::::::: 62 4.4.2 RFOR Fitness Check :::::::::::::::::::::: 66 4.4.3 Subroutine Argument Order Randomization ::::::::: 69 4.5 Evaluation :::::::::::::::::::::::::::::::: 71 4.5.1 Security Benefits :::::::::::::::::::::::: 71 4.5.2 Randomizability Analysis ::::::::::::::::::: 72 4.5.3 Performance ::::::::::::::::::::::::::: 76 4.6 Discussion :::::::::::::::::::::::::::::::: 77 4.7 Summary :::::::::::::::::::::::::::::::: 78 5 Ensuring the Minimality of Included Kernel Components ::::::::: 79 5.1 Introduction ::::::::::::::::::::::::::::::: 79 5.2 Related Work :::::::::::::::::::::::::::::: 80 5.3 Chapter Organization :::::::::::::::::::::::::: 80 5.4 Design :::::::::::::::::::::::::::::::::: 81 5.4.1 Problem ::::::::::::::::::::::::::::: 81 5.4.2 Approach :::::::::::::::::::::::::::: 81 5.5 Implementation ::::::::::::::::::::::::::::: 86 5.6 Evaluation :::::::::::::::::::::::::::::::: 88 5.7 Discussion :::::::::::::::::::::::::::::::: 89 5.8 Summary :::::::::::::::::::::::::::::::: 90 6 Conclusions :::::::::::::::::::::::::::::::::: 91 6.1 Summary :::::::::::::::::::::::::::::::: 91 vi Page 6.1.1 Validation :::::::::::::::::::::::::::: 91 6.1.2 Diversification :::::::::::::::::::::::::: 92 6.1.3 Minimization :::::::::::::::::::::::::: 92 6.2 Future Work ::::::::::::::::::::::::::::::: 92 LIST OF REFERENCES :::::::::::::::::::::::::::: 94 VITA ::::::::::::::::::::::::::::::::::::::: 100 vii LIST OF TABLES Table Page 4.1 RFOR Fitness Report for task struct ::::::::::::::::::: 75 4.2 Performance impact of RFOR ::::::::::::::::::::::: 76 4.3 Time to compile kernel ::::::::::::::::::::::::::: 76 viii LIST OF FIGURES Figure Page 1.1 Computation model :::::::::::::::::::::::::::: 5 1.2 Execution model :::::::::::::::::::::::::::::: 6 1.3 Task memory model :::::::::::::::::::::::::::: 7 1.4 Call stack model :::::::::::::::::::::::::::::: 9 2.1 Return-oriented adder gadget ::::::::::::::::::::::: 17 2.2 Privilege execution modes ::::::::::::::::::::::::: 20 2.3 Microkernel and Monolithic Kernel Designs ::::::::::::::: 30 3.1 Run-Time Patch-Level Validation ::::::::::::::::::::: 40 3.2 Load-Time Optimized Patch-Level Validation :::::::::::::: 40 4.1 Record field order randomization. \R" represents a general-purpose pro- cessor register :::::::::::::::::::::::::::::::: 56 4.2 Subroutine argument order randomization. \SP" represents the stack pointer register. \BP" represents the stack base pointer register. :::: 61 4.3 AST-based randomization ::::::::::::::::::::::::: 63 4.4 Record size variation :::::::::::::::::::::::::::: 64 4.5 Variable initializers ::::::::::::::::::::::::::::: 65 4.6 AST of a cast from one pointer type to another ::::::::::::: 67 5.1 Reactivation of functions over time (log scale) :::::::::::::: 88 5.2 KIS keeps it simple ::::::::::::::::::::::::::::: 89 ix ABSTRACT Stanley, Dannie M. Ph.D., Purdue University, December 2013. Improved Kernel Secu- rity Through Code Validation, Diversification, and Minimization. Major Professors: Eugene H. Spafford and Dongyan Xu. The vast majority of hosts on the Internet, including mobile clients, are running one of three commodity, general-purpose operating system families. In such operating systems the kernel software executes at the highest processor privilege level. If an adversary is able to hijack the kernel software then by extension he has full control of the system. This control includes the ability to disable protection mechanisms and hide evidence of compromise. The lack of diversity in commodity, general-purpose operating systems enables attackers to craft a single kernel exploit that has the potential to infect millions of hosts. If enough variants of the vulnerable software exist, then mass exploitation is much more difficult to achieve. We introduce novel kernel diversification techniques to improve kernel security. Many modern kernels