AN ABSTRACT OF THE THESIS OF Cohn William Van Dyke for the degree of Master of Science in Electrical and Computer Engineering presented on May 6. 2004. Title: An In-Depth Analysis of Common Software Vulnerabilities and their Solutions Abstract Approved: Redacted for privacy cetin K.Koc With the current security climate throughout the Internet, a large emphasis is being placed on protecting software systems from attack by a malicious entity.Most successful attacks against software are the result of lazy or misinformed developers and ase easily prevented given the correct preparation and instruction. This document provides an in-depth introduction to the general methods and used by malicious parties to exploit existing software systems, and is intended be used as a guide for parties interested in preventing attacks against their software in the future. An In-Depth Analysis of Common Software Vulnerabilities and their Solutions by Cohn William Van Dyke A THESIS submitted to Oregon State University In partial fulfillment of the requirements for the degree of Master of Science Presented May 6, 2004 Commencement June 2004 Master of Science thesis of Cohn William Van Dyke presented on May 6. 2004. APPROVED: Redacted for privacy Major Professor, representing Electrical and Computer Engineering Redacted for privacy Director, School of El.óical Engineering and Computer Science Redacted for privacy Dean of the Qdfe School I understand that my thesis will become part of the permanent collection of Oregon State University libraries. My signature below authorizes release of my thesis to any reader upon request. Redacted for privacy LE5Tin)cvilliam Van Dyke, Author ACKNOWLEDGEMENTS The author would like to thank his parents, first and foremost. Without their love and support this would never have been possible.I would like to thank Gokay Saldamh and Onur Aciiçmez for their support throughout this process, as well as Dr. cetin Koç for giving me a chance and believing in me. Finally, I would like to express my sincere gratitude to Dr. Nina Berry of Sandia National Laboratories in Livermore, California for giving me the opportunity to research my very specific interests and her guidance in doing so. 'U TABLE OF CONTENTS Page 1 Introduction . 1 2 Formal Description of an Attack Against Software........................... 2 3 A Brief Review of User Privileges............................................. 3 4 A Brief Review of Program Execution and Memory........................... 5 5 Vulnerability and Solution Summaiy Matrix.................................... 9 6 Common Software Vulnerabilities .............................................13 6.1 The (Basic) Buffer Overflow......................................................13 6.2Advanced Buffer Overflows......................................................27 6.2.1Frame Pointer Overwrites......................................................27 6.2.2Non-Terminated Adjacent Memory ....................................34 6.3The Heap Overflow...............................................................41 6.4The Integer Overflow...............................................................54 6.5Format Strings........................................................................61 6.6External Commands...............................................................65 6.7User IDs ........................................................................68 6.8Environment Variables ......................................................69 6.9Race Conditions ...............................................................73 6.10 Ungraceful Failures...............................................................82 6.11 Cryptography........................................................................83 7 Secure Coding Practices ......................................................84 7.1Design Principles ...............................................................84 111 TABLE OF CONTENTS (Continued) 7.1.1 Least Privilege 84 7.1.2 Fail-Safe Defaults ......................................................84 7.1.3 Economy of Mechanism .............................................85 7.1.4 Complete Mediation......................................................85 7.1.5 Open Design...............................................................85 7.1.6 Separation of Privilege .............................................85 7.1.7 Least Common Mechanism.............................................86 7.1.8 Psychological Acceptability.............................................86 7.2Checklist for C ...............................................................86 8 Conclusion ........................................................................88 Bibliography.................................................................................89 iv LIST OF FIGURES Figure Page 1. Set-UID bit ..................................................................... 4 2. Process Memory Organization .......................................... 5 3. Initial Shellcode Disassembly Using GDB................................. 16 4. Initial Sheilcode Test............................................................21 5. Second Sheilcode Test ...................................................24 6. Third Sheilcode Test............................................................26 7. Framepointer Executable Disassembly Using GDB........................29 8. Stack Contents Prior to Framepointer Overwrite ........................30 9. Data Buffer Contents and Relative Addresses ........................30 10. GDB Disassembly of func()...................................................32 11. Console Outputofframepointei cTest .................................34 12. strncpy()Man Page Excerpt...................................................35 13. Stack Organization of Non-Terminated Buffers ........................ 35 14. Stack Contents Showing Non-Terminated Buffer ........................36 15. General Format of an Exploitable Buffer .................................38 16. Buffer Contents Afterstrncpy() .......................................... 39 17. Console Output of Basic Heap Exploit .................................42 18. Heap Structure Before Exploitation..........................................43 19. Heap Structure After buJ2 Exploitation .................................43 20. Heap Structure After Overrun ..........................................43 21. Pointer Overwrite Code Console Output..................................45 L!i LIST OF FIGURES (Continued) Figure Pa_ge 22. &tmpfile Overwrite Buffer Format...........................................48 23. Console Output for fpointer Exploit Using Stack-based Sheilcode 53 24. Console Output for fpointer Exploit Using Heap-based Sheilcode 54 25. Console Output for Test of expWidth.c ................................. 57 26. Stack Contents for Format String ..........................................64 27. Illustration of a Simple Attack Using Environment Variables 70 28. Functions for Environment Variable Manipulation........................71 29. More Complex Attack on Environment Variables........................72 30. Console Output of Race Conditions Exploit................................. 75 31. Console Output of exrc.c ...................................................75 32. Symbolic Link Creation ...................................................75 33. Termination of exrc.c ...................................................76 34. Effect of Attack ............................................................76 35. Sticky Bit Illustration............................................................79 VI LIST OF TABLES Table Page 1. Buffer Overrun Summary Matrix ..........................................9 2. Non-Buffer-Based Overrun Summary Matrix ........................ 10 3. OS-Based Exploit Summary Matrix.......................................... 11 4. General Exploit Summary Matrix .......................................... 12 5. Shelicode Instruction Substitutions..........................................20 6. Table of Vulnerable Formatting Functions................................. 61 7. Useful String Format Parameters ..........................................62 8. Table of Functions That Rely on a File Descriptor........................78 An In-Depth Analysis of Common Software Vulnerabilities and their Solutions IMPOR TANT NOTE: The material contained in this document is for educational purposes only. The author holds no responsibilily for any consequences of using this material in an illegal manner. 1 Introduction As society becomes more heavily bounded to computing and universal information via the Internet, and increasing concern is being placed on security and protection of data against malicious entities such as hackers. The increase in the number of people who have access to the Internet has also increased the number of attacks against computer systems. These attacks often target exploits in common software packages, and as the software industry grows so do the number of available exploits. As the number of these exploits increases to a critical level, major software vendors and development houses are initiating a new movement to make code-level security the top priority. For example, Microsoft has recently announced that security would be the number one priority in their development, and specific groups have been created to test and fix security vulnerabilities in their existing and future software. Any software included in the Linux Kernel undergoes