The Pennsylvania State University The Graduate School PROTECTING PROGRAMS DURING RESOURCE ACCESS A Dissertation in Computer Science and Engineering by Hayawardh Vijayakumar c 2014 Hayawardh Vijayakumar Submitted in Partial Fulfillment of the Requirements for the Degree of Doctor of Philosophy May 2014 The dissertation of Hayawardh Vijayakumar was reviewed and approved∗ by the following: Trent R. Jaeger Professor of Computer Science and Engineering Dissertation Advisor, Co-Chair of Committee Patrick D. McDaniel Professor of Computer Science and Engineering Co-Chair of Committee Bhuvan Urgaonkar Associate Professor of Computer Science and Engineering Constantino Lagoa Professor of Electrical Engineering Raj Acharya Professor of Computer Science and Engineering Head of the Department, Computer Science and Engineering ∗Signatures are on file in the Graduate School. Abstract With the emergence of targeted malware such as Stuxnet and the continued prevalence of spyware and other types of malicious software, host security has become a critical issue. Attackers break into systems through vulnerabilities in network daemons, malicious insiders, or social engineering, and then attempt to escalate privileges to the administrator to gain complete control of the system by exploiting local vulnerabilities. Thus far, such local vulnerabilities have received little attention, and it has been taken for granted that any attacker break-in can easily be escalated to full control. In this dissertation, we identify a class of previously disjoint local vulnerability attack classes that we call resource access attacks, and provide a framework to detect and defend against them. Programs have to fetch resources, such as files from the operating system (OS) to function. However, local adversaries such as spyware also share this namespace of resources, and can trick programs into retrieving an unintended resource using a variety of resource access attacks that make up 10-15% of vulnerabilities reported each year. Such attacks are challenging to defend for a few reasons. First, program checks to defend against such attacks cause a performance overhead, so programmers have an incentive to omit checks altogether. Second, there is a disconnect between the parties involved in resource access. On the one hand, due to this overhead, programmers omit checks under the expectation that the deployment’s access control policy will protect a subset of resources from adversaries. On the other hand, access control policies are framed by OS distributors and system administrators, who in turn have little idea about programmer expectations, causing mismatches with programmer expectations. Third, even when programmers check resource access, such checks are difficult to get right due to inherent races in the system call API. Previous work handles a subset of resource access attacks but in ad-hoc ways. This dissertation takes several steps to address resource access attacks. First, we present a technique for automated evaluation of a program attack surface in its system deployment, where checks for resource access are required. Second, we present a technique that uses this attack surface to detect a subset of resource access attacks. We found more than 25 previously-unknown vulnerabilities across a variety of both mature and new programs in the widely-used Fedora and Ubuntu Linux distributions, proving the prevalence of such vulnerabilities. Third, we present the Process Firewall, a system to defend against resource access attacks in an efficient manner without requiring program code change. Fourth, we propose a technique to automatically derive the programmer-expected attack surface of a program, and generate Process Firewall rules to enforce that the only adversary-controlled resource accesses in the deployment are part of the iii expected attack surface. The work in this dissertation thus provides a principled starting point to protect programs during resource access, thus reducing the vectors adversaries have to compromise a computer system. iv Contents List of Figures ix List of Tables xi Acknowledgments xiii Chapter 1 Introduction 1 1.1 The Challenge of Defending Resource Access Attacks ................. 2 1.2 Towards a Defense to Resource Access Attacks .................... 3 1.3 Challenges in Defense .................................. 4 1.4 Thesis Statement ..................................... 5 Chapter 2 Background 8 2.1 System Calls and Accessing Resources ......................... 8 2.2 Access Control on Resources ............................... 10 2.2.1 UNIX Discretionary Access Control ...................... 10 2.2.2 Mandatory Access Control ............................ 10 2.2.3 Obtaining Privilege ................................ 10 2.3 Resource Access Attacks ................................. 11 2.3.1 Untrusted Search Path .............................. 12 2.3.2 File/IPC Squat .................................. 13 2.3.3 Directory Traversal and PHP File Inclusion . 14 2.3.4 Link Following and TOCTTOU races ..................... 14 2.3.5 Signal Races ................................... 15 Chapter 3 Related Work 17 3.1 System Defenses ..................................... 17 3.2 Program Defenses .................................... 18 3.3 Detection of Program Vulnerabilities .......................... 19 v Chapter 4 Integrity Walls: Finding Attack Surfaces from Mandatory Access Control Policies 20 4.1 Introduction ........................................ 20 4.2 Problem Definition .................................... 22 4.3 Design ........................................... 24 4.3.1 Building Integrity Walls ............................. 24 4.3.2 Identifying Attack Surfaces ........................... 26 4.4 Implementation ...................................... 28 4.4.1 Integrity Wall Construction ........................... 28 4.4.2 Identifying Attack Surfaces ........................... 29 4.4.3 Finding Attack Surfaces in Interpreted Code . 31 4.4.4 Enforcing Attack Surfaces ............................ 32 4.5 Evaluation ......................................... 32 4.5.1 Policy Analysis .................................. 33 4.5.2 Runtime Analysis ................................. 34 4.5.2.1 System TCB .............................. 34 4.5.2.2 Apache Webserver ........................... 35 4.5.2.3 Secure Shell Daemon ......................... 35 4.5.2.4 Icecat .................................. 38 4.6 Related Work ....................................... 38 4.7 Conclusion ........................................ 39 Chapter 5 STING: Finding Name Resolution Vulnerabilities in Programs 40 5.1 Introduction ........................................ 40 5.2 Problem Definition .................................... 43 5.2.1 Name Resolution Attacks ............................ 43 5.2.2 Detecting Name Resolution Attacks ...................... 45 5.2.2.0.1 Static Analysis ........................ 45 5.2.2.0.2 Dynamic Analysis ...................... 45 5.2.2.0.3 Symbolic Execution ..................... 46 5.2.3 Our Solution ................................... 46 5.3 Testing Model ....................................... 47 5.4 Design ........................................... 48 5.4.1 Attack Phase ................................... 49 5.4.2 Detect Phase ................................... 52 5.5 Implementation ...................................... 54 5.6 Evaluation ......................................... 55 5.6.1 Security Evaluation ............................... 55 5.6.1.1 Finding Vulnerabilities ........................ 56 5.6.1.2 Examples ................................ 58 vi 5.6.1.3 False Positives ............................. 60 5.6.2 Performance Evaluation ............................. 60 5.7 Related Work ....................................... 62 5.8 Discussion ......................................... 63 5.9 Conclusion ........................................ 64 Chapter 6 Process Firewalls: Protecting Processes During Resource Access 65 6.1 Introduction ........................................ 65 6.2 Problem Definition .................................... 68 6.2.1 Challenges in Preventing Resource Access Attacks . 69 6.2.2 Limitations of Prior Defenses .......................... 70 6.3 Solution Overview .................................... 72 6.4 Design ........................................... 73 6.4.1 Defining Attack-Specific Invariants ....................... 74 6.4.2 Checking Invariants ............................... 75 6.4.3 Finding Applicable Invariants .......................... 75 6.4.4 Retrieving Entrypoint Context ......................... 76 6.5 Implementation ...................................... 77 6.5.1 Process Firewall Rule Processing Loop ..................... 77 6.5.2 Process Firewall Rule Language ......................... 79 6.6 Evaluation ......................................... 80 6.6.1 Security Evaluation ............................... 80 6.6.1.1 Common Vulnerabilities ........................ 80 6.6.1.2 New Vulnerabilities Found ...................... 82 6.6.2 Process Firewall Performance .......................... 83 6.6.3 Rule Generation ................................. 84 6.6.3.1 Rule Generation Techniques ..................... 85 6.6.3.2 Rule Generation by OS Distributors . 87 6.7 Related Work ....................................... 88 6.8 Conclusion ........................................ 89 Chapter 7 Protecting Resource Access by Inferring Programmer Expectations 90 7.1 Introduction ........................................ 90 7.2