Enhancing Multi-user OS with Network Provenance for Systematic Malware Defense A Dissertation presented by Wai Kit Sze to The Graduate School in Partial Fulfillment of the Requirements for the Degree of Doctor of Philosophy in Computer Science Stony Brook University May 2016 Stony Brook University The Graduate School Wai Kit Sze We, the dissertation committee for the above candidate for the Doctor of Philosophy degree, hereby recommend acceptance of this dissertation Dr. R. Sekar - Dissertation Advisor Professor of Computer Science Dr. Donald Porter - Chairperson of Defense Assistant Professor of Computer Science Dr. Long Lu - Committee member of Defense Assistant Professor of Computer Science Dr. Trent Jaeger- Committee member of Defense Professor of Computer Science and Engineering Department, Pennsylvania State University This dissertation is accepted by the Graduate School Charles Taber Dean of the Graduate School ii Abstract of the Dissertation Enhancing Multi-user OS with Network Provenance for Systematic Malware Defense by Wai Kit Sze Doctor of Philosophy in Computer Science Stony Brook University 2016 Desktop OSes adopt users as the basic unit of trust. Every file and process owned by the same user has the same userid as the user. This design stems from the very first multi-user OS created in the late 1950s, where computers were self-contained and file contents were entirely under the control of users. With the spread of the Internet, it is common to have those code and data from the Internet. Users do not necessarily have the ability to fully understand every file they download. However, OSes do not distinguish between downloaded files from regular user files, and they simply reuse the same trust model and treat downloaded files as if users own files— OSes assume users understand the consequences of using these files. When malware exploits this design flaw, OSes simply warn users that using files from the Internet can compromise the systems and let the user to decide whether to use the files. In this dissertation, we propose Spif, which generalizes the OS trust-hierarchy with provenance information| Spif extends the trust-hierarchy with principals to encode both local users and remote provenance information. Principals in Spif can have different trust relationships. Spif is general and can model trust-hierarchy of systems like Same-Origin Policy, Android and Bubbles. With just two provenances having a unidirectional trust relationship, Spif can already provide usable integrity protection to defend against unknown, high-profile malware such as Stuxnet and Sandworm. We also demonstrate a secure software installation system based on Spif. Trust-hierarchy and access-controls are enforced deeply inside OSes. Generalizing trust-model in OSes can therefore affect every application and component inside OSes. Instead of building a brand new OS from scratch or instrumenting existing OSes to enforce a new trust-model directly, Spif achieves this generalization by re-purposing security-mechanisms common in contemporary OSes. This re-purposing automatically mediates every resource access, incurs low performance overhead, and is agnostic to both OSes and applications. Spif has been implemented on various OSes, including Linux, BSD, and Windows. Spif also runs large unmodified applications such as Firefox, Microsoft Office, Adobe Reader, and Photoshop. iii Table of Contents Contents 1 Introduction 2 2 Background and Related work5 2.1 Malware detection and avoidance.............................5 2.1.1 Anti-virus......................................5 2.1.2 Origin-based protection..............................6 2.1.3 Code-signing....................................7 2.2 Policy-based confinement.................................8 2.2.1 Drawbacks for policy-based confinement.....................8 2.2.2 Privilege separation................................ 10 2.3 Isolation-based approach.................................. 10 2.3.1 Drawbacks for applying isolation on desktop environment........... 11 2.3.2 Attempts to make isolation more usable for desktop environment....... 11 2.4 Information flow control.................................. 13 2.4.1 Usability problems with IFC policies....................... 13 2.4.2 Modern application of IFC............................ 14 2.4.3 Decentralized information flow control...................... 15 3 Portable Information Flow Tracking 16 3.1 Approach overview..................................... 16 3.1.1 Reliable tracking system.............................. 18 3.1.2 Robust policy enforcement............................ 18 3.1.3 Application transparency............................. 18 3.1.4 Preserving user experience............................. 18 3.1.5 Automating policy development......................... 19 3.1.6 Implementation on contemporary OSes..................... 19 3.2 Threat Model........................................ 19 3.3 Containing Untrusted Processes.............................. 20 3.3.1 Inner Sandbox UI ................................. 21 3.3.2 Transparency Library UL ............................. 22 3.3.3 Helper Process UH ................................. 23 3.3.4 Windows implementation............................. 24 3.4 Protecting Benign Processes................................ 24 3.4.1 Benign Sandboxing Library............................ 25 3.4.2 Secure Context Switching............................. 26 3.4.2.1 Transition on UNIX........................... 27 3.4.2.2 Transition on Windows......................... 27 3.5 Policy Inference....................................... 27 3.5.1 Untrusted Code Policy............................... 28 3.5.1.1 Explicitly specified versus implicit access to files........... 28 3.5.1.2 Computing Implicitly Accessed Files.................. 29 3.5.2 Benign Code Policy................................ 30 3.5.2.1 Logical isolation............................. 30 3.5.2.2 Untrusted execution........................... 30 3.5.2.3 Trust-confined execution........................ 31 3.5.3 Trial-execution based inference.......................... 31 iv 3.6 Security Guarantees.................................... 32 3.6.1 Integrity Preservation............................... 32 3.6.2 Availability Preservation.............................. 34 3.7 Implementation....................................... 35 3.7.1 Spif initialization................................. 35 3.7.2 UL and BL realization............................... 35 3.7.2.1 Policy enforcement mechanics...................... 35 3.7.2.2 Enforcement on system calls...................... 36 3.7.3 Initial file labeling................................. 37 3.7.4 Relabeling...................................... 38 3.7.5 Display server and DBus.............................. 39 3.7.6 File utilities..................................... 39 3.8 Evaluation.......................................... 40 3.8.1 Code complexity.................................. 40 3.8.2 Preserving Functionality of Code......................... 40 3.8.2.1 Benign mode............................... 40 3.8.2.2 Untrusted mode............................. 41 3.8.2.3 Overall.................................. 42 3.8.3 Usage experience.................................. 43 3.8.4 Experience with malicious software........................ 44 3.8.4.1 Real world malware on Ubuntu..................... 44 3.8.4.2 Real world exploit on Ubuntu...................... 44 3.8.4.3 Simulated targeted attacks on Ubuntu................. 44 3.8.5 Real world exploit on Windows.......................... 45 3.8.5.1 Real world malware on Windows.................... 46 3.8.6 Performance.................................... 47 3.8.6.1 Micro-benchmark............................ 47 3.8.6.2 Macro-benchmark............................ 47 3.9 Discussion.......................................... 49 3.9.1 Alternative choices for enforcement........................ 49 3.9.2 Limitations..................................... 50 3.9.3 Other architectural/implementation vulnerabilities............... 50 3.10 Related Work........................................ 51 4 Functionality and usability in policy enforcement mechanisms 53 4.1 Formalizing Functionality, Compatibility and Usability................. 54 4.1.1 Integrity policies.................................. 55 4.1.2 Comparing functionalities of integrity policies.................. 56 4.1.3 Compatibility.................................... 56 4.1.4 Maximizing functionality and compatibility................... 57 4.2 Self-Revocation Free Downgrading............................ 58 4.2.1 Abstractions.................................... 59 4.2.2 Information Flow Policies............................. 59 4.2.3 Forward information flows............................. 60 4.2.4 Constraint propagation.............................. 61 4.2.5 Properties...................................... 61 4.3 SRFD Implementation and Evaluation.......................... 62 4.3.1 Abstractions: Subjects, Objects, and Handles.................. 63 4.3.2 Constraint propagation.............................. 63 4.3.3 Tracking subjects.................................. 64 4.3.4 Limitations..................................... 64 v 4.3.5 Performance.................................... 64 4.3.6 User Experience.................................. 66 4.3.7 Defense against malware.............................. 67 4.4 User-level SRFD...................................... 68 4.4.1 Downgrading mechanism............................. 68 4.4.2 userid propagation and maintenance......................