FILE SYSTEM ISOLATION FOR UNTRUSTED APPLICATIONS by Fareha Shafique A thesis submitted in conformity with the requirements for the degree of Master of Applied Science Graduate Department of Electrical and Computer Engineering University of Toronto Copyright c 2008 by Fareha Shafique Abstract File System Isolation for Untrusted Applications Fareha Shafique Master of Applied Science Graduate Department of Electrical and Computer Engineering University of Toronto 2008 When computer systems are compromised by an attack, it is difficult to determine the precise extent of the damage because the state changes made by an attacker and those made by regular users can be closely intertwined. In particular, the file system provides a single namespace that, when compromised, can have cascading effects on the entire system, making intrusion analysis and recovery a difficult process. This thesis proposes using a copy-on-write file system, called Isolation File System (IFS), to provide a transparent, restricted privilege sandboxing environment for running untrusted applications. The goal of IFS is to limit propagation of attacks by confining each application to its own complete namespace. If a sandboxed application is exploited, a coarse-grained recovery method allows completely removing the footprint of the software. Our approach supports existing applications by providing mechanisms for explicit sharing across IFS environments. Our evaluation shows that IFS is fairly easy to configure. ii Acknowledgements I would first like to thank Dr. Ashvin Goel for his thoughtful guidance, continuous encourage- ment and financial support. Many thanks to Shvetank Jain and Vladan Djeric who worked with me on different parts of this project and who made my experience at graduate school very en- joyable. I am grateful to the University of Toronto as well as the Department of Electrical and Computer Engineering for their financial support. I would also like to thank Dr. Angela Demke Brown, Dr. David Lie and Dr. Shahrokh Valaee for accepting to be on my defense committee. Finally, I would like to thank my husband for supporting me and having confidence in me at all times. iii Contents 1 Introduction 1 1.1 ResearchApproach ................................ 2 1.2 Contributions ................................... 4 1.3 ThesisStructure................................. 5 2 Related Work 6 2.1 SandboxingandVirtualizationTechniques . ......... 7 2.2 AccessControl .................................. 9 2.3 Capability Systems and Restricted Privilege Systems . ............ 11 2.4 FileSystems.................................... 14 3 An Overview of the Solitude System 16 3.1 IFSIsolationEnvironment . ... 16 3.2 SharingPolicies................................. 17 3.3 TaintPropagationandRecovery . .... 17 3.4 UsageModel ................................... 18 4 Isolation File System 20 4.1 Copy-on-WriteFileSystem. ... 21 4.2 ChrootIsolationModel . .. 22 4.3 CapabilityModel ................................. 23 iv 4.4 StructureofPolicyFiles . ... 25 4.5 IPCDiscussion .................................. 26 5 ImplementationofIFSIsolationEnvironment 28 5.1 ImplementationofCopy-on-WriteFileSystem . ........ 28 5.2 ImplementationofChrootIsolationModel . ........ 34 5.3 ImplementationofCapabilityModel . ...... 35 5.4 SpecifyingCapabilities . .... 36 6 Evaluation 38 6.1 PolicyFiles .................................... 38 6.1.1 Web Server: Apache2 and Apache2 + Gallery . ... 39 6.1.2 MTAandMDA:PostfixandProcmail . 41 6.1.3 IMAPServer:Dovecot. 41 6.1.4 FTPServer:vsftpd ............................ 43 6.1.5 DHCPServer:dhcpd3 .. .. .. .. .. .. .. 43 6.1.6 PrinterServer:Cupsd. 44 6.1.7 SVNServer:Svnserve .. .. .. .. .. .. .. 44 6.1.8 Discussion ................................ 45 6.2 PerformanceOverhead . .. .. .. .. .. .. .. .. 46 7 Conclusion 48 7.1 FutureWork.................................... 49 v List of Tables 3.1 IFSsharingmodes ................................ 18 4.1 IFScapabilitymodel .............................. 24 5.1 SummaryofIFSdatabases . 33 vi List of Figures 3.1 TheSolitudearchitecture . .... 16 4.1 ExamplepolicyforApache2webserver . ..... 25 5.1 IFSCopy-on-Write ................................ 30 6.1 PolicyfortheApache2webserver . .... 39 6.2 PolicyforGalleryrunningonApache2. ...... 40 6.3 PolicyforthePostfixMTA . 40 6.4 PolicyforDovecotIMAPserver . ... 42 6.5 PolicyforvsftpdFTPserver . ... 42 6.6 Policyfordhcpd3DHCPserver . .. 43 6.7 Policyforcupsdprinterserver . ..... 44 6.8 PolicyforsvnserveSVNserver. .... 44 6.9 PerformanceOverheadofIFS . .. 47 vii Chapter 1 Introduction Several research efforts in recent years have focused on analysis and recovery of compromised systems [39, 25, 32]. This problem is both very real and hard: once a system is compromised, it is incredibly difficult to untangle the state changes made by an attacker, for instance the replacement of system binaries, from those made by normal users or administrators. While attempting recovery, an administrator is generally left with the choice of either confidently removing all attacker modifications or preserving all valid user activity, but not both. Implicit sharing that exists in modern operating systems is a major cause of this problem. For example, all users and processes share a single common namespace. Compromises that manage to make unauthorized updates to this namespace, for instance by replacing the com- monly used UNIX ps command, can have cascading effects across the entire system. While operating systems provide separate address spaces to protect physical memory, comparable protection is limited for persistent state. This implicit file-system sharing problem is exacerbated as users increasingly download and install software from untrusted sources on the Internet. Users are faced with the choice of either not downloading and running the application, or they risk compromising the integrity and the stability of the system. For instance, a downloaded media player application can have serious vulnerabilities that can allow attackers to attach malicious code and infect computers 1 CHAPTER 1. INTRODUCTION 2 without the user’s knowledge. Additionally, audio and video streams and downloads can be used to hijack or corrupt computers [76]. This thesis focuses on confining untrusted applications into separate isolation environ- ments. Within each environment, an application is bound to its own complete file-system namespace, similar to process address spaces, via a copy-on-write file system that we have de- signed, called Isolation File System (IFS). The benefit of namespace isolation is that malicious changes made by one application cannot inadvertently effect the operation of other applica- tions. In our design, each untrusted application within IFS is run in a sandboxed environment with restricted privileges. As a result, even if the application is compromised, escaping the sandbox is difficult and any damage that can be done to the system is limited because all- powerful super-user capabilities are disallowed within IFS. IFS allows unshared persistent states to diverge freely across isolation environments under the presumption that file sharing across applications is rare. The challenge with such an en- vironment is that since applications cannot share data across the isolation environments, they may not work correctly. Therefore, sharing across environments must be supported, and IFS provides explicit sharing mechanisms for this purpose. However, the detailed description of these mechanisms and the specific sharing policies needed to support IFS applications is the focus of another thesis [36]. 1.1 Research Approach The goal of IFS is to limit the propagation of attacks by running each untrusted application in a separate sandbox that confines it to its own complete file system. IFS offers a transparent view into the base (or regular) file system for reading operations, but any modifications made by the untrusted process or its children processes are confined to the separate namespace. If at any point the user decides that the software is malicious or undesirable, the entire IFS environment can be discarded without concern for the integrity of the base file system. CHAPTER 1. INTRODUCTION 3 A typical usage scenario of this system may involve running a peer-to-peer (P2P) client program within an IFS. For example, a user may download and install a photo editing applica- tion using the P2P client. The user knows that files on P2P networks are sometimes modified to include malicious components and thus installs the application in an IFS. This may be the same IFS as the P2P program or a new IFS, but in both cases, no changes are made to the base file system. If the application exhibits unexpected or suspicious behaviour, the user can remove the program and its changes by discarding its IFS. The basic IFS isolation model enhances chroot isolation with copy-on-write from the base file system. As with any isolation environment, there is a trade-off involved between the secu- rity provided by the IFS isolation environment, application-level functionality and ease-of-use. We rely on two mechanisms to resolve these concerns: support for restricted privileges for running server applications, and support for explicit file sharing policies. Running applications with restricted privileges provides increased security while supporting full application func- tionality. Similarly explicit file sharing, although not as secure as complete isolation, allows us to fully support existing applications that are based on the current access control model