UCAM-CL-TR-818 Technical Report ISSN 1476-2986 Number 818 Computer Laboratory New approaches to operating system security extensibility Robert N. M. Watson April 2012 15 JJ Thomson Avenue Cambridge CB3 0FD United Kingdom phone +44 1223 763500 http://www.cl.cam.ac.uk/ c 2012 Robert N. M. Watson This technical report is based on a dissertation submitted October 2010 by the author for the degree of Doctor of Philosophy to the University of Cambridge, Wolfson College. Technical reports published by the University of Cambridge Computer Laboratory are freely available via the Internet: http://www.cl.cam.ac.uk/techreports/ ISSN 1476-2986 Summary This dissertation proposes new approaches to commodity computer operating system (OS) access control extensibility that address historic problems with concurrency and technology transfer. Access control extensibility addresses a lack of consensus on oper- ating system policy model at a time when security requirements are in flux: OS vendors, anti-virus companies, firewall manufacturers, smart phone developers, and application writers require new tools to express policies tailored to their needs. By proposing prin- cipled approaches to access control extensibility, this work allows OS security to be \designed in" yet remain flexible in the face of diverse and changing requirements. I begin by analysing system call interposition, a popular extension technology used in security research and products, and reveal fundamental and readily exploited con- currency vulnerabilities. Motivated by these failures, I propose two security extension models: the TrustedBSD Mandatory Access Control (MAC) Framework, a flexible ker- nel access control extension framework for the FreeBSD kernel, and Capsicum, practical capabilities for UNIX. The MAC Framework, a research project I began before starting my PhD, allows policy modules to dynamically extend the kernel access control policy. The framework allows policies to integrate tightly with kernel synchronisation, avoiding race condi- tions inherent to system call interposition, as well as offering reduced development and technology transfer costs for new security policies. Over two chapters, I explore the framework itself, and its transfer to and use in several products: the open source Free- BSD operating system, nCircle's enforcement appliances, and Apple's Mac OS X and iOS operating systems. Capsicum is a new application-centric capability security model extending POSIX. Capsicum targets application writers rather than system designers, reflecting a trend towards security-aware applications such as Google's Chromium web browser, that map distributed security policies into often inadequate local primitives. I compare Capsicum with other sandboxing techniques, demonstrating improved performance, programmability, and security. This dissertation makes original contributions to challenging research problems in security and operating system design. Portions of this research have already had a significant impact on industry practice. 3 4 Acknowledgements Writing this dissertation would not have been possible without the support and en- couragement of my family (especially my parents), friends, mentors, and colleagues, to whom I offer my sincerest thanks and appreciation. Ross Anderson, my supervisor, deserves a special note of thanks: he has been supportive throughout my less than typical path through Cambridge's PhD programme, giving me space to pursue a variety of interests, many related to my PhD research, while still shepherding the dissertation to a timely completion! The security research group at the Computer Laboratory has been a critical source of experience and collaboration { especially Jonathan Anderson, Steven Murdoch, and Richard Clayton. Jon's work on Capsicum deserves particular note { the project could not have been completed without his help. Early portions of my research into kernel access control extensibility, including the MAC Framework, were supported by DARPA and the US Navy via research con- tracts N66001-01-C-8035 (CBOSS1) and N66001-04-C-6019 (SEFOS2). This research took place at first TIS, then NAI Labs, later McAfee Research, and finally SPARTA ISSO. My colleagues at those institutions played central roles in helping me to take the idea of an extensible access control framework from concept, to prototype, to widely used system; many of them are acknowledged on page 11. However, I wish especially to thank Russ Mundy and Sandra Murphy, who mentored me through their own re- search projects in the TIS NetSec group. Special thanks are also due to Lee Badger, who at NAI helped me to formulate my first DARPA proposal around the concept of an extensible access control framework, and later at DARPA sponsored my seedling research project that became the inspiration for Capsicum. Industrial collaboration has also played a central role in my PhD research: Google, Apple, nCircle Network Security, and Intel have engaged in elements of my research into access control extensibility and technology transfer. Google's university research grant programme supported my PhD research; Ben Laurie has not only promoted my research at Google, but also been an active collaborator in Capsicum. My thanks also go to Simon Cooper and Richard Gaushell for reviewing drafts of this dissertation. Members of the global computer security research community have provided feed- back on my research, including the anonymous reviewers of my papers. Most notably, Peter Neumann has provided both encouragement and detailed feedback on this dis- sertation. Comments and suggestions from my examiners, Jon Crowcroft (Cambridge) and Mark Handley (UCL), have made the dissertation much stronger, and are greatly appreciated. I have drawn heavily on the open source community, whose products are remarkable assets for research that I have been proud to contribute to. The FreeBSD Project 1Community-Based Open Source Security 2Security Extensibility and Flexibility in Operating Systems 5 has provided seemingly endless assistance; all its contributors deserve my thanks, but especially John Baldwin, Pawel Dawidek, Poul-Henning Kamp, Kris Kennaway, Sam Leffler, George Neville-Neil, Colin Percival, and Bjoern Zeeb for their willingness to support and engage with my research over the last decade. Dedication I dedicate this dissertation to my wife and partner, Leigh Denault { I only hope that I supported you during your PhD as well as you have supported me! 6 Contents 1 Introduction 13 1.1 Context for this research . 14 1.2 What is an operating system? . 15 1.3 Principles of operating system security . 17 1.3.1 Kernel and processes . 18 1.3.2 From isolation to access control policy . 19 1.3.3 Virtualisation . 21 1.3.4 Trusted systems . 22 1.3.5 Capability systems . 23 1.3.6 Of microkernels and security kernels . 24 1.3.7 Language and runtime approaches . 26 1.3.8 Extensible access control frameworks . 28 1.4 Structure of this dissertation . 30 2 Concurrency vulnerabilities in system call interposition 33 2.1 Operating system kernels and concurrency . 34 2.2 Wrappers for security . 34 2.3 Concurrency attacks on wrappers . 35 2.4 Exploit techniques . 36 2.4.1 Concurrency approaches . 37 2.4.2 Racing on uniprocessor systems . 37 2.4.3 Racing on multiprocessor systems . 38 2.5 Exercising real vulnerabilities . 38 2.5.1 Generic Software Wrapper Toolkit (GSWTK) . 38 2.5.2 Systrace . 39 2.5.3 CerbNG . 41 2.6 Preventing wrapper races? . 42 2.6.1 Mitigation techniques . 42 2.6.2 Message passing systems . 43 2.6.3 Integrating security and concurrency . 43 2.7 Impact of the WOOT07 paper . 44 2.8 Conclusion . 45 3 The MAC Framework: extensible kernel access control 47 3.1 History of the MAC Framework . 49 3.2 Past approaches . 52 3.2.1 Direct modification . 52 3.2.2 System call interposition . 52 3.2.3 Stacked file systems . 53 3.3 Limitations of past approaches . 53 3.3.1 Kernel source code access . 53 3.3.2 Tracking vendor development . 54 3.3.3 Concurrency and lock order in threaded kernels . 54 3.3.4 Policy composition . 55 3.3.5 Financial cost of implementation . 56 3.4 Designing for access control extension . 57 3.4.1 Guiding principles . 58 3.5 Architecture of the MAC Framework . 60 3.5.1 Framework startup . 61 3.5.2 Policy registration . 62 3.5.3 Entry point design considerations . 63 3.5.4 Kernel service entry point invocation . 65 3.5.5 Policy entry point invocation . 66 3.5.6 Policy composition . 68 3.5.7 Object labelling . 69 3.5.8 Application-layer approach . 74 3.5.9 Policy-agnostic label management APIs . 74 3.6 MAC Framework policy modules . 75 3.6.1 The Biba integrity policy . 75 3.7 Performance evaluation . 79 3.7.1 System call performance . 80 3.7.2 Network performance . 86 3.7.3 Kernel build performance . 88 3.8 Related work . 89 3.9 Conclusion . 89 4 The MAC Framework: from research to product 91 4.1 FreeBSD operating system . 92 4.1.1 Experimental feature status . 93 4.1.2 Performance . 94 4.1.3 Third-party contributions to the MAC Framework . 98 4.1.4 Additional MAC Framework consumers . 100 4.2 nCircle IP360 monitoring appliance . 101 4.2.1 What are system privileges? . 102 8 4.2.2 System privilege extensions to the MAC Framework . 102 4.2.3 The nCircle MAC policy . 105 4.3 Apple's Mac OS X and iOS . 106 4.3.1 SEDarwin research prototype . 107 4.3.2 Adapting the MAC Framework to Mac OS X . 108 4.3.3 Adoption by Apple . 111 4.3.4 The Sandbox access control policy . 112 4.3.5 Applications constrained by Sandbox . 114 4.3.6 Enforcement in Mach and BSD . 116 4.3.7 Paths in policy expression . 116 4.3.8 Considerations for iOS . 117 4.3.9 Performance optimisations . 117 4.3.10 Policy label data synchronisation requirements . 118 4.3.11 Conclusions on Mac OS X and iOS .