A Theory and Tools for Applying Sandboxes Effectively
Total Page:16
File Type:pdf, Size:1020Kb
A Theory and Tools for Applying Sandboxes Effectively Michael Maass CMU-ISR-16-105 March 2016 Institute for Software Research School of Computer Science Carnegie Mellon University Pittsburgh, PA 15213 Thesis Committee: Jonathan Aldrich (Co-chair) William L. Scherlis (Co-chair) Lujo Bauer Bruno Amizic (The Boeing Company) Submitted in partial fulfillment of the requirements for the degree of Doctor of Philosophy. Copyright c 2016 Michael Maass This material is based upon work supported by the US Department of Defense through the Office of the Assistant Secretary of Defense for Research and Engineering (ASD(R&E)) under Contract HQ0034-13-D- 0004, the Army Research Office under Award No. W911NF-09-1-0273, the Air Force Research Laboratory under Award No. FA87501220139, the National Security Agency under Lablet Contract H98230-14-C- 0140, the National Science Foundation Graduate Research Fellowship Program under Grant No. DGE- 1252522. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of ARL, ARO, ASD (R&E), NSA, or NSF. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript. Keywords: Sandboxing, software protection, supply chain security, Java sandbox, software engineering Abstract It is more expensive and time consuming to build modern software without extensive supply chains. Supply chains decrease these development risks, but typically at the cost of increased security risk. In particular, it is often difficult to understand or verify what a software component delivered by a third party does or could do. Such a component could contain unwanted behaviors, vulner- abilities, or malicious code, many of which become incorporated in applications utilizing the component. Sandboxes provide relief by encapsulating a component and imposing a se- curity policy on it. This limits the operations the component can perform without as much need to trust or verify the component. Instead, a component user must trust or verify the relatively simple sandbox. Given this appealing prospect, researchers have spent the last few decades developing new sandbox- ing techniques and sandboxes. However, while sandboxes have been adopted in practice, they are not as pervasive as they could be. Why are sandboxes not achieving ubiquity at the same rate as extensive supply chains? This thesis ad- vances our understanding of and overcomes some barriers to sandbox adoption. We systematically analyze ten years (2004 { 2014) of sandboxing research from top-tier security and systems conferences. We uncover two barriers: (1) sandboxes are often validated using relatively subjective techniques and (2) usability for sandbox deployers is often ignored by the studied community. We then focus on the Java sandbox to empirically study its use within the open source community. We find features in the sandbox that benign applications do not use, which have promoted a thriving exploit landscape. We develop run time monitors for the Java Virtual Machine (JVM) to turn off these features, stopping all known sandbox escaping JVM exploits without breaking benign applications. Furthermore, we find that the sandbox contains a high degree of complexity benign applications need that hampers sandbox use. When studying the sandbox's use, we did not find a single application that successfully deployed the sandbox for security purposes, which motivated us to overcome benignly-used complexity via tooling. We develop and evaluate a series of tools to automate the most complex tasks, which currently require error-prone manual effort. Our tools help users derive, express, and refine a se- curity policy and impose it on targeted Java application JARs and classes. This tooling is evaluated through case studies with industrial collaborators where we sandbox components that were previously difficult to sandbox securely. Finally, we observe that design and implementation complexity causes sand- box developers to accidentally create vulnerable sandboxes. Thus, we develop and evaluate a sandboxing technique that leverages existing cloud computing environments to execute untrusted computations. Malicious outcomes pro- duced by the computations are contained by ephemeral virtual machines. We describe a field trial using this technique with Adobe Reader and compare the new sandbox to existing sandboxes using a qualitative framework we developed. iv For my father, whose early hacking efforts brought an indelible passion for technology into our family. vi Acknowledgments It is impossible to acknowledge all of the people that helped me find my way into and through graduate school. Growing up, I never dreamed of, or even really considered, attending graduate school, primarily due to a lack of role models. I decided to attend a university after high school thanks to the encouragement of two science and math teachers: Mary Davis and Jennifer Anderson. While studying Electrical Engineering I figured I would graduate, get a software security job somewhere, and carve out a career for myself entirely in industry. However, my views changed when I was invited to lunch with one of my professors, Sandip Roy, who encouraged me to consider graduate school and gave me a peek at what it is all about. After I graduated I got a software security job at Boeing, where I met and later worked for a leader that encouraged me to finally apply for admission to a PhD program: Ron Gabel. Ron gave me the opportunity to travel to universities and conferences to establish research collaborations, and it was through these engagements that I decided I wanted to pursue a PhD at Carnegie Mellon. Shortly before I left for Carnegie Mellon, a PhD Candidate named Bruno Amizic joined our team and offered his wisdom regarding keeping a foot in both academia and industry. Ron and Bruno have been my regular companions and supporters on this journey. Carnegie Mellon is bursting at the seams with influential people, many of whom have had an impact on me and my time here. Bill Scherlis and Jonathan Aldrich have supervised and guided my research progress throughout the entire process. They have both given me unique opportunities outside of typical research experiences to make connections, learn how research is managed, and generally understand how the sausage is made. I can never repay the trust they have placed in me. Josh Sunshine has acted as a third advisor; I cannot emphasize enough the impact he has had on the research in this thesis. Josh has a keen eye for both message and methodology that has improved my work well beyond what I could have achieved without his guidance. I am also thankful for my co-authors, without whom many of the chapters in this thesis would be worse off: 1. Chapter 2: Adam Sales, Benjamin Chung, and Joshua Sunshine. Pub- lished in \PeerJ": https://peerj.com/articles/cs-43/. 2. Chapter 3: Zach Coker, Tianyuan Ding, Claire Le Goues, and Joshua Sunshine. Published in \Annual Computer Security Applications Confer- ence" 2015. 3. Chapter 4: Jonathan Aldrich, William Scherlis, and Joshua Sunshine 4. Chapter 5: William Scherlis and Jonathan Aldrich. Published in \Sym- posium and Bootcamp on the Science of Security" 2014. Many others have motivated, inspired, and educated me including office mates I have shared long discussions with on many topics: Ben Towne and Gabriel Ferreira in particular. I give extra thanks to Ben and Vishal Dwivedi, both of whom have patiently reviewed several drafts of this document to im- prove my clarity and reduce the quantity of mistakes. Mary Shaw taught me foundational skills in research writing that I will always rely on. Charlie Gar- rod showed me what true passion in teaching is and what it can accomplish. Christian Kastner, Claire Le Goues (the best person at venting frustration I have ever met!), and Stephanie Balzer have all improved my communication skills through careful reviews of early paper drafts and practice talks. Others have pushed me to work harder while managing to also keep me social: Michael McCord, Sven Stork, Thomas LaToza, Alex Corn, Ivan Ruchkin, Moses James, Nathan Fulton, David Naylor, Ilari Shafer, Roykrong Sukkerd, Ashwini Rao, Jason Tsay, Mauricio Soto, Darya Kurilova, Blase Ur, Hanan Hibshi, Laurie Jones, Peter Landwehr, Kenny Joseph, Hemank Lamba, Alain Forget, Sudhi Wadkar, Vittorio Perera, and Daniel Smullen. I will not even try to list the many people I have met through service in Dec/5 that could easily make this list as well. I thank my family, particularly my father. My father developed a knack for tinkering with electronics as a child and never gave it up. I credit him with inspiring my passion for learning about and contributing to the advancement of technology. My brothers have brought me a constant sense of support, multiple nieces and nephews to watch grow up, and have taken on the burden of hosting me and my fianc´ee,Erin, for several holidays. My fianc´ee'sparents also deserve acknowledgment for hosting us on several visits and for supporting Erin while she finished law school and sought employment. Finally, I thank my fianc´ee,partner, and constant companion for most of graduate school. Erin, I love you and look forward to taking the many remaining steps in life with you. viii Contents 1 Introduction 1 1.1 The State of Practice: Loose Supply Chains . .1 1.2 Sandboxes: A Path Forward . .2 1.3 Sandboxes and Unfulfilled Promises . .4 1.3.1 Sandbox Complexity Causes Failures in Practice . .5 1.3.2 Practical Challenges are Mirrored in Sandbox Literature . .5 1.4 Thesis Approach: Systematic Sandbox Literature Analysis . .6 1.5 Thesis Approach: Tool Assistance for Applying Sandboxes . .7 1.6 Thesis Approach: Ephemeral Environments . .7 1.7 Contributions . .8 1.8 Thesis . .9 2 A Systematic Analysis of the Science of Sandboxing 11 2.1 What is a sandbox? .