MODEL-CHECKING IN-LINED REFERENCE MONITORS by Meera Sridhar APPROVED by SUPERVISORY COMMITTEE: Dr. Kevin W. Hamlen, Chair Dr. Go
Total Page：16
File Type：pdf, Size：1020Kb
MODEL-CHECKING IN-LINED REFERENCE MONITORS by Meera Sridhar APPROVED BY SUPERVISORY COMMITTEE: Dr. Kevin W. Hamlen, Chair Dr. Gopal Gupta Dr. Bhavani Thuraisingham Dr. I-Ling Yen Copyright © 2014 Meera Sridhar All rights reserved Om Saraswathi Namasthubyam Vardey Kaamarupini Vidhyarambham Karishyami Siddhir Bhavathu Mey Sada MODEL-CHECKING IN-LINED REFERENCE MONITORS by MEERA SRIDHAR, BS, MS DISSERTATION Presented to the Faculty of The University of Texas at Dallas in Partial Fulfillment of the Requirements for the Degree of DOCTOR OF PHILOSOPHY IN COMPUTER SCIENCE THE UNIVERSITY OF TEXAS AT DALLAS August 2014 ACKNOWLEDGMENTS The author expresses her sincere gratitude for being given the opportunity to conduct her doctoral studies under the supervision of her advisor, Dr. Kevin Hamlen. The author is deeply obliged to Dr. Kevin Hamlen for providing superior technical guidance throughout her studies, for being a constant source of inspiration, for setting a fine standard for meticulous science and technical writing, and for emphasizing the importance of conducting research that is backed by a combination of sound theory and strong practical applicability. The author is further indebted to him for providing her tremendous freedom in the pursuit of research, and for providing financial support for innumerable research endeavors, both of which she will always cherish. The author also thanks Dr. Hamlen for his indispensable contributions to this dissertation. The author is obliged to her exemplary educators throughout her schooling for instilling in her a deep passion for mathematics and computer science, and producing the academi- cian she is today. She would especially like to thank Mr. Horkan, Dr. Harry Bennett, Mr. William Marzen, Dr. Helmut Vogel, Dr. Tom Bohman, Dr. Peter Lee, Dr. Edmund Clarke, Dr. Michael Erdmann, Dr. Ramaswamy Chandrasekaran, and Dr. Neeraj Mittal. Special thanks go to Dr. Jeannette Wing at Carnegie Mellon University for introducing the author to computer science research and guiding her through undergraduate and Master's theses in model-checking. The author is extremely grateful to her doctoral committee members, Dr. Gopal Gupta, Dr. Bhavani Thuraisingham, and Dr. I-Ling Yen, for their time, effort, and invaluable guid- ance throughout the writing of this dissertation. The author also thanks her colleagues and friends Dr. Feliks Kluzniak, Dr. Venkat Venkatakrishnan at University of Illinois, Chicago, v and Dr. Matthew Might at the University of Utah for numerous discussions and immensely valuable mentorship. The author is indebted to her co-authors Brian DeVries, Scott Moore, Dr. Gopal Gupta, Dr. Micah Jones, Dr. Richard Wartell, Dr. Phu Phung, Maliheh Monshizadeh, Dr. Venkat Venkatakrishnan, and Dhiraj Karamchandani for their vital contributions to this disserta- tion. The author would also like to thank Peleus Uhley at Adobe, Inc. for several beneficial discussions, grant support letters, and for providing real-world SWF files of interest for testing and certification. The research reported in this dissertation was supported in part by the U.S. Air Force Office of Scientific Research (AFOSR) under grant #FA9550-08-1-0044, the National Science Foun- dation (NSF) under grants #1065216 and #1054629, and by the Office of Naval Research (ONR) under grant #N00014-14-1-0030. Any opinions, findings, conclusions, or recommen- dations expressed are those of the authors and do not necessarily reflect the views of the AFOSR, NSF or ONR. Staff at the Computer Science Department at The University of Texas at Dallas, especially Stacy Morrison and Rhonda Walls, deserve much appreciation for helping with countless administrative tasks throughout. The author's doctoral studies would not have been possible without the priceless compan- ionship of her friends. She is extremely indebted to her long-time friends Dr. Mehmet Tozal, Dr. Sunitha Ramanujam, Dolapo Kukoyi, Neslihan Deniz, Chinyelu Mozie, Shalini Patras, Anna Schlimme, and Charlese Galbraith, who celebrated with her during joyful moments, and helped her in times of need. Special token of appreciation goes to friends Dr. Yan Zhou, Jared Butler, Dr. Aravind Natarajan, Dr. Mustafa Canim, James Garrity, Jian Huang and Binal Kamani for providing assistance during key moments. vi The author will be eternally grateful to her family's support for making this dissertation a possibility. The author is much appreciative of her younger brother, Narasimhan, for keeping her motivated throughout her doctoral journey, and just being a wonderfully kind, humorous, and witty sibling. She expresses her deep appreciation for her mother, who relentlessly supported her throughout her doctoral studies, and whose love and care transcended the physical distance to ensure continual pursuit of the final goal. The author is greatly indebted to her father for his patience, for his guidance, and for his tireless ardor for academics. She thanks both her parents for believing in her dreams absolutely and completely. Most importantly, the author thanks her God Krishna for His divine blessings in enabling her to start and complete this dissertation. May 2014 vii PREFACE This dissertation was produced in accordance with guidelines which permit the inclusion as part of the dissertation the text of an original paper or papers submitted for publication. The dissertation must still conform to all other requirements explained in the \Guide for the Preparation of Master's Theses and Doctoral Dissertations at The University of Texas at Dallas." It must include a comprehensive abstract, a full introduction and literature review, and a final overall conclusion. Additional material (procedural and design data as well as descriptions of equipment) must be provided in sufficient detail to allow a clear and precise judgment to be made of the importance and originality of the research reported. It is acceptable for this dissertation to include as chapters authentic copies of papers already published, provided these meet type size, margin, and legibility requirements. In such cases, connecting texts which provide logical bridges between different manuscripts are mandatory. Where the student is not the sole author of a manuscript, the student is required to make an explicit statement in the introductory material to that manuscript describing the student's contribution to the work and acknowledging the contribution of the other author(s). The signatures of the Supervising Committee which precede all other material in the dissertation attest to the accuracy of this statement. viii MODEL-CHECKING IN-LINED REFERENCE MONITORS Publication No. Meera Sridhar, PhD The University of Texas at Dallas, 2014 Supervising Professor: Dr. Kevin W. Hamlen The formidable growth of the cyber-threat landscape today is accompanied by an imperative need for providing high-assurance software solutions. In the last decade, binary instrumen- tation via In-lined Reference Monitoring (IRMs) has been firmly established as a powerful and versatile technology, providing superior security enforcement for many platforms. IRM frameworks rewrite untrusted binary code, inserting runtime checks to produce safe, self- monitoring code; IRMs are equipped with the ability to enforce a rich set of history-based policies, without requiring access to source code. These immense capabilities, however, come with a price|the power of most real-world IRM infrastructures is usually accompanied by both enormity and complexity, making them prone to implementation errors. In most sce- narios this is highly undesirable; in a mission-critical system, this is unacceptable. Independent certification of two important properties of the IRM, namely soundness and transparency, is extremely important for minimizing the error-space and providing formal assurances for the IRM. Soundness demands that the instrumented code satisfy the policy, whereas transparency demands that the behavior of policy-compliant code is preserved over the instrumentation process. These Certifying In-lined Reference Monitors combine the ix power and flexibility of runtime monitoring with the strong formal guarantees of static analysis, creating automated, machine-verifiable, high-assurance systems. This dissertation demonstrates how the powerful software paradigm of model-checking can be utilized to produce simple, elegant verification algorithms for the special domain of IRM code, providing case-by-case verification of the instrumented code. The dissertation discusses the challenges and subsequent success of developing model-checking IRM frameworks for various platforms. An important result discussed is a new, more powerful class of web security technologies that can be deployed and used immediately, without the need to modify existing web browsers, servers, or web design platforms. Implementations demonstrating security enforcement on a variety of fairly large-scale, real-world Java and ActionScript bytecode applications are also discussed. Theoretical formalizations and proof methodologies are employed to provide strong formal guarantees of the certifying in-lined reference monitoring systems. x TABLE OF CONTENTS ACKNOWLEDGMENTS . v PREFACE . viii ABSTRACT . ix LIST OF FIGURES . xvii LIST OF TABLES . xx CHAPTER 1 INTRODUCTION . 1 1.1 In-lined Reference Monitoring . .2 1.2 Certifying In-lined Reference Monitors . .4 1.3 A Certifying IRM Example . .7 1.4 Policy Specification for Effective Certification . .8 1.5 Certifier Soundness and Completeness . 10 1.6 ActionScript Security . 13 1.7 Roadmap . 16 CHAPTER 2 SECURING MIXED