ABSTRACT OCONNOR, TERRENCE JOHN. Network Access Control through Host and Application Analysis. (Under the direction of William Enck). Despite increasing complexity to networks and applications, the state of practice in network access control has remained largely static for decades. Existing network security solutions provide limited protection against modern and elaborate network compromises where attackers move laterally through the network, because they lack intra-network mediation and distributed information flow control. The omnipresence of Internet-of-Things (IoT) devices with encrypted and propriety protocols obscures network transparency, making it difficult to identify and classify application behaviors. The always- on nature of IoT with perpetual cloud connections complicates policy enforcement because of the heterogeneity of security and privacy designs for offline content caching and device state management. Leveraging host and application behavior analysis provides sufficient context for network access control to address the increasing complexity of networks, opaqueness in applications behaviors, and perpetual cloud connections. In this dissertation, we demonstrate how specialized and sophisticated network access control schemes address the aforementioned challenges by considering host and application context. We implement two security orchestrator application frameworks to address complex attacks, and provide behavior transparency for smart-home IoT devices. First, we present PivotWall which uses a novel combination of information-flow tracking and Software Defined Networking (SDN) to detect a wide range of attacks used by advanced adversaries, including those that abuse both application- and network-layer protocols. It further enables a variety of attack responses including traffic steering, as well as advanced mechanisms for forensic analysis. Through PivotWall, we extend information flow tracking out from a host and through the entire network. Second, we introduce HomeSnitch, a building block for enhancing smart home transparency and control by classifying IoT device communication by behavior. HomeSnitch uses supervised learning to classify features from connection-oriented application data unit exchanges that represent application-layer dialog between clients and servers. This overcomes the limitations of increasingly encrypted payloads in smart home devices. Having laid the groundwork for access control for smart-home devices with our behavior classification, we then answer critical questions that govern access control policy and mediation for smart-home devices. We present a sufficiently broad study of the application behaviors of smart-home actuator and sensor-based IoT devices and analyze their responses to a loss of the communications channel. We investigate weaknesses in the current heterogeneity of smart-home device security designs and describe implications for offline state and cache decisions. In doing so, we provide the context understanding for effective policy enforcement. Network Access Control through Host and Application Analysis by Terrence John OConnor A dissertation submitted to the Graduate Faculty of North Carolina State University in partial fulfillment of the requirements for the Degree of Doctor of Philosophy Computer Science Raleigh, North Carolina 2019 APPROVED BY: Bradley Reaves Alexandros Kapravelos Aydin Aysu William Enck Chair of Advisory Committee DEDICATION For my Ninja Princess and Monkey. ii BIOGRAPHY TJ OConnor received his Bachelor’s Degree in Computer Science from the US Military Academy at West Point in 1999. Following graduation, TJ served 20 years in the US Army, retiring at the rank of Lieutenant Colonel. His service included assignments with the 2nd Infantry Division, the 7th Special Forces Group (Airborne), the 10th Special Forces Group (Airborne), the Office of Special Warfare (Airborne), and the 1st Special Forces Command (Airborne). In 2008, TJ earned a Master’s Degree in Computer Science from North Carolina State University. Subsequently, he served as an assistant professor in the Electrical Engineering and Computer Science department at the US Military Academy and as the Professor of Military Science for the Florida Institute of Technology. He is the first (but hopefully not last) member of his immediate family to earn a doctorate. iii ACKNOWLEDGEMENTS I would like to thank my advisor, Dr. William Enck, for his unparalleled support over the course of my time in graduate school. I cannot express the gratitude, respect, and debt I owe you. I can only promise to do my best to emulate the positive traits you have taught me. I would also like to thank the members of my doctoral committee including Dr. Bradley Reaves, Dr. Alexandros Kapravelos, and Dr. Aydin Aysu for their valuable insight. I owe a debt of gratitude my teammates on the Wolfpack Security and Privacy Research (WSPR) that have provided me the candid feedback, expertise, and emotional support necessary for this endeavor. Specifically, I’d like to thank Akash Verma, Sanket Gooutam, Micah Bushouse, Benjamin Andow, Luke Deshotels, and Isaac Polinsky. I’d also like to thank Michael Petullo and Sam Huddleston for being my sounding boards on so many ideas. I would finally like to thank my family for their unconditional love, support and belief in me. We did this. iv TABLE OF CONTENTS LIST OF TABLES ...................................................... viii LIST OF FIGURES ..................................................... ix LIST OF ABBREVIATIONS ............................................... xi Chapter 1 Introduction ................................................. 1 1.1 Thesis Statement .................................................. 3 1.2 Thesis Contributions ............................................... 4 1.3 Thesis Organization ................................................ 4 Chapter 2 Background and Related Work ................................... 7 2.1 Software Defined Networking ......................................... 7 2.1.1 The Data Plane ............................................. 8 2.1.2 The Control Plane ........................................... 8 2.1.3 The Application Plane ........................................ 9 2.1.4 Extending SDN to Security ..................................... 9 2.1.5 Emulation Environments ....................................... 10 2.2 The Internet of Things .............................................. 10 2.2.1 Overview of IoT ............................................ 10 2.2.2 Cloud Services ............................................. 11 2.2.3 Access Control ............................................. 11 2.2.4 Device and Behavior Transparency ................................ 12 2.2.5 Motivating Examples ......................................... 12 2.3 Machine Learning ................................................. 14 2.3.1 Supervised Learning Models .................................... 14 2.3.2 Unsupervised Learning Models .................................. 15 2.3.3 Scikit-Learn Framework and Evaluation ............................ 15 2.4 Threat Model .................................................... 16 Chapter 3 PivotWall ................................................... 17 3.1 Introduction ..................................................... 17 3.2 Motivation ...................................................... 19 3.2.1 Motivating Example .......................................... 19 3.2.2 Threat Model and Assumptions .................................. 20 3.3 Overview ....................................................... 21 3.3.1 Architecture Overview ........................................ 21 3.3.2 Challenges ................................................ 22 3.3.3 Key Ideas in PivotWall ........................................ 22 3.4 PivotWall ....................................................... 24 3.4.1 Network Layer Design ........................................ 25 3.4.2 Host Design ............................................... 29 3.4.3 Forensic Analysis ............................................ 31 v 3.5 Performance Evaluation ............................................. 32 3.5.1 RQ1: Stealthy Attack Detection .................................. 33 3.5.2 RQ2: Attack Coverage ........................................ 36 3.5.3 RQ3: Network Scalability ...................................... 36 3.5.4 RQ4: Response Capability ..................................... 37 3.6 Discussion ...................................................... 38 3.7 Related Work .................................................... 39 3.7.1 Detection Using Network Flow Correlation .......................... 39 3.7.2 Network Management and Monitoring for SDN ...................... 40 3.7.3 Host Provenance Frameworks ................................... 41 3.8 Conclusion ...................................................... 42 3.9 Acknowledgements ................................................ 42 Chapter 4 HomeSnitch ................................................. 43 4.1 Introduction ..................................................... 43 4.2 Motivation and Problem ............................................. 45 4.3 Overview ....................................................... 46 4.4 Design ......................................................... 48 4.4.1 Classification of Application Behaviors ............................ 49 4.4.2 Context Monitor ............................................ 52 4.4.3 Policy ...................................................