The Security Layer
Total Page:16
File Type:pdf, Size:1020Kb
Brigham Young University BYU ScholarsArchive Theses and Dissertations 2019-01-01 The Security Layer Mark Thomas O'Neill Brigham Young University Follow this and additional works at: https://scholarsarchive.byu.edu/etd Part of the Computer Sciences Commons BYU ScholarsArchive Citation O'Neill, Mark Thomas, "The Security Layer" (2019). Theses and Dissertations. 7761. https://scholarsarchive.byu.edu/etd/7761 This Dissertation is brought to you for free and open access by BYU ScholarsArchive. It has been accepted for inclusion in Theses and Dissertations by an authorized administrator of BYU ScholarsArchive. For more information, please contact [email protected], [email protected]. The Security Layer Mark Thomas O'Neill A dissertation submitted to the faculty of Brigham Young University in partial fulfillment of the requirements for the degree of Doctor of Philosophy Daniel Zappala, Chair Kent Seamons Michael Goodrich Tony Martinez Ryan Farrell Department of Computer Science Brigham Young University Copyright c 2018 Mark Thomas O'Neill All Rights Reserved ABSTRACT The Security Layer Mark Thomas O'Neill Department of Computer Science, BYU Doctor of Philosophy Transport Layer Security (TLS) is a vital component to the security ecosystem and the most popular security protocol used on the Internet today. Despite the strengths of the protocol, numerous vulnerabilities result from its improper use in practice. Some of these vulnerabilities arise from weaknesses in authentication, from the rigidity of the trusted authority system to the complexities of client certificates. Others result from the misuse of TLS by developers, who misuse complicated TLS libraries, improperly validate server certificates, employ outdated cipher suites, or deploy other features insecurely. To make matters worse, system administrators and users are powerless to fix these issues, and lack the ability to properly control how their own machines communicate securely online. In this dissertation we argue that the problems described are the result of an improper placement of security responsibilities. We show that by placing TLS services in the operating system, both new and existing applications can be automatically secured, developers can easily use TLS without intimate knowledge of security, and security settings can be controlled by administrators. This is demonstrated through three explorations that provide TLS features through the operating system. First, we describe and assess TrustBase, a service that repairs and strengthens certificate-based authentication for TLS connections. TrustBase uses traffic interception and a policy engine to provide administrators fine-tuned control over the trust decisions made by all applications on their systems. Second, we introduce and evaluate the Secure Socket API (SSA), which provides TLS as an operating system service through the native POSIX socket API. The SSA enables developers to use modern TLS securely, with as little as one line of code, and also allows custom tailoring of security settings by administrators. Finally, we further explore a modern approach to TLS client authentication, leveraging the operating system to provide a generic platform for strong authentication that supports easy deployment of client authentication features and protects user privacy. We conclude with a discussion of the reasons for the success of our efforts, and note avenues for future work that leverage the principles exhibited in this work, both in and beyond TLS. Keywords: SSL, TLS, man in the middle, certificate, OpenSSL, security, operating system, administrator control, kernel security, policy, certificates, transport layer security, secure sock- ets layer, developer mistakes, vulnerabilities, client authentication, DANE, OSCP, CRLSet, CRL, Convergence, notary, POSIX, socket API, API ACKNOWLEDGMENTS I'd like to thank my advisor, Daniel Zappala, for allowing me the freedom to pursue my degree in the way and areas I wanted. His immediate, unsolicited, and unending support of me and my ideas was a great source of confidence and encouragement, especially during the most frustrating times. A small army of peers assisted me in the development of the works described in this dissertation: Benjamin Davis, Dan Bunker, Elham Vaziripour, Jordan Whitehead, Joshua Reynolds, Justin Wu, Luke Dickinson, Mason Coram, Nick Bonner, Scott Heidbrink, Tanner Perdue, Torstein Collett, and Travis Hendershot. Their work has been exemplary, and I'm proud to call every single one of them my friend. A special thanks goes to Scott Heidbrink, who worked with me from the time this dissertation was a mere fantasy, through its publications, obtaining his master's degree in the process. His dedication, expertise, and wit, did not go unnoticed and will be missed. Another special thanks goes to my fellow PhD students, Elham Vaziripour and Justin Wu, whose companionship during fun weekend outings and all-nighters in the lab will be forever remembered. I'd also like to thank those who were involved behind the scenes, including Jen Bonnett, whose help none of us graduate students could graduate without, and all of the babysitters who volunteered their time to watch my son as I worked. Most of all, I'd like to thank my wife, Leah, who sacrificed her time and talents to support me both financially and mentally during these years. Finally, I want to thank the National Science Foundation, the Department of Homeland Security, and Sandia National Laboratories, for their interest and financial support of this work. Table of Contents List of Figures x List of Tables xii 1 Introduction 1 1.1 TLS Server Authentication . .2 1.1.1 Background . .2 1.1.2 Problems in Practice . .3 1.2 Using TLS in Applications . .7 1.2.1 Background . .7 1.2.2 Problems in Practice . .8 1.3 TLS Client Authentication . .9 1.3.1 Background . .9 1.3.2 Problems in Practice . 11 1.4 Synthesis . 13 1.5 Solution: A Well-formed Security Layer . 15 2 TrustBase 19 2.1 Abstract . 19 2.2 Introduction . 20 2.3 Related Work . 24 2.4 TrustBase . 26 2.4.1 Threat Model . 26 iv 2.4.2 Design Goals . 27 2.4.3 Architecture . 28 2.4.4 Addressing Certificate Pinning . 31 2.4.5 TLS 1.3 and Overriding the CA System . 32 2.4.6 Operating System Support . 33 2.5 Linux Implementation . 34 2.5.1 Traffic Interceptor . 35 2.5.2 TLS Handler . 38 2.5.3 Opportunistic TLS Handler . 39 2.5.4 Policy Engine . 40 2.6 Security Analysis . 40 2.6.1 Centralization . 40 2.6.2 Coverage . 41 2.6.3 Threat Analysis . 42 2.6.4 Hardening . 44 2.7 Evaluation . 46 2.7.1 Performance . 46 2.7.2 Compatibility . 48 2.7.3 Android Prototype . 49 2.7.4 Windows Prototype . 51 2.7.5 Utility . 52 2.8 Future Work . 52 2.9 Conclusion . 54 3 The Secure Socket API 55 3.1 Abstract . 55 3.2 Introduction . 56 3.3 Motivation . 58 v 3.4 SSA Design Goals . 59 3.5 OpenSSL Analysis . 61 3.5.1 Version Selection . 62 3.5.2 Cipher Suite Selection . 63 3.5.3 Extension Management . 64 3.5.4 Certificate/Key Management . 64 3.5.5 Certificate Validation . 65 3.5.6 Session Management . 66 3.5.7 Configuration . 66 3.5.8 Non-TLS Protocol Specific Functions . 67 3.6 The Secure Socket API . 68 3.6.1 Usage . 68 3.6.2 Administrator Options . 71 3.6.3 Developer Options and Use Cases . 74 3.6.4 Porting Applications to the SSA . 76 3.6.5 Language Support . 77 3.6.6 TLS 1.3 0-RTT . 79 3.7 Implementation Details . 79 3.7.1 Basic Operation . 81 3.7.2 Performance . 83 3.8 Coercing Existing Applications . 85 3.9 Discussion . 87 3.9.1 General Benefits . 87 3.9.2 Implementation Benefits . 88 3.9.3 Configuration Considerations . 89 3.9.4 Alternative Implementations . 89 3.9.5 Security Analysis . 90 vi 3.10 Limitations and Future Work . 92 3.11 Related Work . 93 3.12 Conclusion . 94 4 Modernizing TLS Client Authentication 96 4.1 Abstract . 96 4.2 Introduction . 97 4.3 Background . 99 4.3.1 Typical TLS handshake . 99 4.3.2 Handshake with Client Authentication . 100 4.3.3 TLS 1.3 Advances . 101 4.3.4 Secure Socket API . 103 4.4 Problems . 104 4.4.1 Servers and Administration . 104 4.4.2 Clients and Users . ..