Masaryk University Faculty of Informatics Comparing X.509 Certificate Validation Errors Across TLS libraries Bachelor’s Thesis Pavol Žáčik Brno, Spring 2021 Declaration Hereby I declare that this paper is my original authorial work, which I have worked out on my own. All sources, references, and literature used or excerpted during elaboration of this work are properly cited and listed in complete reference to the due source. Pavol Žáčik Advisor: RNDr. Martin Ukrop i Acknowledgements My deep gratitude goes to my advisor, Martin Ukrop, for his invaluable tutorship over the last two years. It is a pleasure to be guided by you. I thank my family for their support, advice and calm. I am grateful to Mrs. Szórádová for her motivational words of worries. Finally, my warmest thanks go to Maťka, for her tender care, avid interest, and ceaseless love. ii Abstract IT professionals often meet certificate validation errors when dealing with TLS. In such situations, their decisions may be crucial for the security of systems they implement. However, error messages differ depending on the used TLS library, and official documentation usually does not help much. This thesis performs a comparison of certificate validation errors occurring in five common TLS libraries. To do so, it employs a custom set of erroneous certificates. Furthermore, a simple TLS connection is implemented in the five libraries. As a result, we establish a mapping between the corresponding errors from different libraries. The mapping is published online, together with the erroneous certificates and TLS source code. All three resources aim to be used by developers when they require guidance. Keywords certificate validation, documentation, usable security, validation error, X.509, TLS library iii Contents 1 Introduction1 2 Crafting malformed certificates4 2.1 Public key certificates....................4 2.1.1 X.509 certificate profile...............4 2.1.2 Certificate path validation.............7 2.1.3 Certificate revocation mechanisms.........8 2.2 Creating a certificate dataset................9 2.2.1 Prototype solution and its limits..........9 2.2.2 Abstract Syntax Notation 1............ 11 2.2.3 Constructing arbitrary ASN.1 structures..... 12 2.2.4 X.509 Python module............... 14 2.2.5 Generating invalid certificate chains........ 15 2.2.6 Final implementation................ 17 3 Validating certificates within TLS 19 3.1 TLS protocol........................ 19 3.1.1 TLS handshake................... 19 3.1.2 Security properties of TLS............. 20 3.1.3 TLS deployment.................. 21 3.2 Implementing certificate validation............ 21 3.2.1 Prototype solution................. 22 3.2.2 Client-side TLS connection............. 23 3.2.3 Library choice.................... 24 3.2.4 TLS client implementation............. 24 3.2.5 Server implementation............... 26 4 Comparing errors across libraries 28 4.1 Goals............................. 28 4.2 Build............................. 30 4.3 Result processing...................... 32 4.3.1 Collecting error data................ 32 4.3.2 Observations.................... 33 4.3.3 Cross-library error linking............. 35 4.4 Results evaluation...................... 37 4.4.1 Error taxonomy improvements........... 40 iv 5 Deployment 41 5.1 Error documentation.................... 41 5.2 Developer guides...................... 43 6 Related work 47 6.1 Malformed certificates................... 47 6.2 Improving certificate infrastructure............ 48 6.3 X.509 and TLS usability.................. 50 7 Conclusion 51 7.1 Limitations......................... 51 7.2 Future work......................... 52 References 53 Appendix: Source code and build 61 v 1 Introduction X.509 public key certificates play a key part within multiple modern security protocols, including those most commonly used, such as TLS. The purpose of certificates in the protocol is rather simple. Still, many end users and developers alike fail to understand the consequences of deploying and trusting the invalid ones [5, 84]. Pitfalls of certificate validation. Before one can rely on a certificate, it must be properly validated. However, due to their complex nature, validating certificates is a rather involved procedure, with its description spanning multiple documents [29, 71, 74, 75]. There are many ways in which a certificate or a certificate chain can become erroneous. During the validation process, all possible miscon- figurations need to be checked. Failing to do so properly may lead to accepting a forged certificate and thus to security threats [35]. Researchers have been assessing certificate validation source code in cryptographic libraries for years, still being able to find new security holes within [19, 24, 49]. Even if we assume that no such holes exist, there are new issues to consider when developers start using the libraries’ source code in their applications. Usable documentation. Some errors occurring during certificate validation pose no or only minimal risks. As a consequence, developers may have to differentiate between “benign” and “malign” errors. These decisions can be crucial for security, as they affect many end users [40]. To help developers make better decisions, cryptographic libraries’ authors should strive to provide detailed and usable documentation regarding both the deployment of certificate validation and the errors occurring therein. Unfortunately, previous research shows that such attempts are scarce [1, 39]. To further complicate the issue, the taxonomy of possible errors is chaotic and not at all unified. Each cryptographic library uses its own set of error codes when referring to certificate deformities. Hence, developers face extra troubles when transitioning from one library to another. Unifying the system would be difficult, but previous successful attempts [34, 44] have proven similar tasks possible. 1 1. Introduction Our contributions. As part of the Usable X.509 Errors project [85], this thesis aims to provide a supplementary resource for developers to consult when dealing with certificate validation. We divide our contributions into three main parts. 1. Erroneous certificate dataset. We establish a dataset of more than 60 distinct malformed cer- tificate chains, which are publicly available for testing purposes. Each chain is initially crafted by hand, but the dataset itself is dynamically generated (Chapter 2). 2. Client-side TLS implementation. We implement basic TLS client-side connection in five crypto- graphic libraries. Each implementation properly validates server certificates. Furthermore, the source code of three TLS clients is published online in the form of well-documented developer guides (Chapter 3). 3. Cross-library error mapping. Merging the previous, we develop an automated system which compares certificate validation errors occurring within the five TLS-enabled libraries. This comparison aims to be one of the initial steps in improving X.509-related documentation. (Chapter 4). Together, the three listed topics form the main scope of this thesis. The rest is laid out as follows. Deployment. All of our newly created resources are deployed on the website x509errors.org. Neither the deployment process nor the website design lie within the main focus of the thesis, but they are briefly described to give an idea of how our results are published (Chapter 5). Related work. The list of previous research related to our work is relatively wide. We present relevant research efforts and locate our contributions among them (Chapter 6). Conclusion. Our work can be further extended in multiple directions. Along with the summary of the thesis, we list its limits and propose possible future extensions and improvements. (Chapter 7). 2 1. Introduction Acknowledgment of collaboration As already mentioned, this thesis is part of the Usable X.509 Errors project [85]. Multiple other students collaborate on the project, and thus the work presented can not fully be my own. Here, I list all of their individual contributions. My thesis advisor and the project lead, Martin Ukrop, helped with prototype development of both certificate generation (Section 2.2.1) and certificate validation (Section 3.2.1). Both were completely replaced, but I include them for illustrative purposes. Additionally, the design of the website where our results are deployed (Chapter 5) is almost exclusively his effort. Therefore, it is presented only in brief. Matěj Grabovský implemented initial versions of three out of five TLS clients (Section 3.2.4), specifically of OpenSSL, GnuTLS, and Mbed TLS. His work was then further improved and refactored. Eric Vincent Valčík helped with website development (Section 5.1) and with collecting library error data (Section 4.3.1). He implemented a prototype of certificate mapping, which is not mentioned in the thesis, since it was fully renewed. Unless explicitly stated otherwise, all further work presented in the following chapters is my own. 3 2 Crafting malformed certificates In order to be able to compare certificate validation error messages occurring within cryptographic libraries, we must prepare a certificate dataset to validate. Such a dataset should be reasonably large and diverse so that it will exhibit as many validation errors as possible. This chapter begins with a description of the X.509 certificate profile (Section 2.1), followed by the tools we implemented to create arbitrary certificate structures (Section 2.2). 2.1 Public key certificates Using asymmetric cryptography gives rise to some major obstacles. When keys are designed to be public,