DRLGENCERT: Deep Learning-based Automated Testing of Certificate Verification in SSL/TLS Implementations Chao Chen∗, Wenrui DiaoyB, Yingpei Zengz, Shanqing Guo∗B, and Chengyu Hu∗ ∗Shandong University, Jinan, China Email: [email protected], fguoshanqing, [email protected] yJinan University, Guangzhou, China Email: [email protected] zChina Mobile (Hangzhou) Information Technology Co., Ltd., Hangzhou, China Email: [email protected] Abstract—The Secure Sockets Layer (SSL) and Transport ifying the validity of a certificate is a complex process that Layer Security (TLS) protocols are the foundation of network includes verifying each certificate in the certificate chain, security. The certificate verification in SSL/TLS implementations checking period of validity, public key, extension, and so forth. is vital and may become the “weak link” in the whole network ecosystem. In previous works, some research focused on the auto- There are many open source SSL/TLS implementations mated testing of certificate verification, and the main approaches available online, and developers can use these implementations rely on generating massive certificates through randomly combin- for certificate validation without implementing the code by ing parts of seed certificates for fuzzing. Although the generated themselves. However, since the certificate validation is com- certificates could meet the semantic constraints, the cost is quite heavy, and the performance is limited due to the randomness. plicated, described in many tedious documents, the authors of To fill this gap, in this paper, we propose DRLgencert, the SSL/TLS implementations may have their own understandings first framework on applying deep reinforcement learning to of how to code the logic. In other words, there is no guarantee the automated testing of certificate verification in SSL/TLS that these SSL/TLS implementations can verify each certificate implementations. DRLgencert accepts ordinary certificates as correctly, which may become the “weak link” in the whole input and outputs the newly generated certificates which could trigger discrepancies with high efficiency. Benefited by the deep network ecosystem. reinforcement learning, when generating certificates, our frame- Some researchers[9], [10] have noticed this issue and tried work could choose the best next action according to the result of to achieve the automated testing of certificate verification in a previous modification, instead of simple random combinations. SSL/TLS implementations. One of the mainstream approaches At the same time, we developed a set of new techniques to support the overall design, like new feature extraction method for X.509 is the differential testing. If multiple certificate verification certificates, fine-grained differential testing, and so forth. Also, codes give different verification results for the same certificate, we implemented a prototype of DRLgencert and carried out a it means some of the certificate verification codes may be series of real-world experiments. The results show DRLgencert flawed. In the differential testing, the effectiveness of test is quite efficient, and we obtained 84,661 discrepancy-triggering results is decided by the quality of input test cases (i.e., certificates from 181,900 certificate seeds, say around 46.5% effectiveness. Also, we evaluated six popular SSL/TLS imple- modified certificates) directly. Better test cases could discover mentations, including GnuTLS, MatrixSSL, MbedTLS, NSS, more design flaws with less time consumption. OpenSSL, and wolfSSL. DRLgencert successfully discovered 23 In previous attempts, it is common to generate massive arXiv:1808.05444v1 [cs.CR] 16 Aug 2018 serious certificate verification flaws, and most of them were previously unknown. certificates (as test cases) through randomly combining parts of seed certificates, like Frankencert [9] and Mucert [10]. I. INTRODUCTION Although the generated certificates could meet the semantic The Transport Layer Security (SSL) [?] and its predeces- constraints, the cost is quite heavy, and the performance is sor Transport Layer Security (TLS) [18] protocols are the limited due to the randomness. It may generate a large number foundation of network security. They are designed to provide of unhelpful certificates which cannot trigger any flaw. On the security and data integrity assurance for Internet commu- other side, the deep learning technology shows the powerful nications. During the SSL/TLS communication, the X.509 capability of information mining and has been widely applied certificate is used to authenticate the identity of communi- in biology, medicine, graphics, and cybersecurity, and so forth. cating party. The semantics and syntax of X.509 certificates We find the deep learning is suitable for the task of automated have many limitations, described semi-formally in dozens of testing of certificate verification. IETF’s RFCs (including RFC 2246[13], 2527[11], 2818[26], Our Approach. In this paper, we propose DRLgencert (using 4346[14], 5246[15], 5280[12], 6101[18], and 6125[27]). Ver- Deep Reinforcement Learning to generate certificates), the first framework on applying deep reinforcement learning to tbsCertificate tbsCertificate tbsCertificate the automated testing of certificate verification in SSL/TLS version version version serialNumber serialNumber serialNumber implementations. DRLgencert accepts ordinary certificates as signature signature signature 1 issuer issuer issuer input and outputs the newly generated certificates which could 2 3 Version Version Validity (Not Before/After) Validity (Not Before/After) Validity (Not Before/After) trigger discrepancies with high efficiency. Benefited by the Version subject subject subject Version Version deep reinforcement learning, when generating certificates, our subjectPublicKeyInfo subjectPublicKeyInfo subjectPublicKeyInfo issuerUniqueID issuerUniqueID issuerUniqueID framework could choose the best next action according to the subjectUniqueID subjectUniqueID subjectUniqueID result of a previous modification, instead of simple random extensions extensions extensions combinations. At the same time, we developed a set of new signatureAlgorithm signatureAlgorithm signatureAlgorithm signature signature signature techniques to support the overall design, like new feature ex- Leaf cert Intermediate CA cert Root CA certificate traction method for X.509 certificates, fine-grained differential testing module, and so forth. Fig. 1. General certificate chain structure. In our research, we implemented DRLgencert and carried out a series of real-world experiments based on six popu- lar SSL/TLS implementations, including GnuTLS [1], Ma- Environment trixSSL [3], mbedTLS [4], NSS [5], OpenSSL [6], and wolf- SSL [7]. The overall performance analysis shows DRLgencert Action is quite efficient, and we could obtain 84,661 discrepancy- Reward State triggering certificates from 181,900 certificate seeds after the first training episode of DRL network. It means more than Agent 46.5% generated certificates are effective, which is better than all existing works. Also, DRLgencert successfully discovered 23 serious certificate verification flaws on six implementations, Fig. 2. The process of reinforcement learning. and most of them were previously unknown. For example, we found GnuTLS, NSS, and OpenSSL accept version 1, 2, Roadmap. The rest of this paper is organized as follows. or 4 certificates, even these v1, v2, v4 certificates have v3 Section II gives the background of certificate validation and extensions that should only exist in version 3 certificate, which deep reinforcement learning. Section III provides the overview violates the RFC documents [12]. The reason is that the ver- design of DRLgencert. Section IV and Section V illustrate the sion testing and extension testing functions are implemented differential testing module and deep reinforcement learning as two independent components in code. We have reported all network module respectively. The evaluation results are sum- these flaws to the corresponding vendors, and the assessments marized in Section VI. Section VII reviews the related work, are in process. and finally Section VIII concludes this paper. Contributions. The main contributions of this paper are: II. BACKGROUND • New framework. We proposed DRLgencert, the first A. Certificate Validation framework on applying deep reinforcement learning to Certificate validation usually requires two inputs: trusted CA the automated testing of certificate verification in SS- certificates and a chain of certificates to be validated. The L/TLS implementations. DRLgencert accepts normal cer- general structure of a certificate chain is shown in Figure 1. tificates as input and outputs the newly generated certifi- Starting from the leaf certificate (end-entity certificate), each cates (as the test cases of differential testing) which could certificate is issued by a superior certificate until a self-signed trigger discrepancies with high efficiency. CA certificate. • New techniques. We developed a set of new techniques In the verification process, the leaf certificate in the certifi- to enable the overall design of DRLgencert, including cate chain is usually verified first to confirm that the content new feature extraction method for X.509 certificates, of the certificate is valid, including the validity period and fine-grained differential testing, and improved certificate extension content. Then the SSL/TLS client checks whether modification actions. the certificate is issued by a higher-level certificate in the • Implementation and findings.