Ca’ Foscari University of Venice and Masaryk University 1 Challenging RSA cryptosystem implementations Doctoral Thesis Matúš Nemec Doctorate Coordinator Advisor (Masaryk University) Prof. Riccardo Focardi Prof. Vashek Matyáš Advisor (Ca’ Foscari University) Consultant Prof. Riccardo Focardi Doc. Petr Švenda Cycle: 32 Semester: Fall 2019 Matricola: 956333 UČO: 396066 SSD: INF/01 Informatica Declaration Hereby I declare that this paper is my original authorial work, which I have worked out on my own. All sources, references, and literature used or excerpted during elaboration of this work are properly cited and listed in complete reference to the due source. Matúš Nemec Advisor (Ca’ Foscari University of Venice): Prof. Riccardo Focardi Advisor (Masaryk University): Prof. Vashek Matyáš Consultant (Masaryk University): Doc. Petr Švenda i Acknowledgements I would like to thank Vashek Matyáš, Riccardo Focardi, Petr Švenda, and Marek Sýs for their guidance and advice. I am thankful to all my co-authors for their fantastic work, and to all the members of CRoCS and ACADIA groups for creating a welcoming atmosphere. I much appreciate the help of Nicola Miotello and Ada Nazarejová with going through the red tape. Last but not least, this thesis would not be possible without the support of my family and friends. iii Abstract The aim of our research was to study security properties of real- world deployments of the RSA cryptosystem. First we analyze RSA key generation methods in cryptographic libraries. We show a prac- tical application of biases in RSA keys for measuring popularity of cryptographic libraries and we develop a factorization method that breaks a proprietary key generation algorithm. Later we examine published implementation issues in the TLS protocol, such as RSA padding oracles, in the wider context of the Web ecosystem. The main conclusion of our previous research on RSA keys was that libraries leak information about their key generation algorithms through the public keys. The bias in the key is often sufficient to identify a library that likely generated the key. In this thesis, we fur- ther demonstrate a practical method for measuring representation of cryptographic libraries in large datasets of RSA public keys. We use statistical inference to approximate a share of libraries matching an observed distribution of RSA keys in an inspected dataset. Such esti- mate also allows us to deploy an improved key classification method that uses prior probabilities of libraries to more accurately determine the source of a key. Next we adapt a key factorization attack to keys generated by a widely deployed proprietary cryptographic library. Our previous re- search hinted at issues in the algorithm by showing an unusual bias in the keys. We reveal the algorithm without the access to the source code by analyzing algebraic properties of the keys. We develop a practical factorization attack that can compute private keys corresponding to the public keys used in electronic identification documents, Trusted Platform Modules, authentication tokens, and other domains that use secure chips with this RSA key generation algorithm. Our findings were disclosed in coordination with the manufacturer of the devices to minimize the resulting security threats. Finally, we focus on implementation issues in the TLS protocol, of which RSA key exchange is an integral part. We survey existing practical attacks that affect current usage of TLS for securing the Web. We specify conditions for the attacks as attack trees. Using measure- ments on popular domains, we confirm that RSA padding oracles are iv currently the most common threat. We show how such security issues get amplified by the complexity of the Web ecosystem. In many cases of vulnerable websites, the issues are caused by external or related- domain hosts. Our work helps to demonstrate how RSA, a seemingly simple and intuitive cryptosystem, requires a lot of knowledge to be implemented correctly. Unlike RSA, elliptic curve cryptography (ECC) uses random sequences as keys and does not require padding. It is more resistant to bad user configurations and provides many other benefits. We conclude that practitioners should follow the example of TLS version 1.3 and stop using RSA in favor of ECC. v Keywords Bleichenbacher’s oracle, Coppersmith’s algorithm, cryptographic li- brary, cryptographic protocol, factorization, key generation, RSA, TLS vi Contents 1 Introduction1 1.1 Problem statement ......................2 1.2 Contributions .........................3 1.2.1 Biased RSA key generation methods.......3 1.2.2 Vulnerabilities in RSA key generation......3 1.2.3 TLS protocol vulnerabilities............4 1.3 Structure of the thesis .....................5 2 State of the art7 2.1 Survey of RSA key generation methods ............7 2.2 Bias in RSA primes and keys .................8 2.2.1 RSA prime types..................9 2.2.2 RSA keypairs....................9 2.2.3 Modular bias.................... 10 2.3 Pseudo-random number generator failures .......... 10 3 Measuring popularity of cryptographic libraries 13 3.1 Introduction .......................... 14 3.2 Method overview ....................... 17 3.2.1 Choice of key features............... 17 3.2.2 Clustering analysis................. 18 3.2.3 Dataset classification – original approach.... 20 3.2.4 Dataset classification – our approach....... 20 3.2.5 Limitations..................... 22 3.3 Methodology in detail ..................... 23 3.3.1 Model........................ 23 3.3.2 Prior probability estimation............ 23 3.3.3 Key classification.................. 24 3.3.4 Evaluation of accuracy............... 24 3.3.5 Additional accuracy considerations....... 28 3.4 Results on relevant datasets .................. 31 3.4.1 Data preparation.................. 31 3.4.2 Internet-wide TLS scans.............. 35 3.4.3 Popularity between usage domains........ 37 3.4.4 TLS to CT comparison............... 37 vii 3.4.5 Detection of transient events........... 38 3.5 Related work .......................... 40 3.6 Additional results ....................... 41 3.7 Conclusions .......................... 42 4 Factorization of widely used RSA moduli 47 4.1 Introduction .......................... 47 4.2 Fingerprinting and factorization ............... 51 4.2.1 Format of the constructed primes......... 52 4.2.2 Fingerprinting.................... 53 4.2.3 Factorization – attack principle.......... 55 4.2.4 Coppersmith’s algorithm in detail........ 57 4.2.5 Application of Coppersmith’s algorithm.... 60 ∗ 4.2.6 Computing the order of a generator in ZM0 ... 61 4.2.7 Optimization of the parameters M0, m, t ..... 62 4.2.8 Guessing strategy.................. 68 4.3 Practical implementation ................... 69 4.3.1 Details and empirical evaluation......... 69 4.3.2 Possible improvements and limitations..... 69 4.4 Analysis of impacts ...................... 71 4.4.1 Summary of results................. 73 4.4.2 Electronic identity documents.......... 74 4.4.3 Code signing.................... 76 4.4.4 Trusted Platform Modules............. 77 4.4.5 PGP with cryptographic tokens.......... 80 4.4.6 TLS and SCADA-related keys........... 82 4.4.7 Certification authorities.............. 83 4.4.8 Generic Java Card platform............ 83 4.4.9 Other domains................... 85 4.5 Mitigation and disclosure ................... 85 4.5.1 Mitigation...................... 86 4.5.2 Future prevention and analysis.......... 88 4.5.3 Responsible disclosure............... 89 4.6 Related work .......................... 90 4.7 Conclusions .......................... 91 5 Amplification of TLS vulnerabilities on the Web 93 5.1 Introduction .......................... 93 viii 5.2 Background on TLS ...................... 98 5.2.1 The Handshake Protocol.............. 98 5.2.2 Ciphersuites..................... 100 5.3 Attack trees for TLS security ................. 101 5.3.1 Threat model.................... 101 5.3.2 Review of known attacks against TLS...... 102 5.3.3 Insecure channels.................. 105 5.3.4 Leaky channels................... 105 5.3.5 Tainted channels.................. 108 5.3.6 Partially leaky channels.............. 109 5.4 Experimental setup ...................... 110 5.4.1 Analysis platform.................. 110 5.4.2 Data collection and findings............ 111 5.4.3 Roadmap...................... 113 5.5 Page integrity ......................... 114 5.5.1 Security analysis.................. 114 5.5.2 Experimental results................ 116 5.6 Authentication credentials .................. 118 5.6.1 Security analysis.................. 118 5.6.2 Experimental results................ 121 5.6.3 Detected attacks.................. 123 5.7 Web tracking ......................... 124 5.7.1 Security analysis.................. 124 5.7.2 Experimental results................ 126 5.8 Closing remarks ........................ 127 5.8.1 Related work.................... 127 5.8.2 Ethics and limitations............... 128 5.8.3 Summary and perspective............. 129 5.9 Additional results ....................... 130 5.9.1 Notable out of scope attacks against TLS.... 130 5.9.2 More detailed attack trees............. 134 6 Conclusions 139 Bibliography 143 A Author’s publications 167 ix 1 Introduction The RSA cryptosystem predates most currently used cryptographic libraries. It is often a necessary part of a new library and perhaps one of the first features to be implemented. In a survey of RSA key generation implementations [Šve+16a], we saw how the source code for RSA in community-driven