Two mathematical security aspects of the rsa cryptosystem: signature padding schemes and key generation with a backdoor Genevi`eve Arboit Doctorate of Philosophy School of Computer Science McGill University Montreal, Quebec February 2008 A thesis submitted to McGill University in partial fulfilment of the requirements of the degree of philosophy doctorate in computer science. Copyright Genevi`eve Arboit, 2008. DEDICATION This document is dedicated to all people dear to me. ii ACKNOWLEDGEMENTS Firstly, I thank my thesis adviser, Claude Cr´epeau, for suggesting the topic of backdoors and for his counsel. I also thank Jean-Marc Robert and Alain Slakmon for their precious reading and comments. I thank my parents, my family, my friends, and Dan Nyborg who put up with my excentric notion of becoming a doctor. I thank the crypto and quantum info lab people, as well as Geza Szamosi, Faith Fich, Charles Rackoff, Laurie Hendren, David Avis, Prakash Panangaden, and Bettina Kemme, for their interest and dis- interested help. In particular, I owe thanks to Isabelle D´ech`ene,Raza Ali Kazmi, Simon Pierre Desrosiers, Martin Courchesne, Natasa Przulj, Nicole Witen (came up with the Casanova story), Valerie Hongoh, Jenifer Thorsley, Shashikala Sivapra- gasam, Paul Dumais, Hugo Touchette, and Fran¸coisPitt. Concerning publications, I would additionally like to thank David Naccache, Jean-S´ebastienCoron, and a few anonymous referees for useful comments. The McGill kendo club and d'Orangeville sensei are a late addition to the people I acknowledge the help of and deeply thank, for making the writing of a philosophy doctorate thesis appear to be relatively easy. iii ABSTRACT This work presents mathematical properties of the rsa cryptosystem. The top- ics of backdoors and padding algorithm are developped. For padding schemes, we give a practical instantiation with a security reduction. It is based on the compression function of sha-1 without any chaining function. Our solution has the advantage over the previous one of removing the relation of the output length of the compression function to the length of the rsa modulus. For backdoors, improvements on definitions, existing algorithms as well as ex- tensions of existing theorems are shown. The definitions pertaining to backdoored key generators are improved as to make their analysis uniform and comparable. New algorithms are presented and compared to existing ones as to show improvements mainly on their running time, the indistinguishability of the keys produced, and that some of these new algorithms are, for all practical purposes, the best that may be called asymmetric. Our theorem on the correctness (or completeness) of one of our better backdoored key generators is a generalization of a theorem of Boneh, Durfee and Frankel's on partial information on the decryption exponent. iv RESUM´ E´ Ce travail pr´esente des propri´et´esmath´ematiquesdu cryptosyst`eme rsa. Les sous-domaines d´evelopp´essont ceux des portes d´erob´eeset des algorithmes de rem- plissage. Pour les algorithmes de remplissage, nous pr´esentons une instantiation pratique avec une r´eductionde s´ecurit´e. Elle est bas´eesur la fonction de compression de sha-1 sans user d'aucune fonction d'encha^ınement. Notre solution a l'avantage, par rapport `ala pr´ec´edente, d'enlever la relation entre la longueur de sortie de la fonction de compression et la longueur du modulo rsa. Pour les portes d´erob´ees,des am´eliorationssur les d´efinitions,les algorithmes existants aussi bien que des prolongements des th´eor`emesexistants sont expos´es.Les d´efinitionsconcernant les g´en´erateursde cl´es`aporte d´erob´eesont am´elior´eesafin que leur analyse soit uniforme et comparable. De nouveaux algorithmes sont pr´esent´es et compar´es`aceux existants afin de d´emontrer des am´eliorations principalement par rapport `aleur temps d'ex´ecution,`al'indistinguabilit´edes clefs produites et par le fait que certains de ces nouveaux algorithmes sont, `atoutes fins pratiques, les meilleurs pouvant ^etre qualifi´esd'asym´etriques.Notre th´eor`emesur l'exactitude d'un de nos meilleurs g´en´erateursde cl´es`aporte d´erob´eeest une g´en´eralisationd'un th´eor`eme de Boneh, Durfee et Frankel sur l'information partielle de l'exposant de d´echiffrage. v TABLE OF CONTENTS DEDICATION . ii ACKNOWLEDGEMENTS . iii ABSTRACT . iv RESUM´ E......................................´ v LIST OF TABLES . x LIST OF FIGURES . xiii NOTATION . xvii CONTRIBUTIONS OF AUTHORS . xix 1 Introduction . 1 1.1 Brief history of public-key cryptosystems . 1 1.2 The rsa cryptosystem . 5 1.2.1 Definition . 5 1.2.2 rsa signature padding . 7 1.2.3 Backdoors in public key generation . 11 1.3 Chapter notes . 14 2 Background on rsa .............................. 15 2.1 History . 15 2.2 Theory . 16 2.2.1 Computational security . 16 2.2.2 Trapdoor one-way function . 17 2.2.3 Toward the definition of rsa . 18 2.2.4 The definition of rsa ..................... 19 2.2.5 Efficiency of key generation . 24 2.2.6 Number of generable keys . 25 vi 2.2.7 Provable security . 26 2.2.8 Distribution properties of rsa . 29 2.3 Application issues and standards . 46 2.3.1 Direct attacks . 46 2.3.2 Indirect attacks . 47 2.4 Chapter notes . 47 3 rsa signature padding . 50 3.1 Introduction . 50 3.1.1 Hash-and-sign paradigm . 51 3.1.2 Chosen-ciphertext attacks . 53 3.1.3 Security goal . 55 3.2 Definitions . 56 3.3 An improved algorithm . 58 3.4 Further developments . 61 3.4.1 A practical hashing family H3·160. 61 3.4.2 Improved communication complexity. 63 3.5 Summary . 64 3.6 Chapter notes . 65 4 Introduction to backdoors . 66 4.1 Background . 66 4.1.1 The two contexts . 66 4.1.2 Intuition on the roles of the parties . 69 4.1.3 History . 71 4.2 Definitions . 74 4.2.1 Informal definition of a backdoor . 74 4.2.2 Model of analysis . 77 4.2.3 Formal definition of a backdoor . 79 4.3 Comparison with setup definitions for backdoors in cryptosystems 85 4.3.1 Definition of setup ...................... 85 4.3.2 Evolution of algorithms w.r.t. definitions in kleptography . 87 4.3.3 White box effect, uniformity and asymmetry . 88 4.3.4 Asymmetry (un)satisfied by setups . 88 4.4 Lexicon . 89 4.5 Chapter notes . 90 vii 5 Measures for backdoored keys . 91 5.1 Nature of the keys . 92 5.1.1 Classical definitions of indistinguishability . 92 5.1.2 Cardinality . 95 5.1.3 Distribution properties . 106 5.1.4 Generalized key regeneration . 110 5.1.5 Diversity . 113 5.2 Interactions with the generator: side channel analyses . 115 5.2.1 Complexity . 115 5.2.2 Memory . 122 5.3 Measures on the design of algorithms . 125 5.3.1 Computational assumptions . 125 5.3.2 Simplicity . 126 5.4 Measures: summary . 126 5.5 Chapter notes . 127 6 Comparison of existing algorithms . 129 6.1 Simple algorithms . 129 6.1.1 Anderson-Kaliski backdoors . 130 6.1.2 Howgrave-Graham backdoors . 137 6.2 Kleptography: Young-Yung algorithms . 146 6.2.1 YY96 . 146 6.2.2 YY97 . 168 6.2.3 YY05a . 175 6.2.4 YY05b . 181 6.3 Cr´epeau-Slakmon algorithms . 189 6.3.1 First algorithm: via Wiener's low decryption exponent attack192 6.3.2 Second algorithm: via upper bits of δ and prime . 196 6.3.3 Third algorithm: via upper and lower bits of δ . 200 6.3.4 Fourth algorithm: via hiding p in n . 206 6.3.5 Fifth algorithm: via Slakmon's variant of Wiener's Theorem 213 6.3.6 Choices of permutation, πβ . 221 6.4 Chapter notes . 225 7 Improved algorithms . 227 7.1 First improvement: complexity, then diversity . 228 7.1.1 Outline of the first improvement . 228 viii 7.1.2 Preliminary results: new extensions of Wiener's Theorem . 231 7.1.3 Improvement of the time complexity . 240 7.1.4 Improvement of the diversity without pseudo-randomness . 257 7.2 Discussion on the first improvement . 278 7.2.1 Structure of the improvement . 278 7.2.2 Failure of simple algorithms . 279 7.2.3 Advantages of a reasonable simplicity . 282 7.2.4 Diversity of the algorithms . 285 7.3 Second improvement: generalized key regeneration for eg keys . 288 7.3.1 Principles . 288 7.3.2 Improvements . 289 7.4 Third improvement: asymmetric algorithms . 298 7.4.1 Principles . 298 7.4.2 Improvements . 307 7.5 Classification of backdoors . 313 7.5.1 Backdoors to rsa key generation . 313 7.5.2 Backdoors to eg key generation . 315 7.5.3 Location of the embedding . 315 7.5.4 Symmetry and asymmetry . 316 7.6 Chapter notes . 317 8 Conclusion . 318 Appendix A: Basic cryptology . 325 A.1 Contemporary secure key lengths . 325 A.2 The ElGamal cryptosystem . ..