Rune Steinsmo Ødegård Hash Functions and Gröbner Bases Cryptanalysis Thesis for the degree of Philosophiae Doctor Trondheim, April 2012 Norwegian University of Science and Technology Faculty of Information Technology, Mathematics and Electrical Engineering Department of Telematics NTNU Norwegian University of Science and Technology Thesis for the degree of Philosophiae Doctor Faculty of Information Technology, Mathematics and Electrical Engineering Department of Telematics © Rune Steinsmo Ødegård ISBN 978-82-471-3501-3 (printed ver.) ISBN 978-82-471-3502-0 (electronic ver.) ISSN 1503-8181 Doctoral theses at NTNU, 2012:111 Printed by NTNU-trykk Abstract Hash functions are being used as building blocks in such diverse primitives as commitment schemes, message authentication codes and digital signatures. These primitives have important applications by themselves, and they are also used in the construction of more complex protocols such as electronic voting systems, online auctions, public-key distribution, mutual authentication hand- shakes and more. Part of the work presented in this thesis has contributed to the \SHA-3 contest" for developing the new standard for hash functions orga- nized by the National Institute of Standards and Technology. We constructed the candidate Edon-R, which is a hash function based on quasigroup string transformation. Edon-R was designed to be much more efficient than SHA-2 cryptographic hash functions, while at the same time offering same or better security. Most notably Edon-R was the most efficient hash function submit- ted to the contest. Another contribution to the contest was our cryptanalysis of the second round SHA-3 candidate Hamsi. In our work we studied Hamsi's resistance to differential and higher-order differential cryptanalysis, with focus on the 256-bit version of Hamsi. Our main results are efficient distinguishers and near-collisions for its full (3-round) compression function, and distinguishers for its full (6-round) finalization function, indicating that Hamsi's building blocks do not behave ideally. Another important part of this thesis is the application of Gr¨obnerbases. In the last decade, Gr¨obnerbases have shown to be a valuable tool for algebraic cryptanalysis. The idea is to set up a system of multivariate equations such that the solution of the system reveals some secret information of the crypto- graphic primitive. The system is then solved with Gr¨obnerbases computation. Staying close to the topic of hash functions, we have applied this tool for crypt- analysis and construction of multivariate digital signature schemes, which is a major hash function application. The result of this is our cryptanalysis of the public-key cryptosystem MQQ, where we show exactly why the multivariate quadratic equation system is so easy to solve in practice. The knowledge we iv HASH FUNCTIONS AND GROBNER¨ BASES CRYPTANALYSIS gained from finding the underlying weakness of the MQQ scheme was used to construct a digital signature scheme. The resulting scheme, MQQ-SIG, is a provably CMA resistant multivariate quadratic digital signature scheme based on multivariate quadratic quasigroups. The scheme is designed to be very fast both in hardware and in software. Compared to some other multivari- ate quadratic digital signature schemes, MQQ-SIG is much better in signing and private key size, while worse in key generation, verification and public key size. This means that MQQ-SIG is a good alternative for protocols where the constrained environment is on the side of the signer. Preface This thesis is submitted in partial fulfillment of the requirements for the de- gree of philosophiae doctor (PhD) at the Norwegian University of Science and Technology (NTNU). The work was performed at the Centre for Quantifiable Quality of Service in Communication Systems (Q2S), Centre of Excellence (CoE), during 2007-2011, and has been supervised by Professor Svein Johan Knapskog and Professor Bjarne Emil Helvik. The document has been formatted in LATEX using a modified version of the document class kapproc.cls provided by Kluwer Academic Publishers. vi HASH FUNCTIONS AND GROBNER¨ BASES CRYPTANALYSIS Acknowledgements First, I would like to thank my supervisor Professor Svein Johan Knapskog for his support throughout my work on this thesis. A particular thanks also goes out to Professor Danilo Gligoroski who, in addition to have coauthored most of my papers, has in practice been my co-supervisor for this thesis. I would also like to thank Research Director Jean-Charles Faug`ere,Assistant Professor Ludovic Perret and the rest of the SALSA-team for hosting me for 7 months. I learned a lot during my stay at the Pierre and Marie Curie University. I also learnt a lot from collaborating with all my coauthors. In addition to the four people mentioned so far I would like to thank Jean-Philippe Au- masson, AleˇsDr´apal,Rune Erlend Jensen, Emilia K¨asper, Vlastimil Klima, Lars Ramkilde Knudsen, Ljupco Kocarev, Smile Markovski, Marija Mihova and Thomas Peyrin for our joint papers. I have made good friends and met a lot of very interesting people during my four years at Q2S. I would like to thank my colleagues for making Q2S a very pleasant environment to work in. Finally, I would like to thank my friends and family. Their continued sup- port has meant a lot to me. viii HASH FUNCTIONS AND GROBNER¨ BASES CRYPTANALYSIS Contents Abstract iii Preface v Acknowledgements vii List of Papers xiii Abbreviations xv Part I Thesis Introduction 1 Introduction 3 2 Background 3 3 Research Goals 31 4 Research Methodology 33 5 Contributions 34 6 Results and Discussion 38 7 Summary and Conclusion 42 Part II Included Papers PAPER A: On Some Properties of Boolean Matrices from Latin Squares 49 Rune Steinsmo Ødeg˚ard, Marija Mihova, Danilo Gligoroski 1 Introduction 49 2 Preliminaries 50 3 Concrete example 52 4 Results 53 5 Open questions 55 6 Conclusion 55 References 55 PAPER B: Cryptographic Hash Function Edon-R 59 Danilo Gligoroski, Rune Steinsmo Ødeg˚ard, Marija Mihova, Svein Johan Knapskog, Ljupco Kocarev, AleˇsDr´apal, Vlastimil Klima 1 Algorithm Specifics 61 2 Description of the Hash Algorithm Edon-R 69 3 Design Rationale 82 4 Estimated Computational Efficiency and Memory Requirements 117 x HASH FUNCTIONS AND GROBNER¨ BASES CRYPTANALYSIS References 121 PAPER C: On the Randomness and Regularity of Reduced Edon-R Compression Function 127 Rune Steinsmo Ødeg˚ard, Danilo Gligoroski 1 Introduction 127 2 Background 128 3 Analysis of 8-bit Edon-R 130 4 Analysis of 16-bit Edon-R 134 5 Conclusion 136 References 136 PAPER D: Distinguishers for the Compression Function and Output Transforma- tion of Hamsi-256 141 Jean-Philippe Aumasson , Emilia K¨asper , Lars Ramkilde Knudsen, Krystian Ma- tusiewicz, Rune Steinsmo Ødeg˚ard, Thomas Peyrin, Martin Schl¨affer 1 Introduction 142 2 Description of Hamsi-256 143 3 Higher-order differential analysis 146 4 First order differential analysis 151 5 Non-randomness of the ouput transformation 155 6 Conclusion 158 7 Acknowledgements 159 References 159 Appendix: The Sbox of Hamsi 162 Appendix: Conforming Message Pairs 163 Appendix: Truncated differential path 164 PAPER E: Analysis of the MQQ Public Key Cryptosystem 167 Jean-Charles Faug`ere, Rune Steinsmo Ødeg˚ard, Ludovic Perret, Danilo Gligoroski 1 Introduction 168 2 Description of the MQQ public key cryptosystem 169 3 Gr¨obnerbases 172 4 Why MQQ is Susceptible to Algebraic Cryptanalysis 175 5 Weaknesses of MQQ 177 6 Conclusion 179 References 180 Appendix: Algorithm for generating random MQQ 184 PAPER F: MQQ-SIG An Ultra-fast and Provably CMA Resistant Digital Signature Scheme 187 Danilo Gligoroski , Rune Steinsmo Ødeg˚ard , Rune Erlend Jensen , Ludovic Perret , Jean-Charles Faug`ere, Svein Johan Knapskog, Smile Markovski 1 Introduction 188 2 Preliminaries - Quasigroups and Multivariate Quadratic Quasigroups 190 3 Description of the MQQ-SIG Digital Signature Scheme 192 4 Design Rationale 194 5 Security analysis of the algorithm 199 6 Operating characteristics 207 Contents xi 7 Conclusions 209 References 209 Bibliography 213 List of Papers Publications Included in the Thesis The following are the papers included in Part II of the thesis. All papers have been subjected to minor editorial changes before their inclusion. PAPER A: Rune Steinsmo Ødeg˚ard,Marija Mihova and Danilo Gligoroski. On Some Properties of Boolean Matrices from Latin Squares. In Proceed- ings of the 1st International Workshop on Security and Communication Networks (IWSCN 2009). Trondheim, Norway. May 20-22, 2009 PAPER B: Danilo Gligoroski, Rune Steinsmo Ødeg˚ard,Marija Mihova, Svein Jo- han Knapskog, Ljupco Kocarev, AleˇsDr`apal,Vlastimil Klima. Crypto- graphic Hash Function Edon-R. Submission to NIST 2008. PAPER C: Rune Steinsmo Ødeg˚ardand Danilo Gligoroski On the Randomness and Regularity of Reduced Edon-R Compression Function. In Proceedings of the 2009 International Conference on Security & Management (SAM 2009). Las Vegas, Nevada, USA. July 13-16, 2009. PAPER D: Jean-Philippe Aumasson, Emilia K¨asper, Lars Ramkilde Knudsen, Krys- tian Matusiewicz, Rune Steinsmo Ødeg˚ard,Thomas Peyrin and Mar- tin Schl¨affer. Distinguishers for the Compression Function and Output Transformation of Hamsi-256. In Proceedings of the 15th Australian Conference on Information Security and Privacy (ACISP 2010). Syd- ney, Australia. July 5-7, 2010. PAPER E: Jean-Charles Faug`ere, Rune Steinsmo Ødeg˚ard,Ludovic Perret and Danilo Gligoroski. Analysis of the MQQ Public Key Cryptosystem. In xiv HASH FUNCTIONS AND GROBNER¨ BASES CRYPTANALYSIS Proceedings of the 9th International Conference on Cryptology and Network Security (CANS 2010). Kuala Lumpur, Malaysia. December 12-14, 2010. PAPER F: Danilo Gligoroski, Rune Steinsmo Ødeg˚ard,Rune Erlend Jensen, Lu- dovic Perret, Jean-Charles Faug`ere,Svein Johan Knapskog and Smile Markovski. MQQ-SIG. An Ultra-fast and provably CMA Resistant Digi- tal Signature Scheme. In Proceedings of the 3rd International Conference on Trusted Systems, INTRUST 2011. Beijing, China. November 27-29, 2011. Other Papers by the Author The following papers were also prepared while working with this thesis.