Transformation and Security Analysis of NLFSR-based Stream Ciphers by Ge Yao ORCID: 0000-0003-4523-8816 A thesis submitted in total fulfillment for the degree of Doctor of Philosophy in the School of of Computing and Information Systems THE UNIVERSITY OF MELBOURNE March 2021 i Abstract The Nonlinear Feedback Shift Register (NLFSR) based stream cipher is becoming the mainstream design of modern stream ciphers. The properties of high operation speed, small footprint in hardware and low power consumption make such ciphers preferable in resource constrained applications requiring secure communications. In the last decade, many NLFSR-based stream ciphers have been proposed, among which the Grain family ciphers are the most mature and well studied ciphers. However, security concerns hinder the development and application of such ciphers. Cryptanalytic attacks like the Time- Memory-Data Trade-Off (TMDTO) attack requires that the size of the internal state should be at least twice of the security level, which is conflict with the requirement of high efficiency. In order to optimise the trade-off between the performance and security, researchers focus on developing new ideas to design stream ciphers inheriting the effi- ciency of Grain family ciphers but remaining resist to the known attacks especially the TMDTO attack. To this end, new design ideas of using shorter Feedback Shift Registers (FSRs) or deploying single Galois NLFSR spark interest in this field. In this thesis, we aim to analyse the security of the newly designed stream ciphers and explore the theory of NLFSR to make progress in studying the NLFSR-based stream ciphers. This research aims to address four research questions. The four research questions and the corresponding contributions are detailed as follows. The first research question is about the security of small-state stream ciphers. As the initial design of small-state stream ciphers, Sprout is proved to be insecure. Its successors including Plantlet, Fruit and Lizard are also not as secure as expected. In this research, we aim to improve the Sprout cipher against the divide-and-conquer key recovery attack and analyze the security of all the small-state stream ciphers. By analyzing the four types of sieving and merging techniques used in the key recovery attack, we identify the design weakness in Sprout. Then we propose countermeasures to resist each type of the sieving and merging techniques. Five experiments are conducted to verify our theoretical improvements. The results of the first four experiments show that our countermeasures are effective and the result of the last experiment shows that the improved cipher resists the key recovery attack. Moreover, we analyze the attack results on Plantlet and Fruit and find that the countermeasures we propose are consistent with the improvements made in these. Finally, we summarize the design principles for small-state Sprout-like stream ciphers. The second research question is how to determine whether a Galois NLFSR is equiva- lent to a Fibonacci NLFSR. We refer to those equivalent ones as transformable Galois NLFSRs. The transformation between Fibonacci and Galois NLFSRs have been studied extensively, but still the equivalence is not fully established. To address this issue, we ii adopt the notion nonlinear recurrence and derive the necessary and sufficient condition for a Galois NLFSR to be equivalent to a Fibonacci NLFSR. We prove that the three types of transformable Galois NLFSRs discovered in literature satisfy this condition. Besides, we study several properties of the nonlinear recurrence and discover a special case where a Galois NLFSR is equivalent to two different Fibonacci NLFSRs. The third research question is how to transform an NLFSR between Fibonacci and Galois configurations. For the three types of transformable Galois NLFSRs, either no transformation algorithm has been proposed or the algorithm has very high complexity. There are several limitations and a common issue in existing algorithms. In this research, we aim to address all the issues. First, we give a formal description of a transformation algorithm. Second, we develop a compensation method. The basic idea is to build relations of the internal states of the NLFSR before and after transformation. According to the established relations, it is possible to construct the output function and compute the initial state for the transformed NLFSR. Based on this unified method, we propose transformation algorithms for all the three types of Galois NLFSRs. Moreover, we discover a new type of transformable Galois NLFSRs, namely Type-IV Galois NLFSRs. We show that this new type also satisfies the necessary and sufficient condition proposed to answer the second research question in Chapter 5. Based on the same compensation method, we propose transformation algorithms for the Type-IV Galois NLFSRs. All the proposed algorithms are easy to program and have polynomial time complexity. We provide a pesudocode for each algorithm. The fourth research question is about the security of maximum period Galois NLFSR- based stream ciphers. We reinterpret the design method and identify a conditional equivalence problem. We find that this problem can be addressed by the Type-II- to-Fibonacci transformation algorithm proposed in Chapter 6. Then we apply this algorithm on Espresso cipher. The Galois NLFSR used in the cipher is transformed to a Linear Feedback Shift Register (LFSR) with a nonlinear output function, which is often referred to as an LFSR filter generator. We mount the fast algebraic attack and the Rønjom-Helleseth attack on the transformed cipher and break it with computation complexity of 268:50 and 248:59 logical operations respectively, which is far lower than the claimed security level of 2128. We then show that not only the Galois NLFSR in Espresso cipher, but also the entire class of maximum period Galois NLFSRs can be transformed back to LFSRs with precise output functions. Therefore, this kind of cipher is always equivalent to an LFSR filter generator. We discuss other related attacks and give suggestions for the future design. Declaration of Authorship I, Ge Yao, declare that this thesis titled, `Transformation and Security Analysis of NLFSR-based Stream Ciphers' and the work presented in it are my own. I confirm that: The thesis comprises only my original work towards the degree of Doctor of Phi- losophy; due acknowledgement has been made in the text to all other material used; and the thesis is fewer than the maximum word limit in length, exclusive of tables, maps, bibliographies and appendices as approved by the Research Higher Degrees Committee. Signed: Date: iii Preface The thesis is written at the School of Computing and Information Systems, The Uni- versity of Melbourne, Australia. The main contributions discussed in Chapters 4 to 7 of the thesis are based on the following publications:: Ge Yao, and Udaya Parampalli. "Improve sprout cipher to resist the divide and conquer based key recovery attack." In Proceedings of the Australasian Computer Science Week Multiconference, pp. 1-7. 2018. Ge Yao, and Udaya Parampalli. "Transformation Algorithm for NLFSRs in Hard- ware Oriented Stream Ciphers." International Conference on Sequences and Their Applications, 6-1: pp. 1-14. 2018. Ge Yao, and Udaya Parampalli. "Improved Transformation Algorithms for Gen- eralized Galois NLFSRs." International Conference on Sequences and Their Ap- plications, pp. 1-21. 2020. (The full version of this paper is submitted to Journal Cryptography and Communications Discrete Structures, Boolean Functions and Sequences (CCDS)) Ge Yao, and Udaya Parampalli. "Cryptanalysis of the Class of Maximum Period Galois NLFSR-based Stream Ciphers." Submitted to Journal Cryptography and Communications Discrete Structures, Boolean Functions and Sequences (CCDS). iv Acknowledgements First and foremost, I would like to express my sincere gratitude to my supervisor, Prof. Udaya Parampalli for his continuous support and guidance throughout my Ph.D. study and research. Without his help, I would not have had the chance to come and study at Melbourne. He has always been nice to me and encouraged me in all the phases of my research. I would like to sincerely thank my co-supervisors, Assoc. Prof. Vanessa Teague and Dr. Olga Ohrimenko. I want to thank Vanessa for giving me insightful comments and suggestions. I want to thank Olga who joined my supervision team at the last six months of my Ph.D. journey but provided practically useful feedback to improve my thesis. I also want to sincerely thank my advisory committee chair, Assoc. Prof. Michael Kirley. I enjoyed every progress review meeting because of his enthusiastic support and valuable comments. It is a privilege to have such a great committee. I would like to thank the University of Melbourne and the China Scholarship Council for funding my study and covering my living expenses. I would also like to express thanks to my friends, Yixin, Karin, Estrid, Tabinda, Lianglu and Jiajia who supported me and exchanged valuable ideas with me during the past four years. Last but not least, I would like to thank my parents, my sister and my boyfriend Meng Wang for their unconditional love, support, understanding and encouragement through- out this journey. v Contents Abstract i Declaration of Authorship iii Preface iv Acknowledgementsv List of Figures ix List of Tablesx 1 Introduction1 1.1 Background...................................1 1.1.1 Structure of Stream Cipher......................2 1.1.2 Security of Stream Cipher.......................4