Heuristic Optimization of Boolean Functions and Substitution Boxes for Cryptography by Linda Burnett Previous qualifications { Bachelor of Applied Science (Honours) 1997 Thesis submitted in accordance with the regulations for Degree of Doctor of Philosophy Information Security Institute Faculty of Information Technology Queensland University of Technology 2005 ii Keywords boolean function, substitution box, heuristic techniques, Genetic Algorithm, Hill Climbing, cryptographic properties, autocorrelation, balance, nonlinearity, corre- lation immunity, resilience, propagation criteria iii iv Abstract Fundamental to the electronic security of information and communication sys- tems, is the correct use and application of appropriate ciphers. The strength of these ciphers, particularly in their ability to resist cryptanalytic attacks, di- rectly influences the overall strength of the entire system. The strength of the underlying cipher is reliant upon a robust structure and the carefully designed interaction between components in its architecture. Most importantly, however, cipher strength is critically dependent on the strength of the individual compo- nents of which it is comprised. Boolean functions and substitution boxes (s-boxes) are among the most com- mon and essential components of ciphers. This is because they are able to provide a cipher with strengthening properties to resist known and potential cryptanalytic attacks. Thus, it is not surprising that significant research effort has been made in trying to develop ways of obtaining boolean functions and substitution boxes with optimal achievable measures of desirable cryptographic properties. Three of the main cryptographic properties required by strong boolean functions and s-boxes are nonlinearity, correlation immunity and propagation criteria, with different cryptographic applications requiring different acceptable measures of these and other properties. As combinations of cryptographic properties exhibited by func- tions can be conflicting, finding cryptographically strong functions often means that a trade-off needs to be made when optimizing property values. Throughout this thesis, the term \optimization" specifically refers to seeking to obtain the best achievable combination of target property values which may be exhibited by boolean functions and s-boxes, regardless of whether the relevant properties are conflicting or complementary. This thesis focusses on a particular class of techniques for obtaining strong functions for cryptographic applications, referred to as heuristic methods or, sim- ply, heuristics. Three new heuristic methods, each aimed at generating boolean functions optimizing one or more of the main cryptographic properties mentioned v above, in addition to other desirable properties, are presented. The first of the new heuristic methods developed for this thesis focusses on generating boolean functions which are balanced and exhibit very high nonlinearities. Highly non- linear balanced functions are critical to many cryptographic applications, as they provide good resistance to linear cryptanalytic attacks. This first method is based on the recursive modification of a starting bent function and is shown to be highly successful and efficient at generating numerous such functions, which also exhibit low autocorrelation values, in a very short computational time. The generation of balanced, correlation immune boolean functions that also exhibit the conflicting property of high nonlinearity is the focus of the second new heuristic method developed for this thesis. By concatenating selected pairs of lower-dimensional boolean functions together in the Walsh Hadamard trans- form domain, direct optimization for both resilience and nonlinearity was able to take place at each level towards and for the final function. This second method was able to generate examples of boolean functions with almost all of the best known optimal combinations of target property values. Experiments have shown the success of this method in consistently generating highly nonlinear resilient boolean functions, for a range of orders of resilience, with such functions possess- ing optimal algebraic degree. A third new heuristic method, which searches for balanced boolean functions which satisfy a non-zero degree of propagation criteria and exhibit high nonlin- earity, is presented. Intelligent bit manipulations in the truth table of starting functions, based on fundamental relationships between boolean function trans- forms and measures, provide the design rationale for this method. Two new function generation schemes have been proposed for this method, to efficiently satisfy the requirements placed on the starting functions utilized in the compu- tational process. An optional process attempts to increase the algebraic degree of the resulting functions, without sacrificing the optimalities that are achiev- able. The validity of this method is demonstrated through the success of various experimental trials. Switching the focus from single output boolean functions to multiple out- put boolean functions (s-boxes), the effectiveness of existing heuristic techniques (namely Genetic Algorithm, Hill Climbing Method and combined Genetic Algo- rithm/Hill Climbing) in primarily being applied to improve the nonlinearity of s-boxes of various dimensions, is investigated. The prior success of these heuristic vi techniques for improving the nonlinearity of boolean functions has been previously demonstrated, as has the success of hill climbing in isolation when applied to bi- jective s-boxes. An extension to the bijective s-box optimization work is presented in this thesis. In this new research, a Genetic Algorithm, Hill Climbing Method and the two in combination are applied to the nonlinearity and autocorrelation optimization of regular NxM s-boxes (N > M) to investigate the effectiveness and efficiency of each of these heuristics. A new breeding scheme, utilized in the Genetic Algorithm and combined Genetic Algorithm/Hill Climbing trials, is also presented. The success of experimental results compared to random regular s-box generation is demonstrated. New research in applying the Hill Climbing Method to construct NxM s- boxes (N < M) required to meet specific property criteria is presented. The consideration of the characteristics desired by the constructed s-boxes largely dictated the generation process. A discussion on the generation process of the component functions is included. Part of the results produced by experimental trials were incorporated into a commonly used family of stream ciphers, thus further supporting the use of heuristic techniques as a useful means of obtaining strong functions suitable for incorporation into practical ciphers. An analysis of the cryptographic properties of the s-box used in the MARS block cipher, the method of generation and the computational time taken to obtain this s-box, led to the new research reported in this thesis on the generation of MARS-like s-boxes. It is shown that the application of the Hill Climbing Method, with suitable requirements placed on the component boolean functions, was able to generate multiple MARS-like s-boxes which satisfied the MARS s- box requirements and provided additional properties. This new work represented an alternative approach to the generation of s-boxes satisfying the MARS s- box property requirements but which are cryptographically superior and can be obtained in a fraction of the time than that which was taken to produce the MARS s-box. An example MARS-like s-box is presented in this thesis. The overall value of heuristic methods in generating strong boolean functions and substitution boxes is clearly demonstrated in this thesis. This thesis has made several significant contributions to the field, both in the development of new, specialized heuristic methods capable of generating strong boolean functions, and in the analysis and optimization of substitution boxes, the latter achieved through applying existing heuristic techniques. vii viii Contents Keywords iii Abstract v Declaration xvii Previously Published Material xix Acknowledgements xxi 1 Introduction 1 1.1 Objectives and Outcomes . 3 1.2 Structure of Thesis . 6 2 Review of Boolean Function and S-Box Theory 11 2.1 Boolean Function Theory . 11 2.1.1 Characteristics of Boolean Functions . 12 2.1.2 Cryptographic Properties of Boolean Functions . 18 Balance . 18 Nonlinearity . 19 Avalanche . 24 Correlation Immunity . 30 2.1.3 Relationships Between Cryptographic Properties of Boolean Functions . 32 Nonlinearity and Avalanche . 32 Nonlinearity and Correlation Immunity . 35 Correlation Immunity and Avalanche . 36 2.1.4 Some Special Boolean Functions . 37 Bent Functions . 37 ix Semi-Bent Functions . 38 Plateaued Functions . 39 2.2 S-Box Theory . 40 2.2.1 S-Box Definitions and Types . 41 2.2.2 Cryptographic Properties of S-Boxes . 42 2.3 Some Common Cryptanalytic Attacks on Cipher Systems . 44 2.3.1 Differential Cryptanalysis . 45 2.3.2 Linear Cryptanalysis . 47 2.3.3 Correlation Attacks . 50 2.4 Summary . 51 3 Heuristic Techniques 53 3.1 Overview of Existing Heuristic Techniques Used . 54 3.1.1 Hill Climbing . 55 Experimental Rationale . 56 Previously Reported Results . 58 Method Applicability . 59 3.1.2 Genetic Algorithms . 59 Experimental Rationale . 62 Previously Reported Results . 63 Method Applicability . 64 3.1.3 Combined Genetic Algorithm and Hill Climbing . 64 Experimental Rationale . 65 Previously Reported Results . 66 Method Applicability . 66 3.2 Summary . 67 4 The Development