A Methodology for the Cryptanalysis of Classical Ciphers with Search
Total Page:16
File Type:pdf, Size:1020Kb
phers with Search Metaheuristics George Lasry A Methodology for the Cryptanalysis of A Methodology for the Cryptanalysis of Classical Ci Classical Ciphers with Search Metaheuristics ISBN 978-3-7376-0458-1 kassel university 9 783737 604581 George Lasry press kassel kassel university press !"# $ % !&' (&)) )*) # + ,)&) - .# +,)/ & + 0123405 / ! & ' & ' & ' 6 # 7 + ))) 8)+$ 9"#)9& )3405 7/':5;.<.5<51.4=>;.0(* 7/':5;.<.5<51.4=>:.;(.* &27++ ?)! ) 04)0:300 $",:5;<5<514=>:; "@'++ . #) +++4443.=4=>:3 '340;9 # !B9$ ) ). ) ,! “After climbing a great hill, one only finds that there are many more hills to climb.” Nelson Mandela Abstract Cryptography, the art and science of creating secret codes, and cryptanalysis, the art and science of breaking secret codes, underwent a similar and parallel course during history. Both fields evolved from manual encryption methods and manual codebreaking techniques, to cipher ma- chines and codebreaking machines in the first half of the 20th century, and finally to computer- based encryption and cryptanalysis from the second half of the 20th century. However, despite the advent of modern computing technology, some of the more challenging classical cipher systems and machines have not yet been successfully cryptanalyzed. For others, cryptanalytic methods exist, but only for special and advantageous cases, such as when large amounts of ciphertext are available. Starting from the 1990s, local search metaheuristics such as hill climbing, genetic algorithms, and simulated annealing have been employed, and in some cases, successfully, for the cryptanal- ysis of several classical ciphers. In most cases, however, results were mixed, and the application of such methods rather limited in their scope and performance. In this work, a robust framework and methodology for the cryptanalysis of classical ciphers using local search metaheuristics, mainly hill climbing and simulated annealing, is described. In an extensive set of case studies conducted as part of this research, this new methodology has been validated and demonstrated as highly effective for the cryptanalysis of several challenging cipher systems and machines, which could not be effectively cryptanalyzed before, and with drastic improvements compared to previously published methods. This work also led to the decipherment of original encrypted messages from WWI, and to the solution, for the first time, of several public cryptographic challenges. v Acknowledgements I would like to thank my supervisor, Prof. Arno Wacker, for his ongoing support, helping me to transform a series of experiments into a full-fledged academic research, for his infinite patience, and his invaluable advice throughout the whole process. I would also like to thank Prof. Bernhard Esslinger for his advice and thorough review, not only catching embarrassing typos and mistakes, but also helping me better formulate some of the key ideas, and filling many gaps. I would also like to thank my (other) co-authors and partners: Nils Kopal, for his ideas, support, good spirit and constant willingness to help, Dr. Ingo Niebel who contributed a well-researched, fascinating historical context, and an in-depth analysis of the original WWI messages deciphered as part of this research, as well as Moshe Rubin, for introducing me to the Chaocipher problem, and for providing a wealth of critical background information. I am also grateful to the managers of the MysteryTwister C3 site (Prof. Esslinger, Wacker and May) for building and maintaining a set of high-quality, stimulating and inspiring crypto challenges. Those include the Double Transposition Challenge, for which I wish to thank Klaus Schmeh, as the work on his tough challenge and its solution generated new ideas that could be applied to additional problems. I also thank Dirk Rijmenants for his highly informative website, his crypto challenges which were the first I tried to solve, and for his kind assistance. I also thank Jean-Franc¸ois Bouchaudy, the owner of the Hagelin M-209 Challenge site, for creating a series of challenging and also entertaining problems, which I used as a testbed for developing new methods. I also thank Jean-Franc¸ois for his ongoing support, pushing me not to give up, and to try and break new limits in the cryptanalysis of the Hagelin M-209, until I was able to complete all of his challenges. I also thank Frode Weierud, Jim Gillogly, Jim Reeds, Geoff Sullivan, Craig Bauer, Paul Gannon, Rene Stein and others to whom I apologize for having omitted, for their assistance in my research in numerous ways, such as providing copies of key documents, and bridging important gaps. I also thank Paul Reuvers and Marc Simons for their wonderful Crypto Museum site, and for their permission to reproduce their photos and diagrams of Enigma. And last but not least, I would like to thank my loving wife Tali, for her ongoing moral and emotional support, and for allowing me to seclude myself for long hours in front of the computer, as well as my wonderful kids, Noam and Shay, who fill every day of my life with happiness and pride for being their father. vi In memory of my parents, Marie and David Lasry and dedicated to my family, my deepest source of inspiration to my wife Tali, my son Noam and my daughter Shay and to the Lasry, Feldman, Prizant, and Bocian families Contents Abstract v Acknowledgements vi Contents viii List of Figures xv List of Tables xvii Abbreviations xix 1 Introduction 1 1.1 Classical Cryptography .............................. 1 1.2 The Development of Cryptanalysis for Classical Ciphers ............ 2 1.3 Cryptanalysis of Classical Ciphers as an Optimization Problem ......... 4 1.4 Challenges ..................................... 6 1.5 Relevance of the Work .............................. 8 1.6 Contributions ................................... 9 1.6.1 Contribution1–ANewMethodology for the Efficient Cryptanalysis of Classical Ciphers using Local Search Metaheuristics .......... 9 1.6.2 Contribution2–NewEffective Cryptanalytic Attacks on Several Chal- lenging Classical Ciphers or Cipher Settings .............. 9 1.6.3 Contribution 3 – Decipherment of Historical Documents and Solutions for Cryptographic Challenges ...................... 10 1.7 Structure of the Thesis .............................. 11 2 Stochastic Local Search 13 2.1 Combinatorial Problems ............................. 13 2.2 Search Algorithms ................................ 15 2.2.1 Search Algorithms for Hard Combinatorial Problems .......... 16 2.2.2 Perturbative vs. Constructive Search ................... 16 2.2.3 Systematic vs. Local Search ....................... 16 2.3 Stochastic Local Search .............................. 17 2.3.1 Overview of Stochastic Local Search .................. 17 2.3.2 Evaluation Functions ........................... 18 2.3.3 Iterative Improvement .......................... 18 2.3.4 Intensification vs. Diversification .................... 19 ix Contents x 2.3.5 Large vs. Small Neighborhoods ..................... 20 2.3.6 Best Improvement vs. First Improvement ................ 20 2.3.7 Probabilistic vs. Deterministic Neighbor Selection ........... 21 2.3.8 Single Candidate vs. Population of Candidates ............. 21 2.3.9 Smooth vs. Rugged Search Landscape .................. 22 2.3.10 Fitness-Distance Correlation ....................... 25 2.3.11 Local Search Metaheuristics vs. Local Search Algorithms . .... 26 3 Cryptanalysis of Classical Ciphers using Local Search Metaheuristics 27 3.1 Cryptanalysis as a Combinatorial Problem .................... 27 3.2 Scoring Functions for Cryptanalysis Problems .................. 28 3.2.1 Introduction ................................ 29 3.2.2 Scoring Functions for Known-Plaintext Attacks ............. 29 3.2.3 Scoring Functions for Ciphertext-Only Attacks ............. 29 3.2.4 Selectivity vs. Resilience to Key Errors ................. 33 3.2.5 The Unicity Distance and Scoring Functions .............. 35 3.2.6 Extended Definitions of the Unicity Distance .............. 37 3.3 Hill Climbing for Classical Ciphers ....................... 38 3.4 Simulated Annealing for Classical Ciphers .................... 39 3.5 Related Work ................................... 41 3.5.1 Ciphertext-Only Cryptanalysis of Enigma ................ 41 3.5.2 Ciphertext-Only Cryptanalysis of Purple ................. 43 3.5.3 Ciphertext-Only Cryptanalysis of Playfair ................ 44 3.5.4 Other Related Work ........................... 45 3.5.5 Cryptanalysis of Modern Ciphers using Local Search Metaheuristics . 47 4 A New Methodology 49 4.1 Motivation ..................................... 49 4.2 Overview of the Methodology Principles ..................... 50 4.3 GP1: Hill Climbing or Simulated Annealing ................... 51 4.3.1 Parallel Search Processes ......................... 52 4.3.2 Nested Search Processes ......................... 53 4.3.3 Sequential Search Processes – Different Key Parts ........... 54 4.3.4 Sequential Search Processes – Applied on the Whole Key ....... 54 4.3.5 Summary of GP1............................. 55 4.4 GP2: Reduction of the Search Space ....................... 55 4.5 GP3: Adaptive Scoring .............................. 56 4.6 GP4: High-Coverage Transformations Preserving a Smooth Landscape . 57 4.7 GP5: Multiple Restarts with Optimal Initial Keys ................ 59 4.8 Conclusion .................................... 60 5 Case