Cryptanalysis of RCPlike Stream Ciphers by Serge Mister A thesis submitted to the Department of EIectrical and Computer Engineering in conformity with the requirements for the degree of Master of Science (Engineering) Queen7sUniversity Kingston, Ontario, Canada May 1998 Copyright (ZJ Serge Mister, 1998 National Library Bibliothèque nationale 1+1 .cm, du Canada Acquisitions and Acquisitions et Bibliographie Services services bibliographiques 395 Wellington Street 395. nie Wellington OttawaON K1AON4 Ottawa ON KI A ON4 Canada Canada The author has granted a non- L'auteur a accordé une licence non exclusive licence allowing the exclusive permettant à la National Library of Canada to Bibliothèque nationale du Canada de reproduce, loan, distribute or sell reproduire, prêter, distribuer ou copies of this thesis in microform, vendre des copies de cette thèse sous paper or electronic formats. la forme de microfiche/fïlm, de reproduction sur papier ou sur format électronique. The author retains ownership of the L'auteur conserve la propriété du copyright in this thesis. Neither the droit d'auteur qui protège cette thèse. thesis nor substantial extracts fkom it Ni la thèse ni des extraits substantiels may be printed or otherwise de celle-ci ne doivent être imprimés reproduced without the author's ou autrement reproduits sans son permission. autorisation. Abstract Cryptography is one important building block used in communication systems to provide confidentiality and authenticity Stream ciphers, ciphers which encrypt data one bit or a few bits at a tirne. have been used for many years in environments where low delay and high speed are a requirement. Until recentl- most stream ciphers were generally constructed using a set of linear feedback shift registers and a nonlinear combiner. Although these offer advantages in terms of ease of analysis and efficient hardware implementation, t heir low performance in software settings and the increasing success of correlation attacks has lead to the proposal of software-oriented stream ciphers. RClo a stream cipher designed by Rivest for RSA Data Security Inc., has found several commercial applications. but little public analysis has been done to date. In this thesis, alleged RC4 (hereafter called RC4') is described and existing analysis outlined. The properties of RCl: and in particular its cycle structure, are discussed. Several variant s of a basic "tracking" attack are analysed, providing experimental results for scaled-dom versions of RC4. For euample, the state of a 5 bit RCClike cipher can be obtained from a portion of the keystream using 242 steps, while the nominal keyspace of the systern is 2l6'. In addition to presenting an interpretation of the results, the thesis contains experimental data offering insight into the RC4 algorithm and its security. This analysis shows that. although the full-size RC4 remains secure against known atta&; keystreams are distinguishable from randomly generated bit streams, and the RC4 key can easily be recovered if a significant fraction of the full cycle of keystream bits is generated. The tracking attacks discussed provide a significant irnprovement over the exhaustive search of the full RC4 keyspace. More work is necessary to make these attacks practical in the case where a reduced keyspace is used. Because the expected RC4 cycle length is vast and only a small portion of the full RC4 keyspace is used in practice, such facts may not pose an immediate concern. lU;hile RC4 remains a trade secret of RSA Data Security Inc., the aigorithm described in [25] is believed to be output-compatible with RC4. This thesis discusses the algorithm given in [25], and is referred to as RC4 for convenience. Acknowledgment s I would like to thank rny supervisor. Dr. S. E. Tavares, for his supervision and guidance throughout this project . Thanks also to Mike Wiener for his helpful suggestions, and to the Natural Sciences and Engineering Research Council (XSERC). the School of Graduate Studies and Research of Queen's University and Communications and Information Technologv Ontario (CITO) for their financial support of this work. Contents Abstract Acknow ledgment s Contents List of Tables v List of Figures vi List of Symbols vii I Introduction 1 1.1 Overview and Motivation ......................... 1 1.2 General Cryptographic Systems ..................... 3 1.3 Stream Ciphers .............................. 5 1.3.1 Linear Feedback Shift Register Based Designs ......... 6 .-. 1.3.2 Other Designs ........................... i 1.4 Propert ies of Stream Ciphers ....................... 7 1.5 RC4 Stream Cipher ............................ 8 1.6 Background on Cycle Structures ..................... 11 2 Literature Review 13 2.1 Stream Cipher Proposais ......................... 13 2.1.1 Linear Feedback Shift Register Based Stream Ciphers ..... 13 2.1.2 SEAL ...............................15 2.2 Cryptanalysis of Stream Ciphers ..................... 15 2.3 Cryptanalysis of RC4 ........................... 16 2.3.1 Sci. Crypt Discussion Sumrnary ................. 17 2.3.2 A class of weak keys in RC4 ................... 18 2.3.3 Linear Statistical Weaknesses in RC4 .............. 20 2.3.4 Cornments Gom RSA ....................... 21 3 Periodicity Analysis of RC4 24 3.1 Charactenzation of RCI Cycles ................. 24 3.2 A Partitioning of RC1 Cycles ...................... 28 4 Systematic Cryptanalysis of RC4 36 1.1 ObtainingaKeyfromtheRC4State .................. 36 -1.2 Fornard Tracking Attack ......................... 37 4.3 Efficient Starting Points for Forward Tracking ............. 44 1.1 Backtracking Attack ........................... 18 4.5 Probabilistic Xttacks ........................... 49 4.5.1 Probabilistic Mode1 of RC4 ................... 50 4.5.2 Laddered Backtracking Attack .................. 51 1.5.3 Most Probable Path Backtracking &ta& ............ 54 4.6 Practicd Attach in Lnusual Scenanos ................. 39 4.7 Sumrnary of Attack Performance .................... 61 5 Conclusion 62 3.1 SummaryandDiscussion ......................... 62 5.2 Suggestions for Further Study ...................... 63 APPENDICES 69 A Short Cycles of RC4-3 69 -4 .1 -1 Cycle of Length 24 ........................... 70 A.2 -4 Second Cycle of Length 24 ...................... 71 A.3 -4 Third Cycle of Length 21 ....................... 72 A.4 -4 Fourth Cycle of Length 24 ....................... 73 B Detailed Key Distribution for n = 2 and n = 3 74 B.1 Key Distribution for n = 2 ........................ 75 B.2 Key Distribution for n = 3 ........................ 83 C Forward Tracking Algorithm in Detail 93 D Probabilistic Information in RC4-4 97 Vit a List of Tables Nominal and Effective Key Sizes for RC3-n . 10 Deilation of RC4 Gap Lengths from those of Random Keystream . 18 Possible Periods for RC1 with Word Length 2 and 3 . 26 Expected Cycle Lengths for a Randomly Chosen Invertible Mapping . 27 Cycle Lengths for a Random Permutation With 2580480 elements (n= 3) 29 Solutions Found by Fonvard Tracking for a Nonzero Keystream . 41 Solutions Found by Fonvard Tracking for a Zero Keystream . 41 Solutions Found by Forward Tracking for a Zero Keystream (approx.) 13 Complexity of the Fomwd Tracking Algorithm . 45 Time to Recover Key, Soned by Path Count (n = 4) . - . 46 Average Fomard Tracking Search Time . 46 Keystreams bhirnizing 'iode Count in the Forward Tracking Attack 48 Solutions Found by Backtracking for a Nonzero Keystrearn . 49 Guess Success Probability of Most Probable Depth 6 Path Backtrack- ing-4ttack.. 59 Complexity of Most Probable Depth 6 Path Backtracking Attack . 60 Time to Recover RCCn Key with Multiple Keystreams . Estimated Upper Bound on the "'T'rue" Keyspace of RCCn . RC4-4 State Sequence for an Initial Key k = {O, 1,1,1) . RC14 State Probability Matrix - 1 Keystream Byte -4nalysed . RC4-4 State Probability Matrix - 3 Keystream Bytes Anaiysed . RC4-4 State Probability Matrix - 6 Keystream Bytes -4nalysed . RC4-4 State Probability Matrix - 9 Keystream Bytes Analysed . RCM State Probability Matrix - 10 Keystream Bytes -4nalysed RC44 State Probability Matrix - 11 Keystream Bytes -4nalysed RC4-4 State Probability Matrix - 12 Keystream Bytes -2nalysed RC4-4 State Probability Yatrix - 13 Keystream Bytes halysed List of Figures - The General Structure of a Stream Cipher . - . - . - - - 3 A Linear Feedback Shift Register Based Stream Cipher . - . - - 6 The -4lleged RC4 Stream Cipher . 10 The Cycle Map for the Function f (z) = x2 + 7 mod 31 . 11 The Cycle Map for the Function f (z) = xi + 7 mod 31 . - 12 LFSR Based Stream Cipher Examples . - 14 RCC3 Cycle Distribution Compared with a Random Permutation . 28 The Right Shift Operation for an RC4n State . 30 Partitioning of an RC4 Cycle . 34 Fonvard Tracking Algorithm for n = 2. and a O Keystream of Length 4 40 Xumber of Nodes Visited During Forward Trachg . 44 Time to Recover Key. Sorted by Node Count at Depths 1 and 2 (n = 4) 45 Time to Recover Key, Sorted by Node Count at Depths 3 and 1 (n = 4) 47 Time to Recover Key. Sorted by Node Count at Depths 5 and 6 (n = 4) 47 Laddered Backtracking Attack . - - - 52 Tree Diagram for blost Probable Depth 2 Path Backtracking Attack . 55 List of Symbols The ith word of the ciphertext. See the definition of Ki for more e'cplanation. If D is an RC4 state (see the entq for (2'' j'? SI)): the state obtained by initialking RC4 to state D and performing t encryptions. (2r.jr. Sf) .ln RC1 state defined by i = ir.j = j'. S = Sr. (rjr-{ - - Sn}An RC4 state defined by i = ir. j = j', S = {SA, - : s;*-l}- In descriptions of RCClike algorithms, the counter i which is part of the state of RC4. It is also used as a general count er when no confusion would result . In descriptions of RC41ike algorithms, the counter j which is part of the state of RC4. It is also used as a general counter mhen no confusion would result. The ith word of the key used to initialize RCglike algo- rithms. This key is of length 1- and the words of the key are ko,. ? kr-l. The ith word of the generated keystream. In an RCP like system, this word is XORed with the corresponding plaintext word Pito produce the ciphertext word Ci.