Heterogeneous Rainbow Table Widths Provide Faster Cryptanalyses Gildas Avoine Xavier Carpent INSA Rennes / IRISA Computer Science Department [email protected] University of California, Irvine [email protected] ABSTRACT by Oechslin. Another notable variant is the distinguished Cryptanalytic time-memory trade-offs are techniques intro- points [8] by Rivest in 1982. A recent analysis [12] shows duced by Hellman in 1980 to speed up exhaustive searches. that the rainbow tables are the fastest variant known today. Oechslin improved the original version with the introduction Several improvements on the rainbow tables were published of rainbow tables in 2003. It is worth noting that this variant during the last decade, including the checkpoints [4] and the is nowadays used world-wide by security experts, notably to fingerprints[1], and techniques to optimize the ending points break passwords, and a key assumption is that rainbow ta- storage [2] and to address non-uniform distributions [3]. bles are of equal width. Whatever the considered variant based on rainbow tables, We demonstrate in this paper that rainbow tables are un- time-memory trade-offs consist of tables sharing the same derexploited due to this assumption never being challenged. width. This assumption has never been challenged since the We stress that the optimal width of each rainbow table original publication of rainbow tables. should be individually { although not independently { calcu- We demonstrate in this paper that rainbow tables are un- lated. So it goes for the memory allocated to each table. We derexploited because considering tables of equal width is far also stress that visiting sequentially the rainbow tables is no from being the optimal configuration. We show that the longer optimal when considering tables with heterogeneous width of each table { and so the memory allocated to each widths. of these tables { should be individually (but not indepen- We provide an algorithm to calculate the optimal con- dently) calculated for each table. This approach lead to cre- figuration and a decision function to visit the tables. Our ate so-called \heterogeneous tables", by opposition to \ho- technique performs very well: it makes any TMTO based on mogeneous tables". We also show that the widely-used rule rainbow tables 40% faster than its classical version. that consists in visiting the tables sequentially is not the op- timal one when considering heterogeneous tables. The paper thus shows that heterogeneous tables are about 40% faster Keywords than their homogeneous counterparts. time-memory tradeoff; rainbow tables Section 2 provides the technical background that is needed to understand our technique. Section 3 describes the tech- 1. INTRODUCTION nique, which includes the description of the heterogenous A cryptanalytic time-memory trade-off is a technique to tables and the interleaving exploring rule. Section 4 intro- find preimages of given outputs of a one-way function. They duces an algorithm to identify the optinal configuration, and were first introduced by Hellman in 1980 [9] and they have Section 5 finally evaluates the technique. been used in many practical attacks such as against A5/1 (used for GSM communications) in 2000 [7], or other stream 2. TECHNICAL BACKGROUND ciphers like LILI-128 in 2002 [15]. The rainbow tables tech- nique [14], a variant on Hellman's, has been illustrated by 2.1 Concept the very efficient cracking of Windows LM Hash passwords A fundamental problem in cryptanalysis is finding the in 2003 [14] and Unix passwords (using FPGA) in 2005 [13]. preimage of a given output of a one-way function. A sim- Hellman's technique has been improved upon in various ple method is applying the function to all possible inputs ways, mostly targeting the efficiency of the online phase. until finding the expected value. Such an exhaustive search The most impactful of these improvements arguably was requires N operations in the worst case to find a preimage, the aforementioned rainbow tables variant [14] introduced where N is the size of the input space. This becomes im- practical when N is very large. The other extreme is to first Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed construct a look-up table including all the preimage values. for profit or commercial advantage and that copies bear this notice and the full cita- Afterwards, finding a preimage is done via a table look-up tion on the first page. Copyrights for components of this work owned by others than operation which requires a negligible amount of time. The ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or re- publish, to post on servers or to redistribute to lists, requires prior specific permission precomputation process however requires an effort equal to and/or a fee. Request permissions from [email protected]. an exhaustive search, but is to be performed only once. Al- ASIA CCS ’17, April 02-06, 2017, Abu Dhabi, United Arab Emirates though this method is quite fast during the online search c 2017 ACM. ISBN 978-1-4503-4944-4/17/04. $15.00 phase, it may require prohibitively large amounts of mem- DOI: http://dx.doi.org/10.1145/3052973.3053030 ory for large problems. 815 Time-memory trade-offs are an intermediate solution to ending point. Then computing rt−1(h(rt−2(h(rt−3(y))))), this problem. They consist in an offline precomputation and so on until either the search succeeds or all columns are phase, and an online search phase, and require some mem- have been searched through. ory. The efficiency of the online phase is proportional to The search procedure for rainbow tables works in parallel. N 2=M 2 where M is the memory associated to the trade-off. That is, all tables are searched through for each column Typically, this translates into both time and memory being rather than sequentially. The reason for this is that the O(N 2=3), but ultimately the more memory is dedicated to search is increasingly more expensive towards the left of the the trade-off, the faster the search phase goes. The memory tables. Result 1 (from [14]) quantifies that cost, in terms of required is typically much smaller than for exhaustive stor- number of cryptographic operations. age, and the online phase is on average typically much faster than for exhaustive search. The precomputation phase how- Result 1. The average number of h evaluations during ever is more expensive than for the exhaustive storage solu- a search in column k (column 0 being the rightmost one) of tion. a clean rainbow table of maximal size with chains of size t We now introduce the notation used in this paper. Let is: h : A ! B be a one-way function. Let ri : B ! A be the Ck = k + (t − k + 1)qt−k+1; reduction function used in column i. The goal of a reduction function is to map a point in B to an arbitrary point in A with in an efficient and uniformly distributed fashion. A typical i(i − 1) qi = 1 − : reduction function is ri : y 7! (y + i) mod N, with N = jAj. t(t + 1) The rainbow tables method is divided into two phases: the offline (or precomputation) and the online phases. It is relatively easy to observe numerically that the quantity written in Result 1 is increasing, but one can be convinced 2.2 Offline phase by observing that the negative term in k is multiplied by During this step, the rainbow tables are computed and qt−k+1, which is both smaller than 1 and decreasing. stored in memory. A table consists in a series of chains built by iterating alternatively h and ri. The first points of 3. OUR TECHNIQUE the chains (called the starting points) are chosen arbitrarily (usually incremental values, see e.g. [2]). Chains are of fixed 3.1 Heterogeneous Tables length t and once all chains are completed, only the starting In order to obtain a clean table, many chains need to be points and the ending points (the last point of each chain) thrown out, which reduces the coverage and thus the prob- are saved. Tables are then usually filtered so as to only ability of success during the online phase. Even tables of 1 keep one chain per different ending point (clean tables ). maximal size have a bounded probability of success, pro- The computation of chains stops when the number of chains vided in Result 2 and proved in [14]. with different ending points m is deemed satisfactory. See Figure 1 for a depiction of the structure of a rainbow table. Result 2. The probability of success of a set of ` clean Multiple clean rainbow tables are usually built for a given rainbow tables of maximal size is: problem (see Section 3.1). P ∗ ≈ 1 − e−2`: A table of maximal size is obtained when all (or almost all) the possible ending points are reached, which happens This implies that in order to obtain a higher probability of when the number of chains computed is sufficiently large (i.e. success while using maximal size clean tables, one must use when any new chain would have a negligible probability of ` independent tables, i.e., tables that use different reduction having a new ending point). Clean tables of maximal size are function families. A typical number for ` is 4, which achieves the most memory-efficient version of the rainbow tables [12]. a total probability of success of about 99:97%. As explained See for instance [5, 12] for an analysis of clean tables and in Section 2, these tables are built separately to one another, tables of maximal size (the results relevant for the analysis and explored in parallel during the online phase.