Statistical Testing of RNGs For a sequence of numbers to be considered a sequence of randomly acquired numbers, it must have two basic statistical properties: • Uniformly distributed values • Independence from previous values drawn A sequence fails because patterning is present in the sequence with some degree of statistical certainty. This may manifest itself by: • Clustering or bias occurs at some point (ie., statistically, the sequence is not uniformly distributed) − This includes sequences of tuples drawn from the sequence (if truly random, there should be n-space uniformity) • Standard statistical measures are too high or too low − Mean value − Standard deviation • Statistically significant patterns occur (indicating dependencies) − Alternating patterns − Distribution patterns (correlation) − Other patterning It is always possible that some underlying patterns in a sequence will go undetected. NIST and RNGs In recognition of the importance of random number generators, particularly for areas such as cryptography, the NIST (http://csrc.nist.gov/rng/) has established guidelines for random number generation and testing with three primary goals: 1. to develop a battery of statistical tests to detect non-randomness in binary sequences constructed using random number generators and pseudo-random number generators utilized in cryptographic applications 2. to produce documentation and software implementation of these tests 3. to provide guidance in the use and application of these tests Typical Tests for Randomness as Reported in 1979 (Bright, H. and R. Enison. Computing Surveys 11(4). 1979. Quasi-Random Number Sequences from a Long-Period TLP Generator with Remarks on Application to Cryptography) 1. Equidistribution or Frequency Test. Count the number of times a member of the sequence falls into a given interval. The number should be approximately the same for intervals of the same length if the sequence is uniform. 2. Serial Test. This is a higher dimensional version of the equidistribution test. Count the number of times a k-tuple of members of the sequence falls into a given k-dimensional cell. If this test is passed, the sequence is said to be k-distributed. Other tests of k-distributivity are: 1. Poker test: Consider groups of k members of the sequence and count the number of distinct values represented in each group (e.g., a hand of 5 cards falls into one of a number of groups, such as one pair, two pairs, three of a kind, etc). 2. Maximum (minimum) of k: Plot the distribution of the function max (min) [of a k-tuple]. 3. Sum of k 3. Gap Test. Plot the distribution of gaps in the sequence of various lengths (typically, a gap is the distance between an item and its next recurrence) 4. Runs Test. Plot the distribution of the runs up (monotonic increasing) and runs down (monotonic decreasing), or the distribution of the runs above the mean and those below the mean, or the distribution of the run lengths. 5. Coupon Collector's Test. Choose a suitably small integer d and divide the universe into d intervals. Then each member of the sequence falls into one such interval. Plot the distribution of runs of various lengths required to have all d intervals represented. The idea is that you are trying to collect all of the “coupons”. 6. Permutation Test. Study the order relations between the members of the sequence in groups of k. Each of the k! possible orders should occur about equally often. If the universe is large, the probability of equality is small; otherwise, equal members may be disregarded. 7. Serial Correlation Test. Compute the correlation coefficient between consecutive members of the sequence. This gives the serial correlation for lag 1. Similarly, one may get the serial correlation for lag k by computing the correlation coefficient between xi and xi+k. This is to show that the members of the sequence are independent. Recall: the correlation coefficient references the method of least squares for fitting a line to a set of points. The closer the correlation coefficient is to 1, the more confidence in using the regression line as a predictor. Hence, the an RNG should exhibit small serial correlation values. Generally, the RNG is assumed to be producing probabilities; ie., numbers in the interval [0,1]. Knuth, Seminumerical Algorithms, (Addison-Wesley, 1997 – 3rd edition) is an excellent source for a discussion on testing random number generators. An often cited battery of tests for random number generators is Marsaglia’s 1995 Diehard collection (http://stat.fsu.edu/pub/diehard/) Both source and object files are provided, but at least some of the source is “an f2c conversion of patched and jumbled Fortran code,” which may limit its usefulness. Background for Hypothesis Testing Given n observations, X1, X2, … , Xn (assumed to be from independent, identically distributed random variables), with (unknown) population mean µ and variance σ2. The sample mean n X ∑ i X()n = i=1 n provides an (unbiased) estimator for µ (meaning that if we perform a large number of independent trials, each producing a value for X(n) , the average of the X(n) ’s will be a close approximation to µ). Likewise the sample variance: n 2 ∑ [Xi − X()n ] S2 ()n = i=1 n −1 is an estimator for σ2 (the population variance is calculated by dividing by n; division by n-1 is used in this case because it can be proved that the sample variance is an unbiased estimator for σ2 when dealing with independent, identically distributed random variables) Basically, two random variables are independent if there is no relationship between them; ie., the values assumed by one random variable tell us nothing about how the values of the other one distribute and vice-versa. Of course, using X(n) to estimate µ has little validity without knowing more information to provide an assessment of how closely X(n) approximates µ. For the mean, it is easy to show that • E(cX) = cE(X) • E(X1 + X2) = E(X1) + E(X2) For the variance, • Var(cX) = c2Var(X), but • Var(X1 + X2) = Var(X1) + Var(X2) only if the Xi’s are independent Evidently, we must examine Var( X(n) ) to see how good an approximation to µ we can expect. Consider that ⎛ 1 n ⎞ 1 ⎛ n ⎞ Var X = Var X Var( X(n) ) = ⎜ ∑ i ⎟ 2 ⎜∑ i ⎟ ⎝ n i=1 ⎠ n ⎝ i=1 ⎠ 1 n = Var(X ) 2 ∑ i if the Xi’s are independent. n i=1 If so, then Var( X(n) ) = σ2/n In other words, when the Xi’s are independent, it is quite clear that with larger values of n, the variance shrinks and averaging the sample means provides a good estimate for µ. Does this matter? Law and Kelton report that simulation output data are almost always correlated; ie., the set of means acquired from multiple simulation runs lack independence. In this case, using the sample variance as an estimator may significantly underestimate the population variance, in turn leading to erroneous estimates of µ. Covariance and Correlation as Measures of Independence Between Two Random Variables For two random variables X1 and X2, the covariance is given by Cov(X1, X2) = E[(X1 - µ1)( X2 - µ2)] = E(X1X2) - µ1µ2 E[(X1 - µ1)( X2 - µ2)] = E(X1X2) - µ1E(X2) - µ2E(X1) + µ1µ2 = E(X1X2) - µ1µ2 µ2 µ1 If X1 and X2 are independent, then E(X1X2) = E(X1)E(X2) (this can be determined from the fact that for independent events x and y, P(xy) = P(x)P(y)). Hence, when X1 and X2 are independent, Cov(X1, X2) = 0. 2 2 2 2 The relationship σ = E[(X-µ) ] = E(X ) - µ is not hard to show (we did 2 2 2 it earlier), so in particular, Cov(X1, X1) = E(X1 ) - µ1 = σ1 (a boundary case, since nothing could be less independent than a random variable compared with itself). When Cov(X1, X2) = 0, the two random variables are said to be uncorrelated. If X1 and X2 are independent, we noted Cov(X1, X2) = 0, which indicates there is some relationship between covariance and independence, but it turns out that this is not a necessary and sufficient condition (so, for those interested in statistical theory, criteria under which this is a necessary and sufficient condition for independence provides a natural object of study). When Cov(X1, X2) > 0, the two random variables are said to be positively correlated (the counterpart being negatively correlated). Intuitively, positive correlation typically occurs when both X1 > µ1 and X2 > µ2 or X1 < µ1 and X2 < µ2 (with the obvious counterpart for negative). In particular, when negative correlation is present, you expect that when one sample mean is on the high side, the other will be on the low side. To normalize the statistic, the correlation ρ12 (also known as the correlation coefficient) is defined by Cov(X1,X 2 ) ρ12 = σ1σ 2 Here we now have -1 ≤ ρ12 ≤ 1 2 2 2 This follows from the fact that E(X1 )E(X2 ) ≥ [E(X1)E(X2)] (Schwarz’s inequality) The closer ρ is to 1 or –1 , the greater the dependence relationship between X1 and X2. Simulation models typically employ random variables for input so the model outputs are also random. A stochastic process is one for which the model outcomes are ordered over time, so in general, the collection of random variables representing one of the outputs of a simulation model over multiple runs, X1, X2, …, Xn is a stochastic process. Reaching conclusions about the model from the implicit stochastic process may require making assumptions about the nature of the stochastic process that may not be true, but are necessary for the application of statistical analysis.