1 Random Number Generation with Multiple Streams for Sequential and Parallel Computing Pierre L'Ecuyer MIT, Operations Research, February 2017 01111010011011010100110110010100011101111?100110?1?101001101100101000111...?... Example: Bit sequence (head or tail): Uniformity: each bit is1 with probability 1/2. Uniformity and independance: Example: 8 possibilities for the 3 bits ??? : 000; 001; 010; 011; 100; 101; 110; 111 Want a probability of 1/8 for each, independently of everything else. For s bits, probability of1 =2s for each of the 2s possibilities. 2 What do we want? Sequences of numbers that look random. 01111?100110?1?101001101100101000111... Uniformity and independance: Example: 8 possibilities for the 3 bits ??? : 000; 001; 010; 011; 100; 101; 110; 111 Want a probability of 1/8 for each, independently of everything else. For s bits, probability of1 =2s for each of the 2s possibilities. 2 What do we want? Sequences of numbers that look random. Example: Bit sequence (head or tail): 011110100110110101001101100101000111?... Uniformity: each bit is1 with probability 1/2. 011110100110110101001101100101000111?... For s bits, probability of1 =2s for each of the 2s possibilities. 2 What do we want? Sequences of numbers that look random. Example: Bit sequence (head or tail): 01111?100110?1?101001101100101000111... Uniformity: each bit is1 with probability 1/2. Uniformity and independance: Example: 8 possibilities for the 3 bits ??? : 000; 001; 010; 011; 100; 101; 110; 111 Want a probability of 1/8 for each, independently of everything else. 011110100110110101001101100101000111?... 2 What do we want? Sequences of numbers that look random. Example: Bit sequence (head or tail): 01111?100110?1?101001101100101000111... Uniformity: each bit is1 with probability 1/2. Uniformity and independance: Example: 8 possibilities for the 3 bits ??? : 000; 001; 010; 011; 100; 101; 110; 111 Want a probability of 1/8 for each, independently of everything else. For s bits, probability of1 =2s for each of the 2s possibilities. Independence: For a random vector U = (U1;:::; Us ), we want P[aj Uj bj for j = 1;:::; s] = (b1 a1) (bs as ): ≤ ≤ − ··· − 1 U2 b2 a2 0 a 1 1 b1 U1 3 Uniform distribution over (0; 1) For simulation in general, we want (to imitate) a sequence U0; U1; U2;::: of independent random variables uniformly distributed over (0; 1). We want P[a Uj b] = b a. ≤ ≤ − 0 a b 1 3 Uniform distribution over (0; 1) For simulation in general, we want (to imitate) a sequence U0; U1; U2;::: of independent random variables uniformly distributed over (0; 1). We want P[a Uj b] = b a. ≤ ≤ − 0 a b 1 Independence: For a random vector U = (U1;:::; Us ), we want P[aj Uj bj for j = 1;:::; s] = (b1 a1) (bs as ): ≤ ≤ − ··· − 1 U2 b2 a2 0 a 1 1 b1 U1 Non-uniform variates: To generate X such that P[X x] = F (x): ≤ −1 X = F (Uj ) = inf x : F (x) Uj : f ≥ g This is inversion. −λx Example: If F (x) = 1 e , take X = [ ln(1 Uj )]/λ. − − − Also other methods such as rejection, etc., when F −1 is costly to compute. 4 This notion of independent uniform random variables is only a mathematical abstraction. Perhaps it does not exist in the real world! We only wish to imitate it (approximately). 4 This notion of independent uniform random variables is only a mathematical abstraction. Perhaps it does not exist in the real world! We only wish to imitate it (approximately). Non-uniform variates: To generate X such that P[X x] = F (x): ≤ −1 X = F (Uj ) = inf x : F (x) Uj : f ≥ g This is inversion. −λx Example: If F (x) = 1 e , take X = [ ln(1 Uj )]/λ. − − − Also other methods such as rejection, etc., when F −1 is costly to compute. 1 2 3 4 6 75 1 3 4 6 7 52 3 4 6 7 521 For n objets, choose an integer from 1 to n, then an integer from 1 to n 1, then from 1 to n 2, ... − − Each permutation should have the same probability. To shuffle a deck of 52 cards: 52! 2226 possibilities. ≈ 5 Random permutation: 1 2 3 4 5 6 7 1 3 4 6 7 52 3 4 6 7 521 For n objets, choose an integer from 1 to n, then an integer from 1 to n 1, then from 1 to n 2, ... − − Each permutation should have the same probability. To shuffle a deck of 52 cards: 52! 2226 possibilities. ≈ 5 Random permutation: 1 2 3 4 5 6 7 1 2 3 4 6 75 3 4 6 7 521 For n objets, choose an integer from 1 to n, then an integer from 1 to n 1, then from 1 to n 2, ... − − Each permutation should have the same probability. To shuffle a deck of 52 cards: 52! 2226 possibilities. ≈ 5 Random permutation: 1 2 3 4 5 6 7 1 2 3 4 6 75 1 3 4 6 7 52 For n objets, choose an integer from 1 to n, then an integer from 1 to n 1, then from 1 to n 2, ... − − Each permutation should have the same probability. To shuffle a deck of 52 cards: 52! 2226 possibilities. ≈ 5 Random permutation: 1 2 3 4 5 6 7 1 2 3 4 6 75 1 3 4 6 7 52 3 4 6 7 521 5 Random permutation: 1 2 3 4 5 6 7 1 2 3 4 6 75 1 3 4 6 7 52 3 4 6 7 521 For n objets, choose an integer from 1 to n, then an integer from 1 to n 1, then from 1 to n 2, ... − − Each permutation should have the same probability. To shuffle a deck of 52 cards: 52! 2226 possibilities. ≈ 6 Physical devices for computers Photon trajectories (sold by id-Quantique): 0 1 0 1 0 0 1 1 1 0 0 1 The signal is sampled periodically. 7 Thermal noise in resistances of electronic circuits time 7 Thermal noise in resistances of electronic circuits time 0 1 0 1 0 0 1 1 1 0 0 1 The signal is sampled periodically. Can reduce the bias and dependence by combining bits. E.g., with a XOR: 0 1 1 0 0 0 1 0 0 1 1 0 1 1 0 1 0 0 |{z} |{z} |{z} |{z} |{z} |{z} |{z} |{z} |{z} 1 1 0 1 1 1 0 1 0 or (this eliminates the bias): 0 1 1 0 0 0 1 0 0 1 1 0 1 1 0 1 0 0 |{z} |{z} |{z} |{z} |{z} |{z} |{z} |{z} |{z} 0 1 1 0 1 0 Physical devices are essential for cryptology, lotteries, etc. But for simulation, it is inconvenient, not always reliable, and has no (or little) mathematical analysis. A much more important drawback: it is not reproducible. 8 Several commercial devices on the market (and hundreds of patents!). None is perfect. or (this eliminates the bias): 0 1 1 0 0 0 1 0 0 1 1 0 1 1 0 1 0 0 |{z} |{z} |{z} |{z} |{z} |{z} |{z} |{z} |{z} 0 1 1 0 1 0 Physical devices are essential for cryptology, lotteries, etc. But for simulation, it is inconvenient, not always reliable, and has no (or little) mathematical analysis. A much more important drawback: it is not reproducible. 8 Several commercial devices on the market (and hundreds of patents!). None is perfect. Can reduce the bias and dependence by combining bits. E.g., with a XOR: 0 1 1 0 0 0 1 0 0 1 1 0 1 1 0 1 0 0 |{z} |{z} |{z} |{z} |{z} |{z} |{z} |{z} |{z} 1 1 0 1 1 1 0 1 0 Physical devices are essential for cryptology, lotteries, etc. But for simulation, it is inconvenient, not always reliable, and has no (or little) mathematical analysis. A much more important drawback: it is not reproducible. 8 Several commercial devices on the market (and hundreds of patents!). None is perfect. Can reduce the bias and dependence by combining bits. E.g., with a XOR: 0 1 1 0 0 0 1 0 0 1 1 0 1 1 0 1 0 0 |{z} |{z} |{z} |{z} |{z} |{z} |{z} |{z} |{z} 1 1 0 1 1 1 0 1 0 or (this eliminates the bias): 0 1 1 0 0 0 1 0 0 1 1 0 1 1 0 1 0 0 |{z} |{z} |{z} |{z} |{z} |{z} |{z} |{z} |{z} 0 1 1 0 1 0 8 Several commercial devices on the market (and hundreds of patents!). None is perfect. Can reduce the bias and dependence by combining bits. E.g., with a XOR: 0 1 1 0 0 0 1 0 0 1 1 0 1 1 0 1 0 0 |{z} |{z} |{z} |{z} |{z} |{z} |{z} |{z} |{z} 1 1 0 1 1 1 0 1 0 or (this eliminates the bias): 0 1 1 0 0 0 1 0 0 1 1 0 1 1 0 1 0 0 |{z} |{z} |{z} |{z} |{z} |{z} |{z} |{z} |{z} 0 1 1 0 1 0 Physical devices are essential for cryptology, lotteries, etc. But for simulation, it is inconvenient, not always reliable, and has no (or little) mathematical analysis. A much more important drawback: it is not reproducible. 9 Reproducibility Simulations are often required to be exactly replicable, and always produce exactly the same results on different computers and architectures, sequential or parallel. Important for debugging and to replay exceptional events in more details, for better understanding. Also essential when comparing systems with slightly different configurations or decision-making rules, by simulating them with common random numbers (CRNs).