Massively Parallel Random Number Generators

Holger Dammertz | September 15, 2010 Page 2 Massively Parallel Random Number Generators | Overview | September 15, 2010 Overview

Applications for Pseudo-Random Numbers

I Monte Carlo Simulation, Integration

I Test and Content Generation I These applications are often easily parallelizable

I MRIP paradigm: multiple replications in parallel

Generating Pseudo-Random Numbers

I Generate i.i.d. uniform random numbers (for example 32bit)

I Transform into (0,1)

I Additional transformation to target distribution (for example, normal distributed) Page 3 Massively Parallel Random Number Generators | Overview | September 15, 2010 Structure of a RNG

Formal Deﬁnition [L’E06] (S, µ,f ,U,g)

I S state space

I µ prob. distr. on S to select initial state (seed) s0 ∈ S

I f : S → S transition function

I g : S → U output function

I U = (0,1) output space

I si = f (si−1),i ≥ 1 and ui = g(si ) Page 4 Massively Parallel Random Number Generators | Overview | September 15, 2010 Structure of a RNG

In a parallel Implementation (S, µ,f ,U,g)

I S needed per generating stream (usually per thread)

I S if possible hold in fast memory

I S store in global memory after ﬁnishing for multiple calls

I f and g are device functions (usually in a single function)

I output space often unsigned int → transform needed

Example: Linear Congruence Generator LCG [Knu81]

I si ∈ S is an integer (for example 32bit), U = S,g = id

I f : si = (asi−1 + c) mod m I Needs well chosen a, c and m Page 5 Massively Parallel Random Number Generators | Overview | September 15, 2010 Structure of a RNG

Required properties of a RNG

I Speed

I Repeatability

I Minimal statistical bias

Additional properties

I Random access on ui I Independent number streams

I Long period Page 6 Massively Parallel Random Number Generators | Overview | September 15, 2010 Structure of a RNG

Why is a long period important? From [SPM05]: for a cycle length of n a single simulation should use at most √ 16 3 n

random numbers (to trust the results of statistical simulation). Assume period of 248 and simple parallel cuda app with 4096 threads: √ 16 3 248 ≈ 256 4096 random numbers per thread. Page 7 Massively Parallel Random Number Generators | Overview | September 15, 2010 Choice of RNG parameters are important

Simple LCG, 210 − 3 points Page 8 Massively Parallel Random Number Generators | Parallelization Techniques | September 15, 2010 Two diﬀerent views on parallel random numbers

1) Single Stream for all Threads

I Each thread computes parts of one problem

I Result should not depend on number of threads and should be repeatable

I Ideally RNG update is also parallel (→ speed) Page 9 Massively Parallel Random Number Generators | Parallelization Techniques | September 15, 2010 Two diﬀerent views on parallel random numbers

2) Each Thread uses it’s own RNG

I Each thread computes individual solutions

I Need to guarantee independence of streams

I If you could have a true RNG both methods would behave exactly the same (but repeatability would be lost).

I Using Pseudo RNGs this needs to be explicitly designed in the program Page 10 Massively Parallel Random Number Generators | Parallelization Techniques | September 15, 2010 Parallelizing RNGs

Random Seeding

I Easy to implement

I Generally very bad parallelization method

I Need to seed valid states I No guarantee of independence

I Can work for generators with a long period

I Better alternative: well chosen seeds (per thread)

I For example: Mersenne Twister dcmt library I New GPU Mersenne Twister (MTGP) provides seed tables

Parametrization

I n diﬀerent RNG conﬁgurations for n threads

I Needs to be especially developed and tested

I Often restricted to speciﬁc n Page 11 Massively Parallel Random Number Generators | Parallelization Techniques | September 15, 2010 Parallelizing RNGs

Block Splitting

I Can guarantee non overlapping sequences of length m

I m needs to be known in advance

I Starting states need to be known or computed Page 12 Massively Parallel Random Number Generators | Parallelization Techniques | September 15, 2010 Parallelizing RNGs

Leap-Frogging

I Needs RNG to be able to skip n numbers (or else quite ineﬃcient) Page 13 Massively Parallel Random Number Generators | Application Scenarios | September 15, 2010 Application Design Considerations

On the ﬂy Computation Compute RN when needed in kernel

I State needs to be stored per stream

I RNG uses additional resources (registers and memory)

I Needs properly parallelized implementation

Pre-computation Store RN in main memory

I Only memory access per RN

I Easily parallelizable (same access as leap frog)

I Memory requirements can be huge

I Bandwidth need to be considered Page 14 Massively Parallel Random Number Generators | Application Scenarios | September 15, 2010 Upload vs. Computation Example

Intel(R) Xeon(R) E5420 @ 2.50GHz, GTX 480, 256MB random numbers, Mersenne Twister: MTGP/SFMT, 214 threads, blocksize 256 Precomputation

I Upload CPU-computed RN

I Precompute on CPU (SFMT): 180ms I Upload: 44ms

I Precompute on GPU

I Precompute on GPU (MTGP): 9.18ms

I Consume (Asian Options): 783.5ms

Produce and consume (MTGP)

I Single Kernel: 1058.1ms Page 15 Massively Parallel Random Number Generators | Overview of some RNGs | September 15, 2010 Overview of some RNGs

LCGs

si = (asi−1 + c) mod m

I Combining multiple LCGs can give longer period

I Independent streams: Wichmann-Hill (273 threads)

Multiple Recursive Generator

k ! si = ∑ aξ si−ξ mod m ξ =1

I Larger period (for k=1 equal to LCG)

I Blocking: MRG32k3a [LSCK02] Page 16 Massively Parallel Random Number Generators | Overview of some RNGs | September 15, 2010 Overview of some RNGs

RNGs based on Cryptographic functions

I Creates white noise from input

I MD5 [TW08]: hash function

I Tiny Encryption Algorithm (TEA) [ZOC10] I Diﬀerent conﬁguration per thread (parametrization)

I Transform counter and thread id

I Random Access in the sequence

Mersenne Twister

I New GPU version [Sai10] 11213 23209 44497 I Long period: 32bit version provides 2 − 1,2 − 1,2 − 1

I Good seeding strategies (see MTGPDC) Page 17 Massively Parallel Random Number Generators | Testing of RNGs | September 15, 2010 Testing RNGs

Why use tests

I Implementation of RNGs is very sensitive

I Before using any RNG implementation it should be tested

I Failed tests: very likely bad sequence

I Passed tests: guarantees nothing

Test Suits

I Use statistical tests to ﬁnd ﬂaws

I DIEHARD + NIST (both integrated in DIEHARDER [Bro09])

I TestU01 [LS07] Page 18 Massively Parallel Random Number Generators | Our Collection | September 15, 2010 Collection of Parallel Random Number Generators

Selecting suitable RNGs

I Domain speciﬁc problem

I Depends on current compiler and hardware

Our Collection

I CUDA implementation of several diﬀerent RNGs

I Easy to use

I Currently tested on Linux

I Most of the code: MIT License

I Copy-paste ready code

Available at http://mprng.sf.net Page 19 Massively Parallel Random Number Generators | Our Collection | September 15, 2010 Collection of Parallel Random Number Generators

Integrated Benchmark

I Benchmarks compiles and runs on your machine

I Conﬁgure threads and blocks according to your target application

Integrated Test Suits

I DIEHARDER and TestU01

I Automatic report generation

Easy to extend

I Integrate your own RNG

I Integrate your own tests Page 20 Massively Parallel Random Number Generators | Some Results | September 15, 2010 Comparisons: TestU01 SmallCrush Battery

RNG Name BirthdaySpacings Collision Gap SimpPoker CouponCollector MaxOft MaxOft AD WeightDistrib MatrixRank HammingIndep RandomWalk1 H RandomWalk1 M RandomWalk1 J RandomWalk1 R RandomWalk1 C combinedLCGTaus P P P P P P P P P P P P P P P drand48gpu P P P P P F F P P P F F F F P kiss07 P P P P P P P P P P P P P P P lfsr113 P P P P P P P P P P P P P P P md5 P P P P P P P P P P P P P P P mtgp P P P P P P P P P P P P P P P park miller F F P P P F F P P P F F F F F ranecu P F P P P F F P P P F F F F F tea P P P P P P P P P P P P P P P tt800 P P P P P P P P P P P P P P P Page 21 Massively Parallel Random Number Generators | Some Results | September 15, 2010 Raw Performance Test

Raw Performance, GeForce GTX 480, 335544320 Random Numbers 0.035

0.03

0.025

0.02

seconds 0.015

0.01

0.005

0 combinedLCGTausdrand48gpu kiss07 lfsr113 mtgp park_miller ranecu tt800 Page 21 Massively Parallel Random Number Generators | Some Results | September 15, 2010 Raw Performance Test

Raw Performance, GeForce GTX 480, 335544320 Random Numbers 0.3

0.25

0.2

0.15 seconds

0.1

0.05

0 combinedLCGTausdrand48gpu kiss07 lfsr113 md5 mtgp park_miller ranecu tea tt800 Page 22 Massively Parallel Random Number Generators | Some Results | September 15, 2010 Performance Test: Asian Option Example from [HT07]

Asian Options, GeForce GTX 480, 96 runs, 163840 simulations 0.5

0.45

0.4

0.35

0.3

0.25 seconds 0.2

0.15

0.1

0.05

0 combinedLCGTausdrand48gpu kiss07 lfsr113 mtgp park_miller ranecu tt800 Page 22 Massively Parallel Random Number Generators | Some Results | September 15, 2010 Performance Test: Asian Option Example from [HT07]

Asian Options, GeForce GTX 480, 96 runs, 163840 simulations 1.6

1.4

1.2

0.8 seconds 0.6

0.4

0.2

0 combinedLCGTausdrand48gpu kiss07 lfsr113 md5 mtgp park_miller ranecu tea tt800 Page 23 Massively Parallel Random Number Generators | Some Results | September 15, 2010 Quality vs. Speed

Reduced rounds of MD5/TEA RNG

I For some applications speed more important than quality I MD5 and TEA allow to reduce number of iterations

I Quality of random numbers may degrade I Avalanche eﬀect

I MD5: passes most of the DIEHARDER tests after 16 rounds Page 24 Massively Parallel Random Number Generators | Some Results | September 15, 2010 Quality vs. Speed, TEA, DIEHARDER tests

Tiny Encryption Algorithm 120 115 weak passed 110 105 100 95 90 85 80 75 70 65 60 55 50 Dieharder tests 45 40 35 30 25 20 15 10 5 0 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32 Number of rounds Page 25 Massively Parallel Random Number Generators | Some Results | September 15, 2010 Quality vs. Speed, TEA performance

Raw Performance, 335544320 Random Numbers 0.35 GTX480, Tiny Encryption Algorithm

0.3

0.25

0.2

0.15 Runtime in ms

0.1

0.05

0 0 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32 Number of rounds Page 26 Massively Parallel Random Number Generators | Some Results | September 15, 2010 Quality vs. Speed, MD5 performance

Raw Performance, 335544320 Random Numbers 0.35 GTX480, MD5

0.3

0.25

0.2

0.15 Runtime in ms

0.1

0.05

0 0 4 8 12 16 20 24 28 32 36 40 44 48 52 56 60 64 Number of rounds Page 27 Massively Parallel Random Number Generators | Conclusion | September 15, 2010 Conclusion

I Many good (parallel) RNGs exist

I Several diﬀerent properties

I Choice of ﬁtting RNG application dependent

Some Picks

I KISS + Simple code and state management − Random seeding: may be ok for non-critical applications

I MTGPU + Very good quality and sophisticated seeding + Long period − Relatively complex code − Fixed block/thread layout

I MD5, TEA + Random access + No Seeding − Slow Questions?

Acknowledgements

I Framework: Christoph Schied I This work was supported by a NVIDIA Professor Partnership Award with Hendrik P.A. Lensch

Page 28 Massively Parallel Random Number Generators | Conclusion | September 15, 2010 Conclusion

Precomputing on GPU

I May be an alternative to in kernel computation

RNG Collection

I Always evaluate your RNG choice and implementation

I Our framework provides an easy platform for testing http://mprng.sf.net Page 28 Massively Parallel Random Number Generators | Conclusion | September 15, 2010 Conclusion

Precomputing on GPU

I May be an alternative to in kernel computation

RNG Collection

I Always evaluate your RNG choice and implementation

I Our framework provides an easy platform for testing http://mprng.sf.net

Questions?

Acknowledgements

I Framework: Christoph Schied I This work was supported by a NVIDIA Professor Partnership Award with Hendrik P.A. Lensch Page 28 Massively Parallel Random Number Generators | References | September 15, 2010 Robert G. Brown. Dieharder: A random number test suite. http://www.phy.duke.edu/~rgb/General/dieharder.php, 2009. Lee Howes and David Thomas. Eﬃcient Random Number Generation and Application Using CUDA. In GPU Gems 3, pages 805–830, 2007. D.E. Knuth. The art of computer programming: Seminumerical algorithms, volume 2, 1981. P. L’Ecuyer. Uniform random number generation. Handbooks in Operations Research and Management Science, 13:55–81, 2006. P. L’Ecuyer and R. Simard. TestU01: AC library for empirical testing of random number generators. ACM Transactions on Mathematical Software (TOMS), 33(4):22, 2007. Pierre L’Ecuyer, Richard Simard, E. Jack Chen, and W. David Kelton. An object-oriented random-number package with many long streams and substreams. Page 28 Massively Parallel Random Number Generators | References | September 15, 2010 Oper. Res., 50(6):1073–1075, 2002. Mutsuo Saito. A variant of mersenne twister suitable for graphic processors. CoRR, abs/1005.4973, 2010. Marcus Schoo, Krzysztof Pawlikowski, and Donald C. Mcnickle. A survey and empirical comparison of modern pseudo-random number generators for distributed stochastic simulations, 2005. S. Tzeng and L.Y. Wei. Parallel white noise generation on a GPU via cryptographic hash. In Proceedings of the 2008 symposium on Interactive 3D graphics and games, pages 79–87. ACM, 2008. F. Zafar, M. Olano, and A. Curtis. GPU Random Numbers via the Tiny Encryption Algorithm. 2010.