Stochastic Simulation APPM 7400

Home , Stochastic simulation

Lesson 1: Random Number Generators

August 27, 2018

Lesson 1: Random Number Generators Stochastic Simulation August27,2018 1/29 Random Numbers

What is a random number?

Lesson 1: Random Number Generators Stochastic Simulation August27,2018 2/29 Random Numbers

What is a random number?

“You mean like... 5?”

Lesson 1: Random Number Generators Stochastic Simulation August27,2018 2/29 Random Numbers

What is a random number?

“You mean like... 5?”

Formal deﬁnition from probability:

A random number is a number uniformly distributed between 0 and 1.

Lesson 1: Random Number Generators Stochastic Simulation August27,2018 2/29 Random Numbers

What is a random number?

“You mean like... 5?”

Formal deﬁnition from probability:

A random number is a number uniformly distributed between 0 and 1.

(It is a realization of a uniform(0,1) random variable.)

Lesson 1: Random Number Generators Stochastic Simulation August27,2018 2/29 Random Numbers

Here are some realizations:

0.8763857

0.2607807 0.7060687 0.0826960 Density 0.7569021 .

. 0 2 4 6 8 10

0.0 0.2 0.4 0.6 0.8 1.0

sample

Lesson 1: Random Number Generators Stochastic Simulation August27,2018 3/29 Random Numbers

Here are some realizations:

0.8763857

0.2607807 0.7060687 0.0826960 Density 0.7569021 .

. 0 2 4 6 8 10

0.0 0.2 0.4 0.6 0.8 1.0

sample

Lesson 1: Random Number Generators Stochastic Simulation August27,2018 3/29 Random Numbers

Here are some realizations:

0.8763857

0.2607807 0.7060687 0.0826960 Density 0.7569021 .

. 0 2 4 6 8 10

0.0 0.2 0.4 0.6 0.8 1.0

sample

Lesson 1: Random Number Generators Stochastic Simulation August27,2018 3/29 Random Numbers

Here are some realizations:

0.8763857

0.2607807 0.7060687 0.0826960 Density 0.7569021 .

. 0 2 4 6 8 10

0.0 0.2 0.4 0.6 0.8 1.0

sample

Lesson 1: Random Number Generators Stochastic Simulation August27,2018 3/29 Random Numbers

Here are some realizations:

0.8763857 An 0.2607807 artifact 0.7060687 of bin width.

0.0826960 Density 0.7569021 . . 0 2 4 6 8 10

0.0 0.2 0.4 0.6 0.8 1.0

sample

Lesson 1: Random Number Generators Stochastic Simulation August27,2018 3/29 Random Numbers

Here are some realizations:

0.8763857

0.2607807 0.7060687

0.0826960 Density 0.7569021 .

. 0 2 4 6 8 10

0.0 0.2 0.4 0.6 0.8 1.0

(100,000 values) sample

Lesson 1: Random Number Generators Stochastic Simulation August27,2018 3/29 Random Numbers

Here are some realizations:

0.8763857

0.2607807 0.7060687

0.0826960 Density 0.7569021 . . 0.0 0.2 0.4 0.6 0.8 1.0

(100,000 values) 0.0 0.2 0.4 0.6 0.8 1.0

sample

Lesson 1: Random Number Generators Stochastic Simulation August27,2018 3/29 Random Numbers

Rough “reverse” conﬁdence bounds:

Let p be the probability the uniforms we are simulating are between 0 and 0.1. Density 0.0 0.2 0.4 0.6 0.8 1.0

0.0 0.2 0.4 0.6 0.8 1.0

sample

Lesson 1: Random Number Generators Stochastic Simulation August27,2018 4/29 Random Numbers

Rough “reverse” conﬁdence bounds:

Let p be the probability the uniforms we are simulating are between 0 and 0.1. We know that p = 0.1. Density 0.0 0.2 0.4 0.6 0.8 1.0

0.0 0.2 0.4 0.6 0.8 1.0

sample

Lesson 1: Random Number Generators Stochastic Simulation August27,2018 4/29 Random Numbers

Rough “reverse” conﬁdence bounds:

Let p be the probability the uniforms we are simulating are between 0 and 0.1. We know that p = 0.1. A rough 95% conﬁdence interval for p based on the normal Density approximation to the binomial (= CLT!) is

pˆ(1 − pˆ) 0.0 0.2 0.4 0.6 0.8 1.0 pˆ ± 2 0.0 0.2 0.4 0.6 0.8 1.0 r n sample

Lesson 1: Random Number Generators Stochastic Simulation August27,2018 4/29 Random Numbers

Rough “reverse” conﬁdence bounds: This comes from the fact that

pˆ − p approx ∼ N(0, 1) p(1−p) q n Density 0.0 0.2 0.4 0.6 0.8 1.0

0.0 0.2 0.4 0.6 0.8 1.0

sample

Lesson 1: Random Number Generators Stochastic Simulation August27,2018 5/29 Random Numbers

Rough “reverse” conﬁdence bounds: This comes from the fact that

pˆ − p approx ∼ N(0, 1) p(1−p) q n So

pˆ − p Density P −2 < < 2 ≈ 0.95 p(1−p)  q n  0.0 0.2 0.4 0.6 0.8 1.0

0.0 0.2 0.4 0.6 0.8 1.0

sample

Lesson 1: Random Number Generators Stochastic Simulation August27,2018 5/29 Random Numbers

Rough “reverse” conﬁdence bounds: This comes from the fact that

pˆ − p approx ∼ N(0, 1) p(1−p) q n So

pˆ − p Density P −2 < < 2 ≈ 0.95 p(1−p)  q n  Usually, we solve for p in the 0.0 0.2 0.4 0.6 0.8 1.0

middle– sometimes after making 0.0 0.2 0.4 0.6 0.8 1.0 the (lazy!) substitution ofp ˆ for sample each p under the square root.

Lesson 1: Random Number Generators Stochastic Simulation August27,2018 5/29 Random Numbers

Rough “reverse” conﬁdence bounds:

Now we are solving forp ˆ in the middle. (No lazy substitutions needed.) Density 0.0 0.2 0.4 0.6 0.8 1.0

0.0 0.2 0.4 0.6 0.8 1.0

sample

Lesson 1: Random Number Generators Stochastic Simulation August27,2018 6/29 Random Numbers

Rough “reverse” conﬁdence bounds:

Now we are solving forp ˆ in the middle. (No lazy substitutions needed.)

− − P p−2 p(1 p)

0.0 0.2 0.4 0.6 0.8 1.0

sample

Lesson 1: Random Number Generators Stochastic Simulation August27,2018 6/29 Random Numbers

Rough “reverse” conﬁdence bounds:

Now we are solving forp ˆ in the middle. (No lazy substitutions needed.)

− − P p−2 p(1 p)

With p = 0.1 and n = 100, 000, Density this means thatp ˆ should (roughly and 95% of the time) be in (0.09810263, 0.10189740). 0.0 0.2 0.4 0.6 0.8 1.0

0.0 0.2 0.4 0.6 0.8 1.0

sample

Lesson 1: Random Number Generators Stochastic Simulation August27,2018 6/29 Random Numbers

Rough “reverse” conﬁdence bounds:

Now we are solving forp ˆ in the middle. (No lazy substitutions needed.)

− − P p−2 p(1 p)

With p = 0.1 and n = 100, 000, Density this means thatp ˆ should (roughly and 95% of the time) be in (0.09810263, 0.10189740).

Or that the height of the 0.0 0.2 0.4 0.6 0.8 1.0 rectangle should be in 0.0 0.2 0.4 0.6 0.8 1.0 sample (0.9810263, 1.018974)

Lesson 1: Random Number Generators Stochastic Simulation August27,2018 6/29 Random Numbers

Rough “reverse” conﬁdence bounds:

Same numbers work for all bins since each one is trying Now we are solving forp ˆ in the to capture p = 0.1. middle. (No lazy substitutions needed.)

− − P p−2 p(1 p)

rectangle should be in 0.0 0.2 0.4 0.6 0.8 1.0 (0.9810263, 1.018974) 0.0 0.2 0.4 0.6 0.8 1.0 sample

Lesson 1: Random Number Generators Stochastic Simulation August27,2018 6/29 Random Numbers Rough “reverse” conﬁdence bounds:

Same numbers work for all bins since each one is trying

to capture p = 0.1.

based on CLT approximation

and

didn’t make any eﬀort to use Density simultaneous conﬁdence intervals– the heights of the bars are dependent on each other 0.0 0.2 0.4 0.6 0.8 1.0

0.0 0.2 0.4 0.6 0.8 1.0

sample

Lesson 1: Random Number Generators Stochastic Simulation August27,2018 7/29 Random Numbers

Rough “reverse” conﬁdence bounds:

Note that even if everything were perfect, each bar is supposed to fall within (0.9810263, 1.018974) for only 95% of samples. Density 0.0 0.2 0.4 0.6 0.8 1.0

0.0 0.2 0.4 0.6 0.8 1.0

sample

Lesson 1: Random Number Generators Stochastic Simulation August27,2018 8/29 Random Numbers

Rough “reverse” conﬁdence bounds:

Note that even if everything were perfect, each bar is supposed to fall within (0.9810263, 1.018974) for only 95% of samples. Even if the ﬁrst bar (for example) falls within those Density bounds 95% of the time (repeated samples/histograms), be wary of “systematic” behavior. 0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0 ex: bar is always on the high sample side of that interval = bad

Lesson 1: Random Number Generators Stochastic Simulation August27,2018 8/29 A General Rough Check...

In general, if you are simulating n values from a density f (~x),

Lesson 1: Random Number Generators Stochastic Simulation August27,2018 9/29 A General Rough Check...

In general, if you are simulating n values from a density f (~x), and if you can integrate f (~x),

Lesson 1: Random Number Generators Stochastic Simulation August27,2018 9/29 A General Rough Check...

In general, if you are simulating n values from a density f (~x), and if you can integrate f (~x), you can perform a rough check by

Lesson 1: Random Number Generators Stochastic Simulation August27,2018 9/29 A General Rough Check...

In general, if you are simulating n values from a density f (~x), and if you can integrate f (~x), you can perform a rough check by

choosing several regions R1, R2,..., Rk

Lesson 1: Random Number Generators Stochastic Simulation August27,2018 9/29 A General Rough Check...

In general, if you are simulating n values from a density f (~x), and if you can integrate f (~x), you can perform a rough check by

choosing several regions R1, R2,..., Rk computing the probabilities

pi = f (~x) d~x ZRi

Lesson 1: Random Number Generators Stochastic Simulation August27,2018 9/29 A General Rough Check...

In general, if you are simulating n values from a density f (~x), and if you can integrate f (~x), you can perform a rough check by

choosing several regions R1, R2,..., Rk computing the probabilities

pi = f (~x) d~x ZRi computing the proportions # of observations in R pˆ = i i n

Lesson 1: Random Number Generators Stochastic Simulation August27,2018 9/29 A General Rough Check...

In general, if you are simulating n values from a density f (~x), and if you can integrate f (~x), you can perform a rough check by

choosing several regions R1, R2,..., Rk computing the probabilities

pi = f (~x) d~x ZRi computing the proportions # of observations in R pˆ = i i n

checking whether or not eachp ˆi is in pi ± 2 pi (1 − pi )/n p

Lesson 1: Random Number Generators Stochastic Simulation August27,2018 9/29 A General Rough Check...

In general, if you are simulating n values from a density f (~x), and if you can integrate f (~x), you can perform a rough check by

choosing several regions R1, R2,..., Rk computing the probabilities Note again ”95%” ”approximate” pi = f (~x) d~x ”non- simultaneous” ZRi computing the proportions # of observations in R pˆ = i i n

checking whether or not eachp ˆi is in pi ± 2 pi (1 − pi )/n p

Lesson 1: Random Number Generators Stochastic Simulation August27,2018 9/29 Random Numbers

A random number stream is a sequence of independent random numbers.

Lesson 1: Random Number Generators Stochastic Simulation August27,2018 10/29 Random Numbers

A random number stream is a sequence of independent random numbers. “Random numbers” are the basic building blocks for all simulation algorithms!

Lesson 1: Random Number Generators Stochastic Simulation August27,2018 10/29 Random Numbers

A random number stream is a sequence of independent random numbers. “Random numbers” are the basic building blocks for all simulation algorithms! So, it is important to us, as simulators, to have a good random number generator. As important as this is, this course will not focus on RNGs. The assumption will be that we are starting with a decent RNG.

Lesson 1: Random Number Generators Stochastic Simulation August27,2018 10/29 Random Numbers

Lesson 1: Random Number Generators Stochastic Simulation August27,2018 10/29 Random Numbers Generators

A RNG is an algorithm used by a computer to generate a stream of independent realizations of uniform(0,1) random variable.

Lesson 1: Random Number Generators Stochastic Simulation August27,2018 11/29 Random Numbers Generators

A RNG is an algorithm used by a computer to generate a stream of independent realizations of uniform(0,1) random variable.

The CPU will do ”this” ... and then a little of ”that”...... and this other thing...... with some iteration...

Lesson 1: Random Number Generators Stochastic Simulation August27,2018 11/29 Random Numbers Generators

A RNG is an algorithm used by a computer to generate a stream of independent realizations of uniform(0,1) random variable.

The CPU will do ”this” ... and then a little of ”that”...... and this other thing...... with some iteration...

but ultimately, it is only following a recipe through a sequence of deterministic steps with absolutely no randomness!

Lesson 1: Random Number Generators Stochastic Simulation August27,2018 11/29 Random Numbers Generators

With a good enough algorithm, the CPU can produce a stream of seemingly independent uniform(0,1) numbers...

Lesson 1: Random Number Generators Stochastic Simulation August27,2018 12/29 Random Numbers Generators

With a good enough algorithm, the CPU can produce a stream of seemingly independent uniform(0,1) numbers...

... but they are actually pseudo-random∗ numbers.

(* I will not say this again!!! Yes, we all get it. It is obnoxious to be the smarmy one in the group who keeps saying, ”Ahem.. I think you mean psuedo-random don’t you?”)

Lesson 1: Random Number Generators Stochastic Simulation August27,2018 12/29 Linear Congruential RNGs

ﬁrst proposed by Lehmer in 1949 relies on modular arithmetic:

Definition For an integer m, (usually > 0), the numbers a and b are congruent modulo m if the diﬀerence a − b is exactly divisible by m.

We write a = b(mod m) or m|(a − b). (m ”divides” a-b)

Lesson 1: Random Number Generators Stochastic Simulation August27,2018 13/29 Linear Congruential RNGs

a can also be thought of as the remainder in the division of b by m:

b(mod m)= a

For example, 7(mod 2) = 1 7.2(mod 2) = 1.2 7(mod 9) = 7

Lesson 1: Random Number Generators Stochastic Simulation August27,2018 14/29 Linear Congruential RNGs

For a linear congruential RNG, we proceed as follows: 1. Choose integers:

a modulus m > 0 a multiplier a such that 0 ≤ a < m an increment b such that 0 ≤ b < m

a seed (starting value) x0 such that 0 ≤ x0 < m

Lesson 1: Random Number Generators Stochastic Simulation August27,2018 15/29 Linear Congruential RNGs

For a linear congruential RNG, we proceed as follows: 2. Produce a sequence

xn = (axn−1 + b)(mod m)

3. Scale the xn numbers to between 0 and 1 by setting

un = xn/m

Lesson 1: Random Number Generators Stochastic Simulation August27,2018 16/29 Linear Congruential RNGs

You’re kidding right?

Lesson 1: Random Number Generators Stochastic Simulation August27,2018 17/29 Linear Congruential RNGs

You’re kidding right?

This formulaic sequence of steps is supposed to be giving us a stream of independent uniformly distributed random numbers?

Lesson 1: Random Number Generators Stochastic Simulation August27,2018 17/29 Linear Congruential RNGs

You’re kidding right?

This formulaic sequence of steps is supposed to be giving us a stream of independent uniformly distributed random numbers?

Well, they might at least appear that way for “good” choices of a, b, and m...

Lesson 1: Random Number Generators Stochastic Simulation August27,2018 17/29 Linear Congruential RNGs

You’re kidding right?

This formulaic sequence of steps is supposed to be giving us a stream of independent uniformly distributed random numbers?

Well, they might at least appear that way for “good” choices of a, b, and m...

... and they might not for ”bad” choices...

Lesson 1: Random Number Generators Stochastic Simulation August27,2018 17/29 Linear Congruential RNGs

Example:

x0 = 0, a = 1, b = 2, m = 3

x1 = (ax0 + b)(mod m) x3 = (ax2 + b)(mod m) = 2(mod 3) = 2 = 3(mod 3) = 0

x2 = (ax1 + b)(mod m) x4 = (ax3 + b)(mod m) = 4(mod 3) = 1 = 2(mod 3) = 2

Lesson 1: Random Number Generators Stochastic Simulation August27,2018 18/29 Linear Congruential RNGs

Example: We get the sequence:

0, 2, 1, 0, 2, 1, 0, 2, 1, 0, 2, 1, 0, 2, 1, 0, 2, 1, 0, 2, 1,...

which we divide by m = 3 to get the “uniform” sequence of “independent” numbers:

2 1 2 1 2 1 2 1 2 1 2 1 2 1 0, , , 0, , , 0, , , 0, , , 0, , , 0, , , 0, , ,... 3 3 3 3 3 3 3 3 3 3 3 3 3 3

Lesson 1: Random Number Generators Stochastic Simulation August27,2018 19/29 Linear Congruential RNGs

Example: We get the sequence:

0, 2, 1, 0, 2, 1, 0, 2, 1, 0, 2, 1, 0, 2, 1, 0, 2, 1, 0, 2, 1,...

which we divide by m = 3 to get the “uniform” sequence of “independent” numbers:

2 1 2 1 2 1 2 1 2 1 2 1 2 1 0, , , 0, , , 0, , , 0, , , 0, , , 0, , , 0, , ,... 3 3 3 3 3 3 3 3 3 3 3 3 3 3

. bad!

Lesson 1: Random Number Generators Stochastic Simulation August27,2018 19/29 Linear Congruential RNGs

The maximum period for a LCRNG of this form is given by m.

Lesson 1: Random Number Generators Stochastic Simulation August27,2018 20/29 Linear Congruential RNGs

The maximum period for a LCRNG of this form is given by m.

So obviously we want m to be very large.

Lesson 1: Random Number Generators Stochastic Simulation August27,2018 20/29 Linear Congruential RNGs

The maximum period for a LCRNG of this form is given by m.

So obviously we want m to be very large.

Also must be careful about the entire combination of a, b, and m.

Lesson 1: Random Number Generators Stochastic Simulation August27,2018 20/29 Linear Congruential RNGs

The maximum period for a LCRNG of this form is given by m.

So obviously we want m to be very large.

Also must be careful about the entire combination of a, b, and m.

Infamous Example: “RANDU” (IBM)

16 31 a = 2 + 3, b = 0, m = 2 , x0 odd

Lesson 1: Random Number Generators Stochastic Simulation August27,2018 20/29 RANDU

Histogram of randu

Looks Density uniform! 0.0 0.2 0.4 0.6 0.8 1.0

0.0 0.2 0.4 0.6 0.8 1.0

Lesson 1: Random Number Generators randu Stochastic Simulation August27,2018 21/29 RANDU

Plot in triples...

15 z parallel planes!

y x

Lesson 1: Random Number Generators Stochastic Simulation August27,2018 22/29 Linear Congruential RNGs

LCRNGs have been studied extensively and are the sole subjects of many a PhD thesis. Things are known that can help in choices for the parameters. ex: For full period, require b and m to be relatively prime and a − 1 to be divisible by all prime factors of m. ex: If using a LCRNG to choose points in n-dimensional space, n-tuples will lie on, at most, m1/n hyperplanes. (Marsaglia’s Theorem) (nice site http://cnx.org/content/m13103/latest/) Defects aside, LCRNGs are fast (especially when choosing m to be a power of 2 since the modulus can then be computed by left truncation) compared to other RNGs and may be suﬃcient for certain applications.

Lesson 1: Random Number Generators Stochastic Simulation August27,2018 23/29 Other RNGs

multiplicative congruential algorithm (b = 0)

shift register generators

Marsaglia’s algorithms (ex:” subtract-with-borrow” algorithm)

Mersenne Twister (generalized feedback shift register generator, Matsumoto and Nishimura, ”Primitive Twisted Generalized Feedback Shift Register Sequence”, uses Mersenne primes, designed to avoid speciﬁc ﬂaws of other RNGs, period 219937 − 1, negligible serial correlation)

Lesson 1: Random Number Generators Stochastic Simulation August27,2018 24/29 Other Software and RNGs

Matlab u = rand(1)

u = rand(1, n) vector of length n

u = rand(n) n × n matrix

Generator: Mersenne Twister Seed: must seed, rand(’seed’,k) where k is an integer you choose, or, rand(’seed’,100*sum(clock))

Lesson 1: Random Number Generators Stochastic Simulation August27,2018 25/29 Other Software and RNGs

Mathematica u = RandomReal[]

u = RandomReal[1, n] vector of length n

(1 = max) Generator: extended cellular automaton generator Seed: internal clock upon startup, can use SeedRandom[]

Lesson 1: Random Number Generators Stochastic Simulation August27,2018 26/29 Other Software and RNGs

C, C++, Visual C++

u = (double)rand()/(double)RAND MAX

(casting!) Generator: linear congruential Seed: srand(k) where k is an integer you choose, or srand(time(NULL))

Lesson 1: Random Number Generators Stochastic Simulation August27,2018 27/29 Other Software and RNGs

S-Plus, R

u < −runif (1)

u < −runif (n) vector of length n

Generator: Mersenne Twister Seed: internal clock, set.seed(k) where k is an integer you choose

Lesson 1: Random Number Generators Stochastic Simulation August27,2018 28/29 Other Software and RNGs

FORTRAN90

CALL RANDOM NUMBER(u)

Generator: Seed: CALL RANDOM_SEED(size=K) ALLOCATE (seed(K)) WRITE(*, ’(1x, a, i4, a)’,ADVANCE="NO")’Enter ’, K,& ’ integers as random number seeds: ’ READ(*, *) seed CALL RANDOM_SEED(put=seed)

Lesson 1: Random Number Generators Stochastic Simulation August27,2018 29/29