Generating pseudo- random numbers

What are pseudo-random numbers?

Numbers exhibiting statistical randomness while being generated by a deterministic process.

Easier to generate than true random numbers?

What is a good pseudo-random number?

From the correct distribution (especially the tails!)

Long periodicity

Independence

Fast to generate

Most standard computing languages have packages or functions that generate either U(0, 1) random numbers or integers on U(0, 232 − 1).

rand or unifrnd in MATLAB

rand in C/C++ – p. 1/?? Linear congruential generator (LCG)

The linear congruential generator is a simple, fast, and popular way of generating random numbers,

xi =(axi−1 + c) mod m. This will generate integer random numbers on [0, m − 1] which are mapped to U(0, 1) after division by m. The period of the generator is m if

c and m are relative prime

a − 1 is divisible by all prime factors of m

a − 1 is divisible by 4 if m is divisible by 4.

The LCG has problems with high serial correlation, and is sensitive to the choice of parameters.

– p. 2/?? Mersenne twister

Developed in 1997 by Makoto Matsumoto and Takuji Nishimura.

Called Mersenne twister since it uses Mersenne prime numbers 2p − 1.

The most commonly used version is the MT19937 algorithm.

Standard generator in MATLAB (from ver. 7.4), Maple, R, and GNU Scientific Library.

Period of the generated numbers is 219937 − 1.

Equidistributed in high dimensions (LCG has problems with d> 5).

– p. 3/?? Simulation algorithms

After having generated a sample of pseudo-random numbers u1,...,un ∈ U(0, 1), we can simulate data from a specified distribution function by either:

inversion and transformation methods

rejection sampling

conditional methods

– p. 4/?? Inversion method

Let U ∈ U(0, 1) be a uniform random variable.

We want random variable X ∈R from a continuous distribution F .

Does a simple transform x = h(u) that is increasing with an inverse h−1 : R → [0, 1] exist?

X = F −1(U) is a r.v with distribution F .

For discrete distributions F −1(u) does not exist. Take instead inf{x; F (x) ≥ u}

Limited to cases where: The inverse of the distribution function exists and is easy to evaluate. We want univariate random numbers.

– p. 5/?? Exponential distribution- an example

To simulate from the exponential distribution F (x) = 1 − exp(−λx), so

−1 −1 F (u)= −λ log(1 − u). u=rand(1,n); x=-(log(1-u))/lambda;

1 500

0.9 450

0.8 400

0.7 350

0.6 300

u 0.5 250 frequency 0.4 200

0.3 150

0.2 100

0.1 50

0 0 0 1 2 3 4 5 6 7 8 0 1 2 3 4 5 6 7 x x

Left : cdf for Exp(1). Right: histogram of 1000 unit exponential – p. 6/?? Poisson(2) distribution - an example

The distribution function of the Poisson(2) distribution is

x F(x) x F(x) 0 0.1353353 5 0.9834364 1 0.4060058 6 0.9954662 2 0.6766764 7 0.9989033 3 0.8571235 8 0.9997626 4 0.9473470 9 0.9999535 10 0.9999917

Generate a seq of standard uniforms (s.u) u1,...,un

∀ui get xi ∈ Po(2): F (xi−1)

For u1 = 0.7352, we get x1 = 3 and for u2 = 0.4256, we get x2 = 2 and so on...

– p. 7/?? Transformation method

The inversion method is a special case of a more general class of methods called: transformation method. Theorem 1: Let X have a continuous density f and let h be a differentiable function with a differentiable inverse g = h−1. Then the random variable Z = h(X) has density

0 f(g(z))|g (z)|.

Can be generalized to multivariate densities and cases where the inverse of h does not exist. Examples: Γ(n, λ), N (µ, σ2),χ2(1) Theorem 2: Let X and Y be independent random variables that take values in R with densities fx and fy, then the density of Z = X + Y is

f (z)= f (z − t)f (t)dt. z Z x y

Examples: Γ(k, 1),χ2(n),Bin(n,p)

– p. 8/?? Rejection sampling

Suppose we can simulate from some density g. we can use this as the basis for simulating from the continuous density f by simulating Y from g and then accepting this simulated value with a prob proportional to f(Y )/g(Y ). Specifically, let c be a constant s.t f(y) ≤ c, ∀y g(y) Algorithm:

1. Simulate Y from g and simulate a random number U.

2. If U ≤ f(Y )/cG(Y ), set X = Y . Otherwise return to step 1.

Remarks:

The way in which we "accept the value Y with prob f(Y )/cg(Y )" is by generating a r.number U and then accepting Y if U ≤ f(Y )/cg(Y )

Since each iteration will, independently, result in an accepted value with prob P (U ≤ f(Y )/cg(Y )) = K = 1/c, the number of iterations follows a geometric distribution with mean c. 2 Examples: Γ(k, 1),χ (n),Bin(n,p) – p. 9/?? An example

Want to simulate from f(x) ∝ x2e−x; 0 ≤ x ≤ 1, a truncated gamma distribution

1 25

0.9

0.8 20

0.7

0.6 15

0.5 exp(−x)

0.4 10

0.3

0.2 5

0.1

0 0 0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 x x

Left: Scaled density and envelope Right: Histogram of simulated data.

– p. 10/?? Conditional Methods

If we can decompose a multivariate density into conditional parts, the problem of sampling from a multivariate density reduces to sampling from several univariate densities.

f(x1,...,xn)f1(x1)f2(x2|x1) ··· fn(xn|xn−1,...,x1). The problem is that usually is hard to find the conditional densities since this involves:

f1(x1)= ... f(x1,...,x )dx2 . . . dx Z Z n n

– p. 11/??