Central Limit Theorem I

Classical Central Theorem Motivation The Classical Central Limit Theorem Slutsky’s Theorem

Chapter 5 Multiple Random Variables Central Limit Theorem

1 / 19 Classical Central Theorem Motivation The Classical Central Limit Theorem Slutsky’s Theorem

Outline

Classical Central Theorem Motivation Bernoulli Random Variables Exponential Random Variables The Classical Central Limit Theorem Exponential Random Variables Bernoulli Trials and the Continuity Correction Slutsky’s Theorem

2 / 19 Classical Central Theorem Motivation The Classical Central Limit Theorem Slutsky’s Theorem

Motivation

For the law of large numbers, the sample means from a sequence of independent random variables converge to their common distributional mean as the number n of random variables increases. 1 S = X¯ → µ as n → ∞. n n n √ Moreover, the standard deviation of X¯n is inversely proportional to n. Thus, we magnify the diﬀerence between the running average and the mean by a √ factor of n and investigate the graph of

√ 1 n S − µ versus n n n

3 / 19 Classical Central Theorem Motivation The Classical Central Limit Theorem Slutsky’s Theorem

Motivation

Does the distribution of the size of these ﬂuctuations have any regular and predictable structure? Let’s begin by examining the distribution for the sum of X1, X2 ... Xn, independent and identically distributed random variables

Sn = X1 + X2 + ··· + Xn.

What distribution do we see? We begin with the simplest case, Xi Bernoulli random variables. The sum Sn is a binomial random variable. We examine two cases. • keep the number of trials the same at n = 100 and vary the success probability p. • keep the success probability the same at p = 1/2, but vary the number of trials.

4 / 19 Classical Central Theorem Motivation The Classical Central Limit Theorem Slutsky’s Theorem

Bernoulli Random Variables 0.10 0.08 0.06 probability 0.04 0.02 0.00

0 20 40 60 80 100

x Successes in 100 Bernoulli trials with p = 0.2, 0.4, 0.6 and 0.8. 5 / 19 Classical Central Theorem Motivation The Classical Central Limit Theorem Slutsky’s Theorem

Bernoulli Random Variables 0.15 0.10 probability 0.05 0.00

0 10 20 30 40 50 60

x Successes in 20, 40, and 80 Bernoulli trials with p = 0.5. 6 / 19 Classical Central Theorem Motivation The Classical Central Limit Theorem Slutsky’s Theorem

Bernoulli Random Variables

The binomial random variable Sn has

mean np and standard deviation pnp(1 − p).

Thus, if we take the standardized version of these sums of Bernoulli random variables

Sn − np Zn = , pnp(1 − p)

then these bell curve graphs would lie on top of each other.

Now, let’s consider exponential random variables . . . .

7 / 19 Classical Central Theorem Motivation The Classical Central Limit Theorem Slutsky’s Theorem

Exponential Random Variables 0.5 0.4 0.3 density 0.2 0.1 0.0

-3 -2 -1 0 1 2 3

x The density of the standardized random variables that result from the sum of2 ,4,8,16, and 32 exponential random variables 8 / 19 Classical Central Theorem Motivation The Classical Central Limit Theorem Slutsky’s Theorem

The Classical Central Limit Theorem

To obtain the standardized random variables, • we can either standardize using the sum S having mean nµ and standard √ n deviation σ n, , or • we can standardize using the sample mean X¯ having mean µ and standard √ n deviation σ/ n. This yields two equivalent versions of the standardized score or z-score. √ S − nµ X¯ − µ n Z = n √ = n √ = (X¯ − µ). n σ n σ/ n σ n The theoretical result behind these numerical explorations is called the classical central limit theorem.

9 / 19 Classical Central Theorem Motivation The Classical Central Limit Theorem Slutsky’s Theorem

The Classical Central Limit Theorem

Theorem. Let {Xi ; i ≥ 1} be independent random variables having a common 2 distribution. Let µ be their mean and σ be their variance. Then Zn, the standardized scores of the sample means, converges in distribution to Z a standard normal random

variable, i.e., the distribution function FZn converges toΦ, the distribution function of the standard normal for every value z. Z z 1 −x2/2 lim FZn (z) = lim P{Zn ≤ z} = √ e dx = Φ(z). n→∞ n→∞ 2π −∞

In practical terms the central limit theorem states that

P{a < Zn ≤ b} ≈ P{a < Z ≤ b} = Φ(b) − Φ(a).

The number value is obtained in R using the command pnorm(b)-pnorm(a).

10 / 19 Classical Central Theorem Motivation The Classical Central Limit Theorem Slutsky’s Theorem

The Classical Central Limit Theorem We will prove this in the case that the Xi have a moment generating function MX (t) for the interval t ∈ (−h, h) by showing that t2 lim MZ (t) = exp n→∞ n 2

or equivalently, show that the cumulant generating functions KZn (t) = log MZn (t) satisfy t2 lim KZ (t) = n→∞ n 2 Write X − µ Y = i i σ then n 1 X Z = √ Y . n n i i=1

11 / 19 Classical Central Theorem Motivation The Classical Central Limit Theorem Slutsky’s Theorem

The Classical Central Limit Theorem

Let MY (t) denote the moment generating function for the Yi .

n ! n n t X Y t t M (t) = E exp(tZ ) = E exp √ Y = E exp √ Y = M √ Zn n n i n i Y n i=1 i=1 and t t K (t) = n log M √ = nK √ . Zn Y n Y n

Recall that for the cumulant generating function KY ,

0 00 KY (0) = EY1 = 0, KY (0) = Var(Y1) = 1.

12 / 19 Classical Central Theorem Motivation The Classical Central Limit Theorem Slutsky’s Theorem

The Classical Central Limit Theorem

Finally, from two applications of L’Hˆopital’s rule, t KY (t) lim KZ (t) = lim nKY √ = lim n→∞ n n→∞ n →0 2 tK 0 (t) t2K 00 (t) = lim Y = lim Y . →0 2 →0 2 t2K 00 (0) t2 = Y = . 2 2

Note that this is continuous at t = 0, thus we have convergence in distribution. The 2 cumulant generating function KZ (t) = t /2. So, the limiting distribution is a standard normal.

13 / 19 Classical Central Theorem Motivation The Classical Central Limit Theorem Slutsky’s Theorem

The Classical Central Limit Theorem

• The more skewed the distribution, the more observations are need to obtain the bell curve shape. • Based on simulations with the goal of having adequate conﬁdence intervals, Sugden et al. (2000) provide a minimum sample size from independent identically distributed observations. ∗ 2 n = 28 + 25γ1 .

where γ1 is the skewness. ∗ • For the exponential distribution, γ1 = 2 and n = 128. p • For the Bernoulli√ distribution, γ1 = (1 − 2p)/ p(1 − p). For p = 1/3, 2/3, γ = ±1/ 2, and n∗ = 40.5.

14 / 19 Classical Central Theorem Motivation The Classical Central Limit Theorem Slutsky’s Theorem

Exponential Random Variables For exponential random variables, the mean, µ = 1/λ and the standard deviation, σ = 1/λ and therefore

S − n/λ X¯ − 1/λ √ Z = n√ = n √ = n(λX¯ − n). n n/λ 1/(λ n) n

Let X¯144 be the mean of 144 independent with parameter λ = 1. Thus, µ = 1 and σ = 1. Note that n = 144 is suﬃciently large for the use of a normal approximation. P{X¯144 < 1.1} = P 12(X¯144 − 1) < 12(1.1 − 1) = P{Z144 < 1.2} ≈ 0.8649.

With S144 ∼ Γ(144, 1), we compare. > pnorm(1.2);pgamma(1.1*144,144,1) [1] 0.8849303 [1] 0.8829258

15 / 19 Classical Central Theorem Motivation The Classical Central Limit Theorem Slutsky’s Theorem

Bernoulli Trials and the Continuity Correction For Bernoulli random variables, µ = p and σ = pp(1 − p). The sample mean is generally denoted byˆp to indicate the fact that it is a sample proportion. The normalized version ofˆp pˆn − p Zn = , pp(1 − p)/n For 100 trials of a fair coin, pˆ − 1/2 Z = = 20(ˆp − 1/2) 100 1/20

and

P{pˆ ≤ 0.40} = P{pˆ − 1/2 ≤ 0.40 − 1/2} = P{20(ˆp − 1/2) ≤ 20(0.4 − 1/2)}

= P{Z100 ≤ −2} ≈ P{Z ≤ −2} = 0.023.

16 / 19 Classical Central Theorem Motivation The Classical Central Limit Theorem Slutsky’s Theorem

Bernoulli Trials and the Continuity Correction

We can improve the normal approximation to the binomial random variable by employing the continuity correction. For a binomial random variable X , the distribution function

x X P{X ≤ x} = P{X < x + 1} = P{X = y} y=0

can be realized as the area of x + 1 rectangles, height P{X = y}, y = 0, 1,..., x and width1. These rectangles look like a Riemann sum for the integral up to the value x + 1/2.

17 / 19 Classical Central Theorem Motivation The Classical Central Limit Theorem Slutsky’s Theorem

Bernoulli Trials and the Continuity Correction Example. For X ∼ Bin(100, 0.3), For the approximating normal Y , this suggests P{X ≤ 32} is the area of 33 rectan- computing P{Y ≤ 32.5}. gles with right side at the value 32.5. > pbinom(32,100,0.3) 0.09 [1] 0.7107186 0.08 0.07 > n<-100; p<-0.3;mu<-n*p 0.06

0.05 > sigma<-sqrt(p*(1-p)) 0.04 > x<-c(32,32.5,33) 0.03

0.02 > prob<-pnorm((x-mu)/(sigma*sqrt(n))) 0.01 > data.frame(x,prob) 0

25 26 27 28 29 30 31 32 33 34 number of successes x prob Figure: Bin(100, 0.3) mass function 1 32.0 0.6687397 (black) and approximating density √ 2 32.5 0.7073105 N(100 · 0.3, 100 · 0.3 · 0.7). 3 33.0 0.7436546

18 / 19 Classical Central Theorem Motivation The Classical Central Limit Theorem Slutsky’s Theorem

Slutsky’s Theorem Some useful extensions of the central limit theorem are based on Slutsky’s theorem. D P Theorem. Let Xn → X and Yn → c, a constant as n → ∞. Then D 1. YnXn → cX , and D 2. Xn + Yn → X + c. For example, by the law of large numbers, the sample variance 2 a.s. 2 Sn → σ , the distribution variance as n → ∞. Thus, σ →a.s. 1. Sn This ratio also converges in probability. So, by Slutsky’s theorem, the t-statistic ¯ ¯ Xn − µ σ Xn − µ σ D √ = · √ = · Zn → 1 · Z, Sn/ n Sn σ/ n Sn a standard normal as n → ∞. 19 / 19