Law of Large Numbers MATH 354

Law of Large Numbers MATH 354

Zun Yin

December 9th

Zun Yin (Macalester College) Law of Large Numbers December 9th 1 / 7 Intuition

The average of results obtained from a large number of the same trials should be close to the expected value, and will tend to become closer as more trials are performed.

Example Rolling a 6-sided die for sufficiently large number of times, we get the average value as close to 3.5, the expected value, as we want.

Zun Yin (Macalester College) Law of Large Numbers December 9th 2 / 7 Weak Law of Large Numbers (WLLN)

Theorem (Weak Law of Large Numbers)

Let X1, X2, ..., be independent and identically distributed random variables 2 ¯ 1 Pn with expected value µ and variance σ < ∞. Define Xn = n i=1 Xi . Then, for every  > 0, ¯ lim P(|Xn − µ| < ) = 1; n→∞

that is, X¯n converges in probability to µ.

Definition (Convergence in Probability)

A sequence of random variables, X1, X2, ..., converges in probability to a random variable X if, for every  > 0,

lim P(|Xn − X | < ) = 1. n→∞

Zun Yin (Macalester College) Law of Large Numbers December 9th 3 / 7 Proof of WLLN

Theorem (Chebyshev’s Inequality) Let W be any random variable with mean µ and variance σ2. For any  > 0,

σ2 P(|W − µ| < ) ≥ 1 − 2 .

Proof (WLLN): According to Chebyshev’s Inequality, ¯ ¯ ¯ Var(Xn) P(|Xn − E(Xn)| < ) ≥ 1 − 2 .

But E(X¯n) = E(Xi ) = µ, ¯ 1 Pn 1 Pn 1 2 σ2 Var(Xn) = Var( n i=1 Xi ) = n2 i=1 Var(Xi ) = n2 ∗ nσ = n , ¯ σ2 so P(|Xn − µ| < ) ≥ 1 − n2 . σ2 ¯ As n → ∞, 2 → 0. Therefore lim P(|Xn − µ| < ) = 1. n n→∞ 

Zun Yin (Macalester College) Law of Large Numbers December 9th 4 / 7 Strong Law of Large Numbers (SLLN)

Theorem (Strong Law of Large Numbers)

Let X1, X2, ..., be independent and identically distributed random variables 2 ¯ 1 Pn with expected value µ and variance σ < ∞. Define Xn = n i=1 Xi . Then, for every  > 0, ¯ P( lim |Xn − µ| < ) = 1; n→∞

that is, X¯n converges almost surely to µ.

Definition (Almost Sure Convergence)

A sequence of random variables, X1, X2, ..., converges almost surely to a random variable X if, for every  > 0,

P( lim |Xn − X | < ) = 1. n→∞

Zun Yin (Macalester College) Law of Large Numbers December 9th 5 / 7 SLLN is Stronger than WLLN SLLN is stronger than WLLN because almost sure convergence is more restrictive than convergence in probability. Convergence in Probability: lim P(|Xn − X | < ) = 1 n→∞ Almost Sure Convergence: P( lim |Xn − X | < ) = 1 n→∞ Example Let the sample space S be the closed interval [0,1] with the uniform probability distribution. Define the sequence X1, X2, ..., as follows:

X1(s) = I[0,1](s), X2(s) = I 1 (s), X3(s) = I 1 (s), [0, 2 ] ( 2 ,1] X4(s) = I 1 (s), X5(s) = I 1 2 (s), X6(s) = I 2 (s), [0, 3 ] ( 3 , 3 ] ( 3 ,1] etc. Let X (s) = 0. Then

lim P(Xn < ) = 1 is true, n→∞

P( lim Xn < ) = 1 is not true. n→∞

Zun Yin (Macalester College) Law of Large Numbers December 9th 6 / 7 Reference

Larsen, R. J., Marx, M. L. (2010). An introduction to mathematical statistics and its applications (5th ed.). Boston, Mass.: Prentice Hall.

Casella, G., Berger, R. L. (2002). Statistical inference (2nd ed.). (Duxbury advanced series; Duxbury advanced series in statistics and decision sciences). For the proof of Chebyshev’s Inequality, see p.332 from Larsen and Marx. For the proof of Strong Law of Large Numbers, see p.268 - 269 from Casella and Berger.

Zun Yin (Macalester College) Law of Large Numbers December 9th 7 / 7