Law of Large Numbers MATH 354 Zun Yin December 9th Zun Yin (Macalester College) Law of Large Numbers December 9th 1 / 7 Intuition The average of results obtained from a large number of the same trials should be close to the expected value, and will tend to become closer as more trials are performed. Example Rolling a 6-sided die for sufficiently large number of times, we get the average value as close to 3.5, the expected value, as we want. Zun Yin (Macalester College) Law of Large Numbers December 9th 2 / 7 Weak Law of Large Numbers (WLLN) Theorem (Weak Law of Large Numbers) Let X1; X2; :::; be independent and identically distributed random variables 2 ¯ 1 Pn with expected value µ and variance σ < 1. Define Xn = n i=1 Xi . Then, for every > 0, ¯ lim P(jXn − µj < ) = 1; n!1 that is, X¯n converges in probability to µ. Definition (Convergence in Probability) A sequence of random variables, X1; X2; :::, converges in probability to a random variable X if, for every > 0, lim P(jXn − X j < ) = 1. n!1 Zun Yin (Macalester College) Law of Large Numbers December 9th 3 / 7 Proof of WLLN Theorem (Chebyshev's Inequality) Let W be any random variable with mean µ and variance σ2. For any > 0, σ2 P(jW − µj < ) ≥ 1 − 2 . Proof (WLLN): According to Chebyshev's Inequality, ¯ ¯ ¯ Var(Xn) P(jXn − E(Xn)j < ) ≥ 1 − 2 . But E(X¯n) = E(Xi ) = µ, ¯ 1 Pn 1 Pn 1 2 σ2 Var(Xn) = Var( n i=1 Xi ) = n2 i=1 Var(Xi ) = n2 ∗ nσ = n , ¯ σ2 so P(jXn − µj < ) ≥ 1 − n2 . σ2 ¯ As n ! 1, 2 ! 0. Therefore lim P(jXn − µj < ) = 1. n n!1 Zun Yin (Macalester College) Law of Large Numbers December 9th 4 / 7 Strong Law of Large Numbers (SLLN) Theorem (Strong Law of Large Numbers) Let X1; X2; :::; be independent and identically distributed random variables 2 ¯ 1 Pn with expected value µ and variance σ < 1. Define Xn = n i=1 Xi . Then, for every > 0, ¯ P( lim jXn − µj < ) = 1; n!1 that is, X¯n converges almost surely to µ. Definition (Almost Sure Convergence) A sequence of random variables, X1; X2; :::, converges almost surely to a random variable X if, for every > 0, P( lim jXn − X j < ) = 1. n!1 Zun Yin (Macalester College) Law of Large Numbers December 9th 5 / 7 SLLN is Stronger than WLLN SLLN is stronger than WLLN because almost sure convergence is more restrictive than convergence in probability. Convergence in Probability: lim P(jXn − X j < ) = 1 n!1 Almost Sure Convergence: P( lim jXn − X j < ) = 1 n!1 Example Let the sample space S be the closed interval [0,1] with the uniform probability distribution. Define the sequence X1; X2; :::; as follows: X1(s) = I[0;1](s); X2(s) = I 1 (s); X3(s) = I 1 (s); [0; 2 ] ( 2 ;1] X4(s) = I 1 (s); X5(s) = I 1 2 (s); X6(s) = I 2 (s); [0; 3 ] ( 3 ; 3 ] ( 3 ;1] etc. Let X (s) = 0. Then lim P(Xn < ) = 1 is true, n!1 P( lim Xn < ) = 1 is not true. n!1 Zun Yin (Macalester College) Law of Large Numbers December 9th 6 / 7 Reference Larsen, R. J., Marx, M. L. (2010). An introduction to mathematical statistics and its applications (5th ed.). Boston, Mass.: Prentice Hall. Casella, G., Berger, R. L. (2002). Statistical inference (2nd ed.). (Duxbury advanced series; Duxbury advanced series in statistics and decision sciences). For the proof of Chebyshev's Inequality, see p.332 from Larsen and Marx. For the proof of Strong Law of Large Numbers, see p.268 - 269 from Casella and Berger. Zun Yin (Macalester College) Law of Large Numbers December 9th 7 / 7.