Introduction to Normal Distribution Nathaniel E. Helwig Assistant Professor of Psychology and Statistics University of Minnesota (Twin Cities) Updated 17-Jan-2017 Nathaniel E. Helwig (U of Minnesota) Introduction to Normal Distribution Updated 17-Jan-2017 : Slide 1 Copyright Copyright c 2017 by Nathaniel E. Helwig Nathaniel E. Helwig (U of Minnesota) Introduction to Normal Distribution Updated 17-Jan-2017 : Slide 2 Outline of Notes 1) Univariate Normal: 3) Multivariate Normal: Distribution form Distribution form Standard normal Probability calculations Probability calculations Affine transformations Affine transformations Conditional distributions Parameter estimation Parameter estimation 2) Bivariate Normal: 4) Sampling Distributions: Distribution form Univariate case Probability calculations Multivariate case Affine transformations Conditional distributions Nathaniel E. Helwig (U of Minnesota) Introduction to Normal Distribution Updated 17-Jan-2017 : Slide 3 Univariate Normal Univariate Normal Nathaniel E. Helwig (U of Minnesota) Introduction to Normal Distribution Updated 17-Jan-2017 : Slide 4 Univariate Normal Distribution Form Normal Density Function (Univariate) Given a variable x 2 R, the normal probability density function (pdf) is 2 1 − (x−µ) f (x) = p e 2σ2 σ 2π (1) 1 (x − µ)2 = p exp − σ 2π 2σ2 where µ 2 R is the mean σ > 0 is the standard deviation( σ2 is the variance) e ≈ 2:71828 is base of the natural logarithm Write X ∼ N(µ, σ2) to denote that X follows a normal distribution. Nathaniel E. Helwig (U of Minnesota) Introduction to Normal Distribution Updated 17-Jan-2017 : Slide 5 Univariate Normal Standard Normal Standard Normal Distribution If X ∼ N(0; 1), then X follows a standard normal distribution: 1 2 f (x) = p e−x =2 (2) 2π 0.4 ) x ( 0.2 f 0.0 −4 −2 0 2 4 x Nathaniel E. Helwig (U of Minnesota) Introduction to Normal Distribution Updated 17-Jan-2017 : Slide 6 Univariate Normal Probability Calculations Probabilities and Distribution Functions Probabilities relate to the area under the pdf: Z b P(a ≤ X ≤ b) = f (x)dx a (3) = F(b) − F(a) where Z x F(x) = f (u)du (4) −∞ is the cumulative distribution function (cdf). Note: F(x) = P(X ≤ x) =) 0 ≤ F(x) ≤ 1 Nathaniel E. Helwig (U of Minnesota) Introduction to Normal Distribution Updated 17-Jan-2017 : Slide 7 Univariate Normal Probability Calculations Normal Probabilities Helpful figure of normal probabilities: 34.1% 34.1% 2.1% 2.1% 0.1% 13.6% 13.6% 0.1% 0.0 0.1 0.2 0.3 0.4 −3σ −2σ −1σ1σ µ 2σ 3σ From http://en.wikipedia.org/wiki/File:Standard_deviation_diagram.svg Nathaniel E. Helwig (U of Minnesota) Introduction to Normal Distribution Updated 17-Jan-2017 : Slide 8 Univariate Normal Probability Calculations Normal Distribution Functions (Univariate) Helpful figures of normal pdfs and cdfs: 1.0 1.0 μ= 0, σ 2= 0.2, μ= 0, σ 2= 0.2, μ= 0, σ 2= 1.0, μ= 0, σ 2= 1.0, 0.8 μ= 0, σ 2= 5.0, 0.8 μ= 0, σ 2= 5.0, μ= −2, σ 2= 0.5, μ= −2, σ 2= 0.5, ) 0.6 ) 0.6 x x ( ( 2 2 μ,σ μ,σ 0.4 -3 -2 -1 0.4 -3 -2 -1 φ Φ 0.2 0.2 0.0 0.0 −5 −3 −2−4 −1 01 2 3 4 5 −5 −3 −2−4 −1 01 2 3 4 5 x x http://en.wikipedia.org/wiki/File:Normal_Distribution_PDF.svg http://en.wikipedia.org/wiki/File:Normal_Distribution_CDF.svg Note that the cdf has an elongated “S” shape, referred to as an ogive. Nathaniel E. Helwig (U of Minnesota) Introduction to Normal Distribution Updated 17-Jan-2017 : Slide 9 Univariate Normal Affine Transformations Affine Transformations of Normal (Univariate) Suppose that X ∼ N(µ, σ2) and a; b 2 R with a 6= 0. If we define Y = aX + b, then Y ∼ N(aµ + b; a2σ2). Suppose that X ∼ N(1; 2). Determine the distributions of. Y = X + 3 Y = 2X + 3 Y = 3X + 2 Nathaniel E. Helwig (U of Minnesota) Introduction to Normal Distribution Updated 17-Jan-2017 : Slide 10 Univariate Normal Affine Transformations Affine Transformations of Normal (Univariate) Suppose that X ∼ N(µ, σ2) and a; b 2 R with a 6= 0. If we define Y = aX + b, then Y ∼ N(aµ + b; a2σ2). Suppose that X ∼ N(1; 2). Determine the distributions of. Y = X + 3 =) Y ∼ N(1(1) + 3; 12(2)) ≡ N(4; 2) Y = 2X + 3 =) Y ∼ N(2(1) + 3; 22(2)) ≡ N(5; 8) Y = 3X + 2 =) Y ∼ N(3(1) + 2; 32(2)) ≡ N(5; 18) Nathaniel E. Helwig (U of Minnesota) Introduction to Normal Distribution Updated 17-Jan-2017 : Slide 10 Univariate Normal Parameter Estimation Likelihood Function Suppose that x = (x1;:::; xn) is an iid sample of data from a normal 2 iid 2 distribution with mean µ and variance σ , i.e., xi ∼ N(µ, σ ). The likelihood function for the parameters (given the data) has the form n n 2 2 Y Y 1 (xi − µ) L(µ, σ jx) = f (xi ) = p exp − 2 2σ2 i=1 i=1 2πσ and the log-likelihood function is given by n n X n n 1 X LL(µ, σ2jx) = log(f (x )) = − log(2π)− log(σ2)− (x −µ)2 i 2 2 2σ2 i i=1 i=1 Nathaniel E. Helwig (U of Minnesota) Introduction to Normal Distribution Updated 17-Jan-2017 : Slide 11 Univariate Normal Parameter Estimation Maximum Likelihood Estimate of the Mean The MLE of the mean is the value of µ that minimizes n n X 2 X 2 2 (xi − µ) = xi − 2nx¯µ + nµ i=1 i=1 Pn where x¯ = (1=n) i=1 xi is the sample mean. Taking the derivative with respect to µ we find that @ Pn (x − µ)2 i=1 i = −2nx¯ + 2nµ ! x¯ =µ ^ @µ i.e., the sample mean x¯ is the MLE of the population mean µ. Nathaniel E. Helwig (U of Minnesota) Introduction to Normal Distribution Updated 17-Jan-2017 : Slide 12 Univariate Normal Parameter Estimation Maximum Likelihood Estimate of the Variance The MLE of the variance is the value of σ2 that minimizes n Pn 2 2 n 1 X n x nx¯ log(σ2) + (x − µ^)2 = log(σ2) + i=1 i − 2 2σ2 i 2 2σ2 2σ2 i=1 Pn where x¯ = (1=n) i=1 xi is the sample mean. Taking the derivative with respect to σ2 we find that n 2 1 Pn 2 n @ log(σ ) + 2 = (xi − µ^) n 1 X 2 2σ i 1 = − (x − µ^)2 @σ2 2σ2 2σ4 i i=1 2 Pn 2 which implies that the sample variance σ^ = (1=n) i=1(xi − x¯) is the MLE of the population variance σ2. Nathaniel E. Helwig (U of Minnesota) Introduction to Normal Distribution Updated 17-Jan-2017 : Slide 13 Bivariate Normal Bivariate Normal Nathaniel E. Helwig (U of Minnesota) Introduction to Normal Distribution Updated 17-Jan-2017 : Slide 14 Bivariate Normal Distribution Form Normal Density Function (Bivariate) Given two variables x; y 2 R, the bivariate normal pdf is n h 2 2 io exp − 1 (x−µx ) + (y−µy ) − 2ρ(x−µx )(y−µy ) 2(1−ρ2) σ2 σ2 σx σy f (x; y) = x y (5) p 2 2πσx σy 1 − ρ where µx 2 R and µy 2 R are the marginal means + + σx 2 R and σy 2 R are the marginal standard deviations 0 ≤ jρj < 1 is the correlation coefficient 2 2 X and Y are marginally normal: X ∼ N(µx ; σx ) and Y ∼ N(µy ; σy ) Nathaniel E. Helwig (U of Minnesota) Introduction to Normal Distribution Updated 17-Jan-2017 : Slide 15 Bivariate Normal Distribution Form p 2 2 Example: µx = µy = 0, σx = 1, σy = 2, ρ = 0:6= 2 0.12 4 0.12 0.10 2 0.10 0.08 ) 0.08 y , 0 ) y x 0.06 y ( , 0.06 f x ( f 0.04 0.04 4 −2 0.02 2 0 0.02 −4 −4 −2 −2 y 0 2 −4 −4 −2 0 2 4 0.00 4 x x http://en.wikipedia.org/wiki/File:MultivariateNormal.png Nathaniel E. Helwig (U of Minnesota) Introduction to Normal Distribution Updated 17-Jan-2017 : Slide 16 Bivariate Normal Distribution Form Example: Different Means µx = 0, µy = 0 µx = 1, µy = 2 µx = −1, µy = −1 0.12 0.12 0.12 4 4 4 0.10 0.10 0.10 2 2 2 0.08 0.08 0.08 ) ) ) y y y , , , 0 0 0 y y y x x x 0.06 0.06 0.06 ( ( ( f f f 0.04 0.04 0.04 −2 −2 −2 0.02 0.02 0.02 −4 −4 −4 −4 −2 0 2 4 0.00 −4 −2 0 2 4 0.00 −4 −2 0 2 4 0.00 x x x p 2 2 Note: for all three plots σx = 1, σy = 2, and ρ = 0:6= 2. Nathaniel E. Helwig (U of Minnesota) Introduction to Normal Distribution Updated 17-Jan-2017 : Slide 17 Example: Different Correlations Nathaniel E. Helwig (U of Minnesota) Note: for all three plots Bivariate Normal Introduction to Normal Distribution µ x = µ Distribution Form ρ y = −0.6/ 2 ρ = 0 ρ = 1.2/ 2 = 0.12 0.20 4 4 4 0.10 0, 0.10 2 2 2 0.08 0.15 σ 0.08 ) ) ) x 2 y y y , , , 0.06 0 0 0 y y y x x x = 0.06 0.10 ( ( ( f f f 0.04 0.04 −2 −2 −2 1, and 0.05 0.02 0.02 −4 −4 −4 Updated 17-Jan-2017 : Slide 18 −4 −2 0 2 4 0.00 −4 −2 0 2 4 0.00 −4 −2 0 2 4 0.00 x x x σ y 2 = 2.