ECE 302: Lecture 4.7 Gaussian Random Variable

Prof Stanley Chan

School of Electrical and Computer Engineering Purdue University

Overall schedule: Continuous random variables, PDF CDF Expectation Mean, mode, median Common random variables Uniform Exponential Gaussian Transformation of random variables How to generate random numbers Today’s lecture: Deﬁnition of Gaussian Mean and variance Skewness and kurtosis Origin of Gaussian c Stanley Chan 2020. All Rights Reserved. 2 / 22 Deﬁnition

Deﬁnition Let X be an Gaussian random variable. The PDF of X is

1 (x−µ)2 − 2 fX (x) = √ e 2σ (1) 2πσ2

where (µ, σ2) are parameters of the distribution. We write

X ∼ Gaussian(µ, σ2) or X ∼ N (µ, σ2)

to say that X is drawn from a Gaussian distribution of parameter (µ, σ2).

0.5 0.5 = -3 = 0.8 = -0.3 = 1 0.4 = 0 0.4 = 2 = 1.2 = 3 = 4 = 4 0.3 0.3

0.2 0.2

0.1 0.1

0 0 -10 -5 0 5 10 -10 -5 0 5 10 µ changes, σ = 1 µ = 0, σ changes Figure: A Gaussian random variable with diﬀerent µ and σ

Theorem If X ∼ N (µ, σ2), then

2 E[X ] = µ, and Var[X ] = σ . (2)

Z ∞ 2 1 − (x−µ) E[X ] = √ xe 2σ2 dx 2πσ2 −∞ Z ∞ 2 (a) 1 − y = √ (y + µ)e 2σ2 dy 2πσ2 −∞ = (b) = (c) = µ.

Theorem If X ∼ N (µ, σ2), then

2 E[X ] = µ, and Var[X ] = σ . (3)

Z ∞ (x−µ)2 1 2 − Var[X ] = √ (x − µ) e 2σ2 dx 2πσ2 −∞ 2 Z ∞ 2 (a) σ 2 − y = √ y e 2 dy, by letting y = 2π −∞ 2 2 ∞ 2 Z ∞ 2 σ − y σ − y 2 2 = √ −ye + √ e dy 2π −∞ 2π −∞ = = σ2

Deﬁnition A standard Gaussian (or standard Normal) random variable X has a PDF

2 1 − x fX (x) = √ e 2 . (4) 2π

That is, X ∼ N (0, 1) is a Gaussian with µ = 0 and σ2 = 1.

Deﬁnition The CDF of the standard Gaussian is deﬁned as the Φ(·) function

Z x 2 def 1 − t Φ(x) = FX (x) = √ e 2 dt. (5) 2π −∞

The standard Gaussian’s CDF is related to a so-called error function

which is deﬁned as x 2 Z 2 erf(x) = √ e−t dt. (6) π 0 It is quite easy to link Φ(x) with erf(x):

1 x √ Φ(x) = 1 + erf √ , and erf(x) = 2Φ(x 2) − 1. 2 2

Theorem (CDF of an arbitrary Gaussian) Let X ∼ N (µ, σ2). Then,

x − µ F (x) = Φ . (7) X σ

We start by expressing FX (x):

FX (x) = .

t−µ Substituting y = σ , and using the deﬁnition of standard Gaussian, we have

x−µ Z x (t−µ)2 Z 2 1 − σ 1 − y √ e 2σ2 dt = √ e 2 dy = −∞ 2πσ2 −∞ 2π

b − µ a − µ [a < X ≤ b] = Φ − Φ . (8) P σ σ To see this, note that b − µ a − µ [a < X ≤ b] = [X ≤ b] − [X ≤ a] = Φ − Φ . P P P σ σ

Corollary Let X ∼ N (µ, σ2). Then, the following results hold: Φ(y) = 1 − Φ(−y). b−µ P[X ≥ b] = 1 − Φ σ . b−µ −b−µ P[|X | ≥ b] = 1 − Φ σ + Φ σ

Deﬁnition

For a random variable X with PDF fX (x), deﬁne the following central moments as

def mean = E[X ] = µ, h 2i def 2 variance = E (X − µ) = σ , " # X − µ3 skewness = def= γ, E σ " # X − µ4 kurtosis = def= κ. E σ

What is skewness? 3 X −µ E σ . Measures how asymmetrical the distribution is. Gaussian has skewness 0.

0.4 positive skewness symmetric 0.3 negative skewness

0.2

0.1

0 0 5 10 15 20 Figure: Skewness of a distribution measures how asymmetric the distribution is. In this example, the skewness are: orange = 0.8943, black = 0, blue = -1.414.

What is kurtosis? 4 X −µ κ = E σ . Measures how heavy tail is. Gaussian has kurtosis 3. Some people prefer excess kurtosis κ − 3. Gaussian has excess kurtosis 0.

1 kurtosis > 0 0.8 kurtosis = 0 kurtosis < 0 0.6

0.4

0.2

0 -5 -4 -3 -2 -1 0 1 2 3 4 5 Figure: Kurtosis of a distribution measures how heavy tail the distribution is. In this example, the (excess) kurtosis are: orange = 2.8567, black = 0, blue = c Stanley Chan 2020. All Rights Reserved. -0.1242. 13 / 22 Skewness and Kurtosis

Random variable Mean Variance Skewness Excess kurtosis µ σ2 γ κ − 3 Bernoulli p p(1 − p) √1−2p 1 + 1 − 6 p(1−p) 1−p p 2 Binomial np np(1 − p) √ 1−2p 6p −6p+1 np(1−p) np(1−p) 2 1 1−p √2−p p −6p+6 Geometric p p2 1−p 1−p Poisson λ λ √1 1 λ λ a+b (b−a)2 6 Uniform 2 12 0 − 5 1 1 Exponential λ λ2 2 6 Gaussian µ σ2 0 0 Table: The ﬁrst few moments of commonly used random variables.

On April 15, 1912, RMS Titanic sank after hitting an iceberg. This has killed 1502 out of 2224 passengers and crew. A hundred years later, we want to analyze the data. On https://www.kaggle.com/c/titanic/ there is a dataset collecting the identities, age, gender, etc of the passengers.

Statistics Group 1 (Died) Group 2 (Survived) Mean 30.6262 28.3437 Standard Deviation 14.1721 14.9510 Skewness 0.5835 0.1795 Excess Kurtosis 0.2652 -0.0772

Mean and standard deviation cannot tell the diﬀerence. Skewness and kurtosis can tell the diﬀerence.

40 40

30 30

20 20

10 10

0 0 0 20 40 60 80 0 20 40 60 80 age age Group 1 (died) Group 2 (survived) Figure: The Titanic dataset https://www.kaggle.com/c/titanic/.

Where does Gaussian come from? Why are they so popular? Why do they have bell shapes?

What is the origin of Gaussian?

When we sum many independent random variables, the resulting random variable is a Gaussian. This is known as the Central Limit Theorem. The theorem applies to any random variable. Summing random variables is equivalent to convolving the PDFs. Convolving PDFs inﬁnitely many times yields the bell shape.

(a) X1 (b) X1 + X2

(c) X1 + ... + X5 (d) X1 + ... + X100 Figure: When adding uniform random variables, the overall distribution is becoming like a Gaussian.

Example: Two rectangles to give a triangle:

We will show this result in a later lecture: Z ∞ (fX ∗ fX )(x) = fX (τ)fX (x − τ)dτ. −∞

Then in Fourier domain you will have

F{(fX ∗ fX ∗ ... ∗ fX )} = F{fX }·F{fX }· ... ·F{fX }.

1.25 (sin x)/x 1 (sin x)2/x2 0.75 (sin x)3/x3 0.5 0.25 0 -0.25 -0.5 -10 -8 -6 -4 -2 0 2 4 6 8 10 Figure: Convolving the PDF of a uniform distribution is equivalent to multiplying their Fourier transforms in the Fourier space. As the number of convolution grows, the product is gradually becoming Gaussian.

What happens if you convolve a PDF inﬁnitely many times? You will get a Gaussian. This is known as the central limit theorem.

Why are Gaussians everywhere?

We seldom look at individual random variables. We often look at the sum/average. Whenever we have a sum, Central Limit Theorem kicks in. Summing random variables is equivalent to convolving the PDFs. Convolving PDFs inﬁnitely many times yields the bell shape. This result applies to any random variable, as long as they are independently summed.