Probability Distributions CEE 201L

Home , Median, Mode (statistics), Probability distribution

Probability Distributions CEE 201L. Uncertainty, Design, and Optimization Department of Civil and Environmental Engineering Duke University Philip Scott Harvey, Henri P. Gavin and Jeﬀrey T. Scruggs Spring 2022

1 Probability Distributions

Consider a continuous, random variable (rv) X with support over the domain X . The probability density function (PDF) of X is the function fX (x) such that for any two numbers a and b in the domain X , with a < b, Z b P [a < X ≤ b] = fX (x) dx a For fX (x) to be a proper distribution, it must satisfy the following two conditions:

1. The PDF fX (x) is positive-valued; fX (x) ≥ 0 for all values of x ∈ X . R 2. The rule of total probability holds; the total area under fX (x) is 1; X fX (x) dx = 1.

Alternately, X may be described by its cumulative distribution function (CDF). The CDF of X is the function FX (x) that gives, for any speciﬁed number x ∈ X , the probability that the random variable X is less than or equal to the number x is written as P [X ≤ x]. For real values of x, the CDF is deﬁned by Z b FX (x) = P [X ≤ b] = fX (x) dx , −∞ so, P [a < X ≤ b] = FX (b) − FX (a)

By the ﬁrst fundamental theorem of calculus, the functions fX (x) and FX (x) are related as d f (x) = F (x) X dx X 2 CEE 201L. Uncertainty, Design, and Optimization – Duke University – Spring 2022 – P.S.H., H.P.G. and J.T.S.

A few important characteristics of CDF’s of X are:

1. CDF’s, FX (x), are monotonic non-decreasing functions of x.

2. For any number a, P [X > a] = 1 − P [X ≤ a] = 1 − FX (a) R b 3. For any two numbers a and b with a < b, P [a < X ≤ b] = FX (b) − FX (a) = a fX (x)dx

2 Descriptors of random variables

The expected or mean value of a continuous random variable X with PDF fX (x) is the centroid of the probability density. Z ∞ µX = E[X] = x fX (x) dx −∞ The expected value of an arbitrary function of X, g(X), with respect to the PDF fX (x) is Z ∞ µg(X) = E[g(X)] = g(x) fX (x) dx −∞

The variance of a continuous rv X with PDF fX (x) and mean µX gives a quantitative measure of how much spread or dispersion there is in the distribution of x values. The variance is calculated as Z ∞ 2 2 σX = V[X] = (x − µX ) fX (x) dx −∞ = = = =

p The standard deviation (s.d.) of X is σX = V[X]. The coeﬃcient of variation (c.o.v.) of X is deﬁned as the ratio of the standard deviation σX to the mean µX :

σX cX = µX for non-zero mean. The c.o.v. is a normalized measure of dispersion (dimensionless).

A mode of a probability density function, fX (x), is a value of x such that the PDF is maximized;

d fX (x) = 0 .

dx x=xmode

The median value, xm, is is the value of x such that

P [X ≤ xm] = P [X > xm] = FX (xm) = 1 − FX (xm) = 0.5 .

CC BY-NC-ND March 25, 2021 PSH, HPG, JTS Probability Distributions 3

3 Some common distributions

The National Institute of Standards and Technology (NIST) lists properties of nineteen commonly used probability distributions in their Engineering Statistics Handbook. This section describes the properties of seven distributions. For each of these distributions, this document provides ﬁgures and equations for the PDF and CDF, equations for the mean and variance, the names of Matlab functions to generate samples, and empirical distributions of such samples.

3.1 The Normal distribution

The Normal (or Gaussian) distribution is perhaps the most commonly used distribution function. 2 The notation X ∼ N (µX , σX ) denotes that X is a normal random variable with mean µX and 2 variance σX . The standard normal random variable, Z, or “z-statistic”, is distributed as N (0, 1). The probability density function of a standard normal random variable is so widely used it has its own special symbol, φ(z), ! 1 z2 φ(z) = √ exp − 2π 2 Any normally distributed random variable can be deﬁned in terms of the standard normal random variable, through the change of variables

X = µX + σX Z.

If X is normally distributed, it has the PDF

2 ! x − µX 1 (x − µX ) fX (x) = φ = q exp − 2 σX 2 2σ 2πσX X

There is no closed-form equation for the CDF of a normal random variable. Solving the integral

z 1 Z 2 Φ(z) = √ e−u /2 du 2π −∞ would make you famous. Try it. The CDF of a normal random variable is expressed in terms of the error function, erf(z). If X is normally distributed, P [X ≤ x] can be found from the standard normal CDF x − µX P [X ≤ x] = FX (x) = Φ . σX Values for Φ(z) are tabulated and can be computed, e.g., the Matlab command . . . Prob_X_le_x = normcdf(x,muX,sigX). The standard normal PDF is symmetric about z = 0, so φ(−z) = φ(z), Φ(−z) = 1 − Φ(z), and P [X > x] = 1 − FX (x) = 1 − Φ ((x − µX )/σX ) = Φ ((µX − x)/σX ).

The linear combination of two independent normal rv’s X1 and X2 (with means µ1 and µ2 and 2 2 variances σ1 and σ2) is also normally distributed,

2 2 2 2 aX1 + bX2 ∼ N aµ1 + bµ2, a σ1 + b σ2 ,

2 2 and more speciﬁcally, aX − b ∼ N aµX − b, a σX .

CC BY-NC-ND March 25, 2021 PSH, HPG, JTS 4 CEE 201L. Uncertainty, Design, and Optimization – Duke University – Spring 2022 – P.S.H., H.P.G. and J.T.S.

Given the probability of a normal rv, i.e., given P [X ≤ x], the associated value of x can be found from the inverse standard normal CDF, x − µ X = z = Φ−1(P [X ≤ x]) . σX Values of the inverse standard normal CDF are tabulated, and can be computed, e.g., the Matlab command . . . x = norminv(Prob_X_le_x,muX,sigX).

3.2 The Log-Normal distribution

The Normal distribution is symmetric and can be used to describe random variables that can take positive as well as negative values, regardless of the value of the mean and standard deviation. For many random quantities a negative value makes no sense (e.g., modulus of elasticity, air pressure, and distance). Using a distribution which admits only positive values for such quantities eliminates any possibility of non-sensical negative values. The log-normal distribution is such a distribution.

If ln X is normally distributed (i.e., ln X ∼ N (µln X , σln X )) then X is called a log-normal random variable. In other words, if Y (= ln X) is normally distributed, eY (= X) is log-normally distributed. P [Y ≤ y] FY (y) Φ y−µY 2 2 σY µY = µln X , σY = σln X , P [ln X ≤ ln x] = Fln X (ln x) = Φ ln x−µln X P [X ≤ x] FX (x) σln X The mean and standard deviation of a log-normal variable X are related to the mean and standard deviation of ln X. 1 µ = ln µ − σ2 σ2 = ln 1 + (σ /µ )2 ln X X 2 ln X ln X X X

If (σX /µX ) < 0.30, σln X ≈ (σX /µX ) = cX

The median, xm, is a useful parameter of log-normal rv’s. By deﬁnition of the median value, half of the population lies above the median, and half lies below, so ln x − µ Φ m ln X = 0.5 σln X ln x − µ m ln X = Φ−1(0.5) = 0 σln X

q 2 and, ln xm = µln X ↔ xm = exp(µln X ) ↔ µX = xm 1 + cX

For the log-normal distribution xmode < xmedian < xmean. If cX < 0.15, xmedian ≈ xmean.

If ln X is normally distributed (X is log-normal) then (for cX < 0.3) ln x − ln x P [X ≤ x] ≈ Φ m cX

2 2 n m If ln X ∼ N (µln X , σln X ), and ln Y ∼ N (µln Y , σln Y ), and Z = aX /Y then 2 ln Z = ln a + n ln X − m ln Y ∼ N (µln Z , σln Z )

where µln Z = ln a + nµln X − mµln Y = ln a + n ln xm − m ln ym 2 2 2 2 2 2 2 2 and σln Z = (nσln X ) + (mσln Y ) = n ln(1 + cX ) + m ln(1 + cY ) = ln(1 + cZ )

CC BY-NC-ND March 25, 2021 PSH, HPG, JTS Probability Distributions 5

Uniform X ∼ U[a, b] Triangular X ∼ T (a, b, c) a ≤ X ≤ b; a ≤ X ≤ b, a ≤ c ≤ b

1/(b-a) 2/(b-a) p.d.f., f(x) p.d.f., f(x)

0 0 a µ-σ µ µ+ σ b a µ-σc µ µ+ σ µ+2 σ b

0.97 0.79 0.82

0.5 0.55 c.d.f., F(x) c.d.f., F(x)

0.21 0.17

a µ-σ µ µ+ σ b a µ-σc µ µ+ σ µ+2 σ b x x

 2(x−a) (  , x ∈ [a, c] 1 , x ∈ [a, b]  (b−a)(c−a) f(x) = b−a f(x) = 2(b−x) , x ∈ [c, b] 0, otherwise  (b−a)(b−c)  0, otherwise

 0, x ≤ a   2  0, x ≤ a  (x−a)  x−a  (b−a)(c−a) , x ∈ [a, c] F (x) = b−a , x ∈ [a, b] F (x) = (b−x)2   1 − , x ∈ [c, b]  1, x ≥ b  (b−a)(b−c)  1, x ≥ b

1 1 µX = 2 (a + b) µX = 3 (a + b + c) 2 1 2 2 1 2 2 2 σX = 12 (b − a) σX = 18 (a + b + c − ab − ac − bc) x = a + (b-a)*rand(1,N); x = triangular rnd(a,b,c,1,N);

0.8 0.8

0.6 0.6

0.4 0.4

empirical p.d.f. 0.2 empirical p.d.f. 0.2

0 0 0 0.5 1 1.5 2 0 0.5 1 1.5 2 2.5

1 1 0.8 0.8 0.6 0.6 0.4 0.4

empirical c.d.f. 0.2 µ=1.0 σ=0.5 empirical c.d.f. 0.2 µ=1.0 σ=0.5 0 0 0 0.5 1 1.5 2 0 0.5 1 1.5 2 2.5 3 x x

CC BY-NC-ND March 25, 2021 PSH, HPG, JTS 6 CEE 201L. Uncertainty, Design, and Optimization – Duke University – Spring 2022 – P.S.H., H.P.G. and J.T.S.

Exponential X ∼ E(µ) Laplace X ∼ L(µ, σ2) + + + X ∈ R , µ ∈ R X ∈ R, µ ∈ R, σ ∈ R

µ p.d.f., f(x) 1/(e ) p.d.f., f(x)

0 0 0 µ 2µ 3µ 4µ µ-3σ µ-2σ µ-σ µ µ+ σ µ+2 σ µ+3 σ

0.95 0.97 0.86 0.88

0.63 0.5 c.d.f., F(x) c.d.f., F(x)

0.12 0.03 0 µ 2µ 3µ 4µ µ - 2σ µ-σ µ µ+ σ µ+2 σ x x √ 1 x 2 √ |x − µ| f(x) = exp − f(x) = exp − 2 µ µ 2σ σ  √ 1 |x−µ| x  2 exp 2 σ x < µ F (x) = 1 − exp − F (x) = √ µ 1 |x−µ|  1 − 2 exp − 2 σ x ≥ µ

µX = µ µX = µ

2 2 2 2 σX = µ σX = σ x = exp rnd(muX,1,N); x = laplace rnd(muX,sigmaX,1,N);

0.8 1 0.8 0.6

0.6 0.4 0.4

empirical p.d.f. empirical p.d.f. 0.2 0.2 0 0 0 1 2 3 4 5 -2 -1 0 1 2 3 4

1 1 0.8 0.8 0.6 0.6 0.4 0.4

empirical c.d.f. 0.2 µ=1.0 empirical c.d.f. 0.2 µ=1.0 σ=1.0 0 0 0 1 2 3 4 5 -2 -1 0 1 2 3 4 x x

CC BY-NC-ND March 25, 2021 PSH, HPG, JTS Probability Distributions 7

2 2 Normal X ∼ N (µ, σ ) Log-Normal ln X ∼ N (µln X , σln X ) + + + + X ∈ R, µ ∈ R, σ ∈ R X ∈ R , µln X ∈ R , σln X ∈ R p.d.f., f(x) p.d.f., f(x)

0 0 µ+3 σ µ-2σ µ-σ µ µ+ σ µ+2 σ µ+3 σ 0 µ-σ µ µ+ σ µ+2 σ µ+3 σ µ+4 σ

0.98 0.96 0.84 0.86

0.59 0.5 c.d.f., F(x) c.d.f., F(x)

0.16 0.11 0.02 µ+3 σ µ-2σ µ-σ µ µ+ σ µ+2 σ µ+3 σ 0 µ-σ µ µ+ σ µ+2 σ µ+3 σ µ+4 σ x x

2 ! 2 ! 1 (x − µ) 1 (ln x − µln X ) f(x) = √ exp − f(x) = q exp − 2 2πσ2 2σ2 2 2σ x 2πσln X ln X    1 x − µ 1 ln x − µln X F (x) = 1 + erf √ F (x) = 1 + erf  q  2 2σ2 2 2 2σln X

q 2 µX = µ µX = xm 1 + cX

2 2 2 2 2 2 σX = σ σX = xm cX 1 + cX

1 2 µln X = ln xm = ln µX − 2 σln X 2 2 σln X = ln 1 + (σX /µX )

x = muX + sigmaX*randn(1,N); x = logn rnd(Xm,Cx,1,N);

0.8 1 0.6 0.8

0.4 0.6 0.4

empirical p.d.f. 0.2 empirical p.d.f. 0.2 0 0 -1 0 1 2 3 0 0.5 1 1.5 2 2.5 3

1 1 0.8 0.8 0.6 0.6 0.4 0.4

empirical c.d.f. 0.2 µ=1.0 σ=0.5 empirical c.d.f. 0.2 µ=1.0 σ=0.5 0 0 -1 0 1 2 3 0 0.5 1 1.5 2 2.5 3 x x CC BY-NC-ND March 25, 2021 PSH, HPG, JTS 8 CEE 201L. Uncertainty, Design, and Optimization – Duke University – Spring 2022 – P.S.H., H.P.G. and J.T.S.

Rayleigh X ∼ R(m) Gamma X ∼ Γ(k, θ) + + + + + X ∈ R , m ∈ R X ∈ R , µ ∈ R , σ ∈ R p.d.f., f(x) p.d.f., f(x)

0 0 0 µ-σ m µ µ+ σ µ+2 σ µ+3 σ µ-2σ µ-σ µ µ+ σ µ+2 σ µ+3 σ µ+4 σ

0.96 0.96 0.84 0.85

0.54 0.57 c.d.f., F(x) c.d.f., F(x)

0.16 0.14 0 0 µ-σ m µ µ+ σ µ+2 σ µ+3 σ µ - 2σ µ-σ µ µ+ σ µ+2 σ µ+3 σ µ+4 σ x x ! x 1 x 2 1 x f(x) = exp − f(x) = xk−1 exp − m2 2 m Γ(k)θk θ ! 1 x 2 1 x F (x) = 1 − exp − F (x) = γ k, 2 m Γ(k) θ

p µX = m π/2 µX = kθ

2 2 2 2 σX = m (4 − π)/2 σX = kθ

x = Rayleigh rnd(modeX,1,N); x = gamma rnd(muX,covX,1,N);

1 1

0.8 0.8

0.6 0.6

0.4 0.4

empirical p.d.f. 0.2 empirical p.d.f. 0.2

0 0 0 0.5 1 1.5 2 2.5 3 0 0.5 1 1.5 2 2.5 3

1 1 0.8 0.8 0.6 0.6 0.4 0.4

empirical c.d.f. 0.2 µ=1.0 empirical c.d.f. 0.2 µ=1.0 σ=0.5 0 0 0 0.5 1 1.5 2 2.5 3 0 0.5 1 1.5 2 2.5 3 x x

CC BY-NC-ND March 25, 2021 PSH, HPG, JTS Probability Distributions 9

Extreme I X ∼ EI [m, s] Extreme II X ∼ EII (m, s, k) + + X ∈ R, m ∈ R, s ∈ R , X > m, m ∈ R, s ∈ R , k > 0, k 6= 1, 2 p.d.f., f(x) p.d.f., f(x)

0 0 µ-2σ µ-σ µ µ+ σ µ+2 σ µ+3 σ µ+4 σ 0 µ-σ µ µ+ σ µ+2 σ µ+3 σ µ+4 σ

0.96 0.96 0.86 0.9 0.65 0.57 c.d.f., F(x) c.d.f., F(x)

0.13 0 0.01 µ-2σ µ-σ µ µ+ σ µ+2 σ µ+3 σ µ+4 σ 0 µ-σ µ µ+ σ µ+2 σ µ+3 σ µ+4 σ x x " # 1 x − m x − m k x − m−1−k x − m−k f(x) = exp − + exp − f(x) = exp − s s s s s s ( h i x − m exp − x−m −k x > m F (x) = exp − exp − F (x) = s s 0 otherwise

µX = m + γs , γ ≈ 0.5772 µX = m + sΓ(1 − 1/k)

2 2 2 2 2 2 σX = (π /6) s σX = s [Γ(1 − 2/k) − (Γ(1 − 1/k)) ] x = extI rnd(mu,cv,1,N); x = extII rnd(m,s,k,1,N);

1 2

0.8 1.5 0.6 1 0.4

empirical p.d.f. empirical p.d.f. 0.5 0.2

0 0 0 0.5 1 1.5 2 2.5 3 0 0.5 1 1.5 2 2.5 3

1 1 0.8 0.8 0.6 0.6 0.4 0.4

empirical c.d.f. 0.2 µ=1.0 σ=0.5 empirical c.d.f. 0.2 µ=1.0 σ=0.5 0 0 0 0.5 1 1.5 2 2.5 3 0 0.5 1 1.5 2 2.5 3 x x

CC BY-NC-ND March 25, 2021 PSH, HPG, JTS 10 CEE 201L. Uncertainty, Design, and Optimization – Duke University – Spring 2022 – P.S.H., H.P.G. and J.T.S.

4 Sums and Diﬀerences of Independent Normal Random Variables

2 2 Consider two normally distributed random variables, X ∼ N (µX , σX ) and Y ∼ N (µY , σY ). Any weighted sum of normal random variables is also normally distributed.

Z = aX − bY

2 2 Z ∼ N aµX − bµY , (aσX ) + (bσY )

2 2 2 µZ = aµX − bµY σZ = (aσX ) + (bσY )

5 Products and Quotients of Independent LogNormal Random Variables

2 2 Consider two log-normally distributed random variables, ln X ∼ N (µln X , σln X ) and ln Y ∼ N (µln Y , σln Y ). Any product or quotient of lognormal random variables is also lognormally distributed.

Z = X/Y

2 2 ln Z ∼ N µln X − µln Y , σln X + σln Y

2 2 2 2 2 2 2 2 µln Z = µln X − µln Y σln Z = σln X + σln Y cZ = cX + cY + cX cY

6 Examples

1. The strength, S, of a particular grade of steel is log-normally distributed with median 36 ksi and c.o.v. of 0.15. What is the probability that the strength of a particular sample is greater than 40 ksi? ln 40 − ln 36 3.69 − 3.58 P [S > 40] = 1 − P [S ≤ 40] = 1 − Φ = 1 − Φ 0.15 0.15 = 1 − Φ(0.702) = 1 − 0.759 = 0.241

0.08

smode = 35.20 ksi 0.07 smedian = 36.00 ksi smean = 36.40 ksi 0.06

0.05

0.04 p.d.f.

0.03

0.02

0.01 P[S>40]

0 20 25 30 35 40 45 50 55 60 strength of steel, s, ksi

CC BY-NC-ND March 25, 2021 PSH, HPG, JTS Probability Distributions 11

2. Highway truck weights in Michigan, W , are assumed to be normally distributed with mean 100 k and standard deviation 40 k. The load capacity of bridges in Michigan, R, are also assumed to be normally distributed with mean 200 k and standard devation 30 k. What is the probability of a truck exceeding a bridge load rating? E = W − R. If E > 0 the truck weight eceededs the bridge capacity. µE = √µW − µR = 100 − 200 = −100 k. 2 2 σE = 40 + 30 = 50 k. 0 − (−100) P [E > 0] = 1 − P [E ≤ 0] = 1 − Φ = 1 − Φ(2) = 1 − 0.977 = 0.023 50 R W E=W−R 40 30

50 −100 0 100 200 k

3. Windows in the Cape Hattaras Lighthouse can withstand wind pressures of R. R is log- normal with median of 40 psf and coefficient of variation of 0.25. The peak wind pressure during a hurricane P in psf is given by the equation P = 1.165 × 10−3CV 2 where C is a log-normal coefficient with median of 1.8 and coefficient of variation of 0.20 and V is the wind speed with median 100 fps and coefficient of variation of 0.30. What is the probability of the wind pressure exceeding the strength of the window? The peak wind pressure is also log-normal. ln P = ln(1.165 × 10−3) + ln C + 2 ln V −3 µln P = ln(1.165 × 10 ) + µln C + 2µln V −3 µln P = ln(1.165 × 10 ) + ln(1.8) + 2 ln(100) = 3.0431 2 2 2 σln P = ln(1 + 0.20 ) + 2 ln(1 + 0.30 ) = 0.2116 . . . σln P = 0.4600

The wind pressure exceeds the resistance if P/R > 1 (that is, if ln P − ln R > 0) ln E = ln P − ln R

µln E = µln P − µln R = 3.0431 − ln(40) = −0.646 2 2 σln E = 0.2116 + ln(1 + 0.25 ) = 0.27220 . . . σln E = 0.5217

The probability of the wind load load exceeding the resistance of the glass is, 0 + 0.646 P [E > 1] = 1 − P [E ≤ 1] = 1 − P [ln E ≤ 0] = 1 − Φ = 1 − Φ(1.2383) = 0.11 0.5217

4. When a strong earthquake (i.e., M > 6) occurs, the intensity of the ground shaking (e.g., peak ground accelration) at a nearby building site is uncertain. Assume that the peak ground acceleration (PGA) from a strong earthquake is log-normally distributed with median of 0.2 g and a coeﬃcient of variation of 0.25. Assume that the building will sustain no damage for ground motion shaking up to 0.3 g. What is the probability the building will be damaged from an earthquake of M > 6? ln(0.3) − ln(0.2) P [D|M > 6] = P [P GA > 0.3] = 1−P [P GA ≤ 0.3] = 1−Φ = 1−0.947 = 0.053. 0.25

CC BY-NC-ND March 25, 2021 PSH, HPG, JTS 12 CEE 201L. Uncertainty, Design, and Optimization – Duke University – Spring 2022 – P.S.H., H.P.G. and J.T.S.

There have been two earthquakes with M > 6 in the last 50 years. What is the probability of no damage from earthquakes with M > 6 in the next 20 years?

P [D0|M > 6] = 1 − 0.053 = 0.947

From the theorem of total probability,

P [D0] in 20 yr = P [D0|0 EQ M > 6 in 20yr] · P [0 EQ M > 6 in 20yr] + P [D0|1 EQ M > 6 in 20yr] · P [1 EQ M > 6 in 20yr] + P [D0|2 EQ M > 6 in 20yr] · P [2 EQ M > 6 in 20yr] + P [D0|3 EQ M > 6 in 20yr] · P [3 EQ M > 6 in 20yr] + ···

where P [D0|n EQ M > 6] = (P [D0|1 EQ M > 6])n (assuming damage from an earthquake does not weaken the building . . . ) So,

∞ (20/25)n P [D0] in 20 yr = X(0.947)n exp(−20/25) n! n=0 " # 0.8 0.82 0.83 = exp(−0.8) 1 + 0.947 + (0.947)2 + (0.947)3 + ··· 1! 2! 3! = exp(−0.8) · exp(0.947 · 0.8) = 0.958

So the probability of damage from earthquakes in the next 20 years (given the assumptions in this example) is close to 4%. Is 4% in 20 years an acceptable level of risk to you for the possibility of earthquake damage to your home?

7 Empirical PDFs, CDFs, and exceedence rates (nonparametric statistics)

The PDF and CDF of a sample of random data can be computed directly from the sample, without assuming any particular probability distribution . . . (such as a normal, exponential, or other kind of distribution).

A random sample of N data points can be sorted into increasing numerical order, so that

x1 ≤ x2 ≤ · · · ≤ xi−1 ≤ xi ≤ xi+1 ≤ · · · ≤ xN−1 ≤ xN .

In the ordered sample there are i data points less than or equal to xi. So, if the sample is representative of the population, and the sample is “big enough” the probability that a random X is less th than or equal to the i ordered value is i/N. In other words, P [X ≤ xi] = i/N. Unless we know that no value of X can exceed xN , we must accept some probability that X > xN . So, P [X ≤ xN ] 1 should be less than 1. In such cases, the unbiased estimate E[FX (xi)] for P [X ≤ xi] is i/(N + 1) The empirical CDF computed from a ordered sample of N values is i Fˆ (x ) = X i N + 1

The empirical PDF is basically a histogram of the data. The following Matlab lines plot empirical CDFs and PDFs from a vector of random data, x.

1E.J. Gumbel, Statistics of extremes, Columbia Univ Press, 1958 Lasse Makkonen, “Problems in the extreme value analysis,” Structural Safety 2008:30:405-419

CC BY-NC-ND March 25, 2021 PSH, HPG, JTS Probability Distributions 13

1 N = length(x); % number of values in the sample 2 nBins = floor (N /50); % number of bins in the histogram 3 [fx ,xx] = hist (x, nBins ); % compute the histogram 4 fx = fx / N * nBins/(max(x)-min(x ))); % scale the histogram toa PDF 5 F_x = ([1:N])/(N+1); % empirical CDF 6 subplot (211); bar(xx ,fx ); % plot empirical PDF 7 subplot (212); s t a i r s ( sort (x), F_x ); % plot empirical CDF 8 probability_of_failure = sum(x >0) / N % probability thatX > 0

The number of values in the sample greater than xi is (N − i). If the sample is representative, the probability of a value exceeding xi is Prob[X > xi] = 1 − FX (xi) ≈ 1 − i/N. If the N samples were collected over a period of time T , the average exceedence rate (number of events greater than xi per unit time) is ν(xi) = N(1 − FX (xi))/T ≈ N(1 − i/N)/T = (N − i)/T . 8 Random variable generation using the Inverse CDF method

A sample of a random variable having virtually any type of CDF, P [X ≤ x] = P = FX (x) can be generated from a sample of a uniformly distributed random variable, U, (0 < U < 1), as −1 long as the inverse CDF, x = FX (P ) can be computed. There are many numerical methods for generating a sample of uniformly distributed random numbers. It is important to be aware that samples from some methods are “more random” than samples from others. The Matlab command u = rand(1,N) computes a (row) vector sample of N uniformly distributed random numbers with 0 < u < 1.

If X is a continuous rv with CDF FX (x) and U has a uniform distribution on (0, 1), then the −1 random variable FX (U) has the distribution FX . Thus, in order to generate a sample of data distributed according to the CDF FX , it suﬃces to generate a sample, u, of the rv U ∼ U[0, 1] and −1 then make the transformation x = FX (u). For example, if X is exponentially distributed, the CDF of X is given by −x/µ FX (x) = 1 − e , so −1 FX (u) = −µ ln(1 − FX (x)). Therefore if u is a value from a uniformly distributed rv in [0, 1], then x = −µ ln(u) is a value from an exponentially distributed random variable. (If U is uniformly distributed in [0,1] then so is 1 − U.)

As another example, if X is log-normally distributed, the CDF of X is ln x − ln xm FX (x) = Φ . σln X If u is a sample from a standard uniform distribution, then h −1 i x = exp ln xm + Φ (u)σln X is a sample from a lognormal distribution.

Note that since expressions for Φ(z) and Φ−1(P ) do not exist, the generation of normally distributed random variables requires other numerical methods. x = muX + sigX*randn(1,N) computes a (row) vector sample of N normally distributed random numbers.

CC BY-NC-ND March 25, 2021 PSH, HPG, JTS 14 CEE 201L. Uncertainty, Design, and Optimization – Duke University – Spring 2022 – P.S.H., H.P.G. and J.T.S.

cdf

1 cdf 1 F(x) F(x) X X u u u = F (x) u = F (x) X X

−1 −1 U x = F (u) U

pdf f (u) x = F (u) X pdf f (u) X 0 0 1 x=a x=b x 1 x=a x=b x

pdf pdf

dFX dFX f (x) = f (x) = X dx X dx

x=a x=b x x=a x=b x

Figure 1. Examples of the generation of uniform random variables from the inverse CDF method.

u cdf u cdf u = F (x) u = FX (x)

X 1 1 F(x) F(x)X X

−1 −1 x = F (u) x = F X (u) U U X pdf f (u) pdf f (u) 0 0 1 1

1 x x

pdf pdf

dFX dFX f (x) = f (x) = X dx X dx

x x

Figure 2. Examples of the generation of random variables from the inverse CDF method. The density of −1 the horizontal arrows u is uniform, whereas the density of the vertical arrows, x = FX (u), is proportional 0 to FX (x), that is, proportional to fX (x).

CC BY-NC-ND March 25, 2021 PSH, HPG, JTS Probability Distributions 15

9 Correlated Random Variables

If the cumulative distribution function FX (x) is invertible, the random variable X can be related to a standard normal random variable Y through ΦY (y) = FX (x), from which a sample of values of Y can be calculated from a sample of values of X as −1 [y1, y2, ..., ym] = Φ (FX ([x1, x2, ..., xm])) and a sample of values of X can be calculated from a sample of values of Y as −1 [x1, x2, ..., xm] = F (ΦY ([y1, y2, ..., ym]))

A sample of m values of a set of n correlated standard normal random variables {Y1,Y2, ..., Yn} with a given correlation matrix Rn×n can be computed from a sample of m values of a set of n uncorrelated standard normal variables {Z1,Z2, ..., Zn} from the eigenvalues and eigenvectors of R, R = V ΛV T.     y11 y12 y13 ··· y1m z11 z12 z13 ··· z1m  y y y ··· y   z z z ··· z   21 22 23 2m  1/2  21 22 23 2m   . . . .  = V Λ  . . . .   . . . .   . . . .   . . . ··· .   . . . ··· .  yn1 yn2 yn3 ··· ynm zn1 zn2 zn3 ··· znm where yij is sample j from random variable Yi and zij is sample j from random variable Zi.

To compute this in Matlab,

[eVec,eVal] = eig(R); Z = randn(n,m); Y = eVec * sqrt(eVal) * Z; X = inv_cdf(normcdf(Y),a,b); where cdf_inv() is a particular inverse cumulative distribution function such as logn_inv(normcdf(Y),medianX,covX); or Rayleigh_inv(normcdf(Y),modeX); .

10 Functions of Random Variables and Monte Carlo Simulation

The probability distributions of virtually any function of random variables can be computed using the powerful method of Monte Carlo Simulation (MCS). MCS involves computing values of functions with large samples of random variables.

For example, consider a function of three random variables, X1, X2, and X3, where X1 is normally distributed with mean of 6 and standard deviation of 2, X2 is log-normally distributed with median of 2 and coeﬃcient of variation of 0.3, and X3 is Rayleigh distributed with mode of 1. The function p Y = sin(X1) + X2 − exp(−X3) − 2 is a function of these three random variables and is therefore also random. The distribution function and statistics of Y may be diﬃcult to derive analytically, especially if the function Y = g(X) is complicated. This is where MCS is powerful. Given samples of N values of X1, X2 and X3, a sample of N values of Y can also be computed. The statistics of Y (mean, variance, PDF, and CDF) can be estimated by computing the average value, sample variance, histogram, and emperical CDF of the sample of Y . The probability P [Y > 0] can be estimated by counting the number of positive values in the sample and dividing by N. The Matlab command P_Y_gt_0 = sum(y>0)/N may be used to estimate this probability.

CC BY-NC-ND March 25, 2021 PSH, HPG, JTS 16 CEE 201L. Uncertainty, Design, and Optimization – Duke University – Spring 2022 – P.S.H., H.P.G. and J.T.S.

11 Monte Carlo Simulation in Matlab

1 %MCS intro.m 2 % Monte Carlo Simulation... an introductory example 3 % 4 %Y=g(X1, X2, X3)= sin(X1)+ sqrt(X2) − exp(−X3) − 2 ; 5 % 6 %H.P. Gavin, Dept. Civil and Environmental Engineering, Duke Univ, Jan. 2012 7 8 % X1 X2 X3 9 % normal lognormal Rayleigh 10 mu1=6; med2=2; mod3=1; 11 sd1=2; cv2=0.3; 12 13 N = 1000; % number of random values in the sample 14 15 % (1) generatea large sample for each random variable in the problem... 16 17 X1 = mu1 + sd1*randn(1,N); 18 X2 = logn_rnd(med2,cv2,1,N); 19 X3 = Rayleigh_rnd(mod3,1,N); 20 21 % (2) evaluate the function for each random variable to computea new sample 22 23 Y = sin (X1) + sqrt (X2) - exp(-X3) - 2; 24 25 % suppose”probability of failure” is Prob[g(X1, X2, X3) > 0]... 26 27 Probability_of_failure = sum(Y >0) / N 28 29 % (3) plot histograms of the random variables 30 31 sort_X1 = sort (X1 ); 32 sort_X2 = sort (X2 ); 33 sort_X3 = sort (X3 ); 34 CDF = [1:N] / (1+N); % empirical CDF of all quantities

CC BY-NC-ND March 25, 2021 PSH, HPG, JTS Probability Distributions 17

0.6 0.6 0.5 0.15 0.4 0.4 0.1 0.3 P.D.F.

0.2 0.2 0.05 0.1

0 0 0 0 2 4 6 8 10 1 2 3 4 5 0.5 1 1.5 2 2.5 3 3.5

0.8 0.8 0.8

0.6 0.6 0.6

C.D.F. 0.4 0.4 0.4

0.2 0.2 0.2

0 2 4 6 8 10 1 2 3 4 5 0.5 1 1.5 2 2.5 3 3.5 X : normal X : log-normal X : Rayleigh 1 2 3

0.8

0.6

0.4 P.D.F.

0.2

0 -3 -2 -1 0 1

0.8

0.6 Y>0 C.D.F. 0.4

0.2

0 -2.5 -2 -1.5 -1 -0.5 0 0.5 Y = g(X ,X ,X ) 1 2 3

Figure 3. Analytical and empirical PDF’s and CDF’s for X1, X2, and X3, and the Empirical PDF and CDF for Y = g(X1,X2,X3) CC BY-NC-ND March 25, 2021 PSH, HPG, JTS 18 CEE 201L. Uncertainty, Design, and Optimization – Duke University – Spring 2022 – P.S.H., H.P.G. and J.T.S.

1 nBins = floor (N /50); 2 figure (1) 3 c l f 4 subplot (231) 5 [fx ,xx] = hist ( X1, nBins, nBins/(max(X1)-min(X1 )) ); % histogram of X1 6 hold on 7 bar(xx,fx, ’FaceColor’,[1 1 1]) 8 plot (sort_X1,normpdf(sort_X1,mu1,sd1), ’LineWidth’,2); 9 hold off 10 axis (’tight ’) 11 ylabel (’P.D.F.’) 12 subplot (234) 13 hold on 14 s t a i r s ( sort_X1,CDF,’-b’,’LineWidth’,2) 15 plot (sort_X1,normcdf(sort_X1,mu1,sd1),’-r’ ) 16 hold off 17 axis (’tight ’) 18 ylabel (’C.D.F.’) 19 xlabel (’X_1 : normal’) 20 subplot (232) 21 [fx ,xx] = hist ( X2, nBins, nBins/(max(X2)-min(X2 )) ); % histogram of X2 22 hold on 23 bar(xx,fx, ’FaceColor’,[1 1 1]); 24 plot (sort_X2,logn_pdf(sort_X2,med2,cv2), ’LineWidth’,2); 25 hold off 26 axis (’tight ’) 27 subplot (235) 28 hold on 29 s t a i r s (sort_X2,CDF, ’-b’, ’LineWidth’,2) 30 plot (sort_X2,logn_cdf(sort_X2,[med2,cv2]),’-r’ ) 31 hold off 32 axis (’tight ’) 33 xlabel (’X_2 : log-normal’) 34 subplot (233) 35 [fx ,xx] = hist ( X3, nBins, nBins/(max(X3)-min(X3 )) ); % histogram of X2 36 hold on 37 bar(xx,fx, ’FaceColor’,[1 1 1]); 38 plot (sort_X3,Rayleigh_pdf(sort_X3,mod3), ’LineWidth’,2 ); 39 hold off 40 axis (’tight ’) 41 subplot (236) 42 hold on 43 s t a i r s (sort_X3,CDF, ’-b’, ’LineWidth’,2) 44 plot (sort_X3,Rayleigh_cdf(sort_X3,mod3),’-r’ ) 45 hold off 46 axis (’tight ’) 47 xlabel (’X_3 : Rayleigh’) 48 nBins = floor (N /20); 49 figure (2) 50 c l f 51 subplot (211) 52 [fx ,xx] = hist ( Y, nBins, nBins/(max(Y)-min(Y))); % histogram ofY 53 hold on 54 bar(xx,fx, ’FaceColor’,[1 1 1]); 55 plot ([0 0],[0 0.5],’-k’,’LineWidth’,3) 56 hold off 57 axis (’tight ’) 58 ylabel (’P.D.F.’) 59 subplot (212) 60 hold on 61 s t a i r s ( sort (Y),CDF) 62 plot ( [0 0],[0 1],’-k’,’LineWidth’,3); 63 hold off 64 axis (’tight ’) 65 text (0.5,0.5,’Y>0’) 66 ylabel (’C.D.F.’) 67 xlabel (’Y = g(X_1,X_2,X_3)’);

CC BY-NC-ND March 25, 2021 PSH, HPG, JTS