Lecture 15: Order Statistics Range of (A, B)

Home , Gumbel distribution, Order statistic, Range (statistics)

Section 4.6 Order Statistics Order Statistics

Let X1, X2, X3, X4, X5 be iid random variables with a distribution F with a Lecture 15: Order Statistics range of (a, b). We can relabel these X’s such that their labels correspond to arranging them in increasing order so that

Statistics 104 X X X X X (1) ≤ (2) ≤ (3) ≤ (4) ≤ (5) Colin Rundel X(1) X(2) X(3) X(4) X(5)

March 14, 2012 a X5 X1 X4 X2 X3 b

In the case where the distribution F is continuous we can make the stronger statement that

X(1) < X(2) < X(3) < X(4) < X(5)

Since P(Xi = Xj ) = 0 for all i = j for continuous random variables. 6 Statistics 104 (Colin Rundel) Lecture 15 March 14, 2012 1 / 24

Section 4.6 Order Statistics Section 4.6 Order Statistics Order Statistics, cont. Notation Detour

For X1, X2,..., Xn iid random variables Xk is the kth smallest X , usually For a continuous random variable we can see that called the kth order statistic. f (x) ≈ P(x ≤ X ≤ x + ) = P(X ∈ [x, x + ]) X(1) is therefore the smallest X and lim f (x) = lim P(X ∈ [x, x + ]) →0 →0 f (x) = lim P(X ∈ [x, x + ])/ X(1) = min(X1,..., Xn) →0

Similarly, X(n) is the largest X and

P (x X x + ✏)=P (X [x, x + ✏]) X(n) = max(X1,..., Xn)   2

f(x) f(x + ✏)

x x + ✏

Statistics 104 (Colin Rundel) Lecture 15 March 14, 2012 2 / 24 Statistics 104 (Colin Rundel) Lecture 15 March 14, 2012 3 / 24 Section 4.6 Order Statistics Section 4.6 Order Statistics Density of the maximum Density of the minimum

For X1, X2,..., Xn iid continuous random variables with pdf f and cdf F For X1, X2,..., Xn iid continuous random variables with pdf f and cdf F the density of the maximum is the density of the minimum is

P(X(n) ∈ [x, x + ]) = P(one of the X ’s ∈ [x, x + ] and all others < x) P(X(1) ∈ [x, x + ]) = P(one of the X ’s ∈ [x, x + ] and all others > x) n n X X = P(Xi ∈ [x, x + ] and all others < x) = P(Xi ∈ [x, x + ] and all others > x) i=1 i=1

= nP(X1 ∈ [x, x + ] and all others < x) = nP(X1 ∈ [x, x + ] and all others > x)

= nP(X1 ∈ [x, x + ])P(all others < x) = nP(X1 ∈ [x, x + ])P(all others > x)

= nP(X1 ∈ [x, x + ])P(X2 < x) ··· P(Xn < x) = nP(X1 ∈ [x, x + ])P(X2 > x) ··· P(Xn > x) = nf (x)F (x)n−1 = nf (x)(1 − F (x))n−1

n−1 n−1 f(n)(x) = nf (x)F (x) f(1)(x) = nf (x)(1 − F (x))

Statistics 104 (Colin Rundel) Lecture 15 March 14, 2012 4 / 24 Statistics 104 (Colin Rundel) Lecture 15 March 14, 2012 5 / 24

Section 4.6 Order Statistics Section 4.6 Order Statistics Density of the kth Order Statistic Cumulative Distribution of the min and max

For X1, X2,..., Xn iid continuous random variables with pdf f and cdf F For X1, X2,..., Xn iid continuous random variables with pdf f and cdf F the density of the kth order statistic is the density of the kth order statistic is

F(1)(x) = P(X(1) < x) = 1 − P(X(1) > x) P(X(k) ∈ [x, x + ]) = P(one of the X ’s ∈ [x, x + ] and exactly k − 1 of the others < x) n = 1 − P(X1 > x,..., Xn > x) = 1 − P(X1 > x) ··· P(Xn > x) X n = P(Xi ∈ [x, x + ] and exactly k − 1 of the others < x) = 1 − (1 − F (x)) i=1

= nP(X1 ∈ [x, x + ] and exactly k − 1 of the others < x) F(n)(x) = P(X(n) < x) = 1 − P(X(n) > x) = nP(X1 ∈ [x, x + ])P(exactly k − 1 of the others < x) = P(X < x,..., X < x) = P(X < x) ··· P(X < x) ! ! ! 1 n 1 n n − 1 n − 1 n = nP(X ∈ [x, x + ]) P(X < x)k−1P(X > x)n−k = nf (x) F (x)k−=1(1F (−x)F (x))n−k 1 k − 1 k − 1

! n − 1 d dF (x) f (x) = nf (x) F (x)k−1(1 − F (x))n−k f (x) = (1 − F (x))n = n(1 − F (x))n−1 = nf (x)(1 − F (x))n−1 (k) k − 1 (1) dx dx d dF (x) f (x) = F (x)n = nF (x)n−1 = nf (x)F (x)n−1 (n) dx dx Statistics 104 (Colin Rundel) Lecture 15 March 14, 2012 6 / 24 Statistics 104 (Colin Rundel) Lecture 15 March 14, 2012 7 / 24 Section 4.6 Order Statistics Section 4.6 Order Statistics Order Statistic of Standard Uniforms Beta Distribution

iid The Beta distribution is a continuous distribution deﬁned on the range Let X1, X2,..., Xn Unif(0, 1) then the density of X(n) is given by ∼ (0, 1) where the density is given by ! n − 1 1 r−1 s−1 f (x) = nf (x) F (x)k−1(1 − F (x))n−k f (x) = x (1 − x) (k) k − 1 B(r, s) ( nn−1x k−1(1 − x)n−k if 0 < x < 1 where B(r, s) is called the Beta function and it is a normalizing constant = k−1 0 otherwise which ensures the density integrates to 1.

Z 1 This is an example of the Beta distribution where r = k and s = n k + 1. 1 = f (x)dx − 0 Z 1 1 1 = x r−1(1 − x)s−1dx B(r, s) X(k) Beta(k, n k + 1) 0 ∼ − 1 Z 1 1 = x r−1(1 − x)s−1dx B(r, s) 0 Z 1 B(r, s) = x r−1(1 − x)s−1dx 0

Statistics 104 (Colin Rundel) Lecture 15 March 14, 2012 8 / 24 Statistics 104 (Colin Rundel) Lecture 15 March 14, 2012 9 / 24

Section 4.6 Order Statistics Section 4.6 Order Statistics Beta Function Beta Function - Expectation

The connection between the Beta distribution and the kth order statistic Let X Beta(r, s) then of n standard Uniform random variables allows us to simplify the Beta ∼ function. Z 1 1 E(X ) = x x r−1(1 − x)s−1dx 0 B(r, s) Z 1 1 Z r−1 s−1 = , 1x (r+1)−1(1 − x)s−1dx B(r, s) = x (1 − x) dx B(r, s) 0 0 1 B(r + 1, s) B(k, n − k + 1) = = n−1 B(r, s) n k−1 (k − 1)!(n − 1 − k + 1)! r!(s − 1)! (r + s − 1)! = = n(n − 1)! (r + s)! (r − 1)!(s − 1)! (r − 1)!(n − k)! r! (r + s − 1)! = = n! (r − 1)! (r + s)! r (r − 1)!(s − 1)! Γ(r)Γ(s) = = = r + s (r + s − 1)! Γ(r + s)

Statistics 104 (Colin Rundel) Lecture 15 March 14, 2012 10 / 24 Statistics 104 (Colin Rundel) Lecture 15 March 14, 2012 11 / 24 Section 4.6 Order Statistics Section 4.6 Order Statistics Beta Function - Variance Beta Distribution - Summary

Let X Beta(r, s) then If X Beta(r, s) then ∼ ∼ Z 1 1 1 E(X 2) = x2 xr−1(1 − x)s−1dx f (x) = x r−1(1 − x)s−1 0 B(r, s) B(r, s) B(r + 2, s) (r + 1)!(s − 1)! (r + s − 1)! Z x 1 B (r, s) = = F (x) = x r−1(1 − x)s−1dx = x B(r, s) (r + s + 1)! (r − 1)!(s − 1)! 0 B(r, s) B(r, s) (r + 1)r = (r + s + 1)(r + s) Z 1 (r − 1)!(s − 1)! Γ(r)Γ(s) B(r, s) = x r−1(1 − x)s−1dx = = (r + s − 1)! Γ(r + s) 2 2 0 Var(X ) = E(X ) − E(X ) Z x r−1 s−1 (r + 1)r r 2 Bx (r, s) = x (1 − x) dx = − 0 (r + s + 1)(r + s) (r + s)2 (r + 1)r(r + s) − r 2(r + s + 1) = r 2 E(X ) = (r + s + 1)(r + s) r + s rs = rs 2 Var(X ) = (r + s + 1)(r + s) (r + s)2(r + s + 1)

Statistics 104 (Colin Rundel) Lecture 15 March 14, 2012 12 / 24 Statistics 104 (Colin Rundel) Lecture 15 March 14, 2012 13 / 24

Section 4.6 Order Statistics Section 4.6 Order Statistics Minimum of Exponentials Maximum of Exponentials

iid iid Let X1, X2,..., Xn Exp(λ), we previously derived a more general result Let X1, X2,..., Xn Exp(λ) then the density of X(n) is given by where the X ’s were∼ not identically distributed and showed that ∼ min(X1,..., Xn) Exp(λ1 + + λn) = Exp(nλ) in this more restricted n−1 ∼ ··· f(n)(x) = nf (x)F (x) case. n−1 = n λe−λx 1 − e−λx Lets conﬁrm that result using our new more general methods

n−1 f(1)(x) = nf (x)(1 − F (x)) n−1 Which we can’t do much with, instead we can try the cdf of the maximum. = n λe−λx 1 − [1 − e−λx ] n−1 = nλe−λx e−λx ] n = nλ e−λx ]

= nλe−nλx

Which is the density for Exp(nλ). Statistics 104 (Colin Rundel) Lecture 15 March 14, 2012 14 / 24 Statistics 104 (Colin Rundel) Lecture 15 March 14, 2012 15 / 24 Section 4.6 Order Statistics Section 4.6 Order Statistics Maximum of Exponentials, cont. Limit Distributions of Maxima and Minima

iid Let X1, X2,..., Xn Exp(λ) then the cdf of X is given by Previous we have shown that ∼ (n)

n F(n)(x) = F (x) n n F(1)(x) = P(X(1) < x) = 1 (1 F (x)) = 1 − e−λx − − F (x) = P(X < x) = F (x)n n (n) (n) ne−λx = 1 − n −λx F(n)(x) ≈ exp(−ne ) When n tends to inﬁnity we get

−λx lim F(n)(x) = lim exp(−ne ) = 0 n→∞ n→∞ ( n 0 if F (x) = 0 lim F(1)(x) = lim 1 (1 F (x)) = n→∞ n→∞ − − 1 if F (x) > 0 This result is not unique to the exponential distribution... ( n 1 if F (x) = 1 lim F(n)(x) = lim F (x) = n→∞ n→∞ 0 if F (x) < 1

Statistics 104 (Colin Rundel) Lecture 15 March 14, 2012 16 / 24 Statistics 104 (Colin Rundel) Lecture 15 March 14, 2012 17 / 24

Section 4.6 Order Statistics Section 4.6 Order Statistics Limit Distributions of Maxima and Minima, cont. Maximum of Exponentials, cont.

iid These results show that the limit distributions are degenerate as they only Let X1, X2,..., Xn Exp(λ) and an = log(n)/λ, bn = 1/λ then the cdf of ∼ take values of 0 or 1. To avoid the degeneracy we would like to use a X(n) is given by simple transform that such that the limit distributions are not degenerate. n F(n)(an + bnx) = F ((log(n) + x)/λ) Let’s consider simple linear transformations n = 1 − e−λ(log(n)+x)/λ n − log(n) −x n 0 = 1 − e e lim F(n)(an + bnx) = lim F (an + bnx) = F (x) n→∞ n→∞ −x n n 00 = 1 − e /n lim F(1)(cn + dnx) = lim 1 − (1 − F (cn + dnx)) = F (x) n→∞ n→∞

−x lim F(n)(an + bnx) = exp(−e ) X(n) − an n→∞ F(n)(an + bnx) = P(X(n) < an + bnx) = P < x bn X(1) − cn This is known as the standard Gumbel distribution. F(1)(cn + dnx) = P(X(1) < cn + dnx) = P < x dn

Statistics 104 (Colin Rundel) Lecture 15 March 14, 2012 18 / 24 Statistics 104 (Colin Rundel) Lecture 15 March 14, 2012 19 / 24 Section 4.6 Order Statistics Section 4.6 Order Statistics Gumbel Distribution Maximum of Exponentials, cont.

iid Let X Gumbel(0, 1) then Let X1, X2,..., Xn Exp(λ) and an = log(n)/λ, bn = 1/λ then if n is ∼ ∼ large we can use the Standard Gumbel to calculate properties of X(n). Standard Gumbel Distribution - pdf Standard Gumbel Distribution - cdf 1.0 Median(X(n)) = m(n) 0.3 0.8 P(X(n) < m(n)) = 1/2 0.6 0.2

f(x) F(x) X(n) − an

0.4 P < mG = 1/2 bn 0.1

0.2 P(X(n) < an + bnmG ) = 1/2 0.0 0.0

-5 0 5 10 -5 0 5 10 x x m(n) = an + bnmG −x 1 1 F (x) = exp( e ) E(X ) = π/√6 = log n − log log 2 − λ λ f (x) = e−x exp( e−x ) Median(X ) = log(log(2)) − −

Statistics 104 (Colin Rundel) Lecture 15 March 14, 2012 20 / 24 Statistics 104 (Colin Rundel) Lecture 15 March 14, 2012 21 / 24

Section 4.6 Order Statistics Section 4.6 Order Statistics Maximum of Exponentials, Example Maximum of Uniforms

iid In 2009 Usain Bolt broke the world record in the 100 meters with a time of Let X1, X2,..., Xn Unif(0, 1) and an = 1, bn = 1/n then the cdf of X(n) 9.58 seconds in Berlin, Germany. If we imagine that the running speed in is given by ∼ m/s of competitive sprinters is given by an Exponential distribution with = 1. How many sprinters would need to run to have a 50/50 chance of n λ F(n)(x) = F (x) beating Usian Bolt’s record (having a faster running speed)? = x n Let X Exp(1) then we need to ﬁnd n such that n ∼ F(n)(an + bnx) = F (an + bnx) Median(X(n)) 100/9.58 ≥ = (1 + x/n)n x lim F(n)(an + bnx) = e n→∞

This is example of the Reverse Weibul distribution.

Statistics 104 (Colin Rundel) Lecture 15 March 14, 2012 22 / 24 Statistics 104 (Colin Rundel) Lecture 15 March 14, 2012 23 / 24 Section 4.6 Order Statistics Maximum of Paretos

This is a distribution we have not seen yet, but is useful for describing many physical processes. It’s key feature is that it has long tails meaning it goes to 0 slower than a distribution like the normal.

iid 1/α Let X1, X2,..., Xn ∼ Pareto(α, k) and an = 0, bn = kn then the cdf of X is ( k α 1 − x if x ≥ k, FX (x) = 0 otherwise

The cdf of X(n) is then given by

k αn F (x) = F (x)n = 1 − (n) x k αn x −α n F (a + b x) = F (a + b x)n = 1 − = 1 − (n) n n n n kxn1/α n −xα lim F(n)(an + bnx) = e n→∞

This is example of the Fr´echetdistribution. Statistics 104 (Colin Rundel) Lecture 15 March 14, 2012 24 / 24