CHAPTER 1 DISTRIBUTION THEORY 1 CHAPTER 1: DISTRIBUTION THEORY CHAPTER 1 DISTRIBUTION THEORY 2 Basic Concepts CHAPTER 1 DISTRIBUTION THEORY 3 • Random Variables (R.V.) { discrete random variable: probability mass function { continuous random variable: probability density function CHAPTER 1 DISTRIBUTION THEORY 4 • \Formal but Scary" Definition of Random Variables { R.V. are some measurable functions from a probability measure space to real space; { probability is some non-negative value assigned to sets of a σ-field; { probability mass ≡ the Radon-Nykodym derivative of the random variable-induced measure w.r.t. to a counting measure; { probability density function ≡ the derivative w.r.t. Lebesgue measure. CHAPTER 1 DISTRIBUTION THEORY 5 • Descriptive quantities of univariate distribution { cumulative distribution function: P (X ≤ x) { moments (mean): E[Xk] { quantiles { mode { centralized moments (variance): E[(X − µ)k] { the skewness: E[(X − µ)3]=V ar(X)3=2 { the kurtosis: E[(X − µ)4]=V ar(X)2 CHAPTER 1 DISTRIBUTION THEORY 6 • Characteristic function (c.f.) { ϕX (t) = E[expfitXg] { c.f. uniquely determines the distribution Z 1 T e−ita − e−itb FX (b) − FX (a) = lim ϕX (t)dt T !1 2π −T it Z 1 1 −itx fX (x) = e ϕ(t)dt: 2π −∞ CHAPTER 1 DISTRIBUTION THEORY 7 • Descriptive quantities of multivariate R.V.s { Cov(X; Y ); corr(X; Y ) { fXjY (xjy) = fX;Y (x; y)=fY (y) R { E[XjY ] = xfXjY (xjy)dx { Independence: P (X ≤ x; Y ≤ y) = P (X ≤ x)P (Y ≤ y), fX;Y (x; y) = fX (x)fY (y) CHAPTER 1 DISTRIBUTION THEORY 8 • Equalities of double expectations E[X] = E[E[XjY ]] V ar(X) = E[V ar(XjY )] + V ar(E[XjY ]) • Useful when the distribution of X given Y is simple! CHAPTER 1 DISTRIBUTION THEORY 9 Examples of Discrete Distributions CHAPTER 1 DISTRIBUTION THEORY 10 • Binomial distribution { Binomial(n; p():) n k − n−k P (Sn = k) = k p (1 p) ; k = 0; :::; n { E[Sn] = np; V ar(Sn) = np(1 − p) it n ϕX (t) = (1 − p + pe ) { Binomial(n1; p) + Binomial(n2; p) ∼ Binomial(n1 + n2; p) CHAPTER 1 DISTRIBUTION THEORY 11 • Negative Binomial distribution ( ) k−1 m − k−m { P (Wm = k) = m−1 p (1 p) k = m; m + 1; ::: 2 { E[Wm] = m=p; V ar(Wm) = m=p − m=p { Neg-Binomial(m1; p) + Neg-Binomial(m2; p) ∼ Neg-Binomial(m1 + m2; p) CHAPTER 1 DISTRIBUTION THEORY 12 • Hypergeometric distribution ( )( ) ( ) M N−M N { P (Sn = k) = k n−k = n ; k = 0; 1; ::; n { E[Sn] = Mn=(M + N) nMN(M+N−n) { V ar(Sn) = (M+N)2(M+N−1) CHAPTER 1 DISTRIBUTION THEORY 13 • Poisson distribution { P (X = k) = λke−λ=k!; k = 0; 1; 2; ::: { E[X] = V ar(X) = λ, it ϕX (t) = exp{−λ(1 − e )g { P oisson(λ1) + P oisson(λ2) ∼ P oisson(λ1 + λ2) CHAPTER 1 DISTRIBUTION THEORY 14 • Poisson vs Binomial { X1 ∼ P oisson(λ1);X2 ∼ P oisson(λ2) λ1 { X1jX1 + X2 = n ∼ Binomial(n; ) λ1+λ2 { if Xn1; :::; Xnn are i.i.d Bernoulli(pn) and npn ! λ, then n! λk P (S = k) = pk (1 − p )n−k ! e−λ: n k!(n − k)! n n k! CHAPTER 1 DISTRIBUTION THEORY 15 • Multinomial Distribution { Nl; 1 ≤ l ≤ k counts the number of times that fY1; :::; Yng fall into Bl ( ) n n1 nk { P (N1 = n1; :::; Nk = nk) = p ··· p n1;:::;nk 1 k n1 + ::: + nk = n { the covariance matrix for (N1; :::; Nk) 0 1 p (1 − p ) ::: −p p B 1 1 1 k C B . C n @ . .. A −p1pk : : : pk(1 − pk) CHAPTER 1 DISTRIBUTION THEORY 16 Examples of Continuous Distribution CHAPTER 1 DISTRIBUTION THEORY 17 • Uniform distribution { Uniform(a; b): fX (x) = I[a;b](x)=(b − a) { E[X] = (a + b)=2 and V ar(X) = (b − a)2=12 CHAPTER 1 DISTRIBUTION THEORY 18 • Normal distribution 2 { N(µ, σ2): f (x) = p 1 exp{− (x−µ) g X 2πσ2 2σ2 { E[X] = µ and V ar(X) = σ2 2 2 { ϕX (t) = expfitµ − σ t =2g CHAPTER 1 DISTRIBUTION THEORY 19 • Gamma distribution 1 θ−1 {− x g { Gamma(θ; β): fX (x) = βθΓ(θ) x exp β ; x > 0 { E[X] = θβ and V ar(X) = θβ2 { θ = 1 equivalent to Exp(β) 2 { θ = n=2; β = 2 equivalent to χn CHAPTER 1 DISTRIBUTION THEORY 20 • Cauchy distribution 1 { Cauchy(a; b): fX (x) = bπf1+(x−a)2=b2g { E[jXj] = 1, ϕX (t) = expfiat − jbtjg { often used as counter example CHAPTER 1 DISTRIBUTION THEORY 21 Algebra of Random Variables CHAPTER 1 DISTRIBUTION THEORY 22 • Assumption { X and Y are independent and Y > 0 { we are interested in d.f. of X + Y; XY; X=Y CHAPTER 1 DISTRIBUTION THEORY 23 • Summation of X and Y { Derivation: F (z) = E[I(X + Y ≤ z)] = E [E [I(X ≤ z − Y )jY ]] X+Y Z Y X = EY [FX (z − Y )] = FX (z − y)dFY (y) { Convolution formula: Z FX+Y (z) = FY (z − x)dFX (x) ≡ FX ∗ FY (z) Z Z fX ∗fY (z) ≡ fX (z−y)fY (y)dy = fY (z−x)fX (x)dx CHAPTER 1 DISTRIBUTION THEORY 24 • Product and quotient of X and Y (Y > 0) R { FXY (z) = E[E[I(XY ≤ z)jY ]] = FX (z=y)dFY (y) Z fXY (z) = fX (z=y)=yfY (y)dy R { FX=Y (z) = E[E[I(X=Y ≤ z)jY ]] = FX (yz)dFY (y) Z fX=Y (z) = fX (yz)yfY (y)dy CHAPTER 1 DISTRIBUTION THEORY 25 • Application of formulae 2 2 ∼ 2 2 { N(µ1; σ1) + N(µ2; σ2) N(µ1 + µ2; σ1 + σ2) { Gamma(r1; θ) + Gamma(r2; θ) ∼ Gamma(r1 + r2; θ) { P oisson(λ1) + P oisson(λ2) ∼ P oisson(λ1 + λ2) { Negative Binomial(m1; p) + Negative Binomial(m2; p) ∼ Negative Binomial(m1 + m2; p) CHAPTER 1 DISTRIBUTION THEORY 26 • Summation of R.V.s using c.f. { Result: if X and Y are independent, then ϕX+Y (t) = ϕX (t)ϕY (t) { Example: X and Y are normal with f − 2 2 g ϕX (t) = exp iµ1t σ1t =2 ; f − 2 2 g ϕY (t) = exp iµ2t σ2t =2 ) f − 2 2 2 g ϕX+Y (t) = exp i(µ1 + µ2)t (σ1 + σ2)t =2 CHAPTER 1 DISTRIBUTION THEORY 27 • Further examples of special distribution ∼ ∼ 2 ∼ 2 { assume X N(0; 1), Y χm and Z χn are independent; { X q ∼ Student's t(m); Y=m Y=m ∼ Snedecor's F ; Z=n m;n Y ∼ Beta(m=2; n=2): Y + Z CHAPTER 1 DISTRIBUTION THEORY 28 • Densities of t−, F − and Beta− distributions Γ((m+1)=2) 1 p −∞ 1 { ft(m)(x) = πmΓ(m=2) (1+x2=m)(m+1)=2 I( ; )(x) Γ(m+n)=2 (m=n)m=2xm=2−1 1 { fFm;n (x) = Γ(m=2)Γ(n=2) (1+mx=n)(m+n)=2 I(0; )(x) Γ(a+b) a−1 − b−1 2 { fBeta(a;b)(x) = Γ(a)Γ(b) x (1 x) I(x (0; 1)) CHAPTER 1 DISTRIBUTION THEORY 29 • Exponential vs Beta- distributions { assume Y1; :::; Yn+1 are i.i.d Exp(θ); Y1+:::+Yi { Zi = ∼ Beta(i; n − i + 1); Y1+:::+Yn+1 { (Z1;:::;Zn) has the same joint distribution as that of the order statistics (ξn:1; :::; ξn:n) of n Uniform(0,1) r.v.s. CHAPTER 1 DISTRIBUTION THEORY 30 Transformation of Random Vector CHAPTER 1 DISTRIBUTION THEORY 31 Theorem 1.3 Suppose that X is k-dimension random vector with density function fX (x1; :::; xk). Let g be a one-to-one and continuously differentiable map from Rk to Rk. Then Y = g(X) is a random vector with density function −1 fX (g (y1; :::; yk))jJg−1 (y1; :::; yk)j; −1 where g is the inverse of g and Jg−1 is the Jacobian of g−1. CHAPTER 1 DISTRIBUTION THEORY 32 • Example { let R2 ∼ Expf2g; R > 0 and Θ ∼ Uniform(0; 2π) be independent; { X = R cos Θ and Y = R sin Θ are two independent standard normal random variables; { it can be applied to simulate normally distributed data. CHAPTER 1 DISTRIBUTION THEORY 33 Multivariate Normal Distribution CHAPTER 1 DISTRIBUTION THEORY 34 • Definition 0 Y = (Y1; :::; Yn) is said to have a multivariate normal 0 distribution with mean vector µ = (µ1; :::; µn) and non-degenerate covariance matrix Σn×n if 1 1 f (y ; :::; y ) = exp{− (y − µ)0Σ−1(y − µ)g Y 1 n (2π)n=2jΣj1=2 2 CHAPTER 1 DISTRIBUTION THEORY 35 • Characteristic function ϕY (t) Z it0Y − n − 1 0 1 0 −1 = E[e ] = (2π) 2 jΣj 2 expfit y − (y − µ) Σ (y − µ)gdy 2 Z p 0 −1 0 −1 − n − 1 y Σ y −1 0 µ Σ µ = ( 2π) 2 jΣj 2 exp{− + (it + Σ µ) y − gdy 2 2 Z { exp{−µ0Σ−1µ/2g 1 = p exp − (y − Σit − µ)0Σ−1(y − Σit − µ) ( 2π)n=2jΣj1=2 2 } +(Σit + µ)0Σ−1(Σit + µ)=2 dy { } 1 = exp −µ0Σ−1µ/2 + (Σit + µ)0Σ−1(Σit + µ) 2 1 = expfit0µ − t0Σtg: 2 CHAPTER 1 DISTRIBUTION THEORY 36 • Conclusions { multivariate normal distribution is uniquely determined by µ and Σ { for standard multivariate normal distribution, 0 ϕX (t) = exp{−t t=2g { the moment generating function for Y is expft0µ + t0Σt=2g CHAPTER 1 DISTRIBUTION THEORY 37 • Linear transformation of normal r.v. Theorem 1.4 If Y = An×kXk×1 where X ∼ N(0;I) (standard multivariate normal distribution), then Y 's characteristic function is given by 0 k ϕY (t) = exp {−t Σt=2g ; t = (t1; :::; tn) 2 R and rank(Σ) = rank(A).