Exponential Families Exponential Families MIT 18.655 Dr. Kempthorne Spring 2016 1 MIT 18.655 Exponential Families One Parameter Exponential Family Exponential Families Multiparameter Exponential Family Building Exponential Families Outline 1 Exponential Families One Parameter Exponential Family Multiparameter Exponential Family Building Exponential Families 2 MIT 18.655 Exponential Families One Parameter Exponential Family Exponential Families Multiparameter Exponential Family Building Exponential Families Definition q Let X be a random variable/vector with sample space X ⊂ R and probability model Pθ: The class of probability models P = fPθ; θ 2 Θg is a one-parameter exponential family if the density/pmf function p(x j θ) can be written: p(x j θ) = h(x)expfη(θ)T (x) − B(θ)g where h : X! R η :Θ ! R B :Θ ! R: Note: By the Factorization Theorem, T (X ) is sufficient for θ p(x j θ) = h(x)g(T (x); θ) Set g(T (x); θ) = expfη(θ)T (x) − B(θ)g T (X ) is the Natural Sufficient Statistic: 3 MIT 18.655 Exponential Families One Parameter Exponential Family Exponential Families Multiparameter Exponential Family Building Exponential Families Examples Poisson Distribution (1.6.1) : X ∼ Poisson(θ); where E[X ] = θ: θx −θ p(x j θ) = x! e ; x = 0; 1;::: 1 = x! expf(log(θ)x − θg = h(x)expfη(θ)T (x) − B(θ) where: 1 h(x) = x! η(θ) = log(θ) T (x) = x 4 MIT 18.655 Exponential Families One Parameter Exponential Family Exponential Families Multiparameter Exponential Family Building Exponential Families Examples Binomial Distribution (1.6.2) : X ∼ Binomial(θ; n) n r p(x j θ) = θ1(1 − θ)n−x x = 0; 1;:::; n x r n θ = expflog()x + nlog(1 − θ)g x 1−θ = h(x)expfη(θ)T (x) − B(θ) where: r n h(x) = x θ η(θ) = log( 1−θ ) T (x) = x B(θ) = −nlog(1 − θ) 5 MIT 18.655 Exponential Families One Parameter Exponential Family Exponential Families Multiparameter Exponential Family Building Exponential Families Examples 2 Normal Distribution : X ∼ N(µ, σ0): (Known variance) 1 2 − (x−µ) 1 2σ2 p 0 p(x j θ) = 2 e 2πσ0 1 x2 µ µ2 = [p exp{− g] × expf x − g 2 2σ2 σ2 2σ2 2πσ0 0 0 0 = h(x)expfη(θ)T (x) − B(θ) where: 1 x2 h(x) = [−p exp{− g] 2 2σ2 2πσ0 0 µ η(θ) = 2 σ0 T (x) = x µ2 B(θ) = 2 2σ0 6 MIT 18.655 Exponential Families One Parameter Exponential Family Exponential Families Multiparameter Exponential Family Building Exponential Families Examples 2 Normal Distribution : X ∼ N(µ0; σ ): (Known mean) 1 2 1 − (x−µ0) p(x j θ) = p e 2σ2 2πσ2 1 1 2 1 2 = [p ] × exp{− 2 (x − µ0) ) − log(σ )g 2πσ2 2σ 2 = h(x)expfη(θ)T (x) − B(θ) where: h(x) = [p1 ] 2π 1 η(θ) = − 2σ2 2 T (x) = (x − µ0) 1 2 B(θ) = 2 log(σ ) 7 MIT 18.655 Exponential Families One Parameter Exponential Family Exponential Families Multiparameter Exponential Family Building Exponential Families Samples from One-Parameter Exponential Family Distribution Consider a sample: X1;:::; Xn; where Xi are iid P where P 2 P= fPθ; θ 2 Θg is a one-parameter exponential family distribution with density function p(x j θ) = h(x)expfη(θ)T (x) − B(θ)g The sample X = (X1;:::; Xn) is a random vector with density/pmf: Qn p(x j θ) = i=1 (h(xi )exp[η(θ)T (xi ) − B(θ)]) Qn Pn = [ i=1 h(xi )] × exp[η(θ) i=1 T (xi ) − nB(θ)] ∗ ∗ ∗ ∗ = h (x)expfη (θ)T (x) − B (θ)g where: ∗ Q n h (x) = i=1 h(xi ) ∗ η (θ) = η(θ) ∗ Pn T (x) = i=1 T (xi ) ∗ B (θ) = nB(θ) ∗ Note: The Sufficient Statsitic T is one-dimensional for all n. 8 MIT 18.655 Exponential Families One Parameter Exponential Family Exponential Families Multiparameter Exponential Family Building Exponential Families Samples from One-Parameter Exponential Family Distribution Theorem 1.6.1 Let fPθg be a one-parameter exponential family of discrete distributions with pmf function: p(x j θ) = h(x)expfη(θ)T (x) − B(θ)g Then the family of distributions of the statistic T (X ) is a one-parameter exponential family of discrete distributions whose frequency functions are ∗∗ Pθ(T (x) = t) = p(t j θ) = h (t)expfη(θ)t − B(θ)g where ∗∗ X h (t) = h(x) fx:T (x)=tg Proof: Immediate 9 MIT 18.655 Exponential Families One Parameter Exponential Family Exponential Families Multiparameter Exponential Family Building Exponential Families Canonical Exponential Family Re-parametrize setting η = η(θ) { the Natural Parameter The density has the form p(x; η) = h(x)expfηT (x) − A(η)g The function A(η) replaces B(θ) and is defined as the normalization constant: log(A(η)) = RR ··· R h(x)expfηT (x)]dx if X continuous or X log(A(η)) = h(x)expfηT (x)] x2X if X discrete The Natural Parameter Space fη : η = η(θ); θ 2 Θg = E (Later, Theorem 1.6.3 gives properties of E) T (x) is the Natural Sufficient Statistic. 10 MIT 18.655 Exponential Families One Parameter Exponential Family Exponential Families Multiparameter Exponential Family Building Exponential Families Canonical Representation of Poisson Family Poisson Distribution (1.6.1) : X ∼ Poisson(θ); where E[X ] = θ: θx −θ p(x j θ) = x! e ; x = 0; 1;::: 1 = x! expf(log(θ)x − θg = h(x)expfη(θ)T (x) − B(θ)g where: 1 h(x) = x! η(θ) = log(θ) T (x) = x Canonical Representation η = log(θ): η A(η) = B(θ) = θ = e . 11 MIT 18.655 Exponential Families One Parameter Exponential Family Exponential Families Multiparameter Exponential Family Building Exponential Families MGFs of Canonical Exponenetial Family Models Theorem 1.6.2 Suppose X is distributued according to a canonical exponential family, i.e., the density/pmf function is given by q p(x j η) = h(x)exp[ηT (x) − A(η)]; for x 2 X ⊂ R : If η is an interior point of E, the natural parameter space, then The moment generating function of T (X ) exists and is given by sT (X ) MT (s) = E [e j η] = expfA(s + η) − A(η)g for s in some neighborhood of 0: 0 E [T (X ) j η] = A (η): 00 Var[T (X ) j η] = A (η): Proof: sT (X ) RR (s+η)T (x)−A(η) MT (s) = E [e ) j η] = ··· h(x)e dx [A(s+η)−A(s)] RR (s+η)T (x)−A(s+η) = [e ] × ··· h(x)e dx = [e[A(s+η)−A(s)]] × 1 Remainder follows from properties of MFGs. 12 MIT 18.655 Exponential Families One Parameter Exponential Family Exponential Families Multiparameter Exponential Family Building Exponential Families Moments of Canonical Exponential Family Distributions η Poisson Distribution: A(η) = B(θ) = θ = e . 0 η E (X j θ) = A (η) = e = θ: 00 η Var(X j θ) = A (η) = e = θ: Binomial Distribution: r n X n−x p(x j θ) = θ (1 − θ) x θ = h(x)expflog()(1−θ) x + nlog(1 − θ)g η = h(x)expfηx − nlog(e + 1)g η θ So A(η) = nlog(e + 1; ) with η = log()1−θ 0 eη A (η) = n eη +1 = nθ 00 1 η η −1 η A (η) = neη +1 e + ne × (eη +1)2 e eη eη = n[]eη +1 × (1 − (eη +1) ) = nθ(1 − θ) 13 MIT 18.655 Exponential Families One Parameter Exponential Family Exponential Families Multiparameter Exponential Family Building Exponential Families Moments of the Gamma Distribution X1;:::; Xn i.i.d Gamma(p; λ) distribution with density λp xp−1 e−λx p(x j λ, p) = Γ(p) , 0 < x < 1 where R 1 p p−1 −λx Γ(p) = 0 λ x e dx xp−1 p(x j λ, p) = [ Γ(p) ]exp{−λx + plog(λ)g = h(x)expfηT (x) − A(η)g where η = −λ A(η) = −plog(λ) = −plog(−η) Thus 0 E (X ) = A (η) = −p/η = p/λ 00 2 2 Var(X ) = A (η) = (p/η ) = p/λ 14 MIT 18.655 Exponential Families One Parameter Exponential Family Exponential Families Multiparameter Exponential Family Building Exponential Families Notes on Gamma Distribution Gamma(p = n=2; λ = 1=2) corresponds to the Chi-Squared distribution with n degrees of freedom. p = 2 corresponds to the Exponential Distribution p For p = 1; Γ(1=2) = π Γ(p + 1) = pΓ(p) for positive integer p. 15 MIT 18.655 Exponential Families One Parameter Exponential Family Exponential Families Multiparameter Exponential Family Building Exponential Families Rayleigh Distribution Sample X1 :::; Xn iid with density function x 2 2 p(x j θ) = θ2 exp(−x =2θ ) 1 2 2 = [x] × exp{− 2θ2 x − log(θ )g = h(x)expfηT (x) − A(η)g where 1 η = − 2θ2 2 T (X ) = X : 2 −1 A(η) = log(θ ) = log( 2η ) = −log(−2η) By the mgf 2 0 2 1 2 E (X ) = A (η) = − 2η= − η = 2θ 2 00 1 4 Var(X ) = A (η) = + η2 = 4θ For the n− sample: X = (X1;:::; Xn) Pn 2 T (X) = 1 X i 2 E [T (X)] = −n/η = 2nθ 1 4 Var[T (X)] = n η2 = 4nθ : x2 Note: P(X ≤ x) = 1 − exp{− 2θ2 g (Failure time model) 16 MIT 18.655 Exponential Families One Parameter Exponential Family Exponential Families Multiparameter Exponential Family Building Exponential Families Outline 1 Exponential Families One Parameter Exponential Family Multiparameter Exponential Family Building Exponential Families 17 MIT 18.655 Exponential Families One Parameter Exponential Family Exponential Families Multiparameter Exponential Family Building Exponential Families Definition k fPθ; θ 2 Θg,Θ ⊂ R , is a k-parameter exponential family if the the density/pmf function of X ∼ Pθ is k X p(x j θ) = h(x)exp[ ηj (θ)Tj (x) − B(θ)], j=1 q where x 2 X ⊂ R ; and η1; : : : ; ηk and B are real-valued functions mappying Θ ! R.