Moment Generating Function T Definition Let X =(X1,...,Xn) be a random vector and T n t =(t1,...,tn) R .Themoment generating function (MGF) is STAT/MTHE 353: 5 – Moment Generating 2 defined by tT X Functions and Multivariate Normal Distribution MX (t)=E e for all t for which the expectation exists (i.e., finite). T. Linder Remarks: n tiXi Queen’s University MX (t)=E e i=1 P T For 0 =(0,..., 0) ,wehave MX (0)=1. Winter 2017 If X is a discrete random variable with finitely many values, then tT X n MX (t)=E e is always finite for all t R . 2 We will always assume that the distribution of X is such that M (t) is finite for all t ( t ,t )n for some t > 0. X 2 − 0 0 0 STAT/MTHE 353: 5 – MGF & Multivariate Normal Distribution 1/34 STAT/MTHE 353: 5 – MGF & Multivariate Normal Distribution 2/34 Connection with moments The single most important property of the MGF is that is uniquely determines the distribution of a random vector: Let k ,...,k be nonnegative integers and k = k + + k . Then 1 n 1 ··· n Theorem 1 k k @ @ t1X1+ +tnXn MX (t)= E e ··· k1 kn k1 kn Assume MX (t) and MY (t) are the MGFs of the random vectors X and @t @tn @t @tn 1 ··· 1 ··· Y and such that M (t)=M (t) for all t ( t ,t )n.Then k X Y 0 0 @ t X + +t X 2 − = E e 1 1 ··· n n k1 kn n @t1 @tn FX (z)=FY (z) for all z ✓ ··· ◆ R k1 k t X + +t X 2 = E X X n e 1 1 ··· n n 1 ··· n where FX and FY are the joint cdfs of X and Y . Setting t = 0 =(0,...,0)T , we get Remarks: k @ k1 kn MX (t) = E X X k1 kn t=0 1 n n @t @tn ··· FX (z)=FY (z) for all z R clearly implies MX (t)=MY (t). 1 2 ··· Thus M (t)=M (t) F (z)=F (z) X Y () X Y Most often we will use the theorem for random variables instead of For a (scalar) random variable X we obtain the kth moment of X: random vectors. In this case, MX (t)=MY (t) for all t ( t0,t0) k 2 − d k implies FX (z)=FY (z) for all z R. MX (t) = E X 2 dtk t=0 STAT/MTHE 353: 5 – MGF & Multivariate Normal Distribution 3/34 STAT/MTHE 353: 5 – MGF & Multivariate Normal Distribution 4/34 Theorem 2 n Assume X1,...,Xm are independent random vectors in R and let X = X + + X .Then 1 ··· m m MX (t)= MXi (t) i=1 Example: MGF for X Gamma(r, λ) and X + + X where the X Y ⇠ 1 ··· m i are independent and X Gamma(r , λ). i ⇠ i Proof: Example: MGF for X Poisson(λ) and X1 + + Xm where the Xi T T ⇠ ··· t X t (X1+ +Xm) are independent and X Gamma(λ ). Also, use the MGF to find MX (t)=E e = E e ··· i i 2 ⇠ T T E(X), E(X ),andVar(X). = Eet X1 et Xm T ··· T = Eet X1 E et Xm ··· = MX1 (t) MXm (t) ··· ⇤ Note: This theorem gives us a powerful tool for determining the distribution of the sum of independent random variables. STAT/MTHE 353: 5 – MGF & Multivariate Normal Distribution 5/34 STAT/MTHE 353: 5 – MGF & Multivariate Normal Distribution 6/34 Theorem 3 Applications to Normal Distribution Assume X is a random vector in Rn, A is an m n real matrix and ⇥ b Rm. Then the MGF of Y = AX + b is given at t Rm by Let X N(0, 1).Then 2 2 ⇠ T t b T tX 1 tx 1 x2/2 MY (t)=e MX (A t) MX (t)=E(e )= e e− dx p2⇡ Z1 1 1 1 (x2 2tx) 1 1 1 (x t)2 t2 Proof: = e− 2 − dx = e− 2 − − dx p2⇡ p2⇡ Z1 Z1 ⇥ ⇤ tT Y tT (AX+b) MY (t)=E e = E e t2/2 1 1 1 (x t)2 = e e− 2 − dx tT b tTAX tT b (AT t)T X p2⇡ = e E e = e E e Z1 T N(t,1) pdf = et bM (AT t) X ⇤ 2 = et /2 | {z } We obtain that for all t R Note: In the scalar case Y = aX + b we obtain 2 t2/2 MX (t)=e tb MY (t)=e MX (at) STAT/MTHE 353: 5 – MGF & Multivariate Normal Distribution 7/34 STAT/MTHE 353: 5 – MGF & Multivariate Normal Distribution 8/34 Moments of a standard normal random variable Sum of independent normal random variables z zk Recall the power series expansion e = 1 valid for all z R. Recall that if X N(0, 1) and Y = σX + µ,whereσ > 0,then k=0 k! 2 ⇠ Apply this to z = tX Y N(µ, σ2). Thus the MGF of a N(µ, σ2) random variable is P ⇠ 2 1 k tµ tµ (σt) /2 tX (tX) MY (t)=e MX (σt) = e e MX (t)=E(e )=E k! tµ+t2σ2/2 ✓ kX=0 ◆ = e 1 E (tX)k 1 E Xk = = tk k! k! k=0 ⇥ ⇤ k=0 2 X X Let X1,...,Xm be independent r.v.’s with Xi N(µi, σ ) and set ⇠ i X = X1 + + Xm. Then and to z = t2/2 ··· m m 2 i 2 2 m m 2 2 2 1 t /2 tµi+t σi /2 t i=1 µi + i=1 σi t /2 t /2 MX (t)= MX (t) = e = e MX (t)=e = i i! i=1 i=1 P P i=0 X Y Y Matching the coefficient of tk,fork =1, 2,... we obtain This implies m m 2 k! X N µi, σi ,keven, ⇠ k 2k/2(k/2)! ✓ i=1 i=1 ◆ E(X )=8 X X i.e., X + + X is normal with mean µ = m µ and <>0,kodd. 1 m X i=1 i 2··· m 2 variance σX = σi . > i=1 P STAT/MTHE 353: 5 – MGF & Multivariate Normal Distribution: 9/34 STAT/MTHE 353: 5 – MGF & Multivariate Normal Distribution 10 / 34 P Multivariate Normal Distributions For any nonnegative definite n n matrix C the following hold: ⇥ Linear Algebra Review (1) C has n nonnegative eigenvalues λ1,...,λn (counting multiplicities) and corresponding n orthogonal unit-length eigenvectors b ,...,b : Recall that an n n real matrix C is called nonnegative definite if it 1 n ⇥ is symmetric and Cbi = λibi,i=1,...,n T n x Cx 0 for all x R ≥ 2 T T where bi bi =1, i =1,...,n and bi bj =0if i = j. and positive definite if it is symmetric and 6 (2) (Spectral decomposition) C can be written as T n x Cx > 0 for all x R such that x = 0 2 6 T C = BDB Let A be an arbitrary n n real matrix. Then C = AT A is ⇥ where D = diag(λ1,...,λn) is the diagonal matrix of the nonnegative definite. If A is nonsingular (invertible), then C is eigenvalues of C,andB is the orthogonal matrix whose ith column positive definite. is bi,i.e.,B =[b1 ...bn]. T T T T T Proof: A A is symmetric since (A A)T =(A )(A )T = A A. (3) C is positive definite C is nonsingular all the eigenvalues () () Thus it is nonnegative definite since λi are positive xT (AT A)x = xT AT Ax =(Ax)T (Ax) = Ax 2 0 k k ≥ ⇤ STAT/MTHE 353: 5 – MGF & Multivariate Normal Distribution 11 / 34 STAT/MTHE 353: 5 – MGF & Multivariate Normal Distribution 12 / 34 (4) C has a unique nonnegative definite square root C1/2,i.e.,there exists a unique nonnegative definite A such that Lemma 4 C = AA If ⌃ is the covariance matrix of some random vector T X =(X1,...,Xn) , then it is nonnegative definite. Proof: We only prove the existence of A. Let n 1/2 1/2 1/2 1/2 1/2 Proof: We know that ⌃ =Cov(X) is symmetric. Let b R be D = diag(λ ,...,λn ) and note that D D = D. Let 2 1 arbitrary. Then A = BD1/2BT . Then A is nonnegative definite and 2 1/2 T 1/2 T bT ⌃b = bT Cov(X)b =Cov(bT X) = Var(bT X) 0 A = AA =(BD B )(BD B ) ≥ 1/2 T 1/2 T 1/2 1/2 T = BD B BD B = BD D B so ⌃ is nonnegative definite = C ⇤ Remark: It can be shown that an n n matrix ⌃ is nonnegative definite ⇥ T if and only if there exists a random vector X =(X1,...,Xn) such that Remarks: Cov(X)=⌃. If C is positive definite, then so is A. If we don’t require that A be nonnegative definite, then in general there are infinitely many solutions A for AAT = C. STAT/MTHE 353: 5 – MGF & Multivariate Normal Distribution 13 / 34 STAT/MTHE 353: 5 – MGF & Multivariate Normal Distribution 14 / 34 Definition Let µ Rn and let ⌃ be an n n nonnegative definite Defining the Multivariate Normal Distribution 2 ⇥ matrix. A random vector X =(X1,...,Xn) is said to have a Let Z1,...,Zn be independent r.v.’s with Zi N(0, 1). The multivariate normal distribution with parameters µ and ⌃ if its T ⇠ multivariate MGF of Z =(Z1,...,Zn) is multivariate MGF is n T 1 T T n t µ+ t ⌃t t Z tiZi tiZi M (t)=e 2 MZ (t)=E e = E e i=1 = E e X P i=1 n Y t2/2 n t2/2 1 tTt = e i = e i=1 i = e 2 i=1 P Notation: X N(µ, ⌃).