SAMPLE MOMENTS
1. POPULATION MOMENTS 1.1. Moments about the origin (raw moments). The rth moment about the origin of a random 0 r variable X, denoted by µr, is the expected value of X ; symbolically,
0 r µr =E(X ) (1) = X xr f(x) (2) x for r = 0, 1, 2, . . . when X is discrete and
0 r µr =E(X ) v (3) when X is continuous. The rth moment about the origin is only defined if E[ Xr ] exists. A mo- 0 ment about the origin is sometimes called a raw moment. Note that µ1 = E(X)=µX , the mean of the distribution of X, or simply the mean of X. The rth moment is sometimes written as function of θ where θ is a vector of parameters that characterize the distribution of X.
If there is a sequence of random variables, X1,X2,...Xn, we will call the rth population moment 0 of the ith random variable µi, r and define it as
0 r µ i,r = E (X i) (4)
1.2. Central moments. The rth moment about the mean of a random variable X, denoted by µr,is r the expected value of ( X − µX ) symbolically,
r µr =E[(X − µX ) ] r (5) = X ( x − µX ) f(x) x for r = 0, 1, 2, . . . when X is discrete and
r µr =E[(X − µX ) ] ∞ r (6) =R−∞ (x − µX ) f(x) dx
r when X is continuous. The rth moment about the mean is only defined if E[ (X - µX) ] exists. The rth moment about the mean of a random variable X is sometimes called the rth central moment of r X. The rth central moment of X about a is defined as E[ (X - a) ]. If a = µX, we have the rth central moment of X about µX.
Date: August 9, 2004. 1 2 SAMPLE MOMENTS
2 Note that µ1 = E[(X - µX)] = 0 and µ2 = E[(X - µX) ] = Var[X]. Also note that all odd moments of X around its mean are zero for symmetrical distributions, provided such moments exist.
If there is a sequence of random variables, X1,X2,...Xn, we will call the rth central population moment of the ith random variable µ i,r and define it as
r 0 r µi, r = E X i − µ i, 1 (7) 0 When the variables are identically distributed, we will drop the i subscript and write µ r and µ r .
2. SAMPLE MOMENTS
2.1. Definitions. Assume there is a sequence of random variables, X1,X2,...Xn. The first sample moment, usually called the average is defined by
1 n X¯ = X X (8) n n i i =1
Corresponding to this statistic is its numerical value, x¯n, which is defined by
1 n x¯ = X x (9) n n i i =1
where xi represents the observed value of Xi. The rth sample moment for any t is defined by
1 n X¯ r = X Xr (10) n n i i =1 This too has a numerical counterpart given by
1 n x¯r = X xr (11) n n i i =1
2.2. Properties of Sample Moments.
¯ r 2.2.1. Expected value of Xn . Taking the expected value of equation 10 we obtain
1 n 1 n E X¯ r = EX¯ r = X EXr = X µ0 (12) n n n i n i,r i =1 i =1 If the X’s are identically distributed, then
1 n E X¯ r = EX¯ r = X µ0 = µ0 (13) n n n r r i =1
¯ r 2.2.2. Variance of Xn. n r 1 0 0 E X¯ r = EX¯ = X µ = µ (14) n n n r r i =1 SAMPLE MOMENTS 3
¯ r 2.2.3. Variance of Xn. First consider the case where we have a sample X1,X2,...Xn.
1 n ! 1 n ! Var X¯ r = Var X Xr = Var X Xr (15) n n i n2 i i =1 i =1 If the X’s are independent, then
1 n Var X¯ r = X Var(Xr ) (16) n n2 i i =1 If the X’s are independent and identically distributed, then
1 Var X¯ r = Var(Xr ) (17) n n where X denotes any one of the random variables (because they are all identical). In the case where r =1, we obtain
2 ¯ 1 σ Var Xn = Var( X )= (18) n n
3. SAMPLE CENTRAL MOMENTS
3.1. Definitions. Assume there is a sequence of random variables, X1,X2,...Xn. We define the sample central moments as
r 1 n − 0 r ··· Cn = n Pi =1 Xi µ i,1 ,r =1, 2 , 3 , , ⇒ 1 1 n − 0 Cn = n Pi =1 Xi µ i,1 (19) ⇒ 2 1 n − 0 2 Cn = n Pi =1 Xi µ i,1
0 These are only defined if µ i,1 is known.
3.2. Properties of Sample Moments.
r r 3.2.1. Expected value of Cn. The expected value of Cn is given by
r 1 n − 0 r 1 n E ( Cn )=n Pi =1 Xi µ i,1 = n Pi =1 µ i,r (20)
The last equality follows from equation 7.
If the Xi are identically distributed, then
r E ( Cn )=µ r 1 (21) E Cn =0 4 SAMPLE MOMENTS
r 3.2.2. Variance of C n. First consider the case where we have a sample X1,X2,...Xn.
n n r 1 0 r ! 1 0 r! Var( C )=Var X Xi − µ = Var X Xi − µ (22) n n i,1 n2 i,1 i =1 i =1 If the X’s are independently distributed, then
n r 1 0 r Var( C )= X Var Xi − µ (23) n n2 i,1 i =1 If the X’s are independent and identically distributed, then
1 r Var( Cr )= Var ( X − µ0 ) (24) n n 1 where X denotes any one of the random variables (because they are all identical). In the case where r =1, we obtain
r 1 − 0 Var( Cn )=n Var[ X µ 1 ] 1 − = n Var[ X µ ] 1 2 − (25) = n σ 2 Cov[ X, µ]+Var[ µ ] 1 2 = n σ
4. SAMPLE ABOUT THE AVERAGE
4.1. Definitions. Assume there is a sequence of random variables, X1,X2,. . . Xn. Define the rth sample moment about the average as
r 1 n − ¯ r ··· Mn = n Pi =1 Xi Xn ,r =1, 2 , 3 , , (26)
This is clearly a statistic of which we can compute a numerical value. We denote the numerical r value by, mn, and define it as
1 n mr = X ( x − x¯ )r (27) n n i n i =1 In the special case where r = 1 we have
1 1 n − ¯ Mn = n Pi =1 Xi Xn 1 n − ¯ = n Pi =1 Xi Xn (28) ¯ ¯ =Xn − Xn =0
4.2. Properties of Sample Moments about the Average when r = 2. SAMPLE MOMENTS 5
r r 4.2.1. Alternative ways to write Mn. We can write Mn in an alternative useful way by expanding the squared term and then simplifying as follows
r 1 n − ¯ r Mn = n Pi =1 Xi Xn ⇒ 2 1 n − ¯ 2 Mn = n Pi =1 Xi Xn 1 n 2 − ¯ ¯ 2 = n Pi =1 Xi 2 XiXn + Xn n ¯ n n (29) 1 2 − 2Xn 1 ¯ 2 = n Pi =1 Xi n Pi =1Xi + n Pi =1Xn 1 n 2 − ¯ 2 ¯ 2 = n Pi =1 Xi 2Xn + Xn 1 n 2 − ¯ 2 = n Pi =1 Xi Xn
r r 4.2.2. Expected value of Mn. The expected value of Mn is then given by 2 1 n 2 − ¯ 2 E Mn = n E Pi =1 Xi E Xn 1 n 2 − ¯ 2 − ¯ = n Pi =1 E Xi E X n Var(Xn ) (30) 1 n 0 − 1 n 0 2 − ¯ = n Pi =1 µ i,2 n Pi =1 µ i,1 Var(Xn )
The second line follows from the alternative definition of variance
2 Var( X )=E X 2 − [ E ( X )] 2 ⇒ E X 2 =[ E ( X )] + Var( X ) (31) ¯ 2 ¯ 2 ¯ ⇒ E Xn = E Xn + Var(Xn )
and the third line follows from equation 12. If the Xi are independent and identically distributed, then
2 1 n 2 − ¯ 2 E Mn = n E Pi =1 Xi E Xn 1 n 0 − 1 n 0 2 − ¯ = n Pi =1 µ i,2 n Pi =1 µ i,1 Var(Xn ) 0 − 0 2 − σ2 (32) =µ2 ( µ1 ) n 2 − 1 2 =σ n σ n − 1 2 = n σ 0 0 where µ1 and µ2 are the first and second population moments, and µ2 is the second central population moment for the identically distributed variables. Note that this obviously implies
h n ¯ 2 i 2 E Pi =1 Xi − X =nE Mn n − 1 2 (33) =n n σ =( n − 1) σ2
2 4.2.3. Variance of Mn. By definition,
2 2 2 2 2 Var Mn =E h Mn i − EMn (34)
The second term on the right on equation 34 is easily obtained by squaring the result in equation 32.
2 n − 1 2 E Mn = n σ 2 2 2 (35) ⇒ 2 2 (n − 1) 4 E Mn = EMn = n2 σ 6 SAMPLE MOMENTS
Now consider the first term on the right hand side of equation 34. Write it as
2 h 2 2i 1 n − ¯ 2 E Mn =E n Pi =1 Xi Xn (36)
1 n − ¯ 2 Now consider writing n Pi =1 Xi Xn as follows
1 n − ¯ 2 1 n − − ¯ − 2 n Pi =1 Xi X = n Pi =1 (Xi µ) (X µ) 2 = 1 n Y − Y¯ n Pi =1 i (37) where Yi =Xi − µ Y¯ =X¯ − µ
Obviously,
n ¯ 2 n ¯ 2 ¯ ¯ Pi =1 Xi − X =Pi =1 Yi − Y , where Yi = Xi − µ,Y = X − µ (38)
Now consider the properties of the random variable Yi which is a transformation of Xi. First the expected value.
Yi =Xi − µ E ( Y )=E (X ) − E ( µ) i i (39) =µ − µ =0
The variance of Yi is
Yi =Xi − µ
Var(Yi)=Var(Xi) (40) 2 =σ if Xi are independently and identically distributed 4 Also consider E(Yi ). We can write this as
4 ∞ 4 E( Y )=R−∞ y f ( x ) dx ∞ 4 =R−∞ ( x − µ) f(x) dx (41) =µ4
Now write equation 36 as follows
2 h 2 2i 1 n − ¯ 2 E Mn =E n Pi =1 Xi Xn 2 1 n − ¯ 2 =E n Pi =1 Xi X 2 (42) 1 n − ¯ 2 =E n Pi =1 Yi Y 2 1 n − ¯ 2 = n2 E Pi =1 Yi Y
1 Ignoring n2 for now, expand equation 42 as follows SAMPLE MOMENTS 7
2 2 n ! n ! ¯ 2 2 ¯ ¯ 2 E X Yi − Y =E X Y − 2 YiY + Y i i =1 i =1 2 n n n ! =E X Y 2 − 2 Y¯ X Y + X Y¯ 2 i i i =1 i =1 i =1 2 n ! =E X Y 2 − 2 n Y¯ 2 + n Y¯ 2 i i =1 2 n ! =E X Y 2 − 2nY¯ 2 (43) i i =1 2 n ! n =E X Y 2 − 2 n Y¯ 2 X Y 2 + n2Y¯ 4 i i i =1 i =1 2 n ! " n # =E X Y 2 − 2 nE Y¯ 2 X Y 2 + n2 E Y¯ 4 i i i =1 i =1
Now consider the first term on the right of 43 which we can write as
2 n ! n n E X Y 2 =E X Y 2 X Y 2 i i j i =1 i =1 j =1 n " 4 2 2# =E X Yi + XX Yi Yj i 6=j i =1 n 4 2 2 = X EYi + XX EYi EYj (44) i 6=j i =1 2 =nµ4 + n (n − 1)µ2 4 =nµ4 + n (n − 1)σ
Now consider the second term on the right of 43 (ignoring 2n for now) which we can write as 8 SAMPLE MOMENTS
" n # 1 n n n E Y¯ 2 X Y 2 = E X Y X Y X Y 2 (45) i n2 j k i i =1 j =1 k =1 i =1 1 n = E X Y 4 + XX Y 2 Y 2 + XX Y Y X Y 2 2 i i j j k i n i 6=j j 6=k i =1 i =6 j i =6 k 1 n = X EY4 + XX EY2 EY2 + XX EYEY X EY2 2 i i j j k i n i 6=j j 6=k i =1 i =6 j i =6 k
1 2 = nµ4 + n (n − 1) µ +0 n2 2 1 4 = µ4 +(n − 1)σ n
The last term on the penultimate line is zero because E(Yj) = E(Yk) = E(Yi)=0.
Now consider the third term on the right side of 43 (ignoring n2 for now) which we can write as
n n n n ¯ 4 1 E Y = E X Yi X Yj X Yk X Y` (46) n4 i =1 j =1 k =1 ` =1
1 n = E X Y 4 + XX Y 2 Y 2 + XX Y 2 Y 2 + X Y 2 Y 2 + ··· n2 i i 6=k i k i 6=j i j i j i =1 i 6=j
where for the first double sum (i = j =6 k=`), for the second (i = k =6 j=`), and for the last (i = ` =6 j = k) and ... indicates that all other terms include Yi in a non-squared form, the expected value of which will be zero. Given that the Yi are independently and identically distributed, the expected value of each of the double sums is the same, which gives
¯ 4 1 h n 4 2 2 2 2 2 2 ···i E Y = n4 E Pi =1 Yi + PPi 6=k Yi Yk + PPi 6=j Yi Yj + Pi 6=j Yi Yj + 1 h n 4 2 2 i = n4 Pi =1 EYi +3PPi 6=j Yi Yj + terms containing E Xi 1 h n 4 2 2i = n4 Pi =1 EYi +3PPi 6=j Yi Yj (47) 1 − 2 = n4 nµ4 +3n (n 1) ( µ2 ) 1 − 4 = n4 nµ4 +3n (n 1) σ 1 − 4 = n3 µ4 +3(n 1) σ
Now combining the information in equations 45, 46, and 47 we obtain SAMPLE MOMENTS 9
2 2 n ! n ! ¯ 2 2 ¯ ¯ 2 E X Yi − Y =E X Y − 2 YiY + Y (48) i i =1 i =1 2 n ! " n # =E X Y 2 − 2 nE Y¯ 2 X Y 2 + n2 E Y¯ 4 i i i =1 i =1
2 1 2 2 1 2 =nµ4 + n(n − 1)µ − 2n µ4 +(n − 1)µ + n µ4 +3(n − 1)µ 2 n 2 n3 2
2 2 1 2 =nµ4 + n (n − 1)µ − 2 µ4 +(n − 1) µ + µ4 +3(n − 1) µ 2 2 n 2 n2 2 n 1 n2(n − 1) 2 n (n − 1) 3(n − 1) = µ − µ + µ + µ2 − µ2 + µ2 n 4 n 4 n 4 n 2 n 2 n 2 n2 − 2 n +1 (n − 1) (n2 − 2 n +3) = µ + µ2 n 4 n 2 n2 − 2 n +1 (n − 1) (n2 − 2 n +3) = µ + σ4 n 4 n
1 Now rewrite equation 42 including n2 as follows
2 n ! 2 2 1 ¯ 2 E h M i = E X Yi − Y n n2 i =1 1 n2 − 2 n +1 (n − 1) (n2 − 2 n +3) = µ + σ n2 n 4 n 4 n2 − 2 n +1 (n − 1) (n2 − 2 n +3) = µ + σ (49) n3 4 n3 4 ( n − 1)2 (n − 1) (n2 − 2 n +3) = µ + σ n3 4 n3 4
Now substitute equations 35 and 49 into equation 34 to obtain
2 2 2 2 2 Var Mn =E h Mn i − EMn 2 2 2 (50) (n−1) (n − 1) (n −2 n+3 ) 4 − ( n − 1) 4 = n3 µ4 + n3 σ n2 σ
We can simplify this as 10 SAMPLE MOMENTS
2 2 2 2 2 Var Mn =E h Mn i − EMn (51) ( n − 1)2 (n − 1) (n2 − 2 n +3) n ( n − 1)2 = µ + σ4 − σ4 n3 4 n3 n3 2 4 2 µ4 ( n − 1) + ( n − 1)σ n − 2 n +3− n ( n − 1) = n3 2 4 2 2 µ4 ( n − 1) + ( n − 1)σ n − 2 n +3− n + n = n3 2 4 µ4 ( n − 1) + ( n − 1)σ (3 − n ) = n3 2 4 µ4 ( n − 1) − ( n − 1)σ (n − 3) = n3 ( n − 1) 2 µ ( n − 1) (n − 3)σ4 = 4 − n3 n3
5. SAMPLE VARIANCE 5.1. Definition of sample variance. The sample variance is defined as
2 1 n − ¯ 2 Sn = n − 1 Pi =1 Xi Xn (52)
We can write this in terms of moments about the mean as
2 1 n ¯ 2 Sn = P Xi − Xn n − 1 i =1 (53) n 2 2 1 n − ¯ 2 = n − 1 Mn where Mn = n Pi =1 Xi Xn
5.2. Expected value of S2. We can compute the expected value of S2 by substituting in from equa- tion 32 as follows
2 n 2 E Sn = n − 1 E Mn n n − 1 2 = n − 1 n σ (54) =σ2
5.3. Variance of S2. We can compute the variance of S2 by substituting in from equation 51 as follows
§2 n2 2 Var n = ( n − 1)2 Var Mn 2 2 4 n ( n − 1) µ4 − ( n − 1) (n−3) σ = ( n − 1)2 n3 n3 (55) 4 µ4 − (n−3) σ = n n ( n − 1)
5.4. Definition of σˆ2. One possible estimate of the population variance is σˆ2 which is given by
2 1 n − ¯ 2 σˆ = n Pi =1 Xi Xn 2 (56) =Mn SAMPLE MOMENTS 11
5.5. Expected value of σˆ2. We can compute the expected value of σˆ2 by substituting in from equa- tion 32 as follows
2 2 E σˆ =E Mn (57) n − 1 = σ2 n 5.6. Variance of σˆ2. We can compute the variance of σˆ2 by substituting in from equation 51 as follows
2 2 Var σˆ =Var Mn 2 4 ( n − 1) µ4 − ( n − 1) (n−3) σ = n3 n3 (58) µ − µ2 2(µ − 2 µ2 ) µ − 3 µ2 4 2 − 4 2 4 2 = n n2 + n3
We can also write this in an alternative fashion
2 2 Var σˆ =Var Mn 2 4 ( n − 1) µ4 − ( n − 1) (n−3)σ = n3 n3 ( n − 1) 2 µ ( n − 1) (n−3)µ2 4 − 2 = n3 n3 2 − n2 µ2 − 4 nµ2+3 µ2 (59) n µ4 2 nµ4+µ4 − 2 2 2 = n3 n3 2 2 2 2 n ( µ4−µ2 )−2 n ( µ4 −2 µ2 )+µ4 −3 µ2 = n3 µ − µ2 2(µ − 2 µ2 ) µ − 3 µ2 4 2 − 4 2 4 2 = n n2 + n3
6. NORMAL POPULATIONS 6.1. Central moments of the normal distribution. For a normal population we can obtain the cen- tral moments by differentiating the moment generating function. The moment generating function for the central moments is as follows
t2 σ2 MX (t)=e 2 . (60) The moments are then as follows. The first central moment is
2 2 d t σ − 2 | E (X µ )=dt e t =0 2 2 2 t σ =tσ e 2 |t =0 (61) =0
The second central moment is
2 2 2 d2 t σ − 2 | E (X µ ) = dt2 e t =0 2 2 d 2 t σ 2 | = dt tσ e t =0 2 2 2 2 (62) 2 4 t σ 2 t σ = t σ e 2 + σ e 2 |t =0 =σ2
The third central moment is 12 SAMPLE MOMENTS
2 2 3 d3 t σ − 2 | E (X µ ) = dt3 e t =0 2 2 2 2 d 2 4 t σ 2 t σ 2 2 | = dt t σ e + σ e t =0 2 2 2 2 2 2 3 6 t σ 4 t σ 4 t σ = t σ e 2 +2tσ e 2 + tσ e 2 |t =0 (63) 2 2 2 2 3 6 t σ 4 t σ = t σ e 2 +3tσ e 2 |t =0 =0
The fourth central moment is
2 2 4 d4 t σ − 2 | E (X µ ) = dt4 e t =0 2 2 2 2 d 3 6 t σ 4 t σ 2 2 | = dt t σ e +3tσ e t =0 2 2 2 2 2 2 2 2 4 8 t σ 2 6 t σ 2 6 t σ 4 t σ = t σ e 2 +3t σ e 2 +3t σ e 2 +3σ e 2 |t =0 (64) 2 2 2 2 2 2 4 8 t σ 2 6 t σ 4 t σ = t σ e 2 +6t σ e 2 +3σ e 2 |t =0 =3 σ4
2 6.2. Variance of S . Let X1,X2,...Xn be a random sample from a normal population with mean µ and variance σ2 < ∞.
We know from equation 55 that
2 n2 2 Var Sn = ( n − 1)2 Var Mn 2 2 4 n ( n − 1) µ4 − ( n − 1) (n−3) σ = ( n − 1)2 n3 n3 (65) − 4 µ4 − (n 3)σ = n n ( n − 1)
If we substitute in for µ4 from equation 64 we obtain
− 4 2 µ4 − (n 3)σ Var Sn = n n (n − 1) 4 3 σ4 − (n−3)σ = n n ( n − 1) (3(n − 1)−(n−3))σ4 = n ( n − 1) (3n−3−n+3 )) σ4 (66) = n ( n − 1) 2 nσ4 = n ( n − 1) 2 σ4 = ( n − 1)
6.3. Variance of σˆ2. It is easy to show that
2 σ4 ( n − 1) Var(ˆσ2 )= n2