Properties of Least Squares Estimators Simple Linear Regression Model: Y = β0 + β1x + is the random error so Y is a random variable too. • Sample: (x1;Y1); (x2;Y2);:::; (xn;Yn) Each (xi;Yi) satisfies Yi = β0 + β1xi + i Least Squares Estimators: n ^ i=1(xi x)(Yi Y ) ^ ^ β1 = P n − −2 ; β0 = Y β1x (xi x) − Pi=1 − 1 Properties of Least Squares Estimators Assumptions on the Random Error E[i] = 0 • 2 V (i) = σ • Implications for the Response Variable Y E[Yi] = β0 + β1xi • 2 V (Yi) = σ • What can be said about: ^ E[β1]? • ^ V (β1)? • ^ E[β0]? • ^ V (β0)? • ^ ^ Cov(β0; β1)? • 2 Properties of Least Squares Estimators ^ ^ Proposition: The estimators β0 and β1 are unbiased; that is, ^ ^ E[β0] = β0;E[β1] = β1: Proof: n ^ i=1(xi x)(Yi Y ) β1 = P n − −2 (xi x) Pi=1 − n n i=1(xi x)Yi Y i=1(xi x) = P − n − P2 − (xi x) Pi=1 − n i=1(xi x)Yi = P n − 2 (xi x) Pi=1 − 3 Thus n ^ (xi x) E[β1] = n − 2 E[Yi] X (xi x) i=1 Pi=1 − n (xi x) = n − 2 (β0 + β1xi) X (xi x) i=1 Pi=1 − n n i=1(xi x) i=1(xi x)xi = Pn − 2 β0 + β1P n − 2 (xi x) (xi x) Pi=1 − Pi=1 − ^ ^ E[β0] = E[Y β1x] − 1 n = E[Y ] E[β^ ]x n X i 1 i=1 − 1 n 1 n = (β + β x ) β x n X 0 1 i 1n X i i=1 − i=1 4 Properties of Least Squares Estimators ^ ^ Proposition: The variances of β0 and β1 are: 2 n 2 2 n 2 ^ σ i=1 xi σ i=1 xi V (β0) = n P 2 = P (xi x) Sxx Pi=1 − and 2 2 ^ σ σ V (β1) = n 2 = : (xi x) Sxx Pi=1 − Proof: n ^ i=1(xi x)Yi V (β1) = V P − Sxx 2 n 1 2 = (xi x) V (Yi) S X xx i=1 − 2 n 1 2 2 = (xi x) ! σ S X xx i=1 − 1 = σ2 Sxx 5 ^ ^ V (β0) = V (Y β1x) − ^ ^ = V (Y ) + V ( β1x) + 2Cov(Y; xβ1) − − 2 ^ ^ = V (Y ) + x V (β1) 2xCov(Y; β1) − 2 2 σ 2 σ ^ = + x 2xCov(Y; β1) n Sxx − Now let's evaluate the covariance term: n n 0 1 xj x 1 Cov(Y; β^ ) = Cov Y ; − Y 1 X n i X S j @ i=1 j=1 xx A n n xj x = − Cov(Y ;Y ) X X nS i j i=1 j=1 xx n xi x = − σ2 + 0 X nS i=1 xx = 0 6 Thus 2 2 ^ σ 2 σ V (β0) = + x n Sxx S + nx2 = σ2 xx nSxx n 2 2 (xi x) + nx = σ2 Pi=1 − nSxx n 2 2 2 (x 2xxi + x ) + nx = σ2 Pi=1 i − nSxx n x2 = σ2 Pi=1 i nSxx 7 Properties of Least Squares Estimators A Practical Matter The variance σ2 of the random error is usually not known. So it is necessary to esti- mate it. Proposition: The estimator n 2 1 2 1 S = (Yi Y^i) = SSE n 2 X − n 2 − i=1 − is an unbiased estimator of σ2. 8 Properties of Least Squares Estimators Assumptions on the Random Error E[i] = 0 • 2 V (i) = σ • Each i is normally distributed. • Implications for the Estimators ^ β1 is normally distributed with mean β1 and variance 2 • σ ; Sxx ^ β0 is normally distributed with mean β0 and variance n 2 • 2 i=1 x σ P i ; nSxx (n 2)S2 2 −σ2 has a χ distribution with n 2 degrees of free- • dom; − 2 ^ ^ S is independent of β0 and β1. • 9 Properties of Least Squares Estimators Multiple Linear Regression Model: Y = β0 + β1x1 + + βkxk + ··· Sample: (x11; x12; : : : ; x1k;Y1) (x21; x22; : : : ; x2k;Y2) . (xn1; xn2; : : : ; xnk;Yn) Each (xi;Yi) satisfies Yi = β0+β1xi+ +βkxk+i ··· Least Squares Estimators: 1 β^ = (X0X)− X0Y 10 Properties of Least Squares Estimators ^ ^ Each βi is an unbiased estimator of βi: E[βi] = βi; • ^ 2 V (βi) = ciiσ , where cii is the element in the ith row • 1 and ith column of (X0X)− ; ^ ^ 2 Cov(βi; βi) = cijσ ; • The estimator • ^ SSE Y0Y β0X0Y S2 = = − n (k + 1) n (k + 1) − − is an unbiased estimator of σ2. 11 Properties of Least Squares Estimators When is normally distributed, ^ Each βi is normally distributed; • The random variable • (n (k + 1))S2 − σ2 has a χ2 distribution with n (k +1) degrees of freee- dom; − 2 ^ The statistics S and βi, i = 0; 1; : : : ; k, are indepen- • dent. 12.