Properties of Least Squares Estimators Simple Linear Regression Model: Y = Β 0 + Β1x + Ε • Ε Is the Random Error So Y Is A

Home , Least squares

Properties of Least Squares Estimators

Simple Linear Regression

Model: Y = β0 + β1x +

is the random error so Y is a random variable too. •

Sample:

(x1,Y1), (x2,Y2),..., (xn,Yn)

Each (xi,Yi) satisﬁes Yi = β0 + β1xi + i

Least Squares Estimators:

n ˆ i=1(xi x)(Yi Y ) ˆ ˆ β1 = P n − −2 , β0 = Y β1x (xi x) − Pi=1 −

1 Properties of Least Squares Estimators

Assumptions on the Random Error

E[i] = 0 • 2 V (i) = σ •

Implications for the Response Variable Y

E[Yi] = β0 + β1xi • 2 V (Yi) = σ •

What can be said about:

ˆ E[β1]? • ˆ V (β1)? • ˆ E[β0]? • ˆ V (β0)? • ˆ ˆ Cov(β0, β1)? • 2 Properties of Least Squares Estimators

ˆ ˆ Proposition: The estimators β0 and β1 are unbiased; that is, ˆ ˆ E[β0] = β0,E[β1] = β1.

Proof: n ˆ i=1(xi x)(Yi Y ) β1 = P n − −2 (xi x) Pi=1 − n n i=1(xi x)Yi Y i=1(xi x) = P − n − P2 − (xi x) Pi=1 − n i=1(xi x)Yi = P n − 2 (xi x) Pi=1 −

3 Thus n ˆ (xi x) E[β1] = n − 2 E[Yi] X (xi x) i=1 Pi=1 −

n (xi x) = n − 2 (β0 + β1xi) X (xi x) i=1 Pi=1 −

n n i=1(xi x) i=1(xi x)xi = Pn − 2 β0 + β1P n − 2 (xi x) (xi x) Pi=1 − Pi=1 −

ˆ ˆ E[β0] = E[Y β1x] − 1 n = E[Y ] E[βˆ ]x n X i 1 i=1 − 1 n 1 n = (β + β x ) β x n X 0 1 i 1n X i i=1 − i=1

4 Properties of Least Squares Estimators

ˆ ˆ Proposition: The variances of β0 and β1 are: 2 n 2 2 n 2 ˆ σ i=1 xi σ i=1 xi V (β0) = n P 2 = P (xi x) Sxx Pi=1 − and 2 2 ˆ σ σ V (β1) = n 2 = . (xi x) Sxx Pi=1 − Proof: n ˆ i=1(xi x)Yi V (β1) = V P − Sxx

2 n 1 2 = (xi x) V (Yi) S X xx i=1 −

2 n 1 2 2 = (xi x) ! σ S X xx i=1 −

1 = σ2 Sxx

5 ˆ ˆ V (β0) = V (Y β1x) − ˆ ˆ = V (Y ) + V ( β1x) + 2Cov(Y, xβ1) − − 2 ˆ ˆ = V (Y ) + x V (β1) 2xCov(Y, β1) − 2 2 σ 2 σ ˆ = + x 2xCov(Y, β1) n Sxx −

Now let’s evaluate the covariance term: n n  1 xj x  Cov(Y, βˆ ) = Cov Y , − Y 1 X n i X S j  i=1 j=1 xx 

n n xj x = − Cov(Y ,Y ) X X nS i j i=1 j=1 xx

n xi x = − σ2 + 0 X nS i=1 xx

= 0

6 Thus

2 2 ˆ σ 2 σ V (β0) = + x n Sxx

S + nx2 = σ2 xx nSxx

n 2 2 (xi x) + nx = σ2 Pi=1 − nSxx

n 2 2 2 (x 2xxi + x ) + nx = σ2 Pi=1 i − nSxx n x2 = σ2 Pi=1 i nSxx

7 Properties of Least Squares Estimators

A Practical Matter The variance σ2 of the random error is usually not known. So it is necessary to esti- mate it.

Proposition: The estimator n 2 1 2 1 S = (Yi Yˆi) = SSE n 2 X − n 2 − i=1 − is an unbiased estimator of σ2.

8 Properties of Least Squares Estimators

Assumptions on the Random Error

E[i] = 0 • 2 V (i) = σ •

Each i is normally distributed. •

Implications for the Estimators

ˆ β1 is normally distributed with mean β1 and variance 2 • σ ; Sxx ˆ β0 is normally distributed with mean β0 and variance n 2 • 2 i=1 x σ P i ; nSxx

(n 2)S2 2 −σ2 has a χ distribution with n 2 degrees of free- • dom; −

2 ˆ ˆ S is independent of β0 and β1. •

9 Properties of Least Squares Estimators

Multiple Linear Regression

Model: Y = β0 + β1x1 + + βkxk + ··· Sample:

(x11, x12, . . . , x1k,Y1) (x21, x22, . . . , x2k,Y2) . (xn1, xn2, . . . , xnk,Yn)

Each (xi,Yi) satisﬁes Yi = β0+β1xi+ +βkxk+i ···

Least Squares Estimators:

1 βˆ = (X0X)− X0Y

10 Properties of Least Squares Estimators

ˆ ˆ Each βi is an unbiased estimator of βi: E[βi] = βi; •

ˆ 2 V (βi) = ciiσ , where cii is the element in the ith row • 1 and ith column of (X0X)− ;

ˆ ˆ 2 Cov(βi, βi) = cijσ ; •

The estimator • ˆ SSE Y0Y β0X0Y S2 = = − n (k + 1) n (k + 1) − − is an unbiased estimator of σ2.

11 Properties of Least Squares Estimators

When is normally distributed,

ˆ Each βi is normally distributed; •

The random variable • (n (k + 1))S2 − σ2 has a χ2 distribution with n (k +1) degrees of freee- dom; −

2 ˆ The statistics S and βi, i = 0, 1, . . . , k, are indepen- • dent.