Generalized Linear Models

Lecture 6.0: Generalized Linear Squares

1 Outline

1 Introduction

2 Generalized least squares (GLS)

3 Weighted least squares (WLS)

4 Iteratively reweighted least squares (IRWLS)

2 Introduction

Think about the assumptions we made in linear regression. How about errors are not constant?

Or not independent? Generalized or weighted least squares (GLS or WLS)

3 Outline

1 Introduction

2 Generalized least squares (GLS)

3 Weighted least squares (WLS)

4 Iteratively reweighted least squares (IRWLS)

4 GLS

Now we have the model Y = Xβ + E() = 0 Var() = σ2V GLS estimation can be obtained by matrix factorization V = KK. We deﬁne z = K−1y B = K−1X g = K−1 We have z = Bβ + g, E(g) = 0, Var(g) = σ2I.

5 GLS

z = Bβ + g

Now we can use OLS to estimate β:

0 S(β) = (z − Bβ) (z − Bβ) 0 = (K−1y − Bβ) (K−1y − Bβ) 0 = (y − Xβ) V−1(y − Xβ) Take the derivative with respect to β and set it to 0, we get 0 0 (X V−1X)β = X V−1y The GLS estimator of β is 0 0 βˆ = (X V−1X)−1X V−1y

6 GLS in R

nlme::gls(y ~ x, data = dat, weights = weights, ...)

7 Outline

1 Introduction

2 Generalized least squares (GLS)

3 Weighted least squares (WLS)

4 Iteratively reweighted least squares (IRWLS)

8 WLS

Some times the errors are uncorrelated, but have unequal variance. In this case we use weighted least squares (WLS). The covariance matrix of has the form:   1/w1 0    1/w2  2 2   σ V = σ  .   ..    0 1/wn −1 Let W = V with elements wi, the WLS estimator of βˆ is 0 0 βˆ = (X WX)−1X Wy

9 WLS

In WLS, observations with large variances get smaller weights than observations with smaller variances. Examples of possible weights are:

−1 Error proportional to a predictor xi suggests wi = xi . When an observation yi is an average of several, ni, observations at 2 that point of the explanatory variable, then, Var(yi) = σ /ni suggests wi = ni.

10 WLS in R

lm(y ~ x, data = dat, weights = weights, ...)

11 Outline

1 Introduction

2 Generalized least squares (GLS)

3 Weighted least squares (WLS)

4 Iteratively reweighted least squares (IRWLS)

12 IRWLS

Sometimes we will have prior information on the weights wi, others we might ﬁnd, looking at residual plots, that the variability is a function of one or more explanatory variables. In these cases we have to estimate the weights, perform the analysis, re-estimate the weights again based on these results and perform the analysis again. This procedure is called iteratively reweighted least squares (IRWLS).

13 IRWLS example

Suppose Var(i) = γ0 + γ1x1 :

1 Start with wi = 1. 2 Use OLS to estimate β. 2 3 Use residuals to estimate γ, perhaps regress ˆ on x1. 4 Recompute wi and repeat go to step 2.