' $

Generalized Linear Models

Objectives:

• Systematic + Random.

• Exponential family.

• Maximum likelihood estimation & inference.

& % 45 Heagerty, Bio/Stat 571 ' $ Generalized Linear Models

• Models for independent observations

Yi, i = 1, 2, . . . , n.

• Components of a GLM: . Random component

Yi ∼ f(Yi, θi, φ) f ∈ exponential family

& % 46 Heagerty, Bio/Stat 571 ' $

. Systematic component

ηi = Xiβ

ηi : linear predictor

Xi : (1 × p) covariate vector β :(p × 1) regression coefficient

. Link function

E(Yi | Xi) = µi

g(µi) = Xiβ g(·): link function

& % 47 Heagerty, Bio/Stat 571 ' $ Generalized Linear Models

• GLMs generalize the standard linear model:

Yi = Xiβ + ²i

. Random: Normal distribution

2 ²i ∼ N (0, σ )

. Systematic: linear combination of covariates

ηi = Xiβ

. Link: identity function

ηi = µi

& % 48 Heagerty, Bio/Stat 571 ' $ Generalized Linear Models

• GLMs extend usefully to overdispersed and correlated data:

. GEE: marginal models / semi-parametric estimation & inference

. GLMM: conditional models / likelihood estimation & inference

& % 49 Heagerty, Bio/Stat 571 ' $ Exponential Family

· ¸ yθ − b(θ) (?) f(y; θ, φ) = exp + c(y, φ) a(φ)

θ = canonical parameter φ = fixed (known) scale parameter

Properties: If Y ∼ f(y; θ, φ) in (?) then,

E(Y ) = µ = b0(θ) var(Y ) = b00(θ) · a(φ)

& % 50 Heagerty, Bio/Stat 571 ' $ Canonical link function: A function g(·) such that:

η = g(µ) = θ (canonical parameter)

Variance function: A function V (·) such that:

var(Y ) = V (µ) · a(φ)

Usually : a(φ) = φ · w φ “scale” parameter w weight

& % 51 Heagerty, Bio/Stat 571 ' $ Examples of GLMS: logistic regression

y = s/m where s=number of successes / m trials

µ ¶ m f(y; θ, φ) = πs(1 − π)m−s s · µ ¶¸ y · log( π ) + log(1 − π) m = exp 1−π + log 1/m s =⇒ θ = log(π/(1 − π)) b(θ) = − log(1 − π) = log[1 + exp(θ)] ∂ µ = b0(θ) = log[1 + exp(θ)] ∂θ = exp(θ)/[1 + exp(θ)] = π

& % 52 Heagerty, Bio/Stat 571 ' $

g(µ) = log[π/(1 − π)] = θ g : logit, log-odds function

1 var(y) = π(1 − π) · m

V (µ) =

a(φ) =

& % 53 Heagerty, Bio/Stat 571 ' $ • Poisson regression

y = number of events (count)

f(y; θ, φ) = λy exp(−λ)/y! = exp [y · log(λ) − λ − log(y!)] =⇒ θ = log(λ) b(θ) = λ = exp(θ) µ = b0(θ) = exp(θ) = λ

g(µ) = θ = log(µ) g : canonical link is log

& % 54 Heagerty, Bio/Stat 571 ' $ • Poisson regression (continued)

var(y) = λ

V (µ) =

a(φ) =

◦ Other examples: . gamma, inverse Gaussian (MN, Table 2.1) . some survival models (MN, Chpt. 13)

& % 55 Heagerty, Bio/Stat 571 ' $ Example: Seizure data (DHLZ ex. 1.6)

• Clinical trial of progabide, evaluating impact on epileptic seizures. • Data: . age = patient age in years . base = 8-week baseline seizure count (pre-tx) . tx = 0 if assigned placebo; 1 if assigned progabide

. Y1,Y2,Y3,Y4 seizure counts in 4 two-week periods following treatment administration • Models:

. linear model: Y4 = age + base + tx + ²

. Poisson GLM: log(µ4) = age + base + tx

& % 56 Heagerty, Bio/Stat 571 ' $ Example: Seizure data (DHLZ ex. 1.6)

linear regression Poisson regression est. s.e. Z est. s.e. Z (Int) -4.97 3.62 -1.37 0.778 0.285 2.73 age 0.12 0.11 1.07 0.014 0.009 1.64 base 0.31 0.03 11.79 0.022 0.001 20.27 tx -1.36 1.37 -0.99 -0.270 0.102 -2.66 • Q: should we use log(base) for Poisson regression? • Q: why does inference regarding significance of TX differ?

& % 57 Heagerty, Bio/Stat 571 • • 4 60 • •

50 • 3 • • • • • • • 40 • • • • • 2 • • • • •

y4 • •• • • • • 30 • ••••• • • • • • • ••• •• • 1

• • log(y4 + 0.5) •• • 20 • • • • • • • • 0 10 • • • • • • ••• • • • • • • • ••••••••••• •• • • 0 ••••• • ••• • ••• • ••

20 40 60 80 100 120 140 20 40 60 80 100 120 140

base base 14 15 12 10 10 8 6 5 4 2 0 0

0 20 40 60 0 20 40 60

y4[trt == 0] y4[trt == 1]

58 Heagerty, Bio/Stat 571 Seizure Residuals vs. Fitted

• • 20 • • 2.0 1.5

10 •• •• •• • • • • • • 1.0 •• •• • •• • • • • • •• • •• • •• •• • • • • •• • • • • • • • •• • • • • •• 0.5 • lmfit$resid •• •• • • glmfit$resid 0 •• • • • • • ••• • •• • • • • • • • • •• • • •••• • •• •••• • • •• • • • ••• •• •• • •• • • • • • • ••• • •• • • •• •

0.0 • • • •• •• • • • •••• • • •• •• • •• • • •• • ••• • •• ••••• •• •• • −10 • •• −0.5 •••• •• •• •• • •••••

• −1.0 ••••• •• 0 10 20 30 40 10 20 30 40 50 60

lmfit$fitted glmfit$fitted

59 Heagerty, Bio/Stat 571 Seizure Residuals vs. Fitted, using predict()

• 4 • 20 • • ••

•• •• •

2 • 10 •• •• •• •• •• • •• • • • • • • • • • • •• • • • • • •• • • • • • • • • •• •• • •• •• • •• • • •• • • • • •• • • ••• •• •• • ••

resid(lmfit) • • • • 0 • • resid(glmfit) 0 •• • • • • • • •• • •• • • • ••• •• • • • • • •• • •• •• • • • •• • •• •••• • ••• • • • ••• ••• • •••• • • ••• • • • • • • • •• • • ••• • •• • •• • • •• •• ••• •• • •• •• • • • −2 • −10 • •• •• •• • • •• •••• •• •• ••

0 10 20 30 40 1.0 1.5 2.0 2.5 3.0 3.5 4.0

predict(lmfit) predict(glmfit)

60 Heagerty, Bio/Stat 571 ' $ Residual Diagnostics

◦ Used to assess model fit similarly as for linear models • Q-Q plots for residuals (may be hard to interpret for discrete data ) • residual plots: ? vs. fitted values ? vs. omitted covariates • assessment of systematic departures • assessment of variance function

& % 61 Heagerty, Bio/Stat 571 ' $ Residual Diagnostics

◦ Types of residuals for GLMs: 1. Pearson residual y − µb rP = pi i V (µbi) X P 2 2 (ri ) = X

2. Deviance residual (see resid(fit)) p D r = sign(y − µb) di X i D 2 (ri ) = D(y, µb)

3. Working residual (see fit$resid)

W ∂ηbi ri = (yi − µbi) = Zi − ηbi ∂µbi & % 62 Heagerty, Bio/Stat 571 ' $ Fitting GLMS by Maximum Likelihood

Solve score equations: ∂ Uj(β) = log L = 0 j = 1, 2, . . . , p ∂βj log-likelihood: · ¸ Xn y · θ − b(θ ) log L = i i i + c(y , φ) a (φ) i i=1 i X = log Li =⇒ ∂ log L X ∂ log L ∂θ ∂µ ∂η U (β) = = i · i · i · i j ∂β ∂θ ∂µ ∂η ∂β j i i i i j

& % 63 Heagerty, Bio/Stat 571 ' $

∂ log Li 1 0 1 = (yi − b (θi)) = (yi − µi) ∂θi ai(φ) ai(φ) µ ¶−1 ∂θi ∂µi = = 1/V (µi) ∂µi ∂θi

∂ηi = Xij ∂βj

Therefore, µ ¶ Xn ∂µ U (β) = X i · [a (φ) · V (µ )]−1 (Y − µ ) j ij ∂η i i i i i=1 i

& % 64 Heagerty, Bio/Stat 571 ' $ GLM Information Matrix

• Either form:

[In](j, k) = cov[Uj(β),Uk(β)]

µ ¶ ∂2 log L = −E ∂βj∂βk

• Let’s consider the second form...

& % 65 Heagerty, Bio/Stat 571 ' $ GLM Information Matrix

· ¸ ∂ [In](j, k) = −E Uj(β) ∂βk " ½µ ¶ ¾# Xn ∂ ∂µ = −E i · [a (φ) · V (µ )]−1 (Y − µ ) β ∂β i i i i i=1 k j µ ¶ µ ¶ Xn ∂µ ∂µ = i · [a (φ) · V (µ )]−1 i ∂β i i ∂β i=1 j k

justify

& % 66 Heagerty, Bio/Stat 571 ' $ Score and Information

• In vector/matrix form we have:   U1(β)      U2(β)  U(β) =    .   . 

Up(β) ³ ´ ∂µi = ∂µi ∂µi ... ∂µi ∂β ∂β1 ∂β2 ∂βp

∂µi = Xi ∂ηi

& % 67 Heagerty, Bio/Stat 571 ' $ Score and Information

µ ¶ Xn ∂µ T U(β) = i · [a (φ) · V (µ )]−1 (Y − µ ) ∂β i i i i i=1 and µ ¶ µ ¶ Xn ∂µ T ∂µ I = i · [a (φ) · V (µ )]−1 i n ∂β i i ∂β i=1

& % 68 Heagerty, Bio/Stat 571 ' $ Fisher Scoring

Goal: Solve the score equations

U(β) = 0

Iterative estimation is required for most GLMs. The score equations can be solved using Newton-Raphson (uses observed derivative of score) or Fisher Scoring which uses the expected derivative of the

score (ie. −In).

& % 69 Heagerty, Bio/Stat 571 ' $ Fisher Scoring

Algorithm:

(0) • Pick an initial value: βb .

(j) • For j → (j + 1) update βb via

³ ´−1 b(j+1) b(j) b(j) b(j) β = β + In U(β )

(j+1) (j) • Evaluate convergence using changes in log L or kβb − βb k.

• Iterate until convergence criterion is satisfied.

& % 70 Heagerty, Bio/Stat 571 ' $

Comments on Fisher Scoring:

1. IWLS is equivalent to Fisher Scoring (Biostat 570). 2. Observed and expected information are equivalent for canonical links. 3. Score equations are an example of an estimating function (more on that to come!) 4. Q: What assumptions make E[U(β)] = 0?

P T 5. Q: What is the relationship between In and U iU i ? 6. Q: What is a 1-step approximation to ∆β(−i)?

& % 71 Heagerty, Bio/Stat 571 ' $ Inference for GLMs

Review of asymptotic likelihood theory:   β (q × 1)  1    β =  − − −  = − − −

β2 (p − q × 1)

0 Goal: Test H0 : β2 = β2

(1) Likelihood Ratio Test: h i b b b0 0 2 2 log L(β1, β2) − log L(β1, β2) ∼ χ (df = p − q)

& % 72 Heagerty, Bio/Stat 571 ' $ Inference for GLMs

(2) Score Test:   U 1(β ) (q × 1)  1    U(β) =  − − −  = − − −

U 2(β2) (p − q × 1)

n o−1 b0 T b0 b0 2 U 2(β ) cov[U 2(β )] U 2(β ) ∼ χ (df = p − q) (3) Wald Test:

n o−1 b 0 T b b 0 2 (β2 − β2) cov(β2) (β2 − β2) ∼ χ (df = p − q)

& % 73 Heagerty, Bio/Stat 571 ' $ Measures of Discrepancy

There are 2 primary measures: • deviance • Pearson’s X2

Deviance: Assume ai(φ) = φ/mi

(eg. normal: φ; binomial: 1/mi; Poisson: 1) Xn b b log L(β) = log fi(yi; θi, φ) i=1 ½ ¾ X m = i [y θb − b(θb )] + c (y , φ) φ i i i i i i

& % 74 Heagerty, Bio/Stat 571 ' $ Now consider log L as a function of µb, using the relationship b0(θ) = µ: ½ ¾ X m l(µb, φ; y) = i [y · θ(µb ) − b[θ(µb )] ] + c (y , φ) φ i i i i i i The deviance is:

D(y, µb) = 2 · φ · [l(y, φ; y) − l(µb, φ; y)]

X = 2 · mi { yi · [θ(yi) − θ(µbi)] − i

( b[θ(yi)] − b[θ(µbi)] ) }

& % 75 Heagerty, Bio/Stat 571 ' $ Deviance

Deviance generalizes the residual sum of squares for linear models:

Model 1 Model 2

    βb (q × 1) βb  1   1       − − −  − − −  − − −  b 0 β2 (p − q × 1) β2

µb1 µb2

& % 76 Heagerty, Bio/Stat 571 ' $ Deviance

Linear Model:

SSE(Model 2) − SSE(Model 1) ∼ χ2(df = p − q) σ2

GLM: D(y, µb ) − D(y, µb ) 2 1 ∼ χ2(df = p − q) φ

& % 77 Heagerty, Bio/Stat 571 ' $

Examples:

2 (yi−µi) 1. Normal: log f(yi; θi, φ) = − 2φ + C X 2 D(y, µb) = (yi − µbi) = SSE i

2. Poisson: log f(yi; θi, φ) = yi · log(µ) − µ + C " µ ¶ # X y D(y, µb) = 2 × y · log i − (y − µb ) i µb i i i i h ³ ´ i µ 3. Binomial: log f(yi; θi, φ) = mi yi · log 1−µ + log(1 − µ) " µ ¶ µ ¶# X y 1 − y D(y, µb) = 2 × y · log i − (1 − y ) · log i i µb i 1 − µb i i i & % 78 Heagerty, Bio/Stat 571 ' $ Pearson’s X2

φ Assume: var(Yi) = V (µi) mi

2 P 2 Define: X = i(yi − µbi) /[V (µbi)/mi]

Examples: 1. Normal: X2 = SSE

2 2 2. Poisson: X = (yi − µbi) /µbi (look familiar?) 2 2 3. Binomial: X = (yi − µbi) /[µbi(1 − µbi)] (??) If the model is correct (mean and variance) then, X2 ≈ φ (n − p)

& % 79 Heagerty, Bio/Stat 571 ' $ e.g. ◦ Normal: SSE/(n − p) ≈ σ2 = φ

◦ Poisson: X2/(n − p) ≈ 1 = φ

◦ Binomial: X2/(n − p) ≈ 1 = φ

& % 80 Heagerty, Bio/Stat 571 ' $

Example: Seizure data (DLZ ex. 1.6)

X2 136.64 = = 2.48 (n − p) 59 − 4

D(y, µb) 147.02 = = 2.67 (n − p) 59 − 4

Q: Poisson???

& % 81 Heagerty, Bio/Stat 571 ' $

Summary:

• GLMs applicable to range of univariate outcomes.

• Systematic variation (regression ) Random variation (variance function, likelihood)

• Score equations of simple form.

• Inference using: likelihood ratios (deviance) score statistics Wald statistics

• Model checking regression structure / variance form V (µ) & % 82 Heagerty, Bio/Stat 571 ' $

References:

McCullagh P., and Nelder J.N. Generalized Linear Models, Second Edition, Chapman and Hall, 1989.

Fahrmeir L., and Tutz G. Multivariate Statistical Modelling Based on Generalized Linear Models, Springer-Verlag, 1991. (see chapter 2)

McCulloch C.E., and Searle S.R. Generalized, Linear, and Mixed Models, Wiley, 2001. (see chapter 5)

Diggle P., Heagerty P.J., Liang K-Y., Zeger S.L. Longitudinal Data Analysis, Second Edition, Oxford, 2002. (see appendix A) & % 83 Heagerty, Bio/Stat 571