Statistics 203: Introduction to Regression and Analysis of Variance Generalized Linear Models I Jonathan Taylor - p. 1/18 Today's class ● Today's class ■ Logistic regression. ● Generalized linear models ● Binary regression example ■ ● Binary outcomes Generalized linear models. ● Logit transform ■ ● Binary regression Deviance. ● Link functions: binary regression ● Link function inverses: binary regression ● Odds ratios & logistic regression ● Link & variance fns. of a GLM ● Binary (again) ● Fitting a binary regression GLM: IRLS ● Other common examples of GLMs ● Deviance ● Binary deviance ● Partial deviance tests 2 ● Wald χ tests - p. 2/18 Generalized linear models ● Today's class ■ All models we have seen so far deal with continuous ● Generalized linear models ● Binary regression example outcome variables with no restriction on their expectations, ● Binary outcomes ● Logit transform and (most) have assumed that mean and variance are ● Binary regression ● Link functions: binary unrelated (i.e. variance is constant). regression ● Link function inverses: binary ■ Many outcomes of interest do not satisfy this. regression ● Odds ratios & logistic ■ regression Examples: binary outcomes, Poisson count outcomes. ● Link & variance fns. of a GLM ● Binary (again) ■ A Generalized Linear Model (GLM) is a model with two ● Fitting a binary regression GLM: IRLS ingredients: a link function and a variance function. ● Other common examples of GLMs ◆ The link relates the means of the observations to ● Deviance ● Binary deviance predictors: linearization ● Partial deviance tests 2 ◆ ● Wald χ tests The variance function relates the means to the variances. - p. 3/18 Binary regression example ● Today's class ■ A local health clinic sent fliers to its clients to encourage ● Generalized linear models ● Binary regression example everyone, but especially older persons at high risk of ● Binary outcomes ● Logit transform complications, to get a flu shot in time for protection against ● Binary regression ● Link functions: binary an expected flu epidemic. regression ● Link function inverses: binary ■ In a pilot follow-up study, 50 clients were randomly selected regression ● Odds ratios & logistic and asked whether they actually received a flu shot. regression ● Link & variance fns. of a GLM ■ ● Binary (again) In addition, data were collected on their age and their health ● Fitting a binary regression GLM: IRLS awareness. ● Other common examples of GLMs ■ Here is the data. ● Deviance ● Binary deviance ● Partial deviance tests 2 ● Wald χ tests - p. 4/18 Binary outcomes ● Today's class ■ Suppose outcome Y is a 0-1 random variable, then ● Generalized linear models i ● Binary regression example ● Binary outcomes µ E Y π : ● Logit transform i = ( i) = i ● Binary regression ● Link functions: binary ■ regression Variance function ● Link function inverses: binary regression ● Odds ratios & logistic Var(Yi) = πi(1 − πi) = µi(1 − µi) regression ● Link & variance fns. of a GLM ● Binary (again) Variance is related to mean! ● Fitting a binary regression GLM: IRLS ■ A convenient way to model the dependence of Y on ● Other common examples of i GLMs ● Deviance covariates Xi1; : : : ; Xi;p−1 is through the logit transform. ● Binary deviance ● Partial deviance tests 2 p−1 ● Wald χ tests logit(πi) = β0 + βj Xij j=1 X - p. 5/18 Logit transform ● Today's class ■ Logit transform: ● Generalized linear models ● Binary regression example ● Binary outcomes π ● Logit transform logit π ; ● Binary regression ( ) = log 2 (−∞ +1) ● Link functions: binary 1 − π regression ● Link function inverses: binary ■ Inverse: regression x ● Odds ratios & logistic −1 e regression logit (x) = 2 (0; 1): ● Link & variance fns. of a GLM x ● Binary (again) 1 + e ● Fitting a binary regression ■ GLM: IRLS Derivative: ● Other common examples of GLMs ● Deviance d 1 1 ● Binary deviance logit(π) = = : ● Partial deviance tests dπ π(1 − π) V (π) 2 ● Wald χ tests Note: special relation between derivatives and variance function – more on this next lecture. - p. 6/18 Binary regression ● Today's class ■ Logistic regression model: ● Generalized linear models ● Binary regression example ● Binary outcomes logit(E(Yi)) = logit(πi) ● Logit transform ● Binary regression p−1 ● Link functions: binary regression ● Link function inverses: binary = β0 + βj Xij regression ● Odds ratios & logistic j=1 regression X ● Link & variance fns. of a GLM ■ ● Binary (again) Probit regression model: ● Fitting a binary regression GLM: IRLS ● Other common examples of p−1 GLMs −1 ● Deviance Φ (E(Yi)) = β0 + βj Xij ● Binary deviance ● Partial deviance tests j=1 2 ● Wald χ tests X where Φ is CDF of N(0; 1), i.e. Φ(t) = pnorm(t). ■ In each case, Var(Yi) = πi(1 − πi) but the regression model is different. ■ Here is an example - p. 7/18 Link functions: binary regression 6 4 logit probit 2 0 f(pi) −2 −4 −6 0.0 0.2 0.4 0.6 0.8 1.0 pi - p. 8/18 Link function inverses: binary regression 1.0 0.8 logit−inv probit−inv 0.6 f(pi) 0.4 0.2 0.0 −4 −2 0 2 4 pi - p. 9/18 Odds ratios & logistic regression ● Today's class ■ For any event A and any probability P ● Generalized linear models ● Binary regression example ● Binary outcomes P(A) ● Logit transform ODDS(A) = : ● Binary regression P A ● Link functions: binary 1 − ( ) regression ● Link function inverses: binary ■ regression In the logistic regression model with outcome Y ● Odds ratios & logistic regression ● ODDS Y : : : ; X x ; : : : Link & variance fns. of a GLM ( = 1j j = j + 1 ) βj ● Binary (again) = e ● Fitting a binary regression ODDS(Y = 1j : : : ; Xj = xj ; : : : ) GLM: IRLS ● Other common examples of GLMs is the (multiplicative) change in odds if variable X increases ● Deviance j ● Binary deviance by 1: βj is known as the ODDS RATIO for . ● Partial deviance tests e Xj 2 ● Wald χ tests ■ If Xj 2 f0; 1g is dichotomous, and P(Y = 1j : : : ) is small (rare event hypothesis) then group with Xj = 1 are approximately eβj more likely to have event, all other parameters being the same. - p. 10/18 Link & variance fns. of a GLM ● Today's class ■ If ● Generalized linear models ● Binary regression example k ● Binary outcomes E ● Logit transform ηi = g( (Yi)) = g(µi) = β0 + βj Xij ● Binary regression j=1 ● Link functions: binary regression X ● Link function inverses: binary then g is called the link function for the model. regression ● Odds ratios & logistic ■ regression If ● Link & variance fns. of a GLM E ● Binary (again) Var(Yi) = φ · V ( (Yi)) = φ · V (µi) ● Fitting a binary regression GLM: IRLS for φ > 0 and some function V , then V is the called variance ● Other common examples of GLMs ● Deviance function for the model. ● Binary deviance ■ ● Partial deviance tests “Canonical”reference: Generalized Linear Models, 2 ● Wald χ tests McCullagh and Nelder. - p. 11/18 Binary (again) ● Today's class ■ For a logistic model, ● Generalized linear models ● Binary regression example ● Binary outcomes g(µ) = logit(µ); V (µ) = µ(1 − µ): ● Logit transform ● Binary regression ● Link functions: binary ■ For a probit model, regression ● Link function inverses: binary regression −1 ● Odds ratios & logistic g(µ) = Φ (µ); V (µ) = µ(1 − µ): regression ● Link & variance fns. of a GLM ● Binary (again) ● Fitting a binary regression GLM: IRLS ● Other common examples of GLMs ● Deviance ● Binary deviance ● Partial deviance tests 2 ● Wald χ tests - p. 12/18 Fitting a binary regression GLM: IRLS ● Today's class ■ Algorithm ● Generalized linear models ● Binary regression example 1. Initialize: set µi = 0:999 or 0.001 depending on whether ● Binary outcomes ● Logit transform Yi = 1 or 0. ● Binary regression 0 ● Link functions: binary 2. Compute Z = g(µ ) + g (µ )(Y − µ ): regression i b i i i i ● Link function inverses: binary −1 0 2 regression 3. Use weights Wi = g (µi) V (µi) to regress Z onto X’s to ● Odds ratios & logistic regression get β using WLS.b b b ● Link & variance fns. of a GLM ● Binary (again) −1 b p b ● Fitting a binary regression 4. Compute µi = g β0 + Xijβj : GLM: IRLS j=1 ● Other common examples of b GLMs 5. Repeat steps 2-4 until convergence. ● Deviance b P b ● Binary deviance ■ b ● Partial deviance tests Approximate distribution 2 ● Wald χ tests β ∼ N(β; φ(XtW X)−1): ■ If φ has to be estimated, a simple choice is Pearson’s X 2: b c n 1 (Y − µ )2 φ = i i : n − p V (µ ) i=1 i X b b - p. 13/18 b Other common examples of GLMs ● Today's class ■ Standard multiple linear regression: g µ µ, Var µ ● Generalized linear models ( ) = ( ) = 1 ● Binary regression example ■ ● Binary outcomes Linear regression with variance tied to mean, for example: ● Logit transform 2 ● Binary regression g(µ) = µ, Var(µ) = µ . ● Link functions: binary ■ regression Poisson log-linear models: g(µ) = log(µ); Var(µ) = µ. ● Link function inverses: binary regression ● Odds ratios & logistic regression ● Link & variance fns. of a GLM ● Binary (again) ● Fitting a binary regression GLM: IRLS ● Other common examples of GLMs ● Deviance ● Binary deviance ● Partial deviance tests 2 ● Wald χ tests - p. 14/18 Deviance ● Today's class ■ Instead of least squares, models are fit on the basis of scaled ● Generalized linear models ● Binary regression example deviance, analogous to SSE when errors are not Gaussian. ● Binary outcomes ● Logit transform ■ ● Binary regression This is not completely clear from the IRLS algorithm, but it is ● Link functions: binary very close to the Newton-Raphson algorithm. Algorithm regression ● Link function inverses: binary often called Fisher scoring. regression ● Odds ratios & logistic ■ regression ● Link & variance fns. of a GLM ● Binary (again) DEV (µ, Y ) = −2 log L(µ, Y ) + −2 log L(Y; Y ) ● Fitting a binary regression GLM: IRLS ● Other common examples of where µ is a location estimator for Y (usually in an GLMs ● Deviance exponential family – more next lecture). ● Binary deviance ■ 2 ● Partial deviance tests If Y is Gaussian with independent N µ ; σ entries 2 ( i ) ● Wald χ tests n 1 DEV (µ, Y ) = (Y − µ )2 σ2 i i i=1 X - p.