Generalized Linear Models Generalized Linear Models MIT 18.655 Dr. Kempthorne Spring 2016 1 MIT 18.655 Generalized Linear Models Linear Predictors and Link Functions Maximum Likelihood Estimation Generalized Linear Models Logistic Regression for Binary Responses Likelihood Ratio Tests Vector Generalized Linear Models Outline 1 Generalized Linear Models Linear Predictors and Link Functions Maximum Likelihood Estimation Logistic Regression for Binary Responses Likelihood Ratio Tests Vector Generalized Linear Models 2 MIT 18.655 Generalized Linear Models Linear Predictors and Link Functions Maximum Likelihood Estimation Generalized Linear Models Logistic Regression for Binary Responses Likelihood Ratio Tests Vector Generalized Linear Models Generalized Linear Model Data: (yi ; xi ); i = 1;:::; n where yi : response variable T xi = (xi;1;:::; xi;p) : p explanatory variables p Linear predictor: For β = (β1; : : : ; βp) 2 R : x β = Pp x β i j=1 i;j j Probability Model: fyi g independent, canonical exponential r.v.'s: ηi yi −A(ηi ) Density: p(yi j ηi ) = e h(x) • Mean Function: µi = E [Yi ] = A(ηi ) Link Function g(·): g(µi ) = xi β ^ ^ With estimate β: xi β = g^(µi ): • −1 T Canonical Link Function: g(µi ) = ηi = [A ] (µi ) = xi β 3 MIT 18.655 Generalized Linear Models Linear Predictors and Link Functions Maximum Likelihood Estimation Generalized Linear Models Logistic Regression for Binary Responses Likelihood Ratio Tests Vector Generalized Linear Models Matrix Notation 232 3 y1 x1;1 x1;2 ··· x1;p 2 3 β1 6 y2 7 6 x2;1 x2;2 ··· x2;p 7 . y = 6 7 X = 6 7 β = 6 . 7 6 . 7 6 ... ... 7 4 . 5 4 . 5 4 .. 5 βp yn xn;1 xn;2 ··· xp;n 2 3 2 • 3 µ1 A(η1) 6 • 7 6 µ2 7 6 A(η2) 7 • E [y] = µ = 6 . 7 = 6 . 7 = A(η) 6 . 7 6 . 7 4 5 4 . 5 µ • n A(ηn) Examples: θi yi ∼ Bernoulli(θi ): ηi = log( ) 1−θi yi ∼ Poisson(λi ): ηi = log(λi ) yi ∼ Gaussian(µi ; 1) : ηi = µi 4 MIT 18.655 Generalized Linear Models Linear Predictors and Link Functions Maximum Likelihood Estimation Generalized Linear Models Logistic Regression for Binary Responses Likelihood Ratio Tests Vector Generalized Linear Models Outline 1 Generalized Linear Models Linear Predictors and Link Functions Maximum Likelihood Estimation Logistic Regression for Binary Responses Likelihood Ratio Tests Vector Generalized Linear Models 5 MIT 18.655 Generalized Linear Models Linear Predictors and Link Functions Maximum Likelihood Estimation Generalized Linear Models Logistic Regression for Binary Responses Likelihood Ratio Tests Vector Generalized Linear Models Log-Likelihood Function for Generalized Linear Model Pn `(β) = i=1(ηi yi − A(ηi ) + log[h(yi )]) T T / η y − 1 A(η) T T = g(µ) y − 1 A(η) T T = [Xβ] y − 1 A(η) (for Canonical Link) T Note: T (y) = X y is sufficient when g(·) is canonical Maximum Likelihood Estimation of β Solve for fβm; m = 1; 2;:::g iteratively: • • •• 0 = `(βm+1) = `(βm) + (βm+1 − βm)`(βm) •• • −1 =) βm+1 = βm + [−`(βm)] `(β m) \Fisher Scoring Algorithm" () Newton-Raphson Pr ^ βm −−−! β (the MLE) 6 MIT 18.655 Generalized Linear Models Linear Predictors and Link Functions Maximum Likelihood Estimation Generalized Linear Models Logistic Regression for Binary Responses Likelihood Ratio Tests Vector Generalized Linear Models For Canonical• Link @ Pn `(β) = @β [(i=1 ηi yi − A(ηi ) + log[h(yi )])] Pn @ = i=1 @β [(ηi yi − A(ηi ) + log[h(yi )])] � � � � Pn @ @ηi = i=1 [(ηi yi − A(ηi ) + log[h(yi )])] @ ηi @β � • � � T � Pn @xi β = i=1 yi − A(ηi ) @β Pn P n T =i=1 (yi − µi ) xi = i=1 xi (yi − µi ) = [X] (y − µ) •• • � • � � T � @ @ Pn @xi β `(β) = [`(β)] = [ y − A(η ) ] @βT @βT i=1 i i @β • @ Pn � � = T [ i=1 yi − A(η i ) xi ] @β � • � = Pn − @ [ A(η )] x ] i=1 @βT i i � • � Pn @ @ηi = − [A(ηi )] T xi ] i=1 @ηi @β � •• � � •• � Pn @ηi Pn T = −[ A (η )] x ] = −[ A(η )] x x i=1 i i @βT i=1 i i i T = X WX 7 MIT 18.655 Generalized Linear Models Linear Predictors and Link Functions Maximum Likelihood Estimation Generalized Linear Models Logistic Regression for Binary Responses Likelihood Ratio Tests Vector Generalized Linear Models For Canonical• Link @ Pn `(β) =@β [ i=1 (ηi yi − A(ηi ) + log[h(yi )])] T = X (y − µ) •• • �• � � T � @ @ Pn @xi β `(β) = [`(β)] = [ y − A(η )] @βT @βT i=1 i i @β •• Pn � � T = i=1 −[A(ηi )] xi xi T = X WX •• where W = Cov(y) is diagonal with Wi;i = A(ηi ) = Var[yi ] 8 MIT 18.655 Generalized Linear Models Linear Predictors and Link Functions Maximum Likelihood Estimation Generalized Linear Models Logistic Regression for Binary Responses Likelihood Ratio Tests Vector Generalized Linear Models Iteratively Re-weighted Least Squares Interpretation • Given βm; and µ^(βm) = A(Xβm); •• • −1 βm+1 = βm + [−`(βm)] `(β m) T −1 T = βm + [X WX] [X] (y − µ^(βm)) Δ = (βm+1 − βm ) is solved as the WLS regression of y∗ = [y − µ^(βm)] on X∗ = WX using Σ∗ = Cov(y∗) = W The WLS estimate of Δ is given by: ^ −1 −1 T −1 Δ = [X∗Σ∗ X∗] X∗ Σ∗ y∗ T −1 −1 T −1 = [X WW WX] X WW [y − µ^(βm)] T −1 T = [X WX] X [y − µ^(βm)] 9 MIT 18.655 Generalized Linear Models Linear Predictors and Link Functions Maximum Likelihood Estimation Generalized Linear Models Logistic Regression for Binary Responses Likelihood Ratio Tests Vector Generalized Linear Models Outline 1 Generalized Linear Models Linear Predictors and Link Functions Maximum Likelihood Estimation Logistic Regression for Binary Responses Likelihood Ratio Tests Vector Generalized Linear Models 10 MIT 18.655 Generalized Linear Models Linear Predictors and Link Functions Maximum Likelihood Estimation Generalized Linear Models Logistic Regression for Binary Responses Likelihood Ratio Tests Vector Generalized Linear Models Logistic Regression for Binary Responses Binomial Data: Yi ∼ Binomial(mi ; πi ); i = 1;:::; k: Log-Likelihood Function � � Qk πi `(π1; : : : ; πk ) = [(Yi log ) + mi log(1 − πi )] i=1 1−πi Covariates: fxi ; i = 1;:::; kg Logistic Regression Parameter: T ηi = log[πi =(1 − π − i)] = xi β W = Cov(Y) = diag(mi πi (1 − πi )), (k × k matrix) 11 MIT 18.655 Generalized Linear Models Linear Predictors and Link Functions Maximum Likelihood Estimation Generalized Linear Models Logistic Regression for Binary Responses Likelihood Ratio Tests Vector Generalized Linear Models Outline 1 Generalized Linear Models Linear Predictors and Link Functions Maximum Likelihood Estimation Logistic Regression for Binary Responses Likelihood Ratio Tests Vector Generalized Linear Models 12 MIT 18.655 Generalized Linear Models Linear Predictors and Link Functions Maximum Likelihood Estimation Generalized Linear Models Logistic Regression for Binary Responses Likelihood Ratio Tests Vector Generalized Linear Models Likelihood Ratio Tests for Generalized Linear Models Consider testing • H0 : µ = µ0 = A(Xβ0): vs HAlt : µ = µ∗ (general n-vector) • e.g., µ∗ = A(X∗β∗) with X∗ = In and β∗ = η: Suppose y is in interior of convex support of fy : p(y j η) > 0g. • −1 Then η^ = η(y) = A (y) is the MLE under HAlt The Likelihood Ratio Test Statistic of H0 vs HAlt : 2 log λ = 2[`(η(Y )) − `(η(µ ))] 0 [ T = 2 [η(y) − η(µ0)] y − [A(η(y)) − A(η(µ0))] = \Deviance" Between y and µ0 13 MIT 18.655 Generalized Linear Models Linear Predictors and Link Functions Maximum Likelihood Estimation Generalized Linear Models Logistic Regression for Binary Responses Likelihood Ratio Tests Vector Generalized Linear Models Deviance Formulas for Distributions P 2 2 Gaussian :(i yi − µi ) /σ0 P Poisson : 2i [yi log(yi /µ^i ) − (yi − µ^i )] P Binomial : 2 i [yi log(yi /µ^i ) + (mi − yi ) log[(mi − yi )=(mi − µ^i )] 14 MIT 18.655 Generalized Linear Models Linear Predictors and Link Functions Maximum Likelihood Estimation Generalized Linear Models Logistic Regression for Binary Responses Likelihood Ratio Tests Vector Generalized Linear Models Outline 1 Generalized Linear Models Linear Predictors and Link Functions Maximum Likelihood Estimation Logistic Regression for Binary Responses Likelihood Ratio Tests Vector Generalized Linear Models 15 MIT 18.655 Generalized Linear Models Linear Predictors and Link Functions Maximum Likelihood Estimation Generalized Linear Models Logistic Regression for Binary Responses Likelihood Ratio Tests Vector Generalized Linear Models Vector Generalized Linear Models Data: (yi ; xi ); i = 1;:::; n where yi : a q-dimensional response vector variable T xi = (xi;1;:::; xi;p) : p explanatory variables Probability Model: The conditional distributions of each yi given xi is of the form p(y j x; B; φ) = f (y; η1; : : : ; ηM ; φ) for some known function f (·); where B = [β1β2 ··· βM ] is a p × M matrix of unknown regression coefficients. M Linear Predictors: For j = 1;:::; M, the jth linear predictor is T Pp ηj = ηj (x) = βj x = k=1 Bkj xk ; T where x = (x1;:::; xp) with x1 = 1 when there is an intercept. 16 MIT 18.655 Generalized Linear Models Linear Predictors and Link Functions Maximum Likelihood Estimation Generalized Linear Models Logistic Regression for Binary Responses Likelihood Ratio Tests Vector Generalized Linear Models Matrix Notation T 32 2 T 23 3 y1 x1 x1;1 x1;2 ··· x1;p T T 6 y 2 7 6 x2 7 6 x2;1 x2;2 ··· x2;p 7 Y = 6 7 X = 6 7 = 6 7 6 . 7 6 . 7 6 ... ... 7 4 . 5 4 . 5 4 .. 5 T T yn xn xn;1 xn;2 ··· xp;n B = [β1jβ2j · · · βM ] Link Function: For each observation i E [yi ] = µi (q × 1 vectors) and T g(µi ) = ηi = B xi (M × 1 vectors) where: 0 01 T 1 η1(xi ) β1 xi B .