Logistic Regression and Generalized Linear Models

Home , Logistic regression, Variance function

University of Massachusetts

©Sridhar Mahadevan: CMPSCI 689 – p. 1/29 Topics

Generative vs. Discriminative models In many cases, it is difficult to model data using a parametric class conditional density P (X|ω, θ) Yet, in many problems, a linear decision boundary is usually adequate to separate classes (also, gaussian densities with a shared covariance matrix produces a linear decision boundary). Logistic regression: discriminative model for classification that produces linear decision boundaries Model fitting problem solved using maximum likelihood Iterative gradient-based algorithm for solving nonlinear maximum likelihood equations Recursive weighted least squares regression Logistic regression is an instance of a generalized linear model (GLM), which consists of a large variety of exponential models.GLMs can also be extended to generalized additive models (GAMs).

©Sridhar Mahadevan: CMPSCI 689 – p. 2/29 Discriminative vs. Generative Models

Both generative and discriminative approaches address the problem of modeling the discriminant function P (y|x) of output labels (or values) y conditioned on the input x. In generative models, we estimate both P (x) and P (x|y), and use Bayes rule to compute the discriminant.

P (y|x) ∝ P (x)P (x|y)

Discriminative approaches model the conditional distribution P (y|x) directly, and ignore the marginal P (x). We now turn to explore several types instances of discriminative models, including logistic regression in this class, and later several other types including support vector machines.

©Sridhar Mahadevan: CMPSCI 689 – p. 3/29 Generalized Linear Models

In linear regression, we model the output y as a linear function of the input variables, with a noise term that is zero mean constant variance Gaussian. y = g(x)+ ǫ, where the conditional mean E(y|x)= g(x), and the noise term is ǫ. T g(x)= β x (where β0 is an offset term). We saw earlier that the maximum likelihood framework justiﬁed the use of a squared error loss function, provided the errors were IID gaussian (the variance does not matter). We want to generalize this idea of specifying a model family by specifying the type of error distribution: When the output variable y is discrete (e.g., binary or multinomial), the noise term is not gaussian, but binomial or multinomial. A change in the mean is coupled by a change in the variance, and we want to be able to couple mean and variance in our model. Generalized linear models provides a rich tool of models based on specifying the error distribution.

©Sridhar Mahadevan: CMPSCI 689 – p. 4/29 Logit Function

Since the output variable y only takes on values ∈ (0, 1) (for binary classiﬁcation), we need a different way of representing E(y|x) so that the range of y ∈ (0, 1). One convenient form to use is the sigmoid or logistic function. Let us assume a vector-valued input variable x =(x1,...,xp). The logistic function is S shaped and approaches 0 (as x → −∞) or 1 (as x → ∞).

T eβ x 1 P (y = 1|x, β)= µ(x|β)= = 1+ eβT x 1+ e−βT x

1 P (y = 0|x, β) = 1 − µ(x|β)= 1+ eβT x

We assume an extra input x0 = 1, so that β0 is an offset. We can invert the above transformation to get the logit function

µ(x|β) g(x|β) = log = βT x 1 − µ(x|β)

©Sridhar Mahadevan: CMPSCI 689 – p. 5/29 Logistic Regression

β0 β2 β1

X2 X1 X0

©Sridhar Mahadevan: CMPSCI 689 – p. 6/29 Example Dataset for Logistic Regression

The data set we are analyzing is coronary heart disease in South Africa. The chd response (output) variable is binary (yes, no), and there are 9 predictor variables: There are 462 instances, out of which 160 are cases (positive instances), and 302 are controls (negative instances). The predictor variables are systolic blood pressure, tobacco, ldl, famhist, obesity, alcohol, age, adiposity, typea, Let’s focus on a subset of the predictors: sbp, tobacco, ldl, famhist, obesity, alcohol, age. We want to ﬁt a model of the following form

1 P (chd = 1|x, β)= 1+ e−βT x where βT x = β0 +β1xsbp +β2xtobacco +β3xldl +β4xfamhist +β5xage +β6xalcohol +β7xobesity

©Sridhar Mahadevan: CMPSCI 689 – p. 7/29 Noise Model for Logistic Regression

Let us try to represent the logistic regression model as y = µ(x|β)+ ǫ and ask ourself what sort of noise model is represented by ǫ. Since y takes on the value 1 with probability µ(x|β), it follows that ǫ can also only take on two possible values, namely If y = 1, then ǫ = 1 − µ(x|β) with probability µ(x|β). Conversely, if y = 0, then ǫ = −µ(x|β) and this happens with probability (1 − µ(x|β)). This analysis shows that the error term in logistic regression is a binomially distributed random variable. Its moments can be computed readily as shown below: E(ǫ)= µ(x|β)(1 − µ(x|β)) − (1 − µ(x|β))µ(x|β) = 0 (the error term has mean 0). V ar(ǫ)= Eǫ2 − (Eǫ)2 = Eǫ2 = µ(x|β)(1 − µ(x|β)) (show this!)

©Sridhar Mahadevan: CMPSCI 689 – p. 8/29 Maximum Likelihood for LR

Suppose we want to ﬁt a logistic regression model to a dataset of n observations X =(x1, y1),..., (xn, yn). We can express the conditional likelihood of a single observation simply as

i i P (yi|xi, β)= µ(xi|β)y (1 − µ(xi|β))1−y

Hence, the conditional likelihood of the entire dataset can be written as

n i i P (Y |X, β)= µ(xi|β)y (1 − µ(xi|β))1−y Y i=1

The conditional log-likelihood is then simply

n l(β|X,Y )= yi log µ(xi|β)+(1 − yi) log(1 − µ(xi|β)) X i=1

©Sridhar Mahadevan: CMPSCI 689 – p. 9/29 Maximum Likelihood for LR

We solve the conditional log-likelihood equation by taking gradients

n ∂l(β|X,Y ) 1 ∂µ(xi|β) 1 ∂µ(xi) = yi − (1 − yi) i i ∂βk X µ(x |β) ∂βk (1 − µ(x |β)) ∂βk i=1

i ∂µ(x |β) ∂ 1 i i i Using the fact that = ( T i )= µ(x |β)(1 − µ(x |β))x , we get βk ∂βk 1+e−β x k

n ∂l(β|X,Y ) i i i = xk(y − µ(x |β)) ∂βk X i=1

Setting this to 0, since x0 = 1 the ﬁrst component of these equations reduces to

n n yi = µ(xi|β) X X i=1 i=1

The expected number of instances of each class must match the observed number. ©Sridhar Mahadevan: CMPSCI 689 – p. 10/29 Newton-Raphson Method

Newton’s method is a general procedure for ﬁnding the roots of an equation f(θ) = 0. Newton’s algorithm is based on the recursion

f(θt) θ = θ − t+1 t ′ f (θt)

Newton’s method ﬁnds the minimum of a function f. We want to ﬁnd the maximum of the log likehood equation. But, the maximum of a function f(θ) is exactly when its derivative f ′(θ) = 0. So, plugging in f ′(θ) for f(θ) above, we get

′ f (θt) θ = θ − t+1 t ′′ f (θt)

©Sridhar Mahadevan: CMPSCI 689 – p. 11/29 Fisher Scoring

In logistic regresion, the parameter β is a vector, so we have to use the Newton-Raphson algorithm

−1 βt+1 = βt − H ∇β l(βt|X,Y )

Here, ∇β l(βt|X,Y ) is the vector of partial derivatives of the log-likelihood equation 2 ∂ l(β|X,Y ) Hij = is the Hessian matrix of second order derivatives. ∂βi∂βj The use of Newton’s method to ﬁnd the solution to the conditional log-likelihood equation is called Fisher scoring.

©Sridhar Mahadevan: CMPSCI 689 – p. 12/29 Fisher Scoring for Maximum Likelihood

Taking the second derivative of the likelihood score equations gives us

n 2 ∂ l(β|X,Y ) i i i i = − xkxmµ(x |β)(1 − µ(x |β)) ∂βk∂βm X i=1

We can use matrix notation to write the Newton-Raphson algorithm for logistic regression. Deﬁne the n × n diagonal matrix

µ(x1|β)(1 − µ(x1|β)) ... 0   0 µ(x2|β)(1 − µ(x2|β)) ... W =    ...     0 ... µ(xn|β)(1 − µ(xn|β)) 

Let Y be an n × 1 column vector of output values, and X be the design matrix of size n × (p + 1) of input values, and P be the column vector of ﬁtted probability values µ(xi|β).

©Sridhar Mahadevan: CMPSCI 689 – p. 13/29 Iterative Weighted Least Squares

The gradient of the log likelihood can be written in matrix form as

n ∂l(β|X,Y ) = xi(yi − µ(xi|β)) = XT (Y − P ) ∂β X i=1

The Hessian can be written as

∂2l(β|X,Y ) = −XT W X ∂β∂βT

The Newton-Raphson algorithm then becomes

βnew = βold +(XT W X)−1XT (Y − P ) = (XT W X)−1XT W Xβold + W −1(Y − P ) = (XT W X)−1XT W Z where Z ≡ Xβold + W −1(Y − P )

©Sridhar Mahadevan: CMPSCI 689 – p. 14/29 Weighted Least Squares Regression

Weighted least squares regression ﬁnds the best least-squares solution to the equation

W Ax ≈ Wb (WA)T WAxˆ = (WA)T Wb xˆ = (AT CA)−1AT Cb where C = W T W

Returning to logistic regression, we now see βnew =(XT W X)−1XT W Z is weighted least squares regression (where X is the matrix A above, W is a diagonal weight vector with entries µ(xi|β)(1 − µ(xi|β)), and Z corresponds to the vector b above). It is termed recursive weighted least squares, because at each step, the weight vector W keeps changing (since the β’s are changing). We can visualize RWLS as solving the following equation

new T β ← argminβ (Z − Xβ) W (Z − Xβ)

©Sridhar Mahadevan: CMPSCI 689 – p. 15/29 Stochastic Gradient Ascent

Newton’s method is often referred to as a 2nd order method, because it involves taking the Hessian. This can be difﬁcult in large problems, because it involves matrix inversion. One way to avoid this is to settle for slower convergence, but less work at each step. For each training instance (x, y) we can derive an incremental gradient update rule.

∂l(β|x, y) = xj (y − µ(x|β)) ∂βj

The stochastic gradient ascent rule can be written as (for instance (xi, yi))

i i i βj ← βj + α(y − µ(x |β))xj

The convergence of this update rule can be iffy, unlike Newton’s method. It depends on tweaking the learning rate α so that the steps are not too small or too large, and also a cooling schedule.

©Sridhar Mahadevan: CMPSCI 689 – p. 16/29 The LMS Algorithm

The LMS (least mean square) algorithm can be used to solve least squares regression problems incrementally taking the gradient of the loss function w.r.t. the parameters, and adjusting the weights for each data instance.

n h(x|β)= β + β x = βT x 0 X j j j=1

Given a data set D, we want to ﬁnd the vector β that minimizes the (mean) 1 2 squared error loss L(h)= (yi − h(x; β)) . 2 Pi LMS algorithm: ﬁnd the gradient on a particular instance

∂L(h) = −(y − h(x|β))xj βj

Adjust the weight in the direction of decreasing the error (negative gradient).

βj ← βj + α(y − h(x|β))xj

©Sridhar Mahadevan: CMPSCI 689 – p. 17/29 Logistic Regression vs LDA

Recall from Bayes decision theory that when the class conditional densities

P (x|ωi, µi, Σ) share the same underlying covariance matrix Σ, the decision boundary that separates the classes is a line (or hyperplane). Such Bayesian classiﬁers are called linear discriminant classiﬁers. We see from above that logistic regression also produces a decision boundary that is a hyperplane, since its discriminant function was shown to be

µ(x|β) g(x|β) = log = βT x 1 − µ(x|β)

So, if both gaussian LDA and logistic regression produce linear decision boundaries, which is preferable? As a general rule, logistic regression works in a larger class of problems because it does not assume the underlying class conditional densities are Gaussian!

A generalized linear model is speciﬁed using two functions A link function that describes how the mean depends on the linear predictor g(µ)= η, where η = βT x. For example, in logistic regression, the link function is the logit function µ T g(µ) = log 1−µ = β x In linear regression, the link function is simply g(µ)= µ The inverse of the logit function g−1(η)= µ describes how the mean µ can be related back to the linear predictor. For logistic regression, the inverse link function is the sigmoid. A variance function that speciﬁes how the variance of the output y depends on the mean φV (µ), where φ is constant. For example, in logistic regression, the variance is µ(1 − µ), because it is a binomial distribution. In linear regression, the variance is simply 1 because the error term is modeled as having constant variance.

Distribution Link Function Variance Function Gaussian µ 1 µ − Binomial log 1−µ µ(1 µ) Poisson log µ µ 1 2 Gamma µ µ

As one more example of a generalized linear model, we generalize the logistic regression model to a multinomial (e.g, sorting email into various categories)

It easily follows that

βT x e k P (Y = k|X = x, β) = , k = 1,...,K − 1 K−1 βT x 1+ e l Pl=1 1 P (Y = K|X = x, β) = , k = 1,...,K − 1 K−1 βT x 1+ e l l=1 P ©Sridhar Mahadevan: CMPSCI 689 – p. 21/29 Multinomial Link function

The multinomial PDF can be written as a member of the exponential family

1{y=1} 2{y=2} 1{y=K} P (y|φ) = φ1 φ2 . . . φK k−1 1− y˜i y˜1 y˜2 i=1 = φ1 φ2 . . . φK P T = eη y˜−a(η)

The last expression above is an instance of the generic form of the exponential family (check Weber handout or earlier notes) Instead of a single link function, we now have a vector

log φ1 φk  φ2  log φ η =  k   .   .   φk−1  log  φk  ©Sridhar Mahadevan: CMPSCI 689 – p. 22/29 Fitting GLM Models in R

The statisics package R has comprehensive built-in features for fitting generalized linear models (and many related models as well). This is a very brief intro to model fitting in R. See the R documentation, as well as the excellent text Statistical Models in S by Chambers and Hastie. The main function is called glm(formula=, family=,...). Here, formula is a symbolic description of the model that is to be fit. family is the description of the appropriate error distribution and link function to be used.

The data set we are analyzing is coronary heart disease in South Africa. The chd response (output) variable is binary (yes, no), and there are 9 predictor variables: systolic blood pressure, tobacco, ldl, famhist, obesity, alcohol, age, adiposity, typea, The R command glm(chd ∼ sbp + tobacco + ldl + famhist + age + alcohol + obesity, family = binomial, data = heart) will ﬁt a logistic regression model to the heart disease data set (assuming the dataset is loaded into the system). The R command glm(chd ∼ sbp + tobacco + ldl + famhist + age + alcohol + obesity, family = binomial(link=probit), data = heart) will ﬁt the same data, now using the probit distribution (inverse CDF of the normal distribution).

The following model was found to be the best ﬁt to the data, using maximum likelihood.

Coefﬁcient Estimate (Intercept) -4.1290787 sbp 0.0057608 tobacco 0.0795237 ldl 0.1847710 famhistPresent 0.9391330 age 0.0425344 alcohol 0.0006058 obesity -0.0345467

Load the data data(Cars93) which contains information about 93 models of cars. names(Cars93) tells you the attributes that make up the dataset.

[1] "Manufacturer" "Model" "Type" [4] "Min.Price" "Price" "Max.Price" [7] "MPG.city" "MPG.highway" "AirBags" [10] "DriveTrain" "Cylinders" "EngineSize" [13] "Horsepower" "RPM" "Rev.per.mile" [16] "Man.trans.avail" "Fuel.tank.capacity" "Passengers" [19] "Length" "Wheelbase" "Width" [22] "Turn.circle" "Rear.seat.room" "Luggage.room" [25] "Weight" "Origin" "Make"

Let’s assume we want to ﬁt a polynomial regression model, and we chose Price as the predictor variable, and Weight as the response variable. We can graph the these two variables using: plot(Weight ∼ Price, data = Cars93)

1 2

The R command to ﬁt a dth degree polynomial is glm(Weight ∼ poly(Price, d), family = gaussian, data = Cars93) Notice how we speciﬁed the link function to be gaussian. W 2 2

1 2

In the regression setting, a generalized additive model has the form

E(Y |X1,...,Xp)= α + f1(X1)+ f2(X2)+ ... + fp(Xp)

Here, the fi are unspeciﬁed “smooth” (nonparametric ) functions. In the classiﬁcation setting, a generalized additive logistic regression model has the form

µ(X|β) log( = α + f1(X1)+ f2(X2)+ ... + fp(Xp) 1 − µ(X|β)

See Section 9.1.2 in Hastie’s book (Statistical Learning) for how to ﬁt additive logistic regression models to an email-spam dataset.