Week 3: Finish SLR Inference Then Multiple Linear Regression

BUS41100 Applied Regression Analysis Week 3: Finish SLR Inference Then Multiple Linear Regression

I. Conﬁdence and Prediction Intervals II. Polynomials, log transformation, categorical variables, interactions & main eﬀects

Max H. Farrell The University of Chicago Booth School of Business Quick Recap

I. We drew a line through a cloud of points Yˆ = b + b X & Y Yˆ = e 0 1 − I It was a good line because: 1. It minimized the SSE 2. It extracted all linear information 3. It implemented the model

II. The regression model helped us understand uncertainty Y = β + β X + ε, ε (0, σ2) 0 1 ∼ N I Sampling distribution: estimates change as data changes σ2 b β , σ2 σ2 1 ( 1 b1 ) b1 = 2 ∼ N (n 1)sx − 1 Our work today

I. Finish SLR I Put sampling distribution to work I Communicable summaries of uncertainty

II. Multiple Linear Regression Y = β + β X +β X + ε, ε (0, σ2) 0 1 1 2 2 ∼ N I Everything carries over from SLR I Interpretation requires one extra piece

2 Summarizing the sampling distribution

Remember the two types of regression questions:

1. Model 2. Prediction

Y = β0 + β1X + ε Yˆ = b0 + b1X

Y = b0 + b1X + e

1. Properties of βk I Sign: Does Y go up when X goes up? I Magnitude: By how much? ⇒ A conﬁdence interval captures uncertainty about β

2. Predicting Y I Best guess for Y given (or “conditional on”) X. ⇒ A prediction interval captures uncertainty about Y 3 Conﬁdence Intervals and Testing

0 Suppose we think that the true βj is equal to some value βj (often 0). Does the data support that guess?

We can rephrase this in terms of competing hypotheses.

0 (Null) H0 : βj = βj (Alternative) H : β = β0 1 j 6 j Our hypothesis test will either reject or fail to reject the null hypothesis I If the hypothesis test rejects the null hypothesis, we have statistical support for our claim I Gives only a “yes” or “no” answer! You choose the “probability” of false rejection: α I 4 We use bj for our test about βj. 0 I Reject H0 if bj is “far” from βj ; assume H0 when close I What we really care about is: 0 how many standard errors bj is away from βj

I standard error = sb1 , cf σb1

0 bj βj H0 The t-statistic is this test is zbj = − (0, 1). sbj ∼ N

“Big” z makes our guess β0 look silly reject | βj | j ⇒ I If H0 is true, then P[ zbj >2] < 0.05 = α | |

0 bj βj 0 But: zβj = − > 2 βj (bj 2sbj ) | | sbj ⇔ 6∈ ±

5 Conﬁdence intervals

Since b (β , σ2 ), j ∼ N j bj

bj βj 1 α = P zα/2 < − < z1−α/2 − sbj h i = P βj (bj zα/2sbj ) ∈ ±

Why should we care about conﬁdence intervals? I The conﬁdence interval completely captures the information in the data about the parameter. I Center is your estimate I Length is how sure you are about your estimate I Any value outside would be rejected by a test! 6 Real life or pretend?

h i P β1 (b1 2σb1 ) = 95% ∈ ± or

h i P β1 (b1 2σb1 ) =0 or 1 ∈ ± ?

True β1 7 Level, Size, and p-values

The p-value is P[ Z > zβj ]. | | | | I Test with size/level = p-value almost rejects

I CI of level 1 (p-value) just excludes zβj − | |

Zα 2 Z1 − α 2

−|zβj| |zβj| 1 − α

p/2 Level α p/2 p−value

8 Example: revisit the CAPM regression for the Windsor fund.

Does Windsor have a non-zero intercept? (i.e., does it make/lose money independent of the market?).

H0 : β0 = 0 H : β = 0 1 0 6 I Recall: the intercept estimate b0 is the stock’s “alpha” > summary(windsor.reg) ## output abbreviated Estimate Std. Error t value Pr(>|t|) (Intercept) 0.003647 0.001409 2.588 0.0105 * mfund$valmrkt 0.935717 0.029150 32.100 <2e-16 *** > 2*pnorm(-abs(0.003647/.001409)) [1] 0.009643399 We reject the null at α = .05, Windsor does have an “alpha” over the market. I Why set α = .05? What about at α = 0.01? 9 Now let’s ask whether or not Windsor moves in a diﬀerent way than the market (e.g., is it more conservative?).

I Recall that the estimate of the slope b1 is the “beta” of the stock. This is a rare case where the null hypothesis is not zero:

H0 : β1 = 1, Windsor is just the market (+ alpha). H : β = 1, Windsor softens or exaggerates market moves. 1 1 6 This time, R’s output t/p values are not what we want (why?). > summary(windsor.reg) ## output abbreviated Estimate Std. Error t value Pr(>|t|) (Intercept) 0.003647 0.001409 2.588 0.0105 * mfund$valmrkt 0.935717 0.029150 32.100 <2e-16 ***

10 But we can get the appropriate values easily: I Test and p-value: > b1 <- 0.935717; sb1 <- 0.029150 > zb1 <- (b1 - 1)/sb1 [1] -2.205249 > 2*pnorm(-abs(zb1)) [1] 0.02743665 I Conﬁdence Interval > confint(windsor.reg, level=0.95) 2.5 % 97.5 % (Intercept) 0.000865657 0.006428105 mfund$valmrkt 0.878193149 0.993240873 Reject at α=.05, so Windsor softens than the market. I What about other values of α? confint(windsor.reg, level=0.99) confint(windsor.reg, level=(1-2*pt(-abs(zb1), df=178))) 11 Forecasting & Prediction Intervals

The conditional forecasting problem:

n I Given covariate Xf and sample data Xi,Yi , predict { }i=1 the “future” observation Yf .

The solution is to use our LS ﬁtted value: Yˆf = b0 + b1Xf . I That’s the easy bit.

The hard (and very important!) part of forecasting is assessing uncertainty about our predictions.

One method is to specify a prediction interval I a range of Y values that are likely, given an X value. 12 The least squares line is a prediction rule: Read Yˆ oﬀ the line for a new X. I It’s not a perfect prediction: Yˆ is what we expect.

●

● Yˆ 160 ● 140

●

120 ● ● price

●

100 ● ● ● ● ● 80 ● ●

● 60

1.0 1.5 2.0 2.5 3.0 3.5 size X 13 If we use Yˆf , our prediction error has two pieces

e = Y Yˆ = Y b b X f f − f f − 0 − 1 f

Yf E[Yf Xf ]=0 + 1Xf " | ef { ﬁt error b0 + b1Xf ˆ Yf }

14 We can decompose ef into two sources of error: I Inherent idiosyncratic randomness (due to ε). I Estimation error in the intercept and slope (i.e., discrepancy between our line and “the truth”).

e = Y Yˆ = (Y E[Y X ]) + E[Y X ] Yˆ f f − f f − f | f f | f − f = ε + (E[Y X ] Yˆ ) f f | f − f = ε + (β b ) + (β b )X . f 0 − 0 1 − 1 f

The variance of our prediction error is thus

var(e ) = var(ε ) + var(E[Y X ] Yˆ ) = σ2 + var(Yˆ ) f f f | f − f f

15 From the sampling distributions derived earlier, var(Yˆf ) is

2 var(b0 + b1Xf ) = var(b0) + Xf var(b1) + 2Xf cov(b0, b1) 1 (X X¯)2 = σ2 + f − . n (n 1)s2 − x 2 2 Replacing σ with s gives the standard error for Yˆf .

And hence the variance of our predictive error is

1 (X X¯)2 var(e ) = σ2 1+ + f − . f n (n 1)s2 − x

16 Putting it all together, we have that

1 (X X¯)2 Yˆ Y , σ2 1 + + f − f ∼ N f n (n 1)s2 − x

A (1 α)100% conﬁdence/prediction interval for Y is thus − f

s ! 1 (X X¯)2 b + b X z s 1 + + f − . 0 1 f ± α/2 × n (n 1)s2 − x

17 Looking closer at what we’ll call s 1 (X X¯)2 q s = s 1 + + f − = s2 + s2 . pred n (n 1)s2 ﬁt − x

A large predictive error variance (high uncertainty) comes from I Large s (i.e., large ε’s). I Small n (not enough data).

I Small sx (not enough observed spread in covariates). ¯ I Large (Xf X). − The ﬁrst three are familiar... what about the last one?

18 For Xf far from our X¯, the space between lines is magniﬁed ... Y (X X¯) f True small (X,¯ Y¯ ) Line point of means

Estimated Line

(Xf X¯) large X

19 The prediction (conf.) interval needs to widen away from X¯ ⇒

20 Returning to our housing data for an example ...

> Xf <- data.frame(size=c(mean(size), 2.5, max(size))) > cbind(Xf,predict(reg, newdata=Xf, interval="prediction")) size fit lwr upr 1 1.853333 104.4667 72.92080 136.0125 2 2.500000 127.3496 95.18501 159.5142 3 3.500000 162.7356 127.36982 198.1013

I interval="prediction" gives lwr and upr, otherwise we just get fit

I spred is not shown in this output

21 We can get spred from the predict output.

> p <- predict(reg, newdata=Xf, se.fit=TRUE) > s <- p$residual.scale > sfit <- p$se.fit > spred <- sqrt(s^2+sfit^2) > b <- reg$coef > b[1] + b[2]*Xf[1,]+ c(0,-1, 1)*qnorm(.975)*spred[1] [,1] [,2] [,3] [1,] 104.4667 75.84713 133.0862 > b[1] + b[2]*Xf[1,]+ c(0,-1, 1)*qt(.975, df=n-2)*spred[1] [1,] 104.4667 72.92080 136.0125 I Or, we can calculate it by hand [see R code].

————— q 2 2 Notice that spred = s + sﬁt; you need to square before summing. 22 Summary

Uncertainty matters!

Captured by the Sampling Distribution. I Quantiﬁes uncertainty from the data I . . . only within the model, assumed before we see data. I Which factors matter for signal-to-noise?

Reporting I Conﬁdence Interval: completely captures the information in the data about the parameter. I Testing/p-value: only a yes/no answer. (Don’t abuse p-values)

23 Multiple Linear Regression

24 Multiple vs simple linear regression

Fundamental model is the same.

Basic concepts and techniques translate directly from SLR. I Individual parameter inference and estimation is the same, conditional on the rest of the model. I We still use lm, summary, predict, etc.

The hardest part would be moving to matrix algebra to translate all of our equations. Luckily, R does all that for you.

25 Polynomial regression

A nice bridge between SLR and MLR is polynomial regression.

Still only one X variable, but we add powers of X:

2 m E[Y X] = β0 + β1X + β2X + + βmX | ··· You can ﬁt any mean function if m is big enough. I Usually, m = 2 does the trick.

This is our ﬁrst “multiple linear regression”!

26 Example: telemarketing/call-center data. I How does length of employment (months) relate to productivity (number of calls placed per day)?

> attach(telemkt <- read.csv("telemarketing.csv")) > tele1 <- lm(calls~months) > xgrid <- data.frame(months = 10:30) > par(mfrow=c(1,2)) > plot(months, calls, pch=20, col=4) > lines(xgrid$months, predict(tele1, newdata=xgrid)) > plot(months, tele1$residuals, pch=20, col=4) > abline(h=0, lty=2)

27 ● ● 3 ● ● ● ● ● ● ● ● ● ● ● ● ● 2 ● ●

30 ● ● 1 ● ● ● ● ●

0 ● ●

calls ● 25

residuals ● ● −1 ● ● ● ● ●

−2 ●

20 ● ● ● −3

10 15 20 25 30 10 15 20 25 30

months months

It looks like there is a polynomial shape to the residuals. I We are leaving some predictability on the table . . . just not linear predictability.

28 Testing for nonlinearity

To see if you need more nonlinearity, try the regression which includes the next polynomial term, and see if it is signiﬁcant.

For example, to see if you need a quadratic term, I ﬁt the model then run the regression 2 E[Y X] = β0 + β1X + β2X . | 2 I If your test implies β2 = 0, you need X in your model. 6 Note: p-values are calculated “given the other β’s are nonzero”; i.e., conditional on X being in the model.

29 Test for a quadratic term:

> months2 <- months^2 > tele2 <- lm(calls~ months + months2) > summary(tele2) ## abbreviated output

Coefficients: Estimate Std. Err t value Pr(>|t|) (Intercept) -0.140471 2.32263 -0.060 0.952 months 2.310202 0.25012 9.236 4.90e-08 *** months2 -0.040118 0.00633 -6.335 7.47e-06 ***

The quadratic months2 term has a very signiﬁcant t-value, so 2 a better model is calls = β0 + β1months + β2months + ε.

30 Everything looks much better with the quadratic mean model.

> xgrid <- data.frame(months=10:30, months2=(10:30)^2) > par(mfrow=c(1,2)) > plot(months, calls, pch=20, col=4) > lines(xgrid$months, predict(tele2, newdata=xgrid)) > plot(months, tele2$residuals, pch=20, col=4) > abline(h=0, lty=2)

● ● ● ● ● ● ●

● ● 1.5 ● ● ● ● ● ●

30 ● ● ● ● ● ● ●

0.5 ●

● ● ● ● calls 25 residuals ● ● ● ● −0.5 ● ● ● 20 ● ● ● ● −1.5

10 15 20 25 30 10 15 20 25 30

months months 31 A few words of caution

We can always add higher powers (cubic, etc.) if necessary. I If you add a higher order term, the lower order term is kept regardless of its individual t-stat. (see handout on website)

Be very careful about predicting outside the data range; I the curve may do unintended things beyond the data.

Watch out for over-ﬁtting. I You can get a “perfect” ﬁt with enough polynomial terms, I but that doesn’t mean it will be any good for prediction or understanding. 32 Beyond SLR

Many problems involve more than one independent variable or factor which aﬀects the dependent or response variable. I Multi-factor asset pricing models (beyond CAPM). I Demand for a product given prices of competing brands, advertising, household attributes, etc. I More than size to predict house price!

In SLR, the conditional mean of Y depends on X. The multiple linear regression (MLR) model extends this idea to include more than one independent variable.

33 The MLR Model

The MLR model is same as always, but with more covariates.

Y X ,...,X β + β X + + β X , σ2 | 1 d ∼ N 0 1 1 ··· d d

Recall the key assumptions of our linear regression model:

(i) The conditional mean of Y is linear in the Xj variables. (ii) The additive errors (deviations from line) I are Normally distributed I independent from each other and all the Xj I identically distributed (i.e., they have constant variance)

34 Our interpretation of regression coeﬃcients can be extended from the simple single covariate regression case:

∂E[Y X1,...,Xd] βj = | ∂Xj

I Holding all other variables constant, βj is the average change in Y per unit change in Xj.

————— ∂ is from calculus and means “change in”

35 If d = 2, we can plot the regression surface in 3D.

Consider sales of a product as predicted by price of this product (P1) and the price of a competing product (P2).

I Everything on log scale -1.0 & 1.1 are elasticities ⇒ 36 Obtaining these estimates in R is very easy:

> salesdata <- read.csv("sales.csv") > attach(salesdata) > salesMLR <- lm(Sales ~ P1 + P2) > salesMLR

Call: lm(formula = Sales ~ P1 + P2)

Coefficients: (Intercept) P1 P2 1.003 -1.006 1.098

37 Same Least Squares Principles

How do we estimate the MLR model parameters? ˆ I ﬁtted values Yi = b0 + b1X1i + b2X2i + + bdXdi ··· ˆ I residuals ei = Yi Yi − 2 1 Pn 2 I residual variance s = n−p i=1 ei , p = d + 1

Then ﬁnd the best ﬁtting plane, i.e., coefs b0, b1, b2, . . . , bd, by minimizing the sum of squared errors, s2.

38 Residuals in MLR

As in the SLR model, the residuals in multiple regression are purged of any linear relationship to the independent variables.

We decompose Y into the part predicted by X and the part due to idiosyncratic error.

Y = Yˆ + e

corr(Xj, e) = 0 corr(Yˆ , e) = 0

These hold by construction, just like SLR.

39 Inference for coeﬃcients

As before in SLR, the LS linear coeﬃcients are random (diﬀerent for each sample) and correlated with each other.

The sampling distribution for the whole vector b = [b , b , , b ] is a multivariate Normal: 0 1 ··· d

b (β, Σb) ∼ Nd+1 0 I With mean β = [β0, β1, , βd] (unbiased, as before) ··· I Variance-covariance matrix Σb I Same as last week:

2 b0 β0 σ cov(b0, b1) , b0 b 2 β cov(b , b ) σ2 1 ∼ N 1 0 1 b1 40 Coeﬃcient covariance matrix

Σb = var(b): the p p covariance matrix for random vector b ×

  var(b0) cov(b0, b1)  cov(b , b ) var(b )   1 0 1   ..  Σb =  .     var(bd−1) cov(bd−1, bd)  cov(bd, bd−1) var(bd)

I Variance decreases with n and var(X); increases with σ2.

Standard errors are the square root of the diagonal of Σb. ⇒ 41 Inference for individual coeﬃcients

Intervals and t-statistics are exactly the same as in SLR.

I A (1 α)100% C.I. for βj is bj zα/2sbj . − ± 0 I zbj = (bj β )/sbj (0, 1) is number of standard − j ∼ N errors between the LS estimate and the null value.

Intervals/testing via bj & sbj are one-at-a-time procedures: I You are evaluating the jth coeﬃcient conditional on the other X’s being in the model, but regardless of the values you’ve estimated for the other b’s.

42 Conveniently, R’s summary gives you all the standard errors. (or do it manually, see week3-Rcode.R)

> summary(salesMLR) ## abbreviated output

Coefficients: Estimate Std. Error t value Pr(>|t|) (Intercept) 1.002688 0.007445 134.7 <2e-16 *** P1 -1.005900 0.009385 -107.2 <2e-16 *** P2 1.097872 0.006425 170.9 <2e-16 ***

Residual standard error: 0.01453 on 97 degrees of freedom Multiple R-squared: 0.998, Adjusted R-squared: 0.9979 F-statistic: 2.392e+04 on 2 and 97 DF, p-value: < 2.2e-16

43 Forecasting in MLR

Prediction follows exactly the same methodology as in SLR.

For new data x = [X X ]0, f 1,f ··· d,f ˆ I Yf = b0 + b1X1f + + bdXdf ··· ˆ 2 2 2 I var[Yf xf ] = var(Yf ) + var(εf ) = s + s = s . | ﬁt pred ˆ I (1 α) level prediction interval is still Yf zα/2spred. − ±

44 The syntax in R is also exactly the same as before:

> predict(salesMLR, data.frame(P1=1, P2=1), + interval="prediction", level=0.95) fit lwr upr 1 1.094661 1.064015 1.125306

> predict(salesMLR, data.frame(P1=1, P2=1), + se.fit=TRUE)$se.fit [1] 0.005227347

45 Glossary and Equations s s LS Estimators: y xy and ¯ ¯. I b1 = rxy = 2 b0 = Y b1X sx sx − ˆ I Yi = b0 + b1Xi is the ith ﬁtted value. ˆ I ei = Yi Yi is the ith residual. − I σˆ, s: standard error of regression residuals ( σ = σε). ≈ I sbj : standard error of regression coeﬃcients. s s s2 1 X¯ 2 s = s = s + b1 (n 1)s2 b0 n (n 1)s2 − x − x

46 I α is the signiﬁcance level (prob of type 1 error).

I zα/2 is the value such that for Z (0, 1), ∼ N

P[Z > zα/2] = P[Z < zα/2] = α/2. −

I zbj is the standardized coeﬃcient:

0 bj βj H0 zbj = − (0, 1). sbj ∼ N

I The (1 α) 100% conﬁdence interval for βj is − ∗ b z s j ± α/2 bj

47 ˆ I Yf = b0 + Xf b1 is a forecast prediction. s 1 (X X¯)2 se(Yˆ ) = s = s + f − f ﬁt n (n 1)s2 − x ˆ 2 2 I Forecast residual is ef = Yf Yf and var(ef ) = s + s . − ﬁt That is, the predictive standard error is s 1 (X X¯)2 s = s 1 + + f − . pred n (n 1)s2 − x and Yˆ z s is the (1 α)100% prediction interval f ± α/2 pred − at Xf .

48 Glossary and equations

MLR: ind 2 I Model: Y X1,...,Xd (β0 + β1X1 + + βdXd, σ ) | ∼ N ··· ˆ I Prediction: Yi = b0 + b1X1i + b2X2i + + bdXdi ··· I b p(β, Sb) ∼ N I Interaction:

I Yi = β0 + β1X1i + β2X2i + β3(X1iX2i) + ... + ε ∂E[Y |X1,X2] = β + β X I ∂X1 1 3 2

49 Glossary and equations

R2 and related terms Pn ¯ 2 I SST = (Yi Y ) i=1 − Pn ˆ ¯ 2 I SSR = (Yi Y ) i=1 − Pn ˆ 2 I SSE = (Yi Yi) i=1 − I (Watch out: sometimes SSE is called SSR or RSS!) 2 2 ˆ 2 I R = SSR/SST = cor (Y,Y ) = ryyˆ

d%Y Elasticity is the slope in a log-log model: β . 1 ≈ d%X (See handout on course website.)