Simple Linear Regression

Example: Body density Aim: Measure body density (weight per unit volume of the body) (Body density indicates the fat content of the human body.) Problem: ◦ Body density is difficult to measure directly. ◦ Research suggests that skinfold thickness can accurately predict body density. ◦ Skinfold thickness is measures by pinching a fold of skin between calipers.

2.0 ) 3 1.8 m kg 3

10 1.6 (

1.4 Body Density 1.2

1.0 1.03 1.04 1.05 1.06 1.07 1.08 1.09 Skinfold Thickness (mm)

Questions: ◦ Are body density and skinfold thickness related? ◦ How accurately can we predict body density from skinfold thickness?

Regression: predict response variable for fixed value of explanatory variable ◦ describe linear relationship in data by regression line ◦ fitted regression line is affected by chance variation in observed data

Statistical inference: accounts for chance variation in data

Simple Linear Regression, Feb 27, 2004 - 1 - Population Regression Line

Simple linear regression studies the relationship between ◦ a response variable Y and ◦ a single explanatory variable X. We expect that different values of X will produce different mean responses of Y . For given X = x, we consider the subpopulation with X = x: ◦ this subpopulation has mean

µY |X=x = E(Y |X = x) (cond. mean of Y given X = x)

◦ and variance

2 σY |X=x = var(Y |X = x) (cond. variance of Y given X = x) Linear regression model with constant variance:

E(Y |X = x) = µY |X=x = a + b x (population regression line) 2 2 var(Y |X = x) = σY |X=x = σ ◦ The population regression line connects the conditional means of the response variable for fixed values of the explanatory variable. ◦ This population regression line tells how the mean response of Y varies with X. ◦ The variance (and standard deviation) does not depend on x.

Simple Linear Regression, Feb 27, 2004 - 2 - Conditional Mean

Sample (x1, y1),..., (xn, yn)

6 1 5 2 3 4 4 5 3 6 7 2 8 9 1 10 11 0 12

Sampling probability f(x, y)

6 0 1 5 2 3 4 4 5 3 6 7 2 8 9 1 10 11 0 12 y fix x = x0

6 0 f(x0, y) 1 5 2 3 4 4 5 3 6 7 2 8 y 9 1 10 rescale by fX (x0) 11 0 12 Conditional probability 6 0 1 5 2 fXY (x0, y) 3 4 4 f(y|x0) = 5 3 f (x ) 6 X 0 7 2 8 9 1 10 11 0 12

Z E(Y |X = x0) = y fY |X(y|x0) dy conditional mean

Simple Linear Regression, Feb 27, 2004 - 3 - The Linear Regression Model

Simple linear regression

Yi = a + b xi + εi, i = 1, . . . , n

where

Yi response (also dependent variable)

xi predictor (also independent variable)

εi error

Assumptions:

◦ Predictor xi is deterministic (fixed values, not random).

◦ Errors have zero mean, E(εi) = 0. 2 ◦ Variation about mean does not depend on xi, i.e. var(εi) = σ .

◦ Errors εi are independent. Often we additionally assume: ◦ The errors are normally distributed,

iid 2 εi ∼ N (0, σ ).

For fixed x the response Y is normally distributed with

Y ∼ N (a + b x, σ2).

Simple Linear Regression, Feb 27, 2004 - 4 - Least Squares Estimation

Data: (Y1, x1),..., (Yn, xn)

Aim: Find straight line which fits data best:

Yˆi = a + b xi fitted values for coefficients a and b

a - intercept b - slope Least Squares Approach:

Minimize squared distance between observed Yi and fitted Yˆi: n n P 2 P 2 L(a, b) = (Yi − Yˆi) = (Yi − a − b xi) i=1 i=1

Set partial derivatives to zero (normal equations):

n ∂L P = 0 ⇔ (Yi − a − b xi) = 0 ∂a i=1 n ∂L P = 0 ⇔ (Yi − a − b xi) · xi = 0 ∂b i=1

Solution: Least squares estimators S aˆ = Y¯ − XY · X¯ SXX S ˆb = XY SXX where

n P SXY = (Yi − Y¯ )(xi − x¯) (sum of squares) i=1 n P 2 SXX = (xi − x¯) i=1

Simple Linear Regression, Feb 27, 2004 - 5 - Least Squares Estimation

Least squares predictor Yˆ ˆ Yˆi =a ˆ + b xi

Residuals εˆi:

εˆi = Yi − Yˆi ˆ = Yi − aˆ − b xi

Residual sum of squares (SS Residual)

n n P 2 P ˆ 2 SS Residual = εˆi = (Yi − Yi) i=1 i=1 Estimation of σ2 n 2 1 P 2 1 σˆ = (Yi − Yˆi) = SS Residual n − 2 i=1 n − 2 Regression standard error p se =σ ˆ = SS Residual/(n − 2)

Variation accounting:

n P 2 SS Total = (Yi − Y¯ ) total variation i=1 n P 2 SS Model = (Yˆi − Y¯ ) variation explained by linear model i=1 n P 2 SS Residual = (Yi − Yˆi) remaining variation i=1

Simple Linear Regression, Feb 27, 2004 - 6 - Least Squares Estimation

Example: Body density Scatter plot with least squares regression line:

2.0 ) 3 1.8 m kg 3

10 1.6 (

1.4 Body Density 1.2

1.0 1.03 1.04 1.05 1.06 1.07 1.08 1.09 Skinfold Thickness (mm)

Calculation of least squares estimates:

x¯ y¯ SXX SXY SYY SS Residual 1.064 1.568 0.0235 -0.2679 4.244 1.187

S −0.267 ˆb = XY = = −11.40 SXX 0.023 aˆ =y ¯ − ˆbx¯ = 1.568 + 11.40 · 1.064 = 13.70 RSS 1.187 σˆ2 = = = 0.0132 n − 2 90 √ √ 2 se = σˆ = 0.0132 = 0.1149

Simple Linear Regression, Feb 27, 2004 - 7 - Least Squares Estimation

Example: Returns on Treasury bills and inflation Using STATA:

. infile ID BODYD SKINT using bodydens.txt, clear (92 observations read) . regress BODYD SKINT Source | SS df MS Number of obs = 92 ------+------F( 1, 90) = 231.89 Model | 3.05747739 1 3.05747739 Prob > F = 0.0000 Residual | 1.18663025 90 .013184781 R-squared = 0.7204 ------+------Adj R-squared = 0.7173 Total | 4.24410764 91 .046638546 Root MSE = .11482 ------BODYD | Coef. Std. Err. t P>|t| [95% Conf. Interval] ------+------SKINT | -11.41345 .7494999 -15.23 0.000 -12.90246 -9.924433 _cons | 13.71221 .7975822 17.19 0.000 12.12768 15.29675 ------. twoway (lfitci BODYD SKINT, range(1 1.1)) (scatter BODYD SKINT), xtitle(Skin thickn > ess) ytitle(Body density) scheme(s1color) legend(off) 2.5 2 Body density 1.5 1 1 1.02 1.04 1.06 1.08 1.1 Skin thickness

Simple Linear Regression, Feb 27, 2004 - 8 -