Chapter 2 Simple Linear Regression Analysis The simple linear regression model We consider the modelling between the dependent and one independent variable. When there is only one independent variable in the linear regression model, the model is generally termed as a simple linear regression model. When there are more than one independent variables in the model, then the linear model is termed as the multiple linear regression model. The linear model Consider a simple linear regression model yX01 where y is termed as the dependent or study variable and X is termed as the independent or explanatory variable. The terms 0 and 1 are the parameters of the model. The parameter 0 is termed as an intercept term, and the parameter 1 is termed as the slope parameter. These parameters are usually called as regression coefficients. The unobservable error component accounts for the failure of data to lie on the straight line and represents the difference between the true and observed realization of y . There can be several reasons for such difference, e.g., the effect of all deleted variables in the model, variables may be qualitative, inherent randomness in the observations etc. We assume that is observed as independent and identically distributed random variable with mean zero and constant variance 2 . Later, we will additionally assume that is normally distributed. The independent variables are viewed as controlled by the experimenter, so it is considered as non-stochastic whereas y is viewed as a random variable with Ey()01 X and Var() y 2 . Sometimes X can also be a random variable. In such a case, instead of the sample mean and sample variance of y , we consider the conditional mean of y given X x as E(|)yx01 x Regression Analysis | Chapter 2 | Simple Linear Regression Analysis | Shalabh, IIT Kanpur 1 and the conditional variance of y given Xx as Var(|) y x 2 . 2 When the values of 01,and are known, the model is completely described. The parameters 01, and 2 are generally unknown in practice and is unobserved. The determination of the statistical model 2 yX01 depends on the determination (i.e., estimation ) of 01, and . In order to know the values of these parameters, n pairs of observations (xii ,yi )( 1,..., n ) on ( Xy , ) are observed/collected and are used to determine these unknown parameters. Various methods of estimation can be used to determine the estimates of the parameters. Among them, the methods of least squares and maximum likelihood are the popular methods of estimation. Least squares estimation Suppose a sample of n sets of paired observations (xii ,yi ) ( 1,2,..., n ) is available. These observations are assumed to satisfy the simple linear regression model, and so we can write yxiniii01 (1,2,...,). The principle of least squares estimates the parameters 01and by minimizing the sum of squares of the difference between the observations and the line in the scatter diagram. Such an idea is viewed from different perspectives. When the vertical difference between the observations and the line in the scatter diagram is considered, and its sum of squares is minimized to obtain the estimates of 01and , the method is known as direct regression. yi (xi, Y 01 X (X , i xi Direct regression Regression Analysis | Chapter 2 | Simple Linear Regression Analysis | Shalabh, IIT Kanpur 2 Alternatively, the sum of squares of the difference between the observations and the line in the horizontal direction in the scatter diagram can be minimized to obtain the estimates of 01and . This is known as a reverse (or inverse) regression method. yi YX 01 (xi, yi) (Xi, Yi) xi, Reverse regression method Instead of horizontal or vertical errors, if the sum of squares of perpendicular distances between the observations and the line in the scatter diagram is minimized to obtain the estimates of 01and , the method is known as orthogonal regression or major axis regression method. yi (xi YX 01 (Xi ) xi Major axis regression method Regression Analysis | Chapter 2 | Simple Linear Regression Analysis | Shalabh, IIT Kanpur 3 Instead of minimizing the distance, the area can also be minimized. The reduced major axis regression method minimizes the sum of the areas of rectangles defined between the observed data points and the nearest point on the line in the scatter diagram to obtain the estimates of regression coefficients. This is shown in the following figure: yi (xi yi) YX 01 (Xi, Yi) xi Reduced major axis method The method of least absolute deviation regression considers the sum of the absolute deviation of the observations from the line in the vertical direction in the scatter diagram as in the case of direct regression to obtain the estimates of 01and . No assumption is required about the form of the probability distribution of i in deriving the least squares estimates. For the purpose of deriving the statistical inferences only, we assume that i 's are random 2 variable with E()ii 0,()Var and Cov (, ij )0forall i j(, i j 1,2,...,). n This assumption is needed to find the mean, variance and other properties of the least-squares estimates. The assumption that i 's are normally distributed is utilized while constructing the tests of hypotheses and confidence intervals of the parameters. Based on these approaches, different estimates of 01and are obtained which have different statistical properties. Among them, the direct regression approach is more popular. Generally, the direct regression estimates are referred to as the least-squares estimates or ordinary least squares estimates. Regression Analysis | Chapter 2 | Simple Linear Regression Analysis | Shalabh, IIT Kanpur 4 Direct regression method This method is also known as the ordinary least squares estimation. Assuming that a set of n paired observations on (xii ,yi ), 1,2,..., n are available which satisfy the linear regression model yX01 . So we can write the model for each observation as yxiii 01, (in 1,2,..., ) . The direct regression approach minimizes the sum of squares nn 22 S(,)01 ii (y 0 1x i ) ii11 with respect to 01and . The partial derivatives of S(,)01 with respect to 0 is n S(,)01 2( yxti 01 ) 0 i1 and the partial derivative of S(,)01 with respect to 1 is n S(,)01 2( yxxiii 01 ). 1 i1 The solutions of 01and are obtained by setting S(,) 01 0 0 S(,) 01 0. 1 The solutions of these two equations are called the direct regression estimators, or usually called as the ordinary least squares (OLS) estimators of 01and . This gives the ordinary least squares estimates bb0011of and of as bybx01 sxy b1 sxx where nnnn 2 11 sxy()(),(),xxyy i i s xx xx i x xy i , y i . iiii1111nn Regression Analysis | Chapter 2 | Simple Linear Regression Analysis | Shalabh, IIT Kanpur 5 Further, we have 2S(,) n 012(1)2, n 2 0 i1 2S(,) n 01 2 x2 2 i 1 i1 2 n S(,)01 22. xt nx 01 i1 The Hessian matrix which is the matrix of second-order partial derivatives, in this case, is given as 22 SS(,)01 (,) 01 2 H* 001 22SS(,) (,) 01 01 2 01 1 nnx 2n 2 nx xi i1 ' 2, x x ' where (1,1,...,1)' is a n -vector of elements unity and x (xx1 ,...,n )' is a n -vector of observations on X . The matrix H * is positive definite if its determinant and the element in the first row and column of H * are positive. The determinant of H * is given by n 222 H *4nxnx i i1 n 2 4(nxx i ) i1 0. n 2 The case when ()0xxi is not interesting because all the observations, in this case, are identical, i.e. i1 xi c (some constant). In such a case, there is no relationship between x and y in the context of regression n 2 analysis. Since ()0,xxi therefore H 0. So H is positive definite for any (,)01 , therefore, i1 S(,)01 has a global minimum at (,).bb01 Regression Analysis | Chapter 2 | Simple Linear Regression Analysis | Shalabh, IIT Kanpur 6 The fitted line or the fitted linear regression model is yb01 bx. The predicted values are ybbxiˆii01(1,2,...,). n The difference between the observed value yi and the fitted (or predicted) value yˆi is called a residual. The ith residual is defined as eyyiiii~ˆ ( 1,2,..., n ) yyiiˆ ybbxii().01 Properties of the direct regression estimators: Unbiased property: sxy Note that bbybx101and are the linear combinations of yii (1,...,). n sxx Therefore n bky1 ii i1 nn wherekxxsii ( ) / xx . Note that k i 0 and kx ii 1, so ii11 n Eb()1 kEyii () i1 n kxii(01 ) . i1 1. This b1 is an unbiased estimator of 1 . Next Eb()01 E y bx E01xbx 1 01xx 1 0. Thus b0 is an unbiased estimator of 0 . Regression Analysis | Chapter 2 | Simple Linear Regression Analysis | Shalabh, IIT Kanpur 7 Variances: Using the assumption that ysi ' are independently distributed, the variance of b1 is n 2 Var() b1 kii Var () y k ijij k Cov (, y y ) iiji1 2 ()xxi 2 i (Cov ( y , y ) 0 as y ,..., y are independent) s2 ij1 n xx 2 sxx = 2 sxx 2 = . sxx The variance of b0 is 2 Var() b011 Var ()y xVarb ()2 xCov (,).y b First, we find that Cov(, y b111 ) E y E () y b E () b Ecy() ii 1 i 1 Eccxc()(ii01 iiiii ) 1 n iiii i 1 0000 n 0 So 2 2 1 x Var() b0 . nsxx Covariance: The covariance between b0 and b1 is Cov(,) b01 b Cov (,) y b 1 xVar () b 1 x 2.