ACTA UNIVERSITATIS LODZIENSIS FOLIA OECONOMICA 269, 2012 Grzegorz Ko Ĕczak * ON TESTING THE SIGNIFICANCE OF THE COEFFICIENTS IN THE MULTIPLE REGRESSION ANALYSIS Abstract. The multiple regression analysis is a statistical tool for the investigation relation- ships between the dependent and independent variables. There are some procedures for selecting a subset of given predictors. These procedures are widely available in statistical computer pack- ages. The most often used are forward selection, backward selection and stepwise selection. In these procedures testing the significance of parameters is used. If some assumptions such as nor- mality errors are not fulfilled, the results of testing significance of the parameters may not be trustworthy. The main goal of this paper is to present a permutation test for testing the significance of the coefficients in the regression analysis. Permutation tests can be used even if the normality assumption is not fulfilled. The properties of this test were analyzed in the Monte Carlo study. Key words: linear regression model, permutation test, Monte Carlo. I. INTRODUCTION AND BASIC NOTATIONS A.C. Rencher (2002) considered multiple linear regression models for fixed and random x's. The errors in these models can be normally or non-normally distributed. Let us consider the multiple linear regression model for fixed x's given by y E 0 E1x1 E2 x2 ... E q xq H (1) where y is the dependent variable x1, x 2 , …, x q represent q different variables (fixed) E 0 is the intercept E 1 ,E2 ,...,Eq represent the corresponding q regression coefficients H is the random error where * Ph.D., Associate Professor, Department of Statistics, Katowice University of Economics, [email protected] [63] 64 Grzegorz Ko Ĕczak E(H) = 0 and D2 ( H) = V2. (2) In this model each y ( y1, y2, ..., yn) in the sample of n observations can be ex- pressed as a linear function of x's plus random error H. The model (1) can be rewritten as follows § y · §1 x x " x1q ·§ E0 · § H · ¨ 1 ¸ ¨ 11 12 ¸¨ ¸ ¨ 1 ¸ ¨ y2 ¸ ¨1 x 21 x22 " x2q ¸¨ E1 ¸ ¨H 2 ¸ ¨ ¸ ¨ ¸¨ ¸ ¨ ¸ (3) ¨ # ¸ ¨# # # % # ¸¨ # ¸ ¨ # ¸ ¨ ¸ ¨ x ¸¨E ¸ ¨ ¸ © yn ¹ ©1 x n 1 xn2 " nq ¹© q ¹ ©H n ¹ or equally Y Xȕ İ. (4) The assumptions (2) can be rewritten as follows 1. E( İi ) ,0 for all i = 1, 2, ..., n. 2 2 2. D ( H i ) V , ,for all i = 1, 2, ..., n. 3. Cov( H i ,H k ) ,0 for all i z k. The hypothesis statements to test the significance of a particular regression coefficient E j ( j = 1, 2, …, q) can be written as follows: H0 : E j 0 (5) H1 : E j z 0 For testing the significance of the individual regression coefficient E j the t test is used. The t test statistics is based on the t distribution and has the form Eˆ t j (6) ˆ S(E j ) ˆ where E j is the least square estimator of the parameter ȕj ( j = 1, 2, ..., q) and ˆ ˆ S(E j ) is the estimated standard error of E j . The standard error of each parame- ˆ ˆ ter E j is given by the square root of the diagonal elements of the matrix Var (E ) where On Testing the Significance of the Coefficients… 65 Var (E ˆ) V 2 (X T X ) 1 The t test could be used if errors are independent and normally distributed. The statistic (6) under H 0 has the t distribution with n - q - 1 degrees of freedom. The null hypothesis (5) is rejected if | t |!tD ,2/ n q1 . If the null hypothesis (5) is not rejected, this indicates that the regressor xj could be removed from the model. If this hypothesis is rejected, then the regressor xj could be added to the model. The multiple linear regression model for random x's has the same form as (1) but it is assumed that x1, x2, ..., xq are not under control of experimenter. Many regression applications involve x's that are random variables. If we assume that the vector ( y, x1, x2, ..., xq) has a multivariate normal distribution (L. Godfrey, 2009 and D.J. Sheskin, 2004), we can proceed with testing in the same way as in the fixed x's case. The statistic (6) could be used if the following assumptions are fulfilled a) the x’s are fixed b) errors are independent and normally distributed or a) the x’s are random and normally distributed b) errors are independent and normally distributed If these assumptions are not fulfilled then testing the significance of parame- ters in the regression analysis can't be performed. II. VARIABLES SELECTION METHODS IN THE LINEAR MODEL One of the most important problems in the multiple regression analysis is the selection of variables. The methods of selecting variables are a way of selecting a particular set of independent variables to be used in the regression model. There is a large number of commonly used methods which are called stepwise techniques. The most often used are forward selection, backward selection and stepwise selection: forward selection starts with no variables selected. Next we add the most significant variable. At each step we add the most significant variable until there are no variables that meet the criterion set by the user, backward selection starts with all variables selected. At each step the least significant variable is removed from the model until none of them meets the criterion set by the user, stepwise selection is a method that is a combination of two previous methods, testing at each stage for including or excluding variables. 66 Grzegorz Ko Ĕczak The methods described above (and some other) are included in statistical packages such as SPSS, Statistica, MiniTab, Statgraphics, R. For each of these methods the significance of parameters is assessed at each step of the procedure. The t test is used many times at each step. If one of the following conditions is fulfilled the errors are not normally distributed the errors are not independent the errors are not homoscedastic the x’s are random and not normally distributed then the t test shouldn't be performed. In this case the permutation test can be used instead. Permutation tests can be performed even if the normality assump- tion is not fulfilled. III. PROBLEMS FOR NORMALITY TESTING In many regression applications x's are not fixed. If the methods of selecting predicting variables described above are used then normality of independent random variables has to be tested. The normality hypothesis can be tested using normality test (for example Shapiro-Wilk's test, Lilliforse's test or chi square goodness of fit test). Even if the null hypothesis in normality testing is not re- jected, it is possible that the sample is taken from non-normal distribution. Nor- mality testing for small sample sizes was analyzed in the Monte Carlo study. X ~ B 2 3 2 1 X2 ~ F10 f(x) f(x) 0.0 0.5 1.0 1.5 0.00 0.04 0.08 0.0 0.2 0.4 0.6 0.8 1.0 0 5 10 15 20 25 30 x x X3 ~ *2 X4 ~ LN 0 1 f(x) f(x) 0.0 0.1 0.2 0.3 0.0 0.2 0.4 0.6 01 2 34 5 6 0 1 2 3 4 x x Fig. 1. The densities of random variables under study On Testing the Significance of the Coefficients… 67 There 4 non-normal distributions were considered: beta (random variable X1), chi-square ( X2), gamma ( X3) and log-normal ( X4). The details of the random variables X1, X2, X3 and X4 are given in table 1. The density functions of ana- lyzed random variables are presented in Fig. 1. The sample of the size n = 15 was generated from these distributions. Then the normality hypotheses were tested using Lillieforse's test and Shapiro-Wilk's test. This procedure was re- peated 1 000 times and the probabilities of the rejection of the null hypothesis were estimated. The simulation study was performed using R (http://www.r- project.org) procedures. The results of computer simulation are presented in table 1. Table 1. Estimated probabilities of failing to reject the normality hypothesis Details Lilliforses Shapiro- R function of the random variable test Wilk's test X 1 : B )3,2( 2 rbeta(n, 2, 3) 0.9454 0.9392 E ( X1) 4,0 D ( X1) ,0 04 2 X 2 : F10 rchisq(n, 10) 0.8737 0.8181 2 E ( X 2 ) 10 D ( X 2 ) 20 X3 : * )2( 2 rgamma(n, 2) 0.8597 0.7914 E ( X 3) 2 D ( X 3 ) 2 X 4 : LN )1,0( 2 rlrnorm(n, 0, 1) 0.7420 0.6123 E ( X 4 ) e D ( X 4 ) e(e )1 Source: Monte Carlo study. It is easy to notice (table 1) that for the analyzed non-normal random vari- ables testing the hypothesis often leads to "no reject the null hypothesis". The standard error of estimation of probabilities included in table 1 is less than 0.016. IV. PERMUTATION TEST VERSUS t TEST – MONTE CARLO STUDY Variables selection methods in the multiple regression analysis are based on the t test. This test can be performed only if the assumptions mentioned above are fulfilled. It is possible that the t test is performed, due to the results of nor- mality testing, even for non-normally distributed x's. The results of the use of the t test and the permutation test were compared. The model analyzed in the Monte Carlo study had the form of: 68 Grzegorz Ko Ĕczak Y E 1 X1 E2 X 2 E3 X 3 E4 X 4 H (7) where H ~ N )1,0( and E j }1,0{ for j = 1, 2, 3, 4.