DOCSLIB.ORG

Prediction, Goodness-Of-Fit, and Modeling Issues 4.1 Least Squares

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Prediction, Goodness-Of-Fit, and Modeling Issues 4.1 Least Squares 12/10/2007 Prediction, Goodness-of-Fit, and Modeling Issues Prepared by Vera Tabakova, East Carolina University 4.1 Least Squares Prediction 4.2 Measuring Goodness-of-Fit 4.3 Modlideling Issues 4.4 Log-Linear Models Principles of Econometrics, 3rd Edition Slide 4-2 yxe01200=+ββ + (4.1) where e0 is a random error. We assume that Ey ( 0120 ) =β +β x and . We also ass ume that 2 and E (e0 ) = 0 var (e0 ) =σ cov(ee0 ,i ) == 0 i 1,2,… , N ybbxˆ0120=+ (4.2) Principles of Econometrics, 3rd Edition Slide 4-3 1 12/10/2007 Figure 4.1 A point prediction Principles of Econometrics, 3rd Edition Slide 4-4 f =−=β+β+yy00ˆ ( 1200 xe) −( bbx 120 + ) (4.3) Ef( ) = β+β120 x + Ee( 0) − ⎣⎦⎡+ ⎡ Eb( 1) + Ebx( 20) ⎤ =β120 +βxx +00 −[] β 120 +β = 2 2 ⎡⎤1 ()xx0 − var(f )=σ⎢⎥ 1 + + 2 (4.4) ⎣⎦Nxx∑()i − Principles of Econometrics, 3rd Edition Slide 4-5 The variance of the forecast error is smaller when i. the overall uncertainty in the model is smaller, as measured by the variance of the random errors ; ii. the sample size N is larger; iii. the variation in the exppylanatory variable is larg g;er; and iv. the value of is small. Principles of Econometrics, 3rd Edition Slide 4-6 2 12/10/2007 ⎡⎤2 2 1(xx0 − ) var(f )=σˆ ⎢⎥ 1 + + 2 ⎣⎦Nxx∑()i − se()f = var ()f (4.5) ytˆ0 ± cse( f) (4.6) Principles of Econometrics, 3rd Edition Slide 4-7 y Figure 4.2 Point and interval prediction Principles of Econometrics, 3rd Edition Slide 4-8 ybbxˆ0120=+ =83.4160 + 10.2096(20) = 287.6089 ⎡⎤2 2 1(xx0 − ) var(f )=σˆ ⎢⎥ 1 + + 2 ⎣⎦Nxx∑()i − 22 22σσˆˆ =σˆ + +()xx0 − 2 N ∑()xxi − σˆ 2 =σˆ 22 + +()varxx − () b N 02 ytˆ0 ±=cse( f ) 287.6069 ± 2.0244(90.6328) =[ 104.1323,471.0854] Principles of Econometrics, 3rd Edition Slide 4-9 3 12/10/2007 yxeiii=β12 +β + (4.7) yEyeiii=+() (4. 8) yyeiii=+ˆˆ (4.9) yyiii−=() yyeˆˆ − + (4.10) Principles of Econometrics, 3rd Edition Slide 4-10 Figure 4.3 Explained and unexplained components of yi Principles of Econometrics, 3rd Edition Slide 4-11 ()yy− 2 σ=ˆ 2 ∑ i y N −1 222(4. 11) ∑∑∑()()yyiii−= yyˆˆ −+ e Principles of Econometrics, 3rd Edition Slide 4-12 4 12/10/2007 2 ∑()yi −y = total sum of squares = SST: a measure of total variation in y about the sample mean. 2 ∑()yˆi − y = sum of squares due to the regression = SSR: that part of total variation in y, about the sample mean, that is explained by, or due to, the regression. Also known as the “explained sum of squares .” 2 ∑eˆi = sum of squares due to error = SSE: that part of total variation in y about its mean that is not explained by the regression. Also known as the unexplained sum of squares, the residual sum of squares, or the sum of squared errors. SST = SSR + SSE Principles of Econometrics, 3rd Edition Slide 4-13 2 SSR SSE R ==−1 (4.12) SST SST 2 The closer R is to one, the closer the sample values yi are to the fitted regression 2 equation ybbxˆ ii =+ 12 . If R = 1, then all the sample data fall exactly on the fitted least squares line, so SSE = 0, and the model fits the data “perfectly.” If the sample data for y and x are uncorrelated and show no linear association, then the least squares fitted line is “horizontal,” so that SSR = 0 and R2 = 0. Principles of Econometrics, 3rd Edition Slide 4-14 cov(xy , ) σxy ρ=xy = (4.13) var(xy ) var( ) σσx y σˆ xy rxy = (4.14) σσˆˆx y σ=ˆ xy∑()()xxyy i − i −( N − 1) 2 σ=ˆ xi∑()xx −() N − 1 (4.15) 2 σ=ˆ yi∑()yy −() N − 1 Principles of Econometrics, 3rd Edition Slide 4-15 5 12/10/2007 22 rRxy = 22 R = ryyˆ 2 R measures the linear association, or goodness-of-fit, between the sample data 2 and their predicted values. Consequently R is sometimes called a measure of “goodness-of-fit.” Principles of Econometrics, 3rd Edition Slide 4-16 2 SST=−=∑() yi y 495132.160 2 2 SSE=−==∑∑() yii yˆˆ e i304505.176 SSE 304505.176 R2 =−11 =− = .385 SST 495132.160 σˆ xy 478.75 rxy == =.62 σσˆˆxy ()()6.848 112.675 Principles of Econometrics, 3rd Edition Slide 4-17 Figure 4.4 Plot of predicted y, yˆ against y Principles of Econometrics, 3rd Edition Slide 4-18 6 12/10/2007 FOOD_EXP = weekly food expenditure by a household of size 3, in dollars INCOME = weekly household income, in $100 units FOOD_EXP =83.42+= 10.21 INCOME R2 .385 (se) (43.41)**** (2.09) * indicates significant at the 10% level ** indicates significant at the 5% level *** indicates significant at the 1% level Principles of Econometrics, 3rd Edition Slide 4-19 4.3.1 The Effects of Scaling the Data Changing the scale of x: ** yxecxcexe= β12+ β + =()(/)β 1+ β 2+ =β 12+ β + ** where ββ22==cxxc and Changing the scale of y: *** * yc/(/)(/)(/)=β12 c +β cx + ec or y =β+β 12 x + e Principles of Econometrics, 3rd Edition Slide 4-20 Variable transformations: Power: if x is a variable then xp means raising the variable to the power p; examples are quadratic (x2) and cubic (x3) transformations. The natural logarithm: if x is a variable then its natural logg(arithm is ln(x). The reciprocal: if x is a variable then its reciprocal is 1/x. Principles of Econometrics, 3rd Edition Slide 4-21 7 12/10/2007 Figure 4.5 A nonlinear relationship between food expenditure and income Principles of Econometrics, 3rd Edition Slide 4-22 The log-log model ln(yx )=β12 +β ln( ) The parameter β is the elasticity of y with respect to x. The log-linear model ln(yxii ) =β12 +β A one-unit increase in x leads to (approximately) a 100 β2 percent change in y. The linear-log model Δβy yx=β +βln() or = 2 12 100()Δ xx 100 A 1% increase in x leads to a β2/100 unit change in y. Principles of Econometrics, 3rd Edition Slide 4-23 The reciprocal model is 1 FOOD_ EXP= β + β + e 12INCOME The linear-log model is FOOD_ln() EXP=β12 +β INCOME + e Principles of Econometrics, 3rd Edition Slide 4-24 8 12/10/2007 Remark: Given this array of models, that involve different transformations of the dependent and independent variables, and some of which have similar shapes, what are some ggguidelines for choosing a functional form? 1. Choose a shape that is consistent with what economic theory tells us about the relationship. 2. Choose a shape that is sufficiently flexible to “fit” the data 3. Choose a shape so that assumptions SR1-SR6 are satisfied, ensuring that the least squares estimators have the desirable properties described in Chapters 2 and 3. Principles of Econometrics, 3rd Edition Slide 4-25 Figure 4.6 EViews output: residuals histogram and summary statistics for food expenditure example Principles of Econometrics, 3rd Edition Slide 4-26 The Jarque-Bera statistic is given by 2 N ⎛⎞()K − 3 JB=+⎜⎟ S 2 ⎜⎟ 64⎝⎠ where N is the sample size, S is skewness, and K is kurtosis. In the food expenditure example 2 40 ⎛⎞()2.99− 3 JB =−⎜⎟.0972 + = .063 ⎜⎟ 64⎝⎠ Principles of Econometrics, 3rd Edition Slide 4-27 9 12/10/2007 Figure 4.7 Scatter plot of wheat yield over time Principles of Econometrics, 3rd Edition Slide 4-28 YIELDttt=β12 +β TIME + e 2 YIELDtt=+.638 .0210 TIME R = .649 (se) (.064) (.0022) Principles of Econometrics, 3rd Edition Slide 4-29 Figure 4.8 Predicted, actual and residual values from straight line Principles of Econometrics, 3rd Edition Slide 4-30 10 12/10/2007 Figure 4.9 Bar chart of residuals from straight line Principles of Econometrics, 3rd Edition Slide 4-31 3 YIELDttt=β12 +β TIME + e TIMECUBE= TIME3 1000000 2 YIELDtt=+0.874 9.68 TIMECUBE R = 0.751 (se) (.036) (.082) Principles of Econometrics, 3rd Edition Slide 4-32 Figure 4.10 Fitted, actual and residual values from equation with cubic term Principles of Econometrics, 3rd Edition Slide 4-33 11 12/10/2007 4.4.1 The Growth Model ln(YIELDt ) =++ ln( YIELD0 ) ln( 1 g) t = β+β12t ln()YIELDt =− .3434 + .0178 t (se) (.0584) (.0021) Principles of Econometrics, 3rd Edition Slide 4-34 4.4.2 A Wage Equation ln(WAGE) =++ ln( WAGE0 ) ln( 1 r) EDUC = βββ12+ β EDUC ln()WAGE=+× .7884 .1038 EDUC (se) (.0849) (.0063) Principles of Econometrics, 3rd Edition Slide 4-35 4.4.3 Prediction in the Log-Linear Model yybbxˆ ==+exp ln exp n ( ()) ()12 22σˆ 2 yEˆˆˆcn==Ebb( y) exp(bb12 ++σ=xye2) ln()WAGE=+× .7884 .1038 EDUC =+×= .7884 .1038 12 2.0335 σˆ 2 2 yEyyeˆˆcn==() =7.6408 × 1.1276 = 8.6161 Principles of Econometrics, 3rd Edition Slide 4-36 12 12/10/2007 4.4.4 A Generalized R2 Measure 2 22ˆ Ryyrgyy=⎡⎣⎦corr() , ⎤ = , ˆ 2 22ˆ Ryygc=⎡⎣⎦corr() , ⎤ = .4739 = .2246 R2 values tend to be small with microeconomic, cross-sectional data, because the variations in individual behavior are difficult to fully explain. Principles of Econometrics, 3rd Edition Slide 4-37 4.4.5 Prediction Intervals in the Log-Linear Model ⎡⎤expln()y −+tf se() ,expln() ytf se() ⎣⎦⎢⎥( cc) ( ) ⎣⎦⎡−×exp( 2.0335 1.96 .4905) ,exp( 2.0335 +×⎤= 1.96 .4905) [ 2.9184,20.0046] Principles of Econometrics, 3rd Edition Slide 4-38 coefficient of linear model determination linear relationship correlation linear-log model data scale log-linear model forecast error log-log model forecast standard log-normal error distribution functional form prediction goodness-of-fit prediction interval growth model R2 Jarque-Bera test residual kurtosis skewness least squares predictor Principles of Econometrics, 3rd Edition Slide 4-39 13 12/10/2007 Principles of Econometrics, 3rd Edition Slide 4-40 f =−=β+β+yy00ˆ ( 1200 xe) −+( bbx 120) 2 var( yˆ0120102012) =+=+ var(bbx) var( b) x var( b) + 2 x cov( bb , ) 22 2 σ ∑ xi 22σ−x =+22xx00 +σ2 2 Nxx∑∑()ii−−() xx ∑() xx i − Principles of Econometrics, 3rd Edition Slide 4-41 ⎡⎤⎡⎤22 ⎧⎫22 222 ⎧⎫ 22 σ ∑ xi ⎪⎪σσNx x0 σ−()2xx0 ⎪⎪ σ Nx var ()yˆ0 =−⎢⎥⎢⎥⎨⎬ +++ ⎨⎬ N xx−−−−−22222 N xx xx xx N xx ⎣⎦⎣⎦⎢⎥⎢⎥∑∑∑∑∑()iiiii⎩⎭⎪⎪() () () ⎩⎭⎪⎪() ⎡⎤2222 2 ∑ xNxi − xxxx00−+2 =σ⎢⎥22 + ⎣⎦⎢⎥Nxx∑∑( ii−−) ( xx) ⎡⎤22 2 ∑()xxi
Load more

Recommended publications
  • Goodness of Fit: What Do We Really Want to Know? I
    PHYSTAT2003, SLAC, Stanford, California, September 8-11, 2003 Goodness of Fit: What Do We Really Want to Know? I. Narsky California Institute of Technology, Pasadena, CA 91125, USA Definitions of the goodness-of-fit problem are discussed. A new method for estimation of the goodness of fit using distance to nearest neighbor is described. Performance of several goodness-of-fit methods is studied for time-dependent CP asymmetry measurements of sin(2β). 1. INTRODUCTION of the test is then defined as the probability of ac- cepting the null hypothesis given it is true, and the The goodness-of-fit problem has recently attracted power of the test, 1 − αII , is defined as the probabil- attention from the particle physics community. In ity of rejecting the null hypothesis given the alterna- modern particle experiments, one often performs an tive is true. Above, αI and αII denote Type I and unbinned likelihood fit to data. The experimenter Type II errors, respectively. An ideal hypothesis test then needs to estimate how accurately the fit function is uniformly most powerful (UMP) because it gives approximates the observed distribution. A number of the highest power among all possible tests at the fixed methods have been used to solve this problem in the confidence level. In most realistic problems, it is not past [1], and a number of methods have been recently possible to find a UMP test and one has to consider proposed [2, 3] in the physics literature. various tests with acceptable power functions. For binned data, one typically applies a χ2 statistic There is a long-standing controversy about the con- to estimate the fit quality.
    [Show full text]
  • Linear Regression: Goodness of Fit and Model Selection
    Linear Regression: Goodness of Fit and Model Selection 1 Goodness of Fit I Goodness of fit measures for linear regression are attempts to understand how well a model fits a given set of data. I Models almost never describe the process that generated a dataset exactly I Models approximate reality I However, even models that approximate reality can be used to draw useful inferences or to prediction future observations I ’All Models are wrong, but some are useful’ - George Box 2 Goodness of Fit I We have seen how to check the modelling assumptions of linear regression: I checking the linearity assumption I checking for outliers I checking the normality assumption I checking the distribution of the residuals does not depend on the predictors I These are essential qualitative checks of goodness of fit 3 Sample Size I When making visual checks of data for goodness of fit is important to consider sample size I From a multiple regression model with 2 predictors: I On the left is a histogram of the residuals I On the right is residual vs predictor plot for each of the two predictors 4 Sample Size I The histogram doesn’t look normal but there are only 20 datapoint I We should not expect a better visual fit I Inferences from the linear model should be valid 5 Outliers I Often (particularly when a large dataset is large): I the majority of the residuals will satisfy the model checking assumption I a small number of residuals will violate the normality assumption: they will be very big or very small I Outliers are often generated by a process distinct from those which we are primarily interested in.
    [Show full text]
  • Chapter 2 Simple Linear Regression Analysis the Simple
    Chapter 2 Simple Linear Regression Analysis The simple linear regression model We consider the modelling between the dependent and one independent variable. When there is only one independent variable in the linear regression model, the model is generally termed as a simple linear regression model. When there are more than one independent variables in the model, then the linear model is termed as the multiple linear regression model. The linear model Consider a simple linear regression model yX01 where y is termed as the dependent or study variable and X is termed as the independent or explanatory variable. The terms 0 and 1 are the parameters of the model. The parameter 0 is termed as an intercept term, and the parameter 1 is termed as the slope parameter. These parameters are usually called as regression coefficients. The unobservable error component accounts for the failure of data to lie on the straight line and represents the difference between the true and observed realization of y . There can be several reasons for such difference, e.g., the effect of all deleted variables in the model, variables may be qualitative, inherent randomness in the observations etc. We assume that is observed as independent and identically distributed random variable with mean zero and constant variance 2 . Later, we will additionally assume that is normally distributed. The independent variables are viewed as controlled by the experimenter, so it is considered as non-stochastic whereas y is viewed as a random variable with Ey()01 X and Var() y 2 . Sometimes X can also be a random variable.
    [Show full text]
  • A Study of Some Issues of Goodness-Of-Fit Tests for Logistic Regression Wei Ma
    Florida State University Libraries Electronic Theses, Treatises and Dissertations The Graduate School 2018 A Study of Some Issues of Goodness-of-Fit Tests for Logistic Regression Wei Ma Follow this and additional works at the DigiNole: FSU's Digital Repository. For more information, please contact [email protected] FLORIDA STATE UNIVERSITY COLLEGE OF ARTS AND SCIENCES A STUDY OF SOME ISSUES OF GOODNESS-OF-FIT TESTS FOR LOGISTIC REGRESSION By WEI MA A Dissertation submitted to the Department of Statistics in partial fulfillment of the requirements for the degree of Doctor of Philosophy 2018 Copyright c 2018 Wei Ma. All Rights Reserved. Wei Ma defended this dissertation on July 17, 2018. The members of the supervisory committee were: Dan McGee Professor Co-Directing Dissertation Qing Mai Professor Co-Directing Dissertation Cathy Levenson University Representative Xufeng Niu Committee Member The Graduate School has verified and approved the above-named committee members, and certifies that the dissertation has been approved in accordance with university requirements. ii ACKNOWLEDGMENTS First of all, I would like to express my sincere gratitude to my advisors, Dr. Dan McGee and Dr. Qing Mai, for their encouragement, continuous support of my PhD study, patient guidance. I could not have completed this dissertation without their help and immense knowledge. I have been extremely lucky to have them as my advisors. I would also like to thank the rest of my committee members: Dr. Cathy Levenson and Dr. Xufeng Niu for their support, comments and help for my thesis. I would like to thank all the staffs and graduate students in my department.
    [Show full text]
  • Testing Goodness Of
    Dr. Wolfgang Rolke University of Puerto Rico - Mayaguez CERN Phystat Seminar 1 Problem statement Hypothesis testing Chi-square Methods based on empirical distribution function Other tests Power studies Running several tests Tests for multi-dimensional data 2 ➢ We have a probability model ➢ We have data from an experiment ➢ Does the data agree with the probability model? 3 Good Model? Or maybe needs more? 4 F: cumulative distribution function 퐻0: 퐹 = 퐹0 Usually more useful: 퐻0: 퐹 ∊ ℱ0 ℱ0 a family of distributions, indexed by parameters. 5 Type I error: reject true null hypothesis Type II error: fail to reject false null hypothesis A: HT has to have a true type I error probability no higher than the nominal one (α) B: probability of committing the type II error (β) should be as low as possible (subject to A) Historically A was achieved either by finding an exact test or having a large enough sample. p value = probability to reject true null hypothesis when repeating the experiment and observing value of test statistic or something even less likely. If method works p-value has uniform distribution. 6 Note above: no alternative hypothesis 퐻1 Different problem: 퐻0: 퐹 = 푓푙푎푡 vs 퐻0: 퐹 = 푙푖푛푒푎푟 → model selection Usually better tests: likelihood ratio test, F tests, BIC etc. Easy to confuse: all GOF papers do power studies, those need specific alternative. Our question: is F a good enough model for data? We want to guard against any alternative. 7 Not again … Actually no, GOF equally important to both (everybody has a likelihood) Maybe more so for Bayesians, no non- parametric methods.
    [Show full text]
  • Measures of Fit for Logistic Regression Paul D
    Paper 1485-2014 SAS Global Forum Measures of Fit for Logistic Regression Paul D. Allison, Statistical Horizons LLC and the University of Pennsylvania ABSTRACT One of the most common questions about logistic regression is “How do I know if my model fits the data?” There are many approaches to answering this question, but they generally fall into two categories: measures of predictive power (like R-square) and goodness of fit tests (like the Pearson chi-square). This presentation looks first at R-square measures, arguing that the optional R-squares reported by PROC LOGISTIC might not be optimal. Measures proposed by McFadden and Tjur appear to be more attractive. As for goodness of fit, the popular Hosmer and Lemeshow test is shown to have some serious problems. Several alternatives are considered. INTRODUCTION One of the most frequent questions I get about logistic regression is “How can I tell if my model fits the data?” Often the questioner is expressing a genuine interest in knowing whether a model is a good model or a not-so-good model. But a more common motivation is to convince someone else--a boss, an editor, or a regulator--that the model is OK. There are two very different approaches to answering this question. One is to get a statistic that measures how well you can predict the dependent variable based on the independent variables. I’ll refer to these kinds of statistics as measures of predictive power. Typically, they vary between 0 and 1, with 0 meaning no predictive power whatsoever and 1 meaning perfect predictions.
    [Show full text]
  • Goodness of Fit of a Straight Line to Data
    5. A linear trend. 10.4 The Least Squares Regression Line LEARNING OBJECTIVES 1. To learn how to measure how well a straight line fits a collection of data. 2. To learn how to construct the least squares regression line, the straight line that best fits a collection of data. 3. To learn the meaning of the slope of the least squares regression line. 4. To learn how to use the least squares regression line to estimate the response variable y in terms of the predictor variablex. Goodness of Fit of a Straight Line to Data Once the scatter diagram of the data has been drawn and the model assumptions described in the previous sections at least visually verified (and perhaps the correlation coefficient r computed to quantitatively verify the linear trend), the next step in the analysis is to find the straight line that best fits the data. We will explain how to measure how well a straight line fits a collection of points by examining how well the line y=12x−1 fits the data set Saylor URL: http://www.saylor.org/books Saylor.org 503 To each point in the data set there is associated an “error,” the positive or negative vertical distance from the point to the line: positive if the point is above the line and negative if it is below theline. The error can be computed as the actual y-value of the point minus the y-value yˆ that is “predicted” by inserting the x-value of the data point into the formula for the line: error at data point (x,y)=(true y)−(predicted y)=y−yˆ The computation of the error for each of the five points in the data set is shown in Table 10.1 "The Errors in Fitting Data with a Straight Line".
    [Show full text]
  • Using Minitab to Run a Goodness‐Of‐Fit Test 1. Enter the Values of A
    Using Minitab to run a Goodness‐of‐fit Test 1. Enter the values of a qualitative variable under C1. 2. Click on “Stat”, choose “Tables” and then “Chi‐square Goodness of Fit Test (One Variable)”. 3. Click on the circle next to “Categorical data:” and enter C1 in the box to the right. 4. Under “Tests”, click on the circle next to “Equal proportions”. 5. Click on the “Graphs” button and uncheck the boxes next to both types of bar chart. 6. Click on “OK” in that window and on “OK” in the next window. The test results will appear in the “Sessions” window under the heading “Chi‐square Goodness of Fit Test for Categorical Variable: … ”. The test statistic will appear under “Chi‐Sq”. The P‐value will appear under “P‐value”. Example (Navidi & Monk, Elementary Statistics, 2nd edition, #23 p.566): The significance level is 0.01. Since 1 – 0.01 = 0.99, the confidence is 99%. The null hypothesis is that all months are equally likely. The alternative is the months are not all equally likely. The data is shown in the table below. Month Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Alarms 32 15 37 38 45 48 46 42 34 36 28 26 Open Minitab and enter the alarms values under C1. A portion of the entered data is shown below. ↓ C1 C2 Alarms 1 32 2 15 3 37 4 38 5 45 6 48 7 46 8 42 9 34 10 36 Now click on “Stat” and then choose “Tables” and “Chi‐Square Goodness‐of‐Fit Test (One Variable) …”.
    [Show full text]
  • Goodness-Of-Fit Tests for Parametric Regression Models
    Goodness-of-Fit Tests for Parametric Regression Models Jianqing Fan and Li-Shan Huang Several new tests are proposed for examining the adequacy of a family of parametric models against large nonparametric alternatives. These tests formally check if the bias vector of residuals from parametric ts is negligible by using the adaptive Neyman test and other methods. The testing procedures formalize the traditional model diagnostic tools based on residual plots. We examine the rates of contiguous alternatives that can be detected consistently by the adaptive Neyman test. Applications of the procedures to the partially linear models are thoroughly discussed. Our simulation studies show that the new testing procedures are indeed powerful and omnibus. The power of the proposed tests is comparable to the F -test statistic even in the situations where the F test is known to be suitable and can be far more powerful than the F -test statistic in other situations. An application to testing linear models versus additive models is also discussed. KEY WORDS: Adaptive Neyman test; Contiguous alternatives; Partial linear model; Power; Wavelet thresholding. 1. INTRODUCTION independent observations from a population, Parametric linear models are frequently used to describe Y m4x5 ˜1 ˜ N 401‘ 251 (1.1) the association between a response variable and its predic- D C tors. The adequacy of such parametric ts often arises. Con- ventional methods rely on residual plots against tted values where x is a p-dimensional vector and m4 5 is a smooth regres- sion surface. Let f 4 1 ˆ5 be a given parametri¢ c family. The or a covariate variable to detect if there are any system- ¢ atic departures from zero in the residuals.
    [Show full text]
  • Measures of Fit for Logistic Regression Paul D
    Paper 1485-2014 Measures of Fit for Logistic Regression Paul D. Allison, Statistical Horizons LLC and the University of Pennsylvania ABSTRACT One of the most common questions about logistic regression is “How do I know if my model fits the data?” There are many approaches to answering this question, but they generally fall into two categories: measures of predictive power (like R-square) and goodness of fit tests (like the Pearson chi-square). This presentation looks first at R-square measures, arguing that the optional R-squares reported by PROC LOGISTIC might not be optimal. Measures proposed by McFadden and Tjur appear to be more attractive. As for goodness of fit, the popular Hosmer and Lemeshow test is shown to have some serious problems. Several alternatives are considered. INTRODUCTION One of the most frequent questions I get about logistic regression is “How can I tell if my model fits the data?” Often the questioner is expressing a genuine interest in knowing whether a model is a good model or a not-so-good model. But a more common motivation is to convince someone else--a boss, an editor, or an audience--that the model is OK. There are two very different approaches to answering this question. One is to get a statistic that measures how well you can predict the dependent variable based on the independent variables. I’ll refer to these kinds of statistics as measures of predictive power. Typically, they vary between 0 and 1, with 0 meaning no predictive power whatsoever and 1 meaning perfect predictions. Predictive power statistics available in PROC LOGISTIC include R-square, the area under the ROC curve, and several rank-order correlations.
    [Show full text]
  • Lecture 10: F-Tests, R2, and Other Distractions
    00:35 Friday 16th October, 2015 See updates and corrections at http://www.stat.cmu.edu/~cshalizi/mreg/ Lecture 10: F -Tests, R2, and Other Distractions 36-401, Section B, Fall 2015 1 October 2015 Contents 1 The F Test 1 1.1 F test of β1 = 0 vs. β1 6=0 .....................2 1.2 The Likelihood Ratio Test . .7 2 What the F Test Really Tests 11 2.1 The Essential Thing to Remember . 12 3 R2 16 3.1 Theoretical R2 ............................ 17 3.2 Distraction or Nuisance? . 17 4 The Correlation Coefficient 19 5 More on the Likelihood Ratio Test 20 6 Concluding Comment 21 7 Further Reading 22 1 The F Test The F distribution with a; b degrees of freedom is defined to be the distribution of the ratio 2 χa=a 2 χb =b 2 2 when χa and χb are independent. Since χ2 distributions arise from sums of Gaussians, F -distributed random variables tend to arise when we are dealing with ratios of sums of Gaussians. The outstanding examples of this are ratios of variances. 1 2 1.1 F test of β1 = 0 vs. β1 6= 0 1.1 F test of β1 = 0 vs. β1 6= 0 Let's consider testing the null hypothesis β1 = 0 against the alternative β1 6= 0, in the context of the Gaussian-noise simple linear regression model. That is, we won't question, in our mathematics, whether or not the assumptions of that model hold, we'll presume that they all do, and just ask how we can tell whether β1 = 0.
    [Show full text]
  • Goodness of Fit Tests for High-Dimensional Linear Models
    Goodness of fit tests for high-dimensional linear models Rajen D. Shah∗ Peter B¨uhlmann University of Cambridge ETH Z¨urich April 8, 2017 Abstract In this work we propose a framework for constructing goodness of fit tests in both low and high-dimensional linear models. We advocate applying regression methods to the scaled residuals following either an ordinary least squares or Lasso fit to the data, and using some proxy for prediction error as the final test statistic. We call this family Residual Prediction (RP) tests. We show that simulation can be used to obtain the critical values for such tests in the low-dimensional setting, and demonstrate using both theoretical results and extensive numerical studies that some form of the parametric bootstrap can do the same when the high- dimensional linear model is under consideration. We show that RP tests can be used to test for significance of groups or individual variables as special cases, and here they compare favourably with state of the art methods, but we also argue that they can be designed to test for as diverse model misspecifications as heteroscedasticity and nonlinearity. 1 Introduction High-dimensional data, where the number of variables may greatly exceed the number of obser- vations, has become increasingly more prevalent across a variety of disciplines. While such data pose many challenges to statisticians, we now have a variety of methods for fitting models to high- dimensional data, many of which are based on the Lasso [Tibshirani, 1996]; see B¨uhlmannand van de Geer [2011] for a review of some of the developments.
    [Show full text]