DOCSLIB.ORG

Linear Regression: Goodness of Fit and Model Selection

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Linear Regression: Goodness of Fit and Model Selection Linear Regression: Goodness of Fit and Model Selection 1 Goodness of Fit I Goodness of fit measures for linear regression are attempts to understand how well a model fits a given set of data. I Models almost never describe the process that generated a dataset exactly I Models approximate reality I However, even models that approximate reality can be used to draw useful inferences or to prediction future observations I ’All Models are wrong, but some are useful’ - George Box 2 Goodness of Fit I We have seen how to check the modelling assumptions of linear regression: I checking the linearity assumption I checking for outliers I checking the normality assumption I checking the distribution of the residuals does not depend on the predictors I These are essential qualitative checks of goodness of fit 3 Sample Size I When making visual checks of data for goodness of fit is important to consider sample size I From a multiple regression model with 2 predictors: I On the left is a histogram of the residuals I On the right is residual vs predictor plot for each of the two predictors 4 Sample Size I The histogram doesn’t look normal but there are only 20 datapoint I We should not expect a better visual fit I Inferences from the linear model should be valid 5 Outliers I Often (particularly when a large dataset is large): I the majority of the residuals will satisfy the model checking assumption I a small number of residuals will violate the normality assumption: they will be very big or very small I Outliers are often generated by a process distinct from those which we are primarily interested in. i.e. the process generating the relationships between the response and the predictors I e.g. Outliers are often generated by measurement or experimental errors I In these circumstances, rather than reject the linear model and search for a simpler one, it is usually better to remove the outliers from the dataset. 6 Outlier Detection I Outliers can be detected graphically using the qqnorm function: 7 Automatic Outlier Detection I Automated outlier detection is built into R I Apply the plot command to an R linear model object: > plot(lm(y∼ x)) 8 Goodness of Fit I Visual checks are important methods for checking the quality of the fit of a linear model to a dataset I However they are qualitative, quantitative measure of goodness of fit are also important I Quantitative measures allow us to compare the goodness of fit of different models to the same dataset 9 Residual Sum of Squares (SSE) I The residual sum of squares is defined as n 2 SSE = ri Xi n 2 = (Yi - αˆ - βˆ x) Xi I This is a simple measure of goodness of fit I The smaller the SSE the better the fit of the model I Note SSE depends on n the number of data points, so it cannot be used to compare the quality of fit of models fit to datasets of different size. 10 R2 2 I You may have noticed that the R value is routinely given in the R software output 2 I R is a measure of variance explained by the predictors I We will now see how to define and interpret this measure. 11 R2 I Consider two equations models: EYi = α + β × xi (1) EYi = α (2) 2 I R of the first model is defined as: sum of squared residuals from model (2) 1 - sum of squared residuals from model (1) I If better model (2) is at explaining the variation compared to model (1). The closer R2 will be to 1. 12 R2 Interpretation I Consider the following two residual plots: I In the first case, the predictor x does not seem to have a large effect, while in the second, the linear model fits well. 13 R2 Properties 2 I R is a number between 0 and 1 2 I The R for a simple linear regression is the squared correlation between x and Y 2 I The larger R the better the fit 2 I If you add a predictor to a multiple linear regression R always goes up 2 I This means R is inappropriate to compare the quality of fit of models with different numbers of predictors I More complex models always fit better 14 Model Selection I Recall there are two main reasons for fitting a statistical model 1 Scientific Inference: estimating an interpretable parameter 2 Prediction: if you give me a new x1, x2, ...xp can I predict the value of the corresponding Y without seeing it. I Models for making scientific inferences are not normally chosen using statistical ‘black box’ model selection procedures. Usually the choice of model(s) depends on the scientific question and knowledge of the data generating process. I Model selection methods are used primarily for finding good models for making predictions. 15 Averaging Over Models I Optimal predictions often come from averaging over predictions from multiple models. I In this lecture we will concentrate on methods for finding a single optimal model amongst a set of possibilities. 16 Complex Models are not Good for Prediction I Problem: find a model using the current dataset that is going to be good at predicting a new observation. I As we’ve seen we can move to a model with improved goodness of fit of by adding a new predictor to the current model, so its easy to find a model which fits well to a given dataset I But really complex models aren’t necessarily good for predicting new observtions, even if they are a good fit to the current dataset. 17 Model Parsimony I Measures of model parsimony take into account goodness of fit to the data and model complexity. I If two models have the same number of parameters the one with the better goodness of fit will be the more parsimonious. I If two models have the same goodness of fit (rarely happens) the model with the fewer parameters will be the more parsimonious. I More parsimonious models should give better predictions (on average). 18 Measures of Model Parsimony I There are many measures of model parsimony. I We will concentrate on AIC and BIC. 19 AIC I The formula for AIC is n log SSE + 2p I SSE is the residual sum of squares, this is a goodness of fit measure I p is the number of parameters of the model (number of regression coefficients). I Smaller values of AIC correspond to more parsimonious models. I AIC tends to be liberal (i.e. can add in too many predictors, overfit) 20 BIC I The formula for BIC in linear regression is 2 log SSE + p log(n) I n is the sample size. I The complexity penalty is stronger than that for AIC. I Smaller values of BIC correspond to more parsimonious models. I BIC tends to be conservative (i.e. it requires quite a bit of evidence before it will include a predictor) 21 Number of Possible Regression Models p I If we have p predictors we can build 2 possible models. p 2 I e.g. p = 2 the 2 = 2 = 4 possible linear regression models have regression equations: EYi= α EYi = α + β1X1 EYi = α + β2X2 EYi= α + β1X1 + β2X2 I The blue model is called the empty model. I The red model is called the saturated (or full) model. 22 All Subsets Selection I All subsets selection is the simplest model search algorithm. 1 Choose a model parsimony criterion. 2 Fit each of the 2p models and compute the criterion. 3 Rank the models by the criterion and choose the most parsimonious. I On modern computers this is doable providing p is not much larger than a number in the late teens. 220 ≈ 1 million. 23 Forward Selection I This algorithm can be run with any model selection criterion. I Start (usually) with the empty model as the current model. I Iterate the following: 1 Fit all the models you can generate by augmenting the current model by one variable. 2 If none of the models fitted in 1 is ranked better by the model selection criterion than the current model terminate the algorithm and output the current model. 3 Update the current model with the model fitted in 1 that is ranked best by the model selection criterion. p I This fits at most p(p + 1)=2 models (faster than 2 ). I May not always find the best model, once a variable is in 24 the model it can’t be removed. Backward Elimination I This algorithm can be run with any model selection criterion. I Start (usually) with the saturated model. I Iterate the following: 1 Fit all the models you can generate by reducing the current model by one variable. 2 If none of the models fitted in 1 is ranked better by the model selection criterion than the current model terminate the algorithm and output the current model. 3 Update the current model with the model fitted in 1 that is ranked best by the model selection criterion. p I This fits at most p(p + 1)=2 models (faster than 2 ). I May not always find the best model, once a variable is out 25 of the model it can’t be returned. Combined Forward and Backwards Selection I This algorithm can be run with any model selection criterion. I Starting point not so important. I Iterate the following: 1 Fit all the models you can generate by augmenting or reducing the current model by one variable.
Load more

Recommended publications
  • Goodness of Fit: What Do We Really Want to Know? I
    PHYSTAT2003, SLAC, Stanford, California, September 8-11, 2003 Goodness of Fit: What Do We Really Want to Know? I. Narsky California Institute of Technology, Pasadena, CA 91125, USA Definitions of the goodness-of-fit problem are discussed. A new method for estimation of the goodness of fit using distance to nearest neighbor is described. Performance of several goodness-of-fit methods is studied for time-dependent CP asymmetry measurements of sin(2β). 1. INTRODUCTION of the test is then defined as the probability of ac- cepting the null hypothesis given it is true, and the The goodness-of-fit problem has recently attracted power of the test, 1 − αII , is defined as the probabil- attention from the particle physics community. In ity of rejecting the null hypothesis given the alterna- modern particle experiments, one often performs an tive is true. Above, αI and αII denote Type I and unbinned likelihood fit to data. The experimenter Type II errors, respectively. An ideal hypothesis test then needs to estimate how accurately the fit function is uniformly most powerful (UMP) because it gives approximates the observed distribution. A number of the highest power among all possible tests at the fixed methods have been used to solve this problem in the confidence level. In most realistic problems, it is not past [1], and a number of methods have been recently possible to find a UMP test and one has to consider proposed [2, 3] in the physics literature. various tests with acceptable power functions. For binned data, one typically applies a χ2 statistic There is a long-standing controversy about the con- to estimate the fit quality.
    [Show full text]
  • Model Selection for Optimal Prediction in Statistical Machine Learning
    Model Selection for Optimal Prediction in Statistical Machine Learning Ernest Fokou´e Introduction science with several ideas from cognitive neuroscience and At the core of all our modern-day advances in artificial in- psychology to inspire the creation, invention, and discov- telligence is the emerging field of statistical machine learn- ery of abstract models that attempt to learn and extract pat- ing (SML). From a very general perspective, SML can be terns from the data. One could think of SML as a field thought of as a field of mathematical sciences that com- of science dedicated to building models endowed with bines mathematics, probability, statistics, and computer the ability to learn from the data in ways similar to the ways humans learn, with the ultimate goal of understand- Ernest Fokou´eis a professor of statistics at Rochester Institute of Technology. His ing and then mastering our complex world well enough email address is [email protected]. to predict its unfolding as accurately as possible. One Communicated by Notices Associate Editor Emilie Purvine. of the earliest applications of statistical machine learning For permission to reprint this article, please contact: centered around the now ubiquitous MNIST benchmark [email protected]. task, which consists of building statistical models (also DOI: https://doi.org/10.1090/noti2014 FEBRUARY 2020 NOTICES OF THE AMERICAN MATHEMATICAL SOCIETY 155 known as learning machines) that automatically learn and Theoretical Foundations accurately recognize handwritten digits from the United It is typical in statistical machine learning that a given States Postal Service (USPS). A typical deployment of an problem will be solved in a wide variety of different ways.
    [Show full text]
  • Scalable Model Selection for Spatial Additive Mixed Modeling: Application to Crime Analysis
    Scalable model selection for spatial additive mixed modeling: application to crime analysis Daisuke Murakami1,2,*, Mami Kajita1, Seiji Kajita1 1Singular Perturbations Co. Ltd., 1-5-6 Risona Kudan Building, Kudanshita, Chiyoda, Tokyo, 102-0074, Japan 2Department of Statistical Data Science, Institute of Statistical Mathematics, 10-3 Midori-cho, Tachikawa, Tokyo, 190-8562, Japan * Corresponding author (Email: [email protected]) Abstract: A rapid growth in spatial open datasets has led to a huge demand for regression approaches accommodating spatial and non-spatial effects in big data. Regression model selection is particularly important to stably estimate flexible regression models. However, conventional methods can be slow for large samples. Hence, we develop a fast and practical model-selection approach for spatial regression models, focusing on the selection of coefficient types that include constant, spatially varying, and non-spatially varying coefficients. A pre-processing approach, which replaces data matrices with small inner products through dimension reduction dramatically accelerates the computation speed of model selection. Numerical experiments show that our approach selects the model accurately and computationally efficiently, highlighting the importance of model selection in the spatial regression context. Then, the present approach is applied to open data to investigate local factors affecting crime in Japan. The results suggest that our approach is useful not only for selecting factors influencing crime risk but also for predicting crime events. This scalable model selection will be key to appropriately specifying flexible and large-scale spatial regression models in the era of big data. The developed model selection approach was implemented in the R package spmoran. Keywords: model selection; spatial regression; crime; fast computation; spatially varying coefficient modeling 1.
    [Show full text]
  • Model Selection Techniques: an Overview
    Model Selection Techniques An overview ©ISTOCKPHOTO.COM/GREMLIN Jie Ding, Vahid Tarokh, and Yuhong Yang n the era of big data, analysts usually explore various statis- following different philosophies and exhibiting varying per- tical models or machine-learning methods for observed formances. The purpose of this article is to provide a compre- data to facilitate scientific discoveries or gain predictive hensive overview of them, in terms of their motivation, large power. Whatever data and fitting procedures are employed, sample performance, and applicability. We provide integrated Ia crucial step is to select the most appropriate model or meth- and practically relevant discussions on theoretical properties od from a set of candidates. Model selection is a key ingredi- of state-of-the-art model selection approaches. We also share ent in data analysis for reliable and reproducible statistical our thoughts on some controversial views on the practice of inference or prediction, and thus it is central to scientific stud- model selection. ies in such fields as ecology, economics, engineering, finance, political science, biology, and epidemiology. There has been a Why model selection long history of model selection techniques that arise from Vast developments in hardware storage, precision instrument researches in statistics, information theory, and signal process- manufacturing, economic globalization, and so forth have ing. A considerable number of methods has been proposed, generated huge volumes of data that can be analyzed to extract useful information. Typical statistical inference or machine- learning procedures learn from and make predictions on data Digital Object Identifier 10.1109/MSP.2018.2867638 Date of publication: 13 November 2018 by fitting parametric or nonparametric models (in a broad 16 IEEE SIGNAL PROCESSING MAGAZINE | November 2018 | 1053-5888/18©2018IEEE sense).
    [Show full text]
  • Least Squares After Model Selection in High-Dimensional Sparse Models.” DOI:10.3150/11-BEJ410SUPP
    Bernoulli 19(2), 2013, 521–547 DOI: 10.3150/11-BEJ410 Least squares after model selection in high-dimensional sparse models ALEXANDRE BELLONI1 and VICTOR CHERNOZHUKOV2 1100 Fuqua Drive, Durham, North Carolina 27708, USA. E-mail: [email protected] 250 Memorial Drive, Cambridge, Massachusetts 02142, USA. E-mail: [email protected] In this article we study post-model selection estimators that apply ordinary least squares (OLS) to the model selected by first-step penalized estimators, typically Lasso. It is well known that Lasso can estimate the nonparametric regression function at nearly the oracle rate, and is thus hard to improve upon. We show that the OLS post-Lasso estimator performs at least as well as Lasso in terms of the rate of convergence, and has the advantage of a smaller bias. Remarkably, this performance occurs even if the Lasso-based model selection “fails” in the sense of missing some components of the “true” regression model. By the “true” model, we mean the best s-dimensional approximation to the nonparametric regression function chosen by the oracle. Furthermore, OLS post-Lasso estimator can perform strictly better than Lasso, in the sense of a strictly faster rate of convergence, if the Lasso-based model selection correctly includes all components of the “true” model as a subset and also achieves sufficient sparsity. In the extreme case, when Lasso perfectly selects the “true” model, the OLS post-Lasso estimator becomes the oracle estimator. An important ingredient in our analysis is a new sparsity bound on the dimension of the model selected by Lasso, which guarantees that this dimension is at most of the same order as the dimension of the “true” model.
    [Show full text]
  • Chapter 2 Simple Linear Regression Analysis the Simple
    Chapter 2 Simple Linear Regression Analysis The simple linear regression model We consider the modelling between the dependent and one independent variable. When there is only one independent variable in the linear regression model, the model is generally termed as a simple linear regression model. When there are more than one independent variables in the model, then the linear model is termed as the multiple linear regression model. The linear model Consider a simple linear regression model yX01 where y is termed as the dependent or study variable and X is termed as the independent or explanatory variable. The terms 0 and 1 are the parameters of the model. The parameter 0 is termed as an intercept term, and the parameter 1 is termed as the slope parameter. These parameters are usually called as regression coefficients. The unobservable error component accounts for the failure of data to lie on the straight line and represents the difference between the true and observed realization of y . There can be several reasons for such difference, e.g., the effect of all deleted variables in the model, variables may be qualitative, inherent randomness in the observations etc. We assume that is observed as independent and identically distributed random variable with mean zero and constant variance 2 . Later, we will additionally assume that is normally distributed. The independent variables are viewed as controlled by the experimenter, so it is considered as non-stochastic whereas y is viewed as a random variable with Ey()01 X and Var() y 2 . Sometimes X can also be a random variable.
    [Show full text]
  • A Study of Some Issues of Goodness-Of-Fit Tests for Logistic Regression Wei Ma
    Florida State University Libraries Electronic Theses, Treatises and Dissertations The Graduate School 2018 A Study of Some Issues of Goodness-of-Fit Tests for Logistic Regression Wei Ma Follow this and additional works at the DigiNole: FSU's Digital Repository. For more information, please contact [email protected] FLORIDA STATE UNIVERSITY COLLEGE OF ARTS AND SCIENCES A STUDY OF SOME ISSUES OF GOODNESS-OF-FIT TESTS FOR LOGISTIC REGRESSION By WEI MA A Dissertation submitted to the Department of Statistics in partial fulfillment of the requirements for the degree of Doctor of Philosophy 2018 Copyright c 2018 Wei Ma. All Rights Reserved. Wei Ma defended this dissertation on July 17, 2018. The members of the supervisory committee were: Dan McGee Professor Co-Directing Dissertation Qing Mai Professor Co-Directing Dissertation Cathy Levenson University Representative Xufeng Niu Committee Member The Graduate School has verified and approved the above-named committee members, and certifies that the dissertation has been approved in accordance with university requirements. ii ACKNOWLEDGMENTS First of all, I would like to express my sincere gratitude to my advisors, Dr. Dan McGee and Dr. Qing Mai, for their encouragement, continuous support of my PhD study, patient guidance. I could not have completed this dissertation without their help and immense knowledge. I have been extremely lucky to have them as my advisors. I would also like to thank the rest of my committee members: Dr. Cathy Levenson and Dr. Xufeng Niu for their support, comments and help for my thesis. I would like to thank all the staffs and graduate students in my department.
    [Show full text]
  • Model Selection and Estimation in Regression with Grouped Variables
    J. R. Statist. Soc. B (2006) 68, Part 1, pp. 49–67 Model selection and estimation in regression with grouped variables Ming Yuan Georgia Institute of Technology, Atlanta, USA and Yi Lin University of Wisconsin—Madison, USA [Received November 2004. Revised August 2005] Summary. We consider the problem of selecting grouped variables (factors) for accurate pre- diction in regression. Such a problem arises naturally in many practical situations with the multi- factor analysis-of-variance problem as the most important and well-known example. Instead of selecting factors by stepwise backward elimination, we focus on the accuracy of estimation and consider extensions of the lasso, the LARS algorithm and the non-negative garrotte for factor selection. The lasso, the LARS algorithm and the non-negative garrotte are recently proposed regression methods that can be used to select individual variables. We study and propose effi- cient algorithms for the extensions of these methods for factor selection and show that these extensions give superior performance to the traditional stepwise backward elimination method in factor selection problems.We study the similarities and the differences between these methods. Simulations and real examples are used to illustrate the methods. Keywords: Analysis of variance; Lasso; Least angle regression; Non-negative garrotte; Piecewise linear solution path 1. Introduction In many regression problems we are interested in finding important explanatory factors in pre- dicting the response variable, where each explanatory factor may be represented by a group of derived input variables. The most common example is the multifactor analysis-of-variance (ANOVA) problem, in which each factor may have several levels and can be expressed through a group of dummy variables.
    [Show full text]
  • Testing Goodness Of
    Dr. Wolfgang Rolke University of Puerto Rico - Mayaguez CERN Phystat Seminar 1 Problem statement Hypothesis testing Chi-square Methods based on empirical distribution function Other tests Power studies Running several tests Tests for multi-dimensional data 2 ➢ We have a probability model ➢ We have data from an experiment ➢ Does the data agree with the probability model? 3 Good Model? Or maybe needs more? 4 F: cumulative distribution function 퐻0: 퐹 = 퐹0 Usually more useful: 퐻0: 퐹 ∊ ℱ0 ℱ0 a family of distributions, indexed by parameters. 5 Type I error: reject true null hypothesis Type II error: fail to reject false null hypothesis A: HT has to have a true type I error probability no higher than the nominal one (α) B: probability of committing the type II error (β) should be as low as possible (subject to A) Historically A was achieved either by finding an exact test or having a large enough sample. p value = probability to reject true null hypothesis when repeating the experiment and observing value of test statistic or something even less likely. If method works p-value has uniform distribution. 6 Note above: no alternative hypothesis 퐻1 Different problem: 퐻0: 퐹 = 푓푙푎푡 vs 퐻0: 퐹 = 푙푖푛푒푎푟 → model selection Usually better tests: likelihood ratio test, F tests, BIC etc. Easy to confuse: all GOF papers do power studies, those need specific alternative. Our question: is F a good enough model for data? We want to guard against any alternative. 7 Not again … Actually no, GOF equally important to both (everybody has a likelihood) Maybe more so for Bayesians, no non- parametric methods.
    [Show full text]
  • Measures of Fit for Logistic Regression Paul D
    Paper 1485-2014 SAS Global Forum Measures of Fit for Logistic Regression Paul D. Allison, Statistical Horizons LLC and the University of Pennsylvania ABSTRACT One of the most common questions about logistic regression is “How do I know if my model fits the data?” There are many approaches to answering this question, but they generally fall into two categories: measures of predictive power (like R-square) and goodness of fit tests (like the Pearson chi-square). This presentation looks first at R-square measures, arguing that the optional R-squares reported by PROC LOGISTIC might not be optimal. Measures proposed by McFadden and Tjur appear to be more attractive. As for goodness of fit, the popular Hosmer and Lemeshow test is shown to have some serious problems. Several alternatives are considered. INTRODUCTION One of the most frequent questions I get about logistic regression is “How can I tell if my model fits the data?” Often the questioner is expressing a genuine interest in knowing whether a model is a good model or a not-so-good model. But a more common motivation is to convince someone else--a boss, an editor, or a regulator--that the model is OK. There are two very different approaches to answering this question. One is to get a statistic that measures how well you can predict the dependent variable based on the independent variables. I’ll refer to these kinds of statistics as measures of predictive power. Typically, they vary between 0 and 1, with 0 meaning no predictive power whatsoever and 1 meaning perfect predictions.
    [Show full text]
  • Model Selection for Production System Via Automated Online Experiments
    Model Selection for Production System via Automated Online Experiments Zhenwen Dai Praveen Ravichandran Spotify Spotify [email protected] [email protected] Ghazal Fazelnia Ben Carterette Mounia Lalmas-Roelleke Spotify Spotify Spotify [email protected] [email protected] [email protected] Abstract A challenge that machine learning practitioners in the industry face is the task of selecting the best model to deploy in production. As a model is often an intermedi- ate component of a production system, online controlled experiments such as A/B tests yield the most reliable estimation of the effectiveness of the whole system, but can only compare two or a few models due to budget constraints. We propose an automated online experimentation mechanism that can efficiently perform model se- lection from a large pool of models with a small number of online experiments. We derive the probability distribution of the metric of interest that contains the model uncertainty from our Bayesian surrogate model trained using historical logs. Our method efficiently identifies the best model by sequentially selecting and deploying a list of models from the candidate set that balance exploration-exploitation. Using simulations based on real data, we demonstrate the effectiveness of our method on two different tasks. 1 Introduction Evaluating the effect of individual changes to machine learning (ML) systems such as choice of algorithms, features, etc., is the key to growth in many internet services and industrial applications. Practitioners are faced with the decision of choosing one model from several candidates to deploy in production. This can be viewed as a model selection problem. Classical model selection paradigms such as cross-validation consider ML models in isolation and focus on selecting the model with the best predictive power on unseen data.
    [Show full text]
  • Goodness of Fit of a Straight Line to Data
    5. A linear trend. 10.4 The Least Squares Regression Line LEARNING OBJECTIVES 1. To learn how to measure how well a straight line fits a collection of data. 2. To learn how to construct the least squares regression line, the straight line that best fits a collection of data. 3. To learn the meaning of the slope of the least squares regression line. 4. To learn how to use the least squares regression line to estimate the response variable y in terms of the predictor variablex. Goodness of Fit of a Straight Line to Data Once the scatter diagram of the data has been drawn and the model assumptions described in the previous sections at least visually verified (and perhaps the correlation coefficient r computed to quantitatively verify the linear trend), the next step in the analysis is to find the straight line that best fits the data. We will explain how to measure how well a straight line fits a collection of points by examining how well the line y=12x−1 fits the data set Saylor URL: http://www.saylor.org/books Saylor.org 503 To each point in the data set there is associated an “error,” the positive or negative vertical distance from the point to the line: positive if the point is above the line and negative if it is below theline. The error can be computed as the actual y-value of the point minus the y-value yˆ that is “predicted” by inserting the x-value of the data point into the formula for the line: error at data point (x,y)=(true y)−(predicted y)=y−yˆ The computation of the error for each of the five points in the data set is shown in Table 10.1 "The Errors in Fitting Data with a Straight Line".
    [Show full text]