Econometric Data Science: A Predictive Modeling Approach Francis X. Diebold University of Pennsylvania Version 2019.08.05 Econometric Data Science: A Predictive Modeling Approach Econometric Data Science Francis X. Diebold Copyright c 2013-2019 by Francis X. Diebold. This work is freely available for your use, but be warned: it is prelim- inary, incomplete, and evolving. It is licensed under the Creative Com- mons Attribution-NonCommercial-NoDerivatives 4.0 International License. (Briefly: I retain copyright, but you can use, copy and distribute non-commercially, so long as you give me attribution and do not modify. To view a copy of the license, go to http://creativecommons.org/licenses/by-nc-nd/ 4.0/.) In return I ask that you please cite the book whenever appropri- ate, as: \Diebold, F.X. (2019), Econometric Data Science: A Predictive Modeling Approach, Department of Economics, University of Pennsylvania, http://www.ssc.upenn.edu/~fdiebold/Textbooks.html." To my undergraduates, who continually surprise and inspire me Brief Table of Contents About the Author xix About the Cover xxi Preface xxix I Beginnings1 1 Introduction to Econometrics3 2 Graphics and Graphical Style 13 II Cross Sections 29 3 Regression Under Ideal Conditions 31 4 Misspecification and Model Selection 73 5 Non-Normality 91 6 Group Heterogeneity and Indicator Variables 113 7 Nonlinearity 121 8 Heteroskedasticity 139 9 Limited Dependent Variables 151 10 Causal Estimation 161 ix x BRIEF TABLE OF CONTENTS III Time Series 177 11 Trend and Seasonality 179 12 Serial Correlation 205 13 Structural Change 251 14 Vector Autoregression 257 15 Dynamic Heteroskedasticity 277 IV Appendices 301 A Probability and Statistics Review 303 B Construction of the Wage Datasets 315 C Some Popular Books Worth Encountering 321 Detailed Table of Contents About the Author xix About the Cover xxi Preface xxix I Beginnings1 1 Introduction to Econometrics3 1.1 Welcome..............................3 1.1.1 Who Uses Econometrics?.................3 1.1.2 What Distinguishes Econometrics?...........5 1.2 Types of Recorded Economic Data...............5 1.3 Online Information and Data..................6 1.4 Software..............................6 1.5 Tips on How to use this book..................8 1.6 Exercises, Problems and Complements............. 10 1.7 Notes................................ 11 2 Graphics and Graphical Style 13 2.1 Simple Techniques of Graphical Analysis............ 13 2.1.1 Univariate Graphics................... 14 2.1.2 Multivariate Graphics.................. 14 2.1.3 Summary and Extension................. 17 2.2 Elements of Graphical Style................... 19 2.3 U.S. Hourly Wages........................ 21 2.4 Concluding Remark........................ 22 2.5 Exercises, Problems and Complements............. 22 xi xii DETAILED TABLE OF CONTENTS 2.6 Notes................................ 26 2.7 Graphics Legend: Edward Tufte................. 27 II Cross Sections 29 3 Regression Under Ideal Conditions 31 3.1 Preliminary Graphics....................... 31 3.2 Regression as Curve Fitting................... 33 3.2.1 Bivariate, or Simple, Linear Regression......... 33 3.2.2 Multiple Linear Regression................ 36 3.2.3 Onward.......................... 37 3.3 Regression as a Probability Model................ 38 3.3.1 A Population Model and a Sample Estimator..... 38 3.3.2 Notation, Assumptions and Results: The Ideal Condi- tions............................ 39 A Bit of Matrix Notation................. 39 Assumptions: The Ideal Conditions (IC)........ 40 Results Under the IC................... 41 3.4 A Wage Equation......................... 42 3.4.1 Mean dependent var 2.342................ 45 3.4.2 S.D. dependent var .561................. 45 3.4.3 Sum squared resid 319.938................ 45 3.4.4 Log likelihood -938.236.................. 46 3.4.5 F statistic 199.626.................... 48 3.4.6 Prob(F statistic) 0.000000................ 49 3.4.7 S.E. of regression .492.................. 49 3.4.8 R-squared .232...................... 50 3.4.9 Adjusted R-squared .231................. 50 3.4.10 Akaike info criterion 1.423................ 51 3.4.11 Schwarz criterion 1.435.................. 51 3.4.12 A Bit More on AIC and SIC............... 52 3.4.13 Hannan-Quinn criter. 1.427............... 52 3.4.14 Durbin-Watson stat. 1.926................ 52 3.4.15 The Residual Scatter................... 53 3.4.16 The Residual Plot..................... 53 3.5 Least Squares and Optimal Point Prediction.......... 55 DETAILED TABLE OF CONTENTS xiii 3.6 Optimal Interval and Density Prediction............ 57 3.7 Regression Output from a Predictive Perspective........ 59 3.8 Multicollinearity......................... 60 3.8.1 Perfect and Imperfect Multicollinearity......... 61 3.9 Beyond Fitting the Conditional Mean: Quantile regression........................ 62 3.10 Exercises, Problems and Complements............. 64 3.11 Notes................................ 70 3.12 Regression's Inventor: Carl Friedrich Gauss........... 71 4 Misspecification and Model Selection 73 4.1 Information Criteria (Hard Thresholding)............ 74 4.2 Cross Validation (Hard Thresholding).............. 80 4.3 Stepwise Selection (Hard Thresholding)............. 81 4.3.1 Forward.......................... 82 4.3.2 Backward......................... 82 4.4 Bayesian Shrinkage (Soft Thresholding)............. 83 4.5 Selection and Shrinkage (Mixed Hard and Soft Thresholding). 84 4.5.1 Penalized Estimation................... 84 4.5.2 The Lasso......................... 84 4.6 Distillation: Principal Components............... 85 4.6.1 Distilling \X Variables" into Principal Components.. 85 4.6.2 Principal Components Regression............ 87 4.7 Exercises, Problems and Complements............. 87 5 Non-Normality 91 5.0.1 Results........................... 91 5.1 Assessing Normality....................... 92 5.1.1 QQ Plots......................... 92 5.1.2 Residual Sample Skewness and Kurtosis........ 93 5.1.3 The Jarque-Bera Test.................. 93 5.2 Outliers.............................. 94 5.2.1 Outlier Detection..................... 94 Graphics.......................... 94 Leave-One-Out and Leverage.............. 95 5.3 Robust Estimation........................ 95 5.3.1 Robustness Iteration................... 95 xiv DETAILED TABLE OF CONTENTS 5.3.2 Least Absolute Deviations................ 96 5.4 Wage Determination....................... 98 5.4.1 W AGE .......................... 98 5.4.2 LW AGE ......................... 98 5.5 Exercises, Problems and Complements............. 110 6 Group Heterogeneity and Indicator Variables 113 6.1 0-1 Dummy Variables....................... 113 6.2 Group Dummies in the Wage Regression............ 115 6.3 Exercises, Problems and Complements............. 117 6.4 Notes................................ 119 6.5 Dummy Variables, ANOVA, and Sir Ronald Fischer...... 120 7 Nonlinearity 121 7.1 Models Linear in Transformed Variables............ 122 7.1.1 Logarithms........................ 122 7.1.2 Box-Cox and GLM.................... 124 7.2 Intrinsically Non-Linear Models................. 125 7.2.1 Nonlinear Least Squares................. 125 7.3 Series Expansions......................... 126 7.4 A Final Word on Nonlinearity and the IC........... 127 7.5 Selecting a Non-Linear Model.................. 128 7.5.1 t and F Tests, and Information Criteria........ 128 7.5.2 The RESET Test .................... 128 7.6 Non-Linearity in Wage Determination.............. 129 7.6.1 Non-Linearity in Continuous and Discrete Variables Si- multaneously....................... 131 7.7 Exercises, Problems and Complements............. 132 8 Heteroskedasticity 139 8.1 Consequences of Heteroskedasticity for Estimation, Inference, and Prediction.......................... 139 8.2 Detecting Heteroskedasticity................... 140 8.2.1 Graphical Diagnostics.................. 140 8.2.2 Formal Tests....................... 141 The Breusch-Pagan-Godfrey Test (BPG)........ 141 White's Test........................ 142 DETAILED TABLE OF CONTENTS xv 8.3 Dealing with Heteroskedasticity................. 143 8.3.1 Adjusting Standard Errors................ 143 8.3.2 Adjusting Density Forecasts............... 144 8.4 Exercises, Problems and Complements............. 145 9 Limited Dependent Variables 151 9.1 Binary Response......................... 151 9.2 The Logit Model......................... 153 9.2.1 Logit............................ 153 9.2.2 Ordered Logit....................... 154 9.2.3 Complications....................... 155 9.3 Classification and \0-1 Forecasting"............... 156 9.4 Exercises, Problems and Complements............. 157 10 Causal Estimation 161 10.1 Predictive Modeling vs. Causal Estimation........... 162 10.1.1 Predictive Modeling and P-Consistency......... 163 10.1.2 Causal Estimation and T-Consistency.......... 163 10.1.3 Correlation vs. Causality, and P-Consistency vs. T- Consistency........................ 164 10.2 Reasons for Failure of IC2.1................... 165 10.2.1 Omitted Variables (\Confounders")........... 165 10.2.2 Misspecified Functional Form.............. 166 10.2.3 Measurement Error.................... 166 10.2.4 Simultaneity........................ 167 10.3 Confronting Failures of IC2.1.................. 168 10.3.1 Controling for Omitted Variables............ 168 10.3.2 Instrumental Variables.................. 169 10.3.3 Randomized Controlled Trials (RCT's)......... 170 10.3.4 Regression Discontinuity Designs (RDD's)....... 171 10.3.5 Propensity-Score Matching................ 172 10.3.6