Local Regression and Likelihood Clive Loader Springer Preface This book, and the associated software, have grown out of the author’s work in the field of local regression over the past several years. The book is designed to be useful for both theoretical work and in applications. Most chapters contain distinct sections introducing methodology, computing and practice, and theoretical results. The methodological and practice sections should be accessible to readers with a sound background in statistical meth- ods and in particular regression, for example at the level of Draper and Smith (1981). The theoretical sections require a greater understanding of calculus, matrix algebra and real analysis, generally at the level found in advanced undergraduate courses. Applications are given from a wide vari- ety of fields, ranging from actuarial science to sports. The extent, and relevance, of early work in smoothing is not widely appre- ciated, even within the research community. Chapter 1 attempts to redress the problem. Many ideas that are central to modern work on smoothing: local polynomials, the bias-variance trade-off, equivalent kernels, likelihood models and optimality results can be found in literature dating to the late nineteenth and early twentieth centuries. The core methodology of this book appears in Chapters 2 through 5. These chapters introduce the local regression method in univariate and multivariate settings, and extensions to local likelihood and density estima- tion. Basic theoretical results and diagnostic tools such as cross validation are introduced along the way. Examples illustrate the implementation of the methods using the locfit software. The remaining chapters discuss a variety of applications and advanced topics: classification, survival data, bandwidth selection issues, computa- vi tion and asymptotic theory. Largely, these chapters are independent of each other, so the reader can pick those of most interest. Most chapters include a short set of exercises. These include theoretical results; details of proofs; extensions of the methodology; some data analysis examples and a few research problems. But the real test for the methods is whether they provide useful answers in applications. The best exercise for every chapter is to find datasets of interest, and try the methods out! The literature on mathematical aspects of smoothing is extensive, and coverage is necessarily selective. I attempt to present results that are of most direct practical relevance. For example, theoretical motivation for standard error approximations and confidence bands is important; the reader should eventually want to know precisely what the error estimates represent, rather than simply asuming software reports the right answers (this applies to any model and software; not just local regression and loc- fit!). On the other hand, asymptotic methods for boundary correction re- ceive no coverage, since local regression provides a simpler, more intuitive and more general approach to achieve the same result. Along with the theory, we also attempt to introduce understanding of the results, along with their relevance. Examples of this include the discussion of non-identifiability of derivatives (Section 6.1) and the problem of bias estimation for confidence bands and bandwidth selectors (Chapters 9 and 10). Software Local fitting should provide a practical tool to help analyse data. This re- quires software, and an integral part of this book is locfit.Thiscanbe run either as a library within R, S and S-Plus, or as a stand-alone appli- cation. Versions of the software for both Windows and UNIX systems can be downloaded from the locfit web page, http://cm.bell-labs.com/stat/project/locfit/ Installation instructions for current versions of locfit and S-Plus are pro- vided in the appendices; updates for future versions of S-Plus will be posted on the web pages. The examples in this book use locfit in S (or S-Plus), which will be of use to many readers given the widespread availability of S within the statistics community. For readers without access to S, the recommended alternative is to use locfit with the R language, which is freely available and has a syntax very similar to S. There is also a stand-alone version, c-locfit, with its own interface and data management facilities. The in- terface allows access to almost all the facilities of locfit’s S interface, and a few additional features. An on-line example facility allows the user to obtain c-locfit code for most of the examples in this book. vii It should also be noted this book is not an introduction to S. The reader using locfit with S should already be familiar with S fundamentals, such as reading and manipulating data and initializing graphics devices. Books such as Krause and Olson (1997), Spector (1994) and Venables and Ripley (1997) cover this material, and much more. Acknowledgements Acknowledgements are many. Foremost, Bill Cleveland introduced me to the field of local fitting, and his influence will be seen in numerous places. Vladimir Katkovnik is thanked for helpful ideas and suggestions, and for providing a copy of his 1985 book. locfit has been distributed, in various forms, over the internet for sev- eral years, and feedback from numerous users has resulted in significant improvements. Kurt Hornik, David James, Brian Ripley, Dan Serachitopol and others have ported locfit to various operating systems and versions of R and S-Plus. This book was used as the basis for a graduate course at Rutgers Uni- versity in Spring 1998, and I thank Yehuda Vardi for the opportunity to teach the course, as well as the students for not complaining too loudly about the drafts inflicted upon them. Of course, writing this book and software required a flawlessly working computer system, and my system administrator Daisy Nguyen recieves the highest marks in this respect! Many of my programming sources also deserve mention. Horspool (1986) has been my usual reference for C programming. John Chambers provided S, and patiently handled my bug reports (which usually turned out as locfit bugs; not S!). Curtin University is an excellent online source for X programming (http://www.cs.curtin.edu.au/units/). This page intentionally left blank Contents 1 The Origins of Local Regression 1 1.1 The Problem of Graduation .................. 1 1.1.1 Graduation Using Summation Formulae ....... 2 1.1.2 The Bias-Variance Trade-Off ............. 7 1.2 Local Polynomial Fitting ................... 7 1.2.1 Optimal Weights .................... 8 1.3 Smoothing of Time Series ................... 10 1.4 Modern Local Regression ................... 11 1.5 Exercises ............................ 12 2 Local Regression Methods 15 2.1 The Local Regression Estimate ................ 15 2.1.1 Interpreting the Local Regression Estimate ..... 18 2.1.2 Multivariate Local Regression ............. 19 2.2 The Components of Local Regression ............. 20 2.2.1 Bandwidth ....................... 20 2.2.2 Local Polynomial Degree ............... 22 2.2.3 The Weight Function .................. 23 2.2.4 The Fitting Criterion ................. 24 2.3 Diagnostics and Goodness of Fit ............... 24 2.3.1 Residuals ........................ 25 2.3.2 Influence, Variance and Degrees of Freedom ..... 27 2.3.3 Confidence Intervals .................. 29 2.4 Model Comparison and Selection ............... 30 x Contents 2.4.1 Prediction and Cross Validation ........... 30 2.4.2 Estimation Error and CP ............... 31 2.4.3 Cross Validation Plots ................. 32 2.5 Linear Estimation ....................... 33 2.5.1 Influence, Variance and Degrees of Freedom ..... 36 2.5.2 Bias ........................... 37 2.6 Asymptotic Approximations .................. 38 2.7 Exercises ............................ 42 3 Fitting with locfit 45 3.1 Local Regression with locfit ................. 46 3.2 Customizing the Local Fit ................... 47 3.3 The Computational Model ................... 48 3.4 Diagnostics ........................... 49 3.4.1 Residuals ........................ 49 3.4.2 Cross Validation .................... 49 3.5 Multivariate Fitting and Visualization ............ 51 3.5.1 Additive Models .................... 53 3.5.2 Conditionally Parametric Models ........... 55 3.6 Exercises ............................ 57 4 Local Likelihood Estimation 59 4.1 The Local Likelihood Model .................. 59 4.2 Local Likelihood with locfit ................. 62 4.3 Diagnostics for Local Likelihood ............... 66 4.3.1 Deviance ........................ 66 4.3.2 Residuals for Local Likelihood ............ 67 4.3.3 Cross Validation and AIC ............... 68 4.3.4 Overdispersion ..................... 70 4.4 Theory for Local Likelihood Estimation ........... 72 4.4.1 Why Maximize the Local Likelihood? ........ 72 4.4.2 Local Likelihood Equations .............. 72 4.4.3 Bias, Variance and Influence ............. 74 4.5 Exercises ............................ 76 5 Density Estimation 79 5.1 Local Likelihood Density Estimation ............. 79 5.1.1 Higher Order Kernels ................. 81 5.1.2 Poisson Process Rate Estimation ........... 82 5.1.3 Discrete Data ...................... 82 5.2 Density Estimation in locfit ................. 83 5.2.1 Multivariate Density Examples ............ 86 5.3 Diagnostics for Density Estimation .............. 87 5.3.1 Residuals for Density Estimation ........... 88 5.3.2 Influence, Cross Validation and AIC ......... 90 Contents xi 5.3.3 Squared Error Methods ................ 92 5.3.4 Implementation ..................... 93 5.4 Some