The search for an organizing physical framework for statistics Colin LaMont A dissertation submitted in partial fulfillment of the requirements for the degree of Doctor of Philosophy University of Washington 2018 Reading Committee: Paul Wiggins, Chair Mathias Drton Jason Detwiler Program Authorized to Offer Degree: Physics ⃝c Copyright 2018 Colin LaMont University of Washington Abstract The search for an organizing physical framework for statistics Colin LaMont Chair of the Supervisory Committee: Paul Wiggins Department of Biophysics Theories of statistical analysis remain in conflict and contradiction. But nature reveals an elegant and coherent formulation of statistics in the thermal properties of physical sys- tems. Demanding that a viable statistical theory share the properties of a viable physical theory|observer independence and coordinate invariance|resolves outstanding controver- sies in statistical model selection. Furthermore, by using constructions taken directly from thermodynamics, a predictive approach to model selection can be reconciled with a Bayesian approach to parameter uncertainty. This approach also solves the longstanding problem of the undetermined Bayesian prior. TABLE OF CONTENTS Page List of Figures ....................................... vi Chapter 1: Introduction ................................ 1 1.0.1 Method of maximum likelihood ..................... 1 1.0.2 A unified framework for statistics? ................... 4 1.1 Model selection .................................. 5 1.1.1 The change point problem ........................ 6 1.1.2 Model selection and predictivity. ..................... 7 1.2 Uncertainty on the parameters .......................... 8 1.3 AIC vs. BIC? ................................... 9 1.4 Layout and contributions of the thesis ..................... 11 Chapter 2: Change-point information criterion supplement ............. 13 2.1 Preliminaries ................................... 15 2.2 Information criterion for change-point analysis ................. 20 2.3 The relation between frequentist and information-based approach ...... 27 2.4 Applications .................................... 30 2.5 Discussion ..................................... 30 Chapter 3: Predictive model selection ......................... 32 3.1 Introduction to Model selection ......................... 33 3.2 Information-based inference ........................... 34 3.2.1 Information criteria ............................ 35 3.2.2 The Akaike Information Criterion .................... 37 3.3 Complexity Landscapes .............................. 38 3.3.1 The finite-sample-size complexity of regular models .......... 38 3.3.2 Singular models .............................. 42 i 3.3.3 Constrained models ............................ 43 3.4 Frequentist Information Criterion (QIC) .................... 44 3.4.1 Measuring model selection performance ................. 45 3.5 Applications of QIC ................................ 45 3.5.1 Small sample size and the step-size analysis .............. 46 3.5.2 Anomalously large complexity and the Fourier regression model ... 49 3.5.3 Anomalously small complexity and the exponential mixture model .. 55 3.6 Discussion ..................................... 57 3.6.1 QIC subsumes extends both AIC and AICc .............. 58 3.6.2 Asymptotic bias of the QIC complexity ................. 59 3.6.3 Advantages of QIC ............................ 60 3.6.4 Conclusion ................................. 62 Chapter 4: From postdiction to prediction ...................... 64 4.1 Introduction .................................... 65 4.1.1 A simple example of the Lindley paradox ................ 66 4.2 Data partition ................................... 68 4.2.1 The definition of frequentism and Bayesian paradigms ......... 68 4.2.2 Prior information content ........................ 69 4.2.3 The Bayesian cross entropy ....................... 70 4.2.4 Pseudo-Bayes factors ........................... 71 4.2.5 Information Criteria ........................... 71 4.2.6 Decision rules and resolution ....................... 73 4.3 The Lindley paradox ............................... 73 4.3.1 Classical Bayes and the Bartlett paradox ................ 74 4.3.2 Minimal training set and Lindley Paradox ............... 75 4.3.3 Frequentist prescription and AIC .................... 76 4.3.4 Log evidence versus AIC ......................... 79 4.3.5 When will a Lindley paradox occur? .................. 79 4.4 Discussion ..................................... 80 4.4.1 A canonical Bayesian perspective on the Lindley paradox ....... 80 4.4.2 Circumventing the Lindley paradox ................... 81 4.4.3 Loss of resolution ............................. 82 ii 4.4.4 The multiple comparisons problem ................... 83 4.4.5 Statistical significance does not imply scientific significance ...... 84 4.4.6 Conclusion ................................. 84 Chapter 5: Thermodynamic correspondence ..................... 87 5.1 Contributions ................................... 87 5.2 Defining the correspondence ........................... 89 5.2.1 Application of thermodynamic identities ................ 90 5.3 Results ....................................... 90 5.3.1 Models ................................... 90 5.3.2 Thermodynamic potentials ........................ 92 5.3.3 The principle of indifference ....................... 94 5.3.4 A generalized principle of indifference .................. 96 5.4 Applications .................................... 97 5.4.1 Learning Capacity ............................ 98 5.4.2 Generalized principle of indifference ................... 108 5.4.3 Inference .................................. 113 5.5 Discussion ..................................... 118 5.5.1 Learning capacity ............................. 118 5.5.2 The generalized principle of indifference ................ 120 5.5.3 Comparison with existing approaches and novel features ....... 124 5.5.4 Conclusion ................................. 126 Chapter 6: Conclusion ................................. 127 6.1 The thermodynamics of inference ........................ 128 6.2 Unanswered questions .............................. 130 Appendix A: QIC for change-point algorithms and Brownian bridges ......... 147 A.1 Type I errors ................................... 147 A.1.1 Asymptotic form of the complexity function .............. 147 Appendix B: QIC calculations .............................. 154 B.1 Complexity Calculations ............................. 154 B.1.1 Exponential families ........................... 154 iii B.1.2 Modified-Centered-Gaussian distribution ................ 155 B.1.3 The component selection model ..................... 155 B.1.4 n-cone ................................... 156 B.1.5 Fourier Regression nesting complexity .................. 157 B.1.6 L1 Constraint ............................... 158 B.1.7 Curvature and QIC unbiasedness under non-realizability ....... 158 B.1.8 Approximations for marginal likelihood ................. 159 B.1.9 Seasonal dependence of the neutrino intensity ............. 160 Appendix C: Lindley Paradox .............................. 162 C.1 Calculation of Complexity ............................ 162 C.2 Definitions and Calculations ........................... 163 C.2.1 Volume of a distribution ......................... 163 C.2.2 Showing I = I0 to zeroth order ...................... 164 C.2.3 Significance level implied by a data partition .............. 164 C.3 Other Methods .................................. 164 C.3.1 Other Predictive Estimators ....................... 165 C.3.2 Data-Validated Posterior and Double use of Data ........... 165 C.3.3 Fractional Bayes Factor ......................... 166 C.4 Efficiency and correct models .......................... 166 Appendix D: Thermodynamics Supplement ....................... 169 D.1 Supplemental results ............................... 169 D.1.1 Definitions of information, cross entropy, Fisher information matrix . 169 D.1.2 An alternate correspondence ....................... 170 D.1.3 Finite difference is equivalent to cross validation ............ 170 D.1.4 Jeffreys prior is proportional to GPI prior in the large-sample-size limit 171 D.1.5 Reparametrization invariance of thermodynamic functions and the GPI prior .................................... 172 D.1.6 Effective temperature of confinement .................. 173 D.1.7 A Bayesian re-interpretation ...................... 173 D.2 Methods ...................................... 174 D.2.1 Computation of learning capacity .................... 174 D.2.2 Direct computation of GPI prior ..................... 174 iv D.2.3 Computation of the free energy using a sufficient statistic ....... 175 D.2.4 Computation of the GPI prior using a recursive approximation .... 176 D.3 Details of applications .............................. 177 D.3.1 Normal model with unknown mean and informative prior ....... 177 D.3.2 Normal model with unknown mean ................... 178 D.3.3 Normal model with unknown discrete mean .............. 179 D.3.4 Normal model unknown mean and variance .............. 181 D.3.5 Exponential model ............................ 182 D.3.6 Uniform distribution ........................... 183 D.3.7 Poisson stoichiometry problem ...................... 184 v LIST OF FIGURES Figure Number Page 1.1 Schematic of a statistical model A statistical model consists of a space of observed data, a space of parameters, and maps between the two. Can- didate distributions describing the data are identified with points in a model or