Statistical Science 2011, Vol. 26, No. 1, 1–9 DOI: 10.1214/10-STS337 c Institute of Mathematical Statistics, 2011 Statistical Inference: The Big Picture1 Robert E. Kass Abstract. Statistics has moved beyond the frequentist-Bayesian con- troversies of the past. Where does this leave our ability to interpret re- sults? I suggest that a philosophy compatible with statistical practice, labeled here statistical pragmatism, serves as a foundation for infer- ence. Statistical pragmatism is inclusive and emphasizes the assump- tions that connect statistical models with observed data. I argue that introductory courses often mischaracterize the process of statistical in- ference and I propose an alternative “big picture” depiction. Key words and phrases: Bayesian, confidence, frequentist, statistical education, statistical pragmatism, statistical significance. 1. INTRODUCTION interpretations of posterior probability in terms of subjective belief, or confidence in terms of long-run The protracted battle for the foundations of statis- frequency, give students a limited and sometimes tics, joined vociferously by Fisher, Jeffreys, Neyman, confusing view of the nature of statistical inference. Savage and many disciples, has been deeply illumi- When used to introduce the expression of uncer- nating, but it has left statistics without a philoso- tainty based on a random sample, these caricatures phy that matches contemporary attitudes. Because forfeit an opportunity to articulate a fundamental each camp took as its goal exclusive ownership of inference, each was doomed to failure. We have all, attitude of statistical practice. or nearly all, moved past these old debates, yet our Most modern practitioners have, I think, an open- textbook explanations have not caught up with the minded view about alternative modes of inference, eclecticism of statistical practice. but are acutely aware of theoretical assumptions and The difficulties go both ways. Bayesians have de- the many ways they may be mistaken. I would sug- nied the utility of confidence and statistical signifi- gest that it makes more sense to place in the center cance, attempting to sweep aside the obvious success of our logical framework the match or mismatch of of these concepts in applied work. Meanwhile, for theoretical assumptions with the real world of data. arXiv:1106.2895v2 [stat.OT] 22 Jun 2011 their part, frequentists have ignored the possibility This, it seems to me, is the common ground that of inference about unique events despite their ubiq- Bayesian and frequentist statistics share; it is more uitous occurrence throughout science. Furthermore, fundamental than either paradigm taken separately; and as we strive to foster widespread understanding Robert E. Kass is Professor, Department of Statistics, of statistical reasoning, it is more important for be- Center for the Neural Basis of Cognition and Machine ginning students to appreciate the role of theoret- Learning Department, Carnegie Mellon University, ical assumptions than for them to recite correctly Pittsburgh, Pennsylvania 15213, USA e-mail: the long-run interpretation of confidence intervals. [email protected]. With the hope of prodding our discipline to right 1Discussed in 10.1214/11-STS337C, 10.1214/11-STS337A, a lingering imbalance, I attempt here to describe the 10.1214/11-STS337D and 10.1214/11-STS337B; rejoinder at dominant contemporary philosophy of statistics. 10.1214/11-STS337REJ. This is an electronic reprint of the original article 2. STATISTICAL PRAGMATISM published by the Institute of Mathematical Statistics in Statistical Science, 2011, Vol. 26, No. 1, 1–9. This I propose to call this modern philosophy statisti- reprint differs from the original in pagination and cal pragmatism. I think it is based on the following typographic detail. attitudes: 1 2 R. E. KASS 1. Confidence, statistical significance, and posterior probability are all valuable inferential tools. 2. Simple chance situations, where counting argu- ments may be based on symmetries that generate equally likely outcomes (six faces on a fair die; 52 cards in a shuffled deck), supply basic intuitions about probability. Probability may be built up to important but less immediately intuitive situ- ations using abstract mathematics, much the way Fig. 1. The big picture of statistical inference. real numbers are defined abstractly based on in- Statistical procedures are abstractly defined in tuitions coming from fractions. Probability is use- terms of mathematics but are used, in conjunc- tion with scientific models and methods, to explain fully calibrated in terms of fair bets: another way observable phenomena. This picture emphasizes the hypothet- to say the probability of rolling a 3 with a fair ical link between variation in data and its description using die is 1/6 is that 5 to 1 odds against rolling a 3 statistical models. would be a fair bet. 3. Long-run frequencies are important mathemati- conclusions most often concern scientific models cally, interpretively, and pedagogically. However, (or theories), so that their “real world” implica- it is possible to assign probabilities to unique tions (involving new data) are somewhat indirect events, including rolling a 3 with a fair die or (the new data will involve new and different ex- having a confidence interval cover the true mean, periments). without considering long-run frequency. Long-run The statistical models in Figure 1 could involve frequencies may be regarded as consequences of large function spaces or other relatively weak prob- the law of large numbers rather than as part of abilistic assumptions. Careful consideration of the the definition of probability or confidence. connection between models and data is a core com- 4. Similarly, the subjective interpretation of poste- ponent of both the art of statistical practice and rior probability is important as a way of under- the science of statistical methodology. The purpose standing Bayesian inference, but it is not funda- of Figure 1 is to shift the grounds for discussion. mental to its use: in reporting a 95% posterior Note, in particular, that data should not be con- interval one need not make a statement such as, fused with random variables. Random variables live “My personal probability of this interval covering in the theoretical world. When we say things like, the mean is 0.95.” “Let us assume the data are normally distributed” 5. Statistical inferences of all kinds use statistical and we proceed to make a statistical inference, we models, which embody theoretical assumptions. do not need to take these words literally as asserting As illustrated in Figure 1, like scientific models, that the data form a random sample. Instead, this statistical models exist in an abstract framework; kind of language is a convenient and familiar short- to distinguish this framework from the real world hand for the much weaker assertion that, for our inhabited by data we may call it a “theoretical specified purposes, the variability of the data is ade- world.” Random variables, confidence intervals, quately consistent with variability that would occur and posterior probabilities all live in this theo- in a random sample. This linguistic amenity is used retical world. When we use a statistical model to routinely in both frequentist and Bayesian frame- make a statistical inference we implicitly assert works. Historically, the distinction between data and that the variation exhibited by data is captured random variables, the match of the model to the reasonably well by the statistical model, so that data, was set aside, to be treated as a separate topic the theoretical world corresponds reasonably well apart from the foundations of inference. But once to the real world. Conclusions are drawn by ap- the data themselves were considered random vari- plying a statistical inference technique, which is ables, the frequentist-Bayesian debate moved into a theoretical construct, to some real data. Fig- the theoretical world: it became a debate about the ure 1 depicts the conclusions as straddling the best way to reason from random variables to infer- theoretical and real worlds. Statistical inferences ences about parameters. This was consistent with may have implications for the real world of new developments elsewhere. In other parts of science, observable phenomena, but in scientific contexts, the distinction between quantities to be measured STATISTICAL INFERENCE 3 (A) (B) Fig. 2. (A) BARS fits to a pair of peri-stimulus time histograms displaying neural firing rate of a particular neuron under two alternative experimental conditions. (B) The two BARS fits are overlaid for ease of comparison. and their theoretical counterparts within a mathe- would report perception with probability p = 0.5. matical theory can be relegated to a different sub- Because the data reported by Hecht et al. involved ject—to a theory of errors. In statistics, we do not fairly large samples, we would obtain essentially the have that luxury, and it seems to me important, same answer if instead we applied Bayesian methods from a pragmatic viewpoint, to bring to center stage to get an interval having 95% posterior probability. the identification of models with data. The purpose But how should such an interval be interpreted? of doing so is that it provides different interpreta- A more recent example comes from DiMatteo, Gen- tions of both frequentist and Bayesian inference, in- ovese and Kass (2001), who illustrated a new non- terpretations which, I believe, are closer to the atti- parametric regression method