Shrinkage Estimation of Rate Statistics Arxiv:1810.07654V1 [Stat.AP] 17 Oct
Total Page:16
File Type:pdf, Size:1020Kb
Published, CS-BIGS 7(1):14-25 http://www.csbigs.fr Shrinkage estimation of rate statistics Einar Holsbø Department of Computer Science, UiT — The Arctic University of Norway Vittorio Perduca Laboratory of Applied Mathematics MAP5, Université Paris Descartes This paper presents a simple shrinkage estimator of rates based on Bayesian methods. Our focus is on crime rates as a motivating example. The estimator shrinks each town’s observed crime rate toward the country-wide average crime rate according to town size. By realistic simulations we confirm that the proposed estimator outperforms the maximum likelihood estimator in terms of global risk. We also show that it has better coverage properties. Keywords : Official statistics, crime rates, inference, Bayes, shrinkage, James-Stein estimator, Monte-Carlo simulations. 1. Introduction that most of the best schools—according to a variety of performance measures—were small. 1.1. Two counterintuitive random phenomena As it turns out, there is nothing special about small schools except that they are small: their It is a classic result in statistics that the smaller over-representation among the best schools is the sample, the more variable the sample mean. a consequence of their more variable perfor- The result is due to Abraham de Moivre and it mance, which is counterbalanced by their over- tells us that the standard deviation of the mean p representation among the worst schools. The is sx¯ = s/ n, where n is the sample size and s observed superiority of small schools was sim- the standard deviation of the random variable ply a statistical fluke. of interest. Although the equation is very sim- ple, its practical implications are not intuitive. arXiv:1810.07654v1 [stat.AP] 17 Oct 2018 People have erroneous intuitions about the laws of Galton(1886) first described another stochastic chance, argue Tversky and Kahneman in their mechanism that is dangerous to ignore. Galton famous paper about the law of small numbers observed that children of tall (or short) par- (Tversky and Kahneman, 1971). ents usually grow up to be not quite as tall (or short), i.e. closer to average height. Today Serious consequences can follow from small- we know this phenomenon as regression to sample inference ignoring deMoivre’s equation. the mean, and we will find it wherever we Wainer(2007) provides a notorious example: in find variation. Imagine a coach who berates a the late 1990s and early 2000s private and pub- runner who had an unusually slow lap time lic institutions provided massive funding to and finds that, indeed, the next lap is faster. small schools. This was due to the observation The coach, who always berates slow runners, -2- has not had the opportunity to realize that the date xi (Stein, 1956). In other words, the maxi- next lap is very likely to be faster no matter mum likelihood estimate is inadmissible. Stein what. As long as there is variability in lap time showed this by introducing a lower-risk estima- we will some times see unusually slow laps tor that biases or shrinks, the xis toward zero. that we can do nothing about and make no James and Stein(1961) introduced an improved inference from. In this case too do our intu- shrinkage estimator, which we will see below. itions about the laws of chance fail us. People, Efron and Morris(1973) show a similar result including scientists, make the mistake of ig- and a similar estimator for shrinking toward noring regression all the time. Mathematically the pooled mean. There are many successful regression to the mean is as simple as imperfect applications of shrinkage estimation, see for correlation between instances. instance the examples from Morris(1983). The common theme is a setting where the statisti- 1.2. These phenomena in official statistics cian wants to estimate many similar variable quantities. The small-schools example is egregious be- cause it led to wasteful public spending. The statistics themselves were probably fine, but 1.4. An almost-Bayesian estimator their interpretation was not careful enough. In this case study we consider the official Nor- Such summary statistics are often presented wegian crime report counts. We assume that without regard for uncertainty. For instance, in a given year the number of crimes reported every year Statistics Norway (ssb.no), the cen- in town i, denoted ki, corresponds to the num- tral bureau of statistics in Norway, presents ber of criminal events in this town. We further crime report counts. The media usually reports assume that each inhabitant can at most be these numbers as rates and inform us that some reported for one crime a year. Our goal is small town that few people know about is the to estimate the crime probability qi: probability most criminal in the country. Often the focus is that a person will commit a crime in this town. on violent crimes. Figure1 below shows these The obvious estimator is the maximum likeli- rates for 2016. Not knowing de Moivre’s result hood estimate (MLE) for a binomial proportion it might be striking to observe that many of the ˆ k qi = i/ni, where ni is the population of town i. towns with the highest rates are small towns. Similarly, not knowing regression it might be striking to observe that, on average, towns with The MLE binomial model rests on an assump- a high rate in one year will have a lower one in tion that inhabitants commit crimes indepen- any other year, see Figure2 below. These are dently according to an identical crime proba- unavoidable stochastic phenomena. Thus there bility. There are reasons to believe that this is is reason to believe that we should somehow not the case. The desperately poor might be adjust our expectations about these numbers. more prone to stealing than the middle class We will see below that such an adjustment also professional. There is a weaker assumption makes statistical sense. called exchangeability that says that individuals are similar but not identical. More precisely we assume that their joint criminal behavior (some 1.3. Shrinkage estimation number of zeros and ones) does not depend on There is an astonishing decision-theoretic re- knowing who the individuals are (the order of sult due to Charles Stein: suppose that we wish the zeros and ones). It is an important theorem to estimate k ≥ 3 parameters q1, ... , qk and ob- in Bayesian inference, due to De Finetti, that serve k independent measurements, x1 ... xk, a sequence of exchangeable variables are inde- such that xi ∼ N(qi, 1). There is an estimator pendent and identically distributed conditional of qi that has uniformly lower risk, in terms on an unknown parameter qi that is distributed of total quadratic loss, than the obvious candi- according to an a priori (or prior) distribution -3- f (qi) (Spiegelhalter et al., 2004). In the bino- as prior. Our likelihood for an observed num- mial sense, qi has the remarkable property that ber of crime reports follows a binomial dis- it is the long-run frequency with which crimes tribution. It is a classic exercise to show that occur regardless of the i.i.d. assumption; the the posterior distribution of qi is then also a prior precisely reflects our opinion about this beta distribution. The problem remains how limit. By virtue of De Finetti’s theorem, the to choose the parameters for the prior. On the exchangeability assumption justifies the intro- idea that a given town is probably not that dif- duction of the unknown parameter qi in a bino- ferent from all the other towns, we will simply mial model for ki, so long as we take the prior pool the observed crime rates for all towns and into account. fit a beta distribution to this ensemble by the method of moments. To make an argument with priors is to make a Bayesian argument. Shrinkage is implicit in Under squared error loss, the posterior mean Bayesian inference: observed data gets pulled as point estimate minimizes Bayes risk. The toward the prior (and indeed the prior is pulled posterior mean serves as our shrinkage esti- ˆs ˆs toward the data likelihood). We propose an mate, qi , for qi. We will see that qi in effect ˆs ˆ almost Bayesian shrinkage estimator, qi , that shrinks the observed crime rate qi toward the ¯ 1 ˆ accounts for the variability due to population country-wide mean q = ∑ m qi by taking into size. Our estimator is almost Bayesian because account the size of town i. we do not treat the prior very formally, as will be clear below. Bayesian inference allows for intuitive uncer- tainty intervals. In contrast to a classical fre- In a Bayesian argument we treat qi as random. quentist confidence interval, which can be The statistician specifies a prior distribution tricky to interpret, we can say that qi lies within f (qi) for the parameter that reflects her knowl- the Bayesian credible interval with a certain edge (and uncertainty) about qi. As in the fre- probability. This probability is necessarily sub- quentist setting, she then selects a parametric jective, as the prior distribution is subjective. model for the data given the parameters, which We will conduct simulations to compare the allows her to compute the likelihood f (xjqi). coverage properties of our estimator to the clas- Inference about qi consists of computing its sical asymptotic confidence interval. posterior distribution by Bayes’ theorem: 1.5. Resources f (xjqi) f (qi) f (qijx) = R . f (xjqi) f (qi) dqi This case-study is written with a pedagogi- cal purpose in mind, and can be used by ad- vanced undergraduate and beginning gradu- There are various assessments we could make ate students in statistics as a tutorial around shrinkage estimation and Bayesian methods.