DOCSLIB.ORG

Lecture 2. Estimation, Bias, and Mean Squared Error Estimators Mean Squared Error

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Lecture 2. Estimation, Bias, and Mean Squared Error Estimators Mean Squared Error 0. 2. Estimation and bias 2.1. Estimators Estimators Suppose that X ,..., X are iid, each with pdf/pmf f (x θ), θ unknown. 1 n X | We aim to estimate θ by a statistic, ie by a function T of the data. Lecture 2. Estimation, bias, and mean squared error If X = x = (x1,..., xn) then our estimate is θˆ = T (x) (does not involve θ). Then T (X) is our estimator of θ, and is a rv since it inherits random fluctuations from those of X. The distribution of T = T (X) is called its sampling distribution. Example Let X1,..., Xn be iid N(µ, 1). 1 A possible estimator for µ is T (X) = n Xi . For any particular observed sample x, our estimate is T (x) = 1 x . P n i We have T (X) N(µ, 1/n). ∼ P Lecture 2. Estimation, bias, and mean squared error 1 (1–7) Lecture 2. Estimation, bias, and mean squared error 2 (1–7) 2. Estimation and bias 2.2. Bias 2. Estimation and bias 2.3. Mean squared error Mean squared error If θˆ = T (X) is an estimator of θ, then the bias of θˆ is the difference between its Recall that an estimator T is a function of the data, and hence is a random expectation and the ’true’ value: i.e. quantity. Roughly, we prefer estimators whose sampling distributions “cluster more closely” around the true value of θ, whatever that value might be. ˆ ˆ bias(θ) = Eθ(θ) θ. − Definition 2.1 An estimator T (X) is unbiased for θ if θT (X) = θ for all θ, otherwise it is ˆ ˆ 2 E The mean squared error (mse) of an estimator θ is Eθ (θ θ) . biased. − In the above example, (T ) = µ so T is unbiased for µ. Eµ For an unbiased estimator, the mse is just the variance. In general ˆ 2 ˆ ˆ ˆ 2 Eθ (θ θ) = Eθ (θ Eθθ + Eθθ θ) − − − ˆ ˆ 2 ˆ 2 ˆ ˆ ˆ [Notation note: when a parameter subscript is used with an expectation or = Eθ(θ Eθθ) + Eθ(θ) θ + 2 Eθ(θ) θ Eθ θ Eθθ variance, it refers to the parameter that is being conditioned on. i.e. the − − − − = var (θˆ) + bias2(θˆ), expectation or variance will be a function of the subscript] θ ˆ ˆ where bias(θ) = Eθ(θ) θ. − [NB: sometimes it can be preferable to have a biased estimator with a low variance - this is sometimes known as the ’bias-variance tradeoff’.] Lecture 2. Estimation, bias, and mean squared error 3 (1–7) Lecture 2. Estimation, bias, and mean squared error 4 (1–7) 2. Estimation and bias 2.4. Example: Alternative estimators for Binomial mean 2. Estimation and bias 2.4. Example: Alternative estimators for Binomial mean Example: Alternative estimators for Binomial mean Suppose X Binomial(n, θ), and we want to estimate θ. ∼ The standard estimator is TU = X /n, which is Unbiassed since Eθ(TU ) = nθ/n = θ. 2 TU has variance varθ(TU ) = varθ(X )/n = θ(1 θ)/n. X +1 − X 1 Consider an alternative estimator TB = n+2 = w n + (1 w) 2 , where 1 − w = n/(n + 2). TB is a weighted average of X /n and 2 . e.g. if X is 8 successes out of 10 trials, we would estimate the underlying success probability as T (8) = 9/12 = 0.75, rather than 0.8. nθ+1 1 Then Eθ(TB ) θ = θ = (1 w) θ , and so it is biased. − n+2 − − 2 − varθ (X ) 2 varθ(TB ) = 2 = w θ(1 θ)/n. (n+2) − 2 Now mse(TU ) = varθ(TU ) + bias (TU ) = θ(1 θ)/n. So the biased estimator has smaller MSE in much of the range of θ − 2 2 2 2 1 TB may be preferable if we do not think θ is near 0 or 1. mse(TB ) = varθ(TB ) + bias (TB ) = w θ(1 θ)/n + (1 w) 2 θ − − − So our prior judgement about θ might affect our choice of estimator. Will see more of this when we come to Bayesian methods,. Lecture 2. Estimation, bias, and mean squared error 5 (1–7) Lecture 2. Estimation, bias, and mean squared error 6 (1–7) 2. Estimation and bias 2.5. Why unbiasedness is not necessarily so great Why unbiasedness is not necessarily so great Suppose X Poisson(λ), and for some reason (which escapes me for the ∼ 2 2λ moment), you want to estimate θ = [P(X = 0)] = e− . Then any unbiassed estimator T (X ) must satisfy Eθ(T (X )) = θ, or equivalently x λ ∞ λ 2λ Eλ(T (X )) = e− T (x) = e− . x! x=0 X The only function T that can satisfy this equation is T (X ) = ( 1)X [coefficients of polynomial must match]. − 2λ Thus the only unbiassed estimator estimates e− to be 1 if X is even, -1 if X is odd. This is not sensible. Lecture 2. Estimation, bias, and mean squared error 7 (1–7).
Load more

Recommended publications
  • Lecture 12 Robust Estimation
    Lecture 12 Robust Estimation Prof. Dr. Svetlozar Rachev Institute for Statistics and Mathematical Economics University of Karlsruhe Financial Econometrics, Summer Semester 2007 Prof. Dr. Svetlozar Rachev Institute for Statistics and MathematicalLecture Economics 12 Robust University Estimation of Karlsruhe Copyright These lecture-notes cannot be copied and/or distributed without permission. The material is based on the text-book: Financial Econometrics: From Basics to Advanced Modeling Techniques (Wiley-Finance, Frank J. Fabozzi Series) by Svetlozar T. Rachev, Stefan Mittnik, Frank Fabozzi, Sergio M. Focardi,Teo Jaˇsic`. Prof. Dr. Svetlozar Rachev Institute for Statistics and MathematicalLecture Economics 12 Robust University Estimation of Karlsruhe Outline I Robust statistics. I Robust estimators of regressions. I Illustration: robustness of the corporate bond yield spread model. Prof. Dr. Svetlozar Rachev Institute for Statistics and MathematicalLecture Economics 12 Robust University Estimation of Karlsruhe Robust Statistics I Robust statistics addresses the problem of making estimates that are insensitive to small changes in the basic assumptions of the statistical models employed. I The concepts and methods of robust statistics originated in the 1950s. However, the concepts of robust statistics had been used much earlier. I Robust statistics: 1. assesses the changes in estimates due to small changes in the basic assumptions; 2. creates new estimates that are insensitive to small changes in some of the assumptions. I Robust statistics is also useful to separate the contribution of the tails from the contribution of the body of the data. Prof. Dr. Svetlozar Rachev Institute for Statistics and MathematicalLecture Economics 12 Robust University Estimation of Karlsruhe Robust Statistics I Peter Huber observed, that robust, distribution-free, and nonparametrical actually are not closely related properties.
    [Show full text]
  • Should We Think of a Different Median Estimator?
    Comunicaciones en Estad´ıstica Junio 2014, Vol. 7, No. 1, pp. 11–17 Should we think of a different median estimator? ¿Debemos pensar en un estimator diferente para la mediana? Jorge Iv´an V´eleza Juan Carlos Correab [email protected] [email protected] Resumen La mediana, una de las medidas de tendencia central m´as populares y utilizadas en la pr´actica, es el valor num´erico que separa los datos en dos partes iguales. A pesar de su popularidad y aplicaciones, muchos desconocen la existencia de dife- rentes expresiones para calcular este par´ametro. A continuaci´on se presentan los resultados de un estudio de simulaci´on en el que se comparan el estimador cl´asi- co y el propuesto por Harrell & Davis (1982). Mostramos que, comparado con el estimador de Harrell–Davis, el estimador cl´asico no tiene un buen desempe˜no pa- ra tama˜nos de muestra peque˜nos. Basados en los resultados obtenidos, se sugiere promover la utilizaci´on de un mejor estimador para la mediana. Palabras clave: mediana, cuantiles, estimador Harrell-Davis, simulaci´on estad´ısti- ca. Abstract The median, one of the most popular measures of central tendency widely-used in the statistical practice, is often described as the numerical value separating the higher half of the sample from the lower half. Despite its popularity and applica- tions, many people are not aware of the existence of several formulas to estimate this parameter. We present the results of a simulation study comparing the classic and the Harrell-Davis (Harrell & Davis 1982) estimators of the median for eight continuous statistical distributions.
    [Show full text]
  • Lecture 13: Simple Linear Regression in Matrix Format
    11:55 Wednesday 14th October, 2015 See updates and corrections at http://www.stat.cmu.edu/~cshalizi/mreg/ Lecture 13: Simple Linear Regression in Matrix Format 36-401, Section B, Fall 2015 13 October 2015 Contents 1 Least Squares in Matrix Form 2 1.1 The Basic Matrices . .2 1.2 Mean Squared Error . .3 1.3 Minimizing the MSE . .4 2 Fitted Values and Residuals 5 2.1 Residuals . .7 2.2 Expectations and Covariances . .7 3 Sampling Distribution of Estimators 8 4 Derivatives with Respect to Vectors 9 4.1 Second Derivatives . 11 4.2 Maxima and Minima . 11 5 Expectations and Variances with Vectors and Matrices 12 6 Further Reading 13 1 2 So far, we have not used any notions, or notation, that goes beyond basic algebra and calculus (and probability). This has forced us to do a fair amount of book-keeping, as it were by hand. This is just about tolerable for the simple linear model, with one predictor variable. It will get intolerable if we have multiple predictor variables. Fortunately, a little application of linear algebra will let us abstract away from a lot of the book-keeping details, and make multiple linear regression hardly more complicated than the simple version1. These notes will not remind you of how matrix algebra works. However, they will review some results about calculus with matrices, and about expectations and variances with vectors and matrices. Throughout, bold-faced letters will denote matrices, as a as opposed to a scalar a. 1 Least Squares in Matrix Form Our data consists of n paired observations of the predictor variable X and the response variable Y , i.e., (x1; y1);::: (xn; yn).
    [Show full text]
  • Bias, Mean-Square Error, Relative Efficiency
    3 Evaluating the Goodness of an Estimator: Bias, Mean-Square Error, Relative Efficiency Consider a population parameter ✓ for which estimation is desired. For ex- ample, ✓ could be the population mean (traditionally called µ) or the popu- lation variance (traditionally called σ2). Or it might be some other parame- ter of interest such as the population median, population mode, population standard deviation, population minimum, population maximum, population range, population kurtosis, or population skewness. As previously mentioned, we will regard parameters as numerical charac- teristics of the population of interest; as such, a parameter will be a fixed number, albeit unknown. In Stat 252, we will assume that our population has a distribution whose density function depends on the parameter of interest. Most of the examples that we will consider in Stat 252 will involve continuous distributions. Definition 3.1. An estimator ✓ˆ is a statistic (that is, it is a random variable) which after the experiment has been conducted and the data collected will be used to estimate ✓. Since it is true that any statistic can be an estimator, you might ask why we introduce yet another word into our statistical vocabulary. Well, the answer is quite simple, really. When we use the word estimator to describe a particular statistic, we already have a statistical estimation problem in mind. For example, if ✓ is the population mean, then a natural estimator of ✓ is the sample mean. If ✓ is the population variance, then a natural estimator of ✓ is the sample variance. More specifically, suppose that Y1,...,Yn are a random sample from a population whose distribution depends on the parameter ✓.The following estimators occur frequently enough in practice that they have special notations.
    [Show full text]
  • A Joint Central Limit Theorem for the Sample Mean and Regenerative Variance Estimator*
    Annals of Operations Research 8(1987)41-55 41 A JOINT CENTRAL LIMIT THEOREM FOR THE SAMPLE MEAN AND REGENERATIVE VARIANCE ESTIMATOR* P.W. GLYNN Department of Industrial Engineering, University of Wisconsin, Madison, W1 53706, USA and D.L. IGLEHART Department of Operations Research, Stanford University, Stanford, CA 94305, USA Abstract Let { V(k) : k t> 1 } be a sequence of independent, identically distributed random vectors in R d with mean vector ~. The mapping g is a twice differentiable mapping from R d to R 1. Set r = g(~). A bivariate central limit theorem is proved involving a point estimator for r and the asymptotic variance of this point estimate. This result can be applied immediately to the ratio estimation problem that arises in regenerative simulation. Numerical examples show that the variance of the regenerative variance estimator is not necessarily minimized by using the "return state" with the smallest expected cycle length. Keywords and phrases Bivariate central limit theorem,j oint limit distribution, ratio estimation, regenerative simulation, simulation output analysis. 1. Introduction Let X = {X(t) : t I> 0 } be a (possibly) delayed regenerative process with regeneration times 0 = T(- 1) ~< T(0) < T(1) < T(2) < .... To incorporate regenerative sequences {Xn: n I> 0 }, we pass to the continuous time process X = {X(t) : t/> 0}, where X(0 = X[t ] and [t] is the greatest integer less than or equal to t. Under quite general conditions (see Smith [7] ), *This research was supported by Army Research Office Contract DAAG29-84-K-0030. The first author was also supported by National Science Foundation Grant ECS-8404809 and the second author by National Science Foundation Grant MCS-8203483.
    [Show full text]
  • In This Segment, We Discuss a Little More the Mean Squared Error
    MITOCW | MITRES6_012S18_L20-04_300k In this segment, we discuss a little more the mean squared error. Consider some estimator. It can be any estimator, not just the sample mean. We can decompose the mean squared error as a sum of two terms. Where does this formula come from? Well, we know that for any random variable Z, this formula is valid. And if we let Z be equal to the difference between the estimator and the value that we're trying to estimate, then we obtain this formula here. The expected value of our random variable Z squared is equal to the variance of that random variable plus the square of its mean. Let us now rewrite these two terms in a more suggestive way. We first notice that theta is a constant. When you add or subtract the constant from a random variable, the variance does not change. So this term is the same as the variance of theta hat. This quantity here, we will call it the bias of the estimator. It tells us whether theta hat is systematically above or below than the unknown parameter theta that we're trying to estimate. And using this terminology, this term here is just equal to the square of the bias. So the mean squared error consists of two components, and these capture different aspects of an estimator's performance. Let us see what they are in a concrete setting. Suppose that we're estimating the unknown mean of some distribution, and that our estimator is the sample mean. In this case, the mean squared error is the variance, which we know to be sigma squared over n, plus the bias term.
    [Show full text]
  • Minimum Mean Squared Error Model Averaging in Likelihood Models
    Statistica Sinica 26 (2016), 809-840 doi:http://dx.doi.org/10.5705/ss.202014.0067 MINIMUM MEAN SQUARED ERROR MODEL AVERAGING IN LIKELIHOOD MODELS Ali Charkhi1, Gerda Claeskens1 and Bruce E. Hansen2 1KU Leuven and 2University of Wisconsin, Madison Abstract: A data-driven method for frequentist model averaging weight choice is developed for general likelihood models. We propose to estimate the weights which minimize an estimator of the mean squared error of a weighted estimator in a local misspecification framework. We find that in general there is not a unique set of such weights, meaning that predictions from multiple model averaging estimators might not be identical. This holds in both the univariate and multivariate case. However, we show that a unique set of empirical weights is obtained if the candidate models are appropriately restricted. In particular a suitable class of models are the so-called singleton models where each model only includes one parameter from the candidate set. This restriction results in a drastic reduction in the computational cost of model averaging weight selection relative to methods which include weights for all possible parameter subsets. We investigate the performance of our methods in both linear models and generalized linear models, and illustrate the methods in two empirical applications. Key words and phrases: Frequentist model averaging, likelihood regression, local misspecification, mean squared error, weight choice. 1. Introduction We study a focused version of frequentist model averaging where the mean squared error plays a central role. Suppose we have a collection of models S 2 S to estimate a population quantity µ, this is the focus, leading to a set of estimators fµ^S : S 2 Sg.
    [Show full text]
  • Calibration: Calibrate Your Model
    DUE TODAYCOMPUTER FILES AND QUESTIONS for Assgn#6 Assignment # 6 Steady State Model Calibration: Calibrate your model. If you want to conduct a transient calibration, talk with me first. Perform calibration using UCODE. Be sure your report addresses global, graphical, and spatial measures of error as well as common sense. Consider more than one conceptual model and compare the results. Remember to make a prediction with your calibrated models and evaluate confidence in your prediction. Be sure to save your files because you will want to use them later in the semester. Suggested Calibration Report Outline Title Introduction describe the system to be calibrated (use portions of your previous report as appropriate) Observations to be matched in calibration type of observations locations of observations observed values uncertainty associated with observations explain specifically what the observation will be matched to in the model Calibration Procedure Evaluation of calibration residuals parameter values quality of calibrated model Calibrated model results Predictions Uncertainty associated with predictions Problems encountered, if any Comparison with uncalibrated model results Assessment of future work needed, if appropriate Summary/Conclusions Summary/Conclusions References submit the paper as hard copy and include it in your zip file of model input and output submit the model files (input and output for both simulations) in a zip file labeled: ASSGN6_LASTNAME.ZIP Calibration (Parameter Estimation, Optimization, Inversion, Regression) adjusting parameter values, boundary conditions, model conceptualization, and/or model construction until the model simulation matches field observations We calibrate because 1. the field measurements are not accurate reflecti ons of the model scale properties, and 2.
    [Show full text]
  • 1 Estimation and Beyond in the Bayes Universe
    ISyE8843A, Brani Vidakovic Handout 7 1 Estimation and Beyond in the Bayes Universe. 1.1 Estimation No Bayes estimate can be unbiased but Bayesians are not upset! No Bayes estimate with respect to the squared error loss can be unbiased, except in a trivial case when its Bayes’ risk is 0. Suppose that for a proper prior ¼ the Bayes estimator ±¼(X) is unbiased, Xjθ (8θ)E ±¼(X) = θ: This implies that the Bayes risk is 0. The Bayes risk of ±¼(X) can be calculated as repeated expectation in two ways, θ Xjθ 2 X θjX 2 r(¼; ±¼) = E E (θ ¡ ±¼(X)) = E E (θ ¡ ±¼(X)) : Thus, conveniently choosing either EθEXjθ or EX EθjX and using the properties of conditional expectation we have, θ Xjθ 2 θ Xjθ X θjX X θjX 2 r(¼; ±¼) = E E θ ¡ E E θ±¼(X) ¡ E E θ±¼(X) + E E ±¼(X) θ Xjθ 2 θ Xjθ X θjX X θjX 2 = E E θ ¡ E θ[E ±¼(X)] ¡ E ±¼(X)E θ + E E ±¼(X) θ Xjθ 2 θ X X θjX 2 = E E θ ¡ E θ ¢ θ ¡ E ±¼(X)±¼(X) + E E ±¼(X) = 0: Bayesians are not upset. To check for its unbiasedness, the Bayes estimator is averaged with respect to the model measure (Xjθ), and one of the Bayesian commandments is: Thou shall not average with respect to sample space, unless you have Bayesian design in mind. Even frequentist agree that insisting on unbiasedness can lead to bad estimators, and that in their quest to minimize the risk by trading off between variance and bias-squared a small dosage of bias can help.
    [Show full text]
  • 11. Parameter Estimation
    11. Parameter Estimation Chris Piech and Mehran Sahami May 2017 We have learned many different distributions for random variables and all of those distributions had parame- ters: the numbers that you provide as input when you define a random variable. So far when we were working with random variables, we either were explicitly told the values of the parameters, or, we could divine the values by understanding the process that was generating the random variables. What if we don’t know the values of the parameters and we can’t estimate them from our own expert knowl- edge? What if instead of knowing the random variables, we have a lot of examples of data generated with the same underlying distribution? In this chapter we are going to learn formal ways of estimating parameters from data. These ideas are critical for artificial intelligence. Almost all modern machine learning algorithms work like this: (1) specify a probabilistic model that has parameters. (2) Learn the value of those parameters from data. Parameters Before we dive into parameter estimation, first let’s revisit the concept of parameters. Given a model, the parameters are the numbers that yield the actual distribution. In the case of a Bernoulli random variable, the single parameter was the value p. In the case of a Uniform random variable, the parameters are the a and b values that define the min and max value. Here is a list of random variables and the corresponding parameters. From now on, we are going to use the notation q to be a vector of all the parameters: Distribution Parameters Bernoulli(p) q = p Poisson(l) q = l Uniform(a,b) q = (a;b) Normal(m;s 2) q = (m;s 2) Y = mX + b q = (m;b) In the real world often you don’t know the “true” parameters, but you get to observe data.
    [Show full text]
  • Bayes Estimator Recap - Example
    Recap Bayes Risk Consistency Summary Recap Bayes Risk Consistency Summary . Last Lecture . Biostatistics 602 - Statistical Inference Lecture 16 • What is a Bayes Estimator? Evaluation of Bayes Estimator • Is a Bayes Estimator the best unbiased estimator? . • Compared to other estimators, what are advantages of Bayes Estimator? Hyun Min Kang • What is conjugate family? • What are the conjugate families of Binomial, Poisson, and Normal distribution? March 14th, 2013 Hyun Min Kang Biostatistics 602 - Lecture 16 March 14th, 2013 1 / 28 Hyun Min Kang Biostatistics 602 - Lecture 16 March 14th, 2013 2 / 28 Recap Bayes Risk Consistency Summary Recap Bayes Risk Consistency Summary . Recap - Bayes Estimator Recap - Example • θ : parameter • π(θ) : prior distribution i.i.d. • X1, , Xn Bernoulli(p) • X θ fX(x θ) : sampling distribution ··· ∼ | ∼ | • π(p) Beta(α, β) • Posterior distribution of θ x ∼ | • α Prior guess : pˆ = α+β . Joint fX(x θ)π(θ) π(θ x) = = | • Posterior distribution : π(p x) Beta( xi + α, n xi + β) | Marginal m(x) | ∼ − • Bayes estimator ∑ ∑ m(x) = f(x θ)π(θ)dθ (Bayes’ rule) | α + x x n α α + β ∫ pˆ = i = i + α + β + n n α + β + n α + β α + β + n • Bayes Estimator of θ is ∑ ∑ E(θ x) = θπ(θ x)dθ | θ Ω | ∫ ∈ Hyun Min Kang Biostatistics 602 - Lecture 16 March 14th, 2013 3 / 28 Hyun Min Kang Biostatistics 602 - Lecture 16 March 14th, 2013 4 / 28 Recap Bayes Risk Consistency Summary Recap Bayes Risk Consistency Summary . Loss Function Optimality Loss Function Let L(θ, θˆ) be a function of θ and θˆ.
    [Show full text]
  • An Analysis of Random Design Linear Regression
    An Analysis of Random Design Linear Regression Daniel Hsu1,2, Sham M. Kakade2, and Tong Zhang1 1Department of Statistics, Rutgers University 2Department of Statistics, Wharton School, University of Pennsylvania Abstract The random design setting for linear regression concerns estimators based on a random sam- ple of covariate/response pairs. This work gives explicit bounds on the prediction error for the ordinary least squares estimator and the ridge regression estimator under mild assumptions on the covariate/response distributions. In particular, this work provides sharp results on the \out-of-sample" prediction error, as opposed to the \in-sample" (fixed design) error. Our anal- ysis also explicitly reveals the effect of noise vs. modeling errors. The approach reveals a close connection to the more traditional fixed design setting, and our methods make use of recent ad- vances in concentration inequalities (for vectors and matrices). We also describe an application of our results to fast least squares computations. 1 Introduction In the random design setting for linear regression, one is given pairs (X1;Y1);:::; (Xn;Yn) of co- variates and responses, sampled from a population, where each Xi are random vectors and Yi 2 R. These pairs are hypothesized to have the linear relationship > Yi = Xi β + i for some linear map β, where the i are noise terms. The goal of estimation in this setting is to find coefficients β^ based on these (Xi;Yi) pairs such that the expected prediction error on a new > 2 draw (X; Y ) from the population, measured as E[(X β^ − Y ) ], is as small as possible.
    [Show full text]