Lecture 2. Estimation, Bias, and Mean Squared Error Estimators Mean Squared Error

0. 2. Estimation and bias 2.1. Estimators Estimators

Suppose that X ,..., X are iid, each with pdf/pmf f (x θ), θ unknown. 1 n X | We aim to estimate θ by a statistic, ie by a function T of the data.

Lecture 2. Estimation, bias, and mean squared error If X = x = (x1,..., xn) then our estimate is θˆ = T (x) (does not involve θ). Then T (X) is our estimator of θ, and is a rv since it inherits random ﬂuctuations from those of X. The distribution of T = T (X) is called its sampling distribution. Example

Let X1,..., Xn be iid N(µ, 1). 1 A possible estimator for µ is T (X) = n Xi . For any particular observed sample x, our estimate is T (x) = 1 x . P n i We have T (X) N(µ, 1/n). ∼ P

Lecture 2. Estimation, bias, and mean squared error 1 (1–7) Lecture 2. Estimation, bias, and mean squared error 2 (1–7)

2. Estimation and bias 2.2. Bias 2. Estimation and bias 2.3. Mean squared error Mean squared error

If θˆ = T (X) is an estimator of θ, then the bias of θˆ is the diﬀerence between its Recall that an estimator T is a function of the data, and hence is a random expectation and the ’true’ value: i.e. quantity. Roughly, we prefer estimators whose sampling distributions “cluster more closely” around the true value of θ, whatever that value might be. ˆ ˆ bias(θ) = Eθ(θ) θ. − Deﬁnition 2.1

An estimator T (X) is unbiased for θ if θT (X) = θ for all θ, otherwise it is ˆ ˆ 2 E The mean squared error (mse) of an estimator θ is Eθ (θ θ) . biased. − In the above example, (T ) = µ so T is unbiased for µ. Eµ For an unbiased estimator, the mse is just the variance. In general

ˆ 2 ˆ ˆ ˆ 2 Eθ (θ θ) = Eθ (θ Eθθ + Eθθ θ) − − − ˆ ˆ 2 ˆ 2 ˆ ˆ ˆ [Notation note: when a parameter subscript is used with an expectation or = Eθ(θ Eθθ) + Eθ(θ) θ + 2 Eθ(θ) θ Eθ θ Eθθ variance, it refers to the parameter that is being conditioned on. i.e. the − − − − = var (θˆ) + bias2(θˆ), expectation or variance will be a function of the subscript] θ ˆ ˆ where bias(θ) = Eθ(θ) θ. − [NB: sometimes it can be preferable to have a biased estimator with a low variance - this is sometimes known as the ’bias-variance tradeoﬀ’.]

Lecture 2. Estimation, bias, and mean squared error 3 (1–7) Lecture 2. Estimation, bias, and mean squared error 4 (1–7) 2. Estimation and bias 2.4. Example: Alternative estimators for Binomial mean 2. Estimation and bias 2.4. Example: Alternative estimators for Binomial mean Example: Alternative estimators for Binomial mean

Suppose X Binomial(n, θ), and we want to estimate θ. ∼ The standard estimator is TU = X /n, which is Unbiassed since Eθ(TU ) = nθ/n = θ. 2 TU has variance varθ(TU ) = varθ(X )/n = θ(1 θ)/n. X +1 − X 1 Consider an alternative estimator TB = n+2 = w n + (1 w) 2 , where 1 − w = n/(n + 2). TB is a weighted average of X /n and 2 . e.g. if X is 8 successes out of 10 trials, we would estimate the underlying success probability as T (8) = 9/12 = 0.75, rather than 0.8. nθ+1 1 Then Eθ(TB ) θ = θ = (1 w) θ , and so it is biased. − n+2 − − 2 − varθ (X ) 2 varθ(TB ) = 2 = w θ(1 θ)/n. (n+2) − 2 Now mse(TU ) = varθ(TU ) + bias (TU ) = θ(1 θ)/n. So the biased estimator has smaller MSE in much of the range of θ − 2 2 2 2 1 TB may be preferable if we do not think θ is near 0 or 1. mse(TB ) = varθ(TB ) + bias (TB ) = w θ(1 θ)/n + (1 w) 2 θ − − − So our prior judgement about θ might aﬀect our choice of estimator. Will see more of this when we come to Bayesian methods,.

Lecture 2. Estimation, bias, and mean squared error 5 (1–7) Lecture 2. Estimation, bias, and mean squared error 6 (1–7)

2. Estimation and bias 2.5. Why unbiasedness is not necessarily so great Why unbiasedness is not necessarily so great

Suppose X Poisson(λ), and for some reason (which escapes me for the ∼ 2 2λ moment), you want to estimate θ = [P(X = 0)] = e− . Then any unbiassed estimator T (X ) must satisfy Eθ(T (X )) = θ, or equivalently

x λ ∞ λ 2λ Eλ(T (X )) = e− T (x) = e− . x! x=0 X The only function T that can satisfy this equation is T (X ) = ( 1)X [coeﬃcients of polynomial must match]. − 2λ Thus the only unbiassed estimator estimates e− to be 1 if X is even, -1 if X is odd. This is not sensible.

Lecture 2. Estimation, bias, and mean squared error 7 (1–7)