ELEC-E7130 Examples on Parameter Estimation

ELEC-E7130 Examples on parameter estimation

Samuli Aalto Department of Communications and Networking Aalto University

September 25, 2018

1 Discrete uniform distribution U d(1,N), one unknown parameter N

Let U d(1,N) denote the discrete uniform distribution, for which N ∈ {1, 2,...} and the point probabilities are given by 1 p(i) = P{X = i} = , i ∈ {1,...,N}. N It follows that

N N X 1 X 1 N(N + 1) N + 1 E(X) = P{X = i}i = i = = , N N 2 2 i=1 i=1 N N X 1 X 1 N(N + 1)(2N + 1) E(X2) = P{X = i}i2 = i2 = N N 6 i=1 i=1 (N + 1)(2N + 1) = , 6 (N + 1)(2N + 1) N + 12 V(X) = E(X2) − (E(X))2 = − 6 2 (N + 1)(N − 1) N 2 − 1 = = . 12 12

1 1.1 Estimation of N: Method of Moments (MoM)

d Consider an IID sample (x1, . . . , xn) of size n from distribution U (1,N). The ﬁrst (theoretical) moment equals N + 1 µ = E(X) = , 1 2 and the corresponding sample moment is the sample meanx ¯n: n 1 X µˆ =x ¯ = x . 1 n n i i=1 Let Nˆ denote the estimator of the unknown parameter N. MoM estimation: By solving the requirement (for the ﬁrst moment) that

Nˆ + 1 µˆ = , 1 2 we get the estimator

MoM Nˆ = 2ˆµ1 − 1 = 2¯xn − 1. (1)

1.2 Estimation of N: Maximum Likelihood (ML)

d Consider again an IID sample (x1, . . . , xn) of size n from distribution U (1,N).

Let mn denote the maximum value of this sample,

mn = max{x1, . . . , xn}.

The likelihood function for this discrete distribution with unknown parameter N equals

L(x1, . . . , xn; N) = P {X1 = x1,...,Xn = xn; N}

= P {X1 = x1; N}··· P {Xn = xn; N} ( n 1 , if N ≥ m ; = N n 0, otherwise.

Let Nˆ denote the estimator of the unknown parameter N.

2 ML estimation: Since 1 n 1 n 1 n max L(x1, . . . , xn; N) = max 0, , ,... = , N∈{1,2,...} mn mn + 1 mn we get the estimator

ML Nˆ = arg max L(x1, . . . , xn; N) = mn. (2) N

1.3 Example

Assume that your IID sample of size n = 5 (from distribution U d(1,N) with unknown N) is as follows:

(x1, . . . , xn) = (5, 2, 7, 5, 3).

22 Now, the sample mean isx ¯n = 5 = 4.4, and the sample maximum equals mn = 7. MoM estimate: By (1), we get

MoM Nˆ = 2¯xn − 1 = 7.8

ML estimate: By (2), we get

ML Nˆ = mn = 7.0

Two following two estimates are borrowed from the German tank problem discussed in the lecture slides 41-42.1 Bayesian estimate: (m − 1)(n − 1) 24 Nˆ Bayes = n = = 8.0 n − 2 3

Minimum-variance unbiased estimate: m (n + 1) 37 Nˆ MinV = n − 1 = = 7.4 n 5 1Note, however, that there is a slight diﬀerence in our estimation problem when compared to the German tank problem. Can you identify the diﬀerence?

3 2 Continuous uniform distribution U C(0, b), one unknown parameter b

Let U c(0, b) denote the continuous uniform distribution, for which b > 0 and the probability density function equals  1  , x ∈ [0, b]; f(x) = b  0, otherwise. It follows that Z b 1 Z b b E(X) = f(x) x dx = x dx = , 0 b 0 2 Z b 1 Z b b2 E(X2) = f(x) x2 dx = x2 dx = , 0 b 0 3 b2 b2 b2 V(X) = E(X2) − (E(X))2 = − = . 3 2 12

2.1 Estimation of b: Method of Moments (MoM)

c Consider an IID sample (x1, . . . , xn) of size n from distribution U (0, b). The ﬁrst (theoretical) moment equals b µ = E(X) = , 1 2 and the corresponding sample moment is the sample meanx ¯n: n 1 X µˆ =x ¯ = x . 1 n n i i=1 Let ˆb denote the estimator of the unknown parameter b. MoM estimation: By solving the requirement (for the ﬁrst moment) that ˆb µˆ = , 1 2 we get the estimator ˆMoM b = 2ˆµ1 = 2¯xn. (3)

4 2.2 Estimation of b: Maximum Likelihood (ML)

c Consider again an IID sample (x1, . . . , xn) of size n from distribution U (0, b).

Let mn denote the maximum value of this sample,

mn = max{x1, . . . , xn}. The likelihood function for this continuous distribution with unknown parameter b equals

L(x1, . . . , xn; b) = f(x1; b) ··· f(xn; b) ( n 1 , if b ≥ m ; = b n 0, otherwise.

Let ˆb denote the estimator of the unknown parameter N. ML estimation: Since 1n 1 n max L(x1, . . . , xn; b) = max = , b>0 b≥mn b mn we get the estimator ˆML b = arg max L(x1, . . . , xn; b) = mn. (4) b

2.3 Example

Assume that your IID sample of size n = 5 (from distribution U c(0, b) with unknown b) is as follows:

(x1, . . . , xn) = (30, 20, 90, 100, 60).

300 Now, the sample mean isx ¯n = 5 = 60.0, and the sample maximum equals mn = 100. MoM estimate: By (3), we get ˆMoM b = 2¯xn = 120.0

ML estimate: By (4), we get ˆML b = mn = 100.0

5 3 Continuous uniform distribution U c(a, b), two unknown parameters a and b

Let U c(a, b) denote the continuous uniform distribution, for which a < b and the probability density function equals  1  , x ∈ [a, b]; f(x) = b − a  0, otherwise. It follows that Z b 1 Z b a + b E(X) = f(x) x dx = x dx = , a b − a a 2 Z b 1 Z b a2 + ab + b2 E(X2) = f(x) x2 dx = x2 dx = , 0 b − a a 3 a2 + ab + b2 a + b2 (b − a)2 V(X) = E(X2) − (E(X))2 = − = . 3 2 12

3.1 Estimation of a and b: Method of Moments (MoM)

c Consider an IID sample (x1, . . . , xn) of size n from distribution U (a, b). The ﬁrst two (theoretical) moments equal a + b a2 + ab + b2 µ = E(X) = , µ = E(X2) = . 1 2 2 3 and the corresponding sample moments are given by n n 1 X 1 X µˆ = x , µˆ = x2. 1 n i 2 n i i=1 i=1 Leta ˆ and ˆb denote the estimators of the two unknown parameters a and b, respectively. MoM estimation: By solving the requirements (for the ﬁrst two moments) that aˆ + ˆb aˆ2 +a ˆˆb + ˆb2 µˆ = , µˆ = , 1 2 2 3 we get the estimators q q MoM 2 ˆMoM 2 aˆ =µ ˆ1 − 3(ˆµ2 − µˆ1), b =µ ˆ1 + 3(ˆµ2 − µˆ1). (5)

6 3.2 Estimation of a and b: Maximum Likelihood (ML)

c Consider again an IID sample (x1, . . . , xn) of size n from distribution U (a, b).

Let un and vn denote the minimum and the maximum value of this sample, respecively, so that

un = min{x1, . . . , xn}, vn = max{x1, . . . , xn}.

The likelihood function for this continuous distribution with unknown parameters a and b equals

L(x1, . . . , xn; a, b) = f(x1; a, b) ··· f(xn; a, b) ( n 1 , if a ≤ u and b ≥ v ; = b−a n n 0, otherwise.

Leta ˆ and ˆb denote the estimators of the unknown parameters a and b, respectively. ML estimation: Since 1 n 1 n max L(x1, . . . , xn; a, b) = max = , a

ML ˆML (ˆa , b ) = arg max L(x1, . . . , xn; a, b) = (un, vn). (6) (a,b)

3.3 Example

Assume that your IID sample of size n = 5 (from distribution U c(a, b) with unknown a and b) is as follows:

(x1, . . . , xn) = (30, 20, 90, 100, 60).

Now, the ﬁrst two sample moments are 300 23000 µˆ = = 60.0, µˆ = = 4600.0 1 5 2 5

In addition, the sample mimimum un and the sample maximum vn equal

un = 20.0, vn = 100.0

7 MoM estimate: By (5), we get q √ MoM 2 aˆ =µ ˆ1 − 3(ˆµ2 − µˆ1) = 60 − 3000 = 5.2, q √ ˆMoM 2 b =µ ˆ1 + 3(ˆµ2 − µˆ1) = 60 + 3000 = 114.8

ML estimate: By (6), we get

ML ˆML aˆ = un = 20.0, b = vn = 100.0