ELEC-E7130 Examples on parameter estimation Samuli Aalto Department of Communications and Networking Aalto University September 25, 2018 1 Discrete uniform distribution U d(1;N), one unknown parameter N Let U d(1;N) denote the discrete uniform distribution, for which N 2 f1; 2;:::g and the point probabilities are given by 1 p(i) = PfX = ig = ; i 2 f1;:::;Ng: N It follows that N N X 1 X 1 N(N + 1) N + 1 E(X) = PfX = igi = i = = ; N N 2 2 i=1 i=1 N N X 1 X 1 N(N + 1)(2N + 1) E(X2) = PfX = igi2 = i2 = N N 6 i=1 i=1 (N + 1)(2N + 1) = ; 6 (N + 1)(2N + 1) N + 12 V(X) = E(X2) − (E(X))2 = − 6 2 (N + 1)(N − 1) N 2 − 1 = = : 12 12 1 1.1 Estimation of N: Method of Moments (MoM) d Consider an IID sample (x1; : : : ; xn) of size n from distribution U (1;N). The first (theoretical) moment equals N + 1 µ = E(X) = ; 1 2 and the corresponding sample moment is the sample meanx ¯n: n 1 X µ^ =x ¯ = x : 1 n n i i=1 Let N^ denote the estimator of the unknown parameter N. MoM estimation: By solving the requirement (for the first moment) that N^ + 1 µ^ = ; 1 2 we get the estimator MoM N^ = 2^µ1 − 1 = 2¯xn − 1: (1) 1.2 Estimation of N: Maximum Likelihood (ML) d Consider again an IID sample (x1; : : : ; xn) of size n from distribution U (1;N). Let mn denote the maximum value of this sample, mn = maxfx1; : : : ; xng: The likelihood function for this discrete distribution with unknown parame- ter N equals L(x1; : : : ; xn; N) = P fX1 = x1;:::;Xn = xn; Ng = P fX1 = x1; Ng · · · P fXn = xn; Ng ( n 1 ; if N ≥ m ; = N n 0; otherwise: Let N^ denote the estimator of the unknown parameter N. 2 ML estimation: Since 1 n 1 n 1 n max L(x1; : : : ; xn; N) = max 0; ; ;::: = ; N2f1;2;:::g mn mn + 1 mn we get the estimator ML N^ = arg max L(x1; : : : ; xn; N) = mn: (2) N 1.3 Example Assume that your IID sample of size n = 5 (from distribution U d(1;N) with unknown N) is as follows: (x1; : : : ; xn) = (5; 2; 7; 5; 3): 22 Now, the sample mean isx ¯n = 5 = 4:4, and the sample maximum equals mn = 7. MoM estimate: By (1), we get MoM N^ = 2¯xn − 1 = 7:8 ML estimate: By (2), we get ML N^ = mn = 7:0 Two following two estimates are borrowed from the German tank problem discussed in the lecture slides 41-42.1 Bayesian estimate: (m − 1)(n − 1) 24 N^ Bayes = n = = 8:0 n − 2 3 Minimum-variance unbiased estimate: m (n + 1) 37 N^ MinV = n − 1 = = 7:4 n 5 1Note, however, that there is a slight difference in our estimation problem when compared to the German tank problem. Can you identify the difference? 3 2 Continuous uniform distribution U C(0; b), one unknown parameter b Let U c(0; b) denote the continuous uniform distribution, for which b > 0 and the probability density function equals 8 1 < ; x 2 [0; b]; f(x) = b : 0; otherwise: It follows that Z b 1 Z b b E(X) = f(x) x dx = x dx = ; 0 b 0 2 Z b 1 Z b b2 E(X2) = f(x) x2 dx = x2 dx = ; 0 b 0 3 b2 b2 b2 V(X) = E(X2) − (E(X))2 = − = : 3 2 12 2.1 Estimation of b: Method of Moments (MoM) c Consider an IID sample (x1; : : : ; xn) of size n from distribution U (0; b). The first (theoretical) moment equals b µ = E(X) = ; 1 2 and the corresponding sample moment is the sample meanx ¯n: n 1 X µ^ =x ¯ = x : 1 n n i i=1 Let ^b denote the estimator of the unknown parameter b. MoM estimation: By solving the requirement (for the first moment) that ^b µ^ = ; 1 2 we get the estimator ^MoM b = 2^µ1 = 2¯xn: (3) 4 2.2 Estimation of b: Maximum Likelihood (ML) c Consider again an IID sample (x1; : : : ; xn) of size n from distribution U (0; b). Let mn denote the maximum value of this sample, mn = maxfx1; : : : ; xng: The likelihood function for this continuous distribution with unknown pa- rameter b equals L(x1; : : : ; xn; b) = f(x1; b) ··· f(xn; b) ( n 1 ; if b ≥ m ; = b n 0; otherwise: Let ^b denote the estimator of the unknown parameter N. ML estimation: Since 1n 1 n max L(x1; : : : ; xn; b) = max = ; b>0 b≥mn b mn we get the estimator ^ML b = arg max L(x1; : : : ; xn; b) = mn: (4) b 2.3 Example Assume that your IID sample of size n = 5 (from distribution U c(0; b) with unknown b) is as follows: (x1; : : : ; xn) = (30; 20; 90; 100; 60): 300 Now, the sample mean isx ¯n = 5 = 60:0, and the sample maximum equals mn = 100. MoM estimate: By (3), we get ^MoM b = 2¯xn = 120:0 ML estimate: By (4), we get ^ML b = mn = 100:0 5 3 Continuous uniform distribution U c(a; b), two unknown parameters a and b Let U c(a; b) denote the continuous uniform distribution, for which a < b and the probability density function equals 8 1 < ; x 2 [a; b]; f(x) = b − a : 0; otherwise: It follows that Z b 1 Z b a + b E(X) = f(x) x dx = x dx = ; a b − a a 2 Z b 1 Z b a2 + ab + b2 E(X2) = f(x) x2 dx = x2 dx = ; 0 b − a a 3 a2 + ab + b2 a + b2 (b − a)2 V(X) = E(X2) − (E(X))2 = − = : 3 2 12 3.1 Estimation of a and b: Method of Moments (MoM) c Consider an IID sample (x1; : : : ; xn) of size n from distribution U (a; b). The first two (theoretical) moments equal a + b a2 + ab + b2 µ = E(X) = ; µ = E(X2) = : 1 2 2 3 and the corresponding sample moments are given by n n 1 X 1 X µ^ = x ; µ^ = x2: 1 n i 2 n i i=1 i=1 Leta ^ and ^b denote the estimators of the two unknown parameters a and b, respectively. MoM estimation: By solving the requirements (for the first two moments) that a^ + ^b a^2 +a ^^b + ^b2 µ^ = ; µ^ = ; 1 2 2 3 we get the estimators q q MoM 2 ^MoM 2 a^ =µ ^1 − 3(^µ2 − µ^1); b =µ ^1 + 3(^µ2 − µ^1): (5) 6 3.2 Estimation of a and b: Maximum Likelihood (ML) c Consider again an IID sample (x1; : : : ; xn) of size n from distribution U (a; b). Let un and vn denote the minimum and the maximum value of this sample, respecively, so that un = minfx1; : : : ; xng; vn = maxfx1; : : : ; xng: The likelihood function for this continuous distribution with unknown pa- rameters a and b equals L(x1; : : : ; xn; a; b) = f(x1; a; b) ··· f(xn; a; b) ( n 1 ; if a ≤ u and b ≥ v ; = b−a n n 0; otherwise: Leta ^ and ^b denote the estimators of the unknown parameters a and b, respectively. ML estimation: Since 1 n 1 n max L(x1; : : : ; xn; a; b) = max = ; a<b a≤un;b≥vn b − a vn − un we get the following pair of estimators: ML ^ML (^a ; b ) = arg max L(x1; : : : ; xn; a; b) = (un; vn): (6) (a;b) 3.3 Example Assume that your IID sample of size n = 5 (from distribution U c(a; b) with unknown a and b) is as follows: (x1; : : : ; xn) = (30; 20; 90; 100; 60): Now, the first two sample moments are 300 23000 µ^ = = 60:0; µ^ = = 4600:0 1 5 2 5 In addition, the sample mimimum un and the sample maximum vn equal un = 20:0; vn = 100:0 7 MoM estimate: By (5), we get q p MoM 2 a^ =µ ^1 − 3(^µ2 − µ^1) = 60 − 3000 = 5:2; q p ^MoM 2 b =µ ^1 + 3(^µ2 − µ^1) = 60 + 3000 = 114:8 ML estimate: By (6), we get ML ^ML a^ = un = 20:0; b = vn = 100:0 8.