<p>Statistics 512 Notes 15: Properties of Maximum Likelihood Estimates Continued</p><p>Computation of maximum likelihood estimates</p><p>Example 2: Logistic distribution. Let X1, , X n be iid with density exp{- (x -q )} f( x ;q ) = , -� <x ��, < q . (1+ exp{ - (x -q )})2 The log of the likelihood simplifies to: n n l(q )= log f ( X ; q ) = n q - nX - 2 log(1 + exp{ - ( X - q )}) 邋i=1i i = 1 i Using this, the first derivative is n exp{- (X -q )} l'(q )= n - 2 i i=1 1+ exp{ - (X i -q )} Setting this equal to 0 and rearranging terms results in teh equation: n exp{- (X -q )} n i = i=1 . (*) 1+ exp{ - (X i -q )} 2 Although this does not simplify, we can show the equation (*) has a unique solution. The derivative of the left hand side of (i) simplifies to nexp{- (X -q )} n exp{ - ( X - q )} 邋 i= i > 0 qi=11- exp{ - (X q )} i = 1 2 � i (1+ exp{ - (X i -q )}) Thus, the left hand side of (*) is a strictly increasing function of q . Finally, the left hand side of (*) approaches 0 as q � and approaches n as q  . Thus, the equation (*) has a unique solution. Also the second derivative of l(q ) is strictly negative for all q ; so the solution is a maximum.</p><p>How do we find the maximum likelihood estimate that is the solution to (*)? </p><p>Newton’s method is a numerical method for approximating solutions to equations. The method produces a sequence of values q(0), q (1) , that, under ideal conditions, converges ˆ to the MLE qMLE . To motivate the method, we expand the derivative of the log likelihood around q (j ) : ˆ(j ) ˆ ( j ) ( j ) 0=l '(qMLE )� l '( q ) - ( q MLE q ) l ''( q ) ˆ Solving for qMLE gives l '(q (j ) ) qˆ � q (j ) MLE l ''(q (j ) ) This suggests the following iterative scheme: l '(q (j ) ) q(j+ 1)= q ( j ) - . l ''(q (j ) )</p><p>The following is an R function that uses Newton’s method to approximate the maximum likelihood estimate for a logistic distribution: mlelogisticfunc=function(xvec,toler=.001){ startvalue=median(xvec); n=length(xvec); thetahatcurr=startvalue; # Compute first deriviative of log likelihood firstderivll=n-2*sum(exp(-xvec+thetahatcurr)/(1+exp(- xvec+thetahatcurr))); </p><p># Continue Newton’s method until the first derivative # of the likelihood is within toler of 0 while(abs(firstderivll)>toler){ # Compute second derivative of log likelihood secondderivll=-2*sum(exp(-xvec+thetahatcurr)/(1+exp(- xvec+thetahatcurr))^2); # Newton’s method update of estimate of theta thetahatnew=thetahatcurr-firstderivll/secondderivll; thetahatcurr=thetahatnew; # Compute first derivative of log likelihood firstderivll=n-2*sum(exp(-xvec+thetahatcurr)/(1+exp(- xvec+thetahatcurr))); } list(thetahat=thetahatcurr); }</p><p>Issues with Newton’s method: 1. The method does not work if l ''(q (j ) ) 0 . 2. The method does not always converge. 3. The method may converge to a local but not global maximum. 4. The starting value is important since different starting values can converge to different local maxima. If the log likelihood is not concave, try different starting values. Another useful method for computing maximum likelihood estimates is the EM algorithm (Chapter 6.6).</p><p>Good book on computation in statistics: Numerical Methods of Statistics, John Monahan</p><p>Confidence interval</p><p>Theorem 6.2.2: Assume X1, , X n are iid with pdf f( x ;q0 ) for q0 蜽 such that the regularity conditions (R0)-(R5) are satisfied. Then D ˆ 骣 1 n(qMLE - q0 ) N 琪0, 桫 I(q0 )</p><p>Approximate (1-a ) confidence interval for q : 1 qˆ z MLE a / 2 ˆ nI(qMLE )</p><p>For the logistic distribution model, 2 轾 轾 n exp{- (X -q )} I(q )= - E log f ( X ; q ) = - E 犏 - 2 i q犏q 2 q i=1 2 臌 臌犏 (1+ exp{ - (X i -q )}) Problem: It is hard to compute I(q ) . Solutions: (1) Use the observed Fisher information rather than the expected information. 2 n Observed Fisher information = - 2 logf ( X i ;q ) ˆ i=1 q q= qMLE Another approximate (1-a ) confidence interval for q besides that given by Theorem 6.2.2 is: 1 qˆ z MLE a / 2 2 n - 2 logf ( X i ;q ) ˆ i=1 q q= qMLE</p><p>(2) Parametric bootstrap: In the nonparametric bootstrap, we approximated the true distribution F by the empirical ˆ distribution Fn . In the parametric bootstrap, we approximate the true distribution F by the density ˆ f( x ;qMLE ) . To obtain a bootstrap percentile confidence interval: * * ˆ 1. Draw an iid sample X1 , , X n from f( x ;qMLE ) . ˆ* ˆ * * 2. Compute , qb= q MLE(X1 , , X n ) 3. Repeat steps 1 and 2 for b=1,...,M. 4. The bootstrap percentile confidence interval endpoints ˆ* ˆ * for q are the a / 2 and 1-a / 2 quantiles of (q1 , , qM ) .</p><p>The function rlogis(n,location=theta,scale=1) generates n iid logistic random variables in R.</p><p># Function that forms percentile bootstrap confidence # interval for parameter theta of logistic distribution percentcibootlogisticfunc=function(X,m,alpha){ # X is a vector containing the original sample # m is the desired number of bootstrap replications thetahat=mlelogisticfunc(X)$thetahat; thetastar=rep(0,m); # stores bootstrap estimates of theta n=length(X);</p><p># Carry out m bootstrap resamples and estimate theta for # each resample for(i in 1:m){ Xstar=rlogis(n,location=thetahat,scale=1); thetastar[i]=mlelogisticfunc(Xstar)$thetahat; } thetastarordered=sort(thetastar); # order the thetastars cutoff=floor((alpha/2)*(m+1)); lower=thetastarordered[cutoff]; # lower CI endpoint upper=thetastarordered[m+1-cutoff]; # upper CI endpoint list(thetahat=thetahat,lower=lower,upper=upper); }</p><p>Optimality (Efficiency) of the MLE:</p><p>We will now show that for large samples, the MLE does as well as any other possible estimator in some sense.</p><p>We start by presenting a lower bound on the variance of an unbiased estimator for an estimator in finite samples. </p><p>Theorem 6.2.1 (Rao-Cramer Lower Bound): Let X1, , X n be iid with pdf f( x ;q ) for q 蜽 . Assume that the regularity conditions (R0)-(R4) hold. Let Y= u( X1 , , X n ) be a statistic with mean</p><p>Eq( Y )= E q [ u ( X1 , , Xn )] = k (q ) . Then [k '(q )]2 Var( Y ) . q nI(q )</p><p>Proof: We will follow the proof in the book.</p>