8.4 Lecture 11 Friday 02/09/01 Homework and Labs. see the logictics section Please hand in your labs to Johan by next Monday.

9 Maximum Likelihood Estimation

X1,X2,X3,...Xn have joint density denoted

fθ(x1, x2, . . . , xn) = f(x1, x2, . . . , xn θ) | Given observed values X1 = x1,X2 = x2,...,Xn = xn, the likelihood of θ is the function

lik(θ) = f(x1, x2, . . . , xn θ) | considered as a function of θ. If the distribution is discrete, f will be the frequency distribution function. In words: lik(θ)=probability of observing the given data as a function of θ. Definition: The maximum likelihood estimate (mle) of θ is that value of θ that maximises lik(θ): it is the value that makes the observed data the “most probable”. If the Xi are iid, then the likelihood simplifies to n lik(θ) = f(x θ) Y i i=1 | Rather than maximising this product which can be quite tedious, we often use the fact that the logarithm is an increasing function so it will be equivalent to maximise the log likelihood: n l(θ) = log(f(x θ)) X i i=1 |

9.0.1 Poisson Example λxe λ P (X = x) = − x! For X1,X2,...,Xn iid Poisson random variables will have a joint frequency function that is a product of the marginal frequency functions, the log likelihood will thus be:

n l(λ) = (Xilogλ λ logXi!) Pi=1 n − − n = logλ Xi nλ logXi! Pi=1 − − Pi=1 We need to find the maximum by finding the derivative: 1 n l (λ) = x n = 0 0 X i λ i=1 −

1 which implies that the estimate should be λˆ = X¯ (as long as we check that the function l is actually concave, which it is). The mle agrees with the method of moments in this case, so does its sampling distribu- tion.

9.0.2 Normal Example

2 If X1,X2,...,Xn are iid (µ, σ ) random variables their density is written: N n 1 1 x µ f(x , . . . , x µ, σ) = exp ( [ i ]2) 1 n Y − | i σ√2π −2 σ Regarded as a function of the two parameters, µ and σ this is the likelihood: n 1 n `(µ, σ) = nlogσ log2π (x µ)2 2 X i − − 2 − 2σ i=1 −

∂` 1 n = (x µ) 2 X i ∂µ σ i=1 −

∂` n 3 n 2 = + σ− (xi µ) ∂σ − σ Pi=1 − so setting these to zero gives X¯ as the mle for µ, andσ ˆ2 as the usual.

9.0.3 Gamma Example

1 α α 1 λx f(x α, λ) = λ x − e− | Γ(α) giving the log-likelihood: n l(x α, λ) = [αlogλ + (α 1)logx λx logΓ(α)] X i i | i=1 − − − One ends up with a nonlinear equation inα ˆ this cannot be solved in closed form, there are basically two methods and they are called root-finding methods, they are based on the calculus theorem that says that when a function is continuous, and changes signs on an interval, it is zero on that interval. For this particular problem there already coded in matlab a mle method called gamfit, that also provides a confidence interval. For general optimization, the function in Matlab is fmin for one variable, and fmins you could also look at how to use optimize in Splus.

2 9.1 Maximum Likelihood of Multinomial Cell Probabilities

X1,X2,...,Xm are counts in cells/ boxes 1 up to m, each box has a different probability (think of the boxes being bigger or smaller) and we fix the number of balls that fall to be n:x1 + x2 + + xm = n. The probability of each box is pi, with also a constraint: ··· p1 + p2 + + pm = 1, this is a case in which the X0s are NOT independent, the joint ··· i probability of a vector x1, x2, . . . , xm is called the multinomial ,and has the form:

n! n xi ! x1 x1 xm f(x1, x2, . . . , xm p1, . . . , pm) = pi = p1 p1 pm | xi! Y x1, x2, xm ··· Q Each box taken separately against all the other boxes is a binomial, this is an extension thereof. (look at page 72) We study the log-likelihood of this :

m m l(p , p , ..., p ) = logn! logx ! + x logp 1 2 m X i X i i − i=1 i=1 However we can’t just go ahead and maximise this we have to take the constraint into account so we have to use the Lagrange multipliers again. We use m L(p , p , ..., p , λ) = l(p , p , ..., p ) + λ(1 p ) 1 2 m 1 2 m X i − i By posing all the derivatives to be 0, we get the most natural estimate x pˆ = i i n Maximising log likelihood, with and without constraints, can be an unsolvable problem in closed form, then we have to use iterative procedures. I explained about how the parametris bootstrap was often the only way to study the sampling distribution of the mle.

3