Likelihood Ratio Test, Most Powerful Test, Uniformly Most Powerful Test

A. We first review the general definition of a hypothesis test. A hypothesis test is like a lawsuit:

H 0 : the defendant is innocent versus

H a : the defendant is guilty The truth

H 0 : the defendant H a the defendant is innocent is guilty

Jury’s H 0 Right decision Type II error Decision H a Type I error Right decision

The significance level and the power of the test.

α = P(Type I error) = P(Reject H 0 | H 0 ) ← signiicance level

β = P(Type II error) = P(Fail to reject H 0 | H a )

Power = 1- β = P(Reject H 0 | H a )

B. Now we derive the likelihood ratio test for the usual two- sided hypotheses for a population mean. (It has a simple null hypothesis and a composite alternative hypothesis.)

Example 1. Please derive the likelihood ratio test for H0: μ = μ0 versus Ha: μ ≠ μ0, when the population is normal and population variance σ2 is known.

Solution: For a 2-sided test of H0: μ = μ0 versus Ha: μ ≠ μ0, when the population is normal and population variance σ2 is known, we have:

1

   :   0  and    :     

The likelihoods are:

L   L0  n 2 n 1  x      1  2  1 n 2   exp i 0     exp  x    i1 2  2  2  2 i1 i 0  2  2   2   2 

There is no free parameter in L , thus Lˆ   L  .

L  L n 2 n 1  x      1  2  1 n 2   exp i     exp  x    i1 2  2  2  2 i1 i  2  2   2   2 

There is only one free parameter μ in L. Now we shall find the value of μ that maximizes the log likelihood n 1 n ln L    ln2 2  x   2 . 2 2 2 i1 i d ln L 1 n By solving  x     0 , we have ˆ  x d  2 i1 i It is easy to verify that ˆ  x indeed maximizes the loglikelihood, and thus the likelihood function.

Therefore the likelihood ratio is: Lˆ  L     0 ˆ L  max L  n  1  2  1 n 2    exp  x    L  2 2  2 2 i1 i 0   0      Lˆ  n  1  2  1 n 2  exp  x  x  2  2 i1i   2   2 

 1 n 2 2   exp  x    x  x  2 i1i 0  i    2   2  1  x  0   1 2   exp     exp z0    2  / n    2 

2

Therefore, the likelihood ratio test that will reject H0 when *    is equivalent to the z-test that will reject H0 when

Z0  c , where c can be determined by the significance level α as c z / 2 .

C. MP Test, UMP Test, and the Neyman-Pearson Lemma

Now considering some one-sided tests where we have :

H0 :   0 versus H a :   0 or, H0 :   0 versus H a :   0 Given the composite null hypothesis for the second one-sided test, we need to expand our definition of the significance level as follows:

For 퐻: 휃 ∈ 휔 versus 퐻: 휃 ∈ 휔

Let the random sample be denoted by 푿 = (푋, 푋,⋯, 푋) . Suppose the rejection region is 푪 such that: (*note: the rejection region can be defined in terms of the sample points directly instead of the test statistic values)

We reject 퐻 if 푿 ∈ 푪 We fail to reject 퐻 if 푿 ∈ 푪

Definition: The size or significance level of the test is defined as: 휶 = 퐦퐚퐱 푷(푿 ∈ 푪) = 퐦퐚퐱 푷휽(푹풆풋풆풄풕 퐻|휃 ∈ 휔) ∈

Definition: The power of a test is given by ퟏ − 휷 = 휸(휃) = 푷휽(푹풆풋풆풄풕 퐻|휃 ∈ 휔)

Definition: A most powerful test corresponds to the best rejection region 푪 such that in testing a simple null hypothesis 퐻: 휃 = 휃 versus a simple alternative hypothesis 퐻: 휃 = 휃 , this test has the largest possible test power (ퟏ − 휷) among all tests of size 휶.

3

Theorem (Neyman-Pearson Lemma). The likelihood ratio test for a simple null hypothesis 퐻: 휃 = 휃 versus a simple alternative hypothesis 퐻: 휃 = 휃 is a most powerful test.

Jerzy Neyman (left) and Egon Pearson (right)

Definition: A uniformly most powerful test in testing a simple null hypothesis 퐻: 휃 = 휃 versus a composite alternative hypothesis 퐻: 휃 ∈ 휔, or in testing a composite null hypothesis 퐻: 휃 ∈ 휔 versus a composite alternative 퐻: 휃 ∈ 휔 , is a size 휶 test such that this test has the largest possible test power 휸(휃) for every simple alternative hypothesis: 퐻: 휃 = 휃 ∈ 휔, among all such tests of size 휶.

Now we derive the likelihood ratio test for the first one- sided hypotheses.

( ) Example 2. Let푓(푥 |휇) = 푒 , 푓표푟 푖 = 1,⋯, 푛, where √ 휎 is known. Derive the likelihood ratio test for the hypothesis 퐻: 휇 = 휇 푣푠. 퐻: 휇 > 휇, and show whether it is a UMP test or not.

Solution: 흎 = {휇: 휇 = 휇} 훀 = {휇: 휇 ≥ 휇} Note: Now that we are not just dealing with the two-sided hypothesis, it is important to know that the more general definition of 흎 is the set of all unknown parameter values under 퐻, while 훀 is the set of all unknown parameter values under the union of 퐻 and 퐻.

4

Therefore we have: ⎧ 퐿 = 푓(푥 , 푥 ,⋯ , 푥 |휇 = 휇 ) = 푓(푥 |휇 = 휇 ) ⎪ ⎪ ( ) ⎪ 1 ∑ ( ) = 푒 = (2휋휎) 푒 ⎪ √2휋휎 ⎨ ⎪ 퐿 = 푓(푥, 푥,⋯ , 푥|휇 ≥ 휇) = 푓(푥|휇 ≥ 휇) ⎪ ⎪ ( ) 1 ∑ ( ) ⎪ = 푒 = (2휋휎) 푒 √2휋휎 ⎩ Since there is no free parameter in 퐿 , ∑() sup(퐿) = (2휋휎 ) 푒 . 휕푙표푔퐿 푛 ⎧ = (푥̅ − 휇) ⎪ 휕휇 휎 ∵ ,∴ sup(퐿) ⎨휕 푙표푔퐿 푛 ⎪ =− <0 ⎩ 휕휇 휎 ∑ ( ̅) (2휋휎) 푒 , 푖푓 푥̅ > 휇 = . ∑() (2휋휎 ) 푒 , 푖푓 푥̅ ≤ 휇 ( ̅ ) sup(퐿) ∴ 퐿푅 = = 푒 , 푖푓 푥̅ > 휇. sup(퐿 ) 1, 푖푓 푥̅ ≤ 휇 푋 − 휇 ∗ ∗ ∴ 푃(퐿푅 ≤ 푐 |퐻) = 훼 ⇔ 푃 푍 = ≥ −2푙표푔푐 퐻 휎⁄√푛 = 훼. ∴ Reject 퐻 in favor of 퐻 if 푍 = ≥ 푍 ⁄√

Furthermore, it is easy to show that for each 휇 > 휇 , the likelihood ratio test of 퐻: 휇 = 휇 푣푠. 퐻: 휇 = 휇 rejects the null hypothesis at the significance level α for 푍 = ≥ 푍 ⁄√

By the Neyman-Pearson Lemma, the likelihood ratio test is also the most powerful test.

Now since for each 1  0 , the most powerful size α test of

퐻: 휇 = 휇 푣푠. 퐻: 휇 = 휇 rejects the null hypothesis for 푍 = ≥ 푍 . Since this same test function is most powerful ⁄√ for each 1  0 , this test is UMP for 퐻: 휇 = 휇 푣푠. 퐻: 휇 > 휇.

5

D. Monotone Likelihood Ratio, the Karlin-Rubin Theorem, and the Exponential Family.

A condition under which the UMP tests exists is when the family of distributions being considered possesses a property called monotone likelihood ratio.

Definition: Let 푓(풙|휃) be the joint pdf of the sample 푿 = (푋, 푋,⋯, 푋) . Then 푓(풙|휃) is said to have a monotone likelihood ratio in the statistic 푇(푿) if for any choice 휃 < 휃 of parameter values, the likelihood ratio 푓(풙|휃)/푓(풙|휃) depends on values of the data X only through the value of statistic 푇(푿) and, in addition, this ratio is a monotone function of 푇(푿).

Theorem (Karlin-Rubin). Suppose that 푓(풙|휃) has an increasing monotone likelihood ratio for the statistic 푇(푿). Let 훼 and 푘 be chosen so that 휶 = 푷휽(푇(푿) ≥ 풌휶). Then

푪 = {풙: 푇(풙) ≥ 풌휶}

is the rejection region (*also called ‘critical region’) for a UMP test for the one-sided tests of: 퐻: 휃 ≤ 휃 versus 퐻: 휃 > 휃.

Theorem. For a Regular Exponential Family with only one unknown parameter θ, and a population p.d.f.: 푓(푥; 휃) = 푐(휃) ⋅ 푔(푥) ⋅ exp[푡(푥)푤(휃)] If 푤(휃) is a is an increasing function of 휃, then we have a monotone likelihood ratio in the statistic 푇푋 = ∑ 푡(푋). {*Recall 푇푋 = ∑ 푡(푋)} is also complete sufficient for 휃}

Examples of a family with monotone likelihood ratio:

For X1,, X n iid Exponential (  ), n  X  e 1 i n  1 n p(x| 1 ) i1   1    n   exp  (1  0 ) X i  p(x| 0 ) 0 Xi   0  i1  0e i1

6

p(x|  ) n 1  X For 1  0 , is an increasing function of  i so the p(x| 0 ) i1 n family has monotone likelihood ratio in T(x )  X i . i1

Example 3. Let 푋, 푋,… 푋 ~푁(휇, 휎 ), where 휎 is known (1) Find the LRT for 퐻: 휇 ≤ 휇 푣푠 퐻: 휇 > 휇 (2) Show the test in (1) is a UMP test.

Solution: 흎 = {휇: 휇 ≤ 휇} 훀 = {휇: −∞< 휇 <∞} Note: Now that we are not just dealing with the two-sided hypothesis, it is important to know that the more general definition of 흎 is the set of all unknown parameter values under 퐻, while 훀 is the set of all unknown parameter values under the union of 퐻 and 퐻.

(1) The ratio of the likelihood is shown as the following, 퐿(휔) 휆(푥)= 퐿(Ω)

퐿(휔) is the maximum likelihood under 퐻: 휇 ≤ 휇 1 / ∑(푥 − 푥̅) ⎧ exp − , 푥̅ < 휇 ⎪ 2휋휎 2휎 퐿(휔) = ⎨ 1 / ∑(푥 − 휇 ) ⎪ exp − , 푥̅ ≥ 휇 ⎩ 2휋휎 2휎

퐿Ω is the maximum likelihood under 퐻 푢푛푖표푛 퐻 1 / ∑(푥 − 푥̅) 퐿Ω = exp − 2휋휎 2휎

The ratio of the maximized likelihood is, 1, 푥̅ < 휇 휆(푥) = (∑(푥 − 휇) − ∑(푥 − 푥̅) ) 푒푥푝 − , 푥̅ ≥ 휇 2휎

⇒ 1, 푥̅ < 휇 휆(푥) = (̅) 푒 , 푥̅ ≥ 휇

7

By LRT, we reject null hypothesis if 휆(푥) < 푐. Generally, c is chosen to be less than 1. The rejection region is,

(̅ ) 2휎 푅 = 푋 ∶ 푒 < 푐 = 푋: 푋 > 휇 + − ln 푐 푛

(2) We can use the Karlin-Rubin theorem to show that the LRT is UMP.

First, we need to show the size for the LRT test. 훼 = 푠푢푝 푃(푋 ∈ 푅 ) In which, 2휎 푃(푋 ∈ 푅 ) = 푃 푋 > 휇 + − ln 푐 푛 푋 − 휇 휇 − 휇 = 푃 > + √−2ln 푐 휎⁄푛 휎⁄푛

휇 − 휇 = 푃 푍 > + √−2ln 푐 휎⁄푛 where z follows standard normal distribution, 푃 푍 > + √−2ln 푐 is an increasing function of 휇. So ⁄ 휇 − 휇 훼 = 푠푢푝 푃(푋 ∈ 푅 ) = 푃 푍 > + √−2ln 푐 | 휇 = 휇 휎 ⁄푛

= 푃푍 > √−2ln 푐 | 휇 = 휇

So the LRT test a size 훼 test where 훼 = 푃푍 > √−2ln 푐

Next, we prove the family has monotone likelihood ratio (MLR) in 푋. It is straightforward to show that 푋 is a sufficient statistics for 휇 because we have an exponential family. For any pair 휇 > 휇, the ratio of the likelihood is:

8

푛 / (푥̅ − 휇 ) exp − 푓(풙|휇 ) 2휋휎 2휎/푛 = 푓(풙|휇 ) 푛 / (푥̅ − 휇 ) exp − 2휋휎 2휎/푛 2휎 = 푒푥푝 ((푥̅ − 휇 ) −(푥̅ − 휇 )) 푛 2휎 = exp ( (휇 − 휇 )(2푥̅ − 휇 − 휇 )) 푛 It is increasing in 푋. So the family has monotone likelihood ratio (MLR) in 푋.

By the Karlin-Rubin theorem, we know that the LRT test is also an UMP test.

9