Likelihood Ratio Test, Most Powerful Test, Uniformly Most Powerful Test A. We first review the general definition of a hypothesis test. A hypothesis test is like a lawsuit: H 0 : the defendant is innocent versus H a : the defendant is guilty The truth H 0 : the defendant H a the defendant is innocent is guilty Jury’s H 0 Right decision Type II error Decision H a Type I error Right decision The significance level and the power of the test. α = P(Type I error) = P(Reject H 0 | H 0 ) ← signiicance level β = P(Type II error) = P(Fail to reject H 0 | H a ) Power = 1- β = P(Reject H 0 | H a ) B. Now we derive the likelihood ratio test for the usual two- sided hypotheses for a population mean. (It has a simple null hypothesis and a composite alternative hypothesis.) Example 1. Please derive the likelihood ratio test for H0: μ = μ0 versus Ha: μ ≠ μ0, when the population is normal and population variance σ2 is known. Solution: For a 2-sided test of H0: μ = μ0 versus Ha: μ ≠ μ0, when the population is normal and population variance σ2 is known, we have: 1 : 0 and : The likelihoods are: L L0 n 2 n 1 x 1 2 1 n 2 exp i 0 exp x i1 2 2 2 2 i1 i 0 2 2 2 2 There is no free parameter in L , thus Lˆ L . L L n 2 n 1 x 1 2 1 n 2 exp i exp x i1 2 2 2 2 i1 i 2 2 2 2 There is only one free parameter μ in L. Now we shall find the value of μ that maximizes the log likelihood n 1 n ln L ln2 2 x 2 . 2 2 2 i1 i d ln L 1 n By solving x 0 , we have ˆ x d 2 i1 i It is easy to verify that ˆ x indeed maximizes the loglikelihood, and thus the likelihood function. Therefore the likelihood ratio is: Lˆ L 0 ˆ L max L n 1 2 1 n 2 exp x L 2 2 2 2 i1 i 0 0 Lˆ n 1 2 1 n 2 exp x x 2 2 i1i 2 2 1 n 2 2 exp x x x 2 i1i 0 i 2 2 1 x 0 1 2 exp exp z0 2 / n 2 2 Therefore, the likelihood ratio test that will reject H0 when * is equivalent to the z-test that will reject H0 when Z0 c , where c can be determined by the significance level α as c z / 2 . C. MP Test, UMP Test, and the Neyman-Pearson Lemma Now considering some one-sided tests where we have : H0 : 0 versus H a : 0 or, H0 : 0 versus H a : 0 Given the composite null hypothesis for the second one-sided test, we need to expand our definition of the significance level as follows: For 퐻: 휃 ∈ 휔 versus 퐻: 휃 ∈ 휔 Let the random sample be denoted by 푿 = (푋, 푋,⋯, 푋) . Suppose the rejection region is 푪 such that: (*note: the rejection region can be defined in terms of the sample points directly instead of the test statistic values) We reject 퐻 if 푿 ∈ 푪 We fail to reject 퐻 if 푿 ∈ 푪 Definition: The size or significance level of the test is defined as: 휶 = 퐦퐚퐱 푷(푿 ∈ 푪) = 퐦퐚퐱 푷휽(푹풆풋풆풄풕 퐻|휃 ∈ 휔) ∈ Definition: The power of a test is given by ퟏ − 휷 = 휸(휃) = 푷휽(푹풆풋풆풄풕 퐻|휃 ∈ 휔) Definition: A most powerful test corresponds to the best rejection region 푪 such that in testing a simple null hypothesis 퐻: 휃 = 휃 versus a simple alternative hypothesis 퐻: 휃 = 휃 , this test has the largest possible test power (ퟏ − 휷) among all tests of size 휶. 3 Theorem (Neyman-Pearson Lemma). The likelihood ratio test for a simple null hypothesis 퐻: 휃 = 휃 versus a simple alternative hypothesis 퐻: 휃 = 휃 is a most powerful test. Jerzy Neyman (left) and Egon Pearson (right) Definition: A uniformly most powerful test in testing a simple null hypothesis 퐻: 휃 = 휃 versus a composite alternative hypothesis 퐻: 휃 ∈ 휔, or in testing a composite null hypothesis 퐻: 휃 ∈ 휔 versus a composite alternative 퐻: 휃 ∈ 휔 , is a size 휶 test such that this test has the largest possible test power 휸(휃) for every simple alternative hypothesis: 퐻: 휃 = 휃 ∈ 휔, among all such tests of size 휶. Now we derive the likelihood ratio test for the first one- sided hypotheses. ( ) Example 2. Let푓(푥 |휇) = 푒 , 푓표푟 푖 = 1,⋯, 푛, where √ 휎 is known. Derive the likelihood ratio test for the hypothesis 퐻: 휇 = 휇 푣푠. 퐻: 휇 > 휇, and show whether it is a UMP test or not. Solution: 흎 = {휇: 휇 = 휇} 훀 = {휇: 휇 ≥ 휇} Note: Now that we are not just dealing with the two-sided hypothesis, it is important to know that the more general definition of 흎 is the set of all unknown parameter values under 퐻, while 훀 is the set of all unknown parameter values under the union of 퐻 and 퐻. 4 Therefore we have: ⎧ 퐿 = 푓(푥 , 푥 ,⋯ , 푥 |휇 = 휇 ) = 푓(푥 |휇 = 휇 ) ⎪ ⎪ ( ) ⎪ 1 ∑ ( ) = 푒 = (2휋휎) 푒 ⎪ √2휋휎 ⎨ ⎪ 퐿 = 푓(푥, 푥,⋯ , 푥|휇 ≥ 휇) = 푓(푥|휇 ≥ 휇) ⎪ ⎪ ( ) 1 ∑ ( ) ⎪ = 푒 = (2휋휎) 푒 √2휋휎 ⎩ Since there is no free parameter in 퐿 , ∑() sup(퐿) = (2휋휎 ) 푒 . 휕푙표푔퐿 푛 ⎧ = (푥̅ − 휇) ⎪ 휕휇 휎 ∵ ,∴ sup(퐿) ⎨휕 푙표푔퐿 푛 ⎪ =− <0 ⎩ 휕휇 휎 ∑ ( ̅) (2휋휎) 푒 , 푖푓 푥̅ > 휇 = . ∑() (2휋휎 ) 푒 , 푖푓 푥̅ ≤ 휇 ( ̅ ) sup(퐿) ∴ 퐿푅 = = 푒 , 푖푓 푥̅ > 휇. sup(퐿 ) 1, 푖푓 푥̅ ≤ 휇 푋 − 휇 ∗ ∗ ∴ 푃(퐿푅 ≤ 푐 |퐻) = 훼 ⇔ 푃 푍 = ≥ −2푙표푔푐 퐻 휎⁄√푛 = 훼. ∴ Reject 퐻 in favor of 퐻 if 푍 = ≥ 푍 ⁄√ Furthermore, it is easy to show that for each 휇 > 휇 , the likelihood ratio test of 퐻: 휇 = 휇 푣푠. 퐻: 휇 = 휇 rejects the null hypothesis at the significance level α for 푍 = ≥ 푍 ⁄√ By the Neyman-Pearson Lemma, the likelihood ratio test is also the most powerful test. Now since for each 1 0 , the most powerful size α test of 퐻: 휇 = 휇 푣푠. 퐻: 휇 = 휇 rejects the null hypothesis for 푍 = ≥ 푍 . Since this same test function is most powerful ⁄√ for each 1 0 , this test is UMP for 퐻: 휇 = 휇 푣푠. 퐻: 휇 > 휇. 5 D. Monotone Likelihood Ratio, the Karlin-Rubin Theorem, and the Exponential Family. A condition under which the UMP tests exists is when the family of distributions being considered possesses a property called monotone likelihood ratio. Definition: Let 푓(풙|휃) be the joint pdf of the sample 푿 = (푋, 푋,⋯, 푋) . Then 푓(풙|휃) is said to have a monotone likelihood ratio in the statistic 푇(푿) if for any choice 휃 < 휃 of parameter values, the likelihood ratio 푓(풙|휃)/푓(풙|휃) depends on values of the data X only through the value of statistic 푇(푿) and, in addition, this ratio is a monotone function of 푇(푿). Theorem (Karlin-Rubin). Suppose that 푓(풙|휃) has an increasing monotone likelihood ratio for the statistic 푇(푿). Let 훼 and 푘 be chosen so that 휶 = 푷휽(푇(푿) ≥ 풌휶). Then 푪 = {풙: 푇(풙) ≥ 풌휶} is the rejection region (*also called ‘critical region’) for a UMP test for the one-sided tests of: 퐻: 휃 ≤ 휃 versus 퐻: 휃 > 휃. Theorem. For a Regular Exponential Family with only one unknown parameter θ, and a population p.d.f.: 푓(푥; 휃) = 푐(휃) ⋅ 푔(푥) ⋅ exp[푡(푥)푤(휃)] If 푤(휃) is a is an increasing function of 휃, then we have a monotone likelihood ratio in the statistic 푇푋 = ∑ 푡(푋). {*Recall 푇푋 = ∑ 푡(푋)} is also complete sufficient for 휃} Examples of a family with monotone likelihood ratio: For X1,, X n iid Exponential ( ), n X e 1 i n 1 n p(x| 1 ) i1 1 n exp (1 0 ) X i p(x| 0 ) 0 Xi 0 i1 0e i1 6 p(x| ) n 1 X For 1 0 , is an increasing function of i so the p(x| 0 ) i1 n family has monotone likelihood ratio in T(x ) X i . i1 Example 3. Let 푋, 푋,… 푋 ~푁(휇, 휎 ), where 휎 is known (1) Find the LRT for 퐻: 휇 ≤ 휇 푣푠 퐻: 휇 > 휇 (2) Show the test in (1) is a UMP test. Solution: 흎 = {휇: 휇 ≤ 휇} 훀 = {휇: −∞< 휇 <∞} Note: Now that we are not just dealing with the two-sided hypothesis, it is important to know that the more general definition of 흎 is the set of all unknown parameter values under 퐻, while 훀 is the set of all unknown parameter values under the union of 퐻 and 퐻. (1) The ratio of the likelihood is shown as the following, 퐿(휔) 휆(푥)= 퐿(Ω) 퐿(휔) is the maximum likelihood under 퐻: 휇 ≤ 휇 1 / ∑(푥 − 푥̅) ⎧ exp − , 푥̅ < 휇 ⎪ 2휋휎 2휎 퐿(휔) = ⎨ 1 / ∑(푥 − 휇 ) ⎪ exp − , 푥̅ ≥ 휇 ⎩ 2휋휎 2휎 퐿Ω is the maximum likelihood under 퐻 푢푛푖표푛 퐻 1 / ∑(푥 − 푥̅) 퐿Ω = exp − 2휋휎 2휎 The ratio of the maximized likelihood is, 1, 푥̅ < 휇 휆(푥) = (∑(푥 − 휇) − ∑(푥 − 푥̅) ) 푒푥푝 − , 푥̅ ≥ 휇 2휎 ⇒ 1, 푥̅ < 휇 휆(푥) = (̅) 푒 , 푥̅ ≥ 휇 7 By LRT, we reject null hypothesis if 휆(푥) < 푐. Generally, c is chosen to be less than 1.