Last Lecture

. Describe the following concepts in your own words 602 - • Hypothesis Lecture 19 • Null Hypothesis Likelihood Ratio Test • . • Hypothesis Testing Procedure • Rejection Region Hyun Min Kang • Type I error • Type II error March 26th, 2013 • Power function • Size α test • Level α test • Likleihood Ratio Test

Hyun Min Kang Biostatistics 602 - Lecture 19 March 26th, 2013 1 / 1 Hyun Min Kang Biostatistics 602 - Lecture 19 March 26th, 2013 2 / 1

Example of Hypothesis Testing Power function

. Let X , , Xn be changes in blood pressure after a treatment. Definition - The power function 1 ··· . The power function of a hypothesis test with rejection region R is the H 0 : θ = 0 function of θ defined by H1 : θ = 0 β(θ) = Pr(X R θ) = Pr(reject H θ) ̸ ∈ | 0| x c The rejection region = x : > 3 . If θ Ω (alternative is true), the probability of rejecting H0 is called the sX/√n ∈ 0 { } .power of test for this particular value of θ. Decision Accept H Reject H • Probability of type I error = β(θ) if θ Ω . 0 0 ∈ 0 Truth H0 Correct Decision Type I error • Probability of type II error = 1 β(θ) if θ Ωc. − ∈ 0 H1 Type II error Correct Decision An ideal test should have power function satisfying β(θ) = 0 for all θ Ω , β(θ) = 1 for all θ Ωc, which is typically not possible in practice. ∈ 0 ∈ 0

Hyun Min Kang Biostatistics 602 - Lecture 19 March 26th, 2013 3 / 1 Hyun Min Kang Biostatistics 602 - Lecture 19 March 26th, 2013 4 / 1 Sizes and Levels of Tests Likelihood Ratio Tests (LRT)

. Size α test . . A test with power function β(θ) is a size α test if Definition sup β(θ) = α . θ Ω0 Let L(θ x) be the of θ. The likelihood ratio test ∈ | for testing H : θ Ω vs. H : θ Ωc is 0 ∈ 0 1 ∈ 0 .In other words, the maximum probability of making a type I error is α. sup L x ˆ θ Ω0 (θ ) L(θ0 x) λ(x) = ∈ | = | . supθ Ω L(θ x) L(θˆ x) Level α test ∈ | | . A test with power function β(θ) is a level α test if where θˆ is the MLE of θ over θ Ω, and θˆ0 is the MLE of θ over θ Ω0 sup β(θ) α (restricted MLE). ∈ ∈ θ Ω0 ≤ ∈ The likelihood ratio test is a test that rejects H if and only if λ(x) c 0 ≤ .where 0 c 1. In other words, the maximum probability of making a type I error is equal ≤ ≤ .or less than α. Any size α test is also a level α test

Hyun Min Kang Biostatistics 602 - Lecture 19 March 26th, 2013 5 / 1 Hyun Min Kang Biostatistics 602 - Lecture 19 March 26th, 2013 6 / 1

Example of LRT MLE of θ over Ω = ( , ) −∞ ∞ . Problem . i.i.d. 2 2 n 2 Consider X1, , Xn (θ, σ ) where σ is known. i=1(xi θ) ··· ∼ N To maximize L(θ x), we need to maximize exp 2− , or H : θ θ | − 2σ 0 ≤ 0 n 2 ∑ equivalently to minimize i=1(xi θ) . [ ] H1 : θ > θ0 − n ∑ n 2 2 2 .For the LRT test and its power function (xi θ) = (x + θ 2θxi) − i − . i=1 i=1 Solution ∑ ∑ n n . 2 2 n 2 = nθ 2θ xi + xi 1 (xi θ) − L(θ x) = exp − i=1 i=1 | √ 2 − 2σ2 ∑ ∑ i=1 2πσ n [ ] xi ∏ n n 2 The equation above minimizes when θ = θˆ = i=1 = x. 1 (xi θ) n = exp i=1 − ∑ √ 2 − 2σ2 ( 2πσ ) [ ∑ ] We need to find MLE of θ over Ω = ( , ) and Ω = ( , θ ]. . −∞ ∞ 0 −∞ 0 Hyun Min Kang Biostatistics 602 - Lecture 19 March 26th, 2013 7 / 1 Hyun Min Kang Biostatistics 602 - Lecture 19 March 26th, 2013 8 / 1 MLE of θ over Ω = ( , θ ] Likelihood ratio test 0 −∞ 0

n 1 if X θ0 xi • i=1 ˆ n (x θ )2 ≤ L(θ x) is maximized at θ = n = x if x θ0. L(θ0 x) exp i=1 i− 0 | ∑ ≤ λ(x) = | =  − 2σ2 ˆ [ ∑ ] if X > θ • However, if x θ , x does not fall into a valid of θˆ , and L(θ x)  n (x x)2 0 0 0  i=1 i− ≥ |  exp 2 − ∑ 2σ θ θ0, the likelihood function will be an increasing function. [ ] ≤ ˆ Therefore θ0 = θ0.  1 if X θ0  2 ≤ = n(x θ0) To summarize, exp − 2 if X > θ0 { − 2σ [ ] X if X θ0 θˆ0 = ≤ θ if X > θ Therefore, the likelihood test rejects the null hypothesis if and only if { 0 0 n(x θ )2 exp − 0 c − 2σ2 ≤ [ ] and x θ . ≥ 0 Hyun Min Kang Biostatistics 602 - Lecture 19 March 26th, 2013 9 / 1 Hyun Min Kang Biostatistics 602 - Lecture 19 March 26th, 2013 10 / 1

Specifying c Specifying c (cont’d)

So, LRT rejects H0 if and only if

2 2 n(x θ0) 2σ log c exp − c x θ0 − 2σ2 ≤ − ≥ − n [ ] √ 2 2σ2 log c n(x θ0) − log c x θ0 n 2σ2 − − = c∗ ⇐⇒ − ≤ ⇐⇒ σ/√n ≥ √ σ/√n 2σ2 log c (x θ )2 ⇐⇒ − 0 ≥ − n Therefore, the rejection region is 2σ2 log c x θ0 (∵ x > θ0) x θ0 ⇐⇒ − ≥ − n x : − c∗ √ σ/√n ≥ { }

Hyun Min Kang Biostatistics 602 - Lecture 19 March 26th, 2013 11 / 1 Hyun Min Kang Biostatistics 602 - Lecture 19 March 26th, 2013 12 / 1 Power function Making size α LRT

X θ0 To make a size α test, β(θ) = Pr (reject H ) = Pr − c∗ 0 σ/√n ≥ ( ) sup β(θ) = α X θ + θ θ0 = Pr − − c∗ θ Ω0 σ/√n ≥ ∈ ( ) θ0 θ sup Pr Z − + c∗ = α X θ θ0 θ ≥ σ/√n = Pr − − + c∗ θ θ0 ( ) σ/√n ≥ σ/√n ≤ ( ) Pr (Z c∗) = α ≥ i.i.d. 2 σ2 Since X , , Xn (θ, σ ), X θ, . Therefore, c∗ = zα 1 ··· ∼ N ∼ N n

( ) θ0 θ X θ Note that Pr Z − + c is maximized when θ is maximum (i.e. − (0, 1) ≥ σ/√n ∗ σ/√n ∼ N θ = θ0). ( ) x θ0 θ0 θ Therefore, size α LRT test rejects H0 if and only if − zα. = β(θ) = Pr Z − + c∗ σ/√n ≥ ⇒ ≥ σ/√n ( ) where Z (0, 1). ∼ N Hyun Min Kang Biostatistics 602 - Lecture 19 March 26th, 2013 13 / 1 Hyun Min Kang Biostatistics 602 - Lecture 19 March 26th, 2013 14 / 1

Another Example of LRT Solution (cont’d) . Problem . i.i.d. (x θ) c X , , Xn f(x θ) = e where x θ and < θ < . Find a When θ Ω , the likelihood is still an increasing function, but bounded by 1 ··· ∼ | − − ≥ −∞ ∞ ∈ 0 LRT testing the following one-sided hypothesis. θ min(x , θ ). Therefore, the likelihood is maximized when ≤ (1) 0 H0 : θ θ0 θ = θˆ = min(x , θ ). The likelihood ratio test statistic is ≤ 0 (1) 0 . H1 : θ > θ0 x +nθ e− i 0 x +nx if θ0 < x(1) . λ(x) = e− ∑ i (1) Solution ∑ . { 1 if θ0 x(1) n ≥ n(θ0 x(1)) (xi θ) e − if θ0 < x(1) L(θ x) = e− − I(xi θ) = | ≥ 1 if θ x i 0 (1) ∏=1 { ≥ xi+nθ = e− I(θ x(1)) ∑ ≤ The likelihood function is a increasing function of θ, bounded by θ x(1). ˆ ≤ Therefore, when θ Ω = R, L(θ x) is maximized when θ = θ = x(1). . ∈ |

Hyun Min Kang Biostatistics 602 - Lecture 19 March 26th, 2013 15 / 1 Hyun Min Kang Biostatistics 602 - Lecture 19 March 26th, 2013 16 / 1 Solution (cont’d) LRT based on sufficient

The LRT rejects H0 if and only if

n(θ0 x ) e − (1) c (and θ < x ) ≤ 0 (1) log c θ0 x . ⇐⇒ − (1) ≤ n Theorem 8.2.4 log c . x θ If T(X) is a sufficient statistic for θ, λ (t) is the LRT statistic based on T, (1) 0 n ∗ ⇐⇒ ≥ − and λ(x) is the LRT statistic based on x then log c So, LRT reject H is x θ and x > θ . The power function is λ∗[T(x)] = λ(x) 0 (1) ≥ 0 − n (1) 0 log c for every x in the space. β(θ) = Pr X θ X > θ . (1) ≤ 0 − n ∧ (1) 0 ( ) To find size α test, we need to find c satisfying the condition sup β(θ) = α θ θ0 ≤

Hyun Min Kang Biostatistics 602 - Lecture 19 March 26th, 2013 17 / 1 Hyun Min Kang Biostatistics 602 - Lecture 19 March 26th, 2013 18 / 1

Proof Proof (cont’d)

LRT statistic based on X is By Factorization Theorem, the joint pdf of x can be written as sup L(θ x) sup f(x θ) f(x θ) = g(T(x) θ)h(x) θ Ω0 θ Ω0 | | λ(x) = ∈ | = ∈ | supθ Ω L(θ x) supθ Ω f(x θ) ∈ | ∈ | and we can choose g(t θ) to be the pdf or pmf of T(x). sup g(T(x) θ)h(x) | θ Ω0 Then, the LRT statistic based on T(X) is defined as = ∈ | supθ Ω g(T(x) θ)h(x) ∈ | sup g(T(x) θ) sup L(θ T(x) = t) sup g(t θ) θ Ω0 θ Ω0 θ Ω0 = ∈ | = λ∗(T(x)) λ∗(t) = ∈ | = ∈ | supθ Ω g(T(x) θ) supθ Ω L(θ T(x) = t) supθ Ω g(t θ) ∈ | ∈ | ∈ | The simplified expression of λ(x) should depend on x only through T(x), where T(x) is a sufficient statistic for θ.

Hyun Min Kang Biostatistics 602 - Lecture 19 March 26th, 2013 19 / 1 Hyun Min Kang Biostatistics 602 - Lecture 19 March 26th, 2013 20 / 1 Example Solution (cont’d) . Problem . The numerator is fixed, and MLE in the denominator is θˆ = t. Therefore i.i.d. 2 2 Consider X , , Xn (θ, σ ) where σ is known. 1 ··· ∼ N the LRT statistic is H0 : θ = θ0 2 n(t θ0) H1 : θ = θ0 λ(t) = exp − ̸ − 2σ2 [ ] .Find a size α LRT. LRT rejects H if and only if . 0 Solution - Using sufficient statistics . T(X) = X is a sufficient statistic for θ. n(t θ )2 λ(t) = exp − 0 c σ2 − 2σ2 ≤ T θ, [ ] ∼ N n t ( ) θ0 2 = − 2 log c = c∗ 1 (t θ0) ⇒ σ/√n ≥ − sup L t 2 exp −2 θ Ω0 (θ ) 2πσ /n 2σ /n λ(t) = ∈ | = − √ (t θ)2 supθ Ω L(θ t) 1 [ ] supθ Ω 2πσ2/n exp 2σ−2/n . ∈ | ∈ − [ ] Hyun Min Kang Biostatistics 602 - Lecture 19 March 26th, 2013 21 / 1 Hyun Min Kang Biostatistics 602 - Lecture 19 March 26th, 2013 22 / 1

Solution (cont’d) LRT with nuisance parameters

Note that σ2 T = X θ, . ∼ N n Problem ( ) . T i.i.d. 2 2 θ0 X , , Xn (θ, σ ) where both θ and σ unknown. Between − (0, 1) 1 ··· ∼ N σ/√n ∼ N H : θ θ and H : θ > θ . 0 ≤ 0 1 0 A size α test satisfies 1. Specify Ω and Ω0 T θ 2. Find size LRT. sup Pr − c∗ = α . α θ Ω0 σ/√n ≥ ∈ ( ) . T θ0 Solution - Ω and Ω0 Pr − c∗ = α . σ/√n ≥ 2 2 ( ) Ω = (θ, σ ): θ R, σ > 0 Pr ( Z c∗) = α { ∈ } | | ≥ 2 2 . Ω0 = (θ, σ ): θ θ0, σ > 0 Pr(Z c∗) + Pr(Z c∗) = α { ≤ } ≥ ≤ − T θ Z = − z | | σ/√n ≥ α/2

Hyun Min Kang Biostatistics 602 - Lecture 19 March 26th, 2013 23 / 1 Hyun Min Kang Biostatistics 602 - Lecture 19 March 26th, 2013 24 / 1

Solution - Size α LRT Solution - Maximizing Numerator

2 sup 2 2 L x (θ,σ ):θ θ0,σ >0 (θ, σ ) x { ≤ } | λ( ) = 2 sup (θ,σ2):θ R,σ2>0 L(θ, σ x) { ∈ } | 2 n 2 Step 1, fix σ , likelihood is maximized when i=1(xi θ) is minimized For the denominator, the MLE of θ and σ2 are − over θ θ0. ≤ ∑ ˆ X θ = x if x θ0 2 ˆ 2 (Xi X) n 1 2 θ0 = ≤ { σ = n− = −n sX θ0 if x > θ0 ∑ { For numerator, we need to maximize L(θ, σ2 x) over the region θ θ and | ≤ 0 σ2 > 0.

n n 2 1 (xi θ) L(θ, σ2 x) = exp i=1 − | √ 2 − 2σ2 ( 2πσ ) [ ∑ ]

Hyun Min Kang Biostatistics 602 - Lecture 19 March 26th, 2013 25 / 1 Hyun Min Kang Biostatistics 602 - Lecture 19 March 26th, 2013 26 / 1

Solution - Maximizing Numerator (cont’d) Solution - Constructing LRT

Step 2 : Now, we need to maximize likelihood (or log-likelihood) with LRT test rejects H0 if and only if x > θ0 and 2 respect to σ and we substitute θˆ0 for θ. n σˆ2 /2 ˆ 2 c 2 n 2 (xi θ0) 2 l ˆ x log log σˆ0 ≤ (θ, σ ) = 2π + σ −2 ( ) | − 2 − 2σ 2 n/2 2 ∑ (xi x) /n ∂ log l n( (xi θˆ )) c 0 − 2 = + − = 0 (xi θ ) /n ≤ ∂σ2 −2σ2 2(σ2)2 ( ∑ − 0 ) ∑ 2 n 2 (xi x) (xi θˆ ) ∑ − c∗ σˆ2 = i=1 − 0 (x θ )2 ≤ 0 n ∑ i 0 2 − ∑ (xi X) −∑ c∗ Combining the results together 2 2 (xi X) + n(x θ ) ≤ −∑ − 0 1 ∑ 2 c∗ 1 if x θ n(x θ0) ≤ 0 1 + − 2 n/2 ≤ (xi x) λ(x) = σˆ2 − { 2 if x > θ0 ∑ σˆ0 ( ) Hyun Min Kang Biostatistics 602 - Lecture 19 March 26th, 2013 27 / 1 Hyun Min Kang Biostatistics 602 - Lecture 19 March 26th, 2013 28 / 1 Solution - Constructing LRT (cont’d) Unbiased Test

. Definition . If a test always satisfies n(x θ )2 Pr(reject H0 when H0 is false ) Pr(reject H0 when H0 is true ) 0 c ≥ − 2 ∗∗ (xi x) ≥ − .Then the test is said to be unbiased x θ0 ∑ − c∗∗∗ . sX/√n ≥ Alternative Definition . x θ0 Recall that β(θ) = Pr(reject H0). A test is unbiased if LRT test reject if − c∗∗∗ sX/√n ≥ β(θ′) β(θ) The next step is specify c to get size α test (omitted). ≥ for every θ Ωc and θ Ω . . ′ ∈ 0 ∈ 0

Hyun Min Kang Biostatistics 602 - Lecture 19 March 26th, 2013 29 / 1 Hyun Min Kang Biostatistics 602 - Lecture 19 March 26th, 2013 30 / 1

Example Example (cont’d)

i.i.d. 2 2 X , , Xn (θ, σ ) where σ is known, testing H : θ θ vs 1 ··· ∼ N 0 ≤ 0 H1 : θ > θ0. x θ0 LRT test rejects H0 if − > c. σ/√n Therefore, for Z (0, 1) ∼ N X θ0 θ θ β(θ) = Pr − > c β(θ) = Pr Z > c + 0 − σ/√n σ/√n ( ) ( ) X θ + θ θ0 = Pr − − > c Because the power function is increasing function of , σ/√n θ ( ) X θ θ θ = Pr − + − 0 > c β(θ′) β(θ) σ/√n σ/√n ≥ ( ) X θ θ θ always holds when θ θ0 < θ′. Therefore the LRTs are unbiased. = Pr − > c + 0 − ≤ σ/√n σ/√n ( )

2 2 X θ Note that Xi (θ, σ ), X (θ, σ /n), and − (0, 1). ∼ N ∼ N σ/√n ∼ N Hyun Min Kang Biostatistics 602 - Lecture 19 March 26th, 2013 31 / 1 Hyun Min Kang Biostatistics 602 - Lecture 19 March 26th, 2013 32 / 1 Summary

. Today . • Examples of LRT • LRT based on sufficient statistics • LRT with nuisance parameters • Unbiased Test . . Next Lecture . • Uniformly Most Powerful Test .

Hyun Min Kang Biostatistics 602 - Lecture 19 March 26th, 2013 33 / 1