Chapter 7 Power & Sample Size Calculation Yibi Huang Chapter 7 Power & Sample Size Calculation for CRDs Section 10.3 Power & Sample Size for Factorial Designs Chapter 7 - 1 Power & Sample Size Calculation When proposing an experiment (applying for funding etc), nowadays one needs to show that the proposed sample size (i.e. the number of experiment units) is I neither so small that scientifically interesting effects will be swamped by random variation (i.e., has enough power to reject a false H0) I nor larger than necessary, wasting resources (time & money) Chapter 7 - 2 Errors and Power in Hypothesis Testing I A Type I error occurs when H0 is true but is rejected. I A Type II error occurs when H0 is false but is accepted. I The (significance) level of a test is the chance of making a Type I error, i.e., the chance to reject a H0 when it is true. I The power of the test is the the chance of rejecting H0 when Ha is true: power = 1 − P(making type II errorjH0 is false) = 1 − β = P(correctly reject H0jH0 is false) I A good test should have small test-level and large power. Ha is rejected H0 is rejected (H0 is accepted) (Ha is accepted) p H is true α = P(Type I error) 0 p H0 is false β = P(Type II error) Chapter 7 - 3 Power Calculation for ANOVA For a CRD design with g treatments, recall the means model: yij = µi + "ij and the effects model: yij = µ + αi + "ij ; for i = 1;:::; g, and j = 1;:::; ni : The two models are related via the relationship µi = µ + αi : For CRDs, we use the means model more often. However, for power and sample size calculation, we need to use the effects model and µ must be defined as g n g 1 X Xi 1 X µ = µ = n µ : N i N i i i=1 j=1 i=1 Pg Pg which implies that i=1 ni αi = 0: Here N = i=1 ni is the total number of experimental units. Chapter 7 - 4 Recall the H0 and Ha for the ANOVA F test are H0 : µ1 = ··· = µg v.s. Ha : µi 's not all equal, in terms of the means model, or equivalently in terms of the effects model, they are H0 : α1 = ··· = αg = 0 v.s. Ha : Not all αi 's are 0. MSTrt Recall we reject H0 at level α if the test statistic F = MSE exceeds the critical value Fα,g−1;N−g : So Power = P(Reject H0jHa is true) = P(F > Fα, g−1; N−g ): To find P(F > Fα, g−1; N−g ), we must know the distribution of F . I What is the distribution of F under H0? Fg−1;N−g . I And under Ha? Chapter 7 - 5 MS SS =(g − 1) For F = Trt = Trt , recall in Ch3 we've shown that MSE SSE=(N − g) I no matter under H0 or Ha, it is always true that SSE ∼ χ2 ; and (MSE) = σ2 σ2 N−g E 2 2 I For the numerator, SSTrt /σ has a χg−1 distribution under H0. But under Ha, it has a non-central Chi-square 2 distribution χg−1,δ with non-central parameter Pg n α2 δ = i=1 i i : σ2 Moreover, 2 Xg 2 (SSTrt ) = (g − 1)σ + ni α E i=1 i | {z } δσ2 ( (g − 1)σ2 under H = 0 2 (g − 1 + δ)σ under Ha Chapter 7 - 6 Non-Central F -Distribution Under Ha, it can be shown that MS F = Trt MSE has a non-central F -distribution on degrees of freedom g − 1 and N − g, with non-centrality parameter δ, denoted as Pg n α2 F ∼ F where δ = i=1 i i : g−1;N−g,δ σ2 Recall that g n g 1 X Xi 1 X α = µ − µ, and µ = µ = n µ : i i N i N i i i=1 j=1 i=1 Non-central F -distribution is also right skewed, and the greater the non-centrality parameter δ, the further away the peak of the distribution from 0. Chapter 7 - 7 Example 1: Power Calculation (1) I g = 5 treatment groups group sizes: n1 = n2 = n3 = 5, n4 = 6, n5 = 4 I assume σ = 0:8 I desired significance level α = 0:05 I find the power of the test when Ha is true with µ1 = 1:6; µ2 = 0:6; µ3 = 2; µ4 = 0; µ5 = 1: Solution. The grand mean µ is Pg n µ 5 × 1:6 + 5 × 0:6 + 5 × 2 + 6 × 0 + 4 × 1 µ = i=1 i i = N 5 + 5 + 5 + 6 + 4 25 = = 1 25 The αi 's are thus αi = µi − µ = µi − 1: α1 = 0:6; α2 = −0:4; α3 = 1; α4 = −1; α5 = 0: Chapter 7 - 8 Example 1: Power Calculation (2) The non-centrality parameter is Pg n α2 5 × 0:62 + 5 × (−0:4)2 + 5 × 12 + 6 × (−1)2 + 4 × 02 δ = i=1 i i = σ2 0:82 13:6 = = 21:25 0:64 So ( MS F = F under H F = Trt ∼ g−1; N−g 4; 25−5 0 MSE Fg−1; N−g,δ = F4; 25−5; 21:25 under Ha 0.8 H0: F ∼ F4,20 0.4 Ha: F ∼ F4,20,21.25 0.0 df(x0, df1 = 4, df2 20) 0 5 10 15 Chapter 7 - 9 Example 1: Power Calculation (3) The critical value to reject H0 keeping the significance level at α = 0:05 is F0:05;4;20 ≈ 2:866: > qf(0.05,4,20, lower.tail=F) [1] 2.866081 When H0 is true, the chance that H0 is rejected is only 0.05 (the red shaded area.) Accept H0 Reject H0 0.8 0.4 Area = 0.05 0.0 df(x0, df1 = 4, df2 20) 0 F0.05,4,20 5 10 15 Reminder: Don't confuse the following two: I in most STAT books, Fα,df 1;df 2 means the area of the upper tail is α: I in R, qf(alpha, df1, df2) means the the area of the lower Chapter 7 - 10 tail is α: Example 1: Power Calculation (4) If Ha is true and µ1 = 1:6; µ2 = 0:6; µ3 = 2; µ4 = 0; µ5 = 1; we know then F ∼ F4;20,δ=21:25. The power to reject H0 is the area under the density of F4;20,δ=21:25 beyond the critical value (the blue shaded area,) which is 0.9249. > pf(qf(.95,4,20),4,20, ncp=21.25, lower.tail=F) [1] 0.9249342 In the R codes, ncp means the \non-centrality parameter." Accept H0 Reject H0 0.8 0.4 Power 0.0 df(x0, df1 = 4, df2 20) 0 F0.05,4,20 5 10 15 Chapter 7 - 11 Example 1: Power Calculation (5) Remark The power of a test is not a single value, but a function of the parameters in Ha. If parameter µi 's change their values, the power of the test also changes. It makes no sense to talk about the power of a test without specifying the parameters in Ha Chapter 7 - 12 Power Of A Test Is Affected By ... The larger the non-centrality parameter Pg n α2 δ = i=1 i i ; σ2 the greater the power. The power of a test will increase if I the number of replicate ni per treatment increases I the magnitude of treatment effects αi 's increase (under the P constraint i ni αi = 0) 2 I the size of noise σ decreases. Chapter 7 - 13 Example 2: Sample Size Calculation (1) I g = 5 treatment groups of equal sample size ni = n for all i I assume σ = 0:8 I desired significance level α = 0:05 I Assuming the design is balanced that all groups have identical sample size n, what is the minimal sample size n per treatment to have power 0:95 when α1 = 0:5; α2 = −0:5; α3 = 1; α4 = −1; α5 = 0? Solution. The non-centrality parameter is Pg n α2 n δ = i=1 i i = [0:52 + (−0:5)2 + 12 + (−1)2 + 02] σ2 0:82 n = × 2:5 = 3:9n 0:82 So ( MS F = F under H F = Trt ∼ g−1; N−g 4; 5n−5 0 MSE Fg−1; N−g,δ = F4; 5n−5; 3:9n under Ha Chapter 7 - 14 ∗ Recall the critical value F for rejecting H0 at level α = 0:05 is ∗ F = Fα, g−1; N−g = F0:05; 4; 5n−5 which can be find in R via the command > F.crit = qf(alpha, g-1, N-g, lower.tail=F) # syntax > F.crit = qf(0.05, 4, 5*n-5, lower.tail=F) # sub-in the values By definition, Power = P(reject H0 jHa is true) = P(the non central F statistic ≥ F ∗) = P(F (g − 1; N − g; δ) ≥ F ∗) = P(F (4; 5n − 5; 3:9n) ≥ F ∗) which can be found in R via the command > pf(F.crit, g-1, N-g, ncp=delta, lower.tail=F) # syntax > pf(F.crit, 4, 5*n-5, ncp=3.9*n, lower.tail=F) # sub-in values In the R codes, ncp means the \non-centrality parameter." Chapter 7 - 15 Now we find the R code to find the power of the ANOVA F -test when n is known.