Chapter 7 Power & Sample Size Calculation

Yibi Huang

Chapter 7 Power & Sample Size Calculation for CRDs Section 10.3 Power & Sample Size for Factorial Designs

Chapter 7 - 1 Power & Sample Size Calculation

When proposing an experiment (applying for funding etc), nowadays one needs to show that the proposed sample size (i.e. the number of experiment units) is

I neither so small that scientiﬁcally interesting eﬀects will be swamped by random variation (i.e., has enough power to reject a false H0) I nor larger than necessary, wasting resources (time & money)

Chapter 7 - 2 Errors and Power in Hypothesis Testing

I A Type I error occurs when H0 is true but is rejected.

I A Type II error occurs when H0 is false but is accepted.

I The (signiﬁcance) level of a test is the chance of making a Type I error, i.e., the chance to reject a H0 when it is true.

I The power of the test is the the chance of rejecting H0 when Ha is true:

power = 1 − P(making type II error|H0 is false) = 1 − β

= P(correctly reject H0|H0 is false)

I A good test should have small test-level and large power.

Ha is rejected H0 is rejected (H0 is accepted) (Ha is accepted) √ H is true α = P(Type I error) 0 √ H0 is false β = P(Type II error)

Chapter 7 - 3 Power Calculation for ANOVA For a CRD design with g treatments, recall

the means model: yij = µi + εij

and the eﬀects model: yij = µ + αi + εij ,

for i = 1,..., g, and j = 1,..., ni . The two models are related via the relationship

µi = µ + αi . For CRDs, we use the means model more often. However, for power and sample size calculation, we need to use the eﬀects model and µ must be deﬁned as g n g 1 X Xi 1 X µ = µ = n µ . N i N i i i=1 j=1 i=1 Pg Pg which implies that i=1 ni αi = 0. Here N = i=1 ni is the total number of experimental units. Chapter 7 - 4 Recall the H0 and Ha for the ANOVA F test are

H0 : µ1 = ··· = µg v.s. Ha : µi ’s not all equal, in terms of the means model, or equivalently in terms of the eﬀects model, they are

H0 : α1 = ··· = αg = 0 v.s. Ha : Not all αi ’s are 0.

MSTrt Recall we reject H0 at level α if the test statistic F = MSE exceeds the critical value Fα,g−1,N−g . So

Power = P(Reject H0|Ha is true) = P(F > Fα, g−1, N−g ).

To ﬁnd P(F > Fα, g−1, N−g ), we must know the distribution of F .

I What is the distribution of F under H0? Fg−1,N−g .

I And under Ha?

Chapter 7 - 5 MS SS /(g − 1) For F = Trt = Trt , recall in Ch3 we’ve shown that MSE SSE/(N − g) I no matter under H0 or Ha, it is always true that SSE ∼ χ2 , and (MSE) = σ2 σ2 N−g E 2 2 I For the numerator, SSTrt /σ has a χg−1 distribution under H0. But under Ha, it has a non-central Chi-square 2 distribution χg−1,δ with non-central parameter

Pg n α2 δ = i=1 i i . σ2 Moreover, 2 Xg 2 (SSTrt ) = (g − 1)σ + ni α E i=1 i | {z } δσ2 ( (g − 1)σ2 under H = 0 2 (g − 1 + δ)σ under Ha

Chapter 7 - 6 Non-Central F -Distribution Under Ha, it can be shown that MS F = Trt MSE has a non-central F -distribution on degrees of freedom g − 1 and N − g, with non-centrality parameter δ, denoted as

Pg n α2 F ∼ F where δ = i=1 i i . g−1,N−g,δ σ2 Recall that

g n g 1 X Xi 1 X α = µ − µ, and µ = µ = n µ . i i N i N i i i=1 j=1 i=1

Non-central F -distribution is also right skewed, and the greater the non-centrality parameter δ, the further away the peak of the distribution from 0. Chapter 7 - 7 Example 1: Power Calculation (1)

I g = 5 treatment groups group sizes: n1 = n2 = n3 = 5, n4 = 6, n5 = 4 I assume σ = 0.8 I desired signiﬁcance level α = 0.05 I ﬁnd the power of the test when Ha is true with

µ1 = 1.6, µ2 = 0.6, µ3 = 2, µ4 = 0, µ5 = 1. Solution. The grand mean µ is Pg n µ 5 × 1.6 + 5 × 0.6 + 5 × 2 + 6 × 0 + 4 × 1 µ = i=1 i i = N 5 + 5 + 5 + 6 + 4 25 = = 1 25

The αi ’s are thus αi = µi − µ = µi − 1:

α1 = 0.6, α2 = −0.4, α3 = 1, α4 = −1, α5 = 0. Chapter 7 - 8 Example 1: Power Calculation (2) The non-centrality parameter is

Pg n α2 5 × 0.62 + 5 × (−0.4)2 + 5 × 12 + 6 × (−1)2 + 4 × 02 δ = i=1 i i = σ2 0.82 13.6 = = 21.25 0.64 So ( MS F = F under H F = Trt ∼ g−1, N−g 4, 25−5 0 MSE Fg−1, N−g,δ = F4, 25−5, 21.25 under Ha 0.8

H0: F ∼ F4,20 0.4

Ha: F ∼ F4,20,21.25 0.0 df(x0, df1 = 4, df2 20) 0 5 10 15

Chapter 7 - 9 Example 1: Power Calculation (3) The critical value to reject H0 keeping the signiﬁcance level at α = 0.05 is F0.05,4,20 ≈ 2.866. > qf(0.05,4,20, lower.tail=F) [1] 2.866081

When H0 is true, the chance that H0 is rejected is only 0.05 (the red shaded area.)

Accept H0 Reject H0 0.8 0.4 Area = 0.05 0.0 df(x0, df1 = 4, df2 20) 0 F0.05,4,20 5 10 15 Reminder: Don’t confuse the following two: I in most STAT books, Fα,df 1,df 2 means the area of the upper tail is α. I in R, qf(alpha, df1, df2) means the the area of the lower Chapter 7 - 10 tail is α. Example 1: Power Calculation (4) If Ha is true and

µ1 = 1.6, µ2 = 0.6, µ3 = 2, µ4 = 0, µ5 = 1,

we know then F ∼ F4,20,δ=21.25. The power to reject H0 is the area under the density of F4,20,δ=21.25 beyond the critical value (the blue shaded area,) which is 0.9249. > pf(qf(.95,4,20),4,20, ncp=21.25, lower.tail=F) [1] 0.9249342 In the R codes, ncp means the “non-centrality parameter.”

Accept H0 Reject H0 0.8

0.4 Power 0.0 df(x0, df1 = 4, df2 20) 0 F0.05,4,20 5 10 15 Chapter 7 - 11 Example 1: Power Calculation (5)

Remark The power of a test is not a single value, but a function of the parameters in Ha. If parameter µi ’s change their values, the power of the test also changes. It makes no sense to talk about the power of a test without specifying the parameters in Ha

Chapter 7 - 12 Power Of A Test Is Aﬀected By ...

The larger the non-centrality parameter

Pg n α2 δ = i=1 i i , σ2 the greater the power. The power of a test will increase if

I the number of replicate ni per treatment increases

I the magnitude of treatment eﬀects αi ’s increase (under the P constraint i ni αi = 0) 2 I the size of noise σ decreases.

Chapter 7 - 13 Example 2: Sample Size Calculation (1)

I g = 5 treatment groups of equal sample size ni = n for all i I assume σ = 0.8 I desired signiﬁcance level α = 0.05 I Assuming the design is balanced that all groups have identical sample size n, what is the minimal sample size n per treatment to have power 0.95 when

α1 = 0.5, α2 = −0.5, α3 = 1, α4 = −1, α5 = 0? Solution. The non-centrality parameter is Pg n α2 n δ = i=1 i i = [0.52 + (−0.5)2 + 12 + (−1)2 + 02] σ2 0.82 n = × 2.5 = 3.9n 0.82 So ( MS F = F under H F = Trt ∼ g−1, N−g 4, 5n−5 0 MSE Fg−1, N−g,δ = F4, 5n−5, 3.9n under Ha Chapter 7 - 14 ∗ Recall the critical value F for rejecting H0 at level α = 0.05 is ∗ F = Fα, g−1, N−g = F0.05, 4, 5n−5 which can be ﬁnd in R via the command

> F.crit = qf(alpha, g-1, N-g, lower.tail=F) # syntax > F.crit = qf(0.05, 4, 5*n-5, lower.tail=F) # sub-in the values

By deﬁnition,

Power = P(reject H0 |Ha is true) = P(the non central F statistic ≥ F ∗) = P(F (g − 1, N − g, δ) ≥ F ∗) = P(F (4, 5n − 5, 3.9n) ≥ F ∗) which can be found in R via the command

> pf(F.crit, g-1, N-g, ncp=delta, lower.tail=F) # syntax > pf(F.crit, 4, 5*n-5, ncp=3.9*n, lower.tail=F) # sub-in values

In the R codes, ncp means the “non-centrality parameter.” Chapter 7 - 15 Now we find the R code to find the power of the ANOVA F -test when n is known. Let’s plug in different values of n and see what is the smallest n to make power ≥ 0.95. > F.crit = qf(alpha, g-1, N-g, lower.tail=F) > pf(F.crit, g-1, N-g, ncp=delta, lower.tail=F) # syntax

> n = 5 > F.crit = qf(0.05, 4, 5*n-5, lower.tail=F) > pf(F.crit, 4, 5*n-5, ncp=3.9*n, lower.tail=F) [1] 0.8994675 # less than 0.95, not high enough

> n = 6 > F.crit = qf(0.05, 4, 5*n-5, lower.tail=F) > pf(F.crit, 4, 5*n-5, ncp=3.9*n, lower.tail=F) [1] 0.9578791 # greater than 0.95, bingo! So we need 6 replicates in each of the 5 treatment groups to ensure a power of 0.95 when

α1 = 0.5, α2 = −0.5, α3 = 1, α4 = −1, α5 = 0.

Remark: Again, we have to specify the Ha fully to to ﬁnd the appropriate sample size. Chapter 7 - 16 But σ2 is Unknown . . .

As σ2 is usually unknown, here are a few ways to make a guess.

I Make a small-sample pilot study to get an estimate of σ.

I Based on prior studies or knowledge about the experimental units, can you think of a range of plausible values for σ? If so, choose the biggest one.

I You could repeat the sample size calculations for various levels of σ to see how it aﬀects the needed sample size.

Chapter 7 - 17 How to Specify the Ha?

As the power of a test depends on the alternative hypothesis Ha, that is, the αi ’s, one might has to try several sets of αi ’s to ﬁnd the appropriate sample size. But how many Ha’s we have to try?

Here is a useful trick. 1. Suppose we would be interested if any two means diﬀered by D or more. 2. The smallest value of δ in this case is when two means diﬀer by exactly, D, and the other g − 2 means are halfway between.

3. So try α1 = D/2, α2 = −D/2, and αi = 0 for all other τ. Assuming equal sample sizes, the non-centrality parameter is

X nα2 n(D2/4 + D2/4) nD2 δ = i = = . σ2 σ2 2σ2 i

Chapter 7 - 18 Power and Sample Size Calculation for Complete Block Designs For complete block designs

yij = µ + αi + βj + εij RCBD

yijk = µ + αi + βj + γk + εijk Latin Square

yijk` = µ + αi + βj + γk + δ` + εijk` Graeco-Latin Square

where αi ’s are the treatment effect, and βj , γk , δ` are effects of blocking factors, when Ha is true, treatments have different effects, the ANOVA F -statistic also has a non-central F -distribution ( MS F under H F = trt ∼ g−1, df of MSE 0 MSE Fg−1, df of MSE,δ under Ha where the non-centrality parameter δ is r P α2 δ = i i , where r = # of replicates per treatment σ2 Note that the df changes from N − g to the df of MSE. Chapter 7 - 19 Power and Sample Size Calculation for Balanced Factorial Designs

I Section 10.3 in Oehlert’s book

I We may ignore factorial structure and compute power and sample size for the overall null hypothesis of no model eﬀects.

I We will address methods for balanced data only because most factorial experiments are designed to be balanced, and simple formulae Power for balanced data for noncentrality parameters exist only for balanced data.

I For factorial data, we usually test H0 about main eﬀects or interactions in addition to the overall H0 of no model eﬀects.

Chapter 7 - 20 Non-Centrality Parameter for Balanced Factorial Designs

Ex, consider a 3-way factorial design

yijk` = µ + αi + βj + γk + αβij + βγjk + αγij + αβγijk + εijk`

to calculate the power of the F -test of a main eﬀect or an interaction eﬀect under Ha, the non-centrality parameter can be calculated as follows:

I Square the factorial eﬀects (using the zero-sum constraint) and sum them,

I Multiply this sum by the total number of data in the design divided by the number of levels in the eﬀect, and

I Divide that product by the error variance.

Chapter 7 - 21 E.g., consider a a × b × c factorial design with r replicates per treatment

yijk` = µ + αi + βj + γk + αβij + βγjk + αγij + αβγijk + εijk`

To test about the B main eﬀects

H0 : β1 = ··· = βb = 0 v.s. Ha : not all βj are 0 the ANOVA F -statistic for B main-eﬀect is ( MS F under H F = B ∼ b−1, df of MSE 0 MSE Fb−1, df of MSE,δ under Ha where the non-centrality parameter δ is

N X X δ = β2 σ2 = rac β2 σ2. b j j j j

Here N = abcr is the total number of observations. Chapter 7 - 22 E.g., consider a a × b × c factorial design with r replicates per treatment

yijk` = µ + αi + βj + γk + αβij + βγjk + αγij + αβγijk + εijk`

To test about the AB interactions

H0 : all αβij = 0 v.s. Ha : not all αβij = 0 the ANOVA F -statistic for AB interactions ( MS F under H F = AB ∼ (a−1)(b−1), df of MSE 0 MSE F(a−1)(b−1), df of MSE,δ under Ha where the non-centrality parameter δ is N X 2 2 X 2 2 δ = (αβij ) σ = rc (αβij ) σ . ab ij ij

Here N = abcr is the total number of observations. Chapter 7 - 23 Sample Size Should Be The Last Thing To Consider

In practice, sample size calculation is the last thing to do in designing an experiment. Consider the followings before calculating the required sample size.

I asking the right question,

I choosing an appropriate response,

I thinking about what factors need to be blocked on,

I choosing a right design,

I setting up the right procedure to recruit subjects, and so on, All the above are more important than doing the right sample size calculation! Sometimes, choosing a better design can substantially reduces the required sample size.

Chapter 7 - 24