Computing Effect Sizes for Clustered Randomized Trials
Terri Pigott, C2 Methods Editor & Co-Chair Professor, Loyola University Chicago [email protected]
The Campbell Collaboration www.campbellcollaboration.org
Computing effect sizes in clustered trials
• In an experimental study, we are interested in the difference in performance between the treatment and control group • In this case, we use the standardized mean difference, given by YYTC− d = gg Control group sp mean
Treatment group mean Pooled sample standard deviation
Campbell Collaboration Colloquium – August 2011 www.campbellcollaboration.org
1 Variance of the standardized mean difference
NNTC+ d2 Sd2 ()=+ NNTC2( N T+ N C )
where NT is the sample size for the treatment group, and NC is the sample size for the control group
Campbell Collaboration Colloquium – August 2011 www.campbellcollaboration.org
TREATMENT GROUP CONTROL GROUP
TT T CC C YY,,..., Y YY12,,..., YC 12 NT N
Overall Trt Mean Overall Cntl Mean T Y C Yg g S 2 2 Trt SCntl 2 S pooled
Campbell Collaboration Colloquium – August 2011 www.campbellcollaboration.org
2 In cluster randomized trials, SMD more complex
• In cluster randomized trials, we have clusters such as schools or clinics randomized to treatment and control • We have at least two means: mean performance for each cluster, and the overall group mean • We also have several components of variance – the within- cluster variance, the variance between cluster means, and the total variance • Next slide is an illustration
Campbell Collaboration Colloquium – August 2011 www.campbellcollaboration.org
TREATMENT GROUP CONTROL GROUP
Cntl Cluster mC Trt Cluster 1 Trt Cluster mT Cntl Cluster 1 TT T T CC C C YY,...ggg Y ,..., Y YY11,... 1n ggg YCC ,..., Y { 11 1n } { mmnTT1 } { } { mmn1 }
C Trt Cluster 1 Mean Trt Cluster mT Mean Cntl Cluster 1 Mean Cntl Cluster m Mean T T C C Y ••• Y T Y ••• Y C 1g m g 1g m g
Overall Cntl Mean Overall Trt Mean T Y C Ygg gg
Assume equal sample size, n, within clusters
Campbell Collaboration Colloquium – August 2011 www.campbellcollaboration.org
3 The problem in cluster randomized trials
• Which mean is the appropriate mean to represent the differences between the treatment and control groups? – The cluster means? – The overall treatment and control group means averaged across clusters? • Which variance is appropriate to represent the differences between the treatment and control groups? – The within-cluster variance? – The variance among cluster means? – The total variance?
Campbell Collaboration Colloquium – August 2011 www.campbellcollaboration.org
Three different components of variance
• The next slides outline the three components of variance that exist in a cluster randomized trial • These components of variance correspond to the variances that we would estimate when analyzing a cluster randomized trial using hierarchical linear models
Campbell Collaboration Colloquium – August 2011 www.campbellcollaboration.org
4 Pooled within-cluster sample variance
• One component of variance is within-cluster variation • This is computed as the pooled differences between each observation and its cluster mean
mnTC mn ()YYTT22 () YY CC ∑∑ij− igg+ ∑∑ ij− i S 2 = ij==11 ij == 11 W NM−
M is number of clusters: M = mT + mC N is total sample size: N = n mT + n mC = NT + NC with equal cluster sample sizes, n
Campbell Collaboration Colloquium – August 2011 www.campbellcollaboration.org
TREATMENT GROUP CONTROL GROUP
Cntl Cluster mC Trt Cluster 1 Trt Cluster mT Cntl Cluster 1 TT T T CC C C YY,...ggg Y ,..., Y YY11,... 1n ggg YCC ,..., Y { 11 1n } { mmnTT1 } { } { mmn1 }
C Trt Cluster 1 Mean Trt Cluster mT Mean Cntl Cluster 1 Mean Cntl Cluster m Mean T T C C Y ••• Y T Y ••• Y C 1g m g 1g m g
Overall Cntl Mean Overall Trt Mean T Y C Ygg gg
2 Within-cluster variance SW
Campbell Collaboration Colloquium – August 2011 www.campbellcollaboration.org
5 Pooled within-group (treatment and control) sample variance of the cluster means • Another component of variation is the variance among the cluster means within the treatment and control groups • This is computed as the pooled difference between each cluster mean and its overall group (treatment and control) mean
mmTC ()()YYTT22 YY CC ∑∑iiggg−+ ggg− S 2 = ii==11 B mmTC+−2
Campbell Collaboration Colloquium – August 2011 www.campbellcollaboration.org
TREATMENT GROUP CONTROL GROUP
Cntl Cluster mC Trt Cluster 1 Trt Cluster mT Cntl Cluster 1 TT T T CC C C YY,...ggg Y ,..., Y YY11,... 1n ggg YCC ,..., Y { 11 1n } { mmnTT1 } { } { mmn1 }
C Trt Cluster 1 Mean Trt Cluster mT Mean Cntl Cluster 1 Mean Cntl Cluster m Mean T T C C Y ••• Y T Y ••• Y C 1g m g 1g m g
Overall Cntl Mean Overall Trt Mean T Y C Ygg gg
2 Variance between cluster S B means
Campbell Collaboration Colloquium – August 2011 www.campbellcollaboration.org
6 Total pooled within-group (treatment and control) variance • The total variance is the variation among the individual observations and their group (treatment and control) overall means
mnTC mn ()YYTT22 () YY CC ∑∑ij−gg+ ∑∑ ij − gg S 2 = ij==11 ij == 11 T N − 2
Campbell Collaboration Colloquium – August 2011 www.campbellcollaboration.org
TREATMENT GROUP CONTROL GROUP
Cntl Cluster mC Trt Cluster 1 Trt Cluster mT Cntl Cluster 1 TT T T CC C C YY,...ggg Y ,..., Y YY11,... 1n ggg YCC ,..., Y { 11 1n } { mmnTT1 } { } { mmn1 }
C Trt Cluster 1 Mean Trt Cluster mT Mean Cntl Cluster 1 Mean Cntl Cluster m Mean T T C C Y ••• Y T Y ••• Y C 1g m g 1g m g
Overall Cntl Mean Overall Trt Mean T Y C Ygg gg 2 S Total pooled within-group variance T (treatment and control)
Campbell Collaboration Colloquium – August 2011 www.campbellcollaboration.org
7 Just a little distributional theory
• As usual, we will assume that each of our observations within the treatment and control group clusters are normally T C distributed about their cluster means, µ i and µ i with 2 common within-cluster variance, σ W, or
TT2 T YNij~ (µ i ,σ W ), i== 1,..., mjn ; 1,..., CC2 C YNij~ (µ i ,σ W ), i== 1,..., mjn ; 1,...,
Campbell Collaboration Colloquium – August 2011 www.campbellcollaboration.org
Just a little more distributional theory
• We also assume that our clusters are sampled from a population of clusters (making them random effects) so that the cluster means have a normal sampling distribution with T C 2 means µ • and µ • and common variance, σ B
TT2 T µµiB~(Nimg ,σ ),1,...,= CC2 C µµiB~(Nimg ,σ ),1,...,=
Campbell Collaboration Colloquium – August 2011 www.campbellcollaboration.org
8 The three component s of variance are related:
22 2 σσTW=+ σ B
Campbell Collaboration Colloquium – August 2011 www.campbellcollaboration.org
The intraclass correlation, ρ
• This parameter summarizes the relationship among the three variance components • We can think of ρ as the ratio of the between cluster variance and the total variance • ρ is given by
22 σσBB ρ ==22 2 σσBW+ σ T
Campbell Collaboration Colloquium – August 2011 www.campbellcollaboration.org
9 Why is the intraclass correlation important?
• Later, we will see that we need the intraclass correlation to obtain the values of our standardized mean difference and its variance • Also, if we know the intraclass correlation, we can obtain the value of one of the variances knowing the other two. For example:
22 σρσρWB=(1− ) / 22 σρσWT=(1− )
Campbell Collaboration Colloquium – August 2011 www.campbellcollaboration.org
So far, we have
• Discussed the structure of the data • Seen how to compute three different sampling variances: 2 2 2 S W, S B, and S T • Discussed the underlying distributions of our clustered data • Seen how the variance components in the distributions of our data are related to one another • Next we will see how to estimate the variance components using our sampling variances • (Yes, we are building up to the effect sizes…)
Campbell Collaboration Colloquium – August 2011 www.campbellcollaboration.org
10 2 Estimator of σ W
2 • As we might expect, we estimate σ W using our estimate of the within-cluster sampling variance, or,
22 σˆWW= S
Campbell Collaboration Colloquium – August 2011 www.campbellcollaboration.org
2 Estimator of σ B
2 • Unfortunately, the estimate of σ B is not as straightforward as 2 for σ W 2 • This is due to the fact that S B includes some of the within- cluster variance. 2 • The expected value for S B is σ 2 σ 2 + W B n
Campbell Collaboration Colloquium – August 2011 www.campbellcollaboration.org
11 2 Thus, an estimator for σ B is
S 2 σˆ 22=S −W BBn
Campbell Collaboration Colloquium – August 2011 www.campbellcollaboration.org
2 And, given prior information, an estimate of σ T is
n −1 ˆ 22⎛⎞ 2 σTB=+SS⎜⎟ W ⎝⎠n
Campbell Collaboration Colloquium – August 2011 www.campbellcollaboration.org
12 Now, we are ready for computing effect sizes
• There are 3 possibilities for computing an effect size in a clustered randomized trial • These correspond to the 3 different components of variance we have been discussing • The choice among them depends on the research design and the information reported in the other studies in our meta- analysis
Campbell Collaboration Colloquium – August 2011 www.campbellcollaboration.org
The most common: δW
• This effect size is comparable to the standardized mean difference computed from single site designs
TC µµgg− δW = σW
Campbell Collaboration Colloquium – August 2011 www.campbellcollaboration.org
13 We estimate δW using dW
TC Trt & Cntl means YYgg− gg dW = SW
Pooled within-cluster variance
Campbell Collaboration Colloquium – August 2011 www.campbellcollaboration.org
Variance of dW
• Variance of dW depends on the intraclass correlation, sample sizes, and dW
TC 2 ⎛⎞NN++⎛⎞1( n− 1)ρ dW Vd{ W } =+⎜⎟TC ⎜⎟ ⎝⎠NN⎝⎠12()−−ρ N M
When SB is 0 (no between-group variation among the cluster means), the variance of dW is equal to the variance for the standardized mean difference in a single-site study
Campbell Collaboration Colloquium – August 2011 www.campbellcollaboration.org
14 A second estimator: δT
• This estimator uses the total variation in the denominator • We may compute this effect size when the other studies in our meta-analysis are also multi-site studies • The general form is:
TC µµgg− δT = σT
Campbell Collaboration Colloquium – August 2011 www.campbellcollaboration.org
There are two ways to compute δT
• Method 1: dT1 2 – Used when the study reports S B , the variance of the cluster 2 means, and S W , the pooled within-cluster variance
• Method 2: dT2 – Used when we the study reports the intraclass correlation, and 2 S T , the total variance
Campbell Collaboration Colloquium – August 2011 www.campbellcollaboration.org
15 2 2 Estimator dT1 : When we have S B and S W
TC YYgg− gg dT1 = , with σˆT n −1 ˆ 22⎛⎞ σTB=+SS⎜⎟ W ⎝⎠n
Campbell Collaboration Colloquium – August 2011 www.campbellcollaboration.org
The variance of dT1 (a little messy)
⎛⎞NNTC+ Vd{ T1} =+⎜⎟TC (1 ( n− 1)ρ ) + ⎝⎠NN ⎡⎤2 22 [1(+n − 1)ρ] (1)(1)n −−ρ 2 ⎢⎥+ d 2(nM22 2) 2( nNM ) T1 ⎣⎦⎢⎥−−
Campbell Collaboration Colloquium – August 2011 www.campbellcollaboration.org
16 2 Estimator dT2 : When we have S T and ρ
TC ⎛⎞YYgg− gg 2(n− 1)ρ dT 2 =⎜⎟1 − ⎝⎠SNT −2
Campbell Collaboration Colloquium – August 2011 www.campbellcollaboration.org
Variance of dT2 (also messy)
⎛⎞NNTC+ Vd{ T 2} =+⎜⎟TC (1 ( n− 1)ρ ) + ⎝⎠NN ⎛⎞(NnNnNn−− 2)(1ρρρρ )22+ (− 2 )+ 2(− 2 ) (1 − ) d 2 T 2 ⎜⎟ ⎝⎠2(NN−−−− 2)[ ( 2) 2( n 1)ρ]
Campbell Collaboration Colloquium – August 2011 www.campbellcollaboration.org
17 2 A third estimator based on S B • We use this when the treatment effect is defined at the level of the clusters, or when the other studies in our meta- analysis have been analyzed using cluster means as the unit of analysis • Though it might not be of general interest, we can use this effect size to obtain other effect sizes of interest • There are two estimates of this effect
Campbell Collaboration Colloquium – August 2011 www.campbellcollaboration.org
2 2 Estimator dB1 : When we have S B and S W
TC YYgg− gg dB1 = , with σˆB S 2 σˆ =S 2 −W BBn
Campbell Collaboration Colloquium – August 2011 www.campbellcollaboration.org
18 The variance of dB1 (a little messy)
⎛⎞mmTC++1( n− 1)ρ Vd{ B1} =+⎜⎟TC ⎝⎠mm nρ ⎡⎤2 2 [1(+n − 1)ρ] (1− ρ ) 2 ⎢⎥+ d 2(Mn 2)22 2( NMn ) 22 B1 ⎣⎦⎢⎥−−ρρ
Campbell Collaboration Colloquium – August 2011 www.campbellcollaboration.org
Estimator dB2 : When we have SB and ρ
TC YYgg− gg 1(+n − 1)ρ dB2 = SnB ρ
Campbell Collaboration Colloquium – August 2011 www.campbellcollaboration.org
19 The variance of dB2 (a little messy)
⎛⎞mmTC++1( n− 1)ρ Vd{ B2} =+⎜⎟TC ⎝⎠mm nρ 2 (1(+nd− 1)ρ ) B2 2(Mn− 2) ρ
Campbell Collaboration Colloquium – August 2011 www.campbellcollaboration.org
IMPORTANT: Can compute any δ from any other effect size and ρ
ρ δT δδWB== 1− ρ 1− ρ
δδρδTB== W1− ρ
Campbell Collaboration Colloquium – August 2011 www.campbellcollaboration.org
20 And, can compute the variance of the effect size using the same transformations
ρ Var{δT } Var{δδWB} == Var{ } 11−−ρρ
Var{δδρδρTB} == Var{ } Var{ W}(1− )
Campbell Collaboration Colloquium – August 2011 www.campbellcollaboration.org
Where are we now?
• We have looked at the structure of the data • We have examined the different components of variation • We have defined estimators for the different components of variation • We have outlined the computation of three different effect
sizes, dW , dT , and dB and their associated variances
Campbell Collaboration Colloquium – August 2011 www.campbellcollaboration.org
21 But we all know it is not that simple: Some common problems
• We have unequal cluster sample sizes – There are formulas for unequal cluster sample sizes that are much more complex – We can assume equal cluster sizes since most studies attempt to use equal cluster sizes in the design • We don’t know the intraclass correlation coefficient – We can estimate it from a number of sources listed in the References – Several researchers have provided estimates from other studies and from large-scale national samples
Campbell Collaboration Colloquium – August 2011 www.campbellcollaboration.org
Let’s look at some examples
Campbell Collaboration Colloquium – August 2011 www.campbellcollaboration.org
22 From the analysis section
Campbell Collaboration Colloquium – August 2011 www.campbellcollaboration.org
Another clue that we have cluster means and sds reported
Campbell Collaboration Colloquium – August 2011 www.campbellcollaboration.org
23 The ICCs are also reported
Campbell Collaboration Colloquium – August 2011 www.campbellcollaboration.org
But I could not find sample sizes within clusters so looked for another report on the intervention
Campbell Collaboration Colloquium – August 2011 www.campbellcollaboration.org
24 Campbell Collaboration Colloquium – August 2011 www.campbellcollaboration.org
So, what do we have in this example?
• We have ρ for SAT Math of 0.72 • There are 10 matched-pairs of schools, with NT = 976, and NC = 738 (conservative common n = 73) • We have the mean of the school (cluster) means and SDs for the treatment and control group at the level of school
Campbell Collaboration Colloquium – August 2011 www.campbellcollaboration.org
25 We have all the information needed to compute dB2
(7.26)22+ (7.48) SS2 ===54.33, 7.37 BB2 82.33− 78.77 1+ (73− 1)0.72 d = B2 7.37 73(0.72) ==0.48 1.005 0.48
Campbell Collaboration Colloquium – August 2011 www.campbellcollaboration.org
To compute Var{dB2 }
⎛⎞10++ 10 1 (73− 1)(0.72) Vd{ B2} = ⎜⎟ ⎝⎠10*10 (73)(0.72) (1+ (73− 1)0.72) (0.48) + 2(20− 2)(73)(0.72) ⎛⎞⎛⎞20 52.84 (52.84)(0.48) =+=+⎜⎟⎜⎟ 0.201 0.013 ⎝⎠⎝⎠100 52.56 1892.16 =0.214
Campbell Collaboration Colloquium – August 2011 www.campbellcollaboration.org
26 Say we want dW in our meta-analysis
ρ 0.72 dd==0.48 WB1−−ρ 1 0.72 ==0.48(1.60) 0.77
Campbell Collaboration Colloquium – August 2011 www.campbellcollaboration.org
The variance of d follows the same transformation W 2 ⎛⎞ρ Var d Var d { WB} = { }⎜⎟ ⎝⎠1− ρ ⎛⎞0.72 =0.214⎜⎟ ⎝⎠(1− 0.72) ==0.214(2.57) 0.55
SD{ dW }=0.74 Campbell Collaboration Colloquium – August 2011 www.campbellcollaboration.org
27 If instead, we want dT
ddTB==ρ 0.48 0.72 ==0.48(0.85) 0.41
Var{ dTW} = Var{ d }ρ ==(0.214)(0.72) 0.154
SD{ dT } = 0.392
Campbell Collaboration Colloquium – August 2011 www.campbellcollaboration.org
Some observations on this example
• The intraclass correlation is large for this measure so we have a lot of between school variability • Thus, it is not surprising that none of the effect sizes we calculated are significantly different from zero • In this example, we have the intraclass correlation reported, but we may not be so lucky
Campbell Collaboration Colloquium – August 2011 www.campbellcollaboration.org
28 Another example
Campbell Collaboration Colloquium – August 2011 www.campbellcollaboration.org
We might have individual level info in Table 2
Campbell Collaboration Colloquium – August 2011 www.campbellcollaboration.org
29 Campbell Collaboration Colloquium – August 2011 www.campbellcollaboration.org
What information do we have?
• mT = 14 schools in the treatment group, mC = 10 schools in the control group • For treatment group, sample size average is nT = 3429/14 = 244.9. For control group, sample size average is nC = 1047/10 = 104.7 • We have individual level means and standard deviations BUT • We don’t have the intraclass correlation, or other estimates of the variance
Campbell Collaboration Colloquium – August 2011 www.campbellcollaboration.org
30 To compute effect sizes, we need either 2 estimates of the variance, or 1 estimate and ρ
2 • We can only compute the estimate of S T from our 2 estimates of the individual level sds • We will need to find an estimate for ρ from another source • From: Murray & Blitstein (2003). Methods to reduce the impact of intraclass correlation in group-randomized trials. Evaluation Review, 27: 79-103 • The upper bound of the ICC for youth cohort studies with outcomes focused on diet is 0.0310
Campbell Collaboration Colloquium – August 2011 www.campbellcollaboration.org
Since we have individual level stats:
(3429− 1)(0.52)22+ (1047− 1)(0.25) S 2 ==0.22 T 3429+ 1047− 2 3.40−− 2.16 2(104 1)0.031 d =1− T 2 0.47 208− 2 ==2.64 0.97 2.59
Campbell Collaboration Colloquium – August 2011 www.campbellcollaboration.org
31 The variance of dT
⎛⎞1456+ 1040 Vd{ T 2} =+⎜⎟(1 (104− 1)*0.031) + ⎝⎠1456*1040 ⎛⎞(2496−− 2)(1 0.031)22 + 104(2496− 2*104)0.031+ 2(2496 −2*104)0.031(1 − 0.031) 2.592 ⎜⎟ ⎝⎠2(2496−−−− 2)[ (2496 2) 2(104 1)0.031] =+0.007 2.592 (0.0002) = 0.0083
SD{ dT 2}=0.091
Campbell Collaboration Colloquium – August 2011 www.campbellcollaboration.org
If we want dW
dT 2.59 dW == =2.64 1−−ρ 1 0.031
Var{ dT } 0.0083 Var{ dW }===0.0086 1−−ρ 1 0.031
SD{ dW }=0.093
Campbell Collaboration Colloquium – August 2011 www.campbellcollaboration.org
32 What about other types of effect sizes, like odds-ratios?
• We do not yet have the same research for odds ratios • What we can do is inflate the odds ratios from a clustered trial using recommendations from the Cochrane Handbook
Campbell Collaboration Colloquium – August 2011 www.campbellcollaboration.org
Recommendations from the Cochrane Handbook
• We first compute the design effect given by
1 + (nave – 1) ρ
where nave is the average cluster size, and ρ is the intraclass correlation coefficient • We divide the total sample sizes and number of events by the design effect to obtain the corrected numbers to compute the odds-ratio • We adjust the variance by computing the variance of the log- odds ratio using the original sample sizes and dividing by the square root of the design effect
Campbell Collaboration Colloquium – August 2011 www.campbellcollaboration.org
33 From the Positive Action program
No smoking Smoked
Intervention 937 4.0% of 976 = 39
Control 682 7.6% of 738 = 56
Campbell Collaboration Colloquium – August 2011 www.campbellcollaboration.org
And the intraclass correlation coefficient
Campbell Collaboration Colloquium – August 2011 www.campbellcollaboration.org
34 Adjusted values
• Average cluster size: (738+976)/20 = 85.7 • Design effect: 1 + (85.7 – 1) 0.05 = 5.235
No smoking Smoked
Intervention 937/5.235=178.99 4.0% of 976 = 39/5.235 = 7.45 Control 682/5.235=130.28 7.6% of 738 = 56/5.235 = 10.7
Campbell Collaboration Colloquium – August 2011 www.campbellcollaboration.org
Variance for the log-odds ratio
• OR = (937*56)/(682*39) = 1.97 • lnOR = ln(1.97) = 0.68 • SE2(lnOR) = 1/937 + 1/56 + 1/682 + 1/39 = 0.046 • SE(lnOR) = 0.214 • Adjusted SE(lnOR) = 0.214/√5.235 = 0.094
Campbell Collaboration Colloquium – August 2011 www.campbellcollaboration.org
35 References Hedges, L. V. (2007). Effect sizes in cluster-randomized designs. Journal of Educational and Behavioral Statistics, 32, 341-370. Hedges, L. V. & Hedberg, E. C. (2007). Intraclass correlation values for planning group randomized experiments in education. Educational Evaluation and Policy Analysis, 29, 60-87. Cochrane Handbook: http://www.cochrane-handbook.org/
Campbell Collaboration Colloquium – August 2011 www.campbellcollaboration.org
P.O. Box 7004 St. Olavs plass 0130 Oslo, Norway
E-mail: [email protected] http://www.campbellcollaboration.org
The Campbell Collaboration www.campbellcollaboration.org
36