Section on Survey Research Methods – JSM 2010 Robust Covariate Control in Cluster-Randomized Trials with MPLUS and WinBUGS Jiaquan Fan1 and David Judkins1 1Westat, 1600 Research Blvd., Rockville, MD 20850 Abstract Two more statistical software packages, MPLUS and WinBUGS, were tested as a continuation of a previous study of analysis method/software robustness in the analysis of clustered data. We want to evaluate robustness in the context of a randomized complete block design, where each “plot” is a small group of children at the same nursery school and a series of measurements of each child are made. We constructed a series of super populations in which the standard assumptions of hierarchical (mixed effects) linear models were violated. The results were compared with HLM, SUDAAN, PROC MIXED and a semi-parametrical analysis of variance procedure we tested before. Key Words: Cluster-randomized trials, multi-level modeling, Bayesian inference using Gibbs Sampling 1. Introduction and Background This work is a continuation of our 2006 study (Fan and Judkins, 2006) in which we undertook a simulation to study the robustness of some standard software options for covariate control in the context of cluster-randomized trials. We also developed and tested a new semi-parametric method which we called semi-parametric ANOVA. In this study, we continue the line of research by including two more software systems, MPLUS and WINBUGS, in the simulation study. We also found that there is a problem of potential serial correlation in the program we used before to generate simulated data because the new seeds were generated by calling SAS random number generation functions. To fix this problem, we changed to a new and improved SAS function for random number generation and a method of drawing new seeds from a pre-determined sequence of numbers. Another change we made in the current study is to increase the variance for treatment effects when researching power. The application of interest was a randomized experiment with alternate preschool instructional paradigms, loosely referred to as curricula. There were four alternate curricula and one control curriculum. All five arms were assigned to a recruited sample of 120 Even Start schools. The schools were deeply stratified into 24 blocks, each containing five schools. Within a block, the five schools were then randomly assigned to the five arms. The curricula involved instructional materials, instructional strategies, teacher training, teacher observation, and teacher consultation. Within the schools, parents of age-eligible children were recruited into the study. Measurements were conducted in the spring of 2004, prior to the introduction of the new curricula, and repeated at one-year intervals in 2005 and 2006. Measurements involved formal assessments of pre-literacy, social competency (teacher observation), parent interviews, and video-taping and behaviour-coding of staged parent-child interactive reading and toy- play sessions to gauge parenting skills. There was considerable turnover in the student- body each year, but there is some overlap of sample across years, and of course, there is considerable organizational and staffing stability. So one set of important covariates 951 Section on Survey Research Methods – JSM 2010 involved school-level past performance and child-teacher ratios. Another important set of covariates involved parent socio-economic status, native language, and child demographics (age, race, sex, and disability status). Native language, in particular, has a huge effect on English pre-literacy. For analysis, we wanted something that was robust to unequal student sample sizes per school, school-level nonresponse, deep stratification, heteroscedasticity, non-Gaussian errors and interactions. We therefore developed superpopulations that had the features of interest, generated samples from them, and tested several alternative analysis procedures on them, using type I error rates and statistical power as evaluation criteria. In section 2, we discuss the superpopulations that we simulated. In section 3, we provide more detail on the analysis methods studied. In section 4, we present results. In section 5, we give some ideas for further research. 2. Simulated Superpopulations Given the application, we built a series of superpopulations with an increasing number of violations of standard models. All shared a common form of having two child-level covariates, one school-level covariate, a random effect at the school level, and student level random error. The project-level covariate was built with a structure similar to the outcome of interest because the way it will be generated in the application is to take the average of students at the school the prior year. All of the superpopulations share a common model structure: yijk =+++µβi α iX ijk θ + Z ij γ ++ ue ij ijk , Zij=++βθ i uX ij ij. + v ij where: The indices stand respectively for block (i), treatment (j), child (k); yijk is the outcome variable; µ is the overall mean; βi is the (fixed) block effect; αi is the treatment effect; X ijk is a vector of two child level covariates ( X1 = FamilyIncome, X 2 = MothersEducation); Zij is the baseline school-level average of the outcome variable measured on a different set of students prior to the intervention; uij is the school level-random effect; eijk is a child level random error; X ij. is a vector of school-level averages of child level covariates; vij is a normally distributed random error term reflecting the error caused by basing the project-level fixed covariate on a small sample from the prior year rather than a long-run average; uij , vij and eijk are mutually independent. Because the theory is better developed for balanced designs, we introduced imbalance both at the school and the child level. Note that standard multi-level software assumes 952 Section on Survey Research Methods – JSM 2010 that all the random errors are normal and homoscedastic. So we developed superpopulations that violated those assumptions. Finally, we allowed interactions. We simulated a series of superpopulations that violated various combinations of these standard assumptions to various degrees while generally keeping the violations within the range that we thought might reasonably occur in our application. Seven different superpopulations with no treatment effect (αi = 0 ) were generated to test robustness of type 1 error rates. Superpopulation 1 satisfies most of the standard assumptions. The numbering of superpopulations 2 through 7 generally reflects increasing severe violations of standard assumptions: Superpopulation 1: There are 24 blocks with five schools per block and each school contains exactly 12 children. There is no school-level nonresponse and the school- and child-level random errors are normally distributed. Residual variances are constant with var( uij )= 12.81 and var( eijk )= 55.26 . The block effect is very large with. βi = 2i . vij is normal in all superpopulations with var( vij ) = 6 . Superpopulation 2: Same as superpopulation 1 except that the number of children per school is allowed to vary. The number of children per school follows a Poisson distribution with mean 12. Superpopulation 3: Same as Superpopulation 2 except that there are two schools missing at random (for a total of 118 schools). The missing schools are from different blocks. Superpopulation 4: Same as Superpopulation 3 except that the school- and child-level random errors have different variances in different blocks: Block 1 – 6 has uij and eijk with variances 3 and 56, Block 7 – 12 has uij and eijk with variances 6 and 42, Block 13 – 18 has uij and eijk with variances 9 and 28, Block 19 – 24 has uij and eijk with variances 12 and 14. Superpopulation 5: Same as Superpopulation 3 except that the school- and child-level random errors have different variances in different treatment groups: Treatment 1 has uij and eijk with variances 3 and 70, Treatment 2 has uij and eijk with variances 6 and 56, Treatment 3 has uij and eijk with variances 9 and 42, Treatment 4 has uij and eijk with variances 12 and 28. Control has uij and eijk with variances 15 and 14. Superpopulation 6: Same as Superpopulation 3 except that school- and child-level random errors have Gamma distributions. uij has shape parameter α = 2 and scale parameter β = 0.395 and eijk has α = 3 and β = 0.233 . Note that in this population, the school-level errors are more seriously non-normal than the student-level errors. Both skew and kurtosis are stronger for the school-level errors. Superpopulation 7: Same as Superpopulation 4 except that there are treatment group effects for individual blocks but no effect on average. That is, within each single block 953 Section on Survey Research Methods – JSM 2010 there are significant differences between the treatment groups, but when schools are aggregated to the treatment level, these differences average out. Another three superpopulations with treatment effect were generated to compare type II error rates. For each of these superpopulations, all four experimental arms are assumed to be equally effective withαi = 2.5 . This number was picked to give power in a range where we thought we might see the largest differences in power among the techniques. Superpopulation 8: Model is the same as Superpopulations 4 except that treatment effect is added. Superpopulation 9: Same as Superpopulations 5 except that treatment effect is added. Superpopulation 10: Same as Superpopulation 6 except that treatment effect is added. The components of variance in the model for the superpopulations are shown in Table 1. Naturally, there positive variance between treatment arms only for superpopulations 8, 9 and 10. All other variance components are constant across superpopulations. Also note that the between-block variance is very large. This was done with the aim of making it large enough to matter. Table 1: Components of Variances Component Magnitude Between block (fixed) 192 Between arm (fixed) 0 or 1 Child-level covariates (fixed) 3.4 School-level covariates (fixed) 18 School-level random effect (random) 13 Child level error (random) 55 3.