Methods Research Report Empirical Evidence of Associations Between Trial Quality and Effect Size Methods Research Report Empirical Evidence of Associations Between Trial Quality and Effect Size Prepared for: Agency for Healthcare Research and Quality U.S. Department of Health and Human Services 540 Gaither Road Rockville, MD 20850 http://www.ahrq.gov Contract No. HHSA 290-2007-10062-I Prepared by: Southern California Evidence-based Practice Center Santa Monica, CA Investigators: Susanne Hempel, Ph.D. Marika J. Suttorp, M.S. Jeremy N.V. Miles, Ph.D. Zhen Wang, M.S. Margaret Maglione, M.P.P. Sally Morton, Ph.D. Breanne Johnsen, B.A. Diane Valentine, J.D. Paul G. Shekelle, M.D., Ph.D. AHRQ Publication No. 11-EHC045-EF June 2011 This report is based on research conducted by the Southern California Evidence-based Practice Center (EPC) under contract to the Agency for Healthcare Research and Quality (AHRQ), Rockville, MD (Contract No. 290-2007-10062-I). The findings and conclusions in this document are those of the author(s), who are responsible for its contents; the findings and conclusions do not necessarily represent the views of AHRQ. Therefore, no statement in this report should be construed as an official position of AHRQ or of the U.S. Department of Health and Human Services. The information in this report is intended to help health care decisionmakers—patients and clinicians, health system leaders, and policymakers, among others—make well-informed decisions and thereby improve the quality of health care services. This report is not intended to be a substitute for the application of clinical judgment. Anyone who makes decisions concerning the provision of clinical care should consider this report in the same way as any medical reference and in conjunction with all other pertinent information, i.e., in the context of available resources and circumstances presented by individual patients. This report may be used, in whole or in part, as the basis for development of clinical practice guidelines and other quality enhancement tools, or as a basis for reimbursement and coverage policies. AHRQ or U.S. Department of Health and Human Services endorsement of such derivative products may not be stated or implied. This document is in the public domain and may be used and reprinted without permission except those copyrighted materials noted for which further reproduction is prohibited without the specific permission of copyright holders. No investigators have any affiliations or financial involvement (e.g., employment, consultancies, honoraria, stock options, expert testimony, grants or patents received or pending, or royalties) that conflict with material presented in this report. Suggested Citation: Hempel S, Suttorp MJ, Miles JNV, Wang Z, Maglione M, Morton S, Johnsen B, Valentine D, Shekelle PG. Empirical Evidence of Associations Between Trial Quality and Effect Sizes. Methods Research Report (Prepared by the Southern California Evidence-based Practice Center under Contract No. 290-2007-10062-I). AHRQ Publication No. 11-EHC045-EF. Rockville, MD: Agency for Healthcare Research and Quality. June 2011. Available at: http://effectivehealthcare.ahrq.gov. i Preface The Agency for Healthcare Research and Quality (AHRQ), through its Evidence-based Practice Centers (EPCs), sponsors the development of evidence reports and technology assessments to assist public- and private-sector organizations in their efforts to improve the quality of health care in the United States. The reports and assessments provide organizations with comprehensive, science-based information on common, costly medical conditions and new health care technologies. The EPCs systematically review the relevant scientific literature on topics assigned to them by AHRQ and conduct additional analyses when appropriate prior to developing their reports and assessments. To improve the scientific rigor of these evidence reports, AHRQ supports empiric research by the EPCs to help understand or improve complex methodologic issues in systematic reviews. These methods research projects are intended to contribute to the research base and be used to improve the science of systematic reviews. They are not intended to be guidance to the EPC program, although may be considered by EPCs along with other scientific research when determining EPC program methods guidance. AHRQ expects that the EPC evidence reports and technology assessments will inform individual health plans, providers, and purchasers; as well as the health care system as a whole by providing important information to help improve health care quality. The reports undergo peer review prior to their release as a final report. We welcome comments on this Methods Research Project. They may be sent by mail to the Task Order Officer named below at: Agency for Healthcare Research and Quality, 540 Gaither Road, Rockville, MD 20850, or by e-mail to [email protected]. Carolyn M. Clancy, M.D. Jean Slutsky, P.A., M.S.P.H. Director Director, Center for Outcomes and Evidence Agency for Healthcare Research and Quality Agency for Healthcare Research and Quality Stephanie Chang, M.D., M.P.H. Director, Task Order Officer Evidence-based Practice Program Center for Outcomes and Evidence Agency for Healthcare Research and Quality ii Acknowledgements We would like to thank the Tufts Medical Center and the Research Triangle Institute – University of North Carolina (RTI-UNC) Evidence-based Practice Center for their collaboration in identifying EPC reports and Kirstin Nyrop and Jonathan V. Todd for quality scoring of selected articles. We also thank Ethan Balk, Nancy Berkman, Isabelle Boutron, Tim Carey, Mark Helfand, David Moher, and Sydne Newberry for comments on an earlier draft of this report. iii Empirical Evidence of Associations Between Trial Quality and Effect Sizes Structured Abstract Objectives. To examine the empirical evidence for associations between a set of proposed quality criteria and estimates of effect sizes in randomized controlled trials across a variety of clinical fields and to explore variables potentially influencing the association. Methods. We applied quality criteria to three large datasets of studies included in a variety of meta-analyses covering a wide range of topics and clinical interventions consisting of 216, 165, and 100 trials. We assessed the relationship between quality and effect sizes for 11 individual criteria (randomization sequence, allocation concealment, similar baseline, assessor blinding, care provider blinding, patient blinding, acceptable dropout rate, intention-to-treat analysis, similar cointerventions, acceptable compliance, similar outcome assessment timing) as well as summary scores. Inter-item relationships were explored using psychometric techniques. We investigated moderators and confounders affecting the association between quality and effect sizes across datasets. Results. Quality levels varied across datasets. Many studies did not report sufficient information to judge methodological quality. Some individual quality features were substantially inter- correlated, but a total score did not show high overall internal consistency (α 0.55 to 0.61). A factor analysis-based model suggested three distinct quality domains. Allocation concealment was consistently associated with slightly smaller treatment effect estimates across all three datasets; other individual criteria results varied. In dataset 1, the 11 individual criteria were consistently associated with lower estimated effect sizes. Dataset 2 showed some unexpected results; for several dimensions, studies meeting quality criteria reported larger effect sizes. Dataset 3 showed some variation across criteria. There was no statistically significant linear association of a summary scale or factor scores with effect sizes. Applying a cutoff of 5 or 6 criteria met (out of 11) differentiated high and low quality studies best. The effect size differences for a cutoff at 5 was -0.20 (95% confidence interval [CI]: -0.34, -0.06) in dataset 1 and the respective ratio of odds ratios in dataset #3 was 0.79 (95% CI: 0.63, 0.95). Associations indicated that low-quality trials tended to overestimate treatment effects. This observation could not be replicated with dataset 2, suggesting the influence of confounders and moderators. The size of the treatment effect, the condition being treated, the type of outcome, and the variance in effect sizes did not sufficiently explain the differential associations between quality and effect sizes but warrant further exploration in explaining variation between datasets. Conclusions. Effect sizes of individual studies depend on many factors. The conditions where quality features lead to biased effect sizes warrant further exploration. iv Contents Executive Summary .................................................................................................................ES-1 Background ....................................................................................................................................1 Methods ...........................................................................................................................................4 Quality Criteria ..........................................................................................................................4 Study Pool Selection ..................................................................................................................4 Dataset 1: Back Pain Trials ..................................................................................................5