Quasi-Experimental Designs in Practice-Based Research Settings: Design and Implementation Considerations
Total Page:16
File Type:pdf, Size:1020Kb
J Am Board Fam Med: first published as 10.3122/jabfm.2011.05.110067 on 7 September 2011. Downloaded from SPECIAL COMMUNICATION Quasi-Experimental Designs in Practice-based Research Settings: Design and Implementation Considerations Margaret A. Handley PhD, MPH, Dean Schillinger, MD, and Stephen Shiboski, PhD Background: Although randomized controlled trials are often a gold standard for determining interven- tion effects, in the area of practice-based research (PBR), there are many situations in which individual randomization is not possible. Alternative approaches to evaluating interventions have received in- creased attention, particularly those that can retain elements of randomization such that they can be considered “controlled” trials. Methods: Methodological design elements and practical implementation considerations for two quasi- experimental design approaches that have considerable promise in PBR settings – the stepped-wedge design, and a variant of this design, a wait-list cross-over design, are presented along with a case study from a recent PBR intervention for patients with diabetes. Results: PBR-relevant design features include: creation of a cohort over time that collects control data but allows all participants (clusters or patients) to receive the intervention; staggered introduction of clusters; multiple data collection points; and one-way cross-over into the intervention arm. Practical considerations include: randomization versus stratification, training run in phases; and extended time period for overall study completion. Conclusion: Several design features of practice based research studies can be adapted to local cir- copyright. cumstances yet retain elements to improve methodological rigor. Studies that utilize these methods, such as the stepped-wedge design and the wait-list cross-over design, can increase the evidence base for controlled studies conducted within the complex environment of PBR. (J Am Board Fam Med 2011;24: 589–596.) Keywords: Clinical Trials, Implementation Science, Practice-based Research, Quasi-Experimental Design Although randomized, controlled trials (RCTs) are situations where individual randomization is not http://www.jabfm.org/ often a gold standard for determining intervention possible.1 For example, an RCT may not be con- effects, in the area of practice-based research, qual- sidered possible for any of the following reasons: ity improvement, and public health, there are many (1) there is a commonly held view among stake- holders that evidence is sufficient in some settings of an established intervention benefit, and it would This article was externally peer reviewed. be unethical to have control groups; (2) the inter- on 26 September 2021 by guest. Protected Submitted 25 February 2011; revised 16 June 2011; ac- cepted 30 June 2011. vention delivery is underway already (e.g., a policy From the University of California San Francisco, Depart- change sets in motion new clinical procedures); or ment of Epidemiology and Biostatistics (MAH, SS) and Department of Medicine, Division of General Internal (3) assignment to a control group is unacceptable to Medicine, San Francisco General Hospital Center for Vul- some groups that would potentially be controls. nerable Populations (MAH, DS). Funding: This work was funded by AHRQ grant R18 HS Consequently, alternative approaches to evaluating 017261, Harnessing Health Information Technology for clinical and community interventions have received Self-Management Support and Medication Activation in a Medicaid Health Plan, McKesson Foundation. increased attention, particularly those that can re- Conflict of interest: none. tain some elements of randomization such that they Corresponding author: Margaret A. Handley, PhD MPH, 2–6 Division of General Internal Medicine, San Francisco can be considered “controlled” trials. Such de- General Hospital Center for Vulnerable Populations, 995 signs are consistent with discussions of “practical Potrero Avenue, Ward 83, San Francisco, CA 94110 (E- 7 mail: handleym@medsfgh.ucsf.edu). clinical trials” in that these designs adapt to local doi: 10.3122/jabfm.2011.05.110067 Quasi-Experimental Designs in Practice-based Research 589 J Am Board Fam Med: first published as 10.3122/jabfm.2011.05.110067 on 7 September 2011. Downloaded from considerations, such as those raised above, and studies), because it is possible to control the roll- therefore are likely to achieve better outcomes. In out and to include elements of randomization that light of the recognition that evidence-based prac- can reduce biases. However, the best design for a tice must be informed by practice-based evidence,8 particular intervention must be determined within practice-based research (PBR), in particular may the local context, through determining what types benefit from approaches that do not require ran- of evidence exist to support other designs and what domization of individuals in settings where it may are the relative merits of alternatives to RCT in not be feasible. Such alternative designs, often different contexts.1 called “quasi-experimental” designs, are increas- ingly used for the evaluation of clinical and/or prac- Methods tice-based interventions applied under real world We present a review of design features and practi- circumstances where individual-level RCT designs cal considerations for PBR implementation for the are not suitable. stepped-wedge and wait-list design, along with a This paper reviews design elements and practi- discussion of published examples from studies of cal implementation considerations for two quasi- clinic-based interventions using these designs. A experimental designs that have considerable prom- PubMed search was conducted using the terms ise in PBR settings: the stepped-wedge design, and quasi-experimental, stepped-wedge, quality improve- a variant of this design, described herein as a wait- ment, and wait-list design to identify methodological list cross-over design, currently used in a large as well as clinic-based publications that describe PBRN project. The stepped-wedge design’s rele- these design approaches. The examples described vance to PBR lies in: (1) it is a cluster-based design in this paper are derived from the literature review suitable to clinic-level interventions that enables all and also the author’s experiences in PBR work sites to receive the intervention, yet (2) it avoids conducted within the San Francisco Bay Collabor- some of the methodological pit-falls associated ative Research Network, a PBRN located in the SF copyright. with before and after designs, as it retains con- Bay Area and affiliated with the University of Cal- trolled data elements.5 The stepped-wedge and ifornia San Francisco, with which the authors are wait-list variant allow all patients to receive the long-time members and research collaborators. intervention while also contributing control time. This differs from a traditional cluster randomized trial done in parallel time, in which some sites are Results randomized to control at the outset and do not have Stepped-Wedge Designs an opportunity to cross over into an intervention Stepped-wedge designs have been in use for several arm. decades, often in the context of settings in which http://www.jabfm.org/ A variant of the stepped wedge, the wait-list there are strong objections to individual level ran- cross-over trial, is also particularly well suited to domization.1 A 2006 review of stepped-wedge de- PBR and quality improvement studies involving signs indicated this design was frequently used in clinic or systems-based disease registries, as it en- developing countries, often in the context of HIV ables staggered implementation of patient-level in- treatment interventions.2 More recently, the liter- terventions over time. Staggered implementation is ature indicates stepped-wedge designs are increas- highly relevant to counseling-based programs and ingly being implemented in clinic-based settings on 26 September 2021 by guest. Protected those requiring considerable staff training, such as world-wide, for example, in a trial of guideline some guideline-based interventions, as the design implementation for therapeutic hypothermia in can alleviate staffing burdens (and costs) that would post-cardiac patients in a network of Emergency be encumbered without staggered implementation. Departments in Canada10 and in evaluating imple- The staggered implementation creates a waiting mentation of a clinician-based psychosocial inter- period for some patients, and this wait time pro- vention for improving treatment of patients with vides useful “control” information for inclusion in cancer pain in Australia.11 Stepped-wedge designs the data analysis. are a cluster-randomized type of cross-over design These designs have stronger methodological in which clusters are all initially assigned to the rigor in comparison with other well-known quasi- control group, then switch to the intervention experimental design options (such as pre and post group at randomly assigned time points. All clus- 590 JABFM September–October 2011 Vol. 24 No. 5 http://www.jabfm.org J Am Board Fam Med: first published as 10.3122/jabfm.2011.05.110067 on 7 September 2011. Downloaded from Figure 1. Stepped-wedge design. Adapted from Brown that control data are collected multiple times for and Lilford, 2006. some units. In Figure 1, cluster 5 contributes data at five time points before receiving the interven- tion. (4) Every site gets the intervention eventually and every site contributes control data