Downloaded from http://qualitysafety.bmj.com/ on February 13, 2018 - Published by group.bmj.com BMJ Quality & Safety Online First, published on 9 February 2018 as 10.1136/bmjqs-2017-007554 EDITORIAL Rigorous evaluations of evolving interventions: can we have our cake and eat it too? Robert E Burke,1 Kaveh G Shojania2 1Research and Hospital The years immediately following the a rigorous evaluation of the Michigan Medicine Sections, Denver widespread interest in patient safety1 and Appropriateness Guide for Intravenous VA Medical Center, Denver, 2 Colorado, USA then healthcare quality saw considerable Catheters (MAGIC) QI intervention, 2Department of Medicine, debate between pragmatically oriented intended to reduce adverse events stem- Sunnybrook Health Sciences improvers and research-oriented evalu- ming from the insertion of peripher- Centre and the University of 3–6 ators —or between ‘evangelists’ and ally inserted venous central catheters Toronto, Toronto, Ontario, Canada ‘snails’ as one longtime observer charac- (PICC). PICCs have become ubiquitous terised the two groups.7 Too often, enthu- as a substitution for a central intrave- Correspondence to siastic improvers (‘evangelists’) relied on nous line when patients need longer term Dr Robert E Burke, Research simple pre-post designs within a single central intravenous access, but clinicians and Hospital Medicine Sections, context leading to erroneous claims often order them unnecessarily or order Denver VA Medical Center, 8 Denver, CO 80220, USA; of efficacy. In contrast, research-ori- inappropriate types—for example, a Robert. Burke5@ va. gov ented investigators (‘snails’) and journals double-lumen PICC when a single-lumen pushed for ever more rigorous designs PICC would work just as well and carry a Accepted 29 January 2018 including randomised trials, potentially at lower risk of complications. The authors the cost of discouraging many improvers implemented MAGIC at a single inter- without this training and leading to vention hospital and used data from nine slower development and deployment of contemporaneous controls drawn from a effective interventions.9 10 Many clini- QI collaborative in the state of Michigan cians, quality improvement (QI) experts (all 10 sites participate in the collabora- and researchers are thus caught in a quan- tive). dary: how best to evaluate a candidate QI The MAGIC intervention included intervention? How can we best balance computerised decision support at the the pragmatic needs of improvement— time of ordering and a much larger role including the frequent need to refine the for PICC nurses to regulate appropriate intervention or its implementation—with PICC placement. Training was also the requirement of most traditional eval- delivered for PICC nurses and ordering uative designs, which typically require a providers. Outcomes included rates of static intervention? inappropriate PICC use and device-re- We believe this question is one of the lated adverse events. The intervention most important issues to consider when achieved a statistically significant but rela- developing a QI intervention and is often tively small decrease in the rate of inap- not considered carefully enough—either propriate PICC use at the intervention site by snails or evangelists. Decisions about after adjustment for measurable potential when and how to evaluate potentially confounders (incidence rate ratio 0.86; promising interventions can have crucial 95% CI 0.74 to 0.99, P=0.048). Fewer ► http:// dx. doi. org/ 10.1136/ implications for the future of the inter- adverse events occurred at the interven- bmjqs-2017-007342 vention and the patients it could affect. tion hospital, but this reduction largely reflected fewer catheter occlusions. Rates TWO RECENT EXAMPLES OF of venous thrombosis and infection rates To cite: Burke RE, IMPROVEMENT INTERVENTIONS remained unchanged, though prior work Shojania KG. BMJ Qual Saf EVALUATED USING TRADITIONAL by the authors has shown low rates for Epub ahead of print: [please include Day Month Year]. DESIGNS both of these complications (5.2% and doi:10.1136/ In this issue of BMJ Quality and Safety, 1.1%, respectively, for thrombosis and bmjqs-2017-007554 Swaminathan and colleagues11 present infection).12 Burke RE, Shojania KG. BMJ Qual Saf 2018;0:1–4. doi:10.1136/bmjqs-2017-007554 1 Copyright Article author (or their employer) 2018. Produced by BMJ Publishing Group Ltd under licence. Downloaded from http://qualitysafety.bmj.com/ on February 13, 2018 - Published by group.bmj.com Editorial Some might characterise these results as disap- common. On the other hand, we question the degree pointing—bordering on a ‘negative trial’. The authors to which these traditional designs provide the appro- (understandably) regard the intervention as having priate balance between rigour and the need to refine achieved some success and probably hope to refine the interventions, since these evaluative designs presume intervention further and test it in other hospitals in a ‘fixed’ or unchanging intervention. None offer an this collaborative. We do not seek to debate this point. obvious way for investigators to modify the interven- Our interest here lies in discussing the tension in QI tion in response to implementation challenges or a between the need to refine interventions, especially disappointing effect size. More adaptive versions of early in their development, and the desire to conduct these traditional evaluative designs do in fact exist, rigorous, compelling evaluations to demonstrate their and we believe are underused. impact. Consider a second example, in which Westbrook HAVING OUR CAKE AND EATING IT, TOO: and colleagues evaluated a bundled intervention to RIGOROUS EVALUATIVE DESIGNS WHICH reduce nurse interruptions during medication admin- PERMIT REFINEMENTS TO THE INTERVENTION istration using a cluster randomised controlled trial Plan-Do-Study-Act (PDSA) cycles offer the most (RCT).13 For every 100 medication administrations, familiar and accessible way to iteratively improve an nurses on intervention wards experienced 15 fewer intervention, but this approach has two main chal- non-medication-related interruptions compared with lenges: (1) it requires specialised expertise and effort control wards. Using results from their previous work to do well,15–17 and (2) improvers may struggle to on the risk of adverse drug events with interruptions successfully pair this with a rigorous evaluative design. during medication administration,14 the authors them- However, pairing high-quality PDSA cycles with run selves acknowledged that the observed reduction in charts or statistical process control (SPC) charts can interruptions would likely achieve little benefit for be a powerful and rigorous approach.18 19 We have patients. Moreover, the nurses hated wearing the ‘do published many studies using SPC, but a particularly not interrupt’ vests, which constituted a core feature compelling example involved an ‘electronic physio- of the intervention. logical surveillance system’ implemented in two hospi- tals in England.20 Nurses recorded patients’ vital signs WHY THESE TWO EXAMPLES? using a handheld device, which then algorithmically What both interventions share, in addition to their determined if another set of vitals should be done small to modest impacts, is the use of rigorous, tradi- sooner, whether care should be escalated to a Rapid tional evaluation paradigms—one an interrupted time Response Team, and displayed the required timing series combined with contemporaneous controls (about of the response automatically on the same handheld as rigorous a non-randomised design as possible) and device. Understandably, given the complexity of the the other a cluster RCT. Yet, the disappointing effect intervention, the investigators permitted flexibility sizes raise the question: did these interventions need in aspects of the roll-out at the two hospitals with further refinement before subjecting them to rigorous built-in cycles for improving fidelity to the interven- evaluation? tion. Importantly, the authors could track how often In the case of the MAGIC, the authors had a reason- the intervention was used at each site. This allowed able idea for an intervention, but much less prior work them to powerfully demonstrate statistically signif- to inform the precise ingredients or implementation. icant decreases in mortality rates that occurred in In the example of the cluster RCT of a bundled ‘do temporal association with successful deployment of the not interrupt’ intervention, substantial prior work (not intervention.20 just by these authors) had explored this type of inter- For larger multisite studies, ‘stepped wedge’ designs vention. Thus, the investigators did not plan modifica- offer an appealing option21 22 that will seem familiar tions to the intervention or its implementation strategy. to those accustomed to PDSA cycles. In their most Consequently, it made sense to randomise wards to a rigorous form, they are cluster randomised trials, fixed intervention or to usual care and focus on eval- except that randomisation determines not whether a uating the impact. That said, it obviously occurred site receives the intervention, but rather when. The to Westbrook et al13 that the nurses might not like total number of sites is randomised into sequential wearing the vests, since they solicited this feedback as cohorts, with all cohorts eventually implementing part of their results. Thus, they might have anticipated the intervention and each providing