BMJ Open BMJ Open: first published as 10.1136/bmjopen-2016-015281 on 10 March 2017. Downloaded from

Process evaluation of the Data-driven Quality Improvement in Primary Care (DQIP) trial: Case-study evaluation of adoption and maintenance of a complex intervention to reduce high-risk primary care prescribing

For peer review only Journal: BMJ Open

Manuscript ID bmjopen-2016-015281

Article Type: Research

Date Submitted by the Author: 25-Nov-2016

Complete List of Authors: Grant, Aileen; University of Stirling, Health Sciences & Sport Dreischulte, Tobias; University of Dundee, Population Health Sciences Guthrie, Bruce; University of Dundee, Population Health Sciences Division, Medical Research Institute

Primary Subject General practice / Family practice Heading:

Health services research, Evidence based practice, Health informatics, Secondary Subject Heading: Qualitative research

PRIMARY CARE, Prescribing, General Practice, Quality and Safety, Keywords:

Randomised Controlled Trials, Process Evaluation http://bmjopen.bmj.com/

on October 3, 2021 by guest. Protected copyright.

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml Page 1 of 55 BMJ Open BMJ Open: first published as 10.1136/bmjopen-2016-015281 on 10 March 2017. Downloaded from 1 2 3 4 1 Process evaluation of the Data-driven Quality Improvement in Primary 5 6 2 Care (DQIP) trial: Case-study evaluation of adoption and maintenance of a 7 8 3 complex intervention to reduce high-risk primary care prescribing 9 4 10 11 12 5 13 14 6 Aileen Grant PhD, Research Fellow1 15 For peer review only 16 2 3 17 7 Tobias Dreischulte PhD, Lead Pharmacist Research and Development 18 19 8 Bruce Guthrie PhD, Professor of Primary Care Medicine3 20 21 9 22 23 24 10 Affiliations: 25 26 11 1. School of Health Sciences & Sport, University of Stirling, Stirling, UK 27 28 12 2. Medicines Governance Unit, NHS Tayside, Dundee, UK 29 13 3. Population Health Sciences Division, Medical Research Institute, University of Dundee, 30 31 14 Dundee, UK 32 33 15 Corresponding author 34 http://bmjopen.bmj.com/ 35 36 16 Aileen Grant: [email protected] 37 38 39 17 Keywords 40 41 18 General practice; family practice; prescribing; quality and safety; randomised controlled trials; 42 on October 3, 2021 by guest. Protected copyright. 43 19 process evaluation 44 45 20 46 47 21 Word count: 4,976 48 49 50 51 52 53 54 55 56 57 58 59 60 1

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml BMJ Open Page 2 of 55 BMJ Open: first published as 10.1136/bmjopen-2016-015281 on 10 March 2017. Downloaded from 1 2 3 22 Abstract 4 5 23 Objective: Explore how different practices responded to the DQIP intervention in terms of their 6 7 8 24 adoption of the work, reorganisation to deliver the intended change in care to patients, and whether 9 10 25 implementation was sustained over time. 11 12 13 26 Design: Mixed methods parallel process evaluation of a cluster trial, reporting the comparative case 14 15 27 study of purposivelyFor selected peer practices. review only 16 17 18 28 Setting: Ten (30%) primary care practices participating in the trial. 19 20 21 29 Results: Four practices were sampled because they had large rapid reductions in targeted 22 23 30 prescribing. They all had internal agreement that the topic mattered, made early plans to implement 24 25 31 including assigning responsibility for work, and regularly evaluated progress. However, how they 26 27 32 internally organised the work varied. Six practices were sampled because they had initial 28 29 33 implementation failure. Implementation failure occurred at different stages depending on practice 30 31 32 34 context, including internal disagreement about whether the work was worthwhile, and intention but 33 35 lack of capacity to implement or sustain implementation due to unfilled posts or sickness. Practice 34 http://bmjopen.bmj.com/ 35 36 36 context was not fixed, and most practices with initial failed implementation adapted to deliver at 37 38 37 least some elements. All interviewed participants valued the intervention because it was an 39 40 38 innovative way to address on an important aspect of safety (although one of the non-interviewed 41 42 39 GPs in one practice disagreed with this). Participants felt that reviewing existing prescribing did on October 3, 2021 by guest. Protected copyright. 43 44 45 40 influence their future initiation of targeted drugs, but raised concerns about sustainability. 46 47 48 41 Conclusions: Variation in implementation and effectiveness was associated with differences in how 49 50 42 practices valued, engaged with, and sustained the work required. Initial implementation failure 51 52 43 varied with practice context, but was not static, with most practices at least partially implementing 53 54 44 by the end of the trial. Practices organised their delivery of changed care to patients in ways which 55 56 45 suited their context, emphasising the importance of flexibility in any future widespread 57 58 59 60 2

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml Page 3 of 55 BMJ Open BMJ Open: first published as 10.1136/bmjopen-2016-015281 on 10 March 2017. Downloaded from 1 2 3 46 implementation. 4 5 6 47 Trial registration: ClinicalTrials.gov number, NCT01425502 7 8 9 48 Strengths and limitations of the study 10 11 49 • This is a comprehensive, pre-planned, process evaluation which includes a third of all 12 50 practices which participated in the DQIP stepped-wedge cluster-randomised trial. 13 14 51 • The evaluation sampled four practices which rapidly implemented the intervention and all 15 For peer review only 16 52 six practices which failed to implement the intervention to some degree. 17 53 • A strength of the study is the use of both qualitative data from interviews and observational 18 19 54 field notes, and quantitative data about key trial processes and practice-level effectiveness 20 21 55 to examine implementation in detail. 22 56 • A limitation is that we did not collect any data from practices prior to them receiving the 23 24 57 intervention. 25 26 58 27 Key words: 28 29 59 Complex intervention, process evaluation, general practice, primary care, prescribing practice. 30 31 60 32 33

34 61 Background http://bmjopen.bmj.com/ 35 36 62 High-risk prescribing in primary care is a major concern for health care systems internationally. 37 38 63 Between 2-4% of emergency hospital admissions are caused by preventable adverse drug events,[1, 39 40 64 2] at significant cost to healthcare systems.[3][4] A large proportion of these admissions are caused 41 42 65 by commonly prescribed drugs, with non-steroidal anti-inflammatory drugs (NSAIDs) and on October 3, 2021 by guest. Protected copyright. 43 44 66 antiplatelets being frequently implicated, causing gastrointestinal, cardiovascular and renal adverse 45 46 67 events.[5-7] 47 48 49 68 The DQIP intervention was systematically developed and optimised[8-10] and comprised three 50 51 52 69 intervention components: (a) professional education about the risks of NSAIDs and antiplatelets via 53 54 70 an outreach visit by a pharmacist; (b) financial incentives to review patients at the highest risk of 55 56 71 NSAID and antiplatelet ADEs, split into a participation fee of £350 and £15 per patient reviewed; and 57 58 59 60 3

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml BMJ Open Page 4 of 55 BMJ Open: first published as 10.1136/bmjopen-2016-015281 on 10 March 2017. Downloaded from 1 2 3 72 (c) access to a web-based IT tool to identify such patients and support structured review. The 4 5 73 intervention was evaluated in a pragmatic stepped-wedge cluster-randomised controlled trial[11] in 6 7 74 33 practices from one Scottish health board, where all participating practices received the 8 9 75 intervention but were randomised to one of ten different start dates.[8] Across all practices, 10 11 12 76 targeted high-risk prescribing fell from 3.7% immediately before to 2.2% at the end of the 13 14 77 intervention period (adjusted OR 0.63 [95%CI 0.57-0.68], p<0.0001). The intervention only 15 For peer review only 16 78 incentivised review of ongoing high-risk prescribing, but led to reductions in both ongoing (adjusted 17 18 79 OR 0.60, 95%CI 0.53-0.67) and ‘new’ high-risk prescribing (adjusted OR 0.77, 95%CI 0.68-0.87). 19 20 80 Notably reductions in high-risk prescribing were sustained in the year after financial incentives 21 22 81 stopped. In addition, there were significant reductions in emergency hospital admissions with 23 24 25 82 gastrointestinal ulcer or bleeding (RR 0.66, 95%CI 0.51-0.86), and heart failure (RR 0.73, 95%CI 0.56- 26 27 83 0.95).[12] 28 29 30 84 Alongside the main trial, we designed a mixed-methods process evaluation, [13, 14] based on a 31 32 85 cluster-randomised trial process-evaluation framework which we developed.[15] Our framework 33

34 86 emphasises the importance of considering two levels of intervention delivery and response that http://bmjopen.bmj.com/ 35 36 87 often characterise cluster-randomised trials of behaviour change interventions (although their 37 38 88 importance will depend on intervention design). The first is the intervention that is delivered to 39 40 89 clusters, which respond by adopting (or not) the intervention and integrating it with existing work. 41 42 on October 3, 2021 by guest. Protected copyright. 43 90 The second is the change in care which the cluster professionals deliver to patients. In DQIP, the 44 45 91 delivery of the intervention to professionals was pre-defined, intended to be delivered with high 46 47 92 fidelity across all practices by the research team, whereas the intervention delivered to patients was 48 49 93 at the discretion of practices, who decided whether and how they reviewed patients and whether to 50 51 94 change prescribing. We used this framework to structure our parallel process evaluation, mapping 52 53 95 data collection to a logic model of how the DQIP intervention was expected to work (figure 1). 54 55 56 96 The focus of this paper is on how different practices responded to the intervention delivered to 57 58 59 60 4

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml Page 5 of 55 BMJ Open BMJ Open: first published as 10.1136/bmjopen-2016-015281 on 10 March 2017. Downloaded from 1 2 3 97 them by the research team in terms of their adoption of the work, reorganisation to deliver the 4 5 98 intended change in care to patients, and whether implementation was sustained over time. 6 7 8 99 Methods 9 10 100 The overall design and methods have been described previously in the published protocol.[16] In 11 12 13 101 brief, we used a mixed-methods parallel process evaluation to examine the implementation of 14 15 102 selected processesFor and their peer associations with review change in high-risk prescribing only at practice level. The 16 17 103 quantitative element examined how change in prescribing at practice level was associated with 18 19 104 practice characteristics and implementation of key processes and is reported separately.[17] The 20 21 105 qualitative element consisted of case studies in 10 of the 33 participating practices. Practice staff 22 23 106 perceptions of the importance of different intervention components and when they had an effect 24 25 26 107 are reported separately.[18] The analysis reported here examines how practices adopted, 27 28 108 implemented and maintained the intervention. The design was a comparative case-study, with 29 30 109 general practices the unit of analysis.[19, 20] The study was reviewed by the Fife and Forth Valley 31 32 110 Research Ethics Committee (11/AL/0251) and informed consent was obtained from all participants 33

34 111 to participate and to published anonymised data. http://bmjopen.bmj.com/ 35 36 37 112 Case study sampling 38 39 113 Practices were purposively sampled to include those which had and had not initially reduced high- 40 41 42 114 risk prescribing, judged by visual inspection of run charts of high-risk prescribing rates approximately on October 3, 2021 by guest. Protected copyright. 43 44 115 four months after starting the intervention. Our assumption was change in high-risk prescribing was 45 46 116 a good proxy for initial response to the intervention delivered to practices. Sampling was additionally 47 48 117 structured to recruit practices starting the trial at different times, and to ensure a mix of small and 49 50 118 large practices (<5000 and ≥5000 registered patients), reflecting the stepped-wedge trial design and 51 52 119 our a-priori belief that smaller practices would more effectively implement the intervention 53 54 55 120 (randomisation was stratified by listsize).[16] Ten (30%) practices were sampled and recruited to the 56 57 121 process evaluation, including all six practices with no initial change in targeted prescribing. 58 59 60 5

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml BMJ Open Page 6 of 55 BMJ Open: first published as 10.1136/bmjopen-2016-015281 on 10 March 2017. Downloaded from 1 2 3 122 Qualitative data collection 4 5 123 In each practice, the GP most involved in the DQIP work, a GP less involved, the practice manager 6 7 124 and pharmacist were invited for interview. All interviews were carried out by AG approximately six 8 9 125 months after the practice started the intervention, and the GP most involved was interviewed again 10 11 12 126 three to six months later to explore changes over time. Interviews lasted approximately one hour, 13 14 127 and were audio-recorded and transcribed verbatim. Additional data generated were field-notes 15 For peer review only 16 128 made by AG during the educational outreach visits (EOV), detailing response to the educational 17 18 129 component and informatics tool training. Data generation took place between September 2011 and 19 20 130 December 2013 in parallel with intervention implementation. All data has been anonymised. 21 22 23 131 Quantitative data collection 24 25 132 More detailed information is given in the accompanying paper[17] but in brief, data from the DQIP IT 26 27 28 133 tool was used to measure reach, delivery to the patient and maintenance. Reach was measured as 29 30 134 the percentage of patients flagged as needing a review who had received at least one review, 31 32 135 delivery to the patient as the percentage of flagged patients who had further action in response to 33

34 136 initial review (eg a medication change, or an invitation to consult), and maintenance as reach in the http://bmjopen.bmj.com/ 35 36 137 final 24 weeks of the intervention period. Effectiveness was defined as the relative change in the 37 38 138 mean rate of high-risk prescribing trial primary outcome measure between baseline (the 48 weeks 39 40 41 139 before the intervention) and the final 24 weeks of the intervention. 42 on October 3, 2021 by guest. Protected copyright. 43 140 Researcher expectations of how the intervention would work 44 45 46 141 During intervention and process evaluation design the research team expected that several 47 48 142 processes had to happen for the intervention to reduce high-risk prescribing: 49 50 51 143 (1) Practices had to adopt the intervention, in the sense of being convinced that the work was 52 53 144 worthwhile, engaging with the education and set-up process, and organising how the work 54 55 145 required would be done.[22] This is represented by the ‘response of clusters’ box in figure 1. 56 57 58 59 60 6

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml Page 7 of 55 BMJ Open BMJ Open: first published as 10.1136/bmjopen-2016-015281 on 10 March 2017. Downloaded from 1 2 3 146 (2) Practices would have to deliver the reviews to patients by using the informatics tool to 4 5 147 identify patients, by a GP initially reviewing the records and taking any action they judged 6 7 148 necessary (for example, to stop the drug, or review the patient in person, or seek advice 8 9 149 from a specialist). This work is represented by the ‘reach’ and ‘delivery to patient’ boxes in 10 11 12 150 figure 1. 13 14 151 (3) Practices would then have to maintain this activity over the 48 weeks the intervention was 15 For peer review only 16 152 active (‘maintenance’ in figure 1). 17 18 19 153 Analysis 20 21 154 Analysis was iterative with data generation to allow issues and themes identified to inform 22 23 155 subsequent data generation and facilitate deeper exploration, and continued until no new themes 24 25 156 emerged. Analysis was carried out by AG with BG contributing through discussion of data and 26 27 28 157 interpretation until he became aware of the trial results, and was completed by AG before she knew 29 30 158 the outcome of the trial.[25] 31 32 33 159 A coding frame was developed inductively from field-notes, initial interviews and topic guides,

34 http://bmjopen.bmj.com/ 35 160 framework[15] and logic model,[16] and revised during analysis through use of the constant 36 37 161 comparative method.[23] NVivo 8 was used to systematically apply the coding frame to all data. An 38 39 162 in-depth description of each case study was constructed detailing characteristics and perceptions of 40 41 163 all practice staff that participated in interviews with additional data from the EOV observation and 42 on October 3, 2021 by guest. Protected copyright. 43 164 informal interviews. This facilitated a detailed exploration on a case-by-case basis. Analysis used the 44 45 46 165 Framework technique which facilitated comparing the data by concept, theme and practice and 47 48 166 facilitated cross and within case comparisons.[24]. Analysis drew on Normalisation Process Theory 49 50 167 (NPT), which focuses attention on how interventions become integrated, embedded and routinized 51 52 168 into social contexts.[22] The data was explored for negative cases. 53 54 55 169 Mixed methods design and analysis 56 57 170 The qualitative data were analysed and interpreted before the outcome of the trial or the 58 59 60 7

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml BMJ Open Page 8 of 55 BMJ Open: first published as 10.1136/bmjopen-2016-015281 on 10 March 2017. Downloaded from 1 2 3 171 quantitative process data was known.[25] The quantitative practice-level process and effectiveness 4 5 172 data were used primarily to examine practice characteristics associated with effectiveness,[19] but 6 7 173 are used in but used in this paper to help understand the cases. 8 9 10 174 Results 11 12 175 Findings are based on data from 38 interviews (ten lead GPs of whom nine were interviewed twice, 13 14 176 seven GPs less involved with DQIP, nine practice managers and three practice pharmacists) and from 15 For peer review only 16 177 approximately 11 hours of field-notes. Table 1 summarises the practice characteristics and intended 17 18 19 178 and actual process for delivering the intervention to patients, ordered by the final qualitative 20 21 179 judgement about the extent to which they implemented the intervention as intended by the 22 23 180 research team (table 2). 24 25 26 181 Perceptions of the work required and initial plans to implement 27 28 182 All GPs interviewed said they believed the targeted prescribing to be a cause of potentially 29 30 183 preventable harm which was worth addressing (although in one practice, one non-interviewed GPs 31 32 184 did not believe this – see below). There was a consistent perception that safety-focused prescribing 33

34 http://bmjopen.bmj.com/ 185 improvement work was more engaging for GPs (and patients) compared to previous experience of 35 36 37 186 interventions to reduce prescribing cost. Some GPs reported they had already reviewed NSAID 38 39 187 prescribing under a local health board initiative but DQIP was more meaningful because it focused 40 41 188 on and identified individuals at higher risk of adverse events. All but one interviewed GP perceived 42 on October 3, 2021 by guest. Protected copyright. 43 189 the DQIP intervention to be clearly differentiated from existing prescribing improvement activity, 44 45 190 and to be of significant value. 46 47 48 191 Although practices were free to organise the work as they saw fit, the research team knew 49 50 192 approximately half of patients requiring review over the year would be identified when the practice 51 52 53 193 started the intervention. At the educational visit, the research team therefore encouraged practices 54 55 194 to divide the work across more than one GP, and most practices initially agreed to do this. However, 56 57 195 many found this was either not feasible or sustainable (for example, where many or most GPs in a 58 59 60 8

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml Page 9 of 55 BMJ Open BMJ Open: first published as 10.1136/bmjopen-2016-015281 on 10 March 2017. Downloaded from 1 2 3 196 practice worked part-time), or in small practices was not necessary since one GP could comfortably 4 5 197 manage the workload. 6 7 8 198 However, all practices found this initial volume of work required dedicated time to complete and 9 10 199 could not be done opportunistically. Three practices (Mingulay, Lingay and Hirta) allocated 11 12 200 administrative time to carrying out the DQIP reviews, in the remaining practices the GPs reviewed 13 14 15 201 patients onceFor routine work peer was finished. review only 16 17 202 “…we ended up doing it in the evenings; it was after six o'clock, so it was sitting here at half 18 19 20 203 past six, seven o'clock in the evening ploughing through patients’ notes.” (Hellisay, GP 2 21 22 204 interview 1). 23 24 25 205 Case study practices which had not initially reduced high-risk prescribing 26 27 206 Six practices were sampled on the basis of having no initial change in high-risk prescribing, and were 28 29 207 assumed by the research team to be initial ‘non-adopters’. However, the reasons for this varied 30 31 208 across practices. 32 33

34 209 Failure to initially adopt the DQIP intervention. Three practices failed to initially adopt the DQIP http://bmjopen.bmj.com/ 35 36 210 intervention, although for different reasons and with different consequences. In Orosay and 37 38 39 211 Boreray, non-adoption was persistent, whereas Lingay did adopt after a delay. 40 41 42 212 The GPs in Orosay never agreed how the work required would be done, partly because of lack of on October 3, 2021 by guest. Protected copyright. 43 44 213 agreement that it was important to do. There was poor attendance at the initial EOV and at a second 45 46 214 visit requested by the practice after approximately six months. One GP in Orosay was highly 47 48 215 motivated and engaged, but one felt they had no obligation to participate despite having received 49 50 216 the initial payment, and another did not believe the targeted prescribing needed review. 51 52 53 217 “How much risk is each individual patient at? What will happen if we don’t do anything? 54 55 218 This GP’s questioning resulted in a long discussion with this GP arguing that he had little 56 57 219 experience of renal failure due to NSAIDs; he said he is an old GP and his experience shapes 58 59 60 9

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml BMJ Open Page 10 of 55 BMJ Open: first published as 10.1136/bmjopen-2016-015281 on 10 March 2017. Downloaded from 1 2 3 220 his prescribing.” (Extract from field notes 17.05.12) 4 5 6 221 During practice visits and during interviews in Orosay, administrative staff indicated that working 7 8 222 relationships among GPs were frayed at times and there was a belief that it was not appropriate for 9 10 223 one GP to change another’s prescribing. However, although patients were not reviewed as 11 12 224 intended, there was some evidence individual GPs in Orosay did change their prescribing as a result 13 14 15 225 of the educationalFor intervention. peer review only 16 17 226 “Yeah it's been because of DQIP really yes, aye. … It's merely made me sit up and pay a bit 18 19 20 227 more attention… It's more now when I see them, and I didn’t even know if their name’s on 21 22 228 the tool.” (Orosay, GP 1 interview 1) 23 24 25 229 In Boreray all GPs legitimised the work. The practice did agree a process by which to do the work but 26 27 230 they never got started due to persistent clinical and administrative understaffing: 28 29 30 231 “…we lost a partner… but the (other) doctors have cut their hours, they havnae got time…so 31 32 232 it's kinda been a wee bit o' upheaval in the last few months. So I was hoping they would take 33

34 233 on a full-time Practice Manager… but they've taken on a part-time one.” (Boreray, http://bmjopen.bmj.com/ 35 36 234 administrative interview) 37 38 39 235 As a result, the practice prioritised routine clinical work, although at the final interview the GP said 40 41 42 236 they had not known that each reviewed patient earned a £15 payment, and that knowing this might on October 3, 2021 by guest. Protected copyright. 43 44 237 have led to different priorities. 45 46 47 238 Lingay experienced a ‘delayed implementation’. When examining the practice data during the EOV, 48 49 239 it was obvious to the two GPs in the practice that many of the patients identified usually consulted 50 51 240 with one of them. This GP was keen to review identified patients, but this was delayed by 52 53 241 informatics access problems and little reviewing was initially done as motivation waned. 54 55 242 Subsequently, one of the GPs was temporarily unable to do clinical work for health reasons which 56 57 243 created time for doing work like DQIP. Once started, the GPs became highly motivated by seeing 58 59 60 10

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml Page 11 of 55 BMJ Open BMJ Open: first published as 10.1136/bmjopen-2016-015281 on 10 March 2017. Downloaded from 1 2 3 244 reductions in high-risk prescribing, and this led them to change their appointment system to protect 4 5 245 specific time to do quality improvement work. 6 7 8 246 “…we are making changes as of this month, we are staggering the times of surgeries so that 9 10 247 there's actually going to be proper paperwork time” (Lingay, GP1, interview 2). 11 12 13 248 Failure to consistently reach patients targeted by the DQIP intervention. In Mingulay all professionals 14 15 249 legitimised DQIP,For and agreed peer that one GP wouldreview do all the work in twoonly hours dedicated 16 17 250 administrative time per month. However, this was not enough time for the GP to complete the initial 18 19 20 251 large volume of reviews, exacerbated by the informatics tool initially running slowly (a problem for 21 22 252 some practices in the first waves of implementation). There was also poor communication with the 23 24 253 other GPs in the practice due to most being part-time with heavy use of locums. The GP doing the 25 26 254 work therefore felt that they never got on top of the reviewing, and perceived that stopped 27 28 255 prescribing was often restarted. 29 30 31 256 “The first few times we sort of went great guns and I got right down to, I only had 5 to 32 33 257 review and then of course the next time I went on there was something like 19 'cause

34 http://bmjopen.bmj.com/ 35 36 258 obviously there had been a couple that had re-triggered for whatever reason and then 37 38 259 others that had obviously come on in that time and it was very frustrating.” (Mingulay, GP 1 39 40 260 interview 1) 41 42 on October 3, 2021 by guest. Protected copyright. 43 261 Failure to deliver the DQIP intervention to patients. In Gighay, one GP committed the practice to 44 45 262 DQIP, attended the EOV and carried out all the review work, but made few changes, instead flagging 46 47 263 the patient’s record for the attention of whichever GP next saw them. As a result, GPs who had not 48 49 264 received the educational intervention (written educational materials were not internally distributed) 50 51 52 265 were responsible for delivering the change in prescribing, and delivery was therefore dependent on 53 54 266 both the patient attending for routine care and on the GP seeing them responding to the flag in the 55 56 267 record. 57 58 59 60 11

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml BMJ Open Page 12 of 55 BMJ Open: first published as 10.1136/bmjopen-2016-015281 on 10 March 2017. Downloaded from 1 2 3 268 Failure to maintain delivery of the DQIP intervention to patients. Hellisay was a two GP practice 4 5 269 where one GP attended the EOV and rapidly started reviewing patients. However, the other GP then 6 7 270 left, and although a new GP replaced them, the remaining original GP then went off sick. 8 9 10 271 “We blitzed the first numbers that we had, we then drifted along for a little bit doing some 11 12 272 (reviews)… roundabout Christmas we stopped doing DQIP 'cause Christmas is, is such a busy 13 14 15 273 rush time,For anyway peerDQIP dropped to review the bottom of the priority only list. Pretty much after 16 17 274 Christmas Dr X went off sick.” (Hellisay, GP 2, interview 1) 18 19 20 275 Reviewing was therefore not maintained, although the new partner said had they been aware of the 21 22 276 financial incentive, it might have encouraged them to prioritise DQIP over other work. 23 24 25 277 Case study practices which had large initial reductions in high-risk prescribing 26 27 278 Four practices were sampled on the basis of having large initial reductions in high-risk prescribing. In 28 29 279 these practices, there was full attendance by virtually all prescribers at the EOVs and there was 30 31 280 strong and widespread legitimation and commitment to deliver. Two were small practices which 32 33 281 initially decided to share the review work among all GPs, but within a couple of months of starting,

34 http://bmjopen.bmj.com/ 35 36 282 one more motivated and/or IT literate GP took responsibility for the work because it was “not too 37 38 283 onerous for one GP” (Interview Scalpay GP1, interview 1). In the two large practices the initial review 39 40 284 work and patient communication was shared by all GPs. In Hirta, one GP then took responsibility for 41 42 285 subsequent review work and consultations with patients, whereas in Taransay the work continued on October 3, 2021 by guest. Protected copyright. 43 44 286 to be shared across GPs and coordinated by a member of administrative staff. 45 46 47 287 Maintenance 48 49 288 No practice felt DQIP put an unfeasible burden on practice resources although it clearly required 50 51 52 289 time to review records and communicate with patients, and two practices (Hellisay and Boreray) 53 54 290 were unable to deliver the intervention due to being understaffed and prioritising competing 55 56 291 demands. Most GPs delivered the required work outside routine opening hours, although in three 57 58 59 60 12

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml Page 13 of 55 BMJ Open BMJ Open: first published as 10.1136/bmjopen-2016-015281 on 10 March 2017. Downloaded from 1 2 3 292 practices GPs used time already allocated for administrative work. As a result all practices primarily 4 5 293 communicated with patients by telephone. DQIP work was therefore done as separate activity 6 7 294 rather than fully integrated with existing work, with only Lingay modifying any broader processes or 8 9 295 systems as a result of the DQIP work. 10 11 12 296 Participants perceived that the review work influenced their future prescribing, saying that the work 13 14 15 297 was more ‘handsFor on’ than otherpeer prescribing reviewinterventions which were only pharmacist-led, and came 16 17 298 with clear and concise prescribing advice. 18 19 20 299 “'cause you're thinking “oh …”, when you're prescribing anti-inflammatory, is there a reason 21 22 300 I shouldn’t be or should I prescribe a PPI as well, so I think it probably does focus your mind.” 23 24 301 (Scalpay GP1, interview 1). 25 26 27 302 Since participants felt this applied less to GPs not doing reviewing, they used a range of strategies to 28 29 303 try to prevent drugs being restarted without due consideration, including informal discussion in the 30 31 304 coffee room or in practice meetings, and flagging the notes with a warning. GPs from the practices 32 33 305 which successfully reduced the targeted high risk prescribing continued to check the tool to ensure

34 http://bmjopen.bmj.com/ 35 36 306 no patients had been incorrectly restarted on high-risk drugs and no new patients had been 37 38 307 prescribed the targeted medication, and expressed concern that when the trial ended they would 39 40 308 not be able to check this. 41 42 on October 3, 2021 by guest. Protected copyright. 43 309 “…the worry is of course when, when the project stops whether things will drift back up 44 45 310 again, I just wondered if, if this software could be left running… … 'cause I think it's very, very 46 47 311 useful … I think, it's like everything, unless you're constantly reminded of things it does, it 48 49 312 does, it does tend to slip a little bit” (Monach GP 1,interview 2). 50 51 52 313 Unintended consequences 53 54 55 314 Unintended consequences were explored in the interviews but none were identified. 56 57 315 Quantitative data in relation to qualitative assessment of the extent of implementation 58 59 60 13

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml BMJ Open Page 14 of 55 BMJ Open: first published as 10.1136/bmjopen-2016-015281 on 10 March 2017. Downloaded from 1 2 3 316 Table 2 summarises the qualitative assessment of the extent of practice adoption and 4 5 317 implementation made before the trial results and quantitative process data were known, and 6 7 318 ordered from the least to most successful adopters/implementers. It additionally shows quantitative 8 9 319 measures of reach, delivery to patients, maintenance and effectiveness to contextualise the 10 11 12 320 qualitative judgement. The four practices sampled as having rapid initial reductions in high-risk 13 14 321 prescribing (Scalpay, Hirta, Monach, Taransay) were all qualitatively judged to have rapid and 15 For peer review only 16 322 sustained adoption and implementation. These four practices also had the highest quantitative 17 18 323 measures of reach, and among the highest measures of delivery and maintenance. They had 19 20 324 amongst the largest reductions in targeted high-risk prescribing across all practices in the trial (figure 21 22 325 2). 23 24 25 326 The six practices sampled as having no initial reduction in high-risk prescribing (assumed to be ‘initial 26 27 28 327 non-adopters’ during sampling) were more varied in both qualitative and quantitative assessment. 29 30 328 Table 2 shows the two practices qualitatively judged to have not adopted the intervention at all 31 32 329 (Orosay and Boreray) had very low rates of reach, delivery and maintenance, all of which were 33

34 330 measured using data from the informatics tool. Orosay did have a reduction in high-risk prescribing http://bmjopen.bmj.com/ 35 36 331 consistent with qualitative data that there was change in prescribing practice by at least some GPs 37 38 332 even though the informatics tool was never systematically used, but high-risk prescribing increased 39 40 333 in Boreray (table 2 and figure 2). The other four practices sampled as having no initial reduction in 41 42 on October 3, 2021 by guest. Protected copyright. 43 334 high-risk prescribing were qualitatively judged to have implemented with delay or in a way which the 44 45 335 practices themselves perceived as sub-optimal. They had generally intermediate levels of reach over 46 47 336 the whole intervention period (table 2). Consistent with the qualitative data, Hellisay had limited 48 49 337 delivery to patients of changed prescribing or follow-up, and poor maintenance with minimal change 50 51 338 in the rate of high-risk prescribing. The remaining three practices had somewhat lower but 52 53 339 overlapping levels of delivery to patients compared to the successful implementers, with similar 54 55 56 340 rates of maintenance, and somewhat lower rates of effectiveness, consistent with the qualitative 57 58 341 interpretation that these practices had delayed but ultimately successful implementation (table 2 59 60 14

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml Page 15 of 55 BMJ Open BMJ Open: first published as 10.1136/bmjopen-2016-015281 on 10 March 2017. Downloaded from 1 2 3 342 and figure 2). 4 5 6 343 Discussion 7 8 344 This study examined implementation of the DQIP intervention in ten practices purposively selected 9 10 345 for variation in initial reduction in targeted high-risk prescribing, which we assumed was a proxy for 11 12 13 346 initial adoption and implementation. For the four practices selected on the basis of early reductions 14 15 347 in high-risk prescribing,For both peer qualitative and review quantitative data showed only evidence of rapid and 16 17 348 sustained adoption and implementation, and they were among the most effective in terms of 18 19 349 practice-level change in high-risk prescribing. Our sample included all the six practices which had no 20 21 350 initial reduction in high-risk prescribing and these were all judged qualitatively to either have not 22 23 351 adopted at all because of disagreement about the value of the work, or to have experienced delay in 24 25 26 352 implementation related to their organisational context, and the quantitative measures were largely 27 28 353 consistent with the qualitative interpretation. 29 30 31 354 The way that practices did the work of implementation did therefore appear to be associated with 32 33 355 effectiveness. An attractive theoretical lens to examine this work is Normalisation Process Theory

34 http://bmjopen.bmj.com/ 35 356 (NPT).[22] The NPT construct coherence refers to participants’ understanding of the intervention. 36 37 357 Interviewed professionals in all practices clearly perceived the intervention and the associated 38 39 358 review work as different from existing quality improvement work because of its focus on safety 40 41 42 359 rather than cost. Cognitive participation relates to the enrolment and engagement of practices and on October 3, 2021 by guest. Protected copyright. 43 44 360 clinicians to the DQIP work. In all practices, there was at least one engaged GP, but collective 45 46 361 engagement varied across other GPs. Most or all GPs in nine of the case-study practices highly 47 48 362 legitimised the DQIP intervention, perceiving this as an innovative way to address an important 49 50 363 safety problem. However, in the one practice where there was openly stated disagreement about 51 52 364 value, implementation of the intervention failed from the outset (although there was evidence of 53 54 55 365 change in prescribing practice by GPs who did legitimise the work and of reduction in high-risk 56 57 366 prescribing, contrary to our belief that use of the informatics tool would be required for 58 59 60 15

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml BMJ Open Page 16 of 55 BMJ Open: first published as 10.1136/bmjopen-2016-015281 on 10 March 2017. Downloaded from 1 2 3 367 effectiveness). 4 5 6 368 Collective action focuses on how practices organised the work required for DQIP, which was usually 7 8 369 as a separate activity rather than fully integrated with existing work or practice routines. In the 9 10 370 smaller practices which successfully implemented the intervention immediately or with delay, one 11 12 371 GP did all the reviewing partly because the numbers made this feasible (contrary to our prior belief 13 14 15 372 that interventionFor effectiveness peer would require review sharing of the work). Inonly the larger practices the initial 16 17 373 bulk of work was shared between GPs which required coordination by an administrator, but 18 19 374 continued reviewing was shared or taken on by the most engaged GP. 20 21 22 375 Reflexive monitoring is about how participants assessed their progress in delivering the intervention, 23 24 376 and responded to problems. Many GPs in practices which successfully implemented the intervention 25 26 377 said that they regularly checked the tool to monitor progress. Interviewed GPs perceived that 27 28 378 reviewing altered their future prescribing but were concerned that this did not apply to their 29 30 379 colleagues. They described using a number of strategies to try to influence colleague’s prescribing to 31 32 33 380 try to ensure sustainability but were uncertain whether this was effective (although the main trial

34 http://bmjopen.bmj.com/ 35 381 analysis found significant reductions in new high-risk prescribing, and that reductions in total high- 36 37 382 risk prescribing were sustained in the 48 weeks after financial incentives ceased).[12] 38 39 40 383 Overall, coherence appeared uniformly high across case-study practices (perhaps unsurprising in 41 42 384 practices volunteering to participate in a trial), and so did not appear to influence implementation. on October 3, 2021 by guest. Protected copyright. 43 44 385 However, varied perceptions of the legitimacy of the work and low engagement (cognitive 45 46 386 participation) significantly contributed to poor adoption in two practices. More successful practices 47 48 49 387 delivered the intervention in a variety of ways which did not always match the research teams’ prior 50 51 388 beliefs about how best to deliver the intended care, highlighting that allowing practices flexibility to 52 53 389 implement in ways that worked for them was important (collective action). Successful implementers 54 55 390 also regularly reviewed their progress, and used a range of strategies to change their colleagues’ 56 57 391 practice to address their own concerns about sustainability (reflexive monitoring). 58 59 60 16

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml Page 17 of 55 BMJ Open BMJ Open: first published as 10.1136/bmjopen-2016-015281 on 10 March 2017. Downloaded from 1 2 3 392 Our other qualitative process evaluation paper reports that practice participants perceived all 4 5 393 primary components of the intervention delivered to them by the research team to be active 6 7 394 (financial incentives, education, informatics).[18] This paper complements and extends those 8 9 395 findings by examining how practices actually responded to the intervention delivered to them by 10 11 12 396 adopting and implementing the work of delivering changed care to patients. They are also consistent 13 14 397 with the quantitative examination of variation in effectiveness.[17] Six of the ten case-study 15 For peer review only 16 398 practices were sampled because they did not initially reduce high-risk prescribing (assumed to be 17 18 399 initial implementation failures), but most had delayed implementation and did achieve reductions in 19 20 400 targeted prescribing. The practice where qualitative and quantitative data was most incongruent 21 22 401 was Orosay, which was qualitatively judged to be an implementation failure (true in the sense that it 23 24 25 402 made no use of the informatics tool and claimed no payments for reviewing), but which had a 19% 26 27 403 reduction in targeted high-risk prescribing, consistent with the educational intervention altering 28 29 404 clinical practice. This supports the idea that multicomponent interventions to change professional 30 31 405 practice may be somewhat more effective than single component interventions, since professionals 32 33 406 or the organisations they work in may vary in how they respond (or not) to different elements.

34 http://bmjopen.bmj.com/ 35 36 407 In the wider literature, the intervention evaluated in the PINCER trial is the closest to DQIP in design 37 38 408 and focus.[25] Significant differences were that PINCER was pharmacist-led and delivered over 12 39 40 409 weeks, and that PINCER had a more standardised intervention in terms of how the pharmacists 41 42 on October 3, 2021 by guest. Protected copyright. 43 410 completed reviews. The PINCER process evaluation found that some of the pharmacists doing 44 45 411 reviews did not feel well integrated into the practice team, found the work repetitive and would 46 47 412 have preferred some flexibility to tailor the intervention to different practice contexts.[26] Both 48 49 413 interventions showed large reductions in targeted prescribing, but the DQIP intervention had 50 51 414 sustained effects 48 weeks after the intervention ceased whereas the PINCER effect at 12 months 52 53 415 was somewhat smaller than at six months.[25] The DQIP findings suggest that GP-led reviewing may 54 55 56 416 lead to more sustainable change by having greater influence on less involved GPs, and by changing 57 58 417 future initiation of high-risk prescribing. However, since both studies were carried out in volunteer 59 60 17

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml BMJ Open Page 18 of 55 BMJ Open: first published as 10.1136/bmjopen-2016-015281 on 10 March 2017. Downloaded from 1 2 3 418 practices, it is likely that non-volunteer practices will require greater effort to engage and greater 4 5 419 support to deliver similar outcomes which has implications for implementation across entire 6 7 420 healthcare systems. From this perspective, pharmacist-led interventions have the potential 8 9 421 advantage that it may be easier to deliver at least some change in practices which are less engaged 10 11 12 422 or which are resource constrained.[26] 13 14 15 423 ConclusionsFor peer review only 16 17 424 The DQIP intervention successfully reduced targeted high-risk prescribing [12] and reductions were 18 19 425 sustained in the 48 weeks after the intervention ceased. Overall, the four case-study practices which 20 21 426 immediately implemented the DQIP intervention had full or nearly full GP attendance at the EOV 22 23 427 consistent with collective engagement, rapidly identified and implement a process for delivering the 24 25 26 428 required work, and had at least one engaged and motivated GP who used a range of strategies to try 27 28 429 to influence their colleagues’ prescribing. In contrast, the six practices in which implementation was 29 30 430 problematic were more variable, with initial implementation failure occurring at different stages of 31 32 431 adoption and often being linked to wider practice context or resources. Context and resources were 33

34 432 however not fixed, with most of the initial implementation ‘failures’ delivering some or all elements http://bmjopen.bmj.com/ 35 36 433 of the intervention, leading to successful reductions in targeted high-risk prescribing. In wider 37 38 39 434 implementation, commissioners should pay careful attention to deploying educational and 40 41 435 persuasive strategies to ensure practices legitimise the work, should allow practices freedom to 42 on October 3, 2021 by guest. Protected copyright. 43 436 tailor how they implement the work of review, and should seek to offer additional support and 44 45 437 facilitation to practices struggling with short-term resource constraint or other contextual barriers 46 47 438 early in implementation. 48 49 50 439 51 52 53 440 54 55 56 441 List of abbreviations 57 58 442 AG – Aileen Grant 59 60 18

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml Page 19 of 55 BMJ Open BMJ Open: first published as 10.1136/bmjopen-2016-015281 on 10 March 2017. Downloaded from 1 2 3 443 BG – Bruce Guthrie 4 5 444 TD – Tobias Dreischulte 6 7 445 DQIP - Data-driven Quality Improvement in Primary care 8 9 446 EOV - Educational outreach visit 10 447 GP – General Practitioner 11 12 448 NICE - National Institute of Clinical Excellence 13 14 449 NPT – Normalisation Process Theory 15 For peer review only 16 450 NSAIDs - Non-steroidal anti-inflammatory drugs 17 18 451 TIDieR – Template for intervention description and replication 19 20 452 PINCER - A Pharmacist-led Information technology Intervention for Medication Errors 21 22 453 Competing interests 23 24 454 The authors declare that they have no competing interests. 25 26 455 Consent for publication 27 28 456 All participants gave consent for the publication of anonymised data. 29 30 457 31 Availability of data and supporting materials section 32 458 No observational or interview qualitative data are available to share because the NHS Research 33 459 Ethics approval for the study was on the basis of only the research team having access to the raw 34 http://bmjopen.bmj.com/ 35 460 data and no quantitative data are available to share because the permissions for data use from the 36 37 461 NHS organisations which are the data controllers only allow access in a Safe Haven environment by 38 39 462 the research team. 40 41 463 Funding 42 on October 3, 2021 by guest. Protected copyright. 43 464 The study was supported by a grant (ARPG/07/02) from the Scottish Government Chief Scientist 44 45 465 Office. The funder had no role in study design, data collection, analysis and interpretation, the 46 466 writing of the manuscript or the decision to publish. 47 48 49 50 51 52 53 54 55 56 57 58 59 60 19

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml BMJ Open Page 20 of 55 BMJ Open: first published as 10.1136/bmjopen-2016-015281 on 10 March 2017. Downloaded from 1 2 3 467 Authors' contributions 4 5 6 468 BG and AG designed the study. AG led the data collection and qualitative analysis, supported by BG 7 469 in interpretation. TD carried out the quantitative analysis. AG wrote the first draft of the manuscript 8 9 470 with all authors commenting on subsequent drafts. 10 11 471 Acknowledgements 12 13 472 We would like to thank all participating practices and the University of Dundee’s Health Informatics 14 15 473 Centre for dataFor management peer and anonymization, review the Advisory and Trialonly Steering Groups, and Debby 16 474 O’Farrell who provided administrative support. 17 18 19 475 Figure 1: DQIP process evaluation framework 20 21 476 Figure 2: All trial practices ranked in order of intervention effectiveness, with case study practices 22 477 identified*FOOTNOTE * Practices marked in green were sampled because they were judged to have initially rapidly 23 24 478 reduced the targeted prescribing; practices marked in red were sample because they were judged to have not initially 25 479 reduced the targeted prescribing 26 27 28 29 30 31 32 33

34 http://bmjopen.bmj.com/ 35 36 37 38 39 40 41 42 on October 3, 2021 by guest. Protected copyright. 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 20

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml BMJ Open: first published as 10.1136/bmjopen-2016-015281 on 10 March 2017. Downloaded from y divided the work. work. the divided y Initially agreed to divide the work, with rapid implementation by all GPs all GPs by implementation rapid with work, the divide to agreed Initially Initially agreed to divide the work, but actually rapid implementation by by implementation rapid actually but work, the divide to agreed Initially reviewing. the out carrying Initially agreed to divide the work and rapidly delivered by all GPs initially initially all GPs by delivered rapidly and work the divide to agreed Initially reviewing. the all doing GP one Did not agree process at EOV, but rapid implementation of one GP one GP of implementation rapid but EOV, at process agree not Did reviewing. maintained GP one done, reviews of bulk initial Once reviewing. Initially agreed to divide the work, but did not implement; one GP one implement; not did but work, the divide to agreed Initially allpatients. reviewing systematically Initially agreed that one GP would review all patients and flag notes for for notes flag and patients all review would GP one that agreed Initially delay. a after reviewed enthusiastically and systematically Initially agreed that one GP would review all patients in set 2 hours/month. hours/month. 2 inset patients all review would GP one that agreed Initially the on acting GPs other and consulting patient on relied so seen, next when flag Initially agreed that one GP would review, but actuall but review, would GP one that agreed Initially reduced further communication to GP GP poor & inadequate, was This impact Initially agreed to divide the work between GPs, but failed to implement implement to failed but GPs, between work the divide to agreed Initially reviewing maintain not they could meant changes Staff Failure to legitimise and no process to implement agreed, but the most most the agreed, but implement to process no and legitimise to Failure work of clinical prioritisation understaffing/ of because The process for delivering the intervention to patients, both planned by the the by planned both patients, to intervention the for delivering process The practice in clinical change wassome there said GP engaged practice and actual (based on interview and observational data). observational and interview on (based actual and practice -

b BMJ Open

http://bmjopen.bmj.com/ 3.7 7.1 4.2 3.7 5.0 3.4 3.2 7.0 2.5 6.6 Overall high Overall risk prescribing prescribing risk at rate baseline

reducing Reducing Reducing Large Reducing Small Reducing Large Not Not Small Not reducing Not Small Not reducing Not Small Not reducing Not Large Not reducing Not Small Not reducing Not Large Sampling (initial (initial Sampling Large change in in change and prescribing size) on October 3, 2021 by guest. Protected copyright. size size - For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml

For peer review only 6,000 FTE 4 3,500 FTE 2.7 5,500 FTE 4.3 3000 FTE 2 3,000 2FTE 2,500 FTE 1.9 9,000 FTE 5 3000 FTE 1.9 6,500 FTE 3.5 Approx list Approx 10,000 FTE 6.5 and full time time full and (FTE) equivalent GPs

a mised mised

5 1 8 6 4 7 3 9 10 Rando 2 group group

Mean practice rate of high-risk prescribing in the two years before starting the intervention intervention the starting before years two the in prescribing high-risk of rate practice Mean Practice group in terms of when started the intervention (1=first group to start, 10=last group to start)to group start, 10=last to group (1=first intervention the started when of interms group Practice a b Taransay Monach Hirta Scalpay Lingay Gighay Mingulay Hellisay Boreray Practice Orosay Table 1: Practice characteristics including planned and actual process for delivering care to patients patients to care fordelivering process andactual planned including characteristics Practice 1: Table 21 Page 21 of 55 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 BMJ Open: first published as 10.1136/bmjopen-2016-015281 on 10 March 2017. Downloaded from

d Page 22 of 55 75 75 59 59 67 67 56 56 77 77 53 53 28 28 6 -24 -24 19 19 Effectiveness % reductionin high- risk prescribing

b 43 43 68 68 50 50 43 43 47 47 44 44 7 48 48 0.7 0 Maintenance % ofpatients eligible with review a recorded in the final 24weeks of the intervention

a c 45 45 33 33 5 5 19 19 14 14 38 38 32 32 29 29 Delivery toDelivery patients % ofpatients eligible with change with in change prescribing recorded

a 90 90 3 2 92 92 64 64 95 95 Reach % of patients eligible 83 83 89 89 78 78 2 0 with review a recorded at point any during the intervention period 62 62 BMJ Open

http://bmjopen.bmj.com/ on October 3, 2021 by guest. Protected copyright. For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml For peer review only GPs byco-ordinatedand an administrative memberof staff. dividedamong GPs all then oneand GP reviewing.maintained process but reviewed EOV one GP at patients. all maintenance. The GPs dealt with immediately the bulk initial butstaff changes meantstruggledthey to consistently reviewing.maintain collectively legitimise the intervention.This was too work muchfor one individual so noprocess for was implementation agreed. Overall qualitative assessmentof implementation lost was DQIP motivation. implemented fullyoneby delay. GP a after to but divide the work one GP actuallythe reviewing. did all divide divide the work but problems with toaccess the informaticstool led to one would patientsGP review all andnotes flag so forseen, when next relied on patientconsulting and on GPs other acting the flag. month allocated to but deliver review, this was inadequate addressto thenumbers identified communication with limited with other GPs. work, butwork, to failed implement changesany because of understaffing and prioritisationa of work.clinical ractice period intervention 48 week the during issued prescriptions high-risk more or one with as patients defined Eligible a. period the intervention of (24weeks) second half the during issued prescriptions high-risk more or one with as patients defined Eligible b. in prescribing change a have to not assumed were recorded a review without c.Patients in increase an indicate numbers (negative started intervention the weeks48 before the to compared intervention the weeks24 of final the in practice each in high-riskprescribing change in relative as the Defined d. prescribing) high-risk Taransay was DQIP fully implemented from was the start. DQIPdelivered by all Monach was DQIP fully implemented from the Thestart. practice initially agreed Hirta was DQIP fully implemented from the Thestart. initial bulk of was work Scalpay Scalpay was DQIP fully implemented from the start. The practice did not agree Lingay intervention DQIP was not adopted. initially GPsinitially to agreed Gighay was DQIP adopted with to delivery limited patients. thatagreed They Mingulay was DQIP adopted with initial reasonable reach.had GP One hours2 per Hellisay was DQIP adopted, delivered initially patientsto but with low Boreray intervention DQIP was not Initially the GPsadopted. agreed tothe share Orosay Orosay intervention DQIP was notbecause there wasadopted failurea to Table 2: Comparison of overall qualitative assessment of implementation and quantitative measures or reach, delivery, maintenance and effectiveness effectiveness and maintenance delivery, reach, or measures quantitative and implementation of assessment qualitative overall of Comparison 2: Table P 22 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 Page 23 of 55 BMJ Open BMJ Open: first published as 10.1136/bmjopen-2016-015281 on 10 March 2017. Downloaded from 1 2 3 4 5 6 References 7 8 1. Hakkarainen KM, Hedna K, Petzold M, Hagg S: Percentage of patients with preventable 9 adverse drug reactions and preventability of adverse drug reactions - a meta analysis. 10 PLOS One 2012, 7(3):e33236. 11 12 2. Howard R, Avery A, Bissell P: Causes of preventable drug-related hospital admissions: 13 14 a qualitative study. Quality and Safety in Health Care 2008, 17(2):109-116. 15 For peer review only 16 3. National Institute for Health and Care Excellence (NICE),: Costing statement: Medicines 17 optimisation - Implementing the NICE guideline on medicines information (NG5). 18 Available at https://www.nice.org.uk/guidance/ng5/resources/costing-statement-6916717 19 [accessed 11/05/2016]. 2015. 20 21 4. Institute for Health Care Informatics, IMS: Responsible Use of Medicines 22 23 Report. http://www.imshealth.com/en/thought-leadership/ims- 24 institute/reports/responsible-use-of-medicines-report. 2012. 25 26 5. Howard RL, Avery A, Slavenburg S, Royal S, Pipe G, Lucassen P, Pirohamed M: Which 27 drugs cause preventable admissions to hospital? A systematic review. British Journal of 28 Clinical Pharmacology 2006, 63:136-147. 29 30 6. Budnitz DS, Lovegrove MC, Shehab N, Richards CL: Emergency Hospitalisation for 31 Adverse Drug Events in Older Americans. N Engl J Med 2011, 365(21):2002-2012. 32 33 7. Leendertse AJ, Egberts ACG, Stoker LJ, van den Bent PM: Frequency of and risk 34 http://bmjopen.bmj.com/ 35 factors for preventable medication-related hospital admissions in the Netherlands. Arch 36 Intern Med 2008, 168(17):1890-1896. 37 38 8. Dreischulte T, Grant A, Donnan P, McCowan C, Davey P, Petrie D, Treweek S, Guthrie B: 39 A cluster randomised stepped wedge trial to evaluate the effectiveness of a multifaceted 40 information technology-bsed intervention in reducing high-risk prescribing of non- 41 steroidal anti-inflammatory drugs and antiplatelets in primary care: the DQIP study 42 on October 3, 2021 by guest. Protected copyright. 43 protocol. Implementation Science 2012, 7:24. 44 45 9. Grant AM, Guthrie B, Dreischulte T: Developing a complex intervention to improve 46 prescribing safety in primary care: mixed methods feasibility and optimisation pilot 47 study. BMJ Open 2014, 4(1). 48 49 10. Guthrie B, McCowan C, Davey P, Simpson CR, Dreischulte T, Barnett K: High risk 50 prescribing in primary care patients particularly vulnerable to adverse drug events: 51 cross sectional population database analysis in Scottish general practice. British Medical 52 53 Journal 2011, 342. 54 55 11. Brown C, Lilford R: The stepped wedge trial design: a systematic review. BMC 56 Medical Research Methodology 2006, 6(1):54. 57 58 59 60 23

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml BMJ Open Page 24 of 55 BMJ Open: first published as 10.1136/bmjopen-2016-015281 on 10 March 2017. Downloaded from 1 2 3 12. Dreischulte T, Donnan P, Grant A, Hapca A, McCowan C, Guthrie B: Safer prescribing 4 - A Trial of Education, Informatics & Financial Incentives. New England Journal of 5 Medicine 2016, 374:1053-64. 6 7 13. Medical Research Council: Developing and evaluating complex interventions: new 8 9 guidance. 2008. 10 11 14. Medical Research Council: Process evaluation of complex interventions: UK Medical 12 Research Council (MRC) guidance. 2015. 13 14 15. Grant A, Treweek S, Dreischulte T, Foy R, Guthrie B: Process evaluations for cluster- 15 randomisedFor trials of complex peer interventions: review a proposed framework only for design and 16 reporting. Trials 2013, 14(1):15. 17 18 19 16. Grant A, Dreischulte T, Treweek S, Guthrie B: Study protocol of a mixed-methods 20 evaluation of a cluster randomised trial to improve the safety of NSAID and antiplatelet 21 prescribing: data-driven quality improvement in primary care. Trials 2012, 13:154. 22 23 17. Dreischulte T, Grant A, Guthrie B: Process evaluation of the Data-driven Quality 24 Improvement in Primary Care (DQIP) trial: Quantitative examination of 25 representativeness of trial participants and heterogeneity of impact. Submitted. 26 Implementation Science 2016. 27 28 29 18. Grant A, Dreischulte T, Guthrie B: Process evaluation of the Data-driven Quality 30 Improvement in Primary Care (DQIP) trial (1): Active and less active ingredients of a 31 multi-component complex intervention to reduce high-risk primary care prescribing. 32 Accepted Implementation Science 2016. 33

34 19. Stake R: The Art of Case Study Research: Thousand Oaks: Sage Publications; 1995. http://bmjopen.bmj.com/ 35 36 20. Crowe S, Cresswell K, Robertson A, Huby G, Avery A, Sheikh A: The case study 37 38 approach. BMC Medical Research Methodology 2011, 11(1):100. 39 40 21. Grant A, Sullivan F, Dowell J: An ethnographic exploration of influences on 41 prescribing in general practice: why is there variation in prescribing practices? 42

Implementation Science 2013, 8(1):72. on October 3, 2021 by guest. Protected copyright. 43 44 22. May C, Finch T: Implementing, Embedding, and Integrating Practices: An Outline 45 of Normalisation Process Theory. Sociology 2009, 43:535. 46 47 48 23. Glaser B, Strauss A: The discovery of grounded theory: strategies for qualitative 49 research: Chicago: Aldine Publishing Co; 1967. 50 51 24. Ritchie J, Spencer L, O'Connor W: Carrying out Qualitative Analysis. In Qualitative 52 Research Practice, A guide for Social Science Students and Researchers. Edited by Ritchie J, 53 Lewis J. London: Sage Publications Ltd; 2003:. 54 55 25. Avery AJ, Rodgers S, Cantrill JA, Armstrong S, Cresswell K, Eden M, Elliott RA, 56 57 Howard R, Kendrick D, Morris CJ, Prescott RJ, Swanwick G, Franklin M, Putman K, Boyd 58 M, Sheikh A: A pharmacist-led information technology intervention for medication 59 60 24

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml Page 25 of 55 BMJ Open BMJ Open: first published as 10.1136/bmjopen-2016-015281 on 10 March 2017. Downloaded from 1 2 3 errors (PINCER): a multicentre, cluster randomised, controlled trial and cost- 4 effectiveness analysis. The Lancet 2012, . 5 6 26. Cresswell K, Sadler S, Rodgers S, Avery A, Cantrill J, Murray S, Sheikh A: An 7 embedded longitudinal multi-faceted qualitative evaluation of a complex cluster 8 9 randomized controlled trial aiming to reduce clinically important errors in medicines 10 management in general practice. Trials 2012, 13(1):78. 11 12 13 14 15 For peer review only 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33

34 http://bmjopen.bmj.com/ 35 36 37 38 39 40 41 42 on October 3, 2021 by guest. Protected copyright. 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 25

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml BMJ Open Page 26 of 55 BMJ Open: first published as 10.1136/bmjopen-2016-015281 on 10 March 2017. Downloaded from 1 2 3 Figure 1: DQIP process evaluation framework 4 5 6 7 8 9 10 11 12 13 14 15 For peer review only 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33

34 http://bmjopen.bmj.com/ 35 36 37 38 39 40 41 42 on October 3, 2021 by guest. Protected copyright. 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml Page 27 of 55 BMJ Open BMJ Open: first published as 10.1136/bmjopen-2016-015281 on 10 March 2017. Downloaded from 1 2 3 4 Monach 5 Hirta 6 7 8 Scalpay 9 10 Taransay 11 12 Gighay 13 14 15 For peer review only Lingay 16 17 18 19 20 21 22 23 24 Mingulay 25 26 Orosay 27 28 Practices rankedin descending oforder intervention effectiveness 29 30 31 32 33 Hellisay

34 http://bmjopen.bmj.com/ 35 36 37 Boreray 38 -30 -20 -10 0 10 20 30 40 50 60 70 80 39 40 Effectiveness (% reduction in high-risk prescribing at end of the 41 intervention compared to baseline [negative values are an increase]) 42 on October 3, 2021 by guest. Protected copyright. 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml BMJ Open: first published as 10.1136/bmjopen-2016-015281 on 10 March 2017. Downloaded from BMJ Open Page 28 of 55 The new england journal of medicine

1 2 3 Special Article 4 5 6 7 Safer Prescribing — A Trial of Education, 8 Informatics, and Financial Incentives 9 10 Tobias Dreischulte, Ph.D., Peter Donnan, Ph.D., Aileen Grant, Ph.D., 11 Adrian Hapca, Ph.D., Colin McCowan, Ph.D., 12 and Bruce Guthrie, M.B., B.Chir., Ph.D.​​ 13 14 For peer review only 15 ABSTRACT 16 17 18 BACKGROUND 19 High-risk prescribing and preventable drug-related complications are common in From the Medicines Governance Unit, primary care. We evaluated whether the rates of high-risk prescribing by primary NHS Tayside (T.D.), and the Population 20 Health Sciences Division, University of 21 care clinicians and the related clinical outcomes would be reduced by a complex Dundee (P.D., A.H., B.G.), Dundee, the 22 intervention. School of Health Sciences, University of Stirling, Stirling (A.G.), and the Robert- 23 METHODS son Centre for Biostatistics, University 24 In this cluster-randomized, stepped-wedge trial conducted in Tayside, Scotland, we of Glasgow, Glasgow (C.M.) — all in Scotland. Address reprint requests to 25 randomly assigned participating primary care practices to various start dates for a 26 Dr. Guthrie at the University of Dundee, 48-week intervention comprising professional education, informatics to facilitate re- Mackenzie Bldg., Kirsty Semple Way, 27 view, and financial incentives for practices to review patients’ charts to assess appro- Dundee DD2 4BF, Scotland, or at 28 ­b​.­guthrie@​­dundee​.­ac​.­uk. priateness. The primary outcome was patient-level exposure to any of nine measures 29 of high-risk prescribing of nonsteroidal antiinflammatory drugs (NSAIDs) or selected N Engl J Med 2016;374:1053-64. 30 DOI: 10.1056/NEJMsa1508955 antiplatelet agents (e.g., NSAID prescription in a patient with chronic kidney disease Copyright © 2016 Massachusetts Medical Society. 31 http://bmjopen.bmj.com/ or coprescription of an NSAID and an oral anticoagulant without gastroprotection). 32 Prespecified secondary outcomes included the incidence of related hospital admis- 33 34 sions. Analyses were performed according to the intention-to-treat principle, with the 35 use of mixed-effect models to account for clustering in the data. 36 RESULTS 37 A total of 34 practices underwent randomization, 33 of which completed the study. 38 Data were analyzed for 33,334 patients at risk at one or more points in the preinterven-

39 tion period and for 33,060 at risk at one or more points in the intervention period. on October 3, 2021 by guest. Protected copyright. 40 Targeted high-risk prescribing was significantly reduced, from a rate of 3.7% (1102 of 41 29,537 patients at risk) immediately before the intervention to 2.2% (674 of 30,187) at 42 the end of the intervention (adjusted odds ratio, 0.63; 95% confidence interval [CI], 43 0.57 to 0.68; P<0.001). The rate of hospital admissions for gastrointestinal ulcer or 44 bleeding was significantly reduced from the preintervention period to the intervention 45 period (from 55.7 to 37.0 admissions per 10,000 person-years; rate ratio, 0.66; 95% CI, 46 0.51 to 0.86; P = 0.002), as was the rate of admissions for heart failure (from 707.7 to 47 513.5 admissions per 10,000 person-years; rate ratio, 0.73; 95% CI, 0.56 to 0.95; 48 P = 0.02), but admissions for acute kidney injury were not (101.9 and 86.0 admissions 49 per 10,000 person-years, respectively; rate ratio, 0.84; 95% CI, 0.68 to 1.09; P = 0.19). 50 51 CONCLUSIONS 52 A complex intervention combining professional education, informatics, and financial 53 incentives reduced the rate of high-risk prescribing of antiplatelet medications and 54 NSAIDs and may have improved clinical outcomes. (Funded by the Scottish Govern- 55 ment Chief Scientist Office; ClinicalTrials.gov number, NCT01425502.) 56 57 n engl j med 374;11 nejm.org March 17, 2016 1053 58 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml 59 60 BMJ Open: first published as 10.1136/bmjopen-2016-015281 on 10 March 2017. Downloaded from Page 29 of 55 BMJ Open The new england journal of medicine

1 2 igh-risk prescribing and prevent- in the Supplementary Appendix, available with 3 able drug-related complications in pri- the full text of this article at NEJM.org), and 4 mary care are major concerns for health structured financial incentives to review these H 1-7 5 care systems internationally. Up to 4% of patients. 6 emergency hospital admissions are caused by We developed and refined this intervention in 7 preventable adverse drug events,8-10 and in the four pilot practices in which participating physi- 8 United States, the cost of avoidable drug-related cians judged that approximately 40% of the tar- 9 hospital admissions, emergency department at- geted high-risk prescribing involving NSAIDs 10 tendances, and outpatient visits was estimated and antiplatelet agents required corrective ac- 11 at $19.6 billion in 2013.11 The majority of drug- tion.21 We evaluated the intervention in a cluster- 12 related emergency admissions are caused by com- randomized trial that examined the effect of the 13 monly prescribedFor drugs, peer with substantial review contri- intervention only on high-risk prescribing of NSAIDs 14 butions from nonsteroidal antiinflammatory and antiplatelet medications and related emer- 15 drugs (NSAIDs) and antiplatelet medications gency hospital admissions. 16 because of gastrointestinal, cardiovascular, and 17 4,5,7,8 renal adverse drug events. Despite routine Methods 18 public reporting of a number of indicators of 19 high-risk prescribing, variation among providers12 Study Design 20 and among Medicare hospital-referral regions13 In brief, the DQIP trial was a pragmatic, cluster- 21 is large and reductions in high-risk prescribing randomized trial that used a stepped-wedge de- 22 are minimal or slow.14,15 sign, in which all the participating practices re- 23 Decisions about prescribing often involve bal- ceived the DQIP intervention but were randomly 24 ancing benefits and risks as well as the prefer- assigned to 1 of 10 designated start dates be- 25 ences of the patient, and high-risk prescribing is tween October 30, 2011, and September 2, 2012. 26 therefore sometimes appropriate, since benefits Each practice received the intervention for a 48- 27 may be judged to outweigh the risk of harm to week period, with continued data collection for 28 12,16-19 29 a person. Nevertheless, the observed high 48 weeks after the active intervention ceased. 30 prevalence of and large variation in high-risk Figure S1 in the Supplementary Appendix shows prescribing patterns are consistent with the the stepped-wedge design of the study. The 31 http://bmjopen.bmj.com/ 32 hypothesis that the safety of prescribing in pri- methods of the study are described in detail in 13-17 33 mary care can be improved, and at a mini- the Supplementary Appendix. The study was ap- 34 mum, regular review to assess appropriateness proved by the National Health Service (NHS) 35 is required. Scotland Fife and Forth Valley research ethics 36 However, persuading primary care practices committee. The study was conducted and re- 37 to allocate scarce staff resources to improving ported with fidelity to the study protocol, which 38 the safety of prescribing is challenging, par- is available at NEJM.org. The authors vouch for ticularly when practices are independent, phy- the accuracy and completeness of the data pre-

39 on October 3, 2021 by guest. Protected copyright. 40 sician-owned small businesses that provide pri- sented. 41 mary medical care under contract to an external 42 payer, as is the case in many countries, includ- Participants 43 ing in the United Kingdom, where this trial was Physician-owned practices with contracts to pro- 44 performed.18 Therefore, the Data-Driven Quality vide NHS primary medical care in the Tayside 45 Improvement in Primary Care (DQIP) program region of Scotland were eligible to participate if 46 systematically developed a multifaceted inter- they used an electronic medical record (EMR) 47 vention for the improvement of prescribing system that was compatible with data extraction 48 safety in primary care practice.12,19-21 This inter- to the DQIP informatics tool.19 In the United 49 vention comprised professional education about Kingdom, patients are registered with one pri- 50 the risks of NSAIDs and antiplatelet medica- mary care practice that is responsible for all 51 tions, access to a Web-based tool to identify prescription of drugs in the community, includ- 52 patients at the highest risk for adverse drug ing any prescribing that is initiated on the rec- 53 events related to NSAIDs and antiplatelet agents ommendation of a specialist. In each practice, 54 (aspirin or clopidogrel, as defined in Table S1 patients were included at each data-measure- 55 56 57 1054 n engl j med 374;11 nejm.org March 17, 2016 58 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml 59 60 BMJ Open: first published as 10.1136/bmjopen-2016-015281 on 10 March 2017. Downloaded from BMJ Open Page 30 of 55 Safer Prescribing

1 2 ment point if they were alive, were permanently these drugs (Table S1 in the Supplementary Ap- 3 registered with the practice on that date, and pendix). The individual measures were related to 4 had one or more risk factors that made them three types of adverse drug events: gastrointesti- 5 particularly vulnerable to adverse drug events nal events (six measures; e.g., aspirin or clopido- 6 related to NSAIDs or antiplatelet agents. grel prescription for a patient taking an oral 7 anticoagulant without coprescription of a gas- 8 Intervention troprotective drug), renal events (two measures; 9 The development and detailed design of the inter- e.g., NSAID prescription for a patient with 10 vention have been described in previous reports19-21 chronic kidney disease), and heart failure (NSAID 11 and are summarized in the Supplementary Ap- prescription for a patient with heart failure).20,22-27 12 pendix. The intervention comprised three com- The composite primary outcome was defined as 13 ponents. First, practicesFor received peer professional review the percentage of patients only with any risk factor 14 education, with each practice receiving a 1-hour (e.g., taking an anticoagulant, having chronic 15 educational outreach visit by a pharmacist at the kidney disease, or having heart failure) who 16 start of the intervention (see Section 4 in the were currently receiving an antiplatelet agent, an 17 Supplementary Appendix for the presentation NSAID, or both in a way that was defined as 18 used), additional written material, and news- “high risk” by one or more individual measures. 19 letters (sent every 8 weeks) tailored to the prog- Prespecified secondary prescribing outcomes in- 20 ress of the practice. cluded ongoing (prescribed within the previous 21 Second, practices received financial incen- year) and new (not prescribed within the previ- 22 tives in the form of an initial fixed payment of ous year) high-risk prescribing and the rates of 23 £350 ($600 U.S.) and a payment of £15 ($25 U.S.) the nine prescribing outcome measures individ- 24 for every patient for whom the targeted high-risk ually. In each practice, all prescribing indicators 25 prescribing was reviewed during the intervention were measured every 8 weeks before and after 26 period (with only one claim allowed for each the practice started the intervention, with the 27 patient). The average expected payment per full- patients considered to be currently receiving a 28 29 time physician was approximately £550 ($910 U.S.), high-risk prescription if it had been issued at any 30 or approximately 0.6% of the average physician point in the preceding 8 weeks. income. Other prespecified secondary outcomes were 31 http://bmjopen.bmj.com/ 32 Third, an informatics tool extracted data from emergency hospital admissions for gastrointesti- 33 the EMR system of each practice, identified in- nal bleeding, acute kidney injury, or heart fail- 34 dividual patients who needed review, facilitated ure; we considered admissions that were pre- 35 reviews of patients by graphically displaying ceded by targeted high-risk prescribing as well 36 relevant drug histories, and provided weekly as all such admissions in patients with risk fac- 37 updates on the rates of high-risk prescribing and tors for adverse drug events related to NSAIDs 38 progress in reviewing. Physicians accessed the and antiplatelet agents, regardless of high-risk tool by means of a password-protected Web- prescribing. We also conducted a post hoc

39 on October 3, 2021 by guest. Protected copyright. 40 based portal. Identified patients were flagged by analysis involving these same patients to exam- 41 the tool as needing review. Physicians could ine changes in the rates of unrelated hospital 42 clear the flag by recording an explicit decision admissions for hip fracture, cancer, and surgical 43 that the prescribing was appropriate, by stop- emergencies (appendicitis, cholecystitis, or pan- 44 ping prescription of the offending drugs, or by creatitis) and in the rates of unrelated ambula- 45 adding a gastroprotective drug (proton-pump tory care sensitive admissions (defined as angi-

46 inhibitor or histamine2-receptor antagonist) for na, asthma, cellulitis, convulsions and epilepsy, 47 measures targeting the risk of gastrointestinal chronic obstructive pulmonary disease, dehydra- 48 bleeding. tion and gastroenteritis, dental conditions, dia- 49 betes complications, infection of the ear, nose, 50 Outcome Measures or throat, gangrene, hypertension, influenza and 51 The primary outcome measure was a composite pneumonia, nutritional deficiency, other vaccine- 52 of nine measures of high-risk prescribing of preventable disease, pelvic inflammatory disease, 53 NSAIDs and antiplatelet agents in people with or pyelonephritis) (Table S2 in the Supplemen- 54 risk factors for adverse drug events related to tary Appendix).28 All the outcomes were mea- 55 56 57 n engl j med 374;11 nejm.org March 17, 2016 1055 58 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml 59 60 BMJ Open: first published as 10.1136/bmjopen-2016-015281 on 10 March 2017. Downloaded from Page 31 of 55 BMJ Open The new england journal of medicine

1 2 sured with the use of routine data that were ex- Finally, in a post hoc analysis, we examined 3 tracted from practice EMRs (for prescribing) or whether the effect of the DQIP intervention was 4 from linked administrative data sets (for hospi- sustained after the end of the intervention. We 5 tal admissions). compared the rate of high-risk prescribing in the 6 last 24 weeks of the intervention period with 7 Randomization and Start-Date Concealment that in the last 24 weeks of the postintervention 8 Practices were assigned to 1 of 10 start dates by period using the same model as that used for the 9 an independent statistician who was unaware of primary outcome (Fig. S1 in the Supplementary 10 the identity of the practices, with randomization Appendix). 11 stratified by thirds of the number of registered 12 patients. Concealment of the start-date assign- Results 13 ment fromFor practices andpeer from the research review team only 14 was not possible, but prescribing measures were Participating Practices 15 calculated at a remote site by the data provider Of 66 Tayside practices we approached, 34 (52%) 16 independently of the research team before the agreed to participate, but 1 withdrew before its 17 data were transferred to the research team, with randomized start date. The intervention was 18 the use of the same algorithms that were used therefore implemented in 33 practices, with a 19 to provide feedback to practices. pooled list size of 202,262 patients on the date 20 that the practices started the intervention. A total 21 Statistical Analysis of 33,334 patients with risk factors for adverse 22 On the basis of data from the pilot practices19 and drug events related to NSAIDs and antiplatelet 23 the Pharmacist-led Information Technology In- agents were included in the analysis at one or 24 tervention for Medication Errors (PINCER) trial,29 more time points during the preintervention 25 we estimated that the study would have 83% period, and 33,060 patients were included in the 26 power to detect a 25% reduction in the primary intervention period (Fig. S2 in the Supplemen- 27 outcome, at an alpha level of 0.05, in 10 prac- tary Appendix). There was a significant but 28 19,30 29 tices randomly assigned to 10 start dates. clinically small rising trend in the rate of high- 30 We report the prespecified primary patient-level risk prescribing during the preintervention period analysis, as well as a secondary practice-level (mean absolute increase, 0.07 percentage points 31 http://bmjopen.bmj.com/ 32 analysis (see the Supplementary Appendix). for every 8 weeks elapsed; 95% confidence inter- 33 The analyses were performed according to val [CI], 0.02 to 0.12). 34 the intention-to-treat principle; we analyzed data 35 from all the eligible patients regardless of Preintervention Data 36 whether they actually received a review. Multi- Table 1 shows the preintervention characteristics 37 level logistic regression was used to estimate odds of the participating practices according to their 38 ratios and 95% confidence intervals for exposure randomly assigned start date. The numbers of to high-risk prescribing across data points dur- patients with risk factors for adverse drug events

39 on October 3, 2021 by guest. Protected copyright. 40 ing the intervention period as compared with related to NSAIDs and antiplatelet agents at the 41 the preintervention period (Fig. S1 in the Supple- start of the intervention ranged from 1894 to 42 mentary Appendix), with adjustment for cluster- 4857 patients across the 10 randomly assigned 43 ing within practices and over time. In prespeci- start dates, with 2.0 to 4.6% of the patients hav- 44 fied analyses of hospital admissions, we summed ing prescriptions for a high-risk NSAID, an anti- 45 events across all the practices during the pre­ platelet medication, or both at the start of the 46 intervention period and the intervention period intervention. Although there were significant dif- 47 and calculated rates by dividing the sums by the ferences in the mean age and sex of the patients 48 total person-time during which patients had risk across start dates, absolute differences were 49 factors for adverse drug events related to NSAIDs small (range of mean age, 72 to 76 years; range 50 and antiplatelet agents. The incidence in the in- of percentage of men, 45 to 48%). There were no 51 tervention period versus the preintervention pe- significant differences in mean practice-list sizes 52 riod was compared with the use of the condi- and quality of care as measured by performance 53 tional maximum-likelihood estimate of the rate on the U.K. Quality and Outcomes Framework,32 54 ratio.31 but the percentages of patients living in the most 55 56 57 1056 n engl j med 374;11 nejm.org March 17, 2016 58 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml 59 60 BMJ Open: first published as 10.1136/bmjopen-2016-015281 on 10 March 2017. Downloaded from BMJ Open Page 32 of 55 Safer Prescribing

1 2 3 1 4 1 10 987

4 5171

5 (977–996) (1962–8506) 6 7 8 9 3 0 1 983 9 5102 (966–995)

10 (2685–9358) 11 12 8 4 3 2

13 982 For peer review7183 only

14 (972–996)

15 (2980–11,643)

16 f clinical and organizational indicators, each of start date of the intervention. The randomized h the use of Scottish Index Multiple 7 3 1 17 2 985 art date. 18 5418 (965–997) 19 (2738–8888) 20 21 6 3 1 3 968 22 6535 (927–996)

23 (3643–9738) 24

25 Study Group 5 3 1 3

26 970 5769

27 (954–988) 28 (3224–7392) 29 30 4 4 1 0 968 759

31 http://bmjopen.bmj.com/ (960–980)

32 (3205–10,052) 33 34 3 3 3 0 971

35 5323 (948–1000)

36 (1367–7596) 37 38 2 3 3 1 966

39 on October 3, 2021 by guest. Protected copyright. 40 6097 (961–1000) 41 (2621–9804) 42 43 1 3 2 1 979 2236 3253 2384 4857 2849 2532 2631 3968 1894 2933 6128 74±12 74±11 72±13 74±12 76±11 74±13 75±12 75±12 74±12 76±12 44 (4.0)90 (4.6) 151 (3.0) 71 (3.8) 184 (4.0) 115 (3.8) 96 (4.1) 108 (4.2) 168 (3.1) 59 (2.0) 60 312 (14.0)312 (12.8) 416 (38.3) 913 (5.1) 246 (7.2) 205 (29.9) 757 (0.4) 10 (6.6) 263 (5.6) 106 (1.6) 48 (968–986) 1009 (45.1)1009 (45.1) 1466 (47.9) 1143 (47.5) 2308 (47.0) 1340 (45.1) 1141 (46.6) 1226 (46.2) 1833 (46.9) 889 (47.9) 1404 45 (3680–8621) 46 47 48 32 49 50 51 52 risk prescribing risk previous in prescribing (%) no. — wk 8 quintile of Scottish postal Scottish of quintile (%)‡ no. — codes mean (range)‖ mean 53 (range) patients Preintervention Characteristics of the Patients and Practices, According to Randomized Start Date.* Start Randomized to According Practices, and Patients the of Characteristics Preintervention 1. Table Patients † high- for factors risk with No. high-risk to Exposure Characteristic Age — yr — Age Male sex — no. (%) no. — sex Male Residence in most deprived most in Residence QOF points achieved — achieved points QOF Practices no. Total Urban practices — no.¶ — practices Urban Training practices — no.§ — practices Training Mean no. of registered of no. Mean Plus–minus values are means ±SD. The 33 participating practices were divided into 10 study groups, according to the randomized Data are the total number of patients (defined as people registered to practices) across practices randomly assigned each st Deprivation was defined as residence in the most socioeconomically deprived fifth of Scottish postal codes and measured wit Training practices were defined as that registered for providing general practitioner training. Urban practices were defined as that based in towns or cities with more than 10,000 residents. The Quality and Outcomes Framework (QOF) financially rewards primary care practices according to their performance on a range o dates all occurred between October 30, 2011, and September 2, 2012, with 1 indicating the earliest group to start 10 la test. There were significant between-group differences in the following characteristics: patients who had high-risk prescribing in previous 8 weeks, age of patients, sex, and pa tients living most socioeconomically deprived quintile Scottish postal codes (all P<0.001). Analyses were performed with the use of one-way analysis variance for means and chi -square test proportions. The proportions training and urban practices in each group were not formally compared owing to small numbers. Deprivation, which is a measure that applied to areas with mean of approximately 750 residents. which is associated with a number of maximum achievable points, each point corresponding to defined payment. QOF scores range from 0 1000, higher indicating better performance.      54  ¶ ‖ ‡ † § 55 * 56 57 n engl j med 374;11 nejm.org March 17, 2016 1057 58 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml 59 60 BMJ Open: first published as 10.1136/bmjopen-2016-015281 on 10 March 2017. Downloaded from Page 33 of 55 BMJ Open The new england journal of medicine

1 2 3 4.0 4 5 3.5 6 7 3.0 8 9 10 2.5 11

12 2.0 13 14 For peer review only 15 1.5 16 in the Previous 8 Wk (%) 17 1.0 18 19 20 0.5 Preintervention period Intervention period Postintervention period 21 Patients with Risk Factors Who Received a High-Risk Prescription 22 0.0 23 −40 −32 −24 −16 −8 0 8 16 24 32 40 48 56 64 72 80 88 96 24 Weeks Relative to the Start of the Intervention in Each Practice 25 26 No. of Patients 965 955 933 977 1036 1102 970 792 744 707 622 674 662 652 630 628 552 560 with High-Risk 27 Prescription 28 29 No. of Patients 27,775 27,642 27,565 27,766 28,801 29,537 30,166 30,224 30,246 30,242 30,204 30,187 30,213 30,245 29,804 29,416 29,053 28,616 Particularly 30 Vulnerable

31 to Adverse http://bmjopen.bmj.com/ 32 Drug Events 33 Figure 1. Overall Trends in the Primary Outcome of High-Risk Prescribing across the Preintervention, Intervention, and Postintervention 34 Periods. 35 Each data point represents the percentage of patients with risk factors who received a high-risk prescription during the 8 weeks previous 36 to the stated time point. Time point 0 is the randomized intervention start date in each practice. 37 38

39 on October 3, 2021 by guest. Protected copyright. 40 socioeconomically deprived areas varied signifi- to be required for 1296 of these cases (69.8%). 41 cantly (range, 0.4 to 38.3%). Follow-up involved contacting patients to discuss 42 appropriateness (515 patients [27.7% of all re- 43 Reviews Conducted by Practices views]), planning discussion at future routine 44 During the intervention period, 2905 patients contact (409 [22.0%]), changing prescriptions 45 with risk factors received at least one high-risk and informing the patient by letter (163 [8.8%]), 46 prescription. Approximately half of all patients or other actions (209 [11.2%]). 47 who needed review during the 48-week interven- 48 tion period were flagged for review at the first Prescribing Outcomes 49 log-in. The remaining patients were flagged Figure 1 shows trends in the primary outcome 50 later during the intervention period; these pa- of high-risk prescribing, relative to the interven- 51 tients had a mixture of prior intermittent use tion start date in each practice (data for the 10 52 and true new use of high-risk prescriptions. Of individual groups, according to the randomly 53 these patients, 1598 (55.0%) underwent a total assigned start dates, are shown in Figs. S3 and 54 of 1858 reviews. At review, follow-up was judged S4 in the Supplementary Appendix). In the 55 56 57 1058 n engl j med 374;11 nejm.org March 17, 2016 58 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml 59 60 BMJ Open: first published as 10.1136/bmjopen-2016-015281 on 10 March 2017. Downloaded from BMJ Open Page 34 of 55 Safer Prescribing

1 2 3 4 0.015 0.040 0.039 0.040 0.021 0.021 0.084 0.014 0.033 0.027 0.053 0.036§ <0.001 5 <0.001 Coefficient Correlation 6 Intracluster 7 8 9 10 0.29 0.01 0.04 <0.001 <0.001 <0.001 <0.001 <0.001 <0.001 <0.001 <0.001 <0.001 <0.001 <0.001 11 P Value 12 13 14 For peer review only 15 16

17 aving been prescribed the same drug during previ - 18 ients over time. Under the stepped-wedge design, ad risk factors for high-risk prescribing and who received a 19 (95% CI)‡ 0.78 (0.49–1.23) 0.78 0.76 (0.62–0.94) 0.76 0.77 (0.65–0.91) 0.77 0.31 (0.10–0.94) 0.31 (0.18–0.55) 0.32 0.44 (0.29–0.67) 0.44 0.39 (0.29–0.51) 0.39 20 (0.21–0.34) 0.27 Multilevel Adjusted Odds 21 Ratio of Intervention Effect 22 23 24 25 26 27 (0.1) 3/2970 28 29 30 Prevalence† http://bmjopen.bmj.com/

31 (%) no. no./total 32 33 34 33/1593 (2.1)33/1593 (2.1) 37/1784 12/2483 (0.5) 12/2483 50/2483 (2.0)50/2483 (1.2) 35/2970 61/8799 (0.7)61/8799 (0.5) 38/8206 205/5841 (3.5)205/5841 (2.4) 145/6004 391/8033 (4.9)391/8033 (3.4) 283/8347 533/12,166 (4.4)533/12,166 (3.0) 381/12,612 (0.69–0.90) 0.78 119/8799 (1.4)119/8799 (0.6) 53/8206 214/17,310 (1.2)214/17,310 (0.6) 109/17,694 (0.38–0.57) 0.46 252/3973 (6.3)252/3973 (2.5) 102/4026 639/24,734 (2.6)639/24,734 (1.2) 308/25,100 (0.50–0.62) 0.55 336/29,537 (1.1)336/29,537 (0.7) 211/30,187 (0.68–0.87) 0.77 766/29,537 (2.6)766/29,537 (1.5) 463/30,187 (0.53–0.67) 0.60

35 (3.7)1102/29,537 (2.2) 674/30,187 (0.57–0.68) 0.63 36 Interventionof Start At Intervention of End At 37

38

39 on October 3, 2021 by guest. Protected copyright. 40 41 42 43 44 45 46 47 48 49 50 system blocker and diuretic and blocker system patient taking an oral anticoagulant oral an taking patient oral anticoagulant oral of age taking aspirin taking age of of age taking aspirin taking age of of age of with prior peptic ulcer peptic prior with 51 factor risk any NSAID in patient with chronic kidney disease kidney chronic with patient in NSAID NSAID in patient taking both renin–angiotensin both taking patient in NSAID Aspirin or clopidogrel without gastroprotection in gastroprotection without clopidogrel or Aspirin NSAID without gastroprotection in patient taking an taking patient in gastroprotection without NSAID Clopidogrel without gastroprotection in patient ≥65 yr ≥65 patient in gastroprotection without Clopidogrel NSAID without gastroprotection in patient ≥65 yr≥65 patient in gastroprotection without NSAID NSAID without gastroprotection in patient ≥75 yr ≥75 patient in gastroprotection without NSAID NSAID or aspirin without gastroprotection in patient in gastroprotection without aspirin or NSAID New 52 Ongoing NSAID in patient with history of heart failure heart of history with patient in NSAID Renal composite Renal Gastrointestinal composite Gastrointestinal 53 factor¶ risk any with patient in prescribing High-risk Effect of the Intervention on Primary and Secondary High-Risk Prescribing Outcomes.* Prescribing High-Risk Secondary and Primary on Intervention the of Effect 2. Table Secondary prescribing outcomes prescribing Secondary Primary outcome: any high-risk prescribing in patient with in patient prescribing high-risk any outcome: Primary Outcome Measure high-risk prescription in the previous 8 weeks. adjusted odds ratios are calculated with the use of all data points in intervention period versus preintervention perio d and so represent average exposure to high-risk prescribing over the entire intervention period versus preintervention period. ous 48 weeks. New high-risk prescribing was defined as the patient receiving a prescription and not having had it dur ing previous NSAID denotes nonsteroidal antiinflammatory drug. Data are the total number of patients across practices, at start and end intervention period in each practice, who h Multilevel adjusted odds ratios account for the clustering of patients within practices and repeated measurement pat The intracluster correlation coefficient is the proportion of variation in outcome that due to between practices. Ongoing high-risk prescribing was defined as the patient receiving a prescription during measurement period and h     54  * ‡ § 55 † ¶ 56 57 n engl j med 374;11 nejm.org March 17, 2016 1059 58 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml 59 60 BMJ Open: first published as 10.1136/bmjopen-2016-015281 on 10 March 2017. Downloaded from Page 35 of 55 BMJ Open The new england journal of medicine

1 2 3 0.004 0.34 0.002 0.19 0.02 4 0.10 <0.001 5 P Value 6 7 8 9 10 (95% CI) 11 Rate Ratio 12 13 14 For peer review only 15 16 17

18 (95% CI) 19 20 Absolute Difference 21 22 23 24 25 (412.4 to 639.3)(412.4 to −38.9)to (−349.5 −194.2 0.95) to (0.56 0.73 5/1558 80/1558 1/23,228 13/11,740 86/23,228

26 101/11740 377/27,878 27 513.5

28 29 30

31 http://bmjopen.bmj.com/ 32 33 14/2374 168/2374 51/14,720 15/32,687 150/14,720 182/32,687 34 465/38,505 35 36 Preintervention Period† Intervention Period† 37

38

39 on October 3, 2021 by guest. Protected copyright. 40 41 42 43 44 45 46 47 48 49 50 51 high-risk prescribing in the 8 wk before admission before wk 8 the in prescribing high-risk gastrointestinal with patients in prescribing high-risk factors risk in patients with heart failure heart with patients in prescribing in patients with renal risk factors risk renal with patients in prescribing preceding high-risk prescribing high-risk preceding with gastrointestinal risk factors risk gastrointestinal with 52 factors No. of events/person-yr of No. No. of events/person-yr of No. Incidence — no. of events/10,000 person-yr (95% CI)(95% person-yr events/10,000 of no. — Incidence 45.6)to (25.8 34.6 18.9)to (5.9 11.1 −12.3)to (−34.8 −23.6 0.58) to (0.17 0.32 No. of events/person-yr of No. Incidence — no. of events/10,000 person-yr (95% CI)(95% person-yr events/10,000 of no. — Incidence 75.7)to (25.7 4.6 2.4)to (0.0 0.4 −1.7)to (−6.6 −4.2 0.52) to (0.00 0.09 Incidence — no. of events/10,000 person-yr (95% CI)(95% person-yr events/10,000 of no. — Incidence 96.6)to (33.6 59.0 71.1)to (11.8 32.1 14.9)to (−68.7 −26.9 1.51) to (0.20 0.54 No. of events/person-yr of No. Incidence — no. of events/10,000 person-yr (95% CI)(95% person-yr events/10,000 of no. — Incidence 64.4)to (48.2 55.7 45.7)to (30.0 37.0 −29.9)to (−7.4 −18.7 0.86) to (0.51 0.66 No. of events/person-yr of No. Incidence — no. of events/10,000 person-yr (95% CI)(95% person-yr events/10,000 of no. — Incidence 119.2)to (86.5 101.9 104.6)to (70.8 86.0 7.5)to (−39.3 −15.9 1.09) to (0.68 0.84 No. of events/person-yr of No. events/person-yr of No. Incidence — no. of events/10,000 person-yr (95% CI)(95% person-yr events/10,000 of no. — Incidence 132.3)to (110.0 120.8 149.6)to (121.9 135.2 32.0)to (−3.0 14.5 1.28) to (0.98 1.12 53 CI)(95% person-yr events/10,000 of no. — Incidence 823.2) to (608.4 707.7 Secondary Outcome Measures Regarding Emergency Hospital Admission in Patients with Risk Factors for Adverse Drug Events from NSAIDs or Antiplatelet Medications.* Antiplatelet or NSAIDs from Events Drug Adverse for Factors Risk with Patients in Admission Hospital Emergency Regarding Measures Outcome Secondary 3. Table Outcome Measure relevant with admission hospital drug-related Potentially by preceded bleeding admission or ulcer Gastrointestinal Heart failure admission preceded by high-risk prescribing high-risk by preceded admission failure Heart Acute kidney injury admission preceded by high-risk by preceded admission injury kidney Acute Potentially drug-related hospital admission regardless of regardless admission hospital drug-related Potentially patients in bleeding admission or ulcer Gastrointestinal Acute kidney injury admission in patients with renal risk renal with patients in admission injury kidney Acute 54 failure heart with patients in admission failure Heart admission hospital Unrelated factor risk any with patients in admission fracture Hip 55 56 57 1060 n engl j med 374;11 nejm.org March 17, 2016 58 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml 59 60 BMJ Open: first published as 10.1136/bmjopen-2016-015281 on 10 March 2017. Downloaded from BMJ Open Page 36 of 55 Safer Prescribing

1 2 analysis of the primary outcome, 674 of 30,187 3 patients (2.2%) with risk factors for related ad- 0.87 0.90 4 0.52 verse drug events had prescriptions for NSAIDs, 5 P Value antiplatelet agents, or both at the end of the in- 6 tervention, as compared with 1102 of 29,537 7 (3.7%) who had prescriptions immediately be- 8 fore the intervention (adjusted odds ratio, 0.63; 9 95% CI, 0.57 to 0.68; P<0.001). 10 There were significant reductions in the rates (95% CI) 11 Rate Ratio of ongoing high-risk prescribing (from 2.6% im- 12 mediately before the intervention to 1.5% at the 13 For peer reviewend onlyof the intervention; adjusted odds ratio, 14 0.60; 95% CI, 0.53 to 0.67; P<0.001) and new 15 high-risk prescribing (from 1.1% to 0.7%; ad- 16 justed odds ratio, 0.77; 95% CI, 0.68 to 0.87;

17 for high-risk prescribing. The analyses of potentially P<0.001). There were significant reductions in der the stepped-wedge design. 18 (95% CI) eight of the nine individual measures of high- 19 risk prescribing (range of adjusted odds ratios, Absolute Difference 20 0.27 to 0.78), but there was no significant reduc- 21 tion in the rate of NSAID prescribing for people 22 with heart failure (Table 2). 23 Reductions in the rates of high-risk prescrib- 24 ing were sustained after financial incentives 25 ceased. In an analysis of the primary outcome,

26 38/27,878 516/27,878 1426/27,878 674 of 30,187 patients (2.2%) with risk factors 27 for related adverse drug events had prescriptions 28 29 for NSAIDs, antiplatelet agents, or both at the 30 end of the intervention, and 560 of 28,616 (2.0%) had such prescriptions at 48 weeks after incen- 31 http://bmjopen.bmj.com/ 32 tives ceased (adjusted odds ratio, 1.14; 95% CI, 33 0.85 to 1.53; P = 0.38). for hospital admissions data in the United Kingdom, but heart failure, perforated peptic ulcer or 54/38,505 706/38,505 1926/38,505 34 28 35 Hospital Admission Outcomes 36 Preintervention Period† Intervention Period† Table 3 shows that among patients with risk fac- 37 tors for adverse drug events related to NSAIDs 38 and antiplatelet agents, the incidence of admis- sions preceded by a high-risk prescription (re-

39 on October 3, 2021 by guest. Protected copyright. 40 ceived in the previous 8 weeks) was significantly 41 reduced from the preintervention period to the 42 intervention period; admissions for gastrointes- 43 tinal bleeding were reduced from 4.6 per 10,000 44 person-years to 0.4 per 10,000 person-years (rate 45 ratio, 0.09; 95% CI, 0.00 to 0.52), and admis- 46 sions for acute kidney injury were reduced from 47 34.6 per 10,000 person-years to 11.1 per 10,000 48 person-years (rate ratio, 0.32; 95% CI, 0.17 to 49 0.58). The incidence of admissions for heart fail- 50 ure that were preceded by an NSAID prescription 51 was not significantly reduced from the preinter- 52 factor risk any with patients factor‡ risk vention period to the intervention period (59.0 No. of events/person-yr of No. Incidence — no. of events/10,000 person-yr (95% CI)(95% person-yr events/10,000 of no. — Incidence 197.4)to (170.1 183.4 200.8)to (169.5 185.1 22.7)to (−19.2 1.7 1.13) to (0.90 1.01 No. of events/person-yr of No. events/person-yr of No. Incidence — no. of events/10,000 person-yr (95% CI)(95% person-yr events/10,000 of no. — Incidence 18.3)to (10.5 14.0 18.7)to (9.6 13.6 5.3)to (−6.1 0.4 1.47) to (0.64 0.97 53 CI)(95% person-yr events/10,000 of no. — Incidence 523.0)to (478.1 500.2 538.8)to (485.3 511.5 46.0)to (−23.4 11.3 1.10) to (0.95 1.02 and 32.1 admissions per 10,000 person-years, re- Outcome Measure Any cancer admission in patients with any risk factor risk any with patients in admission cancer Any Appendicitis, cholecystitis, or pancreatitis admission in admission pancreatitis or cholecystitis, Appendicitis, any with patients in admission care–sensitive ambulatory Any bleeding, and iron-deficiency anemia were excluded because the investigators considered them to be potentially related t argeted prescribing. Ambulatory care–sensitive condi - tions included were angina, asthma, cellulitis, convulsions and epilepsy, chronic obstructive pulmonary disease, dehydration an d gastroenteritis, dental conditions, diabetes complica - tions, infection of the ear, nose, or throat, gangrene, hypertension, influenza and pneumonia, nutritional deficiency, other va ccine-preventable disease, pelvic inflammatory pyelonephritis. drug-related hospital admission with relevant high-risk prescribing in the 8 weeks before and potentially drug-relate d regardless of preceding prescribing were prespecified. The analysis of hospital admission that was considered by the investigators to be unrelated t he specified high-risk patterns targeted intervention was post hoc. Data are the number of outcome events over preintervention and intervention periods, during which patients had risk factors The differences in person-time are due to longer preintervention periods practices randomly assigned later start dates un Ambulatory care–sensitive admissions were as defined by Purdy et al.   54  ‡ † * spectively; rate ratio, 0.54; 95% CI, 0.20 to 1.51). 55 56 57 n engl j med 374;11 nejm.org March 17, 2016 1061 58 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml 59 60 BMJ Open: first published as 10.1136/bmjopen-2016-015281 on 10 March 2017. Downloaded from Page 37 of 55 BMJ Open The new england journal of medicine

1 2 Among patients with risk factors for adverse the intervention led to reductions in the rates of 3 drug events related to NSAIDs and antiplatelet related emergency hospital admissions, with no 4 agents, regardless of preceding high-risk pre- change observed in the rate of unrelated admis- 5 scribing, the incidence of total hospital admis- sions. 6 sions for gastrointestinal ulcer or bleeding was The strengths of the study include the careful 7 significantly reduced (from 55.7 to 37.0 admis- intervention design,12,19-21 evaluation in routine 8 sions per 10,000 person-years; rate ratio, 0.66; primary care practice, and examination of sus- 9 95% CI, 0.51 to 0.86), as was the incidence of tainability after financial incentives ceased. The 10 heart failure (from 707.7 to 513.5 admissions per study also has some important limitations. First, 11 10,000 person-years; rate ratio, 0.73; 95% CI, although half the eligible practices took part, 12 0.56 to 0.95). The incidence of total admissions participating practices may have been more 13 for acuteFor kidney injury peer was not significantly review re- motivated oronly had greater capacity to change than 14 duced from the preintervention period to the nonparticipating practices. Second, the stepped- 15 intervention period (101.9 and 86.0 admissions wedge design can be vulnerable to secular trends 16 per 10,000 person-years, respectively; rate ratio, if outcomes are already improving.30 The rate of 17 0.84; 95% CI, 0.68 to 1.09). targeted high-risk prescribing was rising slowly 18 Among patients with risk factors for adverse during the preintervention period and there were 19 drug events related to NSAIDs and antiplatelet no other relevant nontrial interventions during 20 agents, there was no significant change between the period of implementation, which indicates 21 the preintervention period and the intervention that secular trends are unlikely to have affected 22 period in the incidence of admissions for hip the prescribing outcomes. However, we were un- 23 fracture (120.8 and 135.2 admissions per 10,000 able to reliably examine prior time trends for the 24 person-years, respectively; rate ratio, 1.12; 95% hospital admission outcomes examined, since 25 CI, 0.98 to 1.28), cancer (183.4 and 185.1 admis- they are relatively rare events, but the rates of 26 sions per 10,000 person-years; rate ratio, 1.01; four different types of unrelated admission did 27 95% CI, 0.90 to 1.13), or appendicitis, cholecys- not fall significantly, which increases confidence 28 29 titis, or pancreatitis (14.0 and 13.6 admissions in the observed findings. 30 per 10,000 person-years; rate ratio, 0.97; 95% CI, It should be noted that the reductions in the 0.64 to 1.47), or in the incidence of unrelated total rates of admissions due to gastrointestinal 31 http://bmjopen.bmj.com/ 32 ambulatory care sensitive admissions (500.2 and bleeding and heart failure were larger than can 33 511.5 admissions per 10,000 person-years; rate be explained by reductions in the rates of such 34 ratio, 1.02; 95% CI, 0.95 to 1.10). admissions that were actually preceded by the 35 targeted high-risk prescribing. A recent study of 36 Discussion a large pay-for-performance program in primary 37 care that was conducted in the United Kingdom 38 We found that a complex intervention that com- also showed larger reductions in the rates of bined professional education, informatics to emergency admission than could be explained

39 on October 3, 2021 by guest. Protected copyright. 40 identify high-risk patients, and financial incen- by changes in targeted process measures.33 Al- 41 tives to primary care clinicians significantly re- though such halo effects are conceivable (e.g., 42 duced the rate of high-risk prescribing of NSAIDs owing to safer use of other medicines that cause 43 and antiplatelet medications, which is a common the same adverse effects or recommendations to 44 cause of drug-related emergency hospital admis- patients to avoid over-the-counter NSAIDs and 45 sions internationally.4,5,7,8 The effect on ongoing aspirin), confirmation in larger studies that are 46 high-risk prescribing was somewhat larger than powered to robustly examine hospital admission 47 the effect on new high-risk prescribing — a outcomes is required. 48 finding that was consistent with the fact that the Finally, although it is likely that this interven- 49 intervention prompted review of patients who tion would be effective for similar prescribing 50 were already exposed to high-risk prescribing decisions (those based on an assessment of 51 — although both rates fell significantly, and a benefit and harm to the individual patient), com- 52 lower rate of initiation of prescribing contribut- mon adverse drug events that are associated with 53 ed to the sustained effect in the year after finan- other mechanisms will require different ap- 54 cial incentives ceased. There was evidence that proaches. For example, reducing warfarin-related 55 56 57 1062 n engl j med 374;11 nejm.org March 17, 2016 58 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml 59 60 BMJ Open: first published as 10.1136/bmjopen-2016-015281 on 10 March 2017. Downloaded from BMJ Open Page 38 of 55 Safer Prescribing

1 2 harm is likely to require improvements in the 12 months.29 In contrast, the DQIP intervention 3 reliability of monitoring and patient education. had a sustained effect over the year after the fi- 4 This trial showed that a blend of education, nancial incentives ceased, which is important to 5 financial incentives to review high-risk patients, allow the focus of quality-improvement activity 6 and informatics to support review was effective, to move to other areas of care. 7 but we cannot identify which aspect of the inter- The three components of this intervention are 8 vention mattered most. Financial incentives to feasible in any system in which primary care is 9 stop high-risk prescribing are potentially prob- delivered by physician-owned practices that use 10 lematic because there may be a risk of inappro- EMRs and are under contract with third-party 11 priate cessation to obtain payment in a subgroup payers, which is a common model internation- 12 of patients in whom the balance of benefit and ally. However, complex interventions of this type 13 harm favors continuationFor of thepeer drug. We there review- inevitably require tailoring only to context. For exam- 14 fore chose to provide incentives to review. Since ple, the size of the incentives that are required 15 this approach could lead to “tick-box review” to prompt review is likely to vary according to 16 simply to obtain payment, we sought to ensure primary care physician payment structures and 17 the delivery of meaningful review through the use incomes. Similarly, in some contexts, it may be 18 of education and informatics, because an under- easier to embed the informatics functionality 19 standing of risk and an awareness that current directly in the EMR (e.g., if all targeted providers 20 practice is suboptimal are likely to be prerequi- use one EMR system and the EMR supplier is 21 sites for changing prescribing behavior, and there willing to cooperate). Although we think that it 22 was good evidence from previous studies that is likely that interventions that blend education, 23 both educational outreach visits and audit and incentives to review, and informatics will be ef- 24 feedback practices can lead to small improve- fective in other health care systems, any imple- 25 ments in prescribing.34,35 mentation should be tailored to context and 26 Feedback was one element of the informatics; evaluated for effect. 27 the other was a reduction in barriers to efficient In conclusion, we found that a complex inter- 28 29 structured review. Education and minimization vention that combined professional education, 30 of the barriers to change were also elements in informatics to support patient identification and the successful PINCER intervention (based in the review, and financial incentives to review patients 31 http://bmjopen.bmj.com/ 32 United Kingdom), in which external pharmacists who have been exposed to high-risk prescribing 33 reviewed patients who were exposed to high-risk led to substantial and sustained reductions in 34 prescribing and worked with primary care physi- targeted high-risk prescribing and was associ- 35 cians to implement change. The PINCER trial ated with reductions in the rates of related emer- 36 showed reductions in the rates of high-risk pre- gency admissions. scribing that were similar to those observed Supported by a grant (ARPG/07/02) from the Scottish Govern- 37 ment Chief Scientist Office. 38 with the DQIP intervention at 6 months, but the Disclosure forms provided by the authors are available with effect of pharmacist-led review partly waned by the full text of this article at NEJM.org.

39 on October 3, 2021 by guest. Protected copyright. 40 41 References 42 1. Budnitz DS, Lovegrove MC, Shehab N, 4. Pirmohamed M, James S, Meakin S, 8. Howard RL, Avery AJ, Slavenburg S, 43 Richards CL. Emergency hospitalizations et al. Adverse drug reactions as cause of et al. Which drugs cause preventable ad- 44 for adverse drug events in older Ameri- admission to hospital: prospective analy- missions to hospital? A systematic review. cans. N Engl J Med 2011;​365:​2002-12. sis of 18,820 patients. BMJ 2004;​329:​15-9. Br J Clin Pharmacol 2007;​63:​136-47. 45 2. Leendertse AJ, de Koning FH, Goud- 5. Gurwitz JH, Field TS, Harrold LR, et al. 9. Thomsen LA, Winterstein AG, Søn- 46 swaard AN, et al. Preventing hospital Incidence and preventability of adverse dergaard B, Haugbølle LS, Melander A. 47 admissions by reviewing medication drug events among older persons in the Systematic review of the incidence and (PHARM) in primary care: design of the ambulatory setting. JAMA 2003;​289:​1107- characteristics of preventable adverse drug 48 cluster randomised, controlled, multi- 16. events in ambulatory care. Ann Pharmaco- 49 centre PHARM-study. BMC Health Serv 6. Bates DW, Cullen DJ, Laird N, et al. ther 2007;41:​ 1411-26.​ 50 Res 2011;​11:4.​ Incidence of adverse drug events and po- 10. Hakkarainen KM, Hedna K, Petzold 51 3. Aspden P, Bootman L, Cronenwett L. tential adverse drug events: implications M, Hägg S. Percentage of patients with Preventing medication errors (Quality for prevention. JAMA 1995;​274:​29-34. preventable adverse drug reactions and 52 chasm series). Washington, DC:​ Institute 7. Gandhi TK, Weingart SN, Borus J, preventability of adverse drug reactions 53 of Medicine of the National Academies, et al. Adverse drug events in ambulatory — a meta-analysis. PLoS One 2012;​7(3):​ 54 2007. care. N Engl J Med 2003;​348:​1556-64. e33236. 55 56 57 n engl j med 374;11 nejm.org March 17, 2016 1063 58 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml 59 60 BMJ Open: first published as 10.1136/bmjopen-2016-015281 on 10 March 2017. Downloaded from Page 39 of 55 BMJ Open Safer Prescribing

1 2 11. IMS Institute for Health Care Infor- tory drugs and antiplatelets in primary and management — complementary roles matics. Avoidable costs in U.S. health- medical care: the DQIP study protocol. for COX-2 inhibitors and proton pump in- 3 care: ​the $200 billion opportunity from Implement Sci 2012;7:​ 24.​ hibitors. Gut 2003;​52:600-8.​ 4 using medicines more responsibly. 2013 20. Dreischulte T, Grant AM, McCowan 28. Purdy S, Griffin T, Salisbury C, Sharp 5 (http://www​.imshealth.com/​ ​en/​thought C, McAnaw JJ, Guthrie B. Quality and D. Ambulatory care sensitive conditions: -leadership/ims-institute/​ reports/​ ​avoidable safety of medication use in primary care: terminology and disease coding need to 6 -costs). consensus validation of a new set of ex- be more specific to aid policy makers and 7 12. Guthrie B, McCowan C, Davey P, plicit medication assessment criteria and clinicians. Public Health 2009;​123:​169- 8 Simpson CR, Dreischulte T, Barnett K. prioritisation of topics for improvement. 73. 9 High risk prescribing in primary care BMC Clin Pharmacol 2012;​12:​5. 29. Avery AJ, Rodgers S, Cantrill JA, et al. patients particularly vulnerable to adverse 21. Grant AM, Guthrie B, Dreischulte T. A pharmacist-led information technol­ 10 drug events: cross sectional population Developing a complex intervention to im- ogy intervention for medication errors 11 database analysis in Scottish general prac- prove prescribing safety in primary care: (PINCER): a multicentre, cluster random­ 12 tice. BMJ 2011;342:​ d3514.​ mixed methods feasibility and optimisa- ised, controlled trial and cost-effectiveness 13 13. Zhang Y, Baicker K, Newhouse JP. tion pilot study. BMJ Open 2014;​4(1):​ analysis. Lancet 2012;379:​ 1310-9.​ GeographicFor variation in the peerquality of pre- e004153. review only30. Hussey MA, Hughes JP. Design and 14 scribing. N Engl J Med 2010;​363:​1985-8. 22. Gooch K, Culleton BF, Manns BJ, et analysis of stepped wedge cluster random­ 15 14. Improving quality and patient experi- al. NSAID use and progression of chronic ized trials. Contemp Clin Trials 2007;​28:​ 16 ence: the state of health care quality 2013. kidney disease. Am J Med 2007;​120(3):​ 182-91. Washington, DC: ​National Committee for 280.e1-7. 31. Martin DO, Austin H. Exact estimates 17 Quality Assurance, 2013. 23. Huerta C, Castellsague J, Varas-Lorenzo for a rate ratio. Epidemiology 1996;​7:​29- 18 15. Davidoff AJ, Miller GE, Sarpong EM, C, Garcia Rodriguez LA. Nonsteroidal anti- 33. 19 Yang E, Brandt N, Fick DM. Prevalence of inflammatory drugs and risk of ARF in 32. Roland M, Campbell S. Successes and potentially inappropriate medication use the general population. Am J Kidney Dis failures of pay for performance in the 20 in older adults using the 2012 Beers crite- 2005;​45:531-9.​ United Kingdom. N Engl J Med 2014;​370:​ 21 ria. J Am Geriatr Soc 2015;​63:​486-500. 24. Dreischulte T, Morales DR, Bell S, 1944-9. 22 16. Cribb A, Barber N. Prescribers, pa- Guthrie B. Combined use of nonsteroidal 33. Harrison MJ, Dusheiko M, Sutton M, tients and policy: the limits of technique. anti-inflammatory drugs with diuretics Gravelle H, Doran T, Roland M. Effect of 23 Health Care Anal 1997;​5:​292-8. and/or renin-angiotensin system inhibi- a national primary care pay for perfor- 24 17. Dreischulte T, Guthrie B. High-risk tors in the community increases the risk mance scheme on emergency hospital 25 prescribing and monitoring in primary of acute kidney injury. Kidney Int 2015;​88:​ admissions for ambulatory care sensitive 26 care: how common is it, and how can it 396-403. conditions: controlled longitudinal study. be improved? Ther Adv Drug Saf 2012;​3:​ 25. Hallas J, Dall M, Andries A, et al. Use BMJ 2014;349:​ ​g6423. 27 175-84. of single and combined antithrombotic 34. Forsetlund L, Bjørndal A, Rashidian A, 28 18. Crabtree BF, Nutting PA, Miller WL, therapy and risk of serious upper gastro- et al. Continuing education meetings 29 et al. Primary care practice transforma- intestinal bleeding: population based case- and workshops: effects on profession- tion is hard work: insights from a 15-year control study. BMJ 2006;​333:​726. al practice and health care outcomes. 30 developmental program of research. Med 26. Gislason GH, Rasmussen JN, Abild- Cochrane Database Syst Rev 2009;​2:​

31 Care 2011;​49:Suppl:​ ​S28-35. strom SZ, et al. Increased mortality and CD003030. http://bmjopen.bmj.com/ 32 19. Dreischulte T, Grant A, Donnan P, cardiovascular morbidity associated with 35. O’Brien MA, Rogers S, Jamtvedt G, et al. et al. A cluster randomised stepped wedge use of nonsteroidal anti-inflammatory Educational outreach visits: effects on 33 trial to evaluate the effectiveness of a multi- drugs in chronic heart failure. Arch In- professional practice and health care out- 34 faceted information technology-based tern Med 2009;169:​ 141-9.​ comes. Cochrane Database Syst Rev 2007;​ 35 intervention in reducing high-risk pre- 27. Hawkey CJ, Langman MJ. Non-steroidal 4:​CD000409. 36 scribing of non-steroidal anti-inflamma- anti-inflammatory drugs: overall risks Copyright © 2016 Massachusetts Medical Society. 37 38

39 on October 3, 2021 by guest. Protected copyright. 40 41 42 43 We are seeking a Deputy Editor to join our editorial team at the New England Journal 44 of Medicine. The Deputy Editor will review, select, and edit manuscripts for the 45 Journal as well as participate in planning the Journal’s content. He or she will also ensure that the best available research is submitted to the Journal, identify potential 46 Journal contributors, and invite submissions as appropriate. 47 Applicants must have an M.D. degree and at least 5 years of research experience. 48 The successful candidate will be actively involved in patient care and will continue 49 that involvement (approximately 20% of time) while serving in this role. 50 To apply, visit https://mmscareers.silkroad.com. 51 52 53 54 55 56 57 1064 n engl j med 374;11 nejm.org March 17, 2016 58 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml 59 60 BMJ Open Page 40 of 55

Grant et al. Trials 2012, 13:154 BMJ Open: first published as 10.1136/bmjopen-2016-015281 on 10 March 2017. Downloaded from 1 http://www.trialsjournal.com/content/13/1/154 TRIALS 2 3 4 5 STUDYPROTOCOL Open Access 6 7 8 9 Study protocol of a mixed-methods evaluation of 10 11 a cluster randomized trial to improve the safety 12 13 of NSAID and antiplatelet prescribing: data-driven 14 15 quality improvementFor peer inreview primary only care 16 17 Aileen Grant1*, Tobias Dreischulte2, Shaun Treweek1 and Bruce Guthrie1 18 19 20 Abstract 21 22 Background: Trials of complex interventions are criticized for being ‘black box’, so the UK Medical Research Council 23 recommends carrying out a process evaluation to explain the trial findings. We believe it is good practice to pre-specify 24 and publish process evaluation protocols to set standards and minimize bias. Unlike protocols for trials, little guidance 25 or standards exist for the reporting of process evaluations. This paper presents the mixed-method process evaluation 26 protocol of a cluster randomized trial, drawing on a framework designed by the authors. 27 Methods/design: This mixed-method evaluation is based on four research questions and maps data collection to a 28 logic model of how the data-driven quality improvement in primary care (DQIP) intervention is expected to work. Data 29 collection will be predominately by qualitative case studies in eight to ten of the trial practices, focus groups with 30 patients affected by the intervention and quantitative analysis of routine practice data, trial outcome and questionnaire 31 data and data from the DQIP intervention. 32 33 Discussion: We believe that pre-specifying the intentions of a process evaluation can help to minimize bias arising from potentially misleading post-hoc analysis. We recognize it is also important to retain flexibility to examine the

34 http://bmjopen.bmj.com/ 35 unexpected and the unintended. From that perspective, a mixed-methods evaluation allows the combination of 36 exploratory and flexible qualitative work, and more pre-specified quantitative analysis, with each method contributing 37 to the design, implementation and interpretation of the other. 38 As well as strengthening the study the authors hope to stimulate discussion among their academic colleagues about 39 publishing protocols for evaluations of randomized trials of complex interventions. 40 Data-driven quality improvement in primary care trial registration: ClinicalTrials.gov: NCT01425502 41 42 Keywords: Complex intervention, Process evaluation, Protocol, Mixed methods, Randomized controlled trial 43 on October 3, 2021 by guest. Protected copyright. 44 Background how context influences outcomes, and to provide insights 45 Trials of complex interventions are often criticized as to aid implementation [1].’ However, when designing our 46 being a black box because it is difficult to know exactly own process evaluation of a cluster randomized trial of a 47 why an intervention did (or did not) work. The UK complex intervention, [2] we found little specific guidance. 48 Medical Research Council’s framework for designing and In response, we have proposed a framework to guide 49 evaluating complex interventions recommends conduct- process evaluation design [3]. 50 ing a process evaluation in order to ‘explain discrepancies Pre-specifying and publishing protocols are recognized 51 52 between expected and observed outcomes, to understand as improving the standards of clinical trials, by allowing 53 comparison of what was intended with what was actually done. Although process evaluations are somewhat differ- 54 * Correspondence: [email protected] 55 1Quality, Safety and Informatics Research Group, Population Health Sciences, ent (since flexibility to evaluate unexpected findings quali- University of Dundee, Mackenzie Building, Dundee DD2 4BF, UK tatively will often be useful), we believe that publication of 56 Full list of author information is available at the end of the article 57 58 © 2012 Grant et al.; licensee BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and 59 reproduction in any medium, provided the original work is properly cited. 60 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml Page 41 of 55 BMJ Open

Grant et al. Trials 2012, 13:154 Page 2 of 6 BMJ Open: first published as 10.1136/bmjopen-2016-015281 on 10 March 2017. Downloaded from 1 http://www.trialsjournal.com/content/13/1/154 2 3 4 5 process evaluation protocols is also best practice, and that DQIP process evaluation aim and research questions 6 pre-specifying the intentions of process evaluations can Aim 7 help to minimize bias arising from potentially misleading The aim of this study is to understand DQIP delivery, 8 post-hoc analysis. Unlike trial protocols, there is little implementation and translation into reducing high risk 9 explicit guidance or standards for the reporting of prescribing. 10 process evaluation protocols. We hope publishing this Research questions: 11 protocol will help stimulate discussion among the aca- 12 demic community. 1. How do different practices adopt the intervention and 13 This paper briefly describes the trial being evaluated, how does this influence their delivery to targeted 14 presents a logic model detailing the trial hypothesized patients and whether they continue to use the 15 pathway of change,For and states thepeer research questions reviewintervention? only Are there any positive or negative 16 that arise from our assumed model of trial processes, unintended consequences from a practice perspective? 17 overall study design, and detailed methods for answering 2. How do patients respond to the intervention? Are 18 each question. there any positive or negative unintended 19 consequences from a patient perspective? 20 Data-driven quality improvement in primary care trial 3. How are practice organizational characteristics 21 High-risk prescribing of non-steroidal anti-inflammatory associated with variation in recruitment, adoption, 22 drugs (NSAIDs) and antiplatelet agents account for a reach, delivery to the patient, and maintenance? 23 significant proportion of hospital admissions, due to pre- 4. Does effectiveness vary between practices, and if so, 24 ventable adverse drug events [4,5]. We are conducting a how are practice characteristics and processes 25 randomized controlled trial of a complex intervention to including structure, adoption, reach, delivery to the 26 improve prescribing safety of these drugs in 40 general patient, and maintenance associated with effectiveness? 27 practices in two Scottish Health Boards (the DQIP trial). 28 The trial is fully described in the published protocol [2]. Management and governance 29 In brief, the DQIP intervention comprises three compo- Research ethical approval for this study was granted 30 nents. The first component is a web-based informatics by Fife and Forth Valley Research Ethics Committee 31 tool that provides weekly updated feedback of targeted (11/AL/0251). An external Trial Steering Committee and 32 prescribing at practice level. This prompts the review of an overall Program Advisory Group, consisting of inde- 33 individual patients affected and summarizes each patient’s pendent professional and lay people, have been established

34 http://bmjopen.bmj.com/ relevant risk factors and prescribing. The second compo- and both meet bi-annually. 35 nent is an educational outreach visit that highlights the 36 risks of the targeted prescribing and provides training in Methods/design 37 38 the use of the tool. The final component is a fixed pay- Overall study design 39 ment of 350 GBP (560 USD; 403 EUR) to each practice up The study is a mixed-methods parallel process evalu- 40 front and a fee-for-service payment of 15 GBP (24 USD; ation. It will include a qualitative dominant case-study 41 17 EUR) for each patient reviewed. approach in a purposeful sample of practices and a 42 The trial has a stepped wedge design, [6,7] where all hypothesis-testing quantitative analysis of data from all 43 participating practices receive the DQIP intervention. practices. Data collection and analysis will be conducted on October 3, 2021 by guest. Protected copyright. 44 These practices are randomized to one of ten different in parallel to the DQIP trial itself. 45 start dates, with each practice functioning as its own 46 control in a time series analysis. The primary outcome Methods for research question 1 47 measure is a composite of nine measures of high-risk Study design 48 NSAID and antiplatelet prescribing. A separate eco- Study will be a primarily qualitative, mixed-methods in- 49 nomic evaluation is planned. depth case-study in a sample of eight to ten practices. 50 51 Structure/framework of DQIP parallel process evaluation Sampling 52 protocol Sampling for this study will be purposive and ensure 53 Drawing on our proposed framework for designing process heterogeneity in list size, reflecting our prior expectation 54 evaluations of cluster randomized trials, [3] Figure 1 shows that effectiveness will be greater in small practices; [8] 55 the processes we assume have to occur for the intervention and initial use of the tool (since this is how patients for 56 to be effective, and therefore explicitly identifies targets for review are identified and so a key process if our logic 57 evaluation. This logic model underlies the process evalu- model is true); and the date the practice started the trial 58 ation design and data collection, although resource con- (which is randomized, but workload in general practice is 59 straints inevitably require prioritization. strongly seasonal, so resource to deliver the intervention 60 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml BMJ Open Page 42 of 55

Grant et al. Trials 2012, 13:154 Page 3 of 6 BMJ Open: first published as 10.1136/bmjopen-2016-015281 on 10 March 2017. Downloaded from 1 http://www.trialsjournal.com/content/13/1/154 2 3 4 5 6 7 8 9 10 11 12 13 14 15 For peer review only 16 17 18 19 20 21 22 23 24 25 26 27 28 29 Figure 1 Data-driven quality improvement in primary care (DQIP) trial hypothesized pathway of change and process evaluation model. 30 31 mayvarybystartdate).Wewillsampletwopractices the practice pharmacist will also be interviewed. Partici- 32 from the first two cohorts of starting date and develop pants will be asked about their experience of the interven- 33 theories that can then be tested in the subsequent case tion, the practice process for review, how the intervention

34 http://bmjopen.bmj.com/ studies [9]. fitted with their existing work, barriers and facilitators, 35 maintenance over time, and any perceived unintended 36 Qualitative data collection consequences. Where possible, the GP most involved in 37 38 Data collection will use a variety of methods, in order to DQIP will be re-interviewed three to six months later, to 39 understand how intervention processes were perceived explore changes over time and the factors involved in 40 and implemented in each practice, and how these pro- maintaining and sustaining use of the tool. Interviews will ’ 41 cesses influence observed trial outcomes [10]. As part of be conducted at a time and place of the participants 42 the intervention in all practices, the qualitative re- choosing, facilitated by a topic guide drawing on the logic 43 searcher will assist the research pharmacist in delivering model and Normalization Process Theory (NPT), [12] and on October 3, 2021 by guest. Protected copyright. ’ 44 the education and training to the trial practices, and with the participants permission, audio-recorded for tran- 45 make field notes detailing attendance and the practice’s scribing. Normalization Process Theory is a sociological 46 response. Any additional communication or practice vis- theory designed to understand how practices or interven- 47 its will be recorded in the same way. These data will in- tions are implemented and become embedded into every- 48 form the focus of the interviews in each practice, and day work (normalized and routinized). NPT is made up of 49 additionally be used to help construct a detailed, ‘rich’ four domains, coherence, cognitive participation, collect- 50 description of each practice for cross-case analysis [11]. ive action, reflective monitoring, with each of these 51 General practitioners (GPs), practice pharmacists and domains divided into four categories [12]. 52 practice managers, from the case-study practices, involved 53 in delivering the DQIP intervention to patients will be Qualitative data analysis 54 invited to interview approximately six months after the The professional qualitative data will be inductively and 55 practice starts the trial. Depending on practice size and deductively analyzed in separate NVivo 8 (QSR Inter- 56 how each practice has organized DQIP work, we will national, Southport, UK) databases. Deductive analysis 57 interview one to three professionals per practice. At a will be by examining the data through the lens of NPT 58 minimum, the GP most involved in DQIP will be inter- [12]. Our pilot work has illustrated that this means of 59 viewed, but other GPs involved, the practice manager and analysis sometimes ‘divides’ emergent codes in the data. 60 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml Page 43 of 55 BMJ Open

Grant et al. Trials 2012, 13:154 Page 4 of 6 BMJ Open: first published as 10.1136/bmjopen-2016-015281 on 10 March 2017. Downloaded from 1 http://www.trialsjournal.com/content/13/1/154 2 3 4 5 As a result, the data will be analyzed inductively again, asked about their experience of medication reviews and 6 to allow themes to emerge from the data and to explore the extent to which they were aware of the intervention 7 issues that the NPT deductive analysis may be less suited (response). Then after watching a short presentation 8 to identify and examine. about the way clinical data is being used in this study, 9 patients will be asked about their experience, percep- 10 Case-study analysis tions and any consequences of the intervention triggered 11 For each practice (case), we will have a set of quantitative by review or medication change and their attitudes and 12 and qualitative data which can be synthesized. Quantita- beliefs about the way in which clinical data is being 13 tive data, as described in detail below, will be used to de- used. 14 scriptively characterize each practice in terms of practice 15 structural characteristics,Forand measures peer of adoption, deliv-reviewData analysis only 16 ery to patient, reach, and maintenance. Qualitative data Patient focus group data will be analyzed inductively to 17 includes field notes relating to research team contacts allow themes to emerge from the data. The framework 18 with each practice and interview data. Synthesis of the technique [17] will be used to facilitate analysis. 19 qualitative and quantitative data for the case studies will 20 take two forms; first a comprehensive, ‘rich’ description of Methods for research questions 3 and 4 21 each case-study practice will be created based on all data, Study design 22 then this description will be used to conduct a cross-case The study requires quantitative analysis of routinely 23 comparison based on the logic model [13]. The aim of this available data about practices, trial process data gener- 24 analysis is to examine in detail how practices implement ated by DQIP informatics tool, practice questionnaire 25 (or not) the processes that our logic model assumes are data, and trial outcome data. We hypothesize that practice 26 required to deliver change in high-risk prescribing, and organizational characteristics and adoption will be asso- 27 how these processes appear to influence effectiveness in ciated with the level of reach, delivery to the patient and 28 each practice. The aim of this sequential merging of quali- maintenance achieved, and that lower levels of reach, 29 tative and quantitative data is not triangulation but delivery to the patient and maintenance will be associated 30 crystallization where we are using the qualitative data to with lower effectiveness. The specific hypotheses to be 31 explain the quantitative [14,15]. This offers the advantage tested will be based on findings from the case studies. 32 of allowing the qualitative data to explore complex, unex- 33 pected or contradictory quantitative data collected from Sampling

34 http://bmjopen.bmj.com/ the tool and adoption questionnaires. A constant compari- Data will be collected for all practices. 35 son and deviant case approach will be adopted [16]. Con- 36 stant comparison will allow the researcher to test the Data collection 37 38 emerging hypothesis by comparing different case-study Multiple datasets will be collected and linked. Practice 39 practices. Any deviant examples from the emergent hy- characteristics will be measured using routinely available 40 pothesis will be examined. Analysis will initially be simul- data including list size, rurality , deprivation, proportion 41 taneous with data collection to allow the emerging of older patients, postgraduate training status, dispens- 42 analysis to be tested and revised in subsequent data sam- ing status, contract type (nGMS, section 17c/2c), Health 43 pling, collection and analysis. The framework [17] analysis Board, Community Health Partnership, overall QOF on October 3, 2021 by guest. Protected copyright. 44 technique will be used to support within-case and cross- clinical performance, and QOF performance on relevant 45 case analysis. medicines management indicators (Medicines 6, 10, 11, 46 12){National Institute of Clinical Excellence, 2012}. Base- 47 Methods for research question 2 line levels of the high risk NSAID and antiplatelet 48 Study design prescribing being targeted will also be available for par- 49 The design is that of a focus group study involving a ticipating practices. During the recruitment of clusters 50 sample of patients targeted by the intervention. we will document the trial recruitment process and care- 51 fully record those practices that are approached, those 52 Sampling that respond and those that are recruited. Participating 53 Four to six of the case-study practices will invite patients and non-participating practices will be compared using 54 who have been targeted by the intervention to attend a available practice characteristic data. Adoption will be 55 focus group. measured using a quantified assessment of engagement 56 with the education outreach based on which practice 57 Data collection members attended and field notes of the interaction, and 58 One focus group per practice will be held. The focus by a survey instrument developed for the trial and com- 59 group will have two stages. First, the patients will be pleted at trial start and after six to nine months based 60 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml BMJ Open Page 44 of 55

Grant et al. Trials 2012, 13:154 Page 5 of 6 BMJ Open: first published as 10.1136/bmjopen-2016-015281 on 10 March 2017. Downloaded from 1 http://www.trialsjournal.com/content/13/1/154 2 3 4 5 on NPT. [12] The adoption questionnaire is based on study, and will provide the flexibility either to 6 the four domains of NPT, with the baseline survey fo- identify key processes on the assumed pathway 7 cusing on coherence and cognitive participation, and the quantitatively or to identify unexpected processes 8 follow-up survey collecting additional data on collective that are not in the logic model, or both. 9 action and reflective monitoring. Reach will be measured 3. The quantitative study will test hypotheses generated 10 as the proportion of patients identified by the informat- from the qualitative analysis, and help inform 11 ics tool who are actually reviewed. Delivery to the pa- judgments about generalization of the case and 12 tient will be measured in terms of patterns of review focus group studies. 13 recorded in the tool (which measures were focused on, 4. Overall synthesis will draw on findings from all three 14 how they were carried out [records review, face-to-face studies, which will be used to construct a narrative 15 consultation, telephoneFor consultation]), peer the decisions reviewsynthesis ofonly the process of implementation of the 16 made (the proportion of patients who decide to con- intervention, to contextualize the main trial findings 17 tinue, the proportion who decide to continue with gas- (whether positive or negative) and to inform 18 tric protection and the proportion who decide to stop judgments about external validity and generalizability 19 the drug). Maintenance will be measured in terms of , real-world implementation if appropriate, and 20 how reach changes over time. Effectiveness will be mea- further intervention development [18]. 21 sured in terms of the trial primary outcome (the high 22 risk NSAID/antiplatelet composite) and the two second- Discussion 23 ary outcomes of ‘repeated’ and ‘new’ prescribing (since We have piloted these data collection methods in four 24 the intervention targets the review of repeated prescrib- general practices and in six focus groups with patients 25 ing, with an expectation that the experience of reviewing from these practices and with members of the public. 26 will reduce new prescribing as well, but that this may These methods have been acceptable to practitioners 27 vary by practice and particularly by practice size). and patients and have generated sufficient data to feas- 28 ibly evaluate this complex intervention trial. The pilot 29 Data analysis has influenced the focus and design of this evaluation 30 Initial descriptive analysis will use practice characteristics and the associated topic guides and questionnaires. 31 data to compare participating practices with the wider As in any study, choices have been made in the design 32 population of practices in the two Boards and nationally of this process evaluation protocol that balance ideal re- 33 (to assess the representativeness of recruited practices and search questions with feasibility and resource constraints.

34 http://bmjopen.bmj.com/ the implications for generalizability). The overall extent of [19] We believe that a prior design of the process evalu- 35 reach, delivery to the patient and maintenance will then ation that maps to expectations about how the interven- 36 be examined, and univariate and multivariate associations tion will work strengthens the study. It is also important, 37 38 with practice characteristics and adoption will be exam- however, to retain the flexibility needed for examining the 39 ined using cross-tabulations, comparison of means, and unexpected and the unintended. From that perspective, a 40 logistic/linear regression as appropriate to the data. The mixed-methods evaluation allows the combination of 41 extent of variation between practices in the three specified exploratory and flexible qualitative work, and more pre- 42 measures of effectiveness will be examined using multilevel specified quantitative analysis, with each method contrib- 43 logistic regression, and associations between effectiveness uting to the design, implementation and interpretation of on October 3, 2021 by guest. Protected copyright. 44 and practice characteristics, adoption, reach, delivery to the other [20]. Design is always constrained by the 45 the patient,andmaintenance will be examined. resources available though, and although we have been 46 fortunate in being relatively well-resourced for this elem- 47 Synthesizing results from all three studies ent of the project, our chosen case-study method does 48 Synthesis across the three studies will have several mean having to sample a relatively small number of prac- 49 elements: tices to carry out the qualitative work. Inevitably though, 50 extending the interviewing to a larger number of practices 51 1. The quantitative study will inform case-study would involve less depth in each practice, and we chose to 52 sampling and will help contextualize sampled prioritize more depth in a case-study approach using mul- 53 settings. tiple interviews in selected practices. 54 2. The case and patient focus group studies will provide We have also chosen to conduct a parallel process 55 a rich understanding of how the intervention was evaluation, where data collection is simultaneous with trial 56 perceived and implemented, and how actual implementation [21]. A key advantage is that data collec- 57 implementation related to our model of how the tion is contemporaneous with the practices actually doing 58 intervention was intended to work. These data will the work of implementation, but a potential disadvantage 59 generate hypotheses for testing in the quantitative is the risk that the case-study practices vary little in their 60 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml Page 45 of 55 BMJ Open

Grant et al. Trials 2012, 13:154 Page 6 of 6 BMJ Open: first published as 10.1136/bmjopen-2016-015281 on 10 March 2017. Downloaded from 1 http://www.trialsjournal.com/content/13/1/154 2 3 4 5 overall effectiveness in delivering main trial outcomes. A Received: 4 May 2012 Accepted: 26 July 2012 Published: 28 August 2012 6 post-hoc design that samples based on main trial out- 7 comes would avoid this risk, but at the cost of being References 8 designed after trial effectiveness is known, and conducted 1. Medical Research Council: Developing and evaluating complex interventions; long after the trial completes. Overall, we judged a parallel new guidance. London: Medical Research Council; 2008. 9 2. Dreischulte T, Grant A, Donnan P, McCowan C, Davey P, Petrie D, Treweek S, 10 process evaluation to be more likely to produce robust Guthrie B: A cluster randomised stepped wedge trial to evaluate the 11 findings, and although the protocol is pre-specified, the effectiveness of a multifaceted information technology-based methods proposed are flexible to unexpected findings in intervention in reducing high-risk prescribing of non-steroidal anti- 12 inflammatory drugs and antiplatelets in primary care: the DQIP study 13 that qualitative data collection and analysis is iterative in protocol. Implement Sci 2012, 7:24. 14 nature, and will inform the choice of the actual hypotheses 3. Grant A, Treweek S, Dreischulte T, Foy R, Guthrie B: Process evaluations for 15 to be tested quantitatively.For A final peer potential weakness review is cluster-randomised only trials of complex interventions: a proposed framework for design and reporting. Trials, . in press. 16 that some of the quantitative data is relatively thin. For ex- 4. Pirmohamed MJ, Meakin S, Green C, Scott AK, Walley T, Farrar K, Park BK, 17 ample, the data on the nature of review carried out is col- Breckenridge AM: Adverse drug reactions as a cause of admission to 18 lected routinely in the DQIP informatics tool but lacks hospital: prospective analysis of 18 820 patients. BMJ 2004, 329:15–19. 5. Howard RL, Avery AJ, Slavenburg S, Royal S, Pipe G, Lucassen P, 19 contextual information or much detail. This reflects a Pirmohamed M: Which drugs cause preventable admissions to hospital? 20 trade-off between being able to collect data in all practices A systematic review. Br J Clin Pharmacol 2007, 63:136–147. 21 and avoiding burdening practices with data collection that 6. Brown C, Lilford R: The stepped wedge trial design: a systematic review. 22 BMC Med Res Methodol 2006, 6:54. has no clinical purpose. 7. Hussey MA, Hughes JP: Design and analysis of stepped wedge cluster 23 Our study offers the opportunity to explain why or randomized trials. Contemp Clin Trials 2007, 28:182–191. 24 why not the DQIP intervention reduced high risk pre- 8. Nazareth I, Freemantle N, Duggan C, Mason J, Haines A: Evaluation of a 25 complex intervention for changing professional behaviour: The evidence scribing. If the intervention does not reduce high-risk based out reach (EBOR) trial. J Health Serv Res Policy 2002, 7:230–238. 26 prescribing as intended, then the analysis will help iden- 9. Mason J: Sampling and selection in qualitative research. In Qualitative 27 tify why this happened in terms of where the assump- Researching. London: Sage Publications Ltd; 2002:120–144. 10. Lewin S, Glenton C, Oxman AD: Use of qualitative methods alongside 28 tions of the logic model broke down, or call into 29 randomised controlled trials of complex healthcare interventions: question the underlying model. If the intervention does methodological study. BMJ 2009, 339:b3496. 30 reduce high-risk prescribing, then the process evaluation 11. Hoddinott P, Britten J, Pill R: Why do interventions work in some places and not others: a breastfeeding support group trial. Soc Sci Med 2010, 31 will provide information to inform judgments about 32 70:769–778. likely generalizability, and identify if improvement was 12. May C, Finch T: Implementing, Embedding, and Integrating Practices: An 33 general or restricted to some practices which may influ- Outline of Normalisation Process Theory. Sociology 2009, 43:535. 13. Stake R: The art of case study research. Thousand Oaks, California: Sage

34 http://bmjopen.bmj.com/ ence real-world implementation or identify targets for a 35 Publications Ltd; 1995. modified intervention. 14. Denzin NK: Moments, mixed methods, and paradigm dialogs. Qualtiative 36 Inquiry 2010, 16:416. 37 15. Cresswell JW, Plano CVL: Designing and conducting mixed methods research. Trial Status 38 Thousand Oaks: Sage Publications Ltd; 2007. Active. Trial started 31st October 2011. 16. Silverman D: Interpreting qualitative data. 3rd edition. London: Sage 39 Publications Ltd; 2006. 40 Abbreviations 17. Ritchie J, Spencer L, O’Connor W: Carrying out Qualitative Analysis.In 41 DQIP: data-driven quality improvement in primary care; GP: general Qualitative Research Practice: A guide for Social Science Students and Researchers. 42 practitioner; NPT: Normalization Process Theory; NSAIDS: non-steroidal anti- Edited by Ritchie J, Lewis J. London: Sage Publications Ltd; 2003. inflammatory drugs; QOF: Quality and Outcomes Framework. 18. Dixon-Woods M, Agarwal S, Jones D, Young B, Sutton A: Synthesising on October 3, 2021 by guest. Protected copyright. 43 qualitative and quantitative evidence: a review of possible methods. – 44 Competing interests J Health Serv Res Policy 2005, 10:45 53. 45 The author(s) declare that they have no competing interests. 19. Francis J, Eccles M, Johnston M, Whitty P, Grimshaw J, Kaner E, Smith L, Walker A: Explaining the effects of an intervention designed to promote 46 evidence-based diabetes care: a theory-based process evaluation of a Authors’ contributions 47 pragmatic cluster randomised controlled trial. Implement Sci 2008, 3:50. AG and BG were involved in the initial conceptualization and study design. 20. Protheroe J, Bower P, Chew-Graham C, Protheroe J, Bower P, Chew-Graham 48 AG and BG developed the data collection methods; AG piloted these with C: The use of mixed methodology in evaluating complex interventions: 49 the help of TD. AG and BG wrote the paper. All authors read and approved identifying patient factors that moderate the effects of a decision aid. the final manuscript. AG is responsible for this manuscript. 50 Fam Pract 2007, 24:594–600. 51 21. Craig P, Dieppe P, Macintyre S, Michie S, Nazareth I, Petticrew M: Acknowledgements 52 Developing and evaluating complex interventions: the new Medical AG and TD are funded by the Scottish Government Health Directorates Chief Research Council guidance. BMJ 2008, 337:a1655. 53 Scientist Office Applied Programme Research Grant 07/02 for the DQIP prescribing improvement study. This funding body has had no role in the 54 doi:10.1186/1745-6215-13-154 design and conduct of, or decision to publish this work. 55 Cite this article as: Grant et al.: Study protocol of a mixed-methods evaluation of a cluster randomized trial to improve the safety of NSAID 56 Author details 1 and antiplatelet prescribing: data-driven quality improvement in 57 Quality, Safety and Informatics Research Group, Population Health Sciences, primary care. Trials 2012 13:154. 2 58 University of Dundee, Mackenzie Building, Dundee DD2 4BF, UK. Medicines Management Unit, NHS Tayside, c/o University of Dundee, Mackenzie 59 Building, Dundee DD2 4BF, UK. 60 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml BMJ Open Page 46 of 55

Grant et al. Trials 2013, 14:15 BMJ Open: first published as 10.1136/bmjopen-2016-015281 on 10 March 2017. Downloaded from 1 http://www.trialsjournal.com/content/14/1/15 TRIALS 2 3 4 5 METHODOLOGY Open Access 6 7 8 9 Process evaluations for cluster-randomised trials 10 11 of complex interventions: a proposed framework 12 13 for design and reporting 14 15 Aileen Grant1*, ShaunFor Treweek1, Tobiaspeer Dreischulte 2review, Robbie Foy3 and Bruce only Guthrie1 16 17 18 Abstract 19 20 Background: Process evaluations are recommended to open the ‘black box’ of complex interventions evaluated in 21 trials, but there is limited guidance to help researchers design process evaluations. Much current literature on 22 process evaluations of complex interventions focuses on qualitative methods, with less attention paid to 23 quantitative methods. This discrepancy led us to develop our own framework for designing process evaluations of 24 cluster-randomised controlled trials. 25 Methods: We reviewed recent theoretical and methodological literature and selected published process 26 evaluations; these publications identified a need for structure to help design process evaluations. We drew upon 27 this literature to develop a framework through iterative exchanges, and tested this against published evaluations. 28 Results: The developed framework presents a range of candidate approaches to understanding trial delivery, 29 intervention implementation and the responses of targeted participants. We believe this framework will be useful to 30 31 others designing process evaluations of complex intervention trials. We also propose key information that process 32 evaluations could report to facilitate their identification and enhance their usefulness. 33 Conclusion: There is no single best way to design and carry out a process evaluation. Researchers will be faced with choices about what questions to focus on and which methods to use. The most appropriate design depends

34 http://bmjopen.bmj.com/ 35 on the purpose of the process evaluation; the framework aims to help researchers make explicit their choices of 36 research questions and methods. 37 Trial registration: Clinicaltrials.gov NCT01425502 38 39 Keywords: Process evaluation, Complex intervention, Cluster-randomised controlled trial, Qualitative, Quantitative, 40 Reporting 41 42

Background ences in context, targeted groups and the intervention ac- on October 3, 2021 by guest. Protected copyright. 43 Many interventions to improve healthcare delivery and tually delivered [2,3]. The UK Medical Research Council 44 health are complex in the sense that they possess several guidance for developing and evaluating complex interven- 45 interacting components [1]. Randomised controlled trials tions recommends conducting a process evaluation to 46 of such interventions are often criticised as being ‘black ‘explain discrepancies between expected and observed 47 box,’ since it can be difficult to know why the intervention outcomes, to understand how context influences out- 48 ’ 49 worked (or not) without examining underlying processes. comes, and to provide insights to aid implementation [1]. 50 As a result, reported effects of what appear to be similar Although our focus is on randomised trials, it is important 51 interventions often vary across studies because of differ to recognise that similar issues arise in the evaluation of 52 public health interventions which may use a range of dif- ferent evaluative designs [4-6]. 53 * Correspondence: [email protected] 54 1Quality, Safety and Informatics Research Group, Population Health Sciences, In complex intervention trials, the intervention to be 55 Medical Research Institute, University of Dundee, Mackenzie Building, evaluated may be delivered at different levels: the inter- Dundee DD2 4BF, UK vention can be delivered straight to the patient, such as 56 Full list of author information is available at the end of the article 57 58 © 2013 Grant et al.; licensee BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and 59 reproduction in any medium, provided the original work is properly cited. 60 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml Page 47 of 55 BMJ Open

Grant et al. Trials 2013, 14:15 Page 2 of 10 BMJ Open: first published as 10.1136/bmjopen-2016-015281 on 10 March 2017. Downloaded from 1 http://www.trialsjournal.com/content/14/1/15 2 3 4 5 in trials evaluating the effectiveness of body weight and evaluations to enhance their identification and usefulness 6 physical activity interventions on adults [7]; the interven- to other researchers. 7 tion can be delivered to the healthcare professional, for 8 example in trials of decision support tools [8]; or the inter- Methods 9 vention can be targeted at the organisational level, such Existing literature 10 as in a recent trial testing a pharmacist-led information- To support the design of our own process evaluation, we 11 technology-based intervention aimed at optimising medi- wanted to identify relevant methodological and theore- 12 cation safety systems in general practice [9] (adapted from tical literature. We did not aim for a comprehensive 13 the Medical Research Council guidance 2000) [10]. review of all such studies but rather sought to gain a 14 Process evaluations are studies that run parallel to or general understanding of this literature. We were par- 15 follow intervention trialsFor to understand peer the trial review pro- ticularly interested only in the purpose and the design of 16 cesses or underlying mechanisms in relation to context, process evaluations of cluster-randomised controlled 17 setting [11], professionals and patients [12]. These eva- trials. Our initial search (which included the terms 18 luations provide explanations for the trial results and ‘process evaluation’ or ‘qualitative’ or ‘quantitative’ and 19 enhance understanding on whether or how interventions ‘complex intervention’ or ‘randomised controlled trial’) 20 could move from research to practice. in Medline, Embase, Psycinfo, Cochrane Central Register 21 We planned to conduct a parallel process evaluation of Control Trials and Cochrane Database of Systematic 22 of a cluster-randomised trial of a complex intervention Reviews was informative but identified studies that 23 to improve prescribing safety [13] targeted at general tended to focus on a specific aspect of process evalua- 24 practices. Although there is clear published guidance for tion design or the nature of process evaluations and 25 designing trials where clusters rather than individuals qualitative methods. There was little guidance on how to 26 are the unit of randomisation [14,15], we found the design a process evaluation. We turned to process evalua- 27 Medical Research Council guidance of limited help in tions from cluster-randomised controlled trials identified 28 designing the process evaluation of our trial. by our search from which we could learn. We designed a 29 In our trial, the research team delivers an intervention data extraction form based on the Reach, Efficacy, Adop- 30 to general practices – with the intention of prompting tion, Implementation and Maintenance (RE-AIM) frame- 31 those practices to review patients with particular types work [19] and Northstar [20] to summarise identified 32 of high-risk prescribing (nonsteroidal anti-inflammatory reports of process evaluations in terms of: the aim of the 33 drugs or antiplatelets), to carefully reconsider the risks work; the timing (parallel to the trial or post hoc); its scope

34 http://bmjopen.bmj.com/ and benefits of the offending drugs in each patient and with reference to each RE-AIM construct (reach, efficacy, 35 to take corrective action where possible [13]. There is adoption, implementation and maintenance) and the 36 therefore an intervention delivered to practices (the methods used; and trial and process evaluation findings 37 38 cluster) that hopes to change the behaviour of practi- and how they were reported. This form was systematically 39 tioners. This approach is similar to most trials of audit populated with information from all identified process 40 and feedback [16] and to many trials involving a profes- evaluations by AG, with a second review of each by either 41 sional training intervention, as in the OPERA trial where ST, RF or BG. Disagreements between reviewers were col- 42 a training intervention is delivered to nursing home staff lectively discussed and resolved, and general strengths and 43 and physiotherapists who deliver an intervention of limitations in design and reporting were identified. on October 3, 2021 by guest. Protected copyright. 44 physical activity and physiotherapy to residents of resi- We initially set out to design a critical appraisal form 45 dential and nursing homes [17,18]. In other trials, how- but there was wide variation in the reporting of process 46 ever, the intervention is delivered to individuals by the evaluations that made consistent critical appraisal diffi- 47 research team themselves, as in a trial of body weight cult. We struggled to make judgements based on the 48 and physical activity for adults at risk of colorectal aden- breadth of information reported in our data extraction 49 omas [7]. Process evaluations therefore need to be tai- form because we found studies that chose to report one 50 lored to the trial, the intervention and the outcomes process in detail but did say if other processes had been 51 being studied, but we could not find clear guidance in the considered and an explicit judgement had been made 52 literature on how to do this. Additionally, we noted wide not to evaluate them [21]. Some general features of this 53 variations in the aims, methods, timing and reporting of literature are summarised below. 54 process evaluation studies that made it difficult to draw 55 on them in designing our own process evaluation. Results and discussion 56 We therefore developed a framework to guide the Process evaluation purpose 57 design of our own process evaluation. We believe it may The generally agreed purpose of process evaluations of 58 be of help to other cluster triallists faced with a similar trials of complex and public health interventions is to 59 task. We also make proposals for the reporting of process understand the effects (or not) of interventions [2-4]. 60 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml BMJ Open Page 48 of 55

Grant et al. Trials 2013, 14:15 Page 3 of 10 BMJ Open: first published as 10.1136/bmjopen-2016-015281 on 10 March 2017. Downloaded from 1 http://www.trialsjournal.com/content/14/1/15 2 3 4 5 These effects can help to inform judgements about the va- Framework for designing process evaluations of cluster- 6 lidity by delineating key intervention components (con- randomised trials of complex interventions 7 struct validity), and by demonstrating connections between In developing our framework we drew upon the general li- 8 the intervention and outcomes (internal validity) and be- terature on process evaluation and the RE-AIM framework 9 tween the intervention and other contexts (external vali- [3,11,19,23,26,27,31], and on published process evaluations 10 dity) [4,22]. Implementation and change processes can be [11,12,21,29,32,33]. The framework was developed by all 11 explored [1,23], and factors associated with variations in the authors through iterative meetings, redefining and 12 effectiveness can be examined [24]. Process evaluations can working with the concepts and testing this work against 13 also examine the utility of theories underpinning interven- published evaluations. 14 tion design [25] and generate questions or hypotheses for Cluster-randomised designs are utilised when individual 15 future research. ProcessFor evaluations peer can therefore fulfilreview a randomisation isonly not possible, feasible or appropriate, 16 variety of purposes, including understanding intervention mainly because of potential contamination between the 17 effects, potential generalisability and optimisation in routine intervention and control arms [34]. Following the most 18 practice [26]. recent CONSORT extension for cluster-randomised trials 19 [28], we distinguish between individuals in the target 20 Process evaluation design population on whom outcome data are collected and 21 Published papers on trial process evaluation design largely ‘clusters’, which are some larger social unit within which 22 focus on qualitative research methods [3,23,27], but many individuals are nested and where the cluster is the unit of 23 of the purposes identified above can be assessed both quan- randomisation. Clusters may be defined by geography, 24 titatively and qualitatively; indeed, process evaluation of institution, teams or services within institutions, or by 25 public health interventions often explicitly uses mixed individual professionals delivering a service to patients. 26 methods[4].Thefocusonqualitativemethodsintrialsmay Although outcomes are collected on an individual level, 27 occur because many routinely collect some quantitative the primary unit of inference of the intervention may be 28 process data measures to fulfil Consolidated Standards of the individual or the cluster, depending on the purpose of 29 Reporting Trials (CONSORT) requirements or as secon- the study and the primary outcome measures. The focus 30 dary outcomes, rather than as part of a distinct process of a process evaluation will vary with the nature of the 31 evaluation. For example, data on recruitment are likely to trial and resources available, so although several key pro- 32 be collected and reported as part of the main trial because cesses are candidates for examination in any evaluation of 33 reporting these data is a CONSORT requirement [28], but cluster trials (Figure 1), the importance of studying them

34 http://bmjopen.bmj.com/ for many trials a more detailed examination of the process will vary between trials. Also important to note is the fact 35 of recruitment may be useful, to inform interpretation of that although the emphasis in what follows is on processes 36 trial results and generalisability. in clusters that receive the intervention, it will usually be 37 38 equally important to understand relevant processes in 39 Process evaluation reporting control or usual care clusters. 40 We found it difficult to systematically identify process eva- 41 luations of complex intervention trials, and the reporting Recruitment of clusters (areas, institutions, individual 42 of important information was also variable. Although professionals) 43 some papers were easily identifiable and had clearly stated How clusters are sampled and recruited is important to on October 3, 2021 by guest. Protected copyright. 44 purposes and methods [21,29,30], others: help understand how generalisable the findings are likely 45 to be. At a minimum, recruitment is likely to be moni- 46  were hard to identify because of a lack of explicit tored to fulfil CONSORT requirements for reporting 47 labelling: findings of the main trial [28], but a more in-depth 48  lacked clarity about their overall purpose or did not understanding of why clusters participate (or not) can 49 state specific objectives: inform the interpretation and implementation of trial 50  did not clearly state whether the evaluation was pre- findings. 51 planned and carried out in parallel to the main trial 52 or was conducted post hoc: Delivery to clusters 53  did not provide an explicit statement of the main In some trials, the research team or specially trained 54 trial findings, making it difficult to judge the professionals may deliver a complex intervention to in- 55 relevance and implications of the process evaluation. stitutional or individual professional clusters, which are 56 then expected to change their practice or service deli- 57 Overall, although the existing literature was often very in some way that impacts on the individuals in the 58 thought-provoking, it did not provide an over-arching target population. Understanding the nature of cluster- 59 framework for designing process evaluations. level interventions and variations in their delivery may 60 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml Page 49 of 55 BMJ Open

Grant et al. Trials 2013, 14:15 Page 4 of 10 BMJ Open: first published as 10.1136/bmjopen-2016-015281 on 10 March 2017. Downloaded from 1 http://www.trialsjournal.com/content/14/1/15 2 3 4 5 6 7 8 9 10 11 12 13 14 15 For peer review only 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 Figure 1 Framework model for designing process evaluations of cluster-randomised controlled trials.

34 http://bmjopen.bmj.com/ 35 36 help explain variation in outcomes between organisa- proportion of the population may receive a public health 37 38 tional clusters. intervention [4]. As well as influencing effectiveness, 39 if high or representative recruitment and reach is not 40 Response of clusters achieved within clusters in the trial, then the population 41 Response is the process by which clusters such as institu- impact of implementation in routine practice is likely to 42 tions or individual professionals integrate and adopt new be limited. 43 trial-related or intervention-related work into their existing on October 3, 2021 by guest. Protected copyright. 44 systems and everyday work, and will be influenced by how Delivery to individuals 45 the research team engages and enrols the clusters and by In some studies, a complex intervention is delivered to 46 the extent to which the work is perceived to be legitimate the target population of individuals. Such interventions 47 within organisations or by individual professionals [35]. may be tightly defined by the researchers [36], in which 48 case evaluation of delivery to individuals in the target 49 Recruitment and reach of individuals population may focus on how closely the intervention 50 Interpretation of the intervention’s measured effective- delivered matches what the researchers intended (usually 51 ness in the trial will depend on recruitment and reach, called the fidelity of the intervention) and the impact of 52 in terms of the proportion of the target population that deviation on effectiveness. In some trials, such as those 53 actually receives the intervention, and how representa- on guideline implementation strategies, the intervention 54 tive they are. Variation in the response of clusters will is not directly delivered to individuals (patients) but in- 55 affect how clusters identify and enrol these individuals. stead aims to change professional behaviour. The nature 56 When recruitment of individuals is carried out by orga- of the changes to work patterns or professional beha- 57 nisational or individual professional clusters, there is viour at the individual or patient level can be examined 58 potential for selection bias [34] – although similar issues to better understand why it succeeded or failed. For ex- 59 arise in clusters defined by geography, where a variable ample, in our trial the Data-driven Quality Improvement 60 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml BMJ Open Page 50 of 55

Grant et al. Trials 2013, 14:15 Page 5 of 10 BMJ Open: first published as 10.1136/bmjopen-2016-015281 on 10 March 2017. Downloaded from 1 http://www.trialsjournal.com/content/14/1/15 2 3 4 5 in Primary Care (DQIP) complex intervention is targeted at Context 6 general practitioners to improve prescribing safety. This Irrespective of which specific processes are examined, a 7 intervention prompts practitioners to carry out a targeted careful description of the context in which the trial is 8 medication review based on a guideline provided to them, embedded will usually be helpful for interpreting the 9 but practitioners are free to review medication in any way findings, including how generalisable any findings are 10 they judge appropriate [13]. Understanding how practi- [39]. Additionally, trials are often conducted in different 11 tioners actually carry out these medication reviews is there- contexts – for example, within different health boards or 12 fore likely to be important to explaining the trial results. different countries – and so consideration of these local 13 contexts may be important. Each of the concepts in the 14 Response of individuals model has contextual factors that may act as barriers 15 The effectiveness of manyFor but not allpeer interventions willreview be and/or facilitators only to implementation. For example, the 16 mediated by the responses of the targeted individuals, par- DQIP trial is delivered in two health boards within 17 ticularly where effectiveness is strongly influenced by ad- Scotland that have different pre-existing levels of pre- 18 herence or requires behaviour change in those individuals. scribing improvement support for practices, and this 19 Similar to understanding the response of organisational or support has a different focus in each board. As well as 20 individual professionals to interventions directed at them, the wider context of the organisation of primary health- 21 it may be helpful to draw on prior [37] or other relevant care in the United Kingdom being important, these local 22 theory [38] to examine individual responses. contextual factors may modify how practices respond to 23 the same intervention [13,40]. 24 Maintenance 25 Understanding whether and how each of the above pro- Applying the framework to different types of trial 26 cesses is maintained or sustained over time is important, es- As described above, cluster-randomised trials vary 27 pecially for studies of longer duration. Organisational greatly in their design, including the level at which inter- 28 clusters or individuals may cease to participate completely ventions are targeted. The framework can therefore be 29 or the way in which they adopt and respond to the inter- seen as a set of candidate elements for designers of 30 vention may change over time, such as with declining reach process evaluations to consider. The relevance and rela- 31 or changes in the nature of the intervention delivered. tive importance of candidates will vary between trials, 32 the aim and objectives of the evaluation and whether the 33 Effectiveness authors plan to pre-specify their data collection and ana-

34 http://bmjopen.bmj.com/ Although pre-specified measurement of effectiveness is the lysis (potentially allowing hypothesis testing) or conduct 35 primary function of the main trial, the process evaluation the evaluation post hoc (for example, depending on the 36 can examine associations between trial processes and experience of conducting the trial or on the trial results, 37 – 38 effects on primary and secondary outcomes recognising in which case the evaluation can only be hypothesis 39 that these are exploratory analyses. This can help explain generating). Inevitably, resource limitations will often 40 why an intervention did or did not work, and can explore require a focus on elements that are considered most 41 variations in effectiveness between clusters that may be im- critical in the context being evaluated. In such cases, 42 portant in planning more widespread implementation. emphasis should be placed on collecting quality data for 43 a few key processes rather than collecting a lot of data on October 3, 2021 by guest. Protected copyright. 44 Unintended consequences for each candidate process. The following section dis- 45 All interventions have the potential to produce unintended cusses how the framework may be applied. 46 consequences, which may be beneficial or harmful. Process 47 evaluations provide an opportunity to systematically iden- Research methods for process evaluations of cluster- 48 tifyandquantifyunexpectedunintendedoutcomes. randomised trials 49 The appropriate methods for a process evaluation will 50 Theory depend on the intervention being evaluated and the aims 51 If theory has been used to develop the intervention, then of the evaluation. Table 1 presents examples of research 52 this should be made explicit, and the process evaluation questions that could be asked within each domain and 53 can examine whether predicted relationships and sequences methods to answer these questions. Both qualitative and 54 of changes happen during implementation. Causal model- quantitative methods can be appropriate depending on 55 ling based on psychological theory is an example that the questions asked. Quantitative data collection will be 56 would apply to ‘delivery to the individual’ and ‘response of required if it is judged that all clusters should be evalua- 57 individuals’ [37]. Even if interventions do not have a strong ted in large multisite trials [3], whereas sampling may be 58 theoretical base, then it may be useful to draw on relevant necessary for more in-depth qualitative evaluation. 59 theory to help understand their effects [35]. Mixed methods can add complementary insights. For 60 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml Page 51 of 55 BMJ Open

Grant et al. Trials 2013, 14:15 Page 6 of 10 BMJ Open: first published as 10.1136/bmjopen-2016-015281 on 10 March 2017. Downloaded from 1 http://www.trialsjournal.com/content/14/1/15 2 3 4 5 Table 1 Example research questions and methods for evaluating each process 6 Domain Example research questions Research methods that could be applied Probable best stage of study to 7 collect data 8 Recruitment of How are clusters sampled and recruited? Documentation of recruitment process by research team. Pre-intervention 9 clusters Who agrees to participate? Quantitative comparison of recruited and nonrecruited 10 clusters. 11 Why do clusters agree to participate (or not) Qualitative analysis of interviews with cluster gatekeepers or 12 members. 13 Delivery to What intervention is actually delivered for each Qualitative analysis of observational, interview and Pre-intervention 14 clusters cluster? Is it the one intended by the documentary data relating to the cluster-level intervention. and early 15 researchers?For peer review only intervention 16 Response of How is the intervention adopted by clusters? Quantitative data measuring cluster members’ perceptions Pre-intervention 17 clusters of the intervention and uptake of trial components. and early Qualitative analysis of observational, interview and intervention 18 documentary data about how clusters adopt the 19 intervention. 20 Recruitment Who actually receives the intervention in each Measurement of receipt in target population. Quantitative During 21 and reach in setting? Are they representative? comparison of those receiving vs. not receiving the intervention 22 individuals intervention. 23 Why do clusters achieve the pattern of reach Qualitative analysis of observational, interview and 24 they do? Do they introduce selection bias? documentary data about how clusters achieve reach. 25 Delivery to What intervention is actually delivered for each Qualitative analysis of observational, interview and During individuals cluster? documentary data about what intervention is delivered and intervention 26 why. 27 Is the delivered intervention the one intended Measurement of intervention fidelity across its components. 28 by the researchers? 29 Response of How does the target population respond? Qualitative analysis of observational and interview data During 30 individuals about target population’s experience of and response to intervention and 31 the intervention. post-intervention 32 Maintenance How and why are these processes sustained Any of the above, but probably focused on processes During 33 over time (or not)? identified as critical, or as likely to be difficult to sustain. intervention and post-intervention

34 http://bmjopen.bmj.com/ 35 Unintended Are there unintended changes in processes and Qualitative analysis of observational and interview data for Intervention and consequences outcomes, both related to the trial intervention identification. Quantitative data collection for potential post-intervention 36 and unrelated care? unintended consequences during the trial, or use of routine 37 datasets. 38 Theory What theory has been used to develop the Quantitative process data analysis can assess whether Post-intervention 39 intervention? predicted relationships and sequences of change happened 40 during implementation. 41 Context What is the wider context in which the trial is Qualitative data collection from policy documents or Pre-intervention 42 being conducted? interviews. and early

intervention on October 3, 2021 by guest. Protected copyright. 43 44 45 example, how clusters vary in their response and delivery [11,35]. Even without explicit use of theory, however, a 46 of an intervention to the target population could be process evaluation can productively draw on the resear- 47 examined using qualitative analysis of interview and chers’ implicit models of how an intervention is expected 48 observational data, with variation in outcome across to work [30,41]. Although the right design and method will 49 clusters examined quantitatively. Similarly, a study that therefore depend on the purpose of the evaluation, which 50 seeks to explore individuals’ receipt of and response to will vary with the trial design and context, it is critical that 51 an intervention can initially use quantitative data to the choices made and the rationale behind them is made 52 inform sampling strategies and qualitative methods to explicit. 53 explore different experiences (potentially at different 54 stages of implementation). For some purposes, explicitly Reporting process evaluations 55 theoretical approaches to design and analysis will be The framework systematically identifies cluster-randomised 56 appropriate, such as the use of psychological theory to trial processes that are candidates for evaluation. In prac- 57 understand individual behaviour [21], or sociological tice, choices will often have to be made that balance the 58 approaches to understanding response in different contexts, ideal with the feasible, focusing attention and available 59 such as realistic evaluation or normalisation process theory resources on key research questions. More systematic 60 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml BMJ Open Page 52 of 55

Grant et al. Trials 2013, 14:15 Page 7 of 10 BMJ Open: first published as 10.1136/bmjopen-2016-015281 on 10 March 2017. Downloaded from 1 http://www.trialsjournal.com/content/14/1/15 2 3 4 5 Table 2 Mapping of the reporting framework to three selected process evaluations 6 Summary of trial Clearly Stated purpose Processes Specify Methods used Choice Report being evaluated labelled examined timing of main 7 as a method findings of 8 process justified trial 9 evaluation 10 Nazareth Cluster- Yes ’To describe the Cluster Retrospective/ Reporting of Partly Trial design, 11 and randomised trial steps leading to recruitment, post hoc proportion of intervention, colleagues of a pharmacist- the final primary Delivery to practices recruited. targeted 12 [30] led educational outcome and clusters, Adoption Association outcomes 13 outreach explore the effect and Delivery to between and results 14 intervention to of the intervention target population, proportion of GPs summarised, improve GPs’ on each step of Quantitative in each practice main trial 15 prescribing qualityFor peerthe hypothesised reviewassociations with onlyattending paper 16 pathway of effectiveness education referenced 17 change in (reported in main outreach, the 18 professionals’ trial paper) intervention and prescribing change in 19 behaviour’ prescribing. Mixed- 20 methods 21 assessment of barriers and 22 facilitators to 23 adoption and 24 delivery to target 25 population. 26 Byng and Cluster- Yes ’To unpick the Adoption, Reach Retrospective Realistic Yes Trial design, colleagues randomised trial complexity of the and Delivery to data evaluation, intervention 27 [11] of a complex intervention by target population, collection, qualitative case and results 28 intervention to examining Qualitative unclear if study, analysis of summarised, 29 promote shared interactions associations with planned interview data main trial 30 care for people between effectiveness prospectively with a purposive paper with severe components and sample of referenced 31 mental health context and then participating 32 problems further defining its mental health 33 core functions’ team-practice cases. Case study

34 findings were used http://bmjopen.bmj.com/ 35 to better 36 understand how 37 the intervention changed practice 38 and targeted 39 outcomes. 40 Fretheim Cluster- Yes ’The main Delivery to Prospective / Quantification of Partly Trial design, 41 and randomised trial objective of this clusters, Adoption pre-specified GP perceptions of intervention, 42 colleagues of a multifaceted analysis was to and Quantitative the intervention targeted

[33] intervention identify factors associations with and the trial, and outcomes on October 3, 2021 by guest. Protected copyright. 43 (educational that could explain effectiveness pharmacist and results 44 outreach, audit variation in assessment of the summarised, 45 and feedback, outcomes across quality of main trial computerised practices’ educational paper 46 reminders, patient outreach. referenced 47 information) to Regression analysis 48 improve GPs’ of associations 49 prescribing quality with change in prescribing. 50 51 GP, general practitioner. 52 53 reporting of key information would help make the rationale Process evaluations should clearly state their purpose. 54 for choices more explicit, improving interpretation. We Process evaluations should explicitly state their original 55 believe that this key information includes the following purpose and research questions, the processes being 56 factors: studied and an acknowledgement of what is not being 57 Process evaluations should be clearly labelled. Existing evaluated [21]. Changes to research questions during the 58 process evaluation studies are poorly labelled and hard to study can be legitimate but should be explicit. For ex- 59 identify. ample, a pre-planned examination of recruitment and 60 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml Page 53 of 55 BMJ Open

Grant et al. Trials 2013, 14:15 Page 8 of 10 BMJ Open: first published as 10.1136/bmjopen-2016-015281 on 10 March 2017. Downloaded from 1 http://www.trialsjournal.com/content/14/1/15 2 3 4 5 reach may prove to be less important than understand- framework focuses on the activity of clusters, but pays 6 ing unexpected variation in intervention delivery. less attention to how clusters themselves respond to an 7 Process evaluations should clearly report if they were intervention directed at them, to how patients respond 8 pre-specified or post hoc, and why the selected timing to changes in care, or to unintended consequences [19]. 9 was chosen. Pre-specified and post-hoc process evalua- We believe it is useful for designers of cluster- 10 tions are both legitimate designs. Pre-specified evalua- randomised trials of complex interventions to start with 11 tions are more suited to quantitatively examine prior a broad overview of all potential candidates, and to 12 hypotheses about trial processes, whereas post-hoc eva- clearly justify why some candidates are selected for 13 luations are more flexible for examining unanticipated detailed examination, and the methods and theories 14 problems in implementation or unexpected findings. chosen to examine selected processes. 15 Again, clarity in whatFor was done andpeer why is key to inter-reviewFurther work only is required to develop critical appraisal 16 preting the validity and credibility of the findings. instruments for process evaluation studies; our proposed 17 Process evaluations should state the choice of methods criteria for reporting process evaluations represent mini- 18 and justify them in terms of the stated aims of the eva- mal requirements. We believe they will be helpful whilst 19 luation. The rationale for the methods used should be more empirically-based criteria for quality assessments 20 reported in relation to evaluation aims. are developed, as have been established for the reporting 21 Process evaluations should summarise or refer to the [28] and quality assessment of trials [42]. 22 main findings of the trial. To aid interpretation of evalu- There is no single best way to conduct a process 23 ation findings, trial and evaluation reports should cross- evaluation, and researchers will usually need to make 24 reference each other and process evaluations should choices about which research questions to focus on and 25 summarise the main trial findings. which methods to use. We recommend that researchers 26 Table 2 maps our proposed design and reporting frame- explicitly state the purpose of their process evaluation 27 work to three process evaluations that were broad and and demarcate its limits by clearly stating which pro- 28 detailed in reporting their process evaluation and that we cesses are being examined. We propose our framework 29 judged to significantly add understanding to the main trial to facilitate this. Although we focus on process evalua- 30 findings. These vary considerably in their purpose, tions of cluster-randomised trials, elements of the frame- 31 whether they were prospective or retrospective, the pro- work are applicable to other study designs for evaluating 32 cesses examined and the methods used. We believe our complex interventions. Process evaluations are complex, 33 framework helps clarify what kind of process evaluation diverse and face differing constraints based on the main

34 http://bmjopen.bmj.com/ they were, including the processes they do not examine trial context and study funding. However, a more struc- 35 (which is not always explicit in the original papers). tured approach to process evaluation design and repor- 36 ting will improve the validity and dissemination of such 37 Conclusion 38 studies. 39 There are a number of approaches to the design of 40 process evaluations of randomised trials in the literature Abbreviations [1-3,11,19,23,26,27,31,35,37] and other additional rele- CONSORT: Consolidated Standards of Reporting Trials; DQIP: Data-driven 41 Quality Improvement in Primary Care; RE-AIM: Reach Efficacy Adoption, 42 vant literature related to public health interventions Implementation and Maintenance. 43 where randomisation is often not used. Our framework on October 3, 2021 by guest. Protected copyright. pulls many smaller design considerations together into a Competing interests 44 The authors declare that they received no support from any organisation for 45 comprehensive overview of candidate elements that are the submitted work. RF has received funding for unrelated research with 46 potentially relevant to the design of process evaluations Reckitt Benckiser Group Plc in the previous 3 years. The authors declare that 47 of cluster-randomised trials of complex interventions. they have no competing interests. The framework draws attention to a wider range of pro- 48 Authors’ contributions 49 cesses that can be evaluated than many of the existing AG and BG were involved in the initial conceptualisation and design. AG 50 approaches, which only apply to some of these elements, reviewed the literature, carried out the analysis and interpretation of the data although they do offer useful ways of examining these in and contributed to the design of the model. AG prepared the first 51 manuscript and is responsible for this article. BG, ST and RF contributed to 52 depth. For example, normalisation process theory [35] the analysis, interpretation of data, and design of the model. BG, TD and AG 53 maps well to some candidates of our framework, such as designed the final framework model. All authors iteratively commented on 54 ‘response of clusters’, ‘delivery to individuals’ and ‘main- successive drafts of the manuscript. All authors read and approved the final manuscript. 55 tenance’, and facilitates a more in-depth sociological ex- 56 ploration of these candidates. Hardeman and colleagues’ Acknowledgements 57 causal modelling approach [37] maps well to ‘response AG and TD are funded by the Scottish Government Health Directorates Chief 58 ’ Scientist Office Applied Programme Research Grant 07/02 for the DQIP of individuals and facilitates a more in-depth psycho- prescribing improvement study. This funding body have had no direct role 59 logical exploration of this candidate. The RE-AIM in the design and conduct of, or decision to publish, this work. The authors 60 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml BMJ Open Page 54 of 55

Grant et al. Trials 2013, 14:15 Page 9 of 10 BMJ Open: first published as 10.1136/bmjopen-2016-015281 on 10 March 2017. Downloaded from 1 http://www.trialsjournal.com/content/14/1/15 2 3 4 5 would like to thank Professor Peter Donnan, University of Dundee for his 17. Ellard D, Taylor S, Parsons S, Thorogood M: The OPERA trial: a protocol for the statistical advice. process evaluation of a randomised trial of an exercise intervention for 6 older people in residential and nursing accommodation. Trials 2011, 12:28. 7 Author details 18. Underwood M, Eldridge S, Lamb S, Potter R, Sheehan B, Slowther A-M, 8 1Quality, Safety and Informatics Research Group, Population Health Sciences, Taylor S, Thorogood M, Weich S: The OPERA trial: protocol for a Medical Research Institute, University of Dundee, Mackenzie Building, randomised trial of an exercise intervention for older people in 9 Dundee DD2 4BF, UK. 2Medicines Management Unit, NHS Tayside, c/o residential and nursing accommodation. Trials 2011, 12:27. 10 University of Dundee, Mackenzie Building, Dundee DD2 4BF, UK. 3Leeds 19. Glasgow RE, McKay HG, Piette JD, Reynolds KD: The RE-AIM framework for 11 Institute of Health Sciences, Charles Thackrah Building, University of Leeds, evaluating interventions: what can it tell us about approaches to chronic 12 101 Clarendon Road, Leeds LS2 9LJ, UK. illness management? Patient Educ Couns 2001, 44:119–127. 20. Akl EA, Treweek S, Foy R, Francis J, Oxman AD: NorthStar, a support tool 13 Received: 25 April 2012 Accepted: 18 December 2012 for the design and evaluation of quality improvement interventions in 14 Published: 12 January 2013 healthcare. Implement Sci 2007, 2:19–26. 15 For peer review21. Francis J, Eccles M,only Johnston M, Whitty P, Grimshaw J, Kaner E, Smith L, Walker A: Explaining the effects of an intervention designed to promote References 16 evidence-based diabetes care: a theory-based process evaluation of a 1. Medical Research Council: Developing and Evaluating Complex Interventions: 17 pragmatic cluster randomised controlled trial. Implement Sci 2008, 3:50. New Guidance. London: Medical Research Council; 2008. 22. Durlak J: Why programme implementation is so important. J Prev Interv 18 2. Hawe P, Shiell A, Riley T, Gold L: Methods for exploring implementation Community 1998, 17:5–18. 19 variation and local context within a cluster randomised community 23. Lewin S, Glenton C, Oxman AD: Use of qualitative methods alongside intervention trial. J Epidemiol Community Health 2004, 58:788–793. 20 randomised controlled trials of complex healthcare interventions: 3. Oakley A, Strange V, Bonell C, Allen E, Stephenson J, RIPPLE Study Team: methodological study. Br Med J 2009, 339:b3496. 21 Process evaluation in randomised controlled trials of complex 24. Hulscher M, Laurant M, Grol R: Process evaluation of quality improvement 22 interventions. Br Med J 2006, 332:413–416. interventions.InQuality Improvement Research; Understanding The Science of 4. Linnan L, Steckler A: Process evaluation for public health interventions 23 Change in Healthcare. Edited by Grol R, Baker R, Moss F. London: BMJ and research: an overview.InProcess Evaluation for Public Health 24 Publishing Group; 2004:165–183. Interventions and Research. Edited by Steckler A, Linnan L. San Francisco: 25 Jossey-Bass; 2002:1–23. 25. Cook T: Emergent principles for the design, implementation, and analysis 26 5. Wight D, Obasi A: Unpacking the 'black box': the importance of process of cluster-based experiments in social science. Ann Am Acad Pol Soc Sci 2005, 599:176–198. 27 data to explain outcomes.InEffective Sexual Health Interventions: Issues in Experimental Evaluation. Edited by Stephenson JM, Imrie J, Bonell C. Oxford: 26. Bonell C, Oakley A, Hargreaves J, Strange V, Rees R: Assessment of 28 Oxford University Press; 2003:151–166. generalisability in trials of health interventions: suggested framework – 29 6. Ellard D, Parsons S: Process evaluation: understanding how and why and systematic review. Br Med J 2006, 333:346 349. interventions work.InEvaluating Health Promotion Practice and Methods. 27. Murtagh MJ, Thomson RG, May CR, Rapley T, Heaven BR, Graham RH, Kaner 30 EF, Stobbart L, Eccles MP: Qualitative methods in a randomised controlled 31 Edited by Thorogood M, Coombes Y. Oxford: Oxford University Press; 2010. 7. Craigie A, Caswell S, Paterson C, Treweek S, Belch J, Daly F, Rodger J, trial: the role of an integrated qualitative process evaluation in providing 32 Thompson J, Kirk A, Ludbrook A, Stead M, Wardle J, Steele R, Anderson A: evidence to discontinue the intervention in one arm of a trial of a – 33 Study protocol for BeWEL: the impact of a BodyWEight and physicaL decision support tool. Qual Saf Health Care 2007, 16:224 229. activity intervention on adults at risk of developing colorectal 28. Campbell MK, Elbourne DR, Altman DG: CONSORT statement: extension to

34 – http://bmjopen.bmj.com/ adenomas. BMC Public Health 2011, 11:184. cluster randomised trials. Br Med J 2004, 328:702 708. 35 8. Protheroe J, Bower P, Chew-Graham C: The use of mixed methodology in 29. Hoddinott P, Britten J, Pill R: Why do interventions work in some places and – 36 evaluating complex interventions: identifying patient factors that not others: a breastfeeding support group trial. Soc Sci Med 2010, 70:769 778. 37 moderate the effects of a decision aid. Fam Pract 2007, 24:594–600. 30. Nazareth I, Freemantle N, Duggan C, Mason J, Haines A, Nazareth I, 9. Avery AJ, Rodgers S, Cantrill JA, Armstrong S, Cresswell K, Eden M, Elliott RA, Freemantle N, Duggan C, Mason J, Haines A: Evaluation of a complex 38 Howard R, Kendrick D, Morris CJ, Prescott RJ, Swanwick G, Franklin M, intervention for changing professional behaviour: the Evidence Based 39 Putman K, Boyd M, Sheikh A: A pharmacist-led information technology Out Reach (EBOR) Trial. J Health Serv Res Policy 2002, 7:230–238. 40 intervention for medication errors (PINCER): a multicentre, cluster 31. Craig P, Dieppe P, Macintyre S, Michie S, Nazareth I, Petticrew M: Developing and evaluating complex interventions: the new Medical 41 randomised, controlled trial and cost-effectiveness analysis. Lancet 2012, 379:1310–1319. Research Council guidance. Br Med J 2008, 337:a1655. 42 10. Medical Research Council: A Framework for the Development and Evaluation 32. Salter C, Holland R, Harvey I, Henwood K: ‘I haven't even phoned my on October 3, 2021 by guest. Protected copyright. 43 of RCTs for Complex Interventions to Improve Health. London: Medical doctor yet.’ The advice giving role of the pharmacist during 44 Research Council; 2000. consultations for medication review with patients aged 80 or more: 11. Byng R, Norman I, Redfern S, Jones R: Exposing the key functions of a qualitative discourse analysis. Br Med J 2007, 334:1101. 45 complex intervention for shared care in mental health: case study of a 33. Fretheim A, Havelsrud K, Oxman A: Rational Prescribing in Primary care 46 process evaluation. BMC Health Serv Res 2008, 8:274. (RaPP): process evaluation of an intervention to improve prescribing of 47 12. Jansen Y, de Bont A, Foets M, Bruijnzeels M, Bal R: Tailoring intervention antihypertensive and cholesterol-lowering drugs. Implement Sci 2006, procedures to routine primary health care practice; an ethnographic 1:19. 48 process evaluation. BMC Health Serv Res 2007, 7:125. 34. Farrin A, Russell I, Torgerson DJ, Underwood M: Differential recruitment in 49 13. Dreischulte T, Grant A, Donnan P, McCowan C, Davey P, Petrie D, Treweek S, a cluster randomised trial in primary care: the experience of the UK Back 50 Guthrie B: A cluster randomised stepped wedge trial to evaluate the pain, Exercise, Active management and Manipulation (UK BEAM) – 51 effectiveness of a multifaceted information technology-bsed feasibility study. Clin Trials 2005, 2:119 124. intervention in reducing high-risk prescribing of non-steroidal anti- 35. May C, Finch T: Implementing, embedding, and integrating practices: an 52 inflammatory drugs and antiplatelets in primary care: the DQIP study outline of normalisation process theory. Sociology 2009, 43:535. 53 protocol. Implement Sci 2012, 7:24. 36. Christie D, Hudson L, Mathiot A, Cole T, Karlsen S, Kessel A, Kinra S, Morris S, 54 14. Donnar A, Klar N: Design and Analysis of Cluster Randomization Trials in Nazareth I, Sovio U, Wong I, Viner R: Assessing the efficacy of the healthy Health Research. London: Arnold; 2000. eating and lifestyle programme (HELP) compared with enhanced 55 15. Donnar A, Klar N: Pitfalls of and controversies in cluster randomization standard care of the obese adolescent in the community: study protocol 56 trials. Am J Public Health 2004, 94:416–422. for a randomized controlled trial. Trials 2011, 12:242. 57 16. Ivers N, Jamtvedt G, Flottorp S, Young J, Odgaard-Jensen J, French S, 37. Hardeman W, Sutton S, Griffin S, Johnston M, White AJ, Wareham NJ, 58 O'Brien M, Johansen M, Grimshaw J, Oxman A: Audit and feedback: effects Kinmonth AL: A causal modelling approach to the development of on professional practice and healthcare outcomes. Cochrane Database theory-based behaviour change programmes for trial evaluation. 59 Syst Rev 2012, 6:CD000259. Health Educ Res 2005, 20:676–687. 60 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml Page 55 of 55 BMJ Open

Grant et al. Trials 2013, 14:15 Page 10 of 10 BMJ Open: first published as 10.1136/bmjopen-2016-015281 on 10 March 2017. Downloaded from 1 http://www.trialsjournal.com/content/14/1/15 2 3 4 5 38. Mitchie S, Johnson M, Francis J, Hardeman W, Eccles M: From theory to intervention: mapping theoretically derived behavioural determinants to 6 behaviour change techniques. Appl Psychol Int Rev 2008, 57:660–680. 7 39. Wells M, Williams B, Treweek S, Coyle J, Taylor J: Intervention description is 8 not enough: evidence from an in-depth multiple case study on the untold role and impact of context in randomised controlled trials of 9 seven complex interventions. Trials 2012, 13:95. 10 40. Grant A, Dreischulte T, Treweek S, Guthrie B: Study protocol of a mixed- 11 methods evaluation of a cluster randomised trial to improve the safety of NSAID and antiplatelet prescribing: data-driven quality improvement 12 in primary care. Trials 2012, 13:154. 13 41. Foy R, Francis J, Johnston M, Eccles M, Lecouturier J, Bamford C, Grimshaw J: 14 The development of a theory-based intervention to promote appropriate 15 disclosure of a diagnosisFor of dementia. BMC Healthpeer Serv Res 2007, 7:207. review only 42. 11 Questions to Help You Make Sense of a Trial. http://www.casp-uk.net/wp- 16 content/uploads/2011/11/CASP_RCT_Appraisal_Checklist_14oct10.pdf. 17 18 doi:10.1186/1745-6215-14-15 Cite this article as: Grant et al.: Process evaluations for cluster- 19 randomised trials of complex interventions: a proposed framework for 20 design and reporting. Trials 2013 14:15. 21 22 23 24 25 26 27 28 29 30 31 32 33

34 http://bmjopen.bmj.com/ 35 36 37 38 39 40 41 42 43 on October 3, 2021 by guest. Protected copyright. 44 45 46 47 48 49 Submit your next manuscript to BioMed Central 50 and take full advantage of: 51 52 • Convenient online submission 53 • Thorough peer review 54 • No space constraints or color figure charges 55 • Immediate publication on acceptance 56 • Inclusion in PubMed, CAS, Scopus and Google Scholar 57 • Research which is freely available for redistribution 58 59 Submit your manuscript at www.biomedcentral.com/submit 60 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml BMJ Open BMJ Open: first published as 10.1136/bmjopen-2016-015281 on 10 March 2017. Downloaded from

Process evaluation of the Data-driven Quality Improvement in Primary Care (DQIP) trial: Case-study evaluation of adoption and maintenance of a complex intervention to reduce high-risk primary care prescribing

For peer review only Journal: BMJ Open

Manuscript ID bmjopen-2016-015281.R1

Article Type: Research

Date Submitted by the Author: 07-Feb-2017

Complete List of Authors: Grant, Aileen; University of Stirling, Health Sciences & Sport Dreischulte, Tobias; University of Dundee, Population Health Sciences Guthrie, Bruce; University of Dundee, Population Health Sciences Division, Medical Research Institute

Primary Subject General practice / Family practice Heading:

Health services research, Evidence based practice, Health informatics, Secondary Subject Heading: Qualitative research, General practice / Family practice

PRIMARY CARE, Prescribing, General Practice, Quality and Safety, Keywords:

Randomised Controlled Trials, Process Evaluation http://bmjopen.bmj.com/

on October 3, 2021 by guest. Protected copyright.

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml Page 1 of 61 BMJ Open BMJ Open: first published as 10.1136/bmjopen-2016-015281 on 10 March 2017. Downloaded from 1 2 3 4 1 Process evaluation of the Data-driven Quality Improvement in Primary 5 6 2 Care (DQIP) trial: Case-study evaluation of adoption and maintenance of a 7 8 3 complex intervention to reduce high-risk primary care prescribing 9 4 10 11 12 5 13 14 6 Aileen Grant PhD, Research Fellow1 15 For peer review only 16 2 3 17 7 Tobias Dreischulte PhD, Lead Pharmacist Research and Development 18 19 8 Bruce Guthrie PhD, Professor of Primary Care Medicine3 20 21 9 22 23 24 10 Affiliations: 25 26 11 1. School of Health Sciences & Sport, University of Stirling, Stirling, UK 27 28 12 2. Medicines Governance Unit, NHS Tayside, Dundee, UK 29 13 3. Population Health Sciences Division, Medical Research Institute, University of Dundee, 30 31 14 Dundee, UK 32 33 15 Corresponding author 34 http://bmjopen.bmj.com/ 35 36 16 Aileen Grant: [email protected] 37 38 39 17 Keywords 40 41 18 General practice; family practice; prescribing; quality and safety; randomised controlled trials; 42 on October 3, 2021 by guest. Protected copyright. 43 19 process evaluation 44 45 20 46 47 21 Word count: 5,698 48 49 50 51 52 53 54 55 56 57 58 59 60 1

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml BMJ Open Page 2 of 61 BMJ Open: first published as 10.1136/bmjopen-2016-015281 on 10 March 2017. Downloaded from 1 2 3 22 Abstract 4 5 23 Objective: Explore how different practices responded to the Data driven quality improvement in 6 7 8 24 primary care (DQIP) intervention in terms of their adoption of the work, reorganisation to deliver the 9 10 25 intended change in care to patients, and whether implementation was sustained over time. 11 12 13 26 Design: Mixed methods parallel process evaluation of a cluster trial, reporting the comparative case 14 15 27 study of purposivelyFor selected peer practices. review only 16 17 18 28 Setting: Ten (30%) primary care practices participating in the trial. Scotland, United Kingdom. 19 20 21 29 Results: Four practices were sampled because they had large rapid reductions in targeted 22 23 30 prescribing. They all had internal agreement that the topic mattered, made early plans to implement 24 25 31 including assigning responsibility for work, and regularly evaluated progress. However, how they 26 27 32 internally organised the work varied. Six practices were sampled because they had initial 28 29 33 implementation failure. Implementation failure occurred at different stages depending on practice 30 31 32 34 context, including internal disagreement about whether the work was worthwhile, and intention but 33 35 lack of capacity to implement or sustain implementation due to unfilled posts or sickness. Practice 34 http://bmjopen.bmj.com/ 35 36 36 context was not fixed, and most practices with initial failed implementation adapted to deliver at 37 38 37 least some elements. All interviewed participants valued the intervention because it was an 39 40 38 innovative way to address on an important aspect of safety (although one of the non-interviewed 41 42 39 GPs in one practice disagreed with this). Participants felt that reviewing existing prescribing did on October 3, 2021 by guest. Protected copyright. 43 44 45 40 influence their future initiation of targeted drugs, but raised concerns about sustainability. 46 47 48 41 Conclusions: Variation in implementation and effectiveness was associated with differences in how 49 50 42 practices valued, engaged with, and sustained the work required. Initial implementation failure 51 52 43 varied with practice context, but was not static, with most practices at least partially implementing 53 54 44 by the end of the trial. Practices organised their delivery of changed care to patients in ways which 55 56 45 suited their context, emphasising the importance of flexibility in any future widespread 57 58 59 60 2

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml Page 3 of 61 BMJ Open BMJ Open: first published as 10.1136/bmjopen-2016-015281 on 10 March 2017. Downloaded from 1 2 3 46 implementation. 4 5 6 47 Trial registration: ClinicalTrials.gov number, NCT01425502 7 8 9 48 Strengths and limitations of the study 10 11 49 • This is a comprehensive, pre-planned, process evaluation which includes a third of all 12 50 practices which participated in the DQIP stepped-wedge cluster-randomised trial. 13 14 51 • The evaluation sampled four practices which rapidly implemented the intervention and all 15 For peer review only 16 52 six practices which failed to implement the intervention to some degree. 17 53 • A strength of the study is the use of both qualitative data from interviews and observational 18 19 54 field notes, and quantitative data about key trial processes and practice-level effectiveness 20 21 55 to examine implementation in detail. 22 56 • A limitation is that we did not collect any data from practices prior to them receiving the 23 24 57 intervention. 25 26 58 27 Key words: 28 29 59 Complex intervention, process evaluation, general practice, primary care, prescribing practice. 30 31 60 32 33

34 61 Background http://bmjopen.bmj.com/ 35 36 62 High-risk prescribing in primary care is a major concern for health care systems internationally. 37 38 63 Between 2-4% of emergency hospital admissions are caused by preventable adverse drug events,[1, 39 40 64 2] at significant cost to healthcare systems.[3][4] A large proportion of these admissions are caused 41 42 65 by commonly prescribed drugs, with non-steroidal anti-inflammatory drugs (NSAIDs) and on October 3, 2021 by guest. Protected copyright. 43 44 66 antiplatelets being frequently implicated, causing gastrointestinal, cardiovascular and renal adverse 45 46 67 events.[5-7] 47 48 49 68 The DQIP intervention was systematically developed and optimised[8-10] and comprised three 50 51 52 69 intervention components: (a) professional education about the risks of NSAIDs and antiplatelets via 53 54 70 an outreach visit by a pharmacist; (b) financial incentives to review patients at the highest risk of 55 56 71 NSAID and antiplatelet ADEs, split into a participation fee of £350 and £15 per patient reviewed; and 57 58 59 60 3

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml BMJ Open Page 4 of 61 BMJ Open: first published as 10.1136/bmjopen-2016-015281 on 10 March 2017. Downloaded from 1 2 3 72 (c) access to a web-based IT tool to identify such patients and support structured review. The 4 5 73 intervention was evaluated in a pragmatic stepped-wedge cluster-randomised controlled trial[11] in 6 7 74 33 practices from one Scottish health board, where all participating practices received the 8 9 75 intervention but were randomised to one of ten different start dates.[8] Across all practices, 10 11 12 76 targeted high-risk prescribing fell from 3.7% immediately before to 2.2% at the end of the 13 14 77 intervention period (adjusted OR 0.63 [95%CI 0.57-0.68], p<0.0001). The intervention only 15 For peer review only 16 78 incentivised review of ongoing high-risk prescribing, but led to reductions in both ongoing (adjusted 17 18 79 OR 0.60, 95%CI 0.53-0.67) and ‘new’ high-risk prescribing (adjusted OR 0.77, 95%CI 0.68-0.87). 19 20 80 Notably reductions in high-risk prescribing were sustained in the year after financial incentives 21 22 81 stopped. In addition, there were significant reductions in emergency hospital admissions with 23 24 25 82 gastrointestinal ulcer or bleeding (RR 0.66, 95%CI 0.51-0.86), and heart failure (RR 0.73, 95%CI 0.56- 26 27 83 0.95).[12] 28 29 30 84 Alongside the main trial, we designed a mixed-methods process evaluation, [13, 14] based on a 31 32 85 cluster-randomised trial process-evaluation framework which we developed.[15] Our framework 33

34 86 emphasises the importance of considering two levels of intervention delivery and response that http://bmjopen.bmj.com/ 35 36 87 often characterise cluster-randomised trials of behaviour change interventions (although their 37 38 88 importance will depend on intervention design). The first is the intervention that is delivered to 39 40 89 clusters, which respond by adopting (or not) the intervention and integrating it with existing work. 41 42 on October 3, 2021 by guest. Protected copyright. 43 90 The second is the change in care which the cluster professionals deliver to patients. In DQIP, the 44 45 91 delivery of the intervention to professionals was pre-defined, intended to be delivered with high 46 47 92 fidelity across all practices by the research team, whereas the intervention delivered to patients was 48 49 93 at the discretion of practices, who decided whether and how they reviewed patients and whether to 50 51 94 change prescribing. We used this framework to structure our parallel process evaluation, mapping 52 53 95 data collection to a logic model of how the DQIP intervention was expected to work (figure 1). 54 55 56 96 The aim of this analysis is to examine how different practices responded to the intervention 57 58 59 60 4

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml Page 5 of 61 BMJ Open BMJ Open: first published as 10.1136/bmjopen-2016-015281 on 10 March 2017. Downloaded from 1 2 3 97 delivered to them by the research team in terms of their adoption of the work, their reorganisation 4 5 98 to deliver the intended change in care to patients, and whether implementation was sustained over 6 7 99 time. 8 9 10 100 Methods 11 12 13 101 The design was a mixed-method comparative case-study, with general practices the unit of 14 15 102 analysis.[16, 17]TheFor overall peer design and methods review have been described only previously in the published 16 17 103 protocol.[18] In brief, we used a mixed-methods parallel process evaluation to examine the 18 19 104 implementation of selected processes and their associations with change in high-risk prescribing at 20 21 105 practice level. The quantitative element examined how change in prescribing at practice level was 22 23 106 associated with practice characteristics and implementation of key processes and is reported 24 25 26 107 separately.[19] The qualitative element consisted of case studies in 10 of the 33 participating 27 28 108 practices. Practice staff perceptions of the importance of different intervention components and 29 30 109 when they had an effect are reported separately.[20] The analysis reported here examines how 31 32 110 practices adopted, implemented and maintained the intervention. The study was reviewed by the 33

34 111 Fife and Forth Valley Research Ethics Committee (11/AL/0251) and informed consent was obtained http://bmjopen.bmj.com/ 35 36 112 from all participants to participate and to published anonymised data. 37 38 39 113 Case study sampling 40 41 42 114 Practices were purposively sampled to include those which had and had not initially reduced high- on October 3, 2021 by guest. Protected copyright. 43 44 115 risk prescribing, judged by visual inspection of run charts of high-risk prescribing rates approximately 45 46 116 four months after starting the intervention. Our assumption was change in high-risk prescribing was 47 48 117 a good proxy for initial response to the intervention delivered to practices. Sampling was additionally 49 50 118 structured to recruit practices starting the trial at different times, and to ensure a mix of smaller and 51 52 119 larger practices (<5000 and ≥5000 registered patients), reflecting the stepped-wedge trial design and 53 54 55 120 our a-priori belief that smaller practices would more effectively implement the intervention 56 57 121 (randomisation was stratified by listsize).[18] Ten (30%) practices were sampled and recruited to the 58 59 60 5

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml BMJ Open Page 6 of 61 BMJ Open: first published as 10.1136/bmjopen-2016-015281 on 10 March 2017. Downloaded from 1 2 3 122 process evaluation, including all six practices with no initial change in targeted prescribing. 4 5 6 123 Qualitative data collection 7 8 124 In each practice, the GP most involved in the DQIP work (leading the review work), a GP less involved 9 10 125 (who may have been involved in the review work to some degree), the practice manager and 11 12 126 pharmacist were invited for interview. All interviews were carried out by AG approximately six 13 14 15 127 months after Forthe practice startedpeer the intervention, review and the GP most onlyinvolved was interviewed again 16 17 128 three to six months later to explore changes over time. Interviews lasted approximately one hour, 18 19 129 and were audio-recorded and transcribed verbatim. Additional data generated were field-notes 20 21 130 made by AG during the educational outreach visits (EOV), detailing response to the educational 22 23 131 component and informatics tool training. Data generation took place between September 2011 and 24 25 132 December 2013 in parallel with intervention implementation. All data has been anonymised, 26 27 28 133 including by using pseudonyms for practices. 29 30 134 Quantitative data collection 31 32 33 135 Data from the DQIP IT tool was used to sample practices using run-charts to visually assess change in

34 http://bmjopen.bmj.com/ 35 136 performance in the first four months of implementation, but was not used alongside the qualitative 36 37 137 analysis reported here which was otherwise blind to any quantitative process or outcome data. Data 38 39 138 from the same source was also used after qualitative analysis was complete to measure and 40 41 139 categorise reach, delivery to the patient and maintenance. Reach was measured as the percentage 42 on October 3, 2021 by guest. Protected copyright. 43 140 of patients flagged as needing a review who had received at least one review, delivery to the patient 44 45 46 141 as the percentage of flagged patients who had further action in response to initial review (eg a 47 48 142 medication change, or an invitation to consult), and maintenance as reach in the final 24 weeks of 49 50 143 the intervention period. Effectiveness was defined as the relative change in the mean rate of high- 51 52 144 risk prescribing trial primary outcome measure between baseline (the 48 weeks before the 53 54 145 intervention) and the final 24 weeks of the intervention. Quantitative data was only used to explore 55 56 146 whether the qualitative judgements made about implementation were consistent with observed 57 58 59 60 6

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml Page 7 of 61 BMJ Open BMJ Open: first published as 10.1136/bmjopen-2016-015281 on 10 March 2017. Downloaded from 1 2 3 147 data on reach, delivery, maintenance and effectiveness. Associations between quantitative practice- 4 5 148 level process and effectiveness data will be reported separately. [19] 6 7 8 149 Researcher expectations of how the intervention would work 9 10 150 During intervention and process evaluation design the research team expected that several 11 12 151 processes had to happen for the intervention to reduce high-risk prescribing: 13 14 15 152 (1) PracticesFor had to adopt peer the intervention, review in the sense of being only convinced that the work was 16 17 153 worthwhile, engaging with the education and set-up process, and organising how the work 18 19 20 154 required would be done.[21] This is represented by the ‘response of clusters’ box in figure 1. 21 22 155 (2) Practices would have to deliver the reviews to patients by using the informatics tool to 23 24 156 identify patients, by a GP initially reviewing the records and taking any action they judged 25 26 157 necessary (for example, to stop the drug, or review the patient in person, or seek advice 27 28 158 from a specialist). This work is represented by the ‘reach’ and ‘delivery to patient’ boxes in 29 30 159 figure 1. 31 32 33 160 (3) Practices would then have to maintain this activity over the 48 weeks the intervention was

34 http://bmjopen.bmj.com/ 35 161 active (‘maintenance’ in figure 1). 36 37 38 162 Analysis 39 40 163 Analysis was iterative with data generation to allow issues and themes identified to inform 41 42 164 subsequent data generation and facilitate deeper exploration, and continued until no new themes on October 3, 2021 by guest. Protected copyright. 43 44 165 emerged. Analysis was carried out by AG with BG contributing through discussion of data and 45 46 166 interpretation until he became aware of the trial results, and was completed by AG before she knew 47 48 49 167 the outcome of the trial. 50 51 52 168 A coding frame was developed inductively and deductively from field-notes, initial interviews and 53 54 169 topic guides, framework[15] and logic model,[18] and revised during analysis through use of the 55 56 170 constant comparative method.[22] NVivo 8 was used to systematically apply the coding frame to all 57 58 59 60 7

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml BMJ Open Page 8 of 61 BMJ Open: first published as 10.1136/bmjopen-2016-015281 on 10 March 2017. Downloaded from 1 2 3 171 data. An in-depth description of each case study was constructed detailing characteristics and 4 5 172 perceptions of all practice staff that participated in interviews with additional data from the EOV 6 7 173 observation and informal interviews. This facilitated a detailed exploration on a case-by-case basis. 8 9 174 Analysis used the Framework technique which facilitated comparing the data by concept, theme and 10 11 12 175 practice and facilitated cross and within case comparisons.[23]. Analysis drew on Normalisation 13 14 176 Process Theory (NPT), which focuses attention on how interventions become integrated, embedded 15 For peer review only 16 177 and routinized into social contexts.[21] AG analysed the data collected in the study twice, 17 18 178 inductively and deductively as described above and deductively based on Normalisation Process 19 20 179 Theory. Each NPT construct was defined specifically for DQIP and these definitions can be found in 21 22 180 table 3. Coding charts and memos were developed for each NPT construct and higher order 23 24 25 181 constructs (coherence, cognitive participation, collective action and reflective monitoring). NPT 26 27 182 interpretation and coding reliability was established through a workshop with NPT experts. The data 28 29 183 was explored for negative cases. 30 31 32 184 33

34 http://bmjopen.bmj.com/ 35 185 Results 36 37 186 Findings are based on data from 38 interviews (ten lead GPs of whom nine were interviewed twice, 38 39 187 seven GPs less involved with DQIP, nine practice managers and three practice pharmacists) and from 40 41 188 approximately 11 hours of field-notes. Table 1 summarises the practice characteristics and intended 42 on October 3, 2021 by guest. Protected copyright. 43 189 and actual process for delivering the intervention to patients, ordered by the final qualitative 44 45 190 judgement about the extent to which they implemented the intervention as intended by the 46 47 191 research team (table 2). 48 49 50 192 Perceptions of the work required and initial plans to implement 51 52 53 193 All GPs interviewed said they believed the targeted prescribing to be a cause of potentially 54 55 194 preventable harm which was worth addressing (although in one practice, one non-interviewed GPs 56 57 195 did not believe this – see below). There was a consistent perception that safety-focused prescribing 58 59 60 8

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml Page 9 of 61 BMJ Open BMJ Open: first published as 10.1136/bmjopen-2016-015281 on 10 March 2017. Downloaded from 1 2 3 196 improvement work was more engaging for GPs (and patients) compared to previous experience of 4 5 197 interventions to reduce prescribing cost. Some GPs reported they had already reviewed NSAID 6 7 198 prescribing under a local health board initiative but DQIP was more meaningful because it focused 8 9 199 on and identified individuals at higher risk of adverse events. All but one interviewed GP perceived 10 11 12 200 the DQIP intervention to be clearly differentiated from existing prescribing improvement activity, 13 14 201 and to be of significant value. 15 For peer review only 16 17 202 Although practices were free to organise the work as they saw fit, the research team knew 18 19 203 approximately half of patients requiring review over the year would be identified when the practice 20 21 204 started the intervention. At the educational visit, the research team therefore encouraged practices 22 23 205 to divide the work across more than one GP, and most practices initially agreed to do this. However, 24 25 206 many found this was either not feasible or sustainable (for example, where many or most GPs in a 26 27 28 207 practice worked part-time), or in small practices was not necessary since one GP could comfortably 29 30 208 manage the workload. 31 32 33 209 However, all practices found this initial volume of work required dedicated time to complete and

34 http://bmjopen.bmj.com/ 35 210 could not be done opportunistically. Three practices (Mingulay, Lingay and Hirta) allocated 36 37 211 administrative time to carrying out the DQIP reviews, in the remaining practices the GPs reviewed 38 39 212 patients once routine work was finished. 40 41 42 213 “…we ended up doing it in the evenings; it was after six o'clock, so it was sitting here at half on October 3, 2021 by guest. Protected copyright. 43 44 214 past six, seven o'clock in the evening ploughing through patients’ notes.” (Hellisay, GP 2 45 46 215 interview 1). 47 48 49 216 Case study practices which had not initially reduced high-risk prescribing 50 51 52 217 Six practices were sampled on the basis of having no initial change in high-risk prescribing, and were 53 54 218 assumed by the research team to be initial ‘non-adopters’. However, the reasons for this varied 55 56 219 across practices. 57 58 59 60 9

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml BMJ Open Page 10 of 61 BMJ Open: first published as 10.1136/bmjopen-2016-015281 on 10 March 2017. Downloaded from 1 2 3 220 Failure to initially adopt the DQIP intervention. Three practices failed to initially adopt the DQIP 4 5 221 intervention, although for different reasons and with different consequences. In Orosay and 6 7 222 Boreray, non-adoption was persistent, whereas Lingay did adopt after a delay. 8 9 10 223 The GPs in Orosay never agreed how the work required would be done, partly because of lack of 11 12 224 agreement that it was important to do. There was poor attendance at the initial EOV and at a second 13 14 15 225 visit requestedFor by the practice peer after approximately review six months. One GPonly in Orosay was highly 16 17 226 motivated and engaged, but one felt they had no obligation to participate despite having received 18 19 227 the initial payment, and another did not believe the targeted prescribing needed review. 20 21 22 228 “How much risk is each individual patient at? What will happen if we don’t do anything? 23 24 229 This GP’s questioning resulted in a long discussion with this GP arguing that he had little 25 26 230 experience of renal failure due to NSAIDs; he said he is an old GP and his experience shapes 27 28 231 his prescribing.” (Extract from field notes 17.05.12) 29 30 31 232 During practice visits and during interviews in Orosay, administrative staff indicated that working 32 33 233 relationships among GPs were frayed at times and there was a belief that it was not appropriate for

34 http://bmjopen.bmj.com/ 35 36 234 one GP to change another’s prescribing. However, although patients were not reviewed as 37 38 235 intended, there was some evidence individual GPs in Orosay did change their prescribing as a result 39 40 236 of the educational intervention. 41 42 on October 3, 2021 by guest. Protected copyright. 43 237 “Yeah it's been because of DQIP really yes, aye. … It's merely made me sit up and pay a bit 44 45 238 more attention… It's more now when I see them, and I didn’t even know if their name’s on 46 47 239 the tool.” (Orosay, GP 1 interview 1) 48 49 50 240 In Boreray all GPs legitimised the work. The practice did agree a process by which to do the work but 51 52 241 they never got started due to persistent clinical and administrative understaffing: 53 54 55 242 “…we lost a partner… but the (other) doctors have cut their hours, they havnae got time…so 56 57 243 it's kinda been a wee bit o' upheaval in the last few months. So I was hoping they would take 58 59 60 10

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml Page 11 of 61 BMJ Open BMJ Open: first published as 10.1136/bmjopen-2016-015281 on 10 March 2017. Downloaded from 1 2 3 244 on a full-time Practice Manager… but they've taken on a part-time one.” (Boreray, 4 5 245 administrative interview) 6 7 8 246 As a result, the practice prioritised routine clinical work, although at the final interview the GP said 9 10 247 they had not known that each reviewed patient earned a £15 payment, and that knowing this might 11 12 248 have led to different priorities. 13 14 15 249 Lingay experiencedFor a ‘delayed peer implementation’. review When examining the only practice data during the EOV, 16 17 250 it was obvious to the two GPs in the practice that many of the patients identified usually consulted 18 19 20 251 with one of them. This GP was keen to review identified patients, but this was delayed by 21 22 252 informatics access problems and little reviewing was initially done as motivation waned. 23 24 253 Subsequently, one of the GPs was temporarily unable to do clinical work for health reasons which 25 26 254 created time for doing work like DQIP. Once started, the GPs became highly motivated by seeing 27 28 255 reductions in high-risk prescribing, and this led them to change their appointment system to protect 29 30 256 specific time to do quality improvement work. 31 32 33 257 “…we are making changes as of this month, we are staggering the times of surgeries so that

34 http://bmjopen.bmj.com/ 35 36 258 there's actually going to be proper paperwork time” (Lingay, GP1, interview 2). 37 38 39 259 Failure to consistently reach patients targeted by the DQIP intervention. In Mingulay all professionals 40 41 260 legitimised DQIP, and agreed that one GP would do all the work in two hours dedicated 42 on October 3, 2021 by guest. Protected copyright. 43 261 administrative time per month. However, this was not enough time for the GP to complete the initial 44 45 262 large volume of reviews, exacerbated by the informatics tool initially running slowly (a problem for 46 47 263 some practices in the first waves of implementation). There was also poor communication with the 48 49 264 other GPs in the practice due to most being part-time with heavy use of locums. The GP doing the 50 51 52 265 work therefore felt that they never got on top of the reviewing, and perceived that stopped 53 54 266 prescribing was often restarted. 55 56 57 267 “The first few times we sort of went great guns and I got right down to, I only had 5 to 58 59 60 11

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml BMJ Open Page 12 of 61 BMJ Open: first published as 10.1136/bmjopen-2016-015281 on 10 March 2017. Downloaded from 1 2 3 268 review and then of course the next time I went on there was something like 19 'cause 4 5 269 obviously there had been a couple that had re-triggered for whatever reason and then 6 7 270 others that had obviously come on in that time and it was very frustrating.” (Mingulay, GP 1 8 9 271 interview 1) 10 11 12 272 Failure to deliver the DQIP intervention to patients. In Gighay, one GP committed the practice to 13 14 15 273 DQIP, attendedFor the EOV and peer carried out all thereview review work, but made only few changes, instead flagging 16 17 274 the patient’s record for the attention of whichever GP next saw them. As a result, GPs who had not 18 19 275 received the educational intervention (written educational materials were not internally distributed) 20 21 276 were responsible for delivering the change in prescribing, and delivery was therefore dependent on 22 23 277 both the patient attending for routine care and on the GP seeing them responding to the flag in the 24 25 278 record. 26 27 28 279 Failure to maintain delivery of the DQIP intervention to patients. Hellisay was a two GP practice 29 30 280 where one GP attended the EOV and rapidly started reviewing patients. However, the other GP then 31 32 33 281 left, and although a new GP replaced them, the remaining original GP then went off sick.

34 http://bmjopen.bmj.com/ 35 36 282 “We blitzed the first numbers that we had, we then drifted along for a little bit doing some 37 38 283 (reviews)… roundabout Christmas we stopped doing DQIP 'cause Christmas is, is such a busy 39 40 284 rush time, anyway DQIP dropped to the bottom of the priority list. Pretty much after 41 42 285 Christmas Dr X went off sick.” (Hellisay, GP 2, interview 1) on October 3, 2021 by guest. Protected copyright. 43 44 45 286 Reviewing was therefore not maintained, although the new partner said had they been aware of the 46 47 287 financial incentive, it might have encouraged them to prioritise DQIP over other work. 48 49 50 288 Case study practices which had large initial reductions in high-risk prescribing 51 52 289 Four practices were sampled on the basis of having large initial reductions in high-risk prescribing. In 53 54 55 290 these practices, there was full attendance by virtually all prescribers at the EOVs and there was 56 57 291 strong and widespread legitimation and commitment to deliver. Two were small practices which 58 59 60 12

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml Page 13 of 61 BMJ Open BMJ Open: first published as 10.1136/bmjopen-2016-015281 on 10 March 2017. Downloaded from 1 2 3 292 initially decided to share the review work among all GPs, but within a couple of months of starting, 4 5 293 one more motivated and/or IT literate GP took responsibility for the work because it was “not too 6 7 294 onerous for one GP” (Interview Scalpay GP1, interview 1). In the two large practices the initial review 8 9 295 work and patient communication was shared by all GPs. In Hirta, one GP then took responsibility for 10 11 12 296 subsequent review work and consultations with patients, whereas in Taransay the work continued 13 14 297 to be shared across GPs and coordinated by a member of administrative staff. 15 For peer review only 16 17 298 Maintenance 18 19 299 No practice felt DQIP put an unfeasible burden on practice resources although it clearly required 20 21 300 time to review records and communicate with patients, and two practices (Hellisay and Boreray) 22 23 301 were unable to deliver the intervention due to being understaffed and prioritising competing 24 25 302 demands. Most GPs delivered the required work outside routine opening hours, although in three 26 27 28 303 practices GPs used time already allocated for administrative work. As a result all practices primarily 29 30 304 communicated with patients by telephone. DQIP work was therefore done as separate activity 31 32 305 rather than fully integrated with existing work, with only Lingay modifying any broader processes or 33

34 306 systems as a result of the DQIP work. http://bmjopen.bmj.com/ 35 36 37 307 Participants perceived that the review work influenced their future prescribing, saying that the work 38 39 308 was more ‘hands on’ than other prescribing interventions which were pharmacist-led, and came 40 41 309 with clear and concise prescribing advice. 42 on October 3, 2021 by guest. Protected copyright. 43 44 310 “'cause you're thinking “oh …”, when you're prescribing anti-inflammatory, is there a reason 45 46 311 I shouldn’t be or should I prescribe a PPI as well, so I think it probably does focus your mind.” 47 48 49 312 (Scalpay GP1, interview 1). 50 51 52 313 Since participants felt this applied less to GPs not doing reviewing, they used a range of strategies to 53 54 314 try to prevent drugs being restarted without due consideration, including informal discussion in the 55 56 315 coffee room or in practice meetings, and flagging the notes with a warning. GPs from the practices 57 58 59 60 13

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml BMJ Open Page 14 of 61 BMJ Open: first published as 10.1136/bmjopen-2016-015281 on 10 March 2017. Downloaded from 1 2 3 316 which successfully reduced the targeted high risk prescribing continued to check the tool to ensure 4 5 317 no patients had been incorrectly restarted on high-risk drugs and no new patients had been 6 7 318 prescribed the targeted medication, and expressed concern that when the trial ended they would 8 9 319 not be able to check this. 10 11 12 320 “…the worry is of course when, when the project stops whether things will drift back up 13 14 15 321 again,For I just wondered peer if, if this software review could be left running… only … 'cause I think it's very, very 16 17 322 useful … I think, it's like everything, unless you're constantly reminded of things it does, it 18 19 323 does, it does tend to slip a little bit” (Monach GP 1,interview 2). 20 21 22 324 Unintended consequences 23 24 325 Unintended consequences were explored in the interviews but none were identified. 25 26 27 326 Quantitative data in relation to qualitative assessment of the extent of implementation 28 29 327 Table 2 summarises the qualitative assessment of the extent of practice adoption and 30 31 328 implementation made before the trial results and quantitative process data were known, and 32 33 329 ordered from the least to most successful adopters/implementers. It additionally shows quantitative

34 http://bmjopen.bmj.com/ 35 36 330 measures of reach, delivery to patients, maintenance and effectiveness to contextualise the 37 38 331 qualitative judgement. The four practices sampled as having rapid initial reductions in high-risk 39 40 332 prescribing (Scalpay, Hirta, Monach, Taransay) were all qualitatively judged to have rapid and 41 42 333 sustained adoption and implementation. These four practices also had the highest quantitative on October 3, 2021 by guest. Protected copyright. 43 44 334 measures of reach, and among the highest measures of delivery and maintenance. They had 45 46 335 amongst the largest reductions in targeted high-risk prescribing across all practices in the trial (figure 47 48 49 336 2). 50 51 52 337 The six practices sampled as having no initial reduction in high-risk prescribing (assumed to be ‘initial 53 54 338 non-adopters’ during sampling) were more varied in both qualitative and quantitative assessment. 55 56 339 Table 2 shows the two practices qualitatively judged to have not adopted the intervention at all 57 58 59 60 14

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml Page 15 of 61 BMJ Open BMJ Open: first published as 10.1136/bmjopen-2016-015281 on 10 March 2017. Downloaded from 1 2 3 340 (Orosay and Boreray) had very low rates of reach, delivery and maintenance, all of which were 4 5 341 measured using data from the informatics tool. Orosay did have a reduction in high-risk prescribing 6 7 342 consistent with qualitative data that there was change in prescribing practice by at least some GPs 8 9 343 even though the informatics tool was never systematically used, but high-risk prescribing increased 10 11 12 344 in Boreray (table 2 and figure 2). The other four practices sampled as having no initial reduction in 13 14 345 high-risk prescribing were qualitatively judged to have implemented with delay or in a way which the 15 For peer review only 16 346 practices themselves perceived as sub-optimal. They had generally intermediate levels of reach over 17 18 347 the whole intervention period (table 2). Consistent with the qualitative data, Hellisay had limited 19 20 348 delivery to patients of changed prescribing or follow-up, and poor maintenance with minimal change 21 22 349 in the rate of high-risk prescribing. The remaining three practices had somewhat lower but 23 24 25 350 overlapping levels of delivery to patients compared to the successful implementers, with similar 26 27 351 rates of maintenance, and somewhat lower rates of effectiveness, consistent with the qualitative 28 29 352 interpretation that these practices had delayed but ultimately successful implementation (table 2 30 31 353 and figure 2). 32 33

34 354 Discussion http://bmjopen.bmj.com/ 35 36 355 This study examined implementation of the DQIP intervention in ten practices (30% of trial 37 38 39 356 practices) purposively selected for variation in initial reduction in targeted high-risk prescribing, 40 41 357 which we assumed was a proxy for initial adoption and implementation. For the four practices 42 on October 3, 2021 by guest. Protected copyright. 43 358 selected on the basis of early reductions in high-risk prescribing, both qualitative and quantitative 44 45 359 data showed evidence of rapid and sustained adoption and implementation, and they were among 46 47 360 the most effective in terms of practice-level change in high-risk prescribing. Our sample included all 48 49 361 the six practices which had no initial reduction in high-risk prescribing and these were all judged 50 51 52 362 qualitatively to either have not adopted at all because of disagreement about the value of the work, 53 54 363 or to have experienced delay in implementation related to their organisational context, and the 55 56 364 quantitative measures were largely consistent with the qualitative interpretation. 57 58 59 60 15

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml BMJ Open Page 16 of 61 BMJ Open: first published as 10.1136/bmjopen-2016-015281 on 10 March 2017. Downloaded from 1 2 3 365 The way that practices did the work of implementation did therefore appear to be associated with 4 5 366 effectiveness. An attractive theoretical lens to examine this work is Normalisation Process Theory 6 7 367 (NPT).[21] The NPT construct coherence refers to participants’ understanding of the intervention. 8 9 368 Interviewed professionals in all practices clearly perceived the intervention and the associated 10 11 12 369 review work as different from existing quality improvement work because of its focus on safety 13 14 370 rather than cost. Cognitive participation relates to the enrolment and engagement of practices and 15 For peer review only 16 371 clinicians to the DQIP work. In all practices, there was at least one engaged GP, but collective 17 18 372 engagement varied across other GPs. Most or all GPs in nine of the case-study practices highly 19 20 373 legitimised the DQIP intervention, perceiving this as an innovative way to address an important 21 22 374 safety problem. However, in the one practice where there was openly stated disagreement about 23 24 25 375 value, implementation of the intervention failed from the outset (although there was evidence of 26 27 376 change in prescribing practice by GPs who did legitimise the work and of reduction in high-risk 28 29 377 prescribing, contrary to our belief that use of the informatics tool would be required for 30 31 378 effectiveness). 32 33

34 379 Collective action focuses on how practices organised the work required for DQIP, which was usually http://bmjopen.bmj.com/ 35 36 380 as a separate activity rather than fully integrated with existing work or practice routines. In the 37 38 381 smaller practices which successfully implemented the intervention immediately or with delay, one 39 40 382 GP did all the reviewing partly because the numbers made this feasible (contrary to our prior belief 41 42 on October 3, 2021 by guest. Protected copyright. 43 383 that intervention effectiveness would require sharing of the work). In the larger practices the initial 44 45 384 bulk of work was shared between GPs which required coordination by an administrator, but 46 47 385 continued reviewing was shared or taken on by the most engaged GP. 48 49 50 386 Reflexive monitoring is about how participants assessed their progress in delivering the intervention, 51 52 387 and responded to problems. Many GPs in practices which successfully implemented the intervention 53 54 388 said that they regularly checked the tool to monitor progress. Interviewed GPs perceived that 55 56 389 reviewing altered their future prescribing but were concerned that this did not apply to their 57 58 59 60 16

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml Page 17 of 61 BMJ Open BMJ Open: first published as 10.1136/bmjopen-2016-015281 on 10 March 2017. Downloaded from 1 2 3 390 colleagues. They described using a number of strategies to try to influence colleague’s prescribing to 4 5 391 try to ensure sustainability but were uncertain whether this was effective (although the main trial 6 7 392 analysis found significant reductions in new high-risk prescribing, and that reductions in total high- 8 9 393 risk prescribing were sustained in the 48 weeks after financial incentives ceased).[12] 10 11 12 394 Overall, coherence appeared uniformly high across case-study practices (perhaps unsurprising in 13 14 15 395 practices volunteeringFor to participatepeer in a trial), review and so did not appear only to influence implementation. 16 17 396 However, varied perceptions of the legitimacy of the work and low engagement (cognitive 18 19 397 participation) significantly contributed to poor adoption in two practices. More successful practices 20 21 398 delivered the intervention in a variety of ways which did not always match the research teams’ prior 22 23 399 beliefs about how best to deliver the intended care, highlighting that allowing practices flexibility to 24 25 400 implement in ways that worked for them was important (collective action). Successful implementers 26 27 28 401 also regularly reviewed their progress, and used a range of strategies to change their colleagues’ 29 30 402 practice to address their own concerns about sustainability (reflexive monitoring). 31 32 33 403 Our other qualitative process evaluation paper reports that practice participants perceived all

34 http://bmjopen.bmj.com/ 35 404 primary components of the intervention delivered to them by the research team to be active 36 37 405 (financial incentives, education, informatics).[20] This paper complements and extends those 38 39 406 findings by examining how practices actually responded to the intervention delivered to them by 40 41 407 adopting and implementing the work of delivering changed care to patients. They are also consistent 42 on October 3, 2021 by guest. Protected copyright. 43 408 with the quantitative examination of variation in effectiveness.[19] Six of the ten case-study 44 45 46 409 practices were sampled because they did not initially reduce high-risk prescribing (assumed to be 47 48 410 initial implementation failures), but most had delayed implementation and did achieve reductions in 49 50 411 targeted prescribing. The practice where qualitative and quantitative data was most incongruent 51 52 412 was Orosay, which was qualitatively judged to be an implementation failure (true in the sense that it 53 54 413 made no use of the informatics tool and claimed no payments for reviewing), but which had a 19% 55 56 414 reduction in targeted high-risk prescribing, consistent with the educational intervention altering 57 58 59 60 17

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml BMJ Open Page 18 of 61 BMJ Open: first published as 10.1136/bmjopen-2016-015281 on 10 March 2017. Downloaded from 1 2 3 415 clinical practice. This supports the idea that multicomponent interventions to change professional 4 5 416 practice may be somewhat more effective than single component interventions, since professionals 6 7 417 or the organisations they work in may vary in how they respond (or not) to different elements. 8 9 10 418 In the wider literature, the intervention evaluated in the PINCER trial is the closest to DQIP in design 11 12 419 and focus.[24] Significant differences were that PINCER was pharmacist-led and delivered over 12 13 14 15 420 weeks, and thatFor PINCER had peer a more standardised review intervention in terms only of how the pharmacists 16 17 421 completed reviews. The PINCER process evaluation found that some of the pharmacists doing 18 19 422 reviews did not feel well integrated into the practice team, found the work repetitive and would 20 21 423 have preferred some flexibility to tailor the intervention to different practice contexts.[25] Both 22 23 424 interventions showed large reductions in targeted prescribing, but the DQIP intervention had 24 25 425 sustained effects 48 weeks after the intervention ceased whereas the PINCER effect at 12 months 26 27 28 426 was somewhat smaller than at six months.[24] The DQIP findings suggest that GP-led reviewing may 29 30 427 lead to more sustainable change by having greater influence on less involved GPs, and by changing 31 32 428 future initiation of high-risk prescribing. However, since both studies were carried out in volunteer 33

34 429 practices, it is likely that non-volunteer practices will require greater effort to engage and greater http://bmjopen.bmj.com/ 35 36 430 support to deliver similar outcomes which has implications for implementation across entire 37 38 431 healthcare systems. From this perspective, pharmacist-led interventions have the potential 39 40 432 advantage that it may be easier to deliver at least some change in practices which are less engaged 41 42 on October 3, 2021 by guest. Protected copyright. 43 433 or which are resource constrained.[25] 44 45 46 434 These findings are likely to be generalizable beyond this immediate study and context. For example, 47 48 435 Kennedy et al found that the Whole System Informing Self-management Engagement (WISE) 49 50 436 intervention was not routinely adopted by practices because it was not perceived as relevant or 51 52 437 legitimate activity or a priority for general practice, and it did not fit within existing work. These 53 54 438 findings align with the findings of this study that lack of coherence and cognitive participation were 55 56 439 important barriers to implementation, and therefore may be important targets for intervention.[26] 57 58 59 60 18

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml Page 19 of 61 BMJ Open BMJ Open: first published as 10.1136/bmjopen-2016-015281 on 10 March 2017. Downloaded from 1 2 3 440 Likewise, Berendesen et al found the BeweegKuur (combined lifestyle intervention) was not 4 5 441 implemented according to protocol and had poor sustainability because patient’s expectations of 6 7 442 the intervention were not met and tailoring to general practice and patient contexts was 8 9 443 required.[27] Moore et al have also shown how implementation of an exercise intervention varied 10 11 12 444 by patient characteristics and context.[28] Like this study, the process evaluation of the 13 14 445 implementation of the ESTEEM trial comparing GP versus nurse telephone triage of same-day 15 For peer review only 16 446 appointment requests highlighted the importance of context on intervention implementation, and 17 18 447 suggested that allowing autonomy to deliver the intervention to suit practice and patient contexts 19 20 448 was likely important for effective implementation.[29] 21 22 23 449 Our published paper examining professional perceptions of the key components of the intervention 24 25 450 delivered to practices (education, financial incentives and the informatics tool) found that all 26 27 28 451 components were ‘active’ although at different stages of recruitment and adoption [20]. This paper 29 30 452 adds to that, within a clear definition of the work practices were expected to do with patients, 31 32 453 allowing them the freedom to tailor implementation and develop their own processes to suit their 33

34 454 context was important for effective implementation. The paper also shows that characteristics http://bmjopen.bmj.com/ 35 36 455 associated with implementation varied with context. Having a single motivated individual GP in small 37 38 456 practices appeared to be sufficient for effective implementation, but shared vision and joint working 39 40 41 457 additionally appeared important in larger practices. 42 on October 3, 2021 by guest. Protected copyright. 43 44 458 Strengths and limitations 45 46 459 A strength of the study overall is that we were sufficiently resourced to develop and published a pre- 47 48 460 planned and pre-specified study[18] and carry out a well-resourced and rigorous process evaluation 49 50 461 of a third of all practices included in the trial.[19, 20] A strength of this analysis is the attention to 51 52 462 context, illustrating its influence on intervention implementation including that context is not fixed, 53 54 55 463 for example in relation to variable staffing over time. The sampling method we chose has advantages 56 57 464 and disadvantages. Using run charts showing change in the targeted high-risk prescribing, we chose 58 59 60 19

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml BMJ Open Page 20 of 61 BMJ Open: first published as 10.1136/bmjopen-2016-015281 on 10 March 2017. Downloaded from 1 2 3 465 to sample six practices which did not appear to have implemented the intervention, and four 4 5 466 practices which had. We felt that purposive sampling for heterogeneity in this way would help us 6 7 467 more clearly understand the barriers and facilitators to trial delivery and intervention 8 9 468 implementation. We recognise that sampling from the two ends of the distribution of initial 10 11 12 469 response may limit generalisability to a wider population, but this is mitigated by the fact that we 13 14 470 included one third of practices participating in the trial in the process evaluation. In principle, it also 15 For peer review only 16 471 risks regression to the mean where outliers from both ends of the distribution would be expected to 17 18 472 become more similar to each other because being an outlier at one time point may be due to chance 19 20 473 variation. We do not believe this was the case here, because the four initial responders remained 21 22 474 amongst the most effective at trial end, and the two initial non-responders which delivered the 23 24 25 475 largest reductions in high-risk prescribing by trial end were both judged qualitatively (blind to final 26 27 476 trial results) to have significantly implemented after a delay. 28 29 30 477 Conclusions 31 32 478 The DQIP intervention successfully reduced targeted high-risk prescribing [12] and reductions were 33

34 479 sustained in the 48 weeks after the intervention ceased. Overall, the four case-study practices which http://bmjopen.bmj.com/ 35 36 480 immediately implemented the DQIP intervention had full or nearly full GP attendance at the EOV 37 38 39 481 consistent with collective engagement, rapidly identified and implement a process for delivering the 40 41 482 required work, and had at least one engaged and motivated GP who used a range of strategies to try 42 on October 3, 2021 by guest. Protected copyright. 43 483 to influence their colleagues’ prescribing. In contrast, the six practices in which implementation was 44 45 484 problematic were more variable, with initial implementation failure occurring at different stages of 46 47 485 adoption and often being linked to wider practice context or resources. Context and resources were 48 49 486 however not fixed, with most of the initial implementation ‘failures’ delivering some or all elements 50 51 52 487 of the intervention, leading to successful reductions in targeted high-risk prescribing. In wider 53 54 488 implementation, commissioners should pay careful attention to deploying educational and 55 56 489 persuasive strategies to ensure practices legitimise the work, should allow practices freedom to 57 58 59 60 20

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml Page 21 of 61 BMJ Open BMJ Open: first published as 10.1136/bmjopen-2016-015281 on 10 March 2017. Downloaded from 1 2 3 490 tailor how they implement the work of review, and should seek to offer additional support and 4 5 491 facilitation to practices struggling with short-term resource constraint or other contextual barriers 6 7 492 early in implementation. 8 9 10 493 11 12 13 494 14 15 For peer review only 16 495 List of abbreviations 17 18 496 AG – Aileen Grant 19 20 497 BG – Bruce Guthrie 21 22 498 TD – Tobias Dreischulte 23 24 499 DQIP - Data-driven Quality Improvement in Primary care 25 26 500 EOV - Educational outreach visit 27 501 GP – General Practitioner 28 29 502 NICE - National Institute of Clinical Excellence 30 31 503 NPT – Normalisation Process Theory 32 33 504 NSAIDs - Non-steroidal anti-inflammatory drugs

34 http://bmjopen.bmj.com/ 35 505 TIDieR – Template for intervention description and replication 36 37 506 PINCER - A Pharmacist-led Information technology Intervention for Medication Errors 38 39 507 Competing interests 40 41 508 The authors declare that they have no competing interests. 42 on October 3, 2021 by guest. Protected copyright. 43 509 Consent for publication 44 45 510 All participants gave consent for the publication of anonymised data. 46 47 511 Availability of data and supporting materials section 48 49 512 No observational or interview qualitative data are available to share because the NHS Research 50 51 513 Ethics approval for the study was on the basis of only the research team having access to the raw 52 514 data and no quantitative data are available to share because the permissions for data use from the 53 54 515 NHS organisations which are the data controllers only allow access in a Safe Haven environment by 55 56 516 the research team. 57 58 59 60 21

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml BMJ Open Page 22 of 61 BMJ Open: first published as 10.1136/bmjopen-2016-015281 on 10 March 2017. Downloaded from 1 2 3 517 Funding 4 5 518 The study was supported by a grant (ARPG/07/02) from the Scottish Government Chief Scientist 6 519 Office. The funder had no role in study design, data collection, analysis and interpretation, the 7 8 520 writing of the manuscript or the decision to publish. 9 10 11 521 Authors' contributions 12 522 BG and AG designed the study. AG led the data collection and analysis, supported by BG in analysis 13 14 523 and interpretation. AG wrote the first draft of the manuscript with all authors commenting on 15 For peer review only 16 524 subsequent drafts. 17 18 525 Acknowledgements 19 20 526 We would like to thank all participating practices and the University of Dundee’s Health Informatics 21 527 Centre for data management and anonymisation, the Advisory and Trial Steering Groups, and Debby 22 23 528 O’Farrell who provided administrative support. 24 25 529 Figure 1: DQIP process evaluation framework 26 27 28 530 Figure 2: All trial practices ranked in order of intervention effectiveness, with case study practices 29 531 identified*FOOTNOTE * Practices marked in green were sampled because they were judged to have initially rapidly 30 532 reduced the targeted prescribing; practices marked in red were sample because they were judged to have not initially 31 533 reduced the targeted prescribing 32 33

34 http://bmjopen.bmj.com/ 35 36 37 38 39 40 41 42 on October 3, 2021 by guest. Protected copyright. 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 22

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml BMJ Open: first published as 10.1136/bmjopen-2016-015281 on 10 March 2017. Downloaded from most most

implementation by by implementation all patients in set 2 hours/month. hours/month. 2 inset patients all Initially agreed to divide the work, with rapid implementation by all GPs all GPs by implementation rapid with work, the divide to agreed Initially Initially agreed to divide the work, but actually rapid rapid actually but work, the divide to agreed Initially reviewing. the out carrying Initially agreed to divide the work and rapidly delivered by all GPs initially initially all GPs by delivered rapidly and work the divide to agreed Initially reviewing. the all doing GP one Did not agree process at EOV, but rapid implementation of one GP one GP of implementation rapid but EOV, at process agree not Did reviewing. maintained GP one done, reviews of bulk initial Once reviewing. Initially agreed to divide the work, but did not implement; one GP one implement; not did but work, the divide to agreed Initially allpatients. reviewing systematically Initially agreed that one GP would review all patients and flag notes for for notes flag and patients all review would GP one that agreed Initially delay. a after reviewed enthusiastically and systematically Initially agreed that one GP would review review would GP one that agreed Initially the on acting GPs other and consulting patient on relied so seen, next when flag Initially agreed that one GP would review, but actually divided the work. work. the divided actually but review, would GP one that agreed Initially reduced further communication to GP GP poor & inadequate, was This impact Initially agreed to divide the work between GPs, but failed to implement implement to failed but GPs, between work the divide to agreed Initially reviewing maintain not they could meant changes Staff Failure to legitimise and no process to implement agreed, but the agreed, but implement to process no and legitimise to Failure work of clinical prioritisation understaffing/ of because The process for delivering the intervention to patients, both planned by the the by planned both patients, to intervention the for delivering process The practice in clinical change wassome there said GP engaged practice and actual (based on interview and observational data). observational and interview on (based actual and practice -

c BMJ Open

http://bmjopen.bmj.com/ 3.7 7.1 4.2 3.7 5.0 3.4 3.2 7.0 2.5 6.6 Overall high Overall risk prescribing prescribing risk at rate baseline

ng Reducing Reducing Large Reducing Small Reduci Large Not reducing Not Small Not reducing Not Small Not reducing Not Small Not reducing Not Large Not reducing Not Small Not reducing Not Large Sampling (initial (initial Sampling Large change in in change and prescribing size) on October 3, 2021 by guest. Protected copyright. size size - For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml

For peer review only 6,000 FTE 4 3,500 FTE 2.7 5,500 FTE 4.3 3000 FTE 2 3,000 2FTE 2,500 FTE 1.9 9,000 FTE 5 3000 FTE 1.9 6,500 FTE 3.5 Approx list Approx 10,000 FTE 6.5 and full time time full and (FTE) equivalent GPs

b

5 1 8 6 4 7 3 9 10 Randomised Randomised 2 group

a

a. Ordered a. from top bottom in to ofterms the practices judged from qualitative been analysis have to the least (top) (bottom)most to successful implementers groupb. Practice in ofterms whenstarted the intervention (1=first start, to group 10=last togroup start) Mean ofpractice c. rate high-risk prescribing in the two years beforestarting the intervention Taransay Monach Hirta Scalpay Lingay Gighay Mingulay Hellisay Boreray Practice Orosay Table 1: Practice characteristics including planned and actual process for delivering care to patients patients to care fordelivering process andactual planned including characteristics Practice 1: Table 23 Page 23 of 61 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 BMJ Open: first published as 10.1136/bmjopen-2016-015281 on 10 March 2017. Downloaded from

e Page 24 of 61 75 75 59 59 67 67 56 56 77 77 53 53 28 28 6 -24 -24 19 19 Effectiveness % reductionin high- risk prescribing

c 43 43 68 68 50 50 43 43 47 47 44 44 7 48 48 0.7 0 Maintenance % ofpatients eligible with review a recorded in the final 24weeks of the intervention

c d 45 45 33 33 5 5 19 19 14 14 38 38 32 32 29 29 Delivery toDelivery patients % ofpatients eligible with change with in change prescribing recorded

b 90 90 3 2 92 92 64 64 95 95 83 83 89 89 78 78 Reach % of patients eligible 2 0 with review a recorded at point any during the intervention period 62 62 BMJ Open

http://bmjopen.bmj.com/ a on October 3, 2021 by guest. Protected copyright. For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml For peer review only GPs byco-ordinatedand an administrative memberof staff. dividedamong GPs all then oneand GP reviewing.maintained process but reviewed EOV one GP at patients. all divide divide the work but problems with toaccess the informaticstool led to maintenance. The GPs dealt with immediately the bulk initial butstaff changes meantstruggledthey to consistently reviewing.maintain collectively legitimise the intervention.This was too work muchfor one individual so noprocess for was implementation agreed. Overall qualitative assessmentof implementation lost was DQIP motivation. implemented fullyoneby delay. GP a after to but divide the work one GP actuallythe reviewing. did all one would patientsGP review all andnotes flag so forseen, when next relied on patientconsulting and on GPs other acting the flag. month allocated to but deliver review, this was inadequate addressto thenumbers identified communication with limited with other GPs. work, butwork, to failed implement changesany because of understaffing and prioritisationofa work.clinical ractice c. Eligible defined as patients with one or more high-risk prescriptions issued during the second half (24 weeks) of the intervention period period intervention the of weeks) (24 half second the during issued high-risk prescriptions more or one with as patients defined c.Eligible implementers successful (bottom) most to (top) least the been have analysis to qualitative from judged practices the termsof in bottom to top from Ordered a. period weekintervention 48 the during issued prescriptions high-risk more or one with as patients defined Eligible b. prescribing in a change have to not assumed were recorded review a without Patients d. in increase an indicate numbers (negative started intervention the 48 before weeks the to compared intervention the weeks24 of final the in practice each in prescribing high-risk change in relative as the Defined e. prescribing) high-risk Taransay was DQIP fully implemented from was the start. DQIPdelivered by all Monach was DQIP fully implemented from the Thestart. practice initially agreed Hirta was DQIP fully implemented from the Thestart. initial bulk of was work Scalpay Scalpay was DQIP fully implemented from the start. The practice did not agree Lingay intervention DQIP was notadopted. initially GPsinitially to agreed Gighay was DQIP adopted with to delivery limited patients. thatagreed They Mingulay was DQIP adopted with initial reasonable reach.had GP One hours2 per Hellisay was DQIP adopted, delivered initially patientsto but with low Boreray intervention DQIP was not Initially the GPsadopted. agreed tothe share Orosay Orosay intervention DQIP was notbecause there wasadopted failurea to Table 2: Comparison of overall qualitative assessment of implementation and quantitative measures or reach, delivery, maintenance and effectiveness effectiveness and maintenance delivery, reach, or measures quantitative and implementation of assessment qualitative overall of Comparison 2: Table P 24 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 BMJ Open: first published as 10.1136/bmjopen-2016-015281 on 10 March 2017. Downloaded from

changed? changed? adjustments effectiveness? effectiveness? Reconfiguration Reconfiguration Systematisation Systematisation the conclusions? conclusions? the making necessary necessary making judgements about about judgements Individual appraisal appraisal Individual Communal appraisal appraisal Communal Appraisal may lead to to lead may Appraisal How do practices make make practices do How Unsystematic & informal informal & Unsystematic monitoring and appraisal appraisal and monitoring changes – what have they they have –what changes REFLECTIVE MONITORING MONITORING REFLECTIVE Reflecting on progress and and progress on Reflecting Regular & organised formal formal organised & Regular appraisal of DQIP. What are What DQIP. of appraisal

BMJ Open work around it? around How is DQIP is How http://bmjopen.bmj.com/ operationalised? operationalised? practice context. context. practice Skill set workability workability set Skill COLLECTIVE ACTION COLLECTIVE Relational integration integration Relational Contextual integration integration Contextual mediated by the people people the by mediated Interactional workability workability Interactional Roles and responsibilities responsibilities and Roles Organising and doing the doing and Organising Incorporation of DQIP into into DQIP of Incorporation How is DQIP understood & & understood DQIP is How How is the work allocated? allocated? thework is How

on October 3, 2021 by guest. Protected copyright. part part Initiation Initiation required? required? Activation Activation Enrolment Enrolment about DQIP? DQIP? about Legitimation Legitimation For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml What resources are resources What Agency – capacity of of –capacity Agency and weigh up options. up options. weigh and What process have they have process What of individuals & groups & individuals of Persuading others to take take to others Persuading how or what do they value value they do what or how Buying into the DQIP work: DQIP work: the into Buying decided on to do the work? work? the do to on decided COGNITIVE PARTICIPATION COGNITIVE

Enrolment and engagement engagement and Enrolment For peer review only individuals to make decisions decisions make to individuals

other work? work? other COHERENCE Internalisation Internalisation value to DQIP? value Differentiation Differentiation improvement work? work? improvement How do participants do participants How Individual specification specification Individual understand & attribute & attribute understand understanding of DQIP? DQIP? of understanding Communal specification specification Communal other prescribing quality quality prescribing other they relate DQIP work to? work DQIP relate they What past experiences do do experiences past What How does DQIP differ from differ DQIP does How How does DQIP cohere with with cohere DQIP does How Does the team have a shared shared a team have the Does trial. the DQIP for interpreted constructs theory process Normalisation 3: Table 25 Page 25 of 61 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 BMJ Open Page 26 of 61 BMJ Open: first published as 10.1136/bmjopen-2016-015281 on 10 March 2017. Downloaded from 1 2 3 4 5 6 REFERENCES 7 8 1. Hakkarainen KM, Hedna K, Petzold M, Hagg S: Percentage of patients with preventable adverse 9 drug reactions and preventability of adverse drug reactions - a meta analysis. PLOS One 2012, 10 11 7(3):e33236. 12 13 2. Howard R, Avery A, Bissell P: Causes of preventable drug-related hospital admissions: a 14 qualitative study. Quality and Safety in Health Care 2008, 17(2):109-116. 15 For peer review only 16 3. National Institute for Health and Care Excellence (NICE): Costing statement: Medicines 17 optimisation - Implementing the NICE guideline on medicines information (NG5). Available at 18 https://www.nice.org.uk/guidance/ng5/resources/costing-statement-6916717 [accessed 19 11/05/2016]. 2015. 20 21 22 4. Institute for Health Care Informatics, IMS: Responsible Use of Medicines 23 Report. http://www.imshealth.com/en/thought-leadership/ims-institute/reports/responsible- 24 use-of-medicines-report. 2012. 25 26 5. Howard RL, Avery A, Slavenburg S, Royal S, Pipe G, Lucassen P, Pirohamed M: Which drugs cause 27 preventable admissions to hospital? A systematic review. British Journal of Clinical Pharmacology 28 2006, 63:136-147. 29 30 6. Budnitz DS, Lovegrove MC, Shehab N, Richards CL: Emergency Hospitalisation for Adverse Drug 31 Events in Older Americans. N Engl J Med 2011, 365(21):2002-2012. 32 33 7. Leendertse AJ, Egberts ACG, Stoker LJ, van den Bent PM: Frequency of and risk factors for

34 http://bmjopen.bmj.com/ 35 preventable medication-related hospital admissions in the Netherlands. Arch Intern Med 2008, 36 168(17):1890-1896. 37 38 8. Dreischulte T, Grant A, Donnan P, McCowan C, Davey P, Petrie D, Treweek S, Guthrie B: A cluster 39 randomised stepped wedge trial to evaluate the effectiveness of a multifaceted information 40 technology-bsed intervention in reducing high-risk prescribing of non-steroidal anti-inflammatory 41 drugs and antiplatelets in primary care: the DQIP study protocol. Implementation Science 2012, 42 7:24. on October 3, 2021 by guest. Protected copyright. 43 44 9. Grant AM, Guthrie B, Dreischulte T: Developing a complex intervention to improve prescribing 45 46 safety in primary care: mixed methods feasibility and optimisation pilot study. BMJ Open 2014, 47 4(1). 48 49 10. Guthrie B, McCowan C, Davey P, Simpson CR, Dreischulte T, Barnett K: High risk prescribing in 50 primary care patients particularly vulnerable to adverse drug events: cross sectional population 51 database analysis in Scottish general practice. British Medical Journal 2011, 342. 52 53 11. Brown C, Lilford R: The stepped wedge trial design: a systematic review. BMC Medical Research 54 Methodology 2006, 6(1):54. 55 56 57 12. Dreischulte T, Donnan P, Grant A, Hapca A, McCowan C, Guthrie B: Safer prescribing - A Trial of 58 Education, Informatics & Financial Incentives. New England Journal of Medicine 2016, 374:1053-64. 59 60 26

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml Page 27 of 61 BMJ Open BMJ Open: first published as 10.1136/bmjopen-2016-015281 on 10 March 2017. Downloaded from 1 2 3 13. Medical Research Council: Developing and evaluating complex interventions: new guidance. 4 2008. 5 6 14. Medical Research Council: Process evaluation of complex interventions: UK Medical Research 7 Council (MRC) guidance. 2015. 8 9 15. Grant A, Treweek S, Dreischulte T, Foy R, Guthrie B: Process evaluations for cluster-randomised 10 11 trials of complex interventions: a proposed framework for design and reporting. Trials 2013, 12 14(1):15. 13 14 16. Stake R: The Art of Case Study Research: Thousand Oaks: Sage Publications; 1995. 15 For peer review only 16 17. Crowe S, Cresswell K, Robertson A, Huby G, Avery A, Sheikh A: The case study approach. BMC 17 Medical Research Methodology 2011, 11(1):100. 18 19 18. Grant A, Dreischulte T, Treweek S, Guthrie B: Study protocol of a mixed-methods evaluation of a 20 cluster randomised trial to improve the safety of NSAID and antiplatelet prescribing: data-driven 21 22 quality improvement in primary care. Trials 2012, 13:154. 23 24 19. Dreischulte T, Grant A, Guthrie B: Process evaluation of the Data-driven Quality Improvement 25 in Primary Care (DQIP) trial: Quantitative examination of representativeness of trial participants 26 and heterogeneity of impact. Submitted. Implementation Science 2016. 27 28 20. Grant A, Dreischulte T, Guthrie B: Process evaluation of the Data-driven Quality Improvement 29 in Primary Care (DQIP) trial: active and less active ingredients of a multi-component complex 30 intervention to reduce high-risk primary care prescribing. Implementation Science 2017, 12(4). 31 32 21. May C, Finch T: Implementing, Embedding, and Integrating Practices: An Outline of 33 Normalisation Process Theory. Sociology 2009, 43:535.

34 http://bmjopen.bmj.com/ 35 36 22. Glaser B, Strauss A: The discovery of grounded theory: strategies for qualitative research: 37 Chicago: Aldine Publishing Co; 1967. 38 39 23. Ritchie J, Spencer L, O'Connor W: Carrying out Qualitative Analysis. In Qualitative Research 40 Practice, A guide for Social Science Students and Researchers. Edited by Ritchie J, Lewis J. London: 41 Sage Publications Ltd; 2003. 42 on October 3, 2021 by guest. Protected copyright. 43 24. Avery AJ, Rodgers S, Cantrill JA, Armstrong S, Cresswell K, Eden M, Elliott RA, Howard R, Kendrick 44 45 D, Morris CJ, Prescott RJ, Swanwick G, Franklin M, Putman K, Boyd M, Sheikh A: A pharmacist-led 46 information technology intervention for medication errors (PINCER): a multicentre, cluster 47 randomised, controlled trial and cost-effectiveness analysis. The Lancet 2012. 48 49 25. Cresswell K, Sadler S, Rodgers S, Avery A, Cantrill J, Murray S, Sheikh A: An embedded 50 longitudinal multi-faceted qualitative evaluation of a complex cluster randomized controlled trial 51 aiming to reduce clinically important errors in medicines management in general practice. Trials 52 2012, 13(1):78. 53 54 26. Kennedy A, Rogers A, Chew-Graham C, Blakeman T, Bowen R, Gardner C, Lee V, Morris R, 55 Protheroe J: Implementation of a self-management support approach (WISE) across a health 56 57 system: a process evaluation explaining what did and did not work for organisations, clinicians and 58 patients. Implementation Science 2014, 9(1):129. 59 60 27

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml BMJ Open Page 28 of 61 BMJ Open: first published as 10.1136/bmjopen-2016-015281 on 10 March 2017. Downloaded from 1 2 3 27. Berendsen BAJ, Kremers SPJ, Savelberg HHCM, Schaper NC, Hendriks MRC: The implementation 4 and sustainability of a combined lifestyle intervention in primary care: mixed method process 5 evaluation. BMC Family Practice 2015, 16(1):37. 6 7 28. Moore GF, Moore L, Murphy S: Facilitating adherence to physical activity: exercise 8 professionals' experiences of the National Exercise Referral Scheme in Wales. a qualitative study. 9 BMC Public Health 2011, 11(1):935. 10 11 12 29. Murdoch J, Varley A, Fletcher E, Britten N, Price L, Calitri R, Green C, Lattimer V, Richards SH, 13 Richards DA, Salisbury C, Taylor RS, Campbell JL: Implementing telephone triage in general practice: 14 a process evaluation of a cluster randomised controlled trial. BMC Family Practice 2015, 16(1):47. 15 For peer review only 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33

34 http://bmjopen.bmj.com/ 35 36 37 38 39 40 41 42 on October 3, 2021 by guest. Protected copyright. 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 28

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml Page 29 of 61 BMJ Open BMJ Open: first published as 10.1136/bmjopen-2016-015281 on 10 March 2017. Downloaded from 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 For peer review only 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33

34 http://bmjopen.bmj.com/ 35 36 37 38 39 40 41 42 on October 3, 2021 by guest. Protected copyright. 43 44 45 46 47

48 209x296mm (300 x 300 DPI) 49 50 51 52 53 54 55 56 57 58 59 60 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml BMJ Open Page 30 of 61 BMJ Open: first published as 10.1136/bmjopen-2016-015281 on 10 March 2017. Downloaded from 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 For peer review only 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33

34 http://bmjopen.bmj.com/ 35 36 37 38 39 40 41 42 on October 3, 2021 by guest. Protected copyright. 43 44 45 46 47

48 209x296mm (300 x 300 DPI) 49 50 51 52 53 54 55 56 57 58 59 60 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml Page 31 of 61 BMJ Open

1 2 3 GP topic guide (1). 4 5 BMJ Open: first published as 10.1136/bmjopen-2016-015281 on 10 March 2017. Downloaded from 6 Interview one topic guide; Adoption/initial delivery questions 7 8 What do you think of the intervention? 9 10 Are these views shared across the practice? 11 12 How does DQIP compare to other prescribing improvement work? 13 14 How does DQIP differ to other prescribing improvement work? 15 16 Have they anchored their experience of DQIP to other work they have done? (Internalisation) 17 For peer review only 18 19 What attracted the practice to taking part in the DQIP trial? 20 21 Can you tell me the process by which the practice decided to take part? 22 23 Was the decision to take part mostly driven by one or two enthusiasts, or was it a more communal 24 decision? 25 26 27 Did you or another partner inform and persuade the other GPs in the practice to take part? 28 29 Did the practice have any concerns about the data extraction from the practice system? Look for 30 them talk about the benefits of patient level data, targeted medication review. 31 32 How did the practice decide how to go about organising the DQIP work? 33 34 Can you tell me how the practice is using the tool? What is the practice system or process for using 35 36 the tool? 37 38 Is this how you started using the tool, or has this process evolved slightly? http://bmjopen.bmj.com/ 39 40 How has the practice incorporated the intervention into existing systems of work? 41 42 Were there any barriers to implementation? How have these been overcome? 43 44 What practical steps or resources were needed for implementation? 45 46 47 How often are you personally using the tool? on October 3, 2021 by guest. Protected copyright. 48 49 How often do you do the reviewing? 50 51 Have other members of staff used the tool? What is their role and how did they use it? Did the 52 whole practice buy into the tool? 53 54 55 Have you had to call many patients into the practice? 56 57 Have you told patients you are taking part in a trial? 58 59 How have you explained the medication changes to patients? 60 Have you had any communication with the CHP or HB about DQIP?

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml BMJ Open Page 32 of 61

1 2 3 How have patients responded to the intervention? 4 5 BMJ Open: first published as 10.1136/bmjopen-2016-015281 on 10 March 2017. Downloaded from 6 (Have patients whose NSAIDs were stopped come back? How commonly does this happen?) 7 8 Have you seen any impact on patients? On patient outcomes? 9 10 Has your prescribing changed as a result of DQIP? 11 12 Can potentially show the GPs their data, or ask them to discuss the pattern of data they see. 13 14 Can you please tell me about a situation where you have used the tool and/or intervention? (Get 15 16 them to give at least one scenario, identify purpose, try and get a range of scenarios if possible) 17 For peer review only 18 What did you think of the initial practice visit? 19 20 Was it helpful? How could it have been more helpful? 21 22 What did you think of the educational material? 23 24 Did the payment structure make any difference to your decision to take part? 25 26 27 Has the payment per review completed influenced the way in which you organised or delivered the 28 reviews? 29 30 What do you like or not like about the tool? 31 32 Have you experienced any problems? 33 34 How have these been overcome? 35 36 37 Have you modified any of the work associated with DQIP? 38 http://bmjopen.bmj.com/ 39 Where there any contextual factors which impeded implementation of the tool? 40 41 Where there any contextual factors which facilitated implementation of the tool? 42 43 44 45 46 47 on October 3, 2021 by guest. Protected copyright. 48 49 50 51 52 53 54 55 56 57 58 59 60

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml Page 33 of 61 BMJ Open

1 2 3 GP topic guide (2). 4 5 BMJ Open: first published as 10.1136/bmjopen-2016-015281 on 10 March 2017. Downloaded from 6 Interview two topic guide; delivery/maintenance/sustainability questions 7 8 What did you think of the intervention as a whole; education, informatics and financial incentive? 9 Were these views shared across the practice? 10 11 Can you tell me how the practice used the tool? What was the practice system or process for using 12 13 the tool? Did this change over time? 14 15 What practical steps or resources were needed to maintain use of the intervention? 16 17 How often did youFor personally peer use the tool? Didreview this change over time?only 18 19 Did your experience of the tool/intervention change over time? 20 21 Were there other members of staff using the tool? Did they maintain use? 22 23 24 How did you communicate as a practice about the intervention/tool? 25 26 Did you collectively look at the run charts to monitor prescribing? 27 28 Have you ever made collective judgements about the effectiveness and utility of DQIP? 29 30 Look for formal and informal judgements 31 32 Have you modified the DQIP work or modified any practice processes or systems as a result of DQIP? 33 34 What did you think of the educational material? Did you use it regularly? 35 36 37 Can you please give me an example of when you used the educational material? 38 http://bmjopen.bmj.com/ 39 Can you please tell me about a positive situation using the tool? 40 41 Can you please tell me about a less desirable situation using the tool? 42 43 Is use of the tool and/or intervention sustainable in the long-term? 44 45 Where there any contextual factors which have impacted on use of the tool? 46 47 on October 3, 2021 by guest. Protected copyright. Where there any contextual factors which facilitated use of the tool? 48 49 50 How do you think the tool has impacted on patient outcomes? 51 52 Are there any resources which could have been put in place to improve patient outcomes? 53 54 55 56 57 58 59 60

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml BMJ Open: first published as 10.1136/bmjopen-2016-015281 on 10 March 2017. Downloaded from BMJ Open Page 34 of 61 The new england journal of medicine

1 2 3 Special Article 4 5 6 7 Safer Prescribing — A Trial of Education, 8 Informatics, and Financial Incentives 9 10 Tobias Dreischulte, Ph.D., Peter Donnan, Ph.D., Aileen Grant, Ph.D., 11 Adrian Hapca, Ph.D., Colin McCowan, Ph.D., 12 and Bruce Guthrie, M.B., B.Chir., Ph.D.​​ 13 14 For peer review only 15 ABSTRACT 16 17 18 BACKGROUND 19 High-risk prescribing and preventable drug-related complications are common in From the Medicines Governance Unit, primary care. We evaluated whether the rates of high-risk prescribing by primary NHS Tayside (T.D.), and the Population 20 Health Sciences Division, University of 21 care clinicians and the related clinical outcomes would be reduced by a complex Dundee (P.D., A.H., B.G.), Dundee, the 22 intervention. School of Health Sciences, University of Stirling, Stirling (A.G.), and the Robert- 23 METHODS son Centre for Biostatistics, University 24 In this cluster-randomized, stepped-wedge trial conducted in Tayside, Scotland, we of Glasgow, Glasgow (C.M.) — all in Scotland. Address reprint requests to 25 randomly assigned participating primary care practices to various start dates for a 26 Dr. Guthrie at the University of Dundee, 48-week intervention comprising professional education, informatics to facilitate re- Mackenzie Bldg., Kirsty Semple Way, 27 view, and financial incentives for practices to review patients’ charts to assess appro- Dundee DD2 4BF, Scotland, or at 28 ­b​.­guthrie@​­dundee​.­ac​.­uk. priateness. The primary outcome was patient-level exposure to any of nine measures 29 of high-risk prescribing of nonsteroidal antiinflammatory drugs (NSAIDs) or selected N Engl J Med 2016;374:1053-64. 30 DOI: 10.1056/NEJMsa1508955 antiplatelet agents (e.g., NSAID prescription in a patient with chronic kidney disease Copyright © 2016 Massachusetts Medical Society. 31 http://bmjopen.bmj.com/ or coprescription of an NSAID and an oral anticoagulant without gastroprotection). 32 Prespecified secondary outcomes included the incidence of related hospital admis- 33 34 sions. Analyses were performed according to the intention-to-treat principle, with the 35 use of mixed-effect models to account for clustering in the data. 36 RESULTS 37 A total of 34 practices underwent randomization, 33 of which completed the study. 38 Data were analyzed for 33,334 patients at risk at one or more points in the preinterven-

39 tion period and for 33,060 at risk at one or more points in the intervention period. on October 3, 2021 by guest. Protected copyright. 40 Targeted high-risk prescribing was significantly reduced, from a rate of 3.7% (1102 of 41 29,537 patients at risk) immediately before the intervention to 2.2% (674 of 30,187) at 42 the end of the intervention (adjusted odds ratio, 0.63; 95% confidence interval [CI], 43 0.57 to 0.68; P<0.001). The rate of hospital admissions for gastrointestinal ulcer or 44 bleeding was significantly reduced from the preintervention period to the intervention 45 period (from 55.7 to 37.0 admissions per 10,000 person-years; rate ratio, 0.66; 95% CI, 46 0.51 to 0.86; P = 0.002), as was the rate of admissions for heart failure (from 707.7 to 47 513.5 admissions per 10,000 person-years; rate ratio, 0.73; 95% CI, 0.56 to 0.95; 48 P = 0.02), but admissions for acute kidney injury were not (101.9 and 86.0 admissions 49 per 10,000 person-years, respectively; rate ratio, 0.84; 95% CI, 0.68 to 1.09; P = 0.19). 50 51 CONCLUSIONS 52 A complex intervention combining professional education, informatics, and financial 53 incentives reduced the rate of high-risk prescribing of antiplatelet medications and 54 NSAIDs and may have improved clinical outcomes. (Funded by the Scottish Govern- 55 ment Chief Scientist Office; ClinicalTrials.gov number, NCT01425502.) 56 57 n engl j med 374;11 nejm.org March 17, 2016 1053 58 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml 59 60 BMJ Open: first published as 10.1136/bmjopen-2016-015281 on 10 March 2017. Downloaded from Page 35 of 61 BMJ Open The new england journal of medicine

1 2 igh-risk prescribing and prevent- in the Supplementary Appendix, available with 3 able drug-related complications in pri- the full text of this article at NEJM.org), and 4 mary care are major concerns for health structured financial incentives to review these H 1-7 5 care systems internationally. Up to 4% of patients. 6 emergency hospital admissions are caused by We developed and refined this intervention in 7 preventable adverse drug events,8-10 and in the four pilot practices in which participating physi- 8 United States, the cost of avoidable drug-related cians judged that approximately 40% of the tar- 9 hospital admissions, emergency department at- geted high-risk prescribing involving NSAIDs 10 tendances, and outpatient visits was estimated and antiplatelet agents required corrective ac- 11 at $19.6 billion in 2013.11 The majority of drug- tion.21 We evaluated the intervention in a cluster- 12 related emergency admissions are caused by com- randomized trial that examined the effect of the 13 monly prescribedFor drugs, peer with substantial review contri- intervention only on high-risk prescribing of NSAIDs 14 butions from nonsteroidal antiinflammatory and antiplatelet medications and related emer- 15 drugs (NSAIDs) and antiplatelet medications gency hospital admissions. 16 because of gastrointestinal, cardiovascular, and 17 4,5,7,8 renal adverse drug events. Despite routine Methods 18 public reporting of a number of indicators of 19 high-risk prescribing, variation among providers12 Study Design 20 and among Medicare hospital-referral regions13 In brief, the DQIP trial was a pragmatic, cluster- 21 is large and reductions in high-risk prescribing randomized trial that used a stepped-wedge de- 22 are minimal or slow.14,15 sign, in which all the participating practices re- 23 Decisions about prescribing often involve bal- ceived the DQIP intervention but were randomly 24 ancing benefits and risks as well as the prefer- assigned to 1 of 10 designated start dates be- 25 ences of the patient, and high-risk prescribing is tween October 30, 2011, and September 2, 2012. 26 therefore sometimes appropriate, since benefits Each practice received the intervention for a 48- 27 may be judged to outweigh the risk of harm to week period, with continued data collection for 28 12,16-19 29 a person. Nevertheless, the observed high 48 weeks after the active intervention ceased. 30 prevalence of and large variation in high-risk Figure S1 in the Supplementary Appendix shows prescribing patterns are consistent with the the stepped-wedge design of the study. The 31 http://bmjopen.bmj.com/ 32 hypothesis that the safety of prescribing in pri- methods of the study are described in detail in 13-17 33 mary care can be improved, and at a mini- the Supplementary Appendix. The study was ap- 34 mum, regular review to assess appropriateness proved by the National Health Service (NHS) 35 is required. Scotland Fife and Forth Valley research ethics 36 However, persuading primary care practices committee. The study was conducted and re- 37 to allocate scarce staff resources to improving ported with fidelity to the study protocol, which 38 the safety of prescribing is challenging, par- is available at NEJM.org. The authors vouch for ticularly when practices are independent, phy- the accuracy and completeness of the data pre-

39 on October 3, 2021 by guest. Protected copyright. 40 sician-owned small businesses that provide pri- sented. 41 mary medical care under contract to an external 42 payer, as is the case in many countries, includ- Participants 43 ing in the United Kingdom, where this trial was Physician-owned practices with contracts to pro- 44 performed.18 Therefore, the Data-Driven Quality vide NHS primary medical care in the Tayside 45 Improvement in Primary Care (DQIP) program region of Scotland were eligible to participate if 46 systematically developed a multifaceted inter- they used an electronic medical record (EMR) 47 vention for the improvement of prescribing system that was compatible with data extraction 48 safety in primary care practice.12,19-21 This inter- to the DQIP informatics tool.19 In the United 49 vention comprised professional education about Kingdom, patients are registered with one pri- 50 the risks of NSAIDs and antiplatelet medica- mary care practice that is responsible for all 51 tions, access to a Web-based tool to identify prescription of drugs in the community, includ- 52 patients at the highest risk for adverse drug ing any prescribing that is initiated on the rec- 53 events related to NSAIDs and antiplatelet agents ommendation of a specialist. In each practice, 54 (aspirin or clopidogrel, as defined in Table S1 patients were included at each data-measure- 55 56 57 1054 n engl j med 374;11 nejm.org March 17, 2016 58 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml 59 60 BMJ Open: first published as 10.1136/bmjopen-2016-015281 on 10 March 2017. Downloaded from BMJ Open Page 36 of 61 Safer Prescribing

1 2 ment point if they were alive, were permanently these drugs (Table S1 in the Supplementary Ap- 3 registered with the practice on that date, and pendix). The individual measures were related to 4 had one or more risk factors that made them three types of adverse drug events: gastrointesti- 5 particularly vulnerable to adverse drug events nal events (six measures; e.g., aspirin or clopido- 6 related to NSAIDs or antiplatelet agents. grel prescription for a patient taking an oral 7 anticoagulant without coprescription of a gas- 8 Intervention troprotective drug), renal events (two measures; 9 The development and detailed design of the inter- e.g., NSAID prescription for a patient with 10 vention have been described in previous reports19-21 chronic kidney disease), and heart failure (NSAID 11 and are summarized in the Supplementary Ap- prescription for a patient with heart failure).20,22-27 12 pendix. The intervention comprised three com- The composite primary outcome was defined as 13 ponents. First, practicesFor received peer professional review the percentage of patients only with any risk factor 14 education, with each practice receiving a 1-hour (e.g., taking an anticoagulant, having chronic 15 educational outreach visit by a pharmacist at the kidney disease, or having heart failure) who 16 start of the intervention (see Section 4 in the were currently receiving an antiplatelet agent, an 17 Supplementary Appendix for the presentation NSAID, or both in a way that was defined as 18 used), additional written material, and news- “high risk” by one or more individual measures. 19 letters (sent every 8 weeks) tailored to the prog- Prespecified secondary prescribing outcomes in- 20 ress of the practice. cluded ongoing (prescribed within the previous 21 Second, practices received financial incen- year) and new (not prescribed within the previ- 22 tives in the form of an initial fixed payment of ous year) high-risk prescribing and the rates of 23 £350 ($600 U.S.) and a payment of £15 ($25 U.S.) the nine prescribing outcome measures individ- 24 for every patient for whom the targeted high-risk ually. In each practice, all prescribing indicators 25 prescribing was reviewed during the intervention were measured every 8 weeks before and after 26 period (with only one claim allowed for each the practice started the intervention, with the 27 patient). The average expected payment per full- patients considered to be currently receiving a 28 29 time physician was approximately £550 ($910 U.S.), high-risk prescription if it had been issued at any 30 or approximately 0.6% of the average physician point in the preceding 8 weeks. income. Other prespecified secondary outcomes were 31 http://bmjopen.bmj.com/ 32 Third, an informatics tool extracted data from emergency hospital admissions for gastrointesti- 33 the EMR system of each practice, identified in- nal bleeding, acute kidney injury, or heart fail- 34 dividual patients who needed review, facilitated ure; we considered admissions that were pre- 35 reviews of patients by graphically displaying ceded by targeted high-risk prescribing as well 36 relevant drug histories, and provided weekly as all such admissions in patients with risk fac- 37 updates on the rates of high-risk prescribing and tors for adverse drug events related to NSAIDs 38 progress in reviewing. Physicians accessed the and antiplatelet agents, regardless of high-risk tool by means of a password-protected Web- prescribing. We also conducted a post hoc

39 on October 3, 2021 by guest. Protected copyright. 40 based portal. Identified patients were flagged by analysis involving these same patients to exam- 41 the tool as needing review. Physicians could ine changes in the rates of unrelated hospital 42 clear the flag by recording an explicit decision admissions for hip fracture, cancer, and surgical 43 that the prescribing was appropriate, by stop- emergencies (appendicitis, cholecystitis, or pan- 44 ping prescription of the offending drugs, or by creatitis) and in the rates of unrelated ambula- 45 adding a gastroprotective drug (proton-pump tory care sensitive admissions (defined as angi-

46 inhibitor or histamine2-receptor antagonist) for na, asthma, cellulitis, convulsions and epilepsy, 47 measures targeting the risk of gastrointestinal chronic obstructive pulmonary disease, dehydra- 48 bleeding. tion and gastroenteritis, dental conditions, dia- 49 betes complications, infection of the ear, nose, 50 Outcome Measures or throat, gangrene, hypertension, influenza and 51 The primary outcome measure was a composite pneumonia, nutritional deficiency, other vaccine- 52 of nine measures of high-risk prescribing of preventable disease, pelvic inflammatory disease, 53 NSAIDs and antiplatelet agents in people with or pyelonephritis) (Table S2 in the Supplemen- 54 risk factors for adverse drug events related to tary Appendix).28 All the outcomes were mea- 55 56 57 n engl j med 374;11 nejm.org March 17, 2016 1055 58 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml 59 60 BMJ Open: first published as 10.1136/bmjopen-2016-015281 on 10 March 2017. Downloaded from Page 37 of 61 BMJ Open The new england journal of medicine

1 2 sured with the use of routine data that were ex- Finally, in a post hoc analysis, we examined 3 tracted from practice EMRs (for prescribing) or whether the effect of the DQIP intervention was 4 from linked administrative data sets (for hospi- sustained after the end of the intervention. We 5 tal admissions). compared the rate of high-risk prescribing in the 6 last 24 weeks of the intervention period with 7 Randomization and Start-Date Concealment that in the last 24 weeks of the postintervention 8 Practices were assigned to 1 of 10 start dates by period using the same model as that used for the 9 an independent statistician who was unaware of primary outcome (Fig. S1 in the Supplementary 10 the identity of the practices, with randomization Appendix). 11 stratified by thirds of the number of registered 12 patients. Concealment of the start-date assign- Results 13 ment fromFor practices andpeer from the research review team only 14 was not possible, but prescribing measures were Participating Practices 15 calculated at a remote site by the data provider Of 66 Tayside practices we approached, 34 (52%) 16 independently of the research team before the agreed to participate, but 1 withdrew before its 17 data were transferred to the research team, with randomized start date. The intervention was 18 the use of the same algorithms that were used therefore implemented in 33 practices, with a 19 to provide feedback to practices. pooled list size of 202,262 patients on the date 20 that the practices started the intervention. A total 21 Statistical Analysis of 33,334 patients with risk factors for adverse 22 On the basis of data from the pilot practices19 and drug events related to NSAIDs and antiplatelet 23 the Pharmacist-led Information Technology In- agents were included in the analysis at one or 24 tervention for Medication Errors (PINCER) trial,29 more time points during the preintervention 25 we estimated that the study would have 83% period, and 33,060 patients were included in the 26 power to detect a 25% reduction in the primary intervention period (Fig. S2 in the Supplemen- 27 outcome, at an alpha level of 0.05, in 10 prac- tary Appendix). There was a significant but 28 19,30 29 tices randomly assigned to 10 start dates. clinically small rising trend in the rate of high- 30 We report the prespecified primary patient-level risk prescribing during the preintervention period analysis, as well as a secondary practice-level (mean absolute increase, 0.07 percentage points 31 http://bmjopen.bmj.com/ 32 analysis (see the Supplementary Appendix). for every 8 weeks elapsed; 95% confidence inter- 33 The analyses were performed according to val [CI], 0.02 to 0.12). 34 the intention-to-treat principle; we analyzed data 35 from all the eligible patients regardless of Preintervention Data 36 whether they actually received a review. Multi- Table 1 shows the preintervention characteristics 37 level logistic regression was used to estimate odds of the participating practices according to their 38 ratios and 95% confidence intervals for exposure randomly assigned start date. The numbers of to high-risk prescribing across data points dur- patients with risk factors for adverse drug events

39 on October 3, 2021 by guest. Protected copyright. 40 ing the intervention period as compared with related to NSAIDs and antiplatelet agents at the 41 the preintervention period (Fig. S1 in the Supple- start of the intervention ranged from 1894 to 42 mentary Appendix), with adjustment for cluster- 4857 patients across the 10 randomly assigned 43 ing within practices and over time. In prespeci- start dates, with 2.0 to 4.6% of the patients hav- 44 fied analyses of hospital admissions, we summed ing prescriptions for a high-risk NSAID, an anti- 45 events across all the practices during the pre­ platelet medication, or both at the start of the 46 intervention period and the intervention period intervention. Although there were significant dif- 47 and calculated rates by dividing the sums by the ferences in the mean age and sex of the patients 48 total person-time during which patients had risk across start dates, absolute differences were 49 factors for adverse drug events related to NSAIDs small (range of mean age, 72 to 76 years; range 50 and antiplatelet agents. The incidence in the in- of percentage of men, 45 to 48%). There were no 51 tervention period versus the preintervention pe- significant differences in mean practice-list sizes 52 riod was compared with the use of the condi- and quality of care as measured by performance 53 tional maximum-likelihood estimate of the rate on the U.K. Quality and Outcomes Framework,32 54 ratio.31 but the percentages of patients living in the most 55 56 57 1056 n engl j med 374;11 nejm.org March 17, 2016 58 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml 59 60 BMJ Open: first published as 10.1136/bmjopen-2016-015281 on 10 March 2017. Downloaded from BMJ Open Page 38 of 61 Safer Prescribing

1 2 3 1 4 1 10 987

4 5171

5 (977–996) (1962–8506) 6 7 8 9 3 0 1 983 9 5102 (966–995)

10 (2685–9358) 11 12 8 4 3 2

13 982 For peer review7183 only

14 (972–996)

15 (2980–11,643)

16 f clinical and organizational indicators, each of start date of the intervention. The randomized h the use of Scottish Index Multiple 7 3 1 17 2 985 art date. 18 5418 (965–997) 19 (2738–8888) 20 21 6 3 1 3 968 22 6535 (927–996)

23 (3643–9738) 24

25 Study Group 5 3 1 3

26 970 5769

27 (954–988) 28 (3224–7392) 29 30 4 4 1 0 968 759

31 http://bmjopen.bmj.com/ (960–980)

32 (3205–10,052) 33 34 3 3 3 0 971

35 5323 (948–1000)

36 (1367–7596) 37 38 2 3 3 1 966

39 on October 3, 2021 by guest. Protected copyright. 40 6097 (961–1000) 41 (2621–9804) 42 43 1 3 2 1 979 2236 3253 2384 4857 2849 2532 2631 3968 1894 2933 6128 74±12 74±11 72±13 74±12 76±11 74±13 75±12 75±12 74±12 76±12 44 (4.0)90 (4.6) 151 (3.0) 71 (3.8) 184 (4.0) 115 (3.8) 96 (4.1) 108 (4.2) 168 (3.1) 59 (2.0) 60 312 (14.0)312 (12.8) 416 (38.3) 913 (5.1) 246 (7.2) 205 (29.9) 757 (0.4) 10 (6.6) 263 (5.6) 106 (1.6) 48 (968–986) 1009 (45.1)1009 (45.1) 1466 (47.9) 1143 (47.5) 2308 (47.0) 1340 (45.1) 1141 (46.6) 1226 (46.2) 1833 (46.9) 889 (47.9) 1404 45 (3680–8621) 46 47 48 32 49 50 51 52 risk prescribing risk previous in prescribing (%) no. — wk 8 quintile of Scottish postal Scottish of quintile (%)‡ no. — codes mean (range)‖ mean 53 (range) patients Preintervention Characteristics of the Patients and Practices, According to Randomized Start Date.* Start Randomized to According Practices, and Patients the of Characteristics Preintervention 1. Table Patients † high- for factors risk with No. high-risk to Exposure Characteristic Age — yr — Age Male sex — no. (%) no. — sex Male Residence in most deprived most in Residence QOF points achieved — achieved points QOF Practices no. Total Urban practices — no.¶ — practices Urban Training practices — no.§ — practices Training Mean no. of registered of no. Mean Plus–minus values are means ±SD. The 33 participating practices were divided into 10 study groups, according to the randomized Data are the total number of patients (defined as people registered to practices) across practices randomly assigned each st Deprivation was defined as residence in the most socioeconomically deprived fifth of Scottish postal codes and measured wit Training practices were defined as that registered for providing general practitioner training. Urban practices were defined as that based in towns or cities with more than 10,000 residents. The Quality and Outcomes Framework (QOF) financially rewards primary care practices according to their performance on a range o dates all occurred between October 30, 2011, and September 2, 2012, with 1 indicating the earliest group to start 10 la test. There were significant between-group differences in the following characteristics: patients who had high-risk prescribing in previous 8 weeks, age of patients, sex, and pa tients living most socioeconomically deprived quintile Scottish postal codes (all P<0.001). Analyses were performed with the use of one-way analysis variance for means and chi -square test proportions. The proportions training and urban practices in each group were not formally compared owing to small numbers. Deprivation, which is a measure that applied to areas with mean of approximately 750 residents. which is associated with a number of maximum achievable points, each point corresponding to defined payment. QOF scores range from 0 1000, higher indicating better performance.      54  ¶ ‖ ‡ † § 55 * 56 57 n engl j med 374;11 nejm.org March 17, 2016 1057 58 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml 59 60 BMJ Open: first published as 10.1136/bmjopen-2016-015281 on 10 March 2017. Downloaded from Page 39 of 61 BMJ Open The new england journal of medicine

1 2 3 4.0 4 5 3.5 6 7 3.0 8 9 10 2.5 11

12 2.0 13 14 For peer review only 15 1.5 16 in the Previous 8 Wk (%) 17 1.0 18 19 20 0.5 Preintervention period Intervention period Postintervention period 21 Patients with Risk Factors Who Received a High-Risk Prescription 22 0.0 23 −40 −32 −24 −16 −8 0 8 16 24 32 40 48 56 64 72 80 88 96 24 Weeks Relative to the Start of the Intervention in Each Practice 25 26 No. of Patients 965 955 933 977 1036 1102 970 792 744 707 622 674 662 652 630 628 552 560 with High-Risk 27 Prescription 28 29 No. of Patients 27,775 27,642 27,565 27,766 28,801 29,537 30,166 30,224 30,246 30,242 30,204 30,187 30,213 30,245 29,804 29,416 29,053 28,616 Particularly 30 Vulnerable

31 to Adverse http://bmjopen.bmj.com/ 32 Drug Events 33 Figure 1. Overall Trends in the Primary Outcome of High-Risk Prescribing across the Preintervention, Intervention, and Postintervention 34 Periods. 35 Each data point represents the percentage of patients with risk factors who received a high-risk prescription during the 8 weeks previous 36 to the stated time point. Time point 0 is the randomized intervention start date in each practice. 37 38

39 on October 3, 2021 by guest. Protected copyright. 40 socioeconomically deprived areas varied signifi- to be required for 1296 of these cases (69.8%). 41 cantly (range, 0.4 to 38.3%). Follow-up involved contacting patients to discuss 42 appropriateness (515 patients [27.7% of all re- 43 Reviews Conducted by Practices views]), planning discussion at future routine 44 During the intervention period, 2905 patients contact (409 [22.0%]), changing prescriptions 45 with risk factors received at least one high-risk and informing the patient by letter (163 [8.8%]), 46 prescription. Approximately half of all patients or other actions (209 [11.2%]). 47 who needed review during the 48-week interven- 48 tion period were flagged for review at the first Prescribing Outcomes 49 log-in. The remaining patients were flagged Figure 1 shows trends in the primary outcome 50 later during the intervention period; these pa- of high-risk prescribing, relative to the interven- 51 tients had a mixture of prior intermittent use tion start date in each practice (data for the 10 52 and true new use of high-risk prescriptions. Of individual groups, according to the randomly 53 these patients, 1598 (55.0%) underwent a total assigned start dates, are shown in Figs. S3 and 54 of 1858 reviews. At review, follow-up was judged S4 in the Supplementary Appendix). In the 55 56 57 1058 n engl j med 374;11 nejm.org March 17, 2016 58 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml 59 60 BMJ Open: first published as 10.1136/bmjopen-2016-015281 on 10 March 2017. Downloaded from BMJ Open Page 40 of 61 Safer Prescribing

1 2 3 4 0.015 0.040 0.039 0.040 0.021 0.021 0.084 0.014 0.033 0.027 0.053 0.036§ <0.001 5 <0.001 Coefficient Correlation 6 Intracluster 7 8 9 10 0.29 0.01 0.04 <0.001 <0.001 <0.001 <0.001 <0.001 <0.001 <0.001 <0.001 <0.001 <0.001 <0.001 11 P Value 12 13 14 For peer review only 15 16

17 aving been prescribed the same drug during previ - 18 ients over time. Under the stepped-wedge design, ad risk factors for high-risk prescribing and who received a 19 (95% CI)‡ 0.78 (0.49–1.23) 0.78 0.76 (0.62–0.94) 0.76 0.77 (0.65–0.91) 0.77 0.31 (0.10–0.94) 0.31 (0.18–0.55) 0.32 0.44 (0.29–0.67) 0.44 0.39 (0.29–0.51) 0.39 20 (0.21–0.34) 0.27 Multilevel Adjusted Odds 21 Ratio of Intervention Effect 22 23 24 25 26 27 (0.1) 3/2970 28 29 30 Prevalence† http://bmjopen.bmj.com/

31 (%) no. no./total 32 33 34 33/1593 (2.1)33/1593 (2.1) 37/1784 12/2483 (0.5) 12/2483 50/2483 (2.0)50/2483 (1.2) 35/2970 61/8799 (0.7)61/8799 (0.5) 38/8206 205/5841 (3.5)205/5841 (2.4) 145/6004 391/8033 (4.9)391/8033 (3.4) 283/8347 533/12,166 (4.4)533/12,166 (3.0) 381/12,612 (0.69–0.90) 0.78 119/8799 (1.4)119/8799 (0.6) 53/8206 214/17,310 (1.2)214/17,310 (0.6) 109/17,694 (0.38–0.57) 0.46 252/3973 (6.3)252/3973 (2.5) 102/4026 639/24,734 (2.6)639/24,734 (1.2) 308/25,100 (0.50–0.62) 0.55 336/29,537 (1.1)336/29,537 (0.7) 211/30,187 (0.68–0.87) 0.77 766/29,537 (2.6)766/29,537 (1.5) 463/30,187 (0.53–0.67) 0.60

35 (3.7)1102/29,537 (2.2) 674/30,187 (0.57–0.68) 0.63 36 Interventionof Start At Intervention of End At 37

38

39 on October 3, 2021 by guest. Protected copyright. 40 41 42 43 44 45 46 47 48 49 50 system blocker and diuretic and blocker system patient taking an oral anticoagulant oral an taking patient oral anticoagulant oral of age taking aspirin taking age of of age taking aspirin taking age of of age of with prior peptic ulcer peptic prior with 51 factor risk any NSAID in patient with chronic kidney disease kidney chronic with patient in NSAID NSAID in patient taking both renin–angiotensin both taking patient in NSAID Aspirin or clopidogrel without gastroprotection in gastroprotection without clopidogrel or Aspirin NSAID without gastroprotection in patient taking an taking patient in gastroprotection without NSAID Clopidogrel without gastroprotection in patient ≥65 yr ≥65 patient in gastroprotection without Clopidogrel NSAID without gastroprotection in patient ≥65 yr≥65 patient in gastroprotection without NSAID NSAID without gastroprotection in patient ≥75 yr ≥75 patient in gastroprotection without NSAID NSAID or aspirin without gastroprotection in patient in gastroprotection without aspirin or NSAID New 52 Ongoing NSAID in patient with history of heart failure heart of history with patient in NSAID Renal composite Renal Gastrointestinal composite Gastrointestinal 53 factor¶ risk any with patient in prescribing High-risk Effect of the Intervention on Primary and Secondary High-Risk Prescribing Outcomes.* Prescribing High-Risk Secondary and Primary on Intervention the of Effect 2. Table Secondary prescribing outcomes prescribing Secondary Primary outcome: any high-risk prescribing in patient with in patient prescribing high-risk any outcome: Primary Outcome Measure high-risk prescription in the previous 8 weeks. adjusted odds ratios are calculated with the use of all data points in intervention period versus preintervention perio d and so represent average exposure to high-risk prescribing over the entire intervention period versus preintervention period. ous 48 weeks. New high-risk prescribing was defined as the patient receiving a prescription and not having had it dur ing previous NSAID denotes nonsteroidal antiinflammatory drug. Data are the total number of patients across practices, at start and end intervention period in each practice, who h Multilevel adjusted odds ratios account for the clustering of patients within practices and repeated measurement pat The intracluster correlation coefficient is the proportion of variation in outcome that due to between practices. Ongoing high-risk prescribing was defined as the patient receiving a prescription during measurement period and h     54  * ‡ § 55 † ¶ 56 57 n engl j med 374;11 nejm.org March 17, 2016 1059 58 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml 59 60 BMJ Open: first published as 10.1136/bmjopen-2016-015281 on 10 March 2017. Downloaded from Page 41 of 61 BMJ Open The new england journal of medicine

1 2 3 0.004 0.34 0.002 0.19 0.02 4 0.10 <0.001 5 P Value 6 7 8 9 10 (95% CI) 11 Rate Ratio 12 13 14 For peer review only 15 16 17

18 (95% CI) 19 20 Absolute Difference 21 22 23 24 25 (412.4 to 639.3)(412.4 to −38.9)to (−349.5 −194.2 0.95) to (0.56 0.73 5/1558 80/1558 1/23,228 13/11,740 86/23,228

26 101/11740 377/27,878 27 513.5

28 29 30

31 http://bmjopen.bmj.com/ 32 33 14/2374 168/2374 51/14,720 15/32,687 150/14,720 182/32,687 34 465/38,505 35 36 Preintervention Period† Intervention Period† 37

38

39 on October 3, 2021 by guest. Protected copyright. 40 41 42 43 44 45 46 47 48 49 50 51 high-risk prescribing in the 8 wk before admission before wk 8 the in prescribing high-risk gastrointestinal with patients in prescribing high-risk factors risk in patients with heart failure heart with patients in prescribing in patients with renal risk factors risk renal with patients in prescribing preceding high-risk prescribing high-risk preceding with gastrointestinal risk factors risk gastrointestinal with 52 factors No. of events/person-yr of No. No. of events/person-yr of No. Incidence — no. of events/10,000 person-yr (95% CI)(95% person-yr events/10,000 of no. — Incidence 45.6)to (25.8 34.6 18.9)to (5.9 11.1 −12.3)to (−34.8 −23.6 0.58) to (0.17 0.32 No. of events/person-yr of No. Incidence — no. of events/10,000 person-yr (95% CI)(95% person-yr events/10,000 of no. — Incidence 75.7)to (25.7 4.6 2.4)to (0.0 0.4 −1.7)to (−6.6 −4.2 0.52) to (0.00 0.09 Incidence — no. of events/10,000 person-yr (95% CI)(95% person-yr events/10,000 of no. — Incidence 96.6)to (33.6 59.0 71.1)to (11.8 32.1 14.9)to (−68.7 −26.9 1.51) to (0.20 0.54 No. of events/person-yr of No. Incidence — no. of events/10,000 person-yr (95% CI)(95% person-yr events/10,000 of no. — Incidence 64.4)to (48.2 55.7 45.7)to (30.0 37.0 −29.9)to (−7.4 −18.7 0.86) to (0.51 0.66 No. of events/person-yr of No. Incidence — no. of events/10,000 person-yr (95% CI)(95% person-yr events/10,000 of no. — Incidence 119.2)to (86.5 101.9 104.6)to (70.8 86.0 7.5)to (−39.3 −15.9 1.09) to (0.68 0.84 No. of events/person-yr of No. events/person-yr of No. Incidence — no. of events/10,000 person-yr (95% CI)(95% person-yr events/10,000 of no. — Incidence 132.3)to (110.0 120.8 149.6)to (121.9 135.2 32.0)to (−3.0 14.5 1.28) to (0.98 1.12 53 CI)(95% person-yr events/10,000 of no. — Incidence 823.2) to (608.4 707.7 Secondary Outcome Measures Regarding Emergency Hospital Admission in Patients with Risk Factors for Adverse Drug Events from NSAIDs or Antiplatelet Medications.* Antiplatelet or NSAIDs from Events Drug Adverse for Factors Risk with Patients in Admission Hospital Emergency Regarding Measures Outcome Secondary 3. Table Outcome Measure relevant with admission hospital drug-related Potentially by preceded bleeding admission or ulcer Gastrointestinal Heart failure admission preceded by high-risk prescribing high-risk by preceded admission failure Heart Acute kidney injury admission preceded by high-risk by preceded admission injury kidney Acute Potentially drug-related hospital admission regardless of regardless admission hospital drug-related Potentially patients in bleeding admission or ulcer Gastrointestinal Acute kidney injury admission in patients with renal risk renal with patients in admission injury kidney Acute 54 failure heart with patients in admission failure Heart admission hospital Unrelated factor risk any with patients in admission fracture Hip 55 56 57 1060 n engl j med 374;11 nejm.org March 17, 2016 58 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml 59 60 BMJ Open: first published as 10.1136/bmjopen-2016-015281 on 10 March 2017. Downloaded from BMJ Open Page 42 of 61 Safer Prescribing

1 2 analysis of the primary outcome, 674 of 30,187 3 patients (2.2%) with risk factors for related ad- 0.87 0.90 4 0.52 verse drug events had prescriptions for NSAIDs, 5 P Value antiplatelet agents, or both at the end of the in- 6 tervention, as compared with 1102 of 29,537 7 (3.7%) who had prescriptions immediately be- 8 fore the intervention (adjusted odds ratio, 0.63; 9 95% CI, 0.57 to 0.68; P<0.001). 10 There were significant reductions in the rates (95% CI) 11 Rate Ratio of ongoing high-risk prescribing (from 2.6% im- 12 mediately before the intervention to 1.5% at the 13 For peer reviewend onlyof the intervention; adjusted odds ratio, 14 0.60; 95% CI, 0.53 to 0.67; P<0.001) and new 15 high-risk prescribing (from 1.1% to 0.7%; ad- 16 justed odds ratio, 0.77; 95% CI, 0.68 to 0.87;

17 for high-risk prescribing. The analyses of potentially P<0.001). There were significant reductions in der the stepped-wedge design. 18 (95% CI) eight of the nine individual measures of high- 19 risk prescribing (range of adjusted odds ratios, Absolute Difference 20 0.27 to 0.78), but there was no significant reduc- 21 tion in the rate of NSAID prescribing for people 22 with heart failure (Table 2). 23 Reductions in the rates of high-risk prescrib- 24 ing were sustained after financial incentives 25 ceased. In an analysis of the primary outcome,

26 38/27,878 516/27,878 1426/27,878 674 of 30,187 patients (2.2%) with risk factors 27 for related adverse drug events had prescriptions 28 29 for NSAIDs, antiplatelet agents, or both at the 30 end of the intervention, and 560 of 28,616 (2.0%) had such prescriptions at 48 weeks after incen- 31 http://bmjopen.bmj.com/ 32 tives ceased (adjusted odds ratio, 1.14; 95% CI, 33 0.85 to 1.53; P = 0.38). for hospital admissions data in the United Kingdom, but heart failure, perforated peptic ulcer or 54/38,505 706/38,505 1926/38,505 34 28 35 Hospital Admission Outcomes 36 Preintervention Period† Intervention Period† Table 3 shows that among patients with risk fac- 37 tors for adverse drug events related to NSAIDs 38 and antiplatelet agents, the incidence of admis- sions preceded by a high-risk prescription (re-

39 on October 3, 2021 by guest. Protected copyright. 40 ceived in the previous 8 weeks) was significantly 41 reduced from the preintervention period to the 42 intervention period; admissions for gastrointes- 43 tinal bleeding were reduced from 4.6 per 10,000 44 person-years to 0.4 per 10,000 person-years (rate 45 ratio, 0.09; 95% CI, 0.00 to 0.52), and admis- 46 sions for acute kidney injury were reduced from 47 34.6 per 10,000 person-years to 11.1 per 10,000 48 person-years (rate ratio, 0.32; 95% CI, 0.17 to 49 0.58). The incidence of admissions for heart fail- 50 ure that were preceded by an NSAID prescription 51 was not significantly reduced from the preinter- 52 factor risk any with patients factor‡ risk vention period to the intervention period (59.0 No. of events/person-yr of No. Incidence — no. of events/10,000 person-yr (95% CI)(95% person-yr events/10,000 of no. — Incidence 197.4)to (170.1 183.4 200.8)to (169.5 185.1 22.7)to (−19.2 1.7 1.13) to (0.90 1.01 No. of events/person-yr of No. events/person-yr of No. Incidence — no. of events/10,000 person-yr (95% CI)(95% person-yr events/10,000 of no. — Incidence 18.3)to (10.5 14.0 18.7)to (9.6 13.6 5.3)to (−6.1 0.4 1.47) to (0.64 0.97 53 CI)(95% person-yr events/10,000 of no. — Incidence 523.0)to (478.1 500.2 538.8)to (485.3 511.5 46.0)to (−23.4 11.3 1.10) to (0.95 1.02 and 32.1 admissions per 10,000 person-years, re- Outcome Measure Any cancer admission in patients with any risk factor risk any with patients in admission cancer Any Appendicitis, cholecystitis, or pancreatitis admission in admission pancreatitis or cholecystitis, Appendicitis, any with patients in admission care–sensitive ambulatory Any bleeding, and iron-deficiency anemia were excluded because the investigators considered them to be potentially related t argeted prescribing. Ambulatory care–sensitive condi - tions included were angina, asthma, cellulitis, convulsions and epilepsy, chronic obstructive pulmonary disease, dehydration an d gastroenteritis, dental conditions, diabetes complica - tions, infection of the ear, nose, or throat, gangrene, hypertension, influenza and pneumonia, nutritional deficiency, other va ccine-preventable disease, pelvic inflammatory pyelonephritis. drug-related hospital admission with relevant high-risk prescribing in the 8 weeks before and potentially drug-relate d regardless of preceding prescribing were prespecified. The analysis of hospital admission that was considered by the investigators to be unrelated t he specified high-risk patterns targeted intervention was post hoc. Data are the number of outcome events over preintervention and intervention periods, during which patients had risk factors The differences in person-time are due to longer preintervention periods practices randomly assigned later start dates un Ambulatory care–sensitive admissions were as defined by Purdy et al.   54  ‡ † * spectively; rate ratio, 0.54; 95% CI, 0.20 to 1.51). 55 56 57 n engl j med 374;11 nejm.org March 17, 2016 1061 58 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml 59 60 BMJ Open: first published as 10.1136/bmjopen-2016-015281 on 10 March 2017. Downloaded from Page 43 of 61 BMJ Open The new england journal of medicine

1 2 Among patients with risk factors for adverse the intervention led to reductions in the rates of 3 drug events related to NSAIDs and antiplatelet related emergency hospital admissions, with no 4 agents, regardless of preceding high-risk pre- change observed in the rate of unrelated admis- 5 scribing, the incidence of total hospital admis- sions. 6 sions for gastrointestinal ulcer or bleeding was The strengths of the study include the careful 7 significantly reduced (from 55.7 to 37.0 admis- intervention design,12,19-21 evaluation in routine 8 sions per 10,000 person-years; rate ratio, 0.66; primary care practice, and examination of sus- 9 95% CI, 0.51 to 0.86), as was the incidence of tainability after financial incentives ceased. The 10 heart failure (from 707.7 to 513.5 admissions per study also has some important limitations. First, 11 10,000 person-years; rate ratio, 0.73; 95% CI, although half the eligible practices took part, 12 0.56 to 0.95). The incidence of total admissions participating practices may have been more 13 for acuteFor kidney injury peer was not significantly review re- motivated oronly had greater capacity to change than 14 duced from the preintervention period to the nonparticipating practices. Second, the stepped- 15 intervention period (101.9 and 86.0 admissions wedge design can be vulnerable to secular trends 16 per 10,000 person-years, respectively; rate ratio, if outcomes are already improving.30 The rate of 17 0.84; 95% CI, 0.68 to 1.09). targeted high-risk prescribing was rising slowly 18 Among patients with risk factors for adverse during the preintervention period and there were 19 drug events related to NSAIDs and antiplatelet no other relevant nontrial interventions during 20 agents, there was no significant change between the period of implementation, which indicates 21 the preintervention period and the intervention that secular trends are unlikely to have affected 22 period in the incidence of admissions for hip the prescribing outcomes. However, we were un- 23 fracture (120.8 and 135.2 admissions per 10,000 able to reliably examine prior time trends for the 24 person-years, respectively; rate ratio, 1.12; 95% hospital admission outcomes examined, since 25 CI, 0.98 to 1.28), cancer (183.4 and 185.1 admis- they are relatively rare events, but the rates of 26 sions per 10,000 person-years; rate ratio, 1.01; four different types of unrelated admission did 27 95% CI, 0.90 to 1.13), or appendicitis, cholecys- not fall significantly, which increases confidence 28 29 titis, or pancreatitis (14.0 and 13.6 admissions in the observed findings. 30 per 10,000 person-years; rate ratio, 0.97; 95% CI, It should be noted that the reductions in the 0.64 to 1.47), or in the incidence of unrelated total rates of admissions due to gastrointestinal 31 http://bmjopen.bmj.com/ 32 ambulatory care sensitive admissions (500.2 and bleeding and heart failure were larger than can 33 511.5 admissions per 10,000 person-years; rate be explained by reductions in the rates of such 34 ratio, 1.02; 95% CI, 0.95 to 1.10). admissions that were actually preceded by the 35 targeted high-risk prescribing. A recent study of 36 Discussion a large pay-for-performance program in primary 37 care that was conducted in the United Kingdom 38 We found that a complex intervention that com- also showed larger reductions in the rates of bined professional education, informatics to emergency admission than could be explained

39 on October 3, 2021 by guest. Protected copyright. 40 identify high-risk patients, and financial incen- by changes in targeted process measures.33 Al- 41 tives to primary care clinicians significantly re- though such halo effects are conceivable (e.g., 42 duced the rate of high-risk prescribing of NSAIDs owing to safer use of other medicines that cause 43 and antiplatelet medications, which is a common the same adverse effects or recommendations to 44 cause of drug-related emergency hospital admis- patients to avoid over-the-counter NSAIDs and 45 sions internationally.4,5,7,8 The effect on ongoing aspirin), confirmation in larger studies that are 46 high-risk prescribing was somewhat larger than powered to robustly examine hospital admission 47 the effect on new high-risk prescribing — a outcomes is required. 48 finding that was consistent with the fact that the Finally, although it is likely that this interven- 49 intervention prompted review of patients who tion would be effective for similar prescribing 50 were already exposed to high-risk prescribing decisions (those based on an assessment of 51 — although both rates fell significantly, and a benefit and harm to the individual patient), com- 52 lower rate of initiation of prescribing contribut- mon adverse drug events that are associated with 53 ed to the sustained effect in the year after finan- other mechanisms will require different ap- 54 cial incentives ceased. There was evidence that proaches. For example, reducing warfarin-related 55 56 57 1062 n engl j med 374;11 nejm.org March 17, 2016 58 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml 59 60 BMJ Open: first published as 10.1136/bmjopen-2016-015281 on 10 March 2017. Downloaded from BMJ Open Page 44 of 61 Safer Prescribing

1 2 harm is likely to require improvements in the 12 months.29 In contrast, the DQIP intervention 3 reliability of monitoring and patient education. had a sustained effect over the year after the fi- 4 This trial showed that a blend of education, nancial incentives ceased, which is important to 5 financial incentives to review high-risk patients, allow the focus of quality-improvement activity 6 and informatics to support review was effective, to move to other areas of care. 7 but we cannot identify which aspect of the inter- The three components of this intervention are 8 vention mattered most. Financial incentives to feasible in any system in which primary care is 9 stop high-risk prescribing are potentially prob- delivered by physician-owned practices that use 10 lematic because there may be a risk of inappro- EMRs and are under contract with third-party 11 priate cessation to obtain payment in a subgroup payers, which is a common model internation- 12 of patients in whom the balance of benefit and ally. However, complex interventions of this type 13 harm favors continuationFor of thepeer drug. We there review- inevitably require tailoring only to context. For exam- 14 fore chose to provide incentives to review. Since ple, the size of the incentives that are required 15 this approach could lead to “tick-box review” to prompt review is likely to vary according to 16 simply to obtain payment, we sought to ensure primary care physician payment structures and 17 the delivery of meaningful review through the use incomes. Similarly, in some contexts, it may be 18 of education and informatics, because an under- easier to embed the informatics functionality 19 standing of risk and an awareness that current directly in the EMR (e.g., if all targeted providers 20 practice is suboptimal are likely to be prerequi- use one EMR system and the EMR supplier is 21 sites for changing prescribing behavior, and there willing to cooperate). Although we think that it 22 was good evidence from previous studies that is likely that interventions that blend education, 23 both educational outreach visits and audit and incentives to review, and informatics will be ef- 24 feedback practices can lead to small improve- fective in other health care systems, any imple- 25 ments in prescribing.34,35 mentation should be tailored to context and 26 Feedback was one element of the informatics; evaluated for effect. 27 the other was a reduction in barriers to efficient In conclusion, we found that a complex inter- 28 29 structured review. Education and minimization vention that combined professional education, 30 of the barriers to change were also elements in informatics to support patient identification and the successful PINCER intervention (based in the review, and financial incentives to review patients 31 http://bmjopen.bmj.com/ 32 United Kingdom), in which external pharmacists who have been exposed to high-risk prescribing 33 reviewed patients who were exposed to high-risk led to substantial and sustained reductions in 34 prescribing and worked with primary care physi- targeted high-risk prescribing and was associ- 35 cians to implement change. The PINCER trial ated with reductions in the rates of related emer- 36 showed reductions in the rates of high-risk pre- gency admissions. scribing that were similar to those observed Supported by a grant (ARPG/07/02) from the Scottish Govern- 37 ment Chief Scientist Office. 38 with the DQIP intervention at 6 months, but the Disclosure forms provided by the authors are available with effect of pharmacist-led review partly waned by the full text of this article at NEJM.org.

39 on October 3, 2021 by guest. Protected copyright. 40 41 References 42 1. Budnitz DS, Lovegrove MC, Shehab N, 4. Pirmohamed M, James S, Meakin S, 8. Howard RL, Avery AJ, Slavenburg S, 43 Richards CL. Emergency hospitalizations et al. Adverse drug reactions as cause of et al. Which drugs cause preventable ad- 44 for adverse drug events in older Ameri- admission to hospital: prospective analy- missions to hospital? A systematic review. cans. N Engl J Med 2011;​365:​2002-12. sis of 18,820 patients. BMJ 2004;​329:​15-9. Br J Clin Pharmacol 2007;​63:​136-47. 45 2. Leendertse AJ, de Koning FH, Goud- 5. Gurwitz JH, Field TS, Harrold LR, et al. 9. Thomsen LA, Winterstein AG, Søn- 46 swaard AN, et al. Preventing hospital Incidence and preventability of adverse dergaard B, Haugbølle LS, Melander A. 47 admissions by reviewing medication drug events among older persons in the Systematic review of the incidence and (PHARM) in primary care: design of the ambulatory setting. JAMA 2003;​289:​1107- characteristics of preventable adverse drug 48 cluster randomised, controlled, multi- 16. events in ambulatory care. Ann Pharmaco- 49 centre PHARM-study. BMC Health Serv 6. Bates DW, Cullen DJ, Laird N, et al. ther 2007;41:​ 1411-26.​ 50 Res 2011;​11:4.​ Incidence of adverse drug events and po- 10. Hakkarainen KM, Hedna K, Petzold 51 3. Aspden P, Bootman L, Cronenwett L. tential adverse drug events: implications M, Hägg S. Percentage of patients with Preventing medication errors (Quality for prevention. JAMA 1995;​274:​29-34. preventable adverse drug reactions and 52 chasm series). Washington, DC:​ Institute 7. Gandhi TK, Weingart SN, Borus J, preventability of adverse drug reactions 53 of Medicine of the National Academies, et al. Adverse drug events in ambulatory — a meta-analysis. PLoS One 2012;​7(3):​ 54 2007. care. N Engl J Med 2003;​348:​1556-64. e33236. 55 56 57 n engl j med 374;11 nejm.org March 17, 2016 1063 58 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml 59 60 BMJ Open: first published as 10.1136/bmjopen-2016-015281 on 10 March 2017. Downloaded from Page 45 of 61 BMJ Open Safer Prescribing

1 2 11. IMS Institute for Health Care Infor- tory drugs and antiplatelets in primary and management — complementary roles matics. Avoidable costs in U.S. health- medical care: the DQIP study protocol. for COX-2 inhibitors and proton pump in- 3 care: ​the $200 billion opportunity from Implement Sci 2012;7:​ 24.​ hibitors. Gut 2003;​52:600-8.​ 4 using medicines more responsibly. 2013 20. Dreischulte T, Grant AM, McCowan 28. Purdy S, Griffin T, Salisbury C, Sharp 5 (http://www​.imshealth.com/​ ​en/​thought C, McAnaw JJ, Guthrie B. Quality and D. Ambulatory care sensitive conditions: -leadership/ims-institute/​ reports/​ ​avoidable safety of medication use in primary care: terminology and disease coding need to 6 -costs). consensus validation of a new set of ex- be more specific to aid policy makers and 7 12. Guthrie B, McCowan C, Davey P, plicit medication assessment criteria and clinicians. Public Health 2009;​123:​169- 8 Simpson CR, Dreischulte T, Barnett K. prioritisation of topics for improvement. 73. 9 High risk prescribing in primary care BMC Clin Pharmacol 2012;​12:​5. 29. Avery AJ, Rodgers S, Cantrill JA, et al. patients particularly vulnerable to adverse 21. Grant AM, Guthrie B, Dreischulte T. A pharmacist-led information technol­ 10 drug events: cross sectional population Developing a complex intervention to im- ogy intervention for medication errors 11 database analysis in Scottish general prac- prove prescribing safety in primary care: (PINCER): a multicentre, cluster random­ 12 tice. BMJ 2011;342:​ d3514.​ mixed methods feasibility and optimisa- ised, controlled trial and cost-effectiveness 13 13. Zhang Y, Baicker K, Newhouse JP. tion pilot study. BMJ Open 2014;​4(1):​ analysis. Lancet 2012;379:​ 1310-9.​ GeographicFor variation in the peerquality of pre- e004153. review only30. Hussey MA, Hughes JP. Design and 14 scribing. N Engl J Med 2010;​363:​1985-8. 22. Gooch K, Culleton BF, Manns BJ, et analysis of stepped wedge cluster random­ 15 14. Improving quality and patient experi- al. NSAID use and progression of chronic ized trials. Contemp Clin Trials 2007;​28:​ 16 ence: the state of health care quality 2013. kidney disease. Am J Med 2007;​120(3):​ 182-91. Washington, DC: ​National Committee for 280.e1-7. 31. Martin DO, Austin H. Exact estimates 17 Quality Assurance, 2013. 23. Huerta C, Castellsague J, Varas-Lorenzo for a rate ratio. Epidemiology 1996;​7:​29- 18 15. Davidoff AJ, Miller GE, Sarpong EM, C, Garcia Rodriguez LA. Nonsteroidal anti- 33. 19 Yang E, Brandt N, Fick DM. Prevalence of inflammatory drugs and risk of ARF in 32. Roland M, Campbell S. Successes and potentially inappropriate medication use the general population. Am J Kidney Dis failures of pay for performance in the 20 in older adults using the 2012 Beers crite- 2005;​45:531-9.​ United Kingdom. N Engl J Med 2014;​370:​ 21 ria. J Am Geriatr Soc 2015;​63:​486-500. 24. Dreischulte T, Morales DR, Bell S, 1944-9. 22 16. Cribb A, Barber N. Prescribers, pa- Guthrie B. Combined use of nonsteroidal 33. Harrison MJ, Dusheiko M, Sutton M, tients and policy: the limits of technique. anti-inflammatory drugs with diuretics Gravelle H, Doran T, Roland M. Effect of 23 Health Care Anal 1997;​5:​292-8. and/or renin-angiotensin system inhibi- a national primary care pay for perfor- 24 17. Dreischulte T, Guthrie B. High-risk tors in the community increases the risk mance scheme on emergency hospital 25 prescribing and monitoring in primary of acute kidney injury. Kidney Int 2015;​88:​ admissions for ambulatory care sensitive 26 care: how common is it, and how can it 396-403. conditions: controlled longitudinal study. be improved? Ther Adv Drug Saf 2012;​3:​ 25. Hallas J, Dall M, Andries A, et al. Use BMJ 2014;349:​ ​g6423. 27 175-84. of single and combined antithrombotic 34. Forsetlund L, Bjørndal A, Rashidian A, 28 18. Crabtree BF, Nutting PA, Miller WL, therapy and risk of serious upper gastro- et al. Continuing education meetings 29 et al. Primary care practice transforma- intestinal bleeding: population based case- and workshops: effects on profession- tion is hard work: insights from a 15-year control study. BMJ 2006;​333:​726. al practice and health care outcomes. 30 developmental program of research. Med 26. Gislason GH, Rasmussen JN, Abild- Cochrane Database Syst Rev 2009;​2:​

31 Care 2011;​49:Suppl:​ ​S28-35. strom SZ, et al. Increased mortality and CD003030. http://bmjopen.bmj.com/ 32 19. Dreischulte T, Grant A, Donnan P, cardiovascular morbidity associated with 35. O’Brien MA, Rogers S, Jamtvedt G, et al. et al. A cluster randomised stepped wedge use of nonsteroidal anti-inflammatory Educational outreach visits: effects on 33 trial to evaluate the effectiveness of a multi- drugs in chronic heart failure. Arch In- professional practice and health care out- 34 faceted information technology-based tern Med 2009;169:​ 141-9.​ comes. Cochrane Database Syst Rev 2007;​ 35 intervention in reducing high-risk pre- 27. Hawkey CJ, Langman MJ. Non-steroidal 4:​CD000409. 36 scribing of non-steroidal anti-inflamma- anti-inflammatory drugs: overall risks Copyright © 2016 Massachusetts Medical Society. 37 38

39 on October 3, 2021 by guest. Protected copyright. 40 41 42 43 We are seeking a Deputy Editor to join our editorial team at the New England Journal 44 of Medicine. The Deputy Editor will review, select, and edit manuscripts for the 45 Journal as well as participate in planning the Journal’s content. He or she will also ensure that the best available research is submitted to the Journal, identify potential 46 Journal contributors, and invite submissions as appropriate. 47 Applicants must have an M.D. degree and at least 5 years of research experience. 48 The successful candidate will be actively involved in patient care and will continue 49 that involvement (approximately 20% of time) while serving in this role. 50 To apply, visit https://mmscareers.silkroad.com. 51 52 53 54 55 56 57 1064 n engl j med 374;11 nejm.org March 17, 2016 58 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml 59 60 BMJ Open Page 46 of 61

Grant et al. Trials 2012, 13:154 BMJ Open: first published as 10.1136/bmjopen-2016-015281 on 10 March 2017. Downloaded from 1 http://www.trialsjournal.com/content/13/1/154 TRIALS 2 3 4 5 STUDYPROTOCOL Open Access 6 7 8 9 Study protocol of a mixed-methods evaluation of 10 11 a cluster randomized trial to improve the safety 12 13 of NSAID and antiplatelet prescribing: data-driven 14 15 quality improvementFor peer inreview primary only care 16 17 Aileen Grant1*, Tobias Dreischulte2, Shaun Treweek1 and Bruce Guthrie1 18 19 20 Abstract 21 22 Background: Trials of complex interventions are criticized for being ‘black box’, so the UK Medical Research Council 23 recommends carrying out a process evaluation to explain the trial findings. We believe it is good practice to pre-specify 24 and publish process evaluation protocols to set standards and minimize bias. Unlike protocols for trials, little guidance 25 or standards exist for the reporting of process evaluations. This paper presents the mixed-method process evaluation 26 protocol of a cluster randomized trial, drawing on a framework designed by the authors. 27 Methods/design: This mixed-method evaluation is based on four research questions and maps data collection to a 28 logic model of how the data-driven quality improvement in primary care (DQIP) intervention is expected to work. Data 29 collection will be predominately by qualitative case studies in eight to ten of the trial practices, focus groups with 30 patients affected by the intervention and quantitative analysis of routine practice data, trial outcome and questionnaire 31 data and data from the DQIP intervention. 32 33 Discussion: We believe that pre-specifying the intentions of a process evaluation can help to minimize bias arising from potentially misleading post-hoc analysis. We recognize it is also important to retain flexibility to examine the

34 http://bmjopen.bmj.com/ 35 unexpected and the unintended. From that perspective, a mixed-methods evaluation allows the combination of 36 exploratory and flexible qualitative work, and more pre-specified quantitative analysis, with each method contributing 37 to the design, implementation and interpretation of the other. 38 As well as strengthening the study the authors hope to stimulate discussion among their academic colleagues about 39 publishing protocols for evaluations of randomized trials of complex interventions. 40 Data-driven quality improvement in primary care trial registration: ClinicalTrials.gov: NCT01425502 41 42 Keywords: Complex intervention, Process evaluation, Protocol, Mixed methods, Randomized controlled trial 43 on October 3, 2021 by guest. Protected copyright. 44 Background how context influences outcomes, and to provide insights 45 Trials of complex interventions are often criticized as to aid implementation [1].’ However, when designing our 46 being a black box because it is difficult to know exactly own process evaluation of a cluster randomized trial of a 47 why an intervention did (or did not) work. The UK complex intervention, [2] we found little specific guidance. 48 Medical Research Council’s framework for designing and In response, we have proposed a framework to guide 49 evaluating complex interventions recommends conduct- process evaluation design [3]. 50 ing a process evaluation in order to ‘explain discrepancies Pre-specifying and publishing protocols are recognized 51 52 between expected and observed outcomes, to understand as improving the standards of clinical trials, by allowing 53 comparison of what was intended with what was actually done. Although process evaluations are somewhat differ- 54 * Correspondence: [email protected] 55 1Quality, Safety and Informatics Research Group, Population Health Sciences, ent (since flexibility to evaluate unexpected findings quali- University of Dundee, Mackenzie Building, Dundee DD2 4BF, UK tatively will often be useful), we believe that publication of 56 Full list of author information is available at the end of the article 57 58 © 2012 Grant et al.; licensee BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and 59 reproduction in any medium, provided the original work is properly cited. 60 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml Page 47 of 61 BMJ Open

Grant et al. Trials 2012, 13:154 Page 2 of 6 BMJ Open: first published as 10.1136/bmjopen-2016-015281 on 10 March 2017. Downloaded from 1 http://www.trialsjournal.com/content/13/1/154 2 3 4 5 process evaluation protocols is also best practice, and that DQIP process evaluation aim and research questions 6 pre-specifying the intentions of process evaluations can Aim 7 help to minimize bias arising from potentially misleading The aim of this study is to understand DQIP delivery, 8 post-hoc analysis. Unlike trial protocols, there is little implementation and translation into reducing high risk 9 explicit guidance or standards for the reporting of prescribing. 10 process evaluation protocols. We hope publishing this Research questions: 11 protocol will help stimulate discussion among the aca- 12 demic community. 1. How do different practices adopt the intervention and 13 This paper briefly describes the trial being evaluated, how does this influence their delivery to targeted 14 presents a logic model detailing the trial hypothesized patients and whether they continue to use the 15 pathway of change,For and states thepeer research questions reviewintervention? only Are there any positive or negative 16 that arise from our assumed model of trial processes, unintended consequences from a practice perspective? 17 overall study design, and detailed methods for answering 2. How do patients respond to the intervention? Are 18 each question. there any positive or negative unintended 19 consequences from a patient perspective? 20 Data-driven quality improvement in primary care trial 3. How are practice organizational characteristics 21 High-risk prescribing of non-steroidal anti-inflammatory associated with variation in recruitment, adoption, 22 drugs (NSAIDs) and antiplatelet agents account for a reach, delivery to the patient, and maintenance? 23 significant proportion of hospital admissions, due to pre- 4. Does effectiveness vary between practices, and if so, 24 ventable adverse drug events [4,5]. We are conducting a how are practice characteristics and processes 25 randomized controlled trial of a complex intervention to including structure, adoption, reach, delivery to the 26 improve prescribing safety of these drugs in 40 general patient, and maintenance associated with effectiveness? 27 practices in two Scottish Health Boards (the DQIP trial). 28 The trial is fully described in the published protocol [2]. Management and governance 29 In brief, the DQIP intervention comprises three compo- Research ethical approval for this study was granted 30 nents. The first component is a web-based informatics by Fife and Forth Valley Research Ethics Committee 31 tool that provides weekly updated feedback of targeted (11/AL/0251). An external Trial Steering Committee and 32 prescribing at practice level. This prompts the review of an overall Program Advisory Group, consisting of inde- 33 individual patients affected and summarizes each patient’s pendent professional and lay people, have been established

34 http://bmjopen.bmj.com/ relevant risk factors and prescribing. The second compo- and both meet bi-annually. 35 nent is an educational outreach visit that highlights the 36 risks of the targeted prescribing and provides training in Methods/design 37 38 the use of the tool. The final component is a fixed pay- Overall study design 39 ment of 350 GBP (560 USD; 403 EUR) to each practice up The study is a mixed-methods parallel process evalu- 40 front and a fee-for-service payment of 15 GBP (24 USD; ation. It will include a qualitative dominant case-study 41 17 EUR) for each patient reviewed. approach in a purposeful sample of practices and a 42 The trial has a stepped wedge design, [6,7] where all hypothesis-testing quantitative analysis of data from all 43 participating practices receive the DQIP intervention. practices. Data collection and analysis will be conducted on October 3, 2021 by guest. Protected copyright. 44 These practices are randomized to one of ten different in parallel to the DQIP trial itself. 45 start dates, with each practice functioning as its own 46 control in a time series analysis. The primary outcome Methods for research question 1 47 measure is a composite of nine measures of high-risk Study design 48 NSAID and antiplatelet prescribing. A separate eco- Study will be a primarily qualitative, mixed-methods in- 49 nomic evaluation is planned. depth case-study in a sample of eight to ten practices. 50 51 Structure/framework of DQIP parallel process evaluation Sampling 52 protocol Sampling for this study will be purposive and ensure 53 Drawing on our proposed framework for designing process heterogeneity in list size, reflecting our prior expectation 54 evaluations of cluster randomized trials, [3] Figure 1 shows that effectiveness will be greater in small practices; [8] 55 the processes we assume have to occur for the intervention and initial use of the tool (since this is how patients for 56 to be effective, and therefore explicitly identifies targets for review are identified and so a key process if our logic 57 evaluation. This logic model underlies the process evalu- model is true); and the date the practice started the trial 58 ation design and data collection, although resource con- (which is randomized, but workload in general practice is 59 straints inevitably require prioritization. strongly seasonal, so resource to deliver the intervention 60 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml BMJ Open Page 48 of 61

Grant et al. Trials 2012, 13:154 Page 3 of 6 BMJ Open: first published as 10.1136/bmjopen-2016-015281 on 10 March 2017. Downloaded from 1 http://www.trialsjournal.com/content/13/1/154 2 3 4 5 6 7 8 9 10 11 12 13 14 15 For peer review only 16 17 18 19 20 21 22 23 24 25 26 27 28 29 Figure 1 Data-driven quality improvement in primary care (DQIP) trial hypothesized pathway of change and process evaluation model. 30 31 mayvarybystartdate).Wewillsampletwopractices the practice pharmacist will also be interviewed. Partici- 32 from the first two cohorts of starting date and develop pants will be asked about their experience of the interven- 33 theories that can then be tested in the subsequent case tion, the practice process for review, how the intervention

34 http://bmjopen.bmj.com/ studies [9]. fitted with their existing work, barriers and facilitators, 35 maintenance over time, and any perceived unintended 36 Qualitative data collection consequences. Where possible, the GP most involved in 37 38 Data collection will use a variety of methods, in order to DQIP will be re-interviewed three to six months later, to 39 understand how intervention processes were perceived explore changes over time and the factors involved in 40 and implemented in each practice, and how these pro- maintaining and sustaining use of the tool. Interviews will ’ 41 cesses influence observed trial outcomes [10]. As part of be conducted at a time and place of the participants 42 the intervention in all practices, the qualitative re- choosing, facilitated by a topic guide drawing on the logic 43 searcher will assist the research pharmacist in delivering model and Normalization Process Theory (NPT), [12] and on October 3, 2021 by guest. Protected copyright. ’ 44 the education and training to the trial practices, and with the participants permission, audio-recorded for tran- 45 make field notes detailing attendance and the practice’s scribing. Normalization Process Theory is a sociological 46 response. Any additional communication or practice vis- theory designed to understand how practices or interven- 47 its will be recorded in the same way. These data will in- tions are implemented and become embedded into every- 48 form the focus of the interviews in each practice, and day work (normalized and routinized). NPT is made up of 49 additionally be used to help construct a detailed, ‘rich’ four domains, coherence, cognitive participation, collect- 50 description of each practice for cross-case analysis [11]. ive action, reflective monitoring, with each of these 51 General practitioners (GPs), practice pharmacists and domains divided into four categories [12]. 52 practice managers, from the case-study practices, involved 53 in delivering the DQIP intervention to patients will be Qualitative data analysis 54 invited to interview approximately six months after the The professional qualitative data will be inductively and 55 practice starts the trial. Depending on practice size and deductively analyzed in separate NVivo 8 (QSR Inter- 56 how each practice has organized DQIP work, we will national, Southport, UK) databases. Deductive analysis 57 interview one to three professionals per practice. At a will be by examining the data through the lens of NPT 58 minimum, the GP most involved in DQIP will be inter- [12]. Our pilot work has illustrated that this means of 59 viewed, but other GPs involved, the practice manager and analysis sometimes ‘divides’ emergent codes in the data. 60 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml Page 49 of 61 BMJ Open

Grant et al. Trials 2012, 13:154 Page 4 of 6 BMJ Open: first published as 10.1136/bmjopen-2016-015281 on 10 March 2017. Downloaded from 1 http://www.trialsjournal.com/content/13/1/154 2 3 4 5 As a result, the data will be analyzed inductively again, asked about their experience of medication reviews and 6 to allow themes to emerge from the data and to explore the extent to which they were aware of the intervention 7 issues that the NPT deductive analysis may be less suited (response). Then after watching a short presentation 8 to identify and examine. about the way clinical data is being used in this study, 9 patients will be asked about their experience, percep- 10 Case-study analysis tions and any consequences of the intervention triggered 11 For each practice (case), we will have a set of quantitative by review or medication change and their attitudes and 12 and qualitative data which can be synthesized. Quantita- beliefs about the way in which clinical data is being 13 tive data, as described in detail below, will be used to de- used. 14 scriptively characterize each practice in terms of practice 15 structural characteristics,Forand measures peer of adoption, deliv-reviewData analysis only 16 ery to patient, reach, and maintenance. Qualitative data Patient focus group data will be analyzed inductively to 17 includes field notes relating to research team contacts allow themes to emerge from the data. The framework 18 with each practice and interview data. Synthesis of the technique [17] will be used to facilitate analysis. 19 qualitative and quantitative data for the case studies will 20 take two forms; first a comprehensive, ‘rich’ description of Methods for research questions 3 and 4 21 each case-study practice will be created based on all data, Study design 22 then this description will be used to conduct a cross-case The study requires quantitative analysis of routinely 23 comparison based on the logic model [13]. The aim of this available data about practices, trial process data gener- 24 analysis is to examine in detail how practices implement ated by DQIP informatics tool, practice questionnaire 25 (or not) the processes that our logic model assumes are data, and trial outcome data. We hypothesize that practice 26 required to deliver change in high-risk prescribing, and organizational characteristics and adoption will be asso- 27 how these processes appear to influence effectiveness in ciated with the level of reach, delivery to the patient and 28 each practice. The aim of this sequential merging of quali- maintenance achieved, and that lower levels of reach, 29 tative and quantitative data is not triangulation but delivery to the patient and maintenance will be associated 30 crystallization where we are using the qualitative data to with lower effectiveness. The specific hypotheses to be 31 explain the quantitative [14,15]. This offers the advantage tested will be based on findings from the case studies. 32 of allowing the qualitative data to explore complex, unex- 33 pected or contradictory quantitative data collected from Sampling

34 http://bmjopen.bmj.com/ the tool and adoption questionnaires. A constant compari- Data will be collected for all practices. 35 son and deviant case approach will be adopted [16]. Con- 36 stant comparison will allow the researcher to test the Data collection 37 38 emerging hypothesis by comparing different case-study Multiple datasets will be collected and linked. Practice 39 practices. Any deviant examples from the emergent hy- characteristics will be measured using routinely available 40 pothesis will be examined. Analysis will initially be simul- data including list size, rurality , deprivation, proportion 41 taneous with data collection to allow the emerging of older patients, postgraduate training status, dispens- 42 analysis to be tested and revised in subsequent data sam- ing status, contract type (nGMS, section 17c/2c), Health 43 pling, collection and analysis. The framework [17] analysis Board, Community Health Partnership, overall QOF on October 3, 2021 by guest. Protected copyright. 44 technique will be used to support within-case and cross- clinical performance, and QOF performance on relevant 45 case analysis. medicines management indicators (Medicines 6, 10, 11, 46 12){National Institute of Clinical Excellence, 2012}. Base- 47 Methods for research question 2 line levels of the high risk NSAID and antiplatelet 48 Study design prescribing being targeted will also be available for par- 49 The design is that of a focus group study involving a ticipating practices. During the recruitment of clusters 50 sample of patients targeted by the intervention. we will document the trial recruitment process and care- 51 fully record those practices that are approached, those 52 Sampling that respond and those that are recruited. Participating 53 Four to six of the case-study practices will invite patients and non-participating practices will be compared using 54 who have been targeted by the intervention to attend a available practice characteristic data. Adoption will be 55 focus group. measured using a quantified assessment of engagement 56 with the education outreach based on which practice 57 Data collection members attended and field notes of the interaction, and 58 One focus group per practice will be held. The focus by a survey instrument developed for the trial and com- 59 group will have two stages. First, the patients will be pleted at trial start and after six to nine months based 60 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml BMJ Open Page 50 of 61

Grant et al. Trials 2012, 13:154 Page 5 of 6 BMJ Open: first published as 10.1136/bmjopen-2016-015281 on 10 March 2017. Downloaded from 1 http://www.trialsjournal.com/content/13/1/154 2 3 4 5 on NPT. [12] The adoption questionnaire is based on study, and will provide the flexibility either to 6 the four domains of NPT, with the baseline survey fo- identify key processes on the assumed pathway 7 cusing on coherence and cognitive participation, and the quantitatively or to identify unexpected processes 8 follow-up survey collecting additional data on collective that are not in the logic model, or both. 9 action and reflective monitoring. Reach will be measured 3. The quantitative study will test hypotheses generated 10 as the proportion of patients identified by the informat- from the qualitative analysis, and help inform 11 ics tool who are actually reviewed. Delivery to the pa- judgments about generalization of the case and 12 tient will be measured in terms of patterns of review focus group studies. 13 recorded in the tool (which measures were focused on, 4. Overall synthesis will draw on findings from all three 14 how they were carried out [records review, face-to-face studies, which will be used to construct a narrative 15 consultation, telephoneFor consultation]), peer the decisions reviewsynthesis ofonly the process of implementation of the 16 made (the proportion of patients who decide to con- intervention, to contextualize the main trial findings 17 tinue, the proportion who decide to continue with gas- (whether positive or negative) and to inform 18 tric protection and the proportion who decide to stop judgments about external validity and generalizability 19 the drug). Maintenance will be measured in terms of , real-world implementation if appropriate, and 20 how reach changes over time. Effectiveness will be mea- further intervention development [18]. 21 sured in terms of the trial primary outcome (the high 22 risk NSAID/antiplatelet composite) and the two second- Discussion 23 ary outcomes of ‘repeated’ and ‘new’ prescribing (since We have piloted these data collection methods in four 24 the intervention targets the review of repeated prescrib- general practices and in six focus groups with patients 25 ing, with an expectation that the experience of reviewing from these practices and with members of the public. 26 will reduce new prescribing as well, but that this may These methods have been acceptable to practitioners 27 vary by practice and particularly by practice size). and patients and have generated sufficient data to feas- 28 ibly evaluate this complex intervention trial. The pilot 29 Data analysis has influenced the focus and design of this evaluation 30 Initial descriptive analysis will use practice characteristics and the associated topic guides and questionnaires. 31 data to compare participating practices with the wider As in any study, choices have been made in the design 32 population of practices in the two Boards and nationally of this process evaluation protocol that balance ideal re- 33 (to assess the representativeness of recruited practices and search questions with feasibility and resource constraints.

34 http://bmjopen.bmj.com/ the implications for generalizability). The overall extent of [19] We believe that a prior design of the process evalu- 35 reach, delivery to the patient and maintenance will then ation that maps to expectations about how the interven- 36 be examined, and univariate and multivariate associations tion will work strengthens the study. It is also important, 37 38 with practice characteristics and adoption will be exam- however, to retain the flexibility needed for examining the 39 ined using cross-tabulations, comparison of means, and unexpected and the unintended. From that perspective, a 40 logistic/linear regression as appropriate to the data. The mixed-methods evaluation allows the combination of 41 extent of variation between practices in the three specified exploratory and flexible qualitative work, and more pre- 42 measures of effectiveness will be examined using multilevel specified quantitative analysis, with each method contrib- 43 logistic regression, and associations between effectiveness uting to the design, implementation and interpretation of on October 3, 2021 by guest. Protected copyright. 44 and practice characteristics, adoption, reach, delivery to the other [20]. Design is always constrained by the 45 the patient,andmaintenance will be examined. resources available though, and although we have been 46 fortunate in being relatively well-resourced for this elem- 47 Synthesizing results from all three studies ent of the project, our chosen case-study method does 48 Synthesis across the three studies will have several mean having to sample a relatively small number of prac- 49 elements: tices to carry out the qualitative work. Inevitably though, 50 extending the interviewing to a larger number of practices 51 1. The quantitative study will inform case-study would involve less depth in each practice, and we chose to 52 sampling and will help contextualize sampled prioritize more depth in a case-study approach using mul- 53 settings. tiple interviews in selected practices. 54 2. The case and patient focus group studies will provide We have also chosen to conduct a parallel process 55 a rich understanding of how the intervention was evaluation, where data collection is simultaneous with trial 56 perceived and implemented, and how actual implementation [21]. A key advantage is that data collec- 57 implementation related to our model of how the tion is contemporaneous with the practices actually doing 58 intervention was intended to work. These data will the work of implementation, but a potential disadvantage 59 generate hypotheses for testing in the quantitative is the risk that the case-study practices vary little in their 60 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml Page 51 of 61 BMJ Open

Grant et al. Trials 2012, 13:154 Page 6 of 6 BMJ Open: first published as 10.1136/bmjopen-2016-015281 on 10 March 2017. Downloaded from 1 http://www.trialsjournal.com/content/13/1/154 2 3 4 5 overall effectiveness in delivering main trial outcomes. A Received: 4 May 2012 Accepted: 26 July 2012 Published: 28 August 2012 6 post-hoc design that samples based on main trial out- 7 comes would avoid this risk, but at the cost of being References 8 designed after trial effectiveness is known, and conducted 1. Medical Research Council: Developing and evaluating complex interventions; long after the trial completes. Overall, we judged a parallel new guidance. London: Medical Research Council; 2008. 9 2. Dreischulte T, Grant A, Donnan P, McCowan C, Davey P, Petrie D, Treweek S, 10 process evaluation to be more likely to produce robust Guthrie B: A cluster randomised stepped wedge trial to evaluate the 11 findings, and although the protocol is pre-specified, the effectiveness of a multifaceted information technology-based methods proposed are flexible to unexpected findings in intervention in reducing high-risk prescribing of non-steroidal anti- 12 inflammatory drugs and antiplatelets in primary care: the DQIP study 13 that qualitative data collection and analysis is iterative in protocol. Implement Sci 2012, 7:24. 14 nature, and will inform the choice of the actual hypotheses 3. Grant A, Treweek S, Dreischulte T, Foy R, Guthrie B: Process evaluations for 15 to be tested quantitatively.For A final peer potential weakness review is cluster-randomised only trials of complex interventions: a proposed framework for design and reporting. Trials, . in press. 16 that some of the quantitative data is relatively thin. For ex- 4. Pirmohamed MJ, Meakin S, Green C, Scott AK, Walley T, Farrar K, Park BK, 17 ample, the data on the nature of review carried out is col- Breckenridge AM: Adverse drug reactions as a cause of admission to 18 lected routinely in the DQIP informatics tool but lacks hospital: prospective analysis of 18 820 patients. BMJ 2004, 329:15–19. 5. Howard RL, Avery AJ, Slavenburg S, Royal S, Pipe G, Lucassen P, 19 contextual information or much detail. This reflects a Pirmohamed M: Which drugs cause preventable admissions to hospital? 20 trade-off between being able to collect data in all practices A systematic review. Br J Clin Pharmacol 2007, 63:136–147. 21 and avoiding burdening practices with data collection that 6. Brown C, Lilford R: The stepped wedge trial design: a systematic review. 22 BMC Med Res Methodol 2006, 6:54. has no clinical purpose. 7. Hussey MA, Hughes JP: Design and analysis of stepped wedge cluster 23 Our study offers the opportunity to explain why or randomized trials. Contemp Clin Trials 2007, 28:182–191. 24 why not the DQIP intervention reduced high risk pre- 8. Nazareth I, Freemantle N, Duggan C, Mason J, Haines A: Evaluation of a 25 complex intervention for changing professional behaviour: The evidence scribing. If the intervention does not reduce high-risk based out reach (EBOR) trial. J Health Serv Res Policy 2002, 7:230–238. 26 prescribing as intended, then the analysis will help iden- 9. Mason J: Sampling and selection in qualitative research. In Qualitative 27 tify why this happened in terms of where the assump- Researching. London: Sage Publications Ltd; 2002:120–144. 10. Lewin S, Glenton C, Oxman AD: Use of qualitative methods alongside 28 tions of the logic model broke down, or call into 29 randomised controlled trials of complex healthcare interventions: question the underlying model. If the intervention does methodological study. BMJ 2009, 339:b3496. 30 reduce high-risk prescribing, then the process evaluation 11. Hoddinott P, Britten J, Pill R: Why do interventions work in some places and not others: a breastfeeding support group trial. Soc Sci Med 2010, 31 will provide information to inform judgments about 32 70:769–778. likely generalizability, and identify if improvement was 12. May C, Finch T: Implementing, Embedding, and Integrating Practices: An 33 general or restricted to some practices which may influ- Outline of Normalisation Process Theory. Sociology 2009, 43:535. 13. Stake R: The art of case study research. Thousand Oaks, California: Sage

34 http://bmjopen.bmj.com/ ence real-world implementation or identify targets for a 35 Publications Ltd; 1995. modified intervention. 14. Denzin NK: Moments, mixed methods, and paradigm dialogs. Qualtiative 36 Inquiry 2010, 16:416. 37 15. Cresswell JW, Plano CVL: Designing and conducting mixed methods research. Trial Status 38 Thousand Oaks: Sage Publications Ltd; 2007. Active. Trial started 31st October 2011. 16. Silverman D: Interpreting qualitative data. 3rd edition. London: Sage 39 Publications Ltd; 2006. 40 Abbreviations 17. Ritchie J, Spencer L, O’Connor W: Carrying out Qualitative Analysis.In 41 DQIP: data-driven quality improvement in primary care; GP: general Qualitative Research Practice: A guide for Social Science Students and Researchers. 42 practitioner; NPT: Normalization Process Theory; NSAIDS: non-steroidal anti- Edited by Ritchie J, Lewis J. London: Sage Publications Ltd; 2003. inflammatory drugs; QOF: Quality and Outcomes Framework. 18. Dixon-Woods M, Agarwal S, Jones D, Young B, Sutton A: Synthesising on October 3, 2021 by guest. Protected copyright. 43 qualitative and quantitative evidence: a review of possible methods. – 44 Competing interests J Health Serv Res Policy 2005, 10:45 53. 45 The author(s) declare that they have no competing interests. 19. Francis J, Eccles M, Johnston M, Whitty P, Grimshaw J, Kaner E, Smith L, Walker A: Explaining the effects of an intervention designed to promote 46 evidence-based diabetes care: a theory-based process evaluation of a Authors’ contributions 47 pragmatic cluster randomised controlled trial. Implement Sci 2008, 3:50. AG and BG were involved in the initial conceptualization and study design. 20. Protheroe J, Bower P, Chew-Graham C, Protheroe J, Bower P, Chew-Graham 48 AG and BG developed the data collection methods; AG piloted these with C: The use of mixed methodology in evaluating complex interventions: 49 the help of TD. AG and BG wrote the paper. All authors read and approved identifying patient factors that moderate the effects of a decision aid. the final manuscript. AG is responsible for this manuscript. 50 Fam Pract 2007, 24:594–600. 51 21. Craig P, Dieppe P, Macintyre S, Michie S, Nazareth I, Petticrew M: Acknowledgements 52 Developing and evaluating complex interventions: the new Medical AG and TD are funded by the Scottish Government Health Directorates Chief Research Council guidance. BMJ 2008, 337:a1655. 53 Scientist Office Applied Programme Research Grant 07/02 for the DQIP prescribing improvement study. This funding body has had no role in the 54 doi:10.1186/1745-6215-13-154 design and conduct of, or decision to publish this work. 55 Cite this article as: Grant et al.: Study protocol of a mixed-methods evaluation of a cluster randomized trial to improve the safety of NSAID 56 Author details 1 and antiplatelet prescribing: data-driven quality improvement in 57 Quality, Safety and Informatics Research Group, Population Health Sciences, primary care. Trials 2012 13:154. 2 58 University of Dundee, Mackenzie Building, Dundee DD2 4BF, UK. Medicines Management Unit, NHS Tayside, c/o University of Dundee, Mackenzie 59 Building, Dundee DD2 4BF, UK. 60 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml BMJ Open Page 52 of 61

Grant et al. Trials 2013, 14:15 BMJ Open: first published as 10.1136/bmjopen-2016-015281 on 10 March 2017. Downloaded from 1 http://www.trialsjournal.com/content/14/1/15 TRIALS 2 3 4 5 METHODOLOGY Open Access 6 7 8 9 Process evaluations for cluster-randomised trials 10 11 of complex interventions: a proposed framework 12 13 for design and reporting 14 15 Aileen Grant1*, ShaunFor Treweek1, Tobiaspeer Dreischulte 2review, Robbie Foy3 and Bruce only Guthrie1 16 17 18 Abstract 19 20 Background: Process evaluations are recommended to open the ‘black box’ of complex interventions evaluated in 21 trials, but there is limited guidance to help researchers design process evaluations. Much current literature on 22 process evaluations of complex interventions focuses on qualitative methods, with less attention paid to 23 quantitative methods. This discrepancy led us to develop our own framework for designing process evaluations of 24 cluster-randomised controlled trials. 25 Methods: We reviewed recent theoretical and methodological literature and selected published process 26 evaluations; these publications identified a need for structure to help design process evaluations. We drew upon 27 this literature to develop a framework through iterative exchanges, and tested this against published evaluations. 28 Results: The developed framework presents a range of candidate approaches to understanding trial delivery, 29 intervention implementation and the responses of targeted participants. We believe this framework will be useful to 30 31 others designing process evaluations of complex intervention trials. We also propose key information that process 32 evaluations could report to facilitate their identification and enhance their usefulness. 33 Conclusion: There is no single best way to design and carry out a process evaluation. Researchers will be faced with choices about what questions to focus on and which methods to use. The most appropriate design depends

34 http://bmjopen.bmj.com/ 35 on the purpose of the process evaluation; the framework aims to help researchers make explicit their choices of 36 research questions and methods. 37 Trial registration: Clinicaltrials.gov NCT01425502 38 39 Keywords: Process evaluation, Complex intervention, Cluster-randomised controlled trial, Qualitative, Quantitative, 40 Reporting 41 42

Background ences in context, targeted groups and the intervention ac- on October 3, 2021 by guest. Protected copyright. 43 Many interventions to improve healthcare delivery and tually delivered [2,3]. The UK Medical Research Council 44 health are complex in the sense that they possess several guidance for developing and evaluating complex interven- 45 interacting components [1]. Randomised controlled trials tions recommends conducting a process evaluation to 46 of such interventions are often criticised as being ‘black ‘explain discrepancies between expected and observed 47 box,’ since it can be difficult to know why the intervention outcomes, to understand how context influences out- 48 ’ 49 worked (or not) without examining underlying processes. comes, and to provide insights to aid implementation [1]. 50 As a result, reported effects of what appear to be similar Although our focus is on randomised trials, it is important 51 interventions often vary across studies because of differ to recognise that similar issues arise in the evaluation of 52 public health interventions which may use a range of dif- ferent evaluative designs [4-6]. 53 * Correspondence: [email protected] 54 1Quality, Safety and Informatics Research Group, Population Health Sciences, In complex intervention trials, the intervention to be 55 Medical Research Institute, University of Dundee, Mackenzie Building, evaluated may be delivered at different levels: the inter- Dundee DD2 4BF, UK vention can be delivered straight to the patient, such as 56 Full list of author information is available at the end of the article 57 58 © 2013 Grant et al.; licensee BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and 59 reproduction in any medium, provided the original work is properly cited. 60 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml Page 53 of 61 BMJ Open

Grant et al. Trials 2013, 14:15 Page 2 of 10 BMJ Open: first published as 10.1136/bmjopen-2016-015281 on 10 March 2017. Downloaded from 1 http://www.trialsjournal.com/content/14/1/15 2 3 4 5 in trials evaluating the effectiveness of body weight and evaluations to enhance their identification and usefulness 6 physical activity interventions on adults [7]; the interven- to other researchers. 7 tion can be delivered to the healthcare professional, for 8 example in trials of decision support tools [8]; or the inter- Methods 9 vention can be targeted at the organisational level, such Existing literature 10 as in a recent trial testing a pharmacist-led information- To support the design of our own process evaluation, we 11 technology-based intervention aimed at optimising medi- wanted to identify relevant methodological and theore- 12 cation safety systems in general practice [9] (adapted from tical literature. We did not aim for a comprehensive 13 the Medical Research Council guidance 2000) [10]. review of all such studies but rather sought to gain a 14 Process evaluations are studies that run parallel to or general understanding of this literature. We were par- 15 follow intervention trialsFor to understand peer the trial review pro- ticularly interested only in the purpose and the design of 16 cesses or underlying mechanisms in relation to context, process evaluations of cluster-randomised controlled 17 setting [11], professionals and patients [12]. These eva- trials. Our initial search (which included the terms 18 luations provide explanations for the trial results and ‘process evaluation’ or ‘qualitative’ or ‘quantitative’ and 19 enhance understanding on whether or how interventions ‘complex intervention’ or ‘randomised controlled trial’) 20 could move from research to practice. in Medline, Embase, Psycinfo, Cochrane Central Register 21 We planned to conduct a parallel process evaluation of Control Trials and Cochrane Database of Systematic 22 of a cluster-randomised trial of a complex intervention Reviews was informative but identified studies that 23 to improve prescribing safety [13] targeted at general tended to focus on a specific aspect of process evalua- 24 practices. Although there is clear published guidance for tion design or the nature of process evaluations and 25 designing trials where clusters rather than individuals qualitative methods. There was little guidance on how to 26 are the unit of randomisation [14,15], we found the design a process evaluation. We turned to process evalua- 27 Medical Research Council guidance of limited help in tions from cluster-randomised controlled trials identified 28 designing the process evaluation of our trial. by our search from which we could learn. We designed a 29 In our trial, the research team delivers an intervention data extraction form based on the Reach, Efficacy, Adop- 30 to general practices – with the intention of prompting tion, Implementation and Maintenance (RE-AIM) frame- 31 those practices to review patients with particular types work [19] and Northstar [20] to summarise identified 32 of high-risk prescribing (nonsteroidal anti-inflammatory reports of process evaluations in terms of: the aim of the 33 drugs or antiplatelets), to carefully reconsider the risks work; the timing (parallel to the trial or post hoc); its scope

34 http://bmjopen.bmj.com/ and benefits of the offending drugs in each patient and with reference to each RE-AIM construct (reach, efficacy, 35 to take corrective action where possible [13]. There is adoption, implementation and maintenance) and the 36 therefore an intervention delivered to practices (the methods used; and trial and process evaluation findings 37 38 cluster) that hopes to change the behaviour of practi- and how they were reported. This form was systematically 39 tioners. This approach is similar to most trials of audit populated with information from all identified process 40 and feedback [16] and to many trials involving a profes- evaluations by AG, with a second review of each by either 41 sional training intervention, as in the OPERA trial where ST, RF or BG. Disagreements between reviewers were col- 42 a training intervention is delivered to nursing home staff lectively discussed and resolved, and general strengths and 43 and physiotherapists who deliver an intervention of limitations in design and reporting were identified. on October 3, 2021 by guest. Protected copyright. 44 physical activity and physiotherapy to residents of resi- We initially set out to design a critical appraisal form 45 dential and nursing homes [17,18]. In other trials, how- but there was wide variation in the reporting of process 46 ever, the intervention is delivered to individuals by the evaluations that made consistent critical appraisal diffi- 47 research team themselves, as in a trial of body weight cult. We struggled to make judgements based on the 48 and physical activity for adults at risk of colorectal aden- breadth of information reported in our data extraction 49 omas [7]. Process evaluations therefore need to be tai- form because we found studies that chose to report one 50 lored to the trial, the intervention and the outcomes process in detail but did say if other processes had been 51 being studied, but we could not find clear guidance in the considered and an explicit judgement had been made 52 literature on how to do this. Additionally, we noted wide not to evaluate them [21]. Some general features of this 53 variations in the aims, methods, timing and reporting of literature are summarised below. 54 process evaluation studies that made it difficult to draw 55 on them in designing our own process evaluation. Results and discussion 56 We therefore developed a framework to guide the Process evaluation purpose 57 design of our own process evaluation. We believe it may The generally agreed purpose of process evaluations of 58 be of help to other cluster triallists faced with a similar trials of complex and public health interventions is to 59 task. We also make proposals for the reporting of process understand the effects (or not) of interventions [2-4]. 60 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml BMJ Open Page 54 of 61

Grant et al. Trials 2013, 14:15 Page 3 of 10 BMJ Open: first published as 10.1136/bmjopen-2016-015281 on 10 March 2017. Downloaded from 1 http://www.trialsjournal.com/content/14/1/15 2 3 4 5 These effects can help to inform judgements about the va- Framework for designing process evaluations of cluster- 6 lidity by delineating key intervention components (con- randomised trials of complex interventions 7 struct validity), and by demonstrating connections between In developing our framework we drew upon the general li- 8 the intervention and outcomes (internal validity) and be- terature on process evaluation and the RE-AIM framework 9 tween the intervention and other contexts (external vali- [3,11,19,23,26,27,31], and on published process evaluations 10 dity) [4,22]. Implementation and change processes can be [11,12,21,29,32,33]. The framework was developed by all 11 explored [1,23], and factors associated with variations in the authors through iterative meetings, redefining and 12 effectiveness can be examined [24]. Process evaluations can working with the concepts and testing this work against 13 also examine the utility of theories underpinning interven- published evaluations. 14 tion design [25] and generate questions or hypotheses for Cluster-randomised designs are utilised when individual 15 future research. ProcessFor evaluations peer can therefore fulfilreview a randomisation isonly not possible, feasible or appropriate, 16 variety of purposes, including understanding intervention mainly because of potential contamination between the 17 effects, potential generalisability and optimisation in routine intervention and control arms [34]. Following the most 18 practice [26]. recent CONSORT extension for cluster-randomised trials 19 [28], we distinguish between individuals in the target 20 Process evaluation design population on whom outcome data are collected and 21 Published papers on trial process evaluation design largely ‘clusters’, which are some larger social unit within which 22 focus on qualitative research methods [3,23,27], but many individuals are nested and where the cluster is the unit of 23 of the purposes identified above can be assessed both quan- randomisation. Clusters may be defined by geography, 24 titatively and qualitatively; indeed, process evaluation of institution, teams or services within institutions, or by 25 public health interventions often explicitly uses mixed individual professionals delivering a service to patients. 26 methods[4].Thefocusonqualitativemethodsintrialsmay Although outcomes are collected on an individual level, 27 occur because many routinely collect some quantitative the primary unit of inference of the intervention may be 28 process data measures to fulfil Consolidated Standards of the individual or the cluster, depending on the purpose of 29 Reporting Trials (CONSORT) requirements or as secon- the study and the primary outcome measures. The focus 30 dary outcomes, rather than as part of a distinct process of a process evaluation will vary with the nature of the 31 evaluation. For example, data on recruitment are likely to trial and resources available, so although several key pro- 32 be collected and reported as part of the main trial because cesses are candidates for examination in any evaluation of 33 reporting these data is a CONSORT requirement [28], but cluster trials (Figure 1), the importance of studying them

34 http://bmjopen.bmj.com/ for many trials a more detailed examination of the process will vary between trials. Also important to note is the fact 35 of recruitment may be useful, to inform interpretation of that although the emphasis in what follows is on processes 36 trial results and generalisability. in clusters that receive the intervention, it will usually be 37 38 equally important to understand relevant processes in 39 Process evaluation reporting control or usual care clusters. 40 We found it difficult to systematically identify process eva- 41 luations of complex intervention trials, and the reporting Recruitment of clusters (areas, institutions, individual 42 of important information was also variable. Although professionals) 43 some papers were easily identifiable and had clearly stated How clusters are sampled and recruited is important to on October 3, 2021 by guest. Protected copyright. 44 purposes and methods [21,29,30], others: help understand how generalisable the findings are likely 45 to be. At a minimum, recruitment is likely to be moni- 46  were hard to identify because of a lack of explicit tored to fulfil CONSORT requirements for reporting 47 labelling: findings of the main trial [28], but a more in-depth 48  lacked clarity about their overall purpose or did not understanding of why clusters participate (or not) can 49 state specific objectives: inform the interpretation and implementation of trial 50  did not clearly state whether the evaluation was pre- findings. 51 planned and carried out in parallel to the main trial 52 or was conducted post hoc: Delivery to clusters 53  did not provide an explicit statement of the main In some trials, the research team or specially trained 54 trial findings, making it difficult to judge the professionals may deliver a complex intervention to in- 55 relevance and implications of the process evaluation. stitutional or individual professional clusters, which are 56 then expected to change their practice or service deli- 57 Overall, although the existing literature was often very in some way that impacts on the individuals in the 58 thought-provoking, it did not provide an over-arching target population. Understanding the nature of cluster- 59 framework for designing process evaluations. level interventions and variations in their delivery may 60 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml Page 55 of 61 BMJ Open

Grant et al. Trials 2013, 14:15 Page 4 of 10 BMJ Open: first published as 10.1136/bmjopen-2016-015281 on 10 March 2017. Downloaded from 1 http://www.trialsjournal.com/content/14/1/15 2 3 4 5 6 7 8 9 10 11 12 13 14 15 For peer review only 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 Figure 1 Framework model for designing process evaluations of cluster-randomised controlled trials.

34 http://bmjopen.bmj.com/ 35 36 help explain variation in outcomes between organisa- proportion of the population may receive a public health 37 38 tional clusters. intervention [4]. As well as influencing effectiveness, 39 if high or representative recruitment and reach is not 40 Response of clusters achieved within clusters in the trial, then the population 41 Response is the process by which clusters such as institu- impact of implementation in routine practice is likely to 42 tions or individual professionals integrate and adopt new be limited. 43 trial-related or intervention-related work into their existing on October 3, 2021 by guest. Protected copyright. 44 systems and everyday work, and will be influenced by how Delivery to individuals 45 the research team engages and enrols the clusters and by In some studies, a complex intervention is delivered to 46 the extent to which the work is perceived to be legitimate the target population of individuals. Such interventions 47 within organisations or by individual professionals [35]. may be tightly defined by the researchers [36], in which 48 case evaluation of delivery to individuals in the target 49 Recruitment and reach of individuals population may focus on how closely the intervention 50 Interpretation of the intervention’s measured effective- delivered matches what the researchers intended (usually 51 ness in the trial will depend on recruitment and reach, called the fidelity of the intervention) and the impact of 52 in terms of the proportion of the target population that deviation on effectiveness. In some trials, such as those 53 actually receives the intervention, and how representa- on guideline implementation strategies, the intervention 54 tive they are. Variation in the response of clusters will is not directly delivered to individuals (patients) but in- 55 affect how clusters identify and enrol these individuals. stead aims to change professional behaviour. The nature 56 When recruitment of individuals is carried out by orga- of the changes to work patterns or professional beha- 57 nisational or individual professional clusters, there is viour at the individual or patient level can be examined 58 potential for selection bias [34] – although similar issues to better understand why it succeeded or failed. For ex- 59 arise in clusters defined by geography, where a variable ample, in our trial the Data-driven Quality Improvement 60 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml BMJ Open Page 56 of 61

Grant et al. Trials 2013, 14:15 Page 5 of 10 BMJ Open: first published as 10.1136/bmjopen-2016-015281 on 10 March 2017. Downloaded from 1 http://www.trialsjournal.com/content/14/1/15 2 3 4 5 in Primary Care (DQIP) complex intervention is targeted at Context 6 general practitioners to improve prescribing safety. This Irrespective of which specific processes are examined, a 7 intervention prompts practitioners to carry out a targeted careful description of the context in which the trial is 8 medication review based on a guideline provided to them, embedded will usually be helpful for interpreting the 9 but practitioners are free to review medication in any way findings, including how generalisable any findings are 10 they judge appropriate [13]. Understanding how practi- [39]. Additionally, trials are often conducted in different 11 tioners actually carry out these medication reviews is there- contexts – for example, within different health boards or 12 fore likely to be important to explaining the trial results. different countries – and so consideration of these local 13 contexts may be important. Each of the concepts in the 14 Response of individuals model has contextual factors that may act as barriers 15 The effectiveness of manyFor but not allpeer interventions willreview be and/or facilitators only to implementation. For example, the 16 mediated by the responses of the targeted individuals, par- DQIP trial is delivered in two health boards within 17 ticularly where effectiveness is strongly influenced by ad- Scotland that have different pre-existing levels of pre- 18 herence or requires behaviour change in those individuals. scribing improvement support for practices, and this 19 Similar to understanding the response of organisational or support has a different focus in each board. As well as 20 individual professionals to interventions directed at them, the wider context of the organisation of primary health- 21 it may be helpful to draw on prior [37] or other relevant care in the United Kingdom being important, these local 22 theory [38] to examine individual responses. contextual factors may modify how practices respond to 23 the same intervention [13,40]. 24 Maintenance 25 Understanding whether and how each of the above pro- Applying the framework to different types of trial 26 cesses is maintained or sustained over time is important, es- As described above, cluster-randomised trials vary 27 pecially for studies of longer duration. Organisational greatly in their design, including the level at which inter- 28 clusters or individuals may cease to participate completely ventions are targeted. The framework can therefore be 29 or the way in which they adopt and respond to the inter- seen as a set of candidate elements for designers of 30 vention may change over time, such as with declining reach process evaluations to consider. The relevance and rela- 31 or changes in the nature of the intervention delivered. tive importance of candidates will vary between trials, 32 the aim and objectives of the evaluation and whether the 33 Effectiveness authors plan to pre-specify their data collection and ana-

34 http://bmjopen.bmj.com/ Although pre-specified measurement of effectiveness is the lysis (potentially allowing hypothesis testing) or conduct 35 primary function of the main trial, the process evaluation the evaluation post hoc (for example, depending on the 36 can examine associations between trial processes and experience of conducting the trial or on the trial results, 37 – 38 effects on primary and secondary outcomes recognising in which case the evaluation can only be hypothesis 39 that these are exploratory analyses. This can help explain generating). Inevitably, resource limitations will often 40 why an intervention did or did not work, and can explore require a focus on elements that are considered most 41 variations in effectiveness between clusters that may be im- critical in the context being evaluated. In such cases, 42 portant in planning more widespread implementation. emphasis should be placed on collecting quality data for 43 a few key processes rather than collecting a lot of data on October 3, 2021 by guest. Protected copyright. 44 Unintended consequences for each candidate process. The following section dis- 45 All interventions have the potential to produce unintended cusses how the framework may be applied. 46 consequences, which may be beneficial or harmful. Process 47 evaluations provide an opportunity to systematically iden- Research methods for process evaluations of cluster- 48 tifyandquantifyunexpectedunintendedoutcomes. randomised trials 49 The appropriate methods for a process evaluation will 50 Theory depend on the intervention being evaluated and the aims 51 If theory has been used to develop the intervention, then of the evaluation. Table 1 presents examples of research 52 this should be made explicit, and the process evaluation questions that could be asked within each domain and 53 can examine whether predicted relationships and sequences methods to answer these questions. Both qualitative and 54 of changes happen during implementation. Causal model- quantitative methods can be appropriate depending on 55 ling based on psychological theory is an example that the questions asked. Quantitative data collection will be 56 would apply to ‘delivery to the individual’ and ‘response of required if it is judged that all clusters should be evalua- 57 individuals’ [37]. Even if interventions do not have a strong ted in large multisite trials [3], whereas sampling may be 58 theoretical base, then it may be useful to draw on relevant necessary for more in-depth qualitative evaluation. 59 theory to help understand their effects [35]. Mixed methods can add complementary insights. For 60 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml Page 57 of 61 BMJ Open

Grant et al. Trials 2013, 14:15 Page 6 of 10 BMJ Open: first published as 10.1136/bmjopen-2016-015281 on 10 March 2017. Downloaded from 1 http://www.trialsjournal.com/content/14/1/15 2 3 4 5 Table 1 Example research questions and methods for evaluating each process 6 Domain Example research questions Research methods that could be applied Probable best stage of study to 7 collect data 8 Recruitment of How are clusters sampled and recruited? Documentation of recruitment process by research team. Pre-intervention 9 clusters Who agrees to participate? Quantitative comparison of recruited and nonrecruited 10 clusters. 11 Why do clusters agree to participate (or not) Qualitative analysis of interviews with cluster gatekeepers or 12 members. 13 Delivery to What intervention is actually delivered for each Qualitative analysis of observational, interview and Pre-intervention 14 clusters cluster? Is it the one intended by the documentary data relating to the cluster-level intervention. and early 15 researchers?For peer review only intervention 16 Response of How is the intervention adopted by clusters? Quantitative data measuring cluster members’ perceptions Pre-intervention 17 clusters of the intervention and uptake of trial components. and early Qualitative analysis of observational, interview and intervention 18 documentary data about how clusters adopt the 19 intervention. 20 Recruitment Who actually receives the intervention in each Measurement of receipt in target population. Quantitative During 21 and reach in setting? Are they representative? comparison of those receiving vs. not receiving the intervention 22 individuals intervention. 23 Why do clusters achieve the pattern of reach Qualitative analysis of observational, interview and 24 they do? Do they introduce selection bias? documentary data about how clusters achieve reach. 25 Delivery to What intervention is actually delivered for each Qualitative analysis of observational, interview and During individuals cluster? documentary data about what intervention is delivered and intervention 26 why. 27 Is the delivered intervention the one intended Measurement of intervention fidelity across its components. 28 by the researchers? 29 Response of How does the target population respond? Qualitative analysis of observational and interview data During 30 individuals about target population’s experience of and response to intervention and 31 the intervention. post-intervention 32 Maintenance How and why are these processes sustained Any of the above, but probably focused on processes During 33 over time (or not)? identified as critical, or as likely to be difficult to sustain. intervention and post-intervention

34 http://bmjopen.bmj.com/ 35 Unintended Are there unintended changes in processes and Qualitative analysis of observational and interview data for Intervention and consequences outcomes, both related to the trial intervention identification. Quantitative data collection for potential post-intervention 36 and unrelated care? unintended consequences during the trial, or use of routine 37 datasets. 38 Theory What theory has been used to develop the Quantitative process data analysis can assess whether Post-intervention 39 intervention? predicted relationships and sequences of change happened 40 during implementation. 41 Context What is the wider context in which the trial is Qualitative data collection from policy documents or Pre-intervention 42 being conducted? interviews. and early

intervention on October 3, 2021 by guest. Protected copyright. 43 44 45 example, how clusters vary in their response and delivery [11,35]. Even without explicit use of theory, however, a 46 of an intervention to the target population could be process evaluation can productively draw on the resear- 47 examined using qualitative analysis of interview and chers’ implicit models of how an intervention is expected 48 observational data, with variation in outcome across to work [30,41]. Although the right design and method will 49 clusters examined quantitatively. Similarly, a study that therefore depend on the purpose of the evaluation, which 50 seeks to explore individuals’ receipt of and response to will vary with the trial design and context, it is critical that 51 an intervention can initially use quantitative data to the choices made and the rationale behind them is made 52 inform sampling strategies and qualitative methods to explicit. 53 explore different experiences (potentially at different 54 stages of implementation). For some purposes, explicitly Reporting process evaluations 55 theoretical approaches to design and analysis will be The framework systematically identifies cluster-randomised 56 appropriate, such as the use of psychological theory to trial processes that are candidates for evaluation. In prac- 57 understand individual behaviour [21], or sociological tice, choices will often have to be made that balance the 58 approaches to understanding response in different contexts, ideal with the feasible, focusing attention and available 59 such as realistic evaluation or normalisation process theory resources on key research questions. More systematic 60 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml BMJ Open Page 58 of 61

Grant et al. Trials 2013, 14:15 Page 7 of 10 BMJ Open: first published as 10.1136/bmjopen-2016-015281 on 10 March 2017. Downloaded from 1 http://www.trialsjournal.com/content/14/1/15 2 3 4 5 Table 2 Mapping of the reporting framework to three selected process evaluations 6 Summary of trial Clearly Stated purpose Processes Specify Methods used Choice Report being evaluated labelled examined timing of main 7 as a method findings of 8 process justified trial 9 evaluation 10 Nazareth Cluster- Yes ’To describe the Cluster Retrospective/ Reporting of Partly Trial design, 11 and randomised trial steps leading to recruitment, post hoc proportion of intervention, colleagues of a pharmacist- the final primary Delivery to practices recruited. targeted 12 [30] led educational outcome and clusters, Adoption Association outcomes 13 outreach explore the effect and Delivery to between and results 14 intervention to of the intervention target population, proportion of GPs summarised, improve GPs’ on each step of Quantitative in each practice main trial 15 prescribing qualityFor peerthe hypothesised reviewassociations with onlyattending paper 16 pathway of effectiveness education referenced 17 change in (reported in main outreach, the 18 professionals’ trial paper) intervention and prescribing change in 19 behaviour’ prescribing. Mixed- 20 methods 21 assessment of barriers and 22 facilitators to 23 adoption and 24 delivery to target 25 population. 26 Byng and Cluster- Yes ’To unpick the Adoption, Reach Retrospective Realistic Yes Trial design, colleagues randomised trial complexity of the and Delivery to data evaluation, intervention 27 [11] of a complex intervention by target population, collection, qualitative case and results 28 intervention to examining Qualitative unclear if study, analysis of summarised, 29 promote shared interactions associations with planned interview data main trial 30 care for people between effectiveness prospectively with a purposive paper with severe components and sample of referenced 31 mental health context and then participating 32 problems further defining its mental health 33 core functions’ team-practice cases. Case study

34 findings were used http://bmjopen.bmj.com/ 35 to better 36 understand how 37 the intervention changed practice 38 and targeted 39 outcomes. 40 Fretheim Cluster- Yes ’The main Delivery to Prospective / Quantification of Partly Trial design, 41 and randomised trial objective of this clusters, Adoption pre-specified GP perceptions of intervention, 42 colleagues of a multifaceted analysis was to and Quantitative the intervention targeted

[33] intervention identify factors associations with and the trial, and outcomes on October 3, 2021 by guest. Protected copyright. 43 (educational that could explain effectiveness pharmacist and results 44 outreach, audit variation in assessment of the summarised, 45 and feedback, outcomes across quality of main trial computerised practices’ educational paper 46 reminders, patient outreach. referenced 47 information) to Regression analysis 48 improve GPs’ of associations 49 prescribing quality with change in prescribing. 50 51 GP, general practitioner. 52 53 reporting of key information would help make the rationale Process evaluations should clearly state their purpose. 54 for choices more explicit, improving interpretation. We Process evaluations should explicitly state their original 55 believe that this key information includes the following purpose and research questions, the processes being 56 factors: studied and an acknowledgement of what is not being 57 Process evaluations should be clearly labelled. Existing evaluated [21]. Changes to research questions during the 58 process evaluation studies are poorly labelled and hard to study can be legitimate but should be explicit. For ex- 59 identify. ample, a pre-planned examination of recruitment and 60 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml Page 59 of 61 BMJ Open

Grant et al. Trials 2013, 14:15 Page 8 of 10 BMJ Open: first published as 10.1136/bmjopen-2016-015281 on 10 March 2017. Downloaded from 1 http://www.trialsjournal.com/content/14/1/15 2 3 4 5 reach may prove to be less important than understand- framework focuses on the activity of clusters, but pays 6 ing unexpected variation in intervention delivery. less attention to how clusters themselves respond to an 7 Process evaluations should clearly report if they were intervention directed at them, to how patients respond 8 pre-specified or post hoc, and why the selected timing to changes in care, or to unintended consequences [19]. 9 was chosen. Pre-specified and post-hoc process evalua- We believe it is useful for designers of cluster- 10 tions are both legitimate designs. Pre-specified evalua- randomised trials of complex interventions to start with 11 tions are more suited to quantitatively examine prior a broad overview of all potential candidates, and to 12 hypotheses about trial processes, whereas post-hoc eva- clearly justify why some candidates are selected for 13 luations are more flexible for examining unanticipated detailed examination, and the methods and theories 14 problems in implementation or unexpected findings. chosen to examine selected processes. 15 Again, clarity in whatFor was done andpeer why is key to inter-reviewFurther work only is required to develop critical appraisal 16 preting the validity and credibility of the findings. instruments for process evaluation studies; our proposed 17 Process evaluations should state the choice of methods criteria for reporting process evaluations represent mini- 18 and justify them in terms of the stated aims of the eva- mal requirements. We believe they will be helpful whilst 19 luation. The rationale for the methods used should be more empirically-based criteria for quality assessments 20 reported in relation to evaluation aims. are developed, as have been established for the reporting 21 Process evaluations should summarise or refer to the [28] and quality assessment of trials [42]. 22 main findings of the trial. To aid interpretation of evalu- There is no single best way to conduct a process 23 ation findings, trial and evaluation reports should cross- evaluation, and researchers will usually need to make 24 reference each other and process evaluations should choices about which research questions to focus on and 25 summarise the main trial findings. which methods to use. We recommend that researchers 26 Table 2 maps our proposed design and reporting frame- explicitly state the purpose of their process evaluation 27 work to three process evaluations that were broad and and demarcate its limits by clearly stating which pro- 28 detailed in reporting their process evaluation and that we cesses are being examined. We propose our framework 29 judged to significantly add understanding to the main trial to facilitate this. Although we focus on process evalua- 30 findings. These vary considerably in their purpose, tions of cluster-randomised trials, elements of the frame- 31 whether they were prospective or retrospective, the pro- work are applicable to other study designs for evaluating 32 cesses examined and the methods used. We believe our complex interventions. Process evaluations are complex, 33 framework helps clarify what kind of process evaluation diverse and face differing constraints based on the main

34 http://bmjopen.bmj.com/ they were, including the processes they do not examine trial context and study funding. However, a more struc- 35 (which is not always explicit in the original papers). tured approach to process evaluation design and repor- 36 ting will improve the validity and dissemination of such 37 Conclusion 38 studies. 39 There are a number of approaches to the design of 40 process evaluations of randomised trials in the literature Abbreviations [1-3,11,19,23,26,27,31,35,37] and other additional rele- CONSORT: Consolidated Standards of Reporting Trials; DQIP: Data-driven 41 Quality Improvement in Primary Care; RE-AIM: Reach Efficacy Adoption, 42 vant literature related to public health interventions Implementation and Maintenance. 43 where randomisation is often not used. Our framework on October 3, 2021 by guest. Protected copyright. pulls many smaller design considerations together into a Competing interests 44 The authors declare that they received no support from any organisation for 45 comprehensive overview of candidate elements that are the submitted work. RF has received funding for unrelated research with 46 potentially relevant to the design of process evaluations Reckitt Benckiser Group Plc in the previous 3 years. The authors declare that 47 of cluster-randomised trials of complex interventions. they have no competing interests. The framework draws attention to a wider range of pro- 48 Authors’ contributions 49 cesses that can be evaluated than many of the existing AG and BG were involved in the initial conceptualisation and design. AG 50 approaches, which only apply to some of these elements, reviewed the literature, carried out the analysis and interpretation of the data although they do offer useful ways of examining these in and contributed to the design of the model. AG prepared the first 51 manuscript and is responsible for this article. BG, ST and RF contributed to 52 depth. For example, normalisation process theory [35] the analysis, interpretation of data, and design of the model. BG, TD and AG 53 maps well to some candidates of our framework, such as designed the final framework model. All authors iteratively commented on 54 ‘response of clusters’, ‘delivery to individuals’ and ‘main- successive drafts of the manuscript. All authors read and approved the final manuscript. 55 tenance’, and facilitates a more in-depth sociological ex- 56 ploration of these candidates. Hardeman and colleagues’ Acknowledgements 57 causal modelling approach [37] maps well to ‘response AG and TD are funded by the Scottish Government Health Directorates Chief 58 ’ Scientist Office Applied Programme Research Grant 07/02 for the DQIP of individuals and facilitates a more in-depth psycho- prescribing improvement study. This funding body have had no direct role 59 logical exploration of this candidate. The RE-AIM in the design and conduct of, or decision to publish, this work. The authors 60 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml BMJ Open Page 60 of 61

Grant et al. Trials 2013, 14:15 Page 9 of 10 BMJ Open: first published as 10.1136/bmjopen-2016-015281 on 10 March 2017. Downloaded from 1 http://www.trialsjournal.com/content/14/1/15 2 3 4 5 would like to thank Professor Peter Donnan, University of Dundee for his 17. Ellard D, Taylor S, Parsons S, Thorogood M: The OPERA trial: a protocol for the statistical advice. process evaluation of a randomised trial of an exercise intervention for 6 older people in residential and nursing accommodation. Trials 2011, 12:28. 7 Author details 18. Underwood M, Eldridge S, Lamb S, Potter R, Sheehan B, Slowther A-M, 8 1Quality, Safety and Informatics Research Group, Population Health Sciences, Taylor S, Thorogood M, Weich S: The OPERA trial: protocol for a Medical Research Institute, University of Dundee, Mackenzie Building, randomised trial of an exercise intervention for older people in 9 Dundee DD2 4BF, UK. 2Medicines Management Unit, NHS Tayside, c/o residential and nursing accommodation. Trials 2011, 12:27. 10 University of Dundee, Mackenzie Building, Dundee DD2 4BF, UK. 3Leeds 19. Glasgow RE, McKay HG, Piette JD, Reynolds KD: The RE-AIM framework for 11 Institute of Health Sciences, Charles Thackrah Building, University of Leeds, evaluating interventions: what can it tell us about approaches to chronic 12 101 Clarendon Road, Leeds LS2 9LJ, UK. illness management? Patient Educ Couns 2001, 44:119–127. 20. Akl EA, Treweek S, Foy R, Francis J, Oxman AD: NorthStar, a support tool 13 Received: 25 April 2012 Accepted: 18 December 2012 for the design and evaluation of quality improvement interventions in 14 Published: 12 January 2013 healthcare. Implement Sci 2007, 2:19–26. 15 For peer review21. Francis J, Eccles M,only Johnston M, Whitty P, Grimshaw J, Kaner E, Smith L, Walker A: Explaining the effects of an intervention designed to promote References 16 evidence-based diabetes care: a theory-based process evaluation of a 1. Medical Research Council: Developing and Evaluating Complex Interventions: 17 pragmatic cluster randomised controlled trial. Implement Sci 2008, 3:50. New Guidance. London: Medical Research Council; 2008. 22. Durlak J: Why programme implementation is so important. J Prev Interv 18 2. Hawe P, Shiell A, Riley T, Gold L: Methods for exploring implementation Community 1998, 17:5–18. 19 variation and local context within a cluster randomised community 23. Lewin S, Glenton C, Oxman AD: Use of qualitative methods alongside intervention trial. J Epidemiol Community Health 2004, 58:788–793. 20 randomised controlled trials of complex healthcare interventions: 3. Oakley A, Strange V, Bonell C, Allen E, Stephenson J, RIPPLE Study Team: methodological study. Br Med J 2009, 339:b3496. 21 Process evaluation in randomised controlled trials of complex 24. Hulscher M, Laurant M, Grol R: Process evaluation of quality improvement 22 interventions. Br Med J 2006, 332:413–416. interventions.InQuality Improvement Research; Understanding The Science of 4. Linnan L, Steckler A: Process evaluation for public health interventions 23 Change in Healthcare. Edited by Grol R, Baker R, Moss F. London: BMJ and research: an overview.InProcess Evaluation for Public Health 24 Publishing Group; 2004:165–183. Interventions and Research. Edited by Steckler A, Linnan L. San Francisco: 25 Jossey-Bass; 2002:1–23. 25. Cook T: Emergent principles for the design, implementation, and analysis 26 5. Wight D, Obasi A: Unpacking the 'black box': the importance of process of cluster-based experiments in social science. Ann Am Acad Pol Soc Sci 2005, 599:176–198. 27 data to explain outcomes.InEffective Sexual Health Interventions: Issues in Experimental Evaluation. Edited by Stephenson JM, Imrie J, Bonell C. Oxford: 26. Bonell C, Oakley A, Hargreaves J, Strange V, Rees R: Assessment of 28 Oxford University Press; 2003:151–166. generalisability in trials of health interventions: suggested framework – 29 6. Ellard D, Parsons S: Process evaluation: understanding how and why and systematic review. Br Med J 2006, 333:346 349. interventions work.InEvaluating Health Promotion Practice and Methods. 27. Murtagh MJ, Thomson RG, May CR, Rapley T, Heaven BR, Graham RH, Kaner 30 EF, Stobbart L, Eccles MP: Qualitative methods in a randomised controlled 31 Edited by Thorogood M, Coombes Y. Oxford: Oxford University Press; 2010. 7. Craigie A, Caswell S, Paterson C, Treweek S, Belch J, Daly F, Rodger J, trial: the role of an integrated qualitative process evaluation in providing 32 Thompson J, Kirk A, Ludbrook A, Stead M, Wardle J, Steele R, Anderson A: evidence to discontinue the intervention in one arm of a trial of a – 33 Study protocol for BeWEL: the impact of a BodyWEight and physicaL decision support tool. Qual Saf Health Care 2007, 16:224 229. activity intervention on adults at risk of developing colorectal 28. Campbell MK, Elbourne DR, Altman DG: CONSORT statement: extension to

34 – http://bmjopen.bmj.com/ adenomas. BMC Public Health 2011, 11:184. cluster randomised trials. Br Med J 2004, 328:702 708. 35 8. Protheroe J, Bower P, Chew-Graham C: The use of mixed methodology in 29. Hoddinott P, Britten J, Pill R: Why do interventions work in some places and – 36 evaluating complex interventions: identifying patient factors that not others: a breastfeeding support group trial. Soc Sci Med 2010, 70:769 778. 37 moderate the effects of a decision aid. Fam Pract 2007, 24:594–600. 30. Nazareth I, Freemantle N, Duggan C, Mason J, Haines A, Nazareth I, 9. Avery AJ, Rodgers S, Cantrill JA, Armstrong S, Cresswell K, Eden M, Elliott RA, Freemantle N, Duggan C, Mason J, Haines A: Evaluation of a complex 38 Howard R, Kendrick D, Morris CJ, Prescott RJ, Swanwick G, Franklin M, intervention for changing professional behaviour: the Evidence Based 39 Putman K, Boyd M, Sheikh A: A pharmacist-led information technology Out Reach (EBOR) Trial. J Health Serv Res Policy 2002, 7:230–238. 40 intervention for medication errors (PINCER): a multicentre, cluster 31. Craig P, Dieppe P, Macintyre S, Michie S, Nazareth I, Petticrew M: Developing and evaluating complex interventions: the new Medical 41 randomised, controlled trial and cost-effectiveness analysis. Lancet 2012, 379:1310–1319. Research Council guidance. Br Med J 2008, 337:a1655. 42 10. Medical Research Council: A Framework for the Development and Evaluation 32. Salter C, Holland R, Harvey I, Henwood K: ‘I haven't even phoned my on October 3, 2021 by guest. Protected copyright. 43 of RCTs for Complex Interventions to Improve Health. London: Medical doctor yet.’ The advice giving role of the pharmacist during 44 Research Council; 2000. consultations for medication review with patients aged 80 or more: 11. Byng R, Norman I, Redfern S, Jones R: Exposing the key functions of a qualitative discourse analysis. Br Med J 2007, 334:1101. 45 complex intervention for shared care in mental health: case study of a 33. Fretheim A, Havelsrud K, Oxman A: Rational Prescribing in Primary care 46 process evaluation. BMC Health Serv Res 2008, 8:274. (RaPP): process evaluation of an intervention to improve prescribing of 47 12. Jansen Y, de Bont A, Foets M, Bruijnzeels M, Bal R: Tailoring intervention antihypertensive and cholesterol-lowering drugs. Implement Sci 2006, procedures to routine primary health care practice; an ethnographic 1:19. 48 process evaluation. BMC Health Serv Res 2007, 7:125. 34. Farrin A, Russell I, Torgerson DJ, Underwood M: Differential recruitment in 49 13. Dreischulte T, Grant A, Donnan P, McCowan C, Davey P, Petrie D, Treweek S, a cluster randomised trial in primary care: the experience of the UK Back 50 Guthrie B: A cluster randomised stepped wedge trial to evaluate the pain, Exercise, Active management and Manipulation (UK BEAM) – 51 effectiveness of a multifaceted information technology-bsed feasibility study. Clin Trials 2005, 2:119 124. intervention in reducing high-risk prescribing of non-steroidal anti- 35. May C, Finch T: Implementing, embedding, and integrating practices: an 52 inflammatory drugs and antiplatelets in primary care: the DQIP study outline of normalisation process theory. Sociology 2009, 43:535. 53 protocol. Implement Sci 2012, 7:24. 36. Christie D, Hudson L, Mathiot A, Cole T, Karlsen S, Kessel A, Kinra S, Morris S, 54 14. Donnar A, Klar N: Design and Analysis of Cluster Randomization Trials in Nazareth I, Sovio U, Wong I, Viner R: Assessing the efficacy of the healthy Health Research. London: Arnold; 2000. eating and lifestyle programme (HELP) compared with enhanced 55 15. Donnar A, Klar N: Pitfalls of and controversies in cluster randomization standard care of the obese adolescent in the community: study protocol 56 trials. Am J Public Health 2004, 94:416–422. for a randomized controlled trial. Trials 2011, 12:242. 57 16. Ivers N, Jamtvedt G, Flottorp S, Young J, Odgaard-Jensen J, French S, 37. Hardeman W, Sutton S, Griffin S, Johnston M, White AJ, Wareham NJ, 58 O'Brien M, Johansen M, Grimshaw J, Oxman A: Audit and feedback: effects Kinmonth AL: A causal modelling approach to the development of on professional practice and healthcare outcomes. Cochrane Database theory-based behaviour change programmes for trial evaluation. 59 Syst Rev 2012, 6:CD000259. Health Educ Res 2005, 20:676–687. 60 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml Page 61 of 61 BMJ Open

Grant et al. Trials 2013, 14:15 Page 10 of 10 BMJ Open: first published as 10.1136/bmjopen-2016-015281 on 10 March 2017. Downloaded from 1 http://www.trialsjournal.com/content/14/1/15 2 3 4 5 38. Mitchie S, Johnson M, Francis J, Hardeman W, Eccles M: From theory to intervention: mapping theoretically derived behavioural determinants to 6 behaviour change techniques. Appl Psychol Int Rev 2008, 57:660–680. 7 39. Wells M, Williams B, Treweek S, Coyle J, Taylor J: Intervention description is 8 not enough: evidence from an in-depth multiple case study on the untold role and impact of context in randomised controlled trials of 9 seven complex interventions. Trials 2012, 13:95. 10 40. Grant A, Dreischulte T, Treweek S, Guthrie B: Study protocol of a mixed- 11 methods evaluation of a cluster randomised trial to improve the safety of NSAID and antiplatelet prescribing: data-driven quality improvement 12 in primary care. Trials 2012, 13:154. 13 41. Foy R, Francis J, Johnston M, Eccles M, Lecouturier J, Bamford C, Grimshaw J: 14 The development of a theory-based intervention to promote appropriate 15 disclosure of a diagnosisFor of dementia. BMC Healthpeer Serv Res 2007, 7:207. review only 42. 11 Questions to Help You Make Sense of a Trial. http://www.casp-uk.net/wp- 16 content/uploads/2011/11/CASP_RCT_Appraisal_Checklist_14oct10.pdf. 17 18 doi:10.1186/1745-6215-14-15 Cite this article as: Grant et al.: Process evaluations for cluster- 19 randomised trials of complex interventions: a proposed framework for 20 design and reporting. Trials 2013 14:15. 21 22 23 24 25 26 27 28 29 30 31 32 33

34 http://bmjopen.bmj.com/ 35 36 37 38 39 40 41 42 43 on October 3, 2021 by guest. Protected copyright. 44 45 46 47 48 49 Submit your next manuscript to BioMed Central 50 and take full advantage of: 51 52 • Convenient online submission 53 • Thorough peer review 54 • No space constraints or color figure charges 55 • Immediate publication on acceptance 56 • Inclusion in PubMed, CAS, Scopus and Google Scholar 57 • Research which is freely available for redistribution 58 59 Submit your manuscript at www.biomedcentral.com/submit 60 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml