RESEARCH

Treatment effects in randomised trials using routinely collected BMJ: first published as 10.1136/bmj.n450 on 3 March 2021. Downloaded from data for outcome assessment versus traditional trials: meta-research study Kimberly A Mc Cord,1 Hannah Ewald,1,2 Arnav Agarwal,3 Dominik Glinz,1 Soheila Aghlmandi,1 John P A Ioannidis,4,9 Lars G Hemkens1,5,9

1Basel Institute for Clinical ABSTRACT were similar across various types of outcomes and , OBJECTIVE (mortality outcomes: 0.92, 0.74 to 1.15, I2=12%; Department of , To compare effect estimates of randomised clinical non-mortality outcomes: 0.71, 0.60 to 0.84, I2=8%), University Hospital Basel, University of Basel, 4031 Basel, trials that use routinely collected data (RCD-RCT) for data sources (electronic health records: 0.81, 0.59 to Switzerland outcome ascertainment with traditional trials not 1.11, I2=28%; registries: 0.86, 0.75 to 0.99, I2=20%; 2University Medical Library, using routinely collected data. administrative data: 0.84, 0.72 to 0.99, I2=0%), and University of Basel, Basel, data quality (high data quality: 0.82, 0.72 to 0.93, Switzerland DESIGN I2=0%). 3Department of Medicine, Meta-research study. University of Toronto, Toronto, DATA SOURCE CONCLUSIONS ON, Canada Randomised clinical trials using routinely collected 4 Studies included in the same meta-analysis in a Stanford Prevention Research data for outcome ascertainment show smaller Center, Department of Medicine, Cochrane review. treatment benefits than traditional trials not using Stanford University School of ELIGIBILITY CRITERIA FOR STUDY SELECTION routinely collected data. These differences could have Medicine, Stanford, CA, USA Randomised clinical trials using any type of routinely 5 implications for healthcare decision making and the Meta-Research Innovation Center collected data for outcome ascertainment, including at Stanford (METRICS), Stanford application of real world evidence. University, Palo Alto, CA, USA from registries, electronic health records, and 6Department of Health Research administrative databases, that were included in a and Policy, Stanford University meta-analysis of a Cochrane review on any clinical Introduction School of Medicine, Stanford, question and any health outcome together with Clinical trials increasingly use health data that are CA, USA traditional trials not using routinely collected data for not collected for the purposes of research.1 2 Such 7Department of Biomedical Data http://www.bmj.com/ Science, Stanford University outcome measurement. routinely collected data from registries, electronic School of Medicine, Stanford, REVIEW METHODS health records, administrative claims, or even mobile CA, USA devices might be used to identify trial participants and 8 Effect estimates from trials using or not using routinely Department of , to assess treatment outcomes.2 Readily available data Stanford University School collected data were summarised in random effects of Humanities and Sciences, meta-analyses. Agreement of (summary) treatment are typically more affordable than actively collected Stanford, CA, USA effect estimates from trials using routinely collected research data.3 Cost reduction might make larger and 9 Meta-Research Innovation Center data and those not using such data was expressed as longer trials more feasible. during Berlin (METRIC-B), Berlin Institute the ratio of ratios. Subgroup analyses explored usual care also avoids artificial research settings, and on 27 September 2021 by guest. Protected copyright. of Health, Berlin, Germany effects in trials based on different types of routinely this could increase pragmatism and applicability of Correspondence to: 4 L G Hemkens collected data. Two investigators independently trial results to routine care. Databases of routinely [email protected] assessed the quality of each data source. collected data include many outcomes that are (or @lghemkens on Twitter; relevant in practice and matter to clinicians and ORCID 0000-0002-3444-1432) RESULTS Additional material is published 84 RCD-RCTs and 463 traditional trials on 22 clinical patients (eg, mortality, disability, hospital admission), online only. To view please visit questions were included. Trials using routinely whereas they typically lack outcomes that are more the journal online. collected data for outcome ascertainment showed relevant for explanatory trials aiming to understand Cite this as: BMJ 2021;372:n450 20% less favourable treatment effect estimates than the biological processes underpinning treatment http://dx.doi.org/10.1136/bmj.n450 traditional trials (ratio of odds ratios 0.80, 95% effects (eg, biomarkers).5 Cutting out research driven Accepted: 27 January 2021 0.70 to 0.91, 2I =14%). Results follow-up visits and relying only on patient during usual care probably better reflects real world treatment effects, and patient adherence might be less WHAT IS ALREADY KNOWN ON THIS TOPIC faithful in such a setting compared with traditional, Routinely collected data are increasingly used in randomised clinical trials to more explanatory trials. Overall, trials embedded in measure outcomes existing data collection structures might provide real Data collection during usual care can reduce costs and avoid artificial research world evidence, being more informative for guiding settings, which might increase pragmatism and applicability of trial results treatment decisions and sharing more features of pragmatic trials than do many traditional trials.6-8 WHAT THIS STUDY ADDS Data quality is a key problem of using routinely Our study suggests that randomised clinical trials using routinely collected data collected data for clinical research.1 2 On the one hand, to assess outcomes provide systematically less favourable treatment effects than for some outcomes the quality of routinely collected trials using traditional methods data might be lower, in particular as a result of non- Differences might exist between traditional trials and trial designs using routinely uniform data collection and potential measurement collected data beyond data quality issues that would explain this finding errors.9-13 On the other hand, healthcare professionals the bmj | BMJ 2021;372:n450 | doi: 10.1136/bmj.n450 1 RESEARCH

collecting routine data during usual care might have The other randomised clinical trials (ie, not RCD- more clinical expertise than research staff who often RCTs) that were included with an eligible RCD-RCT in BMJ: first published as 10.1136/bmj.n450 on 3 March 2021. Downloaded from collect trial data only for a narrow time frame and the same Cochrane review meta-analysis were eligble scope, sometimes only for a few participants or even comparators. a single patient in each centre.14 Since routine data We considered any health intervention in any are collected independently of the trial from people population. We did not consider outcomes that unaware of treatment allocations, biases related to were uniquely cost related, but we kept outcomes outcome ascertainment might be even less likely than that measured uptake of interventions, such as in traditional trials. Moreover, quality of routinely vaccinations, drug treatments, and screening. collected data can vary enormously for different Routinely collected data was defined as any health outcomes. For mortality, the quality might be high15: information not collected primarily for a specific complete, accurate information can be achieved with research question.19 Trials that Cochrane reviewers proper linkage to death registries, whereas other described as quasi-randomised or as controlled before trials not linked to routinely collected data sources and after design were excluded. We considered trials might lack information on survival status for many reported as cluster randomised trials and crossover participants. Conversely, the quality of routinely trials (data from first period only), but excluded them collected data might be highly insufficient for other in a sensitivity anlysis. outcomes, such as specific adverse events or some patient reported endpoints. The impact of using Search routinely collected data for outcome ascertainment and To identify the index RCD-RCTs, we searched PubMed the impact of potential inaccuracies on trial results is using text words and medical subject headings focusing unclear. Misclassification of clinical events or missing on terms around routine data (see appendix 1). We information that occurs randomly—for example, due searched for randomised clinical trials published to coding errors or problems with database linkage,16 in English between 2000 and 2015 because of the could diminish the treatment effect point estimates.17 emerging availability of electronic health records and Larger sample sizes achieved by using routinely other sources of routinely collected data in the past two collected data might increase precision of treatment decades and because more recent trials were less likely effect estimates,18 but these could still be biased to be included in Cochrane reviews. Two reviewers underestimations. independently screened titles and abstracts (KAM and http://www.bmj.com/ Here, we provide empirical insights on the agreement AL or HE). Articles found to be potentially eligible by of findings from trials using routinely collected data one reviewer were considered for further analysis. for measuring outcomes compared with traditional One reviewer (KAM) then identified Cochrane reviews randomised clinical trials not using routinely collected citing any of these potentially eligible RCD-RCTs using data. the “cited in systematic reviews” function on PubMed. We also searched ISI Web of Science and perused the Methods citing articles (from Web of Science Core Collection). No was published for this study. We The last searches for RCD-RCTs in literature databases on 27 September 2021 by guest. Protected copyright. systematically obtained a large sample of randomised and citing Cochrane reviews were in March 2016 and clinical trials that used routinely collected data to September 2017 (see appendix 1 for details). We used measure study outcomes (RCD-RCTs), identified trials the most recent updated version (last search January that explored the same clinical question but measured 2020) of each Cochrane review for all pertinent clinical outcomes using traditional methods (not based on questions, and updated our searches, classifications, routinely collected data), and then we compared their and extractions using these most recent versions. treatment effect estimates. We assumed that studies included in the same meta-analysis in a Cochrane Study selection review would be on the same clinical question. We obtained all full texts of cited randomised Cochrane reviews were a main information source for clinical trials and citing reviews. One reviewer this study. (KAM) determined if the trial was an index RCD-RCT (ie, measured at least one pertinent outcome using Eligibility criteria routinely collected data and was included in a meta- RCD-RCTs were eligible if they used the data for analysis evaluating this outcome together with other measurement of any binary clinical outcome and trials). This was verified by a second reviewer (LGH). were included in a Cochrane review meta-analysis We obtained the full texts for all other trials in together with at least one other trial not using routinely the meta-analysis, and one reviewer (KAM or DG) collected data for measuring the same outcome. determined if they were eligble RCD-RCTs or they Eligible RCD-RCTs were either identified directly by were categorised as traditional trials. Whenever searching PubMed and subsequent citation analysis to uncertainty occurred in these steps, a second reviewer determine if they were included in a Cochrane review was consulted (LGH) and the decision made was (index RCD-RCTs) or indirectly by perusing the other based on consensus. Eligibility of all RCD-RCTs was trials that were included with the index RCD-RCTs in confirmed by a second reviewer (LGH, AA, or KAM). the same Cochrane review meta-analysis. Any uncertainties were resolved by discussion.

2 doi: 10.1136/bmj.n450 | BMJ 2021;372:n450 | the bmj RESEARCH

Data collection process authors provided a statement that led us to assume From each Cochrane review, we selected only one clinical that the routinely collected data would adequately BMJ: first published as 10.1136/bmj.n450 on 3 March 2021. Downloaded from question addressed by one meta-analysis including the measure the outcome of interest, a high quality mark index RCD-RCT. We selected the meta-analysis with the was given. If this was not reported, but the source was largest number of randomised clinical trials (if multiple specifically designed to collect the endpoint (eg, breast ones existed, we selected the one with the greatest total cancer cases through a comprehensive national breast sample size). Some meta-analyses were reported with cancer registry), a high quality mark was still given. summary estimates for subsets of studies but without If a statement indicating low quality was provided an overall summary effect. In such cases, we took the (which we expected to be rare, but such statements subset including the highest number of RCD-RCTs. In could have been made in the limitations section of some cases, when the same RCD-RCTs were included the studies) or the reviewer thought that the routinely in multiple subsets (eg, for different lengths of follow- collected data source was unlikely to specifically up) but there was an overall summary presented, collect such outcome data with little missingness and we also used only the largest subset to avoid double little measurement error (eg, adverse events extracted counting of participants or events. We preferred any from administrative databases), a low mark was given. primary analysis over sensitivity or subgroup analyses. Other cases were rated unclear. We quantified the Sensitivity analyses on methodological features (eg, by agreement between the two reviewers (KAM versus AA publication year) were always excluded. These steps or DG) using κ statistics and the total agreement. were conducted by one reviewer (KAM) and verified by a For sensitivity analyses, we extracted the risk of second (LGH). We applied a different selection approach reported for each bias domain of all individual trials. as secondary analysis whenever the meta-analysis We categorised the trials as having one domain or more selected for the main analysis was not on mortality at high risk (if any bias domain was deemed by the (which was the case for 14 reviews) but there was a Cochrane reviewers to be high risk), all domains at low relevant mortality analysis included in the Cochrane risk (if all domains were deemed to be low risk), and review (which was the case in four of the 14 reviews); all domains at low or unclear risk (in all other cases). then we selected this one instead. We applied the same We also specifically extracted the risk of bias due to the approach for primary outcomes, but in the three cases blinding status (or participant blinding when several where the selected outcome was not a primary outcome blinding domains were presented). of the Cochrane review, no eligible alternative existed. http://www.bmj.com/ For each included trial, one reviewer (KM, LGH, AA, Summary measures and synthesis of results HE, or DG) extracted from the Cochrane review the We used a two stage process to synthesise the results. treatment effects (ie, number of events and no events Firstly, we calculated two summary odds ratios for each per study arm), trial characteristics (parallel group clinical question using random effects meta-analyses design, crossover design, cluster design, country, year (Hartung-Knapp-Sidik-Jonkman method21): the of publication), the age of the study population summary odds ratios of the RCD-RCTs, and separately the (when not reported, we used other available pertinent summary odds ratios of all the traditional trials. In cases information (eg, ) for approximation when when only one trial was available, the summary odds on 27 September 2021 by guest. Protected copyright. possible), and the Cochrane reviewer’s risk of bias ratio was actually the odds ratio of the trial. Subsequently, assessment. A second reviewer (KM or LGH) verified for each pair of summary odds ratios, we calculated their the extracted treatment effects. respective ratio—that is, ratio of odds ratios (summary For each eligible RCD-RCT, one reviewer (KAM, odds ratios of the traditional trials divided by summary DG, or LGH) extracted general characteristics and odds ratios of RCD-RCTs). The of the ratio of the types of routinely collected data utilised. We also odds ratios was calculated as the sum of the noted whether the routinely collected data source of the summary odds ratios (after log transformation). was the only form of outcome data source, or if a We ensured that for all clinical questions odds ratios of hybrid approach was reported (ie, when the routinely less than 1 indicate favourable effects for the evaluated collected data were complemented by additional active treatment. We inverted effects when necessary (ie, if a data collection). Trials using routinely collected data meta-analysis reported survival, we inverted the effect within a hybrid approach were considered as RCD- estimate by taking its reciprocal so that estimates <1 RCTs but were excluded in a sensitivity analysis. indicate mortality benefits). For consistency, we ensured One reviewer (KAM or DG) extracted any statement that the second comparator was the control (that is, no on quality of the routinely collected data in the broader intervention or usual care—in three cases when two sense (eg, statements related to measurement errors, active interventions were compared,22-24 we defined the reliability, accuracy, or completeness) and a second control as the older treatment; we left these cases out in reviewer (KAM or AA) verified the extractions. As a a sensitivity analysis). A ratio of odds ratios of less than 1 working definition, we deemed data quality to be indicated that the RCD-RCTs estimated a less favourable high when the routinely collected data would be treatment effect for the evaluated treatment than did the adequate to reliably measure the outcomes of interest traditional trials. for this clinical question.20 Two reviewers assessed Secondly, we combined all ratios of odds ratios this independently. We fully acknowledge that such across all clinical questions in a meta-analysis (random an assessment from the outside is difficult.20 When effects, Hartung-Knapp-Sidik-Jonkman) to provide an the bmj | BMJ 2021;372:n450 | doi: 10.1136/bmj.n450 3 RESEARCH

overall summary of the relation of treatment effects trials, including only clinical questions on mortality obtained from trials using routinely collected data outcomes or non-mortality outcomes (subsets of main BMJ: first published as 10.1136/bmj.n450 on 3 March 2021. Downloaded from versus trials not using routinely collected data. analysis), excluding clinical questions with active controls, using only clinical questions with effect Additional analyses estimates from RCD-RCTs and traditional randomised We conducted several sensitivity analyses: including clinical trials that had no largely different precision (ie, only RCD-RCTs with low risk of bias in all domains, ratio of summary odds ratio standard errors >0.33 and including only RCD-RCTs with low risk of bias related <3), excluding clinical questions with fewer than three to blinding, excluding RCD-RCTs with some active data RCD-RCTs, excluding clinical questions with more than collection (hybrid approaches), excluding older RCD- 10 RCD-RCTs, comparing the index RCD-RCTs with all RCTs (published before 2005), including only more other trials in the meta-analysis (including traditional recent RCD-RCTs (published in 2010 or later), stratified trials and misclassifying the indirectly identified RCD- by number of participants and number of events (thirds RCTs) to evaluate the robustness of the classification across all RCD trials), including only RCD-RCTs when and procedure, using DerSimonian-Laird the median age of the RCD-RCT population was within random effects meta-analyses, and using only fixed 1 of the median age of the other effect meta-analyses.

Table 1 | Overview of characteristics of randomised clinical trials that use routinely collected data for outcome ascertainment and traditional trials not using routinely collected data for the outcome ascertainment. Values are numbers (percentages) unless stated otherwise Source of RCD Characteristics Overall RCD-RCTs Registry Administrative database Electronic health records Traditional RCTs Trials 84 (100) 36 (43) 18 (21) 30 (36) 463 (100) Publication year: Median (interquartile ) 2005 (1998-2009) 2003 (1992-2009) 2007 (2003-12) 2006 (2000-11) 2003 (1997-2006) Range 1976-2017 1976-2015 1998-2015 1989-2017 1963-2016 No of participants: Median () 721 (275-2729) 2037 (524-17 066) 1403 (414-3406) 286 (146-534) 121 (60-359) Range 16-89 699 99-89 699 45-24 743 16-12 205 16-160 840 No of events:

Median (interquartile range) 194 (50-1266) 440 (65-1383) 559 (98-1734) 124 (33-271) 27 (6-100) http://www.bmj.com/ Range 0-86 201 4-86 201 0-18 146 0-5562 0-3364 Cluster randomised design* 4 (5) 1 (3) 2 (11) 1 (3) 18 (4) Age†: Median (interquartile range) 52 (25-68) 16 (1-59) 67 (45-73) 55 (46-63) 62 (57-66) Range 0-87 0-79 2-85 1-89 0-87 Country: Australia 0 (0) 0 (0) 0 (0) 0 (0) 20 (4) Brazil 0 (0) 0 (0) 0 (0) 0 (0) 9 (2) on 27 September 2021 by guest. Protected copyright. China 0 (0) 0 (0) 0 (0) 0 (0) 5 (1) Continental Europe 3 (4) 0 (0) 1 (5.5) 2 (7) 60 (13) North America 56 (67) 16 (44) 14 (78) 26 (87) 125 (27) Scandinavia‡ 14 (17) 13 (36) 1 (5.5) 0 (0) 31 (7) United Kingdom 6 (7) 4 (11) 1 (5.5) 1 (3) 43 (9) Other§ 5 (6) 3 (8) 1 (5.5) 1 (3) 45 (10) Not reported 0 (0) 0 (0) 0 (0) 0 (0) 125 (27) Risk of bias: High in ≥1 domain 35 (42) 16 (44) 5 (28) 14 (47) 218 (47) Low in all domains 16 (19) 8 (22) 3 (17) 5 (17) 79 (17) Unclear or low in all domains 33 (39) 12 (33) 10 (56) 11 (37) 166 (36) Blinding: High 21 (25) 5 (14) 3 (17) 13 (43) 158 (34) Low 32 (38) 18 (50) 6 (33) 8 (27) 194 (42) Unclear 31 (37) 13 (36) 9 (50) 9 (30) 111 (24) Estimated data quality: NA High 56 (67) 31 (86) 9 (50) 16 (53) - Low 24 (29) 3 (8) 9 (50) 12 (40) - Unclear 4 (5) 2 (6) 0 () 2 (7) - RCD collection level: NA Complete RCD¶ 57 (68) 29 (81) 10 (56) 18 (60) - Hybrid** 27 (32) 7 (19) 8 (44) 12 (40) - NA=not applicable; RCT=randomised ; RCD=routinely collected data. *Cochrane reviewers described two trials (0.5%) as crossover. †Information reported for 345 of traditional RCTs and 48 RCD-RCTs. ‡Includes Sweden, Norway, Denmark, Finland, and Iceland. §Includes Europe (more than one country, multicentre), worldwide (more than one country outside of Europe, multicentre), Taiwan, Syria, India, Iran, Belarus, Malaysia, Egypt, Turkey, Zimbawe, New Zealand, Chile, Colombia, Israel, Venezuela, Japan, Hong Kong, Pakistan, Argentina, Korea, Singapore, South Africa, and Georgia. ¶Fully RCD based data collection. **Routine data collection with supportive active data collection.

4 doi: 10.1136/bmj.n450 | BMJ 2021;372:n450 | the bmj RESEARCH

We conducted exploratory subgroup analyses Results including only RCD-RCTs using registries, electronic Overall, 4649 publications were screened and 29 index BMJ: first published as 10.1136/bmj.n450 on 3 March 2021. Downloaded from health records, or administrative data, and when the RCD-RCTs identified (see appendices 1, 2, and 5a) from data quality of RCD-RCTs was assumed to be high. 22 Cochrane reviews. Among the corresponding trials We report with interquartile ranges if not in the selected Cochrane review analyses, 55 other stated otherwise. We used the meta package (version RCD-RCTs were identified (see appendix 5b) and 463 4.11-0) for meta-analyses25 (RStudio version 1.2.5033; were eligible traditional randomised clinical trials (see R version 3.6.2).26 27 appendix 6). The median number of participants in each of the Patient and public involvement 84 RCD-RCTs was 721 (interquartile range 275-2729), We did not involve patients or members of the public most (56/84, 67%) originating from North America, when we selected the research question, designed the followed by Scandinavia (14/84, 17%; table 1). The study, interpreted the results, or wrote the manuscript. trials were published between 1976 and 2017: median

Table 2 | Clinical questions and corresponding trials providing treatment effects measured with or without routinely collected data Cochrane review identifier; Median trial size (interquartile Clinical questions Outcomes meta-analysis No No of trials range); range Individualised discharge plan for all patients Unscheduled readmissions CD00031337; 2.1.0 3 RCD-RCTs: 14 other RCTs 575 (336-637); 96-698: 205 (97- admitted to hospital 278); 50-738 Breastfeeding support for healthy pregnant Stopping breastfeeding CD00114138; 1.1.0 1 RCD-RCT: 48 other RCTs 990: 329 (136-520); 41-1660 women intending to breastfeed or already breastfeeding Mammography screening in women without Breast cancer mortality CD00187739; 1.1.0 7 RCD-RCTs: 4 other RCTs 39 405 (24 767-46 357); 17 793- previous breast cancer diagnosis 59 176: 69 485 (60 974-97 937); 57 897-160 840 Antifibrinolytic agents in patients undergoing Need for allogeneic blood CD00188640; 1.1.0 1 RCD-RCT: 107 other RCTs 16: 59 (40-98); 17-1784 surgery transfusion Interventions to increase uptake of cervical cancer Uptake of screening CD00283441; 1.1.1 6 RCD-RCTs: 6 other RCTs 1157 (358-2335); 314-89 699: screening 482 (162-1317); 97-1794 Self-management interventions in patients with Mortality CD00299042; 1.21.0 1 RCD-RCT: 8 other RCTs 191: 164 (145-211); 135-743 chronic obstructive pulmonary 43 Exercised based interventions in patients with Hospital admission CD003331 ; 1.4.0 1 RCD-RCT: 6 other RCTs 2330: 47 (28-87); 23-123 http://www.bmj.com/ heart failure Fast track interventions for early extubation Mortality CD00358723; 2.1.4 3 RCD-RCTs: 7 other RCTs 120 (84-359); 48-597: 98 (66- (time directed extubation protocol) in patients 172); 60-404 undergoing cardiac surgery Levonorgestrel intrauterine device versus surgery Additional surgery received CD00385522; 2.13.0 1 RCD-RCT: 5 other RCTs 225: 60 (57-69); 57-72 in women with heavy menstrual bleeding Reminder and recall immunisation interventions in Immunisations CD00394144; 1.1.0 30 RCD-RCTs: 27 other RCTs 1888 (751-4598); 204-24743: adults and children 304 (173-555); 96-3006

Routine invasive versus conservative selective Mortality or non-fatal CD00481545; 1.13.0 1 RCD-RCT: 2 other RCTs 2457: 1505 (1353-1658); 1200- on 27 September 2021 by guest. Protected copyright. treatment in patients with unstable angina and myocardial infarction 1810 non-ST elevation myocardial infarction Interventions to reduce falls in those aged 60 Falls CD00546546; 4.2.0 2 RCD-RCT: 4 other RCTs 1883 (965-2800); 48-3717: 353 years or older in care facilities and hospitals (114-594); 91-625 Collaborative care interventions for people with drug use CD00652547; 1.3.1 13 RCD-RCTs: 31 other RCTs 208 (88-285); 45-372: 179 (83- depression and anxiety 292); 34-1570 Antioxidant supplementation in healthy Mortality CD00717648; 1.1.0 2 RCD-RCTs: 76 other RCTs 15 022 (7966-22 077); 910- participants and in patients with various stable 29 133: 357 (99-1667); 19- 39 876 On-pump surgery in patients undergoing coronary Mortality CD00722449; 1.1.0 1 RCD-RCT: 73 other RCTs 339: 60 (40-120); 20-2203 artery bypass graft surgery Structured telephone support or non-invasive Mortality CD00722850; 1.2.0 3 RCD-RCTs: 14 other RCTs 319 (263-515); 206-710: 141 telemonitoring interventions in patients with heart (91-259); 20-460 failure Mycophenolic acid versus azathioprine as Graft loss CD00774624; 1.3.3 1 RCD-RCT: 3 other RCTs 133: 76 (72-162); 68-248 primary immunosuppression for kidney transplant recipients (adults and children) Statins in patients with chronic kidney disease not Mortality CD00778451; 1.2.0 2 RCD-RCTs: 8 other RCTs 9565 (5936-13 195); 2306- requiring dialysis 16 824: 722 (255-1472); 87-3267 Case management interventions in people with Hospital admissions CD00834552; 1.5.2 2 RCD-RCTs: 3 other RCTs 141 (133-149); 125-157: 89 (89- dementia 108); 88-126 Drug review in patients admitted to hospital Mortality CD00898653; 1.1.0 1 RCD-RCT: 8 other RCTs 99: 368 (120-485); 66-936 Interventions to reduce dietary salt in hypertensive Mortality CD00921754; 1.1.0 1 RCD-RCT: 6 other RCTs 1981: 519 (401-710); 67-2382 patients Fish oil for pregnant or breastfeeding women to Allergies CD01008555; 6.2.1 1 RCD-RCT: 3 other RCT 528: 531 (324-619); 117-706 prevent allergies in offspring RCD=routinely collected data; RCT=randomised clinical trial. All comparators were no intervention or usual care if not stated otherwise. All but three outcomes (CD002990, CD003587, CD006525) were primary outcomes of the Cochrane review. In four Cochrane reviews (CD000313, CD001886, CD004815, CD007746) a pertinent mortality outcome was also reported, which was used for the secondary analysis. the bmj | BMJ 2021;372:n450 | doi: 10.1136/bmj.n450 5 RESEARCH

2005 (interquartile range 1998-2009). The sources 13%) and were published between 1963 and 2016 of routinely collected data were registries (36/84, (median 2003 (interquartile range 1997-2006); table BMJ: first published as 10.1136/bmj.n450 on 3 March 2021. Downloaded from 43%), electronic health records (30/84, 36%), and 1 and see appendix 4). administrative databases (18/84, 21%). In 27 RCD- Of the 22 clinical questions, eight (36%) were RCTs (32%), a hybrid approach with elements of active related to screening and preventive medicine, data collection was applied. five (23%) to community medicine, five (23%) The quality of the data was considered high for 56 of to cardiology, and four (18%) to surgery. Eleven the 84 RCD-RCTs (67%; moderate interrater agreement comparisons had only one RCD-RCT, four 77.4%; κ=0.50, weighted κ=0.48). comparisons had two RCD-RCTs, three comparisons The 463 traditional RCTs had a median 121 had three RCD-RCTs, and four comparison had four (interquartile range 60-359) participants in each RCD-RCTs or more (table 2). Outcomes were diverse, trial. The trials were primarily from North America with a large proportion related to mortality (9 of (125/463, 27%) and continental Europe (60/463, 22 in the main analysis; 41%). In 19 of 22 cases

Clinical topic Odds ratio Summary CI (95%) Range of (95% CI) odds ratio odds ratios of clinical trials

Individualised discharge plans on readmissions RCD-RCT 1.18 1.00 to 1.38 1.05 to 1.23 Traditional RCT 0.74 0.63 to 0.89 0.31 to 1.80 Breastfeeding support on stopping breastfeeding RCD-RCT 1.11 0.84 to 1.47 1.11 to 1.11 Traditional RCT 0.75 0.66 to 0.86 0.03 to 1.53 Mammography screening on breast cancer mortality RCD-RCT 0.91 0.74 to 1.11 0.71 to 1.36 Traditional RCT 0.74 0.60 to 0.91 0.65 to 0.83 Anti brinolytic agents on need for allogenic blood transfusions RCD-RCT 0.02 0.00 to 0.56 0.02 to 0.02 http://www.bmj.com/ Traditional RCT 0.37 0.32 to 0.43 0.01 to 3.32 Interventions to increase cervical cancer screening uptake RCD-RCT 0.70 0.53 to 0.93 0.17 to 0.95 Traditional RCT 0.59 0.38 to 0.92 0.20 to 1.20 COPD self-management interventions on mortality RCD-RCT 0.67 0.31 to 1.46 0.67 to 0.67 on 27 September 2021 by guest. Protected copyright. Traditional RCT 0.81 0.60 to 1.08 0.30 to 1.36 Excercise based interventions on hospital admissions RCD-RCT 0.92 0.77 to 1.09 0.92 to 0.92 Traditional RCT 0.47 0.19 to 1.14 0.20 to 1.88 Fast track interventions for early extubation on mortality RCD-RCT 0.33 0.27 to 0.40 0.32 to 0.33 Traditional RCT 0.85 0.14 to 4.97 0.14 to 3.20 IUD for heavy menstrual bleeding on additional surgery received RCD-RCT 55.74 3.34 to 929.15 55.74 to 55.74 Traditional RCT 2.54 0.84 to 7.73 0.33 to 6.50 Immunisation reminder and recalls on immunisations RCD-RCT 0.71 0.64 to 0.78 0.16 to 0.96 Traditional RCT 0.54 0.43 to 0.68 0.15 to 1.16 Routine coronary interventions in UA/NSTEMI on mortality or MI RCD-RCT 0.77 0.63 to 0.94 0.77 to 0.77 Traditional RCT 0.96 0.06 to 15.92 0.78 to 1.22

0.1 0.2 0.5 11250 Favours Favours experimental control

Fig 1 | Overview of summary results of treatment effects measured with or without routinely collected data (RCD) in randomised clinical trials for 22 clinical questions. COPD=chronic obstructive pulmonary disease; IUD=intrauterine device; sOR=summary odds ratio; traditional RCT=randomised clinical trial not using RCD for outcome collection; MI=myocardial infarction; ROR=ratio of odds ratios; UA/NSTEMI=unstable angina/non-ST- elevation MI. Ordered by ROR

6 doi: 10.1136/bmj.n450 | BMJ 2021;372:n450 | the bmj RESEARCH

(86%) the outcomes were a primary outcome of the 0.70 to 0.91, I2=14%) (fig 3 and table 3; see appendix Cochrane review. 3). In four of the 22 clinical questions (individualised BMJ: first published as 10.1136/bmj.n450 on 3 March 2021. Downloaded from discharge plans on readmissions, intrauterine device Agreement of treatment effects for heavy menstrual bleeding, breastfeeding support In 19 of 22 cases (86%), treatment effect estimates for healthy women, and immunisation reminders and from RCD-RCTs and from traditional trials were in the recalls), the 95% confidence intervals of the ratio of same direction. In 14 of 22 cases (63%), the summary odds ratios excluded the null, and in all four clinical point estimate of the RCD-RCT was less favourable (fig questions, trials using routinely collected data had less 1 and fig 2). favourable results than traditional trials (fig 3). Overall, trials using routinely collected data for The results were similar when including only any outcome ascertainment systematically showed available primary outcomes of Cochrane reviews (ratio less favourable estimates of treatment effects than of odds ratios 0.79, 95% confidence interval 0.70 to traditional trials not using routinely collected data 0.90, I2=9%) or mortality outcomes (0.92, 0.74 to 1.15, (ratio of odds ratios 0.80, 95% confidence interval I2=12%), or studies with routinely collected data when

Clinical topic Odds ratio Summary CI (95%) Range of (95% CI) odds ratio odds ratios of clinical trials

Any interventions to reduce falls RCD-RCT 0.80 0.01 to 109.30 0.44 to 1.03 Traditional RCT 0.97 0.81 to 1.15 0.70 to 1.03 Collaborative care on antidepressant drug use RCD-RCT 0.56 0.46 to 0.68 0.20 to 0.93 Traditional RCT 0.41 0.29 to 0.56 0.04 to 1.87 Antioxidant supplementation on mortality RCD-RCT 1.09 0.76 to 1.55 1.09 to 2.01 Traditional RCT 1.01 0.97 to 1.05 0.09 to 3.42

On-pump CABG on mortality http://www.bmj.com/ RCD-RCT 0.56 0.32 to 0.97 0.56 to 0.56 Traditional RCT 0.86 0.69 to 1.08 0.19 to 20.21 Telephone support or telemonitoring on mortality RCD-RCT 0.99 0.58 to 1.67 0.56 to 1.13 Traditional RCT 0.64 0.47 to 0.87 0.25 to 7.44 Mycophenolic acid v azathioprine on gra loss on 27 September 2021 by guest. Protected copyright. RCD-RCT 1.04 0.49 to 2.17 1.04 to 1.04 Traditional RCT 0.63 0.17 to 2.31 0.37 to 1.00 Statins on mortality RCD-RCT 0.77 0.41 to 1.45 0.68 to 0.79 Traditional RCT 0.75 0.55 to 1.02 0.20 to 1.50 Case management on hospital admissions RCD-RCT 0.81 0.06 to 10.68 0.65 to 0.98 Traditional RCT 0.86 0.14 to 5.13 0.46 to 1.59 Drug review on mortality RCD-RCT 1.68 0.51 to 5.54 1.68 to 1.68 Traditional RCT 1.02 0.84 to 1.24 0.88 to 1.73 Interventions to reduce dietary salt on mortality RCD-RCT 0.96 0.78 to 1.19 0.96 to 0.96 Traditional RCT 0.90 0.61 to 1.31 0.63 to 3.0 Fish oil on allergies RCD-RCT 0.64 0.29 to 1.39 0.64 to 0.64 Traditional RCT 0.94 0.63 to 1.41 0.70 to 1.09

0.1 0.2 0.5 11250 Favours Favours experimental control

Fig 2 | Overview of summary results of treatment effects measured with or without routinely collected data (RCD) in clinical trials for 22 clinical questions. CABG=coronary artery bypass grafting; sOR=summary odds ratio; traditional RCT=randomised clinical trial not using RCD for outcome collection; ROR=ratio of odds ratios. Ordered by ROR the bmj | BMJ 2021;372:n450 | doi: 10.1136/bmj.n450 7 RESEARCH

Clinical topic Odds ratio Odds ratio Weight (%) (95% CI) (95% CI) BMJ: first published as 10.1136/bmj.n450 on 3 March 2021. Downloaded from

Individualised discharge plans on readmissions 0.63 (0.50 to 0.80) 15.4 Breastfeeding support on stopping breastfeeding 0.68 (0.50 to 0.92) 10.9 Mammography screening on breast cancer mortality 0.81 (0.61 to 1.08) 12.3 Anti brinolytic agents on need for allogenic blood transfusions 16.42 (0.66 to 405.72) 0.1 Interventions to increase cervical cancer screening uptake 0.85 (0.50 to 1.43) 4.7 COPD self-management interventions on mortality 1.21 (0.53 to 2.77) 2.0 Excercise based interventions on hospital admissions 0.51 (0.21 to 1.26) 1.7 Fast track interventions for early extubation on mortality 2.58 (0.43 to 15.35) 0.5 IUD for heavy menstrual bleeding on additional surgery received 0.05 (0.00 to 0.94) 0.2 Immunisation reminder and recalls on immunisations 0.76 (0.59 to 0.98) 14.6 Routine coronary interventions in UA/NSTEMI on mortality or MI 1.24 (0.07 to 20.78) 0.2 Any interventions to reduce falls 1.20 (0.01 to 164.21) 0.1 Collaborative care on antidepressant drug use 0.73 (0.50 to 1.07) 8.1 Antioxidant supplementation on mortality 0.93 (0.65 to 1.33) 8.8 On-pump CABG on mortality 1.54 (0.85 to 2.78) 3.8 Telephone support or telemonitoring on mortality 0.64 (0.35 to 1.19) 3.6 Mycophenolic acid v azathioprine on gra loss 0.61 (0.14 to 2.71) 0.7 Statins on mortality 0.97 (0.48 to 1.95) 2.8 Case management on hospital admissions 1.06 (0.05 to 24.67) 0.2 Drug review on mortality 0.61 (0.18 to 2.04) 1.0 Interventions to reduce dietary salt on mortality 0.93 (0.60 to 1.44) 6.5 Fish oil on allergies 1.48 (0.61 to 3.56) 1.8 Random effects model 0.80 (0.70 to 0.91) 100.0

0.8 0.9 1.1 1.251 http://www.bmj.com/ Smaller effects Larger effects in RCD-RCTs in RCD-RCTs

Fig 3 | of main analysis showing agreement of treatment effects measured with or without routinely collected data (RCD) in clinical trials. CABG=coronary artery bypass grafting; COPD=chronic obstructive pulmonary disease; IUD=intrauterine device; sOR=summary odds ratio; traditional RCT=randomised clinical trial not using RCD for outcome collection; MI=myocardial infarction; ROR=ratio of odds ratios; UA/ NSTEMI=unstable angina/non-ST-elevation MI. Ordered by ROR on 27 September 2021 by guest. Protected copyright. the data quality were considered to be high (0.82, 0.72 effect estimates.2 For example, such trials might be to 0.93, I2=0%). The results were also similar when more pragmatic than traditional trials.2 5 18 28 More analysing electronic health records (0.81, 0.59 to 1.11, natural care settings with less eagerness to artificially I2=28%), registries (0.86, 0.75 to 0.99, I2=20%), and increase treatment adherence might result in smaller administrative data sources (0.84, 0.72 to 0.99, I2=0%; treatment effect estimates. table 3). All other sensitivity analyses corroborated the This interpretation agrees with empirical research main findings (table 3). indicating that procedures to standardise and increase data quality could have a smaller impact on trial effect Discussion estimates than is often assumed: a review indicated In this systematic analysis of various clinical topics that central outcome adjudication committees used to and outcomes, randomised clinical trials that used increase data quality typically did not influence effect routinely collected data for outcome ascertainment estimates compared with onsite assessments in the very showed less favourable treatment effects than same trial.29 Direct comparisons of treatment estimates traditional randomised clinical trials not using based on separate ways of outcome ascertainments are routinely collected data. This might be due to problems helpful to understand better the underlying mechanisms with data quality and measurement errors leading to of outcome measurements.30 In contrast with such dilution of effects by misclassified outcomes. However, research, we did not aim to isolate the “clean” effect of the results remained similar across sensitivity analyses using compared with not using routinely collected data dealing with this possibility, including data source within the same trial as alternative data ascertainment type and estimated data quality, or when including methods. Conversely, we aimed to empirically describe only mortality outcomes where misclassification how results from trials designed to provide randomised is probably less likely. Thus, trials using routinely real world evidence31 (by using ) agree collected data for outcome ascertainment might have with those from traditionally designed trials relying on other features that are associated with less pronounced their own, active data collection procedures.

8 doi: 10.1136/bmj.n450 | BMJ 2021;372:n450 | the bmj RESEARCH

Table 3 | Results of analyses comparing treatment effects measured with or without routinely collected data (RCD) in clinical trials BMJ: first published as 10.1136/bmj.n450 on 3 March 2021. Downloaded from No of clinical Analysis questions Ratio of odds ratios (95% CI) I2 (95% CI) (%) Main analysis 22 0.80 (0.70 to 0.91) 14 (0 to 48) Secondary analyses Mortality outcomes used when available 12 0.92 (0.74 to 1.15) 12 (0 to 52) Primary outcomes used when available 19 0.79 (0.70 to 0.90) 9 (0 to 45) Subgroup analyses Source of data collection: Registries 14 0.86 (0.75 to 0.99) 20 (0 to 57) Electronic health records 9 0.81 (0.59 to 1.11) 28 (0 to 67) Administrative claims data 9 0.84 (0.72 to 0.99) 0 (0 to 58) High data quality 17 0.82 (0.72 to 0.93) 0 (0 to 50) Sensitivity analyses Study design excluded: Hybrid data collection 18 0.88 (0.78 to 1.00) 0 (0 to 49) Cluster randomisation or crossover design 20 0.84 (0.69 to 1.02) 28 (0 to 58) Year of publication: ≥2005 15 0.73 (0.63 to 0.85) 4 (0 to 56) ≥2010 6 0.81 (0.55 to 1.19) 39 (0 to 76) <5 years before or after traditional RCTs 15 0.78 (0.69 to 0.88) 0 (0 to 21) Low risk of bias: All domains 10 0.85 (0.60 to 1.23) 49 (0 to 75) Blinding 13 0.87 (0.69 to 1.08) 28 (0 to 63) Trial size: Smallest (lowest third)* 13 0.82 (0.57 to 1.17) 0 (0 to 53) Medium (middle third)* 11 0.85 (0.72 to 1.02) 0 (0 to 43) Largest (largest third)* 4 0.92 (0.86 to 0.99) 0 (0 to 60) No of events status: Lowest (lowest third)† 14 1.12 (0.79 to 1.60) 1 (0 to 55) Medium (middle third)† 7 0.78 (0.65 to 0.92) 0 (0 to 11) Largest (largest third)† 4 0.93 (0.85 to 1.01) 0 (0 to 74) Age of RCD-RCT population‡ 13 0.81 (0.66 to 0.99) 49 (3 to 73) http://www.bmj.com/ Subsets: Mortality outcomes 9 0.91 (0.76 to 1.10) 0 (0 to 60) Non-mortality outcomes 12 0.71 (0.60 to 0.84) 8 (0 to 62) Reason for exclusion of clinical questions: Active comparators 19 0.80 (0.70 to 0.91) 14 (0 to 49) Largely different precision for each summary estimates 15 0.78 (0.68 to 0.91) 18 (0 to 55) <3 RCD-RCTs 7 0.73 (0.64 to 0.84) 0 (0 to 61)

>10 RCD-RCTs 20 0.82 (0.70 to 0.97) 22 (0 to 54) on 27 September 2021 by guest. Protected copyright. Index RCD-RCTs v all other§ 22 0.91 (0.78 to 1.06) 17 (0 to 51) Meta-analysis model: DerSimonian Laird random effects 22 0.84 (0.75 to 0.94) 40 (1 to 64) Fixed effect 22 0.88 (0.83 to 0.93) 40 (1 to 64) A ratio of odds ratios <1 indicates that the RCD-RCT estimated a less favourable treatment effect of the evaluated treatment than the traditional RCT. RCTs=randomised clinical trials. *Tertiles for participants were 333 and 1997, based on RCD-RCTs. †Tertiles for events were 75 and 502, based on RCD-RCTs. ‡Within standard deviation of median age of traditional RCTs. §Indirectly identified RCD-RCTs and traditional randomised clinical trials.

Comparison with other studies within some of these RCD-RCTs were still determined We are aware of only one other similar study that traditionally, thus introducing artificial settings that compared effects from 30 registry based trials with deviate from routine care. Therefore, some of the RCD- that from traditional trials on 12 different topics in RCTs might reflect the “real world” more and others less. cardiology or cancer screening.32 The reported ratio of Secondly, we did not directly evaluate the impact of odds ratios were 0.97 (95% confidence interval 0.92 trial pragmatism on treatment effects. The applicability of to 1.03) for mortality and 0.95 (0.89 to 1.02) for other research findings to real world settings can be determined outcomes (reported ratio of odds ratios inverted to by other factors, such as the representativeness of the facilitate comparison), compatible with our findings trial population or the treatment setting, which we have for registry based trials. not assessed. A deeper investigation of all RCD-RCTs and their comparators would be beyond the scope of this Limitations of this study project, and a valid retrospective assessment of each Several limitations need to be considered. Firstly, trial’s pragmatism and representativeness is difficult for although the outcome selected for our analysis was researchers outside of the original trial team, requiring routinely collected in the RCD-RCTs, other outcomes further information such as study protocols33 34 or details the bmj | BMJ 2021;372:n450 | doi: 10.1136/bmj.n450 9 RESEARCH

on the study population and target population that are JPAI analysed the data. KAM and LGH wrote the first draft of the typically unavailable. manuscript. All the authors interpreted the data, critically revised the BMJ: first published as 10.1136/bmj.n450 on 3 March 2021. Downloaded from manuscript for important intellectual content, and gave final approval Thirdly, although we individually assessed and of the version to be published. LGH and KAM are guarantors. The graded data quality and expected accuracies in corresponding author attests that all listed authors meet authorship duplicate, assessing the quality of the sources of criteria and that no others meeting the criteria have been omitted. routinely collected data is inherently subjective and Funding: The Basel Institute for Clinical Epidemiology and Biostatistics is supported by the Stiftung Institut für klinische limited because of widely insufficient reporting of Epidemiologie (KAM, LGH, EH, SA, and DG). METRICS has been critical details (such as results of data validation supported by grants from the Laura and John Arnold Foundation (JPAI studies). We are not aware of an established instrument and LGH). METRIC-B has been supported by an Einstein fellowship award to JPAI from the Stiftung Charite and the Einstein Stiftung (JPAI that would allow the “data quality” on an outcome and LGH). The funders had no role in the design and conduct of the level to be unambiguously determined using trial study; collection, management, analysis, and interpretation of the reports. Thus, interpretations in this regard need to be data; and preparation, review, or approval of the manuscript or its submission for publication. made very cautiously. Competing interests: All authors have completed the ICMJE uniform Fourthly, although our topics were evaluated in disclosure form at www.icmje.org/coi_disclosure.pdf and declare: Cochrane reviews and probably explore questions KAM, JPAI, and LGH support the RCD for RCT initiative, which aims to of interest for healthcare decision makers, they do explore the use of routinely collected data for clinical trials. KAM and LGH are members of the MARTA-Group, which aims to explore how not cover the full spectrum of clinical research. The to make randomised trials more affordable. Since 1 June 2020, DG statistical heterogeneity across topics was small, and has been employed by Roche Pharma (Schweiz), Basel, Switzerland. issues related to data quality and trial design vary The first draft of this manuscript was submitted before his current employment and his current employer had no role in the design across clinical specialties. It remains uncertain how and conduct of the project; preparation, review, and approval of the the results can be extrapolated to specific medical manuscript, and decision to submit the manuscript for publication. disciplines, and more evidence is needed to better The authors declare no other relationships or activities that could appear to have influenced the submitted work. assess the generalisability of our findings. However, Ethical approval: Not required. our assessment covers areas of clinical research where Data sharing: Available on request from the corresponding author using routinely collected data for outcome assessment and on the Open Science Framework.36 is a realistic alternative, indicated by the existence The lead author (the manuscript’s guarantor) affirms that the of trials using routinely collected data based and manuscript is an honest, accurate, and transparent account of the traditional outcome measurement. study being reported; that no important aspects of the study have been omitted; and that any relevant discrepancies from the study as Finally, some of our analyses rely on sometimes

planned have been explained. http://www.bmj.com/ 35 insufficiently reported details. Although we Dissemination to participants and related patient and public systematically ensured that the trials were actually communities: We plan to disseminate the results through measuring the analysed outcomes through routinely publications, conference presentations, and social media to international stakeholders and healthcare decision makers who use or collected data, poor reporting of such data use generate evidence based on routinely collected data. in the traditional trials could have led to some Provenance and peer review: Not commissioned; externally peer misclassification or we might have overlooked some reviewed. hybrid approaches. We have no reason to believe that This is an Open Access article distributed in accordance with the possible misclassifications are associated with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, on 27 September 2021 by guest. Protected copyright. which permits others to distribute, remix, adapt, build upon this work investigated agreement; hence, such errors would non-commercially, and license their derivative works on different have led to a dilution of the difference between the terms, provided the original work is properly cited and the use is non- compared study designs and not change our overall commercial. See: http://creativecommons.org/licenses/by-nc/4.0/. conclusion. 1 Hemkens LG, Contopoulos-Ioannidis DG, Ioannidis JPA. Routinely collected data and comparative effectiveness evidence: promises and Conclusions limitations. CMAJ 2016;188:E158-64. doi:10.1503/cmaj.150653 2 Mc Cord KA, Al-Shahi Salman R, Treweek S, et al. Routinely collected Randomised clinical trials utilising any form of data for randomized trials: promises, barriers, and implications. routinely collected data for outcome ascertainment Trials 2018;19:29. doi:10.1186/s13063-017-2394-5 found systematically less favourable treatment effects 3 Mc Cord KA, Ewald H, Ladanie A, et al, RCD for RCTs initiative and the Making Randomized Trials More Affordable Group. Current use than randomised clinical trials utilising traditional and costs of electronic health records for clinical trial research: methods. Differences might exist between traditional a descriptive study. CMAJ Open 2019;7:E23-32. doi:10.9778/ cmajo.20180096 trials and trial designs utilising routinely collected 4 Weisberg HI, Hayden VC, Pontes VP. Selection criteria and data beyond data quality problems that would explain generalizability within the counterfactual framework: explaining this finding. We need a better understanding of these the paradox of antidepressant-induced suicidality?Clin Trials 2009;6:109-18. doi:10.1177/1740774509102563 factors, to optimise the use of such emerging designs 5 Zwarenstein M, Treweek S. What kind of randomized trials do we for comparative effectiveness research and to increase need?CMAJ 2009;180:998-1000. doi:10.1503/cmaj.082007 6 Schwartz D, Lellouch J. Explanatory and pragmatic attitudes the applicability of real world evidence derived from in therapeutical trials. J Chronic Dis 1967;20:637-48. randomised trials. doi:10.1016/0021-9681(67)90041-0 We thank Aviv Ladanie for contributing to the literature screening 7 Zuidgeest MGP, Goetz I, Groenwold RHH, Irving E, van Thiel and data extraction and Julie Jacobson Vann for providing details on GJMW, Grobbee DE, GetReal Work Package 3. Series: Pragmatic trials and real world evidence: Paper 1. Introduction. J Clin included trials. Epidemiol 2017;88:7-13. doi:10.1016/j.jclinepi.2016.12.023 Contributors: LGH and JPAI conceived the study. LGH, KAM, and HE 8 Ramsberg J, Neovius M. Register or electronic health records enriched designed the search strategy. KAM performed the literature search. randomized pragmatic trials: The future of clinical effectiveness and KAM, LGH, HE, and DG screened the studies for eligibility. KAM, LGH, cost-effectiveness trials?Nordic Journal of Health Economics 2017;5. HE, DG, and AA performed the data extractions. LGH, KAM, SA, and doi:10.5617/njhe.1386

10 doi: 10.1136/bmj.n450 | BMJ 2021;372:n450 | the bmj RESEARCH

9 Nicholls SG, Langan SM, Benchimol EI. Routinely collected data: features of medicine trials with the PRECIS-2 tool. Eur J Clin Pharmacol

the importance of high-quality diagnostic coding to research. 2020;76:1093-102. doi:10.1007/s00228-020-02901-4 BMJ: first published as 10.1136/bmj.n450 on 3 March 2021. Downloaded from CMAJ 2017;189:E1054-5. doi:10.1503/cmaj.170807 35 Kwakkenbos L, Juszczak E, Hemkens LG, et al. Protocol for the 10 Arts DG, De Keizer NF, Scheffer GJ. Defining and improving data quality in development of a CONSORT extension for RCTs using cohorts and medical registries: a literature review, , and generic framework. routinely collected health data. Res Integr Peer Rev 2018;3:9. J Am Med Inform Assoc 2002;9:600-11. doi:10.1197/jamia.M1087 doi:10.1186/s41073-018-0053-3 11 Capurro D, Yetisgen M, van Eaton E, Black R, Tarczy-Hornoch P. 36 Janiaud P, Dueblin P, Hemkens LG. PragMeta: Generalizability, Availability of structured and unstructured clinical data for comparative applicability and pragmatism of clinical trials and their impact on effectiveness research and quality improvement: a multisite assessment. treatment effect estimates: a metaepidemiological study. https://osf. EGEMS (Wash DC) 2014;2:1079. doi:10.13063/2327-9214.1079 io/cdbhn/ 12 Feder SL. Data Quality in Electronic Health Records Research: Quality 37 Gonçalves-Bradley DC, Lannin NA, Clemson LM, Cameron ID, Shepperd Domains and Assessment Methods. West J Nurs Res 2018;40:753- S. Discharge planning from hospital. Cochrane Database Syst Rev 66. doi:10.1177/0193945916689084 2016;(1):CD000313. doi:10.1002/14651858.CD000313.pub5. 13 Weiskopf NG, Weng C. Methods and dimensions of electronic 38 McFadden A, Gavine A, Renfrew MJ, et al. Support for healthy health record data quality assessment: enabling reuse for clinical breastfeeding mothers with healthy term babies. Cochrane Database research. J Am Med Inform Assoc 2013;20:144-51. doi:10.1136/ Syst Rev 2017;2:CD001141. doi:10.1002/14651858.CD001141. amiajnl-2011-000681 pub5 14 Ladanie A, Speich B, Briel M, et al. Single pivotal trials with few 39 Gøtzsche PC, Jørgensen KJ. Screening for breast cancer with corroborating characteristics were used for FDA approval of cancer mammography. Cochrane Database Syst Rev 2013;(6):CD001877. therapies. J Clin Epidemiol 2019;114:49-59. doi:10.1016/j. doi:10.1002/14651858.CD001877.pub5. jclinepi.2019.05.033 40 Henry DA, Carless PA, Moxey AJ, et al. Anti-fibrinolytic use for 15 Emilsson L, Lindahl B, Köster M, Lambe M, Ludvigsson JF. minimising perioperative allogeneic blood transfusion. Cochrane Review of 103 Swedish Healthcare Quality Registries. J Intern Database Syst Rev 2011;(1):CD001886. doi:10.1002/14651858. Med 2015;277:94-136. doi:10.1111/joim.12303 CD001886.pub3 16 Harron K, Wade A, Gilbert R, Muller-Pebody B, Goldstein H. Evaluating 41 Everett T, Bryant A, Griffin MF, Martin-Hirsch PP, Forbes CA, Jepson RG. bias due to data linkage error in electronic healthcare records. BMC Interventions targeted at women to encourage the uptake of cervical Med Res Methodol 2014;14:36. doi:10.1186/1471-2288-14-36 screening. Cochrane Database Syst Rev 2011;(5):CD002834. 17 van Smeden M, Lash TL, Groenwold RHH. Reflection on modern doi:10.1002/14651858.CD002834.pub2 methods: five myths about measurement error in epidemiological 42 Zwerink M, Brusse-Keizer M, van der Valk PDLPM, et al. Self research. Int J Epidemiol 2020;49:338-47. doi:10.1093/ije/dyz251 management for patients with chronic obstructive pulmonary 18 Dechartres A, Trinquart L, Boutron I, Ravaud P. Influence of trial disease. Cochrane Database Syst Rev 2014;(3):CD002990. sample size on treatment effect estimates: meta-epidemiological doi:10.1002/14651858.CD002990.pub3 study. BMJ 2013;346:f2304. doi:10.1136/bmj.f2304 43 Long L, Mordi IR, Bridges C, et al. Exercise-based cardiac 19 Benchimol EI, Smeeth L, Guttmann A, et al, RECORD Working rehabilitation for adults with heart failure. Cochrane Database Syst Committee. The REporting of studies Conducted using Observational Rev 2019;1:CD003331. doi:10.1002/14651858.CD003331.pub5 Routinely-collected health Data (RECORD) statement. PLoS 44 Jacobson Vann JC, Jacobson RM, Coyne-Beasley T, Asafu-Adjei JK, Med 2015;12:e1001885. doi:10.1371/journal.pmed.1001885 Szilagyi PG. Patient reminder and recall interventions to improve 20 Hemkens LG. Commentary on Bertagnolli et al: Clinical trial designs with immunization rates. Cochrane Database Syst Rev 2018;1:CD003941. routinely collected real-world data-issues of data quality and beyond. doi:10.1002/14651858.CD003941.pub3 Clin Trials 2020;17:247-50. doi:10.1177/1740774520913845 45 Fanning JP, Nyong J, Scott IA, Aroney CN, Walters DL. Routine invasive

21 IntHout J, Ioannidis JP, Borm GF. The Hartung-Knapp-Sidik-Jonkman strategies versus selective invasive strategies for unstable angina http://www.bmj.com/ method for random effects meta-analysis is straightforward and and non-ST elevation myocardial infarction in the stent era. Cochrane considerably outperforms the standard DerSimonian-Laird method. Database Syst Rev 2016;(5):CD004815. doi:10.1002/14651858. BMC Med Res Methodol 2014;14:25. doi:10.1186/1471-2288-14-25 CD004815.pub4 22 Marjoribanks J, Lethaby A, Farquhar C. Surgery versus medical 46 Cameron ID, Gillespie LD, Robertson MC, et al. Interventions therapy for heavy menstrual bleeding. Cochrane Database Syst Rev for preventing falls in older people in care facilities and 2016;(1):CD003855. doi:10.1002/14651858.CD003855.pub3 hospitals. Cochrane Database Syst Rev 2012;12:CD005465. 23 Wong W-T, Lai VK, Chee YE, Lee A. Fast-track cardiac care doi:10.1002/14651858.CD005465.pub3 for adult cardiac surgical patients. Cochrane Database Syst 47 Archer J, Bower P, Gilbody S, et al. Collaborative care for Rev 2016;9:CD003587. doi:10.1002/14651858.CD003587.pub3 depression and anxiety problems. Cochrane Database Syst

24 Wagner M, Earley AK, Webster AC, Schmid CH, Balk EM, Uhlig K. Rev 2012;10:CD006525. doi:10.1002/14651858.CD006525.pub2 on 27 September 2021 by guest. Protected copyright. Mycophenolic acid versus azathioprine as primary immunosuppression 48 Bjelakovic G, Nikolova D, Gluud LL, Simonetti RG, Gluud C. Antioxidant for kidney transplant recipients. Cochrane Database Syst Rev supplements for prevention of mortality in healthy participants 2015;(12):CD007746. doi:10.1002/14651858.CD007746.pub2 and patients with various diseases. Cochrane Database Syst 25 Balduzzi S, Rücker G, Schwarzer G. How to perform a meta-analysis Rev 2012;(3):CD007176. doi:10.1002/14651858.CD007176.pub2 with R: a practical tutorial. Evid Based Ment Health 2019;22:153-60. 49 Møller CH, Penninga L, Wetterslev J, Steinbrüchel DA, Gluud C. Off- doi:10.1136/ebmental-2019-300117 pump versus on-pump coronary artery bypass grafting for ischaemic 26 RStudio. Open source & professional software for data science teams heart disease. Cochrane Database Syst Rev 2012;(3):CD007224. - RStudio. Accessed January 11, 2020. https://rstudio.com/ doi:10.1002/14651858.CD007224.pub2 27 R: The R Project for Statistical Computing. Accessed January 11, 50 Inglis SC, Clark RA, Dierckx R, Prieto-Merino D, Cleland JGF. Structured 2020. https://www.r-project.org/ telephone support or non-invasive telemonitoring for patients with 28 Patsopoulos NA. A pragmatic view on pragmatic trials. Dialogues heart failure. Cochrane Database Syst Rev 2015;(10):CD007228. Clin Neurosci 2011;13:217-24. doi:10.31887/DCNS.2011.13.2/ doi:10.1002/14651858.CD007228.pub3 npatsopoulos 51 Palmer SC, Navaneethan SD, Craig JC, et al. HMG CoA reductase 29 Ndounga Diakou LA, Trinquart L, Hróbjartsson A, et al. Comparison inhibitors (statins) for people with chronic kidney disease not of central adjudication of outcomes and onsite outcome requiring dialysis. Cochrane Database Syst Rev 2014;(5):CD007784. assessment on treatment effect estimates. Cochrane Database Syst doi:10.1002/14651858.CD007784.pub2. Rev 2016;3:MR000043. doi:10.1002/14651858.MR000043.pub2 52 Reilly S, Miranda-Castillo C, Malouf R, et al. Case management 30 Strom JB, Faridi KF, Butala NM, et al. Use of Administrative Claims approaches to home support for people with dementia. Cochrane to Assess Outcomes and Treatment Effect in Randomized Clinical Database Syst Rev 2015;1:CD008345. doi:10.1002/14651858. Trials for Transcatheter Aortic Valve Replacement: Findings From CD008345.pub2 the EXTEND Study. Circulation 2020;142:203-13. doi:10.1161/ 53 Christensen M, Lundh A. Medication review in hospitalised patients CIRCULATIONAHA.120.046159. to reduce morbidity and mortality. Cochrane Database Syst 31 U.S. Food and Drug Administration (FDA). Framework for FDA’s real- Rev 2016;2:CD008986. doi:10.1002/14651858.CD008986.pub3 world evidence program. Published online December 2018. https:// 54 Adler AJ, Taylor F, Martin N, Gottlieb S, Taylor RS, Ebrahim S. Reduced www.fda.gov/media/120060/download dietary salt for the prevention of cardiovascular disease. Cochrane 32 Mathes T, Klaßen P, Pieper D. No differences were found between Database Syst Rev 2014;(12):CD009217. doi:10.1002/14651858. effect estimates from conventional and registry-based randomized CD009217.pub3 controlled trials. J Clin Epidemiol 2019;105:80-91. doi:10.1016/j. 55 Gunaratne AW, Makrides M, Collins CT. Maternal prenatal and/ jclinepi.2018.09.011 or postnatal n-3 long chain polyunsaturated fatty acids (LCPUFA) 33 Dal-Ré R, Janiaud P, Ioannidis JPA. Real-world evidence: How supplementation for preventing allergies in early childhood. Cochrane pragmatic are randomized controlled trials labeled as pragmatic? Database Syst Rev 2015;(7):CD010085. doi:10.1002/14651858. BMC Med 2018;16:49. doi:10.1186/s12916-018-1038-2 CD010085.pub2 34 Dal-Ré R. Articles provided insufficient information to conduct an appropriate retrospective assessment of the pragmatic/explanatory Supplementary information: appendices 1-6

No commercial reuse: See rights and reprints http://www.bmj.com/permissions Subscribe: http://www.bmj.com/subscribe