Confidential: for Review Only

BMJ

Confidential: For Review Only

Data sharing and re-analysis for randomised controlled trials in leading biomedical journals with a full data-sharing policy: a survey of studies published in The BMJ and PLOS Medicine

Journal: BMJ

Manuscript ID BMJ.2017.041198.R1

Article Type: Research

BMJ Journal: BMJ

Date Submitted by the Author: 30-Nov-2017

Complete List of Authors: Naudet, Florian; INSERM Paris France, U669 Sakarovich, Charlotte; Stanford University, Quantitative Science Unit Janniaud, Perrine; Stanford University, METRICS Cristea, Ioana Alina; Babes Bolyai University, Clinical Psychology and Psychotherapy; University of Pisa, Department of Surgical, Medical, Molecular and Critical Pathology Fanelli, Daniele; Stanford University Moher, David; Ottawa Hospital Research Institute, Ottawa Methods Centre Ioannidis, John; Stanford University, Stanford Prevention Research Center, Department of Medicine and Department of Health Research and Policy

Keywords: Data sharing, Randomised Controlled Trials, Reproducibility, Transparency

https://mc.manuscriptcentral.com/bmj Page 1 of 63 BMJ

1 2 3 1 Data sharing and re-analysis for randomised controlled trials in leading biomedical 4 2 journals with a full data-sharing policy: a survey of studies published in The BMJ and 5 6 3 PLOS Medicine 7 4 8 5 9 6 Florian Naudet, Charlotte Sakarovitch, Perrine Janiaud, Ioana Cristea, Daniele Fanelli, David 10 7 Moher, JohnConfidential: P.A. Ioannidis For Review Only 11 8 12 9 13 1 14 10 Florian Naudet (Postdoctoral Fellow) ORCID: 0000-0003-3760-3801 2 15 11 Charlotte Sakarovitch, (Senior Statistician) 16 12 Perrine Janiaud, (Postdoctoral Fellow)1 ORCID: 0000-0001-7684-8014 17 13 Ioana Cristea, (Visiting Scholar)1-3 ORCID: 0000-0002-9854-7076 18 14 Daniele Fanelli, (Senior Scientist)1 ORCID: 0000-0003-1780-1958 19 15 David Moher, (Visiting Scholar)1, 4 ORCID: 0000-0003-2434-4206 20 1, 5 21 16 John P.A. Ioannidis (Professor) ORCID: 0000-0003-3118-6859 22 17 23 18 1/ Meta-Research Innovation Center at Stanford, Stanford University, Stanford, California, 24 19 USA; 2/ Quantitative Sciences Unit, Division of Biomedical Informatics Research, Department 25 20 of Medicine, Stanford University, Stanford, CA, USA ; 3/ Department of Clinical Psychology and 26 27 21 Psychotherapy, Babes-Bolyai University, Romania ; 4/ Centre for Journalology, Clinical 28 22 Epidemiology Program, Ottawa Hospital Research Institute, Ottawa, Canada; 5/ Departments of 29 23 Medicine, of Health Research and Policy, of Biomedical Data Science, and of Statistics, Stanford 30 24 University, Stanford, California, USA. 31 25 32 26 33 27 34 35 28 36 29 Correspondence to: 37 30 John P.A. Ioannidis ([email protected]) 38 31 Meta-Research Innovation Center at Stanford (METRICS), Stanford University School of 39 32 Medicine, Stanford University, Stanford, California, USA 40 33 Tel: +1 650 725-5465; fax: +1 650 725-6247. 41 42 43 34 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 1 60 https://mc.manuscriptcentral.com/bmj BMJ Page 2 of 63

1 2 3 35 Copyright/licence for publication 4 36 5 37 The Corresponding Author has the right to grant on behalf of all authors and does grant on behalf of all authors, a 6 38 worldwide licence to the Publishers and its licensees in perpetuity, in all forms, formats and media (whether known 7 39 now or created in the future), to i) publish, reproduce, distribute, display and store the Contribution, ii) translate the 8 40 Contribution into other languages, create adaptations, reprints, include within collections and create summaries, 9 41 extracts and/or, abstracts of the Contribution, iii) create any other derivative work(s) based on the Contribution, iv) 10 42 to exploit all subsidiary rights in the Contribution, v) the inclusion of electronic links from the Contribution to third 11 43 party materialConfidential: where-ever it may be located; and, vi)For licence any Review third party to do any orOnly all of the above. 12 44 13 45 Declaration of competing interests 14 46 15 47 All authors have completed the Unified Competing Interest form at http://www.icmje.org/coi_disclosure.pdf 16 48 (available on request from the corresponding author) and declare that (1) No authors have support from any company for the submitted work; (2) FN has relationships (Travel/accommodations expenses covered/reimbursed) 17 49 50 with Servier, BMS, Lundbeck and Janssen who might have an interest in the work submitted in the previous 3 years. 18 51 In the past three years PJ received a fellowship/grant from GSK for her PhD as part of a public-private 19 52 collaboration. CS, IC, DF, DM and JPAI have no relationship with any company who might have an interest in the 20 53 work submitted. (3) No author’s spouse, partner, or children have any financial relationships that could be relevant 21 54 to the submitted work; and (4) none of the authors has any non-financial interests that could be relevant to the 22 55 submitted work. 23 56 24 57 Authorship & contributorship 25 58 26 59 Conceived and designed the experiments: FN, DF, JI. 27 60 Performed the experiments: FN, PJ, CS, IC, JI. 28 61 Analysed the data: FN, CS. 29 62 Interpreted the results: FN, CS, PJ, IC, DF, DM, JI. 30 63 Wrote the first draft of the manuscript: FN. 31 64 Contributed to the writing of the manuscript: CS, PJ, IC, DF, DM, JI. 32 65 Agreed with the results and conclusions of the manuscript: FN, CS, PJ, IC, DF, DM, JI. 33 66 All authors have read, and confirm that they meet ICMJE criteria for authorship. 34 67 35 68 All authors had full access to all of the data (including statistical reports and tables) in the study and can take 36 69 responsibility for the integrity of the data and the accuracy of the data analysis. 37 70 38 71 Florian Naudet is the guarantor. He affirms that the manuscript is an honest, accurate, and transparent account of the 39 72 study being reported; that no important aspects of the study have been omitted; and that any discrepancies from the study as planned have been explained. 40 73

41 74 75 Data sharing statement 42 76 43 77 Data sharing: code will be shared on the Open Science Framework when the manuscript is accepted. All datasets 44 78 that were used are retrievable following the instruction of the original papers. 45 79 46 80 Ethics approval 47 81 48 82 Ethics approval was not required. 49 83 50 84 Funding 51 85 52 86 METRICS has been funded by Laura and John Arnold Foundation. In addition, FN received grants from La 53 87 Fondation Pierre Deniker, Rennes University Hospital, France (CORECT: COmité de la Recherche Clinique et 54 88 Translationelle) and Agence Nationale de la Recherche (ANR), PJ is supported by a post-doctoral fellowship from 55 89 the Laura and John Arnold Foundation, IC was supported by the Laura and John Arnold Foundation and the 56 90 Romanian National Authority for Scientific Research and Innovation, CNCS–UEFISCDI, project number PN-II- 57 58 59 2 60 https://mc.manuscriptcentral.com/bmj Page 3 of 63 BMJ

1 2 3 91 RU-TE-2014-4-1316 (awarded to Dr. Cristea), and the work of J.I. is supported by an unrestricted gift from Sue and 4 92 Bob O’Donnell. The sponsors had no role concerning preparation, review or approval of the manuscript. 5 93 6 94 Acknowledgments 7 95 8 96 This study was made possible through sharing of anonymized individual participant data from all studies’ authors. 9 97 We thanks all authors that were contacted for this study: C. Bullen and the National Institute for Health Innovation, 10 98 S. Gilbody, C. Hewitt, L. Littlewood, C. van der Meulen, H. van der Aa, S. Cohen, M. Bicket, T. Harris, the STOP 11 99 GAP studyConfidential: investigators including Kim Thomas, AlanFor Montgomery Review and Nicola Greenlaw, Only Nottingham University 12 100 Hospitals NHS Trust, NIHR Programme Grants for Applied Research, the Nottingham Clinical Trials Unit, C. 13 101 Poylac, K. Yuhas, C. Adrion, G. Greisen, S. Hyttel-Sørensen A. Barker, R. Morello, K. Luedtke, M. Paul, D. Yahav, 14 102 L. Chesterton, the Arthritis Research UK Primary Care Centre and C. Hanson. 15 103 16 17 104 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 3 60 https://mc.manuscriptcentral.com/bmj BMJ Page 4 of 63

1 2 3 105 PRINT ABSTRACT 4 5 6 106 Study question 7 8 107 To explore the effectiveness of randomised controlled trial (RCT) data sharing in journals with a 9 10 108 full data-sharing policy. 11 Confidential: For Review Only 12 109 Methods 13 14 15 110 RCTs that had been submitted and published by The BMJ and PLOS Medicine subsequent to the 16 17 111 adoption of data sharing policies by these journals were identified from PubMed/Medline. Data 18 19 112 availability was defined as the eventual receipt of complete data with clear labelling. All primary 20 21 22 113 outcomes were then re-analysed to assess to what extent studies were reproduced. 23 24 114 Study answer and limitations 25 26 115 Thirty-seven RCTs (21 from The BMJ and 16 from PLOS Medicine) published between 2013 27 28 29 116 and 2016 met our eligibility criteria. 17/37 (46% (95% CI [30% to 62%]) satisfied our definition 30 31 117 of data availability and 14 of the 17 (82%, 95% CI [59% to 94%]) were fully reproduced on all 32 33 118 their primary outcomes. Of the remaining, we identified errors in two RCTs but reached similar 34 35 119 conclusions and one paper did not provide enough information in the Methods section to 36 37 38 120 reproduce the analyses. We identified difficulties, such as problems in contacting corresponding 39 40 121 authors and lack of resources on their behalf in preparing the datasets. However, our study might 41 42 122 be perceived by some authors as imposing unreasonable requests on them. 43 44 45 123 What this study adds 46 47 124 Data availability was not optimal in journals with a full data-sharing policy, but the 46% data 48 49 125 sharing rate we observed is much higher than elsewhere in the biomedical literature. When re- 50 51 52 126 analyses are possible, these mostly yield similar results to the original analysis. 53 54 127 Funding, competing interests, data sharing 55 56 128 Our center has been funded by Laura and John Arnold Foundation, there is no relevant 57 58 59 4 60 https://mc.manuscriptcentral.com/bmj Page 5 of 63 BMJ

1 2 3 129 competing interest. Data will be shared on the Open Science Framework (OSF). 4 5 6 130 Study registration 7 8 131 OSF (osf.io/c4zke) 9 10 132 11 Confidential: For Review Only 12 133 13 14 134 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 5 60 https://mc.manuscriptcentral.com/bmj BMJ Page 6 of 63

1 2 3 135 ABSTRACT 4 136 5 6 7 137 Objective – To explore the effectiveness of randomised controlled trial (RCT) data sharing in 8 9 138 journals with a full data-sharing policy and to describe potential difficulties encountered in the 10 Confidential: For Review Only 11 139 process of performing re-analyses of the primary outcome(s). 12 13 14 140 Design – Survey of published RCTs. Data availability was defined as the eventual receipt of 15 16 141 complete data with clear labelling. All primary outcomes were then re-analysed to assess to what 17 18 142 extent studies were reproduced. All difficulties encountered were described. 19 20 21 143 Data sources – Eligible studies were identified from PubMed/Medline. 22 23 144 Eligibility criteria for selecting studies – RCTs that had been submitted and published by The 24 25 145 BMJ and PLOS Medicine subsequent to the adoption of data sharing policies by these journals. 26 27 146 Results – Thirty-seven RCTs (21 from The BMJ and 16 from PLOS Medicine) published 28 29 30 147 between 2013 and 2016 met our eligibility criteria. 17/37 (46% (95% CI [30% to 62%]) satisfied 31 32 148 our definition of data availability and 14 of the 17 (82%, 95% CI [59% to 94%]) were fully 33 34 149 reproduced on all their primary outcomes. Of the remaining, we identified errors in two RCTs 35 36 37 150 but reached similar conclusions and one paper did not provide enough information in the 38 39 151 Methods section to reproduce the analyses. We identified difficulties, such as problems in 40 41 152 contacting corresponding authors and lack of resources on their behalf in preparing the datasets. 42 43 44 153 In addition, we found a range of different data sharing practices across study groups. 45 46 154 Conclusions - Data availability was not optimal even in journals with a strong policy for data 47 48 155 sharing. When investigators shared data, most re-analyses largely reproduced the original results. 49 50 156 Data sharing practices need to become more widespread and streamlined to allow meaningful re- 51 52 53 157 analyses and re-use of data. 54 55 158 Registration number - Open Science Framework (osf.io/c4zke) 56 57 58 59 6 60 https://mc.manuscriptcentral.com/bmj Page 7 of 63 BMJ

1 2 3 159 4 5 6 160 7 8 9 10 11 Confidential: For Review Only 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 7 60 https://mc.manuscriptcentral.com/bmj BMJ Page 8 of 63

1 2 3 161 What is already known on this subject 4 5 6 162 7 8 163 • In 2017, the International Committee of Medical Journal Editors mandate that a data 9 10 164 sharing plan to be included in each paper (and pre-specified in study registration). 11 Confidential: For Review Only 12 13 165 • Two leading general medical journals, The BMJ and PLOS Medicine, already have a 14 15 166 stronger policy expressly requiring data-sharing as a condition for publication of 16 17 167 randomised controlled trials (RCTs). 18 19 20 168 • Only a small number of re-analyses of RCTs have been published to date; of these, only a 21 22 169 minority was conducted by entirely independent authors. 23 24 170 25 26 171 What this study adds 27 28 29 172 30 31 173 • Data availability was not optimal even in journals with a stronger policy, but the 46% 32 33 174 data sharing rate we observed is much higher than elsewhere in the biomedical literature. 34 35 36 175 • When re-analyses are possible, these mostly yield similar results to the original analysis. 37 38 176 However, these re-analyses used data at a very mature analytical stage. Exploring the 39 40 177 impact of data processing, coding, cleaning and re-categorization of events on the results 41 42 43 178 requires further research. 44 45 179 • Problems in contacting corresponding authors, lack of resources in preparing the datasets 46 47 180 and important heterogeneity in data sharing practices are barriers to overcome. 48 49 50 181 51 52 182 53 183 54 184 55 56 57 58 59 8 60 https://mc.manuscriptcentral.com/bmj Page 9 of 63 BMJ

1 2 3 185 INTRODUCTION 4 5 186 6 7 187 Ideally, patients, medical practitioners and health policy analysts feel more confident when the 8 188 results and conclusions of scientific studies can be verified. Value of medical care increases 9 10 189 when people waste less time pursuing wrong claims. However, for a long time, verifying the 11 Confidential: For Review Only 12 190 results of clinical trials was not possible, since the data on which their conclusions are based 13 14 191 were not available. 15 192 16 17 193 In 2016, the International Committee of Medical Journal Editors (ICMJE) published an editorial1 18 19 194 stating that “it is an ethical obligation to responsibly share data generated by interventional 20 21 195 clinical trials because participants have put themselves at risk.” The ICJME proposed to require 22 196 that de-identified individual-patient data (IPD) are made publicly available no later than 6 23 24 197 months after publication of the trial results. This proposal triggered debate.2-7 In June 2017, the 25 26 198 ICJME stepped back from their proposal. The new requirements do not mandate data sharing, 27 199 but only a data sharing plan to be included in each paper (and pre-specified in study 28 29 200 registration).8 30 31 201 32 33 202 Because of this trend toward a new norm where data sharing for randomized controlled trials 34 203 (RCTs) becomes a standard, it seems important to assess how accessible the data are in journals 35 36 204 with existing data-sharing policies. Two leading general medical journals, The BMJ9 10 and 37 11 38 205 PLOS Medicine, already have a policy expressly requiring data-sharing as a condition for 39 206 publication of clinical trials: data sharing became a requirement after January 2013 for RCTs on 40 41 207 drugs and devices9 and July 2015 for all therapeutics10 at The BMJ, and after March 2014 for all 42 43 208 types of interventions at PLOS Medicine. 44 45 209 46 210 The aim of our study was to explore the effectiveness of RCT data sharing in both journals in 47 48 211 terms of data availability, feasibility and accuracy of re-analyses, and to describe potential 49 50 212 difficulties encountered in the process of performing re-analyses of the primary outcome(s). We 51 52 213 focused on RCTs because they are considered to represent high quality evidence and because 53 214 availability of their data is crucial in the evaluation of health interventions. RCTs represent the 54 55 215 most firmly codified methodology, which also allows data to be most easily analysed. Moreover, 56 57 58 59 9 60 https://mc.manuscriptcentral.com/bmj BMJ Page 10 of 63

1 2 3 216 RCTs have been the focus of transparency and data sharing initiatives for a longer time,12 due to 4 5 217 the importance of primary data availability in the evaluation of therapeutics (e.g. for individual 6 7 218 patient data meta-analyses). 8 219 9 10 220 METHODS 11 Confidential: For Review Only 12 221 13 14 222 Methods were specified in advance and documented in a protocol submitted for review on 2016- 15 223 11-12 and subsequently registered with the Open Science Framework on 2016-11-15 16 17 224 (registration number: osf.io/c4zke; see https://osf.io/u6hcv/register/565fb3678c5e4a66b5582f67). 18 19 225 20 21 226 Eligibility criteria 22 227 23 24 228 We surveyed publications of RCTs, including cluster trials and cross-over studies, non- 25 26 229 inferiority design and superiority designs, that had been submitted and published by The BMJ 27 230 and PLOS Medicine subsequently to the adoption of data sharing policies by these journals. Data 28 29 231 sharing became a requirement after January 2013 for RCTs on drugs and devices9 and July 2015 30 10 31 232 for all therapeutics at The BMJ, and after March 2014 for all types of interventions at PLOS 32 33 233 Medicine. 34 234 35 36 235 Search strategy 37 38 236 39 237 Eligible studies were identified from PubMed/Medline. For The BMJ the following search 40 41 238 strategy was used: "BMJ"[jour] AND ("2013/01/01"[PDAT]: "2017/01/01"[PDAT]) AND 42 43 239 Randomized Controlled Trial[ptyp]. For PLOS Medicine the following search strategy was used: 44 45 240 ""PLoS Med"[jour]) AND ("2014/03/01"[PDAT]: "2017/01/01"[PDAT]) AND Randomized 46 241 Controlled Trial[ptyp]. 47 48 242 49 50 243 Study selection 51 52 244 53 245 The eligibility assessment was performed independently by two reviewers (FN and PJ). 54 55 246 Disagreements were resolved by consensus or in consultation with a third reviewer (JPAI or 56 57 58 59 10 60 https://mc.manuscriptcentral.com/bmj Page 11 of 63 BMJ

1 2 3 247 DM). More specifically, the eligibility assessment was based on the date of submission (and not 4 5 248 of publication). When these dates were not available they were obtained by contacting the 6 7 249 journal editors. 8 250 9 10 251 Data extraction and datasets retrieval 11 Confidential: For Review Only 12 252 13 14 253 A data extraction sheet was developed. For each included study, information was extracted on 15 254 study characteristics (country of the corresponding author, design, sample size, condition and 16 17 255 funding), type of intervention (drug, device, other) and procedure to gather the data. Two authors 18 19 256 (FN and PJ) independently extracted the data from the included studies. Disagreements were 20 21 257 resolved by consensus or in consultation with a third reviewer (JPAI). One reviewer (FN) was in 22 258 charge of retrieving the individual patient data for all included studies by following the 23 24 259 instructions found in the data-sharing statement of the included studies. More specifically, when 25 26 260 data were available upon request, a standardised e-mail was sent (see https://osf.io/h9cas/). All 27 261 initial emails were sent from a professional email address ([email protected]), and 3 28 29 262 additional reminders were sent to each author 2-3 weeks apart, in case of non-response. 30 31 263 32 33 264 Data availability 34 265 35 36 266 Our primary outcome was data availability, defined as the eventual receipt of data presented with 37 38 267 sufficient information to reproduce the analysis of the primary outcome(s) of the included RCTs 39 268 (i.e. complete data with clear labelling). Additional information was collected on type of data 40 41 269 sharing (upon request by e-mail, upon request on a specific website, upon request on a specific 42 43 270 register, available on a public register, other); time for collecting the data (in days, time between 44 45 271 first attempt to success of getting a database); de-identification of data (concerning name, 46 272 birthdate and address); type of data shared (from case report forms to directly analysable 47 48 273 datasets);13 sharing of analysis code; and reasons for non-availability in case data were not 49 50 274 shared. 51 52 275 53 276 Reproducibility 54 55 277 56 57 58 59 11 60 https://mc.manuscriptcentral.com/bmj BMJ Page 12 of 63

1 2 3 278 When data were available, a re-analysis of the trial was performed by a single researcher (FN). 4 5 279 For each study, analyses were repeated exactly as described in the published report of the study. 6 7 280 Whenever insufficient details about the analysis was provided in the study report, we sought 8 281 clarifications from the trial investigators. Only analyses concerning the primary outcome (or 9 10 282 outcomes, if multiple primary outcomes existed) of each trial were considered. Any discrepancy 11 Confidential: For Review Only 12 283 between results obtained in the re-analysis and those reported in the publication was examined in 13 14 284 consultation with a statistician (CS). This examination aimed to determine if, based on both 15 285 quantitative (effect size, p-values) and qualitative (clinical judgment) consideration, the 16 17 286 discrepant results of the re-analysis entailed a different conclusion from the one reported in the 18 19 287 original publication. Any disagreement or uncertainty over such conclusions was resolved by 20 21 288 consulting a third co-author with expertise in both clinical medicine and statistical methodology 22 289 (JPAI). If, following this assessment process, results (and eventually conclusions) were still 23 24 290 determined to be not reproduced, CS independently re-analysed the data to confirm such 25 26 291 conclusion. Once the “not reproduced” status of a publication was confirmed, the authors of the 27 292 study were contacted by FN to discuss the source of the discrepancy. Following this assessment 28 29 293 procedure, studies were classified into 4 categories: fully reproduced; not fully reproduced but 30 31 294 same conclusion; not reproduced and different conclusion; and not reproduced (or partially 32 33 295 reproduced) because of missing information. 34 296 35 36 297 Difficulties encountered in getting data and/or code, using them and performing the re- 37 38 298 analyses 39 299 40 41 300 We noted whether the sharing of data and/or analytical code required clarifications for which 42 43 301 additional queries had to be made to the authors in order to obtain the relevant data and/or code, 44 45 302 clarify their labels and/or use, and reproduce the original analysis of the primary outcomes. A 46 303 catalogue of these queries was created and similar clarifications were grouped for descriptive 47 48 304 purposes in order to generate a list of some common challenges and to help address these 49 50 305 challenges pre-emptively in future published trials. 51 52 306 53 307 Statistical analyses 54 55 308 56 57 58 59 12 60 https://mc.manuscriptcentral.com/bmj Page 13 of 63 BMJ

1 2 3 309 Percentages of data sharing or reproducibility were computed with 95% confidence intervals 4 5 310 based on binomial approximation or on Wilson score method without continuity correction if 6 14 7 311 necessary. For the purposes of registration, we hypothesized that, if effective, these data 8 312 sharing policies would lead to more than 80 % of studies sharing their data (i.e. the lower 9 10 313 boundary of its confidence interval had to be above 80 %). One should expect high rates of data 11 Confidential: For Review Only 12 314 sharing resulting from full data sharing policies. However, based on previous experience of 13 15 14 315 explicit data sharing policies in Psychological Science we knew that a rate of 100% was not 15 316 realistic and judged that an 80% rate could be a desirable outcome. 16 17 317 When data was available, re-analyses were performed, using the open source statistical software 18 19 318 R (R Development Core Team) for one researcher (FN) and SAS (SAS Institute Inc) for the 20 21 319 senior statistician (CS). In addition, when authors shared their codes (in R, SAS, STATA 22 320 (StataCorp. 2017) or other), these were checked and used. Estimates of effect sizes, 95% 23 24 321 confidence intervals and p-values were obtained for each re-analysis. 25 26 322 27 323 Changes from the initial protocol 28 29 324 30 31 325 Initially we planned to include all studies published after the data sharing policies were in place. 32 33 326 Nevertheless, some authors that we contacted suggested that their studies were not eligible 34 327 because the papers were submitted to the journal before the policy. We contacted editors who 35 36 328 confirmed that policies applied for papers submitted (and not published) after the policy was 37 38 329 adopted. In accordance, and to avoid any underestimation of data sharing rates, we changed our 39 330 selection criteria to those presented above. For a few additional studies submitted before the 40 41 331 policy, data were collected and re-analyzed but only described in the web-appendix. 42 43 332 44 45 333 Concerning the re-analysis, we initially planned to consider non-reproducibility as a 46 334 disagreement between re-analysed results and results reported in the original publication by more 47 48 335 than 2% in the point estimate or 95% confidence interval. After reanalysis of a couple of studies 49 50 336 we felt that such a definition was sometimes meaningless. Interpreting a RCT involves clinical 51 52 337 expertise, and cannot be reduced to solely quantitative factors. Accordingly, we changed our 53 338 definition as stated above and provided a detailed description of re-analyses and published 54 55 339 results, mentioning the nature of effect size and the type of outcome considered. 56 57 58 59 13 60 https://mc.manuscriptcentral.com/bmj BMJ Page 14 of 63

1 2 3 340 4 5 341 Patient involvement 6 7 342 8 343 We had no established contacts with specific patient groups who might be involved in this 9 10 344 project. No patients were involved in setting the research question or the outcome measures, nor 11 Confidential: For Review Only 12 345 were they involved in the design and implementation of the study. There are no plans to involve 13 14 346 patients in the dissemination of results, nor will we disseminate results directly to patients. 15 347 16 17 348 RESULTS 18 19 349 20 21 350 Characteristics of included studies 22 351 23 24 352 A flow-chart detailing the study selection process is provided in Figure 1. The searches 25 26 353 performed on 2016-11-12 resulted in 159 citations. Of these, 134 full-texts were considered for 27 354 eligibility. Thirty-seven RCTs (21 from The BMJ and 16 from PLOS Medicine) published 28 29 355 between 2013 and 2016 met our eligibility criteria. Characteristics of these studies are presented 30 31 356 in Table 1. These RCTs had a median sample size of 432 participants (interquartile range = 213 32 33 357 to 1070), had no industry funding in 26 cases (70 %) and were led by teams from Europe in 25 34 358 cases (67 %). Twenty RCTs (54 %) evaluated pharmacological interventions, 9 (24 %) complex 35 36 359 interventions (e.g. psychotherapeutic program) and 8 (22 %) devices. 37 38 360 39 40 361 Data availability 41 362 42 43 363 We were able to access data for 19/37 studies (51%). Among these 19 studies, median time for 44 45 364 collecting the data was 4 days (range= [0 – 191]). Two of these studies, however, did not provide 46 365 sufficient information within the dataset to enable direct re-analysis (e.g., had unclear labels), 47 48 366 therefore 17 studies satisfied our definition of data availability. Data availability rate was thus 49 50 367 46% (95% CI [30% to 62%]). 51 52 368 Data was in principle available for two additional studies, not included in the count above and 53 369 both authored by the same research team. However, their authors asked us to cover the financial 54 55 370 costs of preparing the data for sharing (£607). Since other teams shared the data for free, we 56 57 58 59 14 60 https://mc.manuscriptcentral.com/bmj Page 15 of 63 BMJ

1 2 3 371 considered that it would not have been fair to pay some and not others for similar work in the 4 5 372 context of our project and we classified these two studies as not sharing data. For a third study, 6 7 373 the authors were in correspondence with us and discussing conditions for sharing data, but we 8 374 did not receive their data by the time our data collection process was determined to be over (7 9 10 375 months). If these three studies were included, the proportion of data sharing would be 54% (95% 11 Confidential: For Review Only 12 376 CI [38% to 70%]). 13 14 15 377 For the remaining 15 studies classified as not sharing data, reasons for non-availability were: no 16 378 answer to the different e-mails (n=7); no answer after an initial agreement (n=2); and refusal to 17 18 379 share data (n=6). Explanations for refusal to share data included: lack of endorsement of the 19 20 380 objectives of our study (n=1); personal reason (e.g. sick leave, n=2); restrictions due to an 21 22 381 embargo on data sharing (n=1); no specific reason offered (n=2). The existence of possible 23 382 privacy concerns was never put forward as a reason for not sharing data. 24 25 383 26 27 384 Among the 19 studies sharing some data (analysable datasets and non-analysable datasets), 16 28 385 (84 %) datasets were totally de-identified. Birthdates were found in 3 datasets and geographical 29 30 386 information (country and postcode) in 1 of these 3. Most datasets were data ready for analysis 31 32 387 (n= 17), whereas 2 required some additional processing before the analysis could be repeated. In 33 34 388 these two cases, such processing was difficult to implement (even with the code being available) 35 389 and authors were then contacted to share analysable data. Statistical analysis code was available 36 37 390 for 7 studies (including 2 that were obtained after a second specific request). 38 39 391 40 41 392 Reproducibility 42 393 43 44 394 Among the 17 studies providing sufficient data for re-analysis of their primary outcome(s) 14 45 46 395 (82%, 95% CI [59% to 94%]) studies were fully reproduced on all their primary outcomes. One 47 396 of the 17 studies did not provide enough information in the Methods section to reproduce the 48 49 397 analyses (specifically, the methods used for adjustment were unclear). We contacted the authors 50 51 398 of this study to obtain clarifications but received no reply. Of the remaining 16 studies, we re- 52 53 399 analyzed 47 different primary analyses. Two of these studies were considered not fully 54 400 reproduced. For one study, we identified an error in the statistical code and for the second one 55 56 401 we found slightly different numerical values for the effect sizes measured as well as slight 57 58 59 15 60 https://mc.manuscriptcentral.com/bmj BMJ Page 16 of 63

1 2 3 402 differences in terms of numbers of patients included in the analyses (a difference of 1 patient in 4 5 403 one of 4 analyses). Nevertheless, similar conclusions were reached in both cases (these two 6 7 404 studies were categorized as not fully reproduced but reaching the same conclusion). Therefore, 8 405 we found no results contradicting the initial publication, neither in terms of magnitude of the 9 10 406 effect (Table 2) nor in terms of statistical significance of the finding (Figure 2). 11 Confidential: For Review Only 12 407 We retrieved the data of 3 additional studies, published in The BMJ after its data sharing policy 13 14 408 was in place (but submitted before the policy). Although these studies were ineligible for our 15 409 main analysis, re-analyses were performed and also reached the same conclusions as the initial 16 17 410 study (e-Table 1). 18 19 411 20 21 412 Difficulties encountered in getting data and/or code, using them and performing the re- 22 413 analyses 23 24 414 25 26 415 Based on our correspondence with authors, we identified several difficulties in getting the data. 27 416 A frequent concern pertained to the costs involved in the data-sharing process, for example, the 28 29 417 costs of preparing the data or translating the database from one language to another. Some 30 31 418 authors wondered whether their team or our team should assume these costs. In addition, some of 32 33 419 the authors balanced these additional costs with their perceived benefits of sharing data for the 34 420 purpose of this study and seemed to rather value data sharing for the purpose of a meta-analysis 35 36 421 than for re-analyses, possibly because of the risk acknowledged by one investigator we contacted 37 38 422 about “naming and shaming” individual studies/investigators. 39 423 Getting prepared and preplanning for data sharing seems still a challenge for many trial groups; 40 41 424 data sharing proved to be novel for some authors who were unsure how to proceed. Indeed, there 42 43 425 was considerable heterogeneity between different procedures to share data: provided in an open 44 45 426 repository (n=5), downloadable on a secured website (n=1) after registration, included as 46 427 appendix of the published paper (n=3), or sent by e-mail (n=10). In 3 occasions, we signed a 47 48 428 data-sharing request/agreement. In these agreements, the sponsor and recipient parties specified 49 50 429 the terms that bound them in the data sharing process (e.g. concerning the use of data, 51 52 430 intellectual property, etc). In addition, typically there was no standard in what type of data were 53 431 shared (at what level of cleaning and processing). In one case, authors mentioned explicitly that 54 55 432 they followed standardized guidelines16 to prepare the dataset. 56 57 58 59 16 60 https://mc.manuscriptcentral.com/bmj Page 17 of 63 BMJ

1 2 3 433 Some analyses were complex and it was sometimes challenging to re-analyze data from very 4 5 434 specific designs or when unusual measures were used (e.g. relative change in percentage). 6 7 435 Obtaining more information about the analysis by contacting authors was necessary for 6 of 17 8 436 studies to replicate the findings. In one case, specific exploration of the code revealed that, in a 9 10 437 survival analysis, authors treated repeated events in the same patient (multiple events) as distinct 11 Confidential: For Review Only 12 438 observations. This was in disagreement with the methods section describing usual survival 13 14 439 analysis (taking into account the only first event for each patient). However, alternative analyses 15 440 did not contradict the published results. 16 17 441 Three databases did not provide sufficient information to reproduce the analyses. Missing data 18 19 442 concerned variables used for adjustment, definition of the analysis population, and 20 21 443 randomization groups. Communication with authors was therefore necessary and was fruitful in 22 444 one these 3 cases. 23 24 445 25 26 446 DISCUSSION 27 447 28 29 448 Statement of principal findings 30 31 449 32 33 450 In two prominent medical journals with a strict data-sharing policy for RCTs, we found that for 34 451 46% (95% CI [30% to 62%]) of published articles the original investigators shared their data 35 36 452 with sufficient information to enable re-analyses. This rate was under the 80% boundary that we 37 38 453 pre-specified as an acceptable threshold for papers submitted under a policy that makes data 39 454 sharing an explicit condition for publication. However, despite being lower than might be 40 41 455 expected, a 46% data sharing rate is much higher than the average rate of biomedical literature at 42 17 43 456 large, in which data sharing is almost non-existent (with few exceptions in some specific fields 44 18 45 457 such as genetics) . Moreover, our analyses focused on publications that were submitted right 46 458 after the implementation of new data-sharing policies, which might be expected to have practical 47 48 459 and cultural barriers to their full implementation. Indeed, our correspondence with the authors 49 50 460 helped identify several practical difficulties connected to data sharing, including difficulties in 51 52 461 contacting corresponding authors, and lack of time and financial resources on their behalf in 53 462 preparing the datasets for us. In addition, we found a wide variety of data sharing practices 54 55 463 between study groups (i.e. regarding the type of data that can be shared and the procedures that 56 57 58 59 17 60 https://mc.manuscriptcentral.com/bmj BMJ Page 18 of 63

1 2 3 464 are necessary to follow in order to get the data). Data sharing practices could evolve in the future 4 5 465 to address these barriers to data sharing (Table 3). 6 7 8 466 For all results that we were able to re-analyze, we reached similar conclusions (despite 9 467 occasional slight differences in the numerical estimations) to those reported in the original 10 Confidential: For Review Only 11 468 publication and this result is reassuring that at least the available data shared do correspond 12 13 469 closely to the reported results. Of course, there is a large amount of diversity on what exactly 14 15 470 “raw data” mean and they can involve various transformations (from the case report forms to 16 471 coded and analyzable data).13 Here, we relied on late-stage, coded, and cleaned data and 17 18 472 therefore, the potential for leading to a different conclusion was probably small. Data processing, 19 20 473 coding, cleaning and re-categorization of events can have substantial impact on the results in 21 22 474 some trials. For example, SmithKline Beecham’s Study 329 was a well-known study on 23 475 paroxetine in adolescent depression presenting the drug as safe and effective19 while a re- 24 25 476 analysis starting from the case report forms demonstrated a lack of efficacy and some serious 26 20 27 477 safety issues. 28 478 29 30 479 Strengths and weaknesses of the study 31 32 480 33 34 481 Some leading general medical journals, New England Journal of Medicine, Lancet, JAMA and 35 482 JAMA Internal Medicine, have had no specific policy for data sharing in RCTs until recently. 36 37 483 Annals of Internal Medicine has encouraged (but not demanded) data-sharing since 2007.21 BMC 38 39 484 Medicine adopted a similar policy in 2015. The BMJ and PLOS Medicine have adopted stronger 40 485 policies, beyond the ICMJE policy, that mandate data sharing for RCTs. Our survey of RCTs 41 42 486 published in these two journals might therefore give a taste of the impact and caveats of such full 43 44 487 policies. However, one should be careful and not generalize these results to other journals. First, 45 46 488 we have a very selected sample of studies. Our sample includes studies (mostly from Europe) 47 489 that are larger and less likely to be funded by the industry than the average published RCT in the 48 49 490 medical literature.22 Most of these re-analyzed RCTs were large, and sometimes very large. 50 51 491 Several, especially those published in PLOS Medicine, were cluster randomized studies, and 52 53 492 many explored infectious disease or important public health issues, characteristics that are not 54 493 very common in RCTs overall. Similarly, few of the evaluated studies were sponsored by 55 56 494 industry. Some public funders (or charities) involved in their funding have already open access 57 58 59 18 60 https://mc.manuscriptcentral.com/bmj Page 19 of 63 BMJ

1 2 3 495 policies such as the Bill and Melinda Gates foundation or the National Institute for Health 4 5 496 Research in the United Kingdom. 6 7 497 These reasons also explain why we did not compare journals. Qualitatively they don’t publish 8 498 the same kind of RCTs and quantitatively, The BMJ and PLOS Medicine publish few RCTs in 9 10 499 comparison to other leading journals, such as the New England Journal of Medicine and Lancet. 11 Confidential: For Review Only 12 500 Any comparisons might have been subject to confounding. Similarly, we believed that pre-post 13 14 501 studies at The BMJ and PLOS Medicine might have been subject to historical bias and 15 502 confounding factors since such policies might have changed the profile of submitting authors: 16 17 503 researchers with a specific interest in open science and reproducibility might have been attracted 18 19 504 and others might have opted for another journal. We think comparative studies will be easier to 20 21 505 conduct when all journals adopt data sharing standards. Even with the current proposed ICJME 22 506 requirements, The BMJ and PLOS Medicine will be journals with stronger policies. In addition, 23 24 507 The BMJ and PLOS Medicine are both major journals with many resources such as in-depth 25 26 508 discussion of papers in editorial meetings, and statistical peer review. Our findings concerning 27 509 reproducibility might not apply to smaller journals with more limited resources. We cannot 28 29 510 generalize this finding to studies that did not share their data: authors who are confident in their 30 31 511 results might more readily agree to share their data and this may lead to overestimation of 32 33 512 reproducibility. In addition, we might have missed a few references using the filter "Randomized 34 513 Controlled Trial[ptyp]", however, this is unlikely to have affected data sharing and 35 36 514 reproducibility rates. 37 38 515 Finally, in our study, the notion of data sharing was restricted to a request by a researcher group 39 516 while, in theory, other types of requestors (patients, clinicians, health and academic institutions, 40 41 517 etc.) may also be interested in the data. Moreover, we followed the strict procedure presented in 42 43 518 the paper and without sending any correspondence (e.g. rapid response) on the journals’ website. 44 45 519 In addition, our re-analyses were based on primary outcomes whereas secondary and safety 46 520 outcomes may be more problematic in their reproduction. These points need to be explored in 47 48 521 further studies. 49 50 522 51 52 523 Strengths and weaknesses in relation to other studies, discussing important differences in 53 524 results 54 55 525 56 57 58 59 19 60 https://mc.manuscriptcentral.com/bmj BMJ Page 20 of 63

1 2 3 526 In a precedent survey of 160 randomly sampled BMJ research articles from 2009 to 2015, 4 5 527 excluding meta-analysis and systematic reviews,23 the authors found that only 5% shared their 6 7 528 data sets. Nevertheless, this survey assessed data sharing among all studies with original raw data 8 529 while The BMJ data sharing policy specifically applied to clinical trial data. When considering 9 10 530 clinical trials bound by The BMJ data sharing policy (n=21), the percent shared was 24% (8%- 11 Confidential: For Review Only 12 531 47%). Our study identified higher rates of shared datasets in accordance with an increase in the 13 14 532 rate of ‘data shared’ for every additional year between 2009 and 2015 found in the previous 15 533 survey. It is not known whether it results directly from the policy implementation or of a slow 16 17 534 and positive cultural swift of trialists. 18 19 535 20 21 536 Lack of response from authors was also identified as a caveat in the previous evaluation, which 22 537 suggested that the wording of The BMJ policy such as availability on ‘reasonable request’ might 23 24 538 be interpreted in different ways.23 Previous research across PLOS also suggests that data requests 25 24 26 539 by contacting authors might be sometimes ineffective. Despite writing a data sharing 27 540 agreement in their paper, corresponding authors are still free to decline data requests. Our study 28 29 541 might be perceived by some authors as imposing unreasonable requests on them. For example, 30 31 542 one may question whether sharing data for the purposes of this project constitutes a reasonable 32 33 543 request. One might consider that this meta-research project exploring the effectiveness of data 34 544 sharing policies lies outside the purpose for which the initial trial was done and for which the 35 36 545 participants gave consent. A survey found that authors are generally less willing to share their 37 25 38 546 data for the purpose of re-analyses than, for instance, for individual patient data meta-analyses. 39 547 Nevertheless, reproducibility checks, based on independent re-analysis are perfectly aligned with 40 41 548 the primary objective of clinical trials and indeed with patient interests. Conclusions that persist 42 43 549 through substantial re-analyses are becoming more credible. In order to explain and support the 44 45 550 interest of our study, we have registered a protocol and have transparently described our 46 551 intentions in our e-mail, but data sharing rates were still lower than we expected. Active auditing 47 48 552 of data sharing policies by journal editors may facilitate data sharing implementation. 49 50 553 51 52 554 Concerning reproducibility of clinical trials, an empirical analysis suggests that only a small 53 555 number of re-analyses of RCTs have been published to date; of these, only a minority was 54 55 556 conducted by entirely independent authors. In a previous empirical evaluation of published re- 56 57 58 59 20 60 https://mc.manuscriptcentral.com/bmj Page 21 of 63 BMJ

1 2 3 557 analyses, thirty-five percent (13/37) of published re-analyses yielded changes in findings that 4 5 558 implied conclusions different from those of the original article as to whether patients should be 6 26 7 559 treated or not or about which patients should be treated. In our assessment, we found no 8 560 differences in conclusions pertaining to treatment decisions. The difference between the previous 9 10 561 empirical evaluation and the current one is probably due to many factors. Published re-analyses 11 Confidential: For Review Only 12 562 in the past would be unlikely to be published if they had found the very same results and had 13 14 563 reached the very same conclusions as the original analysis. Therefore, the set of published re- 15 564 analyses is enriched with discrepant results and conclusions. Moreover, past published re- 16 17 565 analyses addressed the same question on the same data but using typically different analytical 18 19 566 methods. Conversely, we used the very same analysis employed by the original paper. In 20 21 567 addition, the good computational reproducibility (replication of analyses) we found is only one 22 568 aspect of reproducibility27 and should not be over interpreted. For example, in psychology, 23 24 569 numerous labs have volunteered to re-run experiments (not solely analyses), with the methods 25 28 26 570 used by the original researchers. 39/100 studies were considered as successfully replicated. But 27 571 attempts to replicate psychological studies are often easier to implement than attempts to 28 29 572 replicate RCTs that are often costly and difficult to run. This is especially true for large RCTs. 30 31 573 32 33 574 Meaning of the study: possible explanations and implications for clinicians and 34 575 policymakers 35 36 576 37 38 577 The ICJME’s requirements adopted in 2017 mandate that a data sharing plan will have to be 39 578 included in each paper (and pre-specified in study registration).8 Our results suggest that this not 40 41 579 likely to be sufficient to achieve high rates of data sharing. One can imagine that individual 42 43 580 authors will agree to write such a statement in line with the promise of publication, but the time 44 45 581 and costs involved into such data preparation might lead them to be reluctant to answer data 46 582 sharing requests. Interestingly the ICJME also mandates that clinical trials that begin enrolling 47 48 583 participants on or after January 1, 2019, must include a data sharing plan in the trial’s 49 50 584 registration. An a priori data sharing plan might push more authors to pre-emptively address and 51 52 585 find funding for sharing, but its eventual impact on actual sharing is unknown. Funders are well 53 586 positioned to facilitate data sharing. Some such as the Wellcome Trust already allow 54 55 587 investigators to use funds towards open access article processing charges which is typically a 56 57 58 59 21 60 https://mc.manuscriptcentral.com/bmj BMJ Page 22 of 63

1 2 3 588 very small fraction of awarded funding. Funders could extend their funding policy and also allow 4 5 589 investigators to use a similar small fraction of funding towards enabling data sharing. In addition, 6 5 7 590 patients can help promote a culture of data sharing for clinical trials. 8 9 591 In addition, reproducible research does not solely imply sharing data, but also reporting all steps 10 Confidential: For Review Only 11 592 of the statistical analysis. This is one core principle of the CONSORT statement “Describe 12 13 593 statistical methods with enough detail to enable a knowledgeable reader with access to the 14 29 15 594 original data to verify the reported results”. It is nonetheless sometimes difficult to provide 16 595 such detailed information in a restricted number of lines (i.e. a paper). We suggest that details of 17 18 596 the statistical analysis plan have to be provided as well as the detailed labels used in the table and 19 30 20 597 efficient analytical code sharing is also essential. 21 22 23 598 We suggest that if journals ask for data and code to be shared, they should ensure that this 24 599 material is also reviewed by editorial staff (and/or peer reviewers) or at a minimum checked for 25 26 600 completeness and basic usability.30 This could also be positively translated in specific incentives 27 15 28 601 for the paper; e.g. many psychology journals use badges as signs of good research practices 29 602 (including data sharing) adopted by papers. And, beyond incentivizing authors, journals adopting 30 31 603 such practices could also be incentivized by a gain in reputation and credibility. 32 33 34 604 Though ensuring patient privacy and lack of explicit consent for sharing are often cited as major 35 31 36 605 barriers to sharing RCT data (and generally accepted as valid exemptions), none of the 37 606 investigators we approached mentioned this reason. This could suggest that technical constraints, 38 39 607 lack of incentives and dedicated funding, and a general diffidence towards re-analyses might be 40 41 608 more germane obstacles to be addressed. 42 43 609 Finally, data sharing practices were very different from one team to another. There is thus a need 44 45 610 for standardization and for drafting specific guidelines for best practices in data sharing. 46 47 611 48 49 612 Unanswered questions and future research 50 613 51 52 614 We recommend some prospective monitoring of data sharing practices to ensure that the new 53 54 615 ICJME’s requirements are effective and useful. To this end, one also should keep in mind that 55 56 616 data-availability is a surrogate of the expected benefit of having open data. Proving that data- 57 58 59 22 60 https://mc.manuscriptcentral.com/bmj Page 23 of 63 BMJ

1 2 3 617 sharing rates impacts re-use32 of the data is a further step. Proving that this reuse might translate 4 5 618 into discovery that can change care without generating false positive findings (e.g. in series of 6 7 619 unreliable a posteriori subgroup analyses) is even more challenging. 8 9 620 10 Confidential: For Review Only 11 621 REFERENCES 12 13 622 14 15 623 1. Taichman DB, Backus J, Baethge C, et al. Sharing Clinical Trial Data: A Proposal from the 16 624 International Committee of Medical Journal Editors. PLoS Med 2016;13(1):e1001950. doi: 17 625 10.1371/journal.pmed.1001950 [doi] 18 626 PMEDICINE-D-15-03676 [pii] [published Online First: 2016/01/21] 19 627 2. Platt R, Ramsberg J. Challenges for Sharing Data from Embedded Research. N Engl J Med 20 628 2016;374(19):1897. doi: 10.1056/NEJMc1602016 [doi] [published Online First: 2016/04/21] 21 22 629 3. Kalager M, Adami HO, Bretthauer M. Recognizing Data Generation. N Engl J Med 23 630 2016;374(19):1898. doi: 10.1056/NEJMc1603789 [doi] [published Online First: 2016/04/21] 24 631 4. Devereaux PJ, Guyatt G, Gerstein H, et al. Toward Fairness in Data Sharing. N Engl J Med 25 632 2016;375(5):405-7. doi: 10.1056/NEJMp1605654 [doi] [published Online First: 2016/08/16] 26 633 5. Haug CJ. Whose Data Are They Anyway? Can a Patient Perspective Advance the Data- 27 634 Sharing Debate? N Engl J Med 2017;376(23):2203-05. doi: 10.1056/NEJMp1704485 [published 28 635 Online First: 2017/04/27] 29 30 636 6. Rockhold F, Nisen P, Freeman A. Data Sharing at a Crossroads. N Engl J Med 31 637 2016;375(12):1115-7. doi: 10.1056/NEJMp1608086 [doi] [published Online First: 2016/09/23] 32 638 7. Rosenbaum L. Bridging the Data-Sharing Divide - Seeing the Devil in the Details, Not the 33 639 Other Camp. N Engl J Med 2017;376(23):2201-03. doi: 10.1056/NEJMp1704482 [doi] 34 640 [published Online First: 2017/04/27] 35 641 8. Taichman DB, Sahni P, Pinborg A, et al. Data sharing statements for clinical trials. BMJ 36 642 2017;357:j2372. [published Online First: 2017/06/07] 37 38 643 9. Godlee F, Groves T. The new BMJ policy on sharing data from drug and device trials. BMJ 39 644 2012;345:e7888. [published Online First: 2012/11/22] 40 645 10. Loder E, Groves T. The BMJ requires data sharing on request for all trials. BMJ 41 646 2015;350:h2373. [published Online First: 2015/05/09] 42 647 11. Bloom T, Ganley E, Winker M. Data access for the open access literature: PLOS's data 43 648 policy. PLoS Biology 2014;12(2):e1001797. 44 45 649 12. Mello MM, Francer JK, Wilenzick M, et al. Preparing for responsible sharing of clinical trial 46 650 data. N Engl J Med 2013;369(17):1651-8. doi: 10.1056/NEJMhle1309073 [doi] [published 47 651 Online First: 2013/10/23] 48 652 13. Zarin DA, Tse T. Sharing Individual Participant Data (IPD) within the Context of the Trial 49 653 Reporting System (TRS). PLoS Med 2016;13(1):e1001946. doi: 10.1371/journal.pmed.1001946 50 654 [doi] 51 655 PMEDICINE-D-15-03309 [pii] [published Online First: 2016/01/20] 52 53 656 14. Newcombe RG. Two-sided confidence intervals for the single proportion: comparison of 54 657 seven methods. Stat Med 1998;17(8):857-72. doi: 10.1002/(SICI)1097- 55 658 0258(19980430)17:8<857::AID-SIM777>3.0.CO;2-E [pii] [published Online First: 1998/05/22] 56 57 58 59 23 60 https://mc.manuscriptcentral.com/bmj BMJ Page 24 of 63

1 2 3 659 15. Kidwell MC, Lazarevic LB, Baranski E, et al. Badges to Acknowledge Open Practices: A 4 660 Simple, Low-Cost, Effective Method for Increasing Transparency. PLoS Biol 5 6 661 2016;14(5):e1002456. doi: 10.1371/journal.pbio.1002456 [doi] 7 662 PBIOLOGY-D-16-00142 [pii] [published Online First: 2016/05/14] 8 663 16. Hrynaszkiewicz I, Norton ML, Vickers AJ, et al. Preparing raw clinical data for publication: 9 664 guidance for journal editors, authors, and peer reviewers. BMJ 2010;340:c181. [published Online 10 665 First: 2010/01/30]Confidential: For Review Only 11 666 17. Iqbal SA, Wallach JD, Khoury MJ, et al. Reproducible Research Practices and Transparency 12 667 across the Biomedical Literature. PLoS Biol 2016;14(1):e1002333. doi: 13 14 668 10.1371/journal.pbio.1002333 [doi] 15 669 PBIOLOGY-D-15-02900 [pii] [published Online First: 2016/01/05] 16 670 18. Mailman MD, Feolo M, Jin Y, et al. The NCBI dbGaP database of genotypes and 17 671 phenotypes. Nature genetics 2007;39(10):1181-6. doi: 10.1038/ng1007-1181 [published Online 18 672 First: 2007/09/28] 19 673 19. Keller MB, Ryan ND, Strober M, et al. Efficacy of paroxetine in the treatment of adolescent 20 21 674 major depression: a randomized, controlled trial. J Am Acad Child Adolesc Psychiatry 22 675 2001;40(7):762-72. doi: S0890-8567(09)60309-9 [pii] 23 676 10.1097/00004583-200107000-00010 [doi] [published Online First: 2001/07/05] 24 677 20. Le Noury J, Nardo JM, Healy D, et al. Restoring Study 329: efficacy and harms of paroxetine 25 678 and imipramine in treatment of major depression in adolescence. BMJ 2015;351:h4320. 26 679 [published Online First: 2015/09/18] 27 680 21. Laine C, Goodman SN, Griswold ME, et al. Reproducible research: moving toward research 28 29 681 the public can really trust. Ann Intern Med 2007;146(6):450-3. doi: 0000605-200703200-00154 30 682 [pii] [published Online First: 2007/03/07] 31 683 22. Chan AW, Altman DG. Epidemiology and reporting of randomised trials published in 32 684 PubMed journals. Lancet (London, England) 2005;365(9465):1159-62. doi: 10.1016/s0140- 33 685 6736(05)71879-1 [published Online First: 2005/03/30] 34 686 23. Rowhani-Farid A, Barnett AG. Has open data arrived at the British Medical Journal (BMJ)? 35 687 An observational study. BMJ Open 2016;6(10):e011784. doi: bmjopen-2016-011784 [pii] 36 37 688 10.1136/bmjopen-2016-011784 [doi] [published Online First: 2016/10/16] 38 689 24. Savage CJ, Vickers AJ. Empirical study of data sharing by authors publishing in PLoS 39 690 journals. PLoS One 2009;4(9):e7078. doi: 10.1371/journal.pone.0007078 [doi] [published Online 40 691 First: 2009/09/19] 41 692 25. Early Experiences With Journal Data Sharing Policies: A Survey of Published Clinical Trial 42 693 Investigators. International Congress on Peer Review and Scientific Publication; 2017; Chicago. 43 44 694 26. Ebrahim S, Sohani ZN, Montoya L, et al. Reanalyses of randomized clinical trial data. JAMA 45 695 2014;312(10):1024-32. doi: 1902230 [pii] 46 696 10.1001/jama.2014.9646 [doi] [published Online First: 2014/09/10] 47 697 27. Goodman SN, Fanelli D, Ioannidis JP. What does research reproducibility mean? Sci Transl 48 698 Med 2016;8(341):341ps12. doi: 8/341/341ps12 [pii] 49 699 10.1126/scitranslmed.aaf5027 [doi] [published Online First: 2016/06/03] 50 700 28. PSYCHOLOGY. Estimating the reproducibility of psychological science. Science 51 52 701 2015;349(6251):aac4716. doi: 349/6251/aac4716 [pii] 53 702 10.1126/science.aac4716 [doi] [published Online First: 2015/09/01] 54 55 56 57 58 59 24 60 https://mc.manuscriptcentral.com/bmj Page 25 of 63 BMJ

1 2 3 703 29. Schulz KF, Altman DG, Moher D. CONSORT 2010 statement: updated guidelines for 4 704 reporting parallel group randomised trials. BMJ 2010;340:c332. [published Online First: 5 6 705 2010/03/25] 7 706 30. Stodden V, McNutt M, Bailey DH, et al. Enhancing reproducibility for computational 8 707 methods. Science 2016;354(6317):1240-41. doi: 10.1126/science.aah6168 [published Online 9 708 First: 2016/12/13] 10 709 31. LewandowskyConfidential: S, Bishop D. Research integrity:For Don'tReview let transparency Only damage science. 11 710 Nature 2016;529(7587):459-61. doi: 529459a [pii] 12 711 10.1038/529459a [doi] [published Online First: 2016/01/29] 13 14 712 32. Navar AM, Pencina MJ, Rymer JA, et al. Use of Open Access Platforms for Clinical Trial 15 713 Data. JAMA 2016;315(12):1283-4. doi: 2504799 [pii] 16 714 10.1001/jama.2016.2374 [doi] [published Online First: 2016/03/24] 17 715 18 19 716 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 25 60 https://mc.manuscriptcentral.com/bmj BMJ Page 26 of 63

1 2 3 717 FIGURES 4 5 718 6 719 Figure 1: Study flow diagram 7 720 8 9 721 Figure 2: p-values in initial analyses and in re-analyses. The axes are on a log scale. Blue colour 10 722 indicates identical conclusion between the initial analysis and the re-analysis. 11 723 Confidential: For Review Only 12 724 13 14 725 15 726 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 26 60 https://mc.manuscriptcentral.com/bmj Page 27 of 63 BMJ

1 2 3 727 TABLES 4 5 6 All BMJ PLOS Medicine

7 (37 studies) (21 studies) (16 studies) 8 Geographical area of the lead country 9 Europe 25 (67 %) 17 (80 %) 8 (50 %) 10 11 AustraliaConfidential: and New Zealand For4 (11 %) Review1 (5 %) Only 3 (19 %) 12 Northern America 3 (8 %) 1 (5 %) 2 (12.5 %) 13 Africa 3 (8 %) 1 (5 %) 2 (12.5 %) 14 East Asia 1 (3 %) 0 (0 %) 1 (6 %) 15 Middle East 1 (3 %) 1 (5 %) 0 (0 %) 16 Type of intervention 17 Drug 20 (54 %) 13 (62 %) 7 (44 %) 18 Device 8 (22 %) 8 (38 %) 0 (0 %) 19 Complex intervention 9 (24 %) 0 (0 %) 9 (56 %) 20 Medical specialty 21 Infectious disease 12 (33 %) 4 (19 %) 8 (50 %) 22 Rheumatology 5 (14 %) 5 (24 %) 0 (0 %) 23 24 Endocrinology/nutrition 4 (11 %) 1 (5 %) 3 (19 %) 25 Paediatrics 3 (8 %) 2 (9 %) 1 (6 %) 26 Mental health / addiction 2 (5 %) 1 (5 %) 1 (6 %) 27 Obstetrics 2 (5 %) 1 (5 %) 1 (6 %) 28 Emergency medicine 2 (5 %) 2 (9 %) 0 (0 %) 29 Geriatrics 2 (5 %) 0 (0 %) 2 (13 %) 30 Other 5 (14 %) 5 (24 %) 0 (0 %) 31 Designs 32 Superiority (Head to head) 18 (49 %) 15 (71 %) 3 (19 %) 33 Superiority (Factorial) 1 (3 %) 1 (5 %) 0 (0 %) 34 Superiority (Clusters) 8 (21 %) 1 (5 %) 7 (43 %) 35 Non-inferiority + Superiority (Head to head) 4 (11 %) 1 (5 %) 3 (19 %) 36 Non-inferiority (Head to head) 6 (16 %) 3 (14 %) 3 (19 %) 37 38 Sample size 432 (213 – 1070)† 221 (159 – 494) 1047 (433 – 2248)† 39 Private sponsorship 40 No 26 (70 %) 15 (71 %) 11 (69 %) 41 Provided the device 1 (3 %) 1 (5 %) 0 (0 %) 42 Provided the intervention 1 (3 %) 0 (0 %) 1 (6 %) 43 Provided the drug 5 (13 %) 1 (5 %) 4 (25 %) 44 Provided the drug and some financial support 2 (5 %) 2 (9 %) 0 (0 %) 45 Provided partial financial support 1 (3 %) 1 (5 %) 0 (0 %) 46 Provided total financial support 1 (3 %) 1 (5 %) 0 (0 %) 47 Statement of availability 48 Ask to contact by e-mail 23 (62 %) 17 (81 %) 6 (38 %) 49 Explain how to retrieve the data (e.g. platform) 9 (24 %) 0 (0 %) 9 (56 %) 50 51 State ‘no additional data available‘ 2 (5 %) 2 (9 %) 0 (0 %) 52 Ask to contact by mail 1 (3 %) 0 (0 %) 1 (6 %) 53 Embargo 1 (3 %) 1 (5 %) 0 (0 %) 54 No statement 1 (3 %) 1 (5 %) 0 (0 %) 55 728 56 57 58 59 27 60 https://mc.manuscriptcentral.com/bmj BMJ Page 28 of 63

1 2 3 729 Table 1: Characteristics of the included studies. Numbers (and percentages) are presented (rounded 4 730 percentages add up to 100% for each variable). For sample size medians and interquartile ranges are 5 6 731 presented. † For one cluster trial in PLOS Medicine the exact sample size was not reported. 7 8 9 10 11 Confidential: For Review Only 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 28 60 https://mc.manuscriptcentral.com/bmj 29 p-value Re- analysis 0.26 0.45 0.095 0.047* 0.328 0.157* 0.239 0.339* 0.778 0.858* 0.170 0.253* 0.805 0.814* . . 0.001< 0.001< 0.777 0.777 0.58 0.58 0.13 p-value Published paper 0.25 0.43 0.075 0.024* 0.326 0.265* 0.219 0.216* 0.775 0.713* 0.199 0.187* 0.804 0.809* . . 0.001< 0.001< 0.759 0.777 5.19 [-3.62 5.19 to 14.00] Effectsizemeasure Re-analysis [-0.3 to 0.4 1.2] [-0.5 to 0.3 1.2] [-2.5 10.2 22.9] to [-5.3 to 4.3 13.9] [-3.3 to 3.6 0.1] [-7.6 to 1.0 9.7] [-2.1 to 4.0 10.2] [-8.4 to 1.0 10.5] [-1.66 -0.92 - to 0.18] [-1.63 -0.89 - to 0.15] [0.13 0.17 0.24] to [0.11 0.16 0.22] to [0.942 1.033 to 1.133] [0.921 1.011 to 1.109] 0.17 [0.07 0.17 0.46] to [0.06 0.17 0.49] to 0.001† < 0.001 < 3 [-10.32 3 to 6.73] [-1.73 1 1.98] to [-2.45 0.01 2.62] to [-1.99 -0.12 to 1.75] [-3.15 -0.49 to 2.18] 0.86 0.92 0.87 0.64 [1.52 2.27 3.39] to [1.52 2.27 3.40] to 0.001 < 0.001 < 1 [-8.69 1 6.30] to [-6.06 1.42 8.90] to 0.68 0.62 [-35 -58 -74] to [-36 -58 -72] to 0.001 < 0.001 < [1.52 2.27 3.38] to [1.51 2.27 3.39] to 0.001 < 0.001 < [513 1037 1560] to [513 1037 1560] to 0.001 < 0.001 < [1.52 1.75 2.03] to [1.52 1.75 2.03] to 0.001 < 0.001 < Effectsizemeasure Publishedpaper [-0.3 to 0.4 1.2] [-0.5 to 0.3 1.2] [-1.3 10.2 21.7] to [-5.1 to 4.2 13.8] [-2.6 to 3.6 9.7] [-7 1.0 to 9.1] [-2.4 to 3.8 10.0] [-8.3 to 1.0 10.4] [-1.67 -0.92 - to 0.16] [-1.62 -0.87 - to 0.13] [0.14 0.18 0.25] to [0.12 0.17 0.24] to [0.942 1.036 to 1.140] [0.919 1.012 to 1.114] HR, HR, 95%CI MD, 99%CI MD, 99%CI MD, 99%CI IRR, 95 %CI MD, 99%CI RC%, 95%CI IRR, 95 %CI MD, 95%CI RR, 95%CI Effect size (type) MD, 95%CI MD, 95%CI MD, 95%CI MD, 95%CI MD, 95%CI MD, 95%CI MD, 95%CI MD, 95%CI MD, 95%CI MD, 95%CI HR, 95%CI HR, 95%CI RaR, 95%CI RaR, 95%CI BMJ https://mc.manuscriptcentral.com/bmj Primary outcomePrimary measures analogue Visual scale, 2 time Disability Oswestry Index, time 1 Disability Oswestry Index, time 2 leg Average pain, time 1 leg Average pain, time 2 parasite nPCR prevalence, time 1, zone1 parasite nPCR prevalence, time 2, 1 zone parasite nPCR prevalence, time1, 2 zone parasite nPCR prevalence, time 22,zone parasite nPCR prevalence, time 1, 3 zone parasite nPCR prevalence, time 2, 3 zone in Change (unadjusted) weight in Change (adjusted) weight Malaria,pneumonia, diarrhea or (PP) mortality P.vivaxinfection by qPCR, 1 time P.vivaxinfection by qPCR, 2 time attackVertigo attackVertigo Visual analogue Visual scale, 1 time of Burden and hypoxia hyperoxia Malaria,pneumonia, diarrhea or (ITT) mortality step count Daily ofUse antiplatelet, statin, 2≥ BP lowering drugs

Confidential: For Review Only TranscranialDirect Current stimulation (TDCs) VSSHAM Chronic TDCs/ low back pain InfraRed Near Spectroscopy VScareusual / Preterm infants Cotrimoxazole prophylaxiscessationVS continuation/ HIV Nurse-delivered walkingintervention VS usual careadults / Older Fixed dosecombination VS treatment usual Patients care/ at highrisk of cardiovascular Treatmentcomparison / condition Epiduralsteroid injections VSgabapentin / Lumbosacral radicular pain Hotspot-targetedinterventions VS usual interventions Malaria / Low intensityintervention VS general health intervention / Obesity Blood-liver-stage plus or drugs blood-stage only Malariadrugs / Low dosebetahistine VSplacebo / Meniere’s disease High dosebetahistine VS Meniere’splacebo / disease

Study Study ID Luedtke,2015† Cohen, 2015 Hyttel-SorensenS, 2015 Bousema, 2016 Gysin-Maillart, 2016†† BrieftherapyVS careusual Suicide / attempt Lombard, 2016 attempt Suicide 2016 Polyak, Robinson, 2015 2015 Harris, Adrion, 2016 Selak, 2014 Page 29 of 63 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 Page 30 of 63 30

0.33 0.33 0.08 0.36 0.57 1 0.339 0.97 0.97 0.97 0.31 0.31 0.08 0.36 0.57 1 0.339 0.003 [−0.20 0.003 to 0.21] -0.33 [-1.01 -0.33 to 0.34] [0.96 1.38 1.99] to [0.82 1.24 1.89] to [0.65 1.27 2.45] to [0.47 1.05 2.32] to [0.9 1.0 1.2] to -1.2 [-3.2 -1.2 to 0.8] [-0.17 -0.05 to 0.08] [-3.2 -1.2 to 0.8] [-0.17 -0.05 to 0.08] 0.46 0.22 0.46 0.22 1.64 [1.32 1.64 2.05] to [1.32 1.65 2.05] to 0.001 < 0.001 < [−0.20 0.003 to 0.21] -2.2 [-5.6 -2.2 to 1.2] [-5.6 -2.2 to 1.2] 0.21 0.21 -0.33 [-0.96 -0.33 to 0.31] [0.96 1.38 1.99] to [0.82 1.24 1.89] to [0.65 1.27 2.45] to [0.47 1.05 2.32] to [0.9 1.0 1.2] to MD, 95%CI MD, 95%CI RR, 95%CI MD, 95%CI MD, 95%CI MD, 95%CI RR, 95%CI RR, 95%CI RR, 95%CI RR, 95%CI OR, 95%CI BMJ https://mc.manuscriptcentral.com/bmj Change in Change diastolic blood pressure in Change LDL-C elbowin Change intensity pain failure Treatment (ITT) failure Treatment (PP) cause All (ITT) mortality cause All (PP) mortality Neonatal mortality Bloodloss 500> ml of Speed six healingover weeks (cm2/day) Change in systolicin Change bloodpressure

Confidential: For Review Only Sublingual Sublingual misoprostolVS intramuscular Prevention oxytocin/ of postpartum hemorrhage VSCiclosporin prednisolonePyoderma / gangrenosum disease indisease careprimary Transcutaneouselectrical nervestimulation VS careusual / Tennis elbow Trimethoprim-sulfamethoxazoleVS vancomycin Severe / infectionscaused bymeticillin Staphylococcus resistant aureus Home-BasedCounselling Strategy VS usual care Neonatal / Care † In Luedke, 2015, we judged the study as not fully reproduced but reaching the same conclusion because of differences in numerical estimates estimates numerical in differences of because conclusion same the reaching but reproduced fully not as the study judged we 2015, Luedke, † In our re-analysis]). in more [1 patient analysed of patient the number in a difference found we analyses of these 1 (for take not did analysis initial the because conclusion same reaching but reproduced fully not as the study we judged 2016, Gysin-Maillart, †† In estimates. in differences major without re-analysis our in corrected was This the data. of nature the repeated into account

Table 2: Results of the re-analyses re-analyses the of Results 2: Table Intention ITT: ratio; Odds OR: Risk; Relative RR: Ratio; Rate RaR: Ratio; HR: Hazard Percentage; in Change Relative RC%: Differences; Mean MD: Protocol. Per PP: Treat; To analyses. non-adjusted of complement in computed were analyses adjusted of p-values 2016, Bousema, * In Chesterton, 2013 Atukunda, 2014 2015 Paul, Thomas, 2015 2015Hanson, 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 31 identified before before identified -

data deposition when it is is it when deposition data

should systematically address data data address systematically should BMJ should provide guidance on the on the guidance provide should should favor favor should

should help to develop awareness of a a of awareness develop to help should

should develop comprehensive educational educational comprehensive develop should should develop comprehensive educational educational comprehensive develop should educational comprehensive develop should should adopt some clear and homogeneous rules rules homogeneous and clear some adopt should

and editors and de are databases that ensure must should pre-emptively address and find funding for for funding find and address pre-emptively should labels detailed the concerning details provide should analysis statistical detailed their share also should

should allow investigators to use funds towards data data towards funds use to investigators allow should should adopt more binding policies than the current current the than policies binding more adopt should who ask for data and code to be shared should ensure ensure should toshared be code and data for ask who

https://mc.manuscriptcentral.com/bmj Suggested solution for various stakeholders various for solution Suggested clinicians and Patients stakeholders All Editors Researchers Researchers groups trials Clinical groups trials Clinical common ownership of the data, intended as common common as intended data, the of ownership common to resources necessary the all inproviding also responsibility use. ethical and effective their for sharable data the make Researchers sharing. data Funders sharing. in data practices best for guidelines) (e.g. groups trials Clinical sharing data about outreach boards review Institutional issues. sharing requirement. ICJME possible. ethically sharing. board review Institutional study. individual each for de-identification of level requested sharing data about outreach Researchers code. analytical and the table the in used Editors isreviewed. material this that sharing data about outreach Researchers plan. plan. sharing. sharing. activities sharing data reward should institutions Academic tenure. and promotion through ,

Confidential: For Review Only

Identified issues Identified the to burden and responsibility the leaves policies sharing Data in still is sharing data for preplanning and prepared Getting of discretion exclusive the to leaves request upon sharing Data corresponding contacting in difficulties some be could There in befound can birth of as date such information identifying Some complete, (e.g. sharable effectively be to need databases Shared data sharing to limited not are practices research Reproducible money, (time, resources necessary the all obtain to researchers, andlegal ethical andservices, tools organizational and technical etc.). compliance, heterogeneity is considerable There units. trials in progress data of types the and data share to procedures different between shared. are that own their share to whether about decision the researchers the under and objectives which for groups, research other to data conditions. and terms which request. upon sharing data limiting authors databases. some labels, descriptive (e.g. meta-data including homogeneous), analysis). of methods tools, processing pre-analysis of description the and methods the of reporting complete code, and materials relevant. also are analyses statistical re-analyses and data-sharing for suggestions) (and challenges identified 3: Some Table Page 31 of 63 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 BMJ Page 32 of 63

1 2 3 91 RU-TE-2014-4-1316 (awarded to Dr. Cristea), and the work of J.I. is supported by an unrestricted gift from Sue and 4 92 Bob O’Donnell. The sponsors had no role concerning preparation, review or approval of the manuscript. 5 93 6 94 Acknowledgments 7 95 8 96 This study was made possible through sharing of anonymized individual participant data from all studies’ authors. 9 97 We thanks all authors that were contacted for this study: C. Bullen and the National Institute for Health Innovation, 10 98 S. Gilbody, C. Hewitt, L. Littlewood, C. van der Meulen, H. van der Aa, S. Cohen, M. Bicket, T. Harris, the STOP 11 99 GAP studyConfidential: investigators including Kim Thomas, AlanFor Montgomery Review and Nicola Greenlaw, Only Nottingham University 12 100 Hospitals NHS Trust, NIHR Programme Grants for Applied Research, the Nottingham Clinical Trials Unit, C. 13 101 Poylac, K. Yuhas, C. Adrion, G. Greisen, S. Hyttel-Sørensen A. Barker, R. Morello, K. Luedtke, M. Paul, D. Yahav, 14 102 L. Chesterton, the Arthritis Research UK Primary Care Centre and C. Hanson. 15 103 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 3 60 https://mc.manuscriptcentral.com/bmj Page 35 of 63 BMJ

1 2 3 104 ABSTRACT 4 105 5 6 7 106 Objective – To explore the effectiveness of randomised controlled trial (RCT) data sharing in 8 9 107 journals with a full data-sharing policy and to describe potential difficulties encountered in the 10 Confidential: For Review Only 11 108 process of performing re-analyses of the primary outcome(s). 12 13 14 109 Design – Survey of published RCTs. Data availability was defined as the eventual receipt of 15 16 110 complete data with clear labelling. All primary outcomes were then re-analysed to assess to what 17 18 111 extent studies were reproduced. All difficulties encountered were described. 19 20 21 112 Data sources – Eligible studies were identified from PubMed/Medline. 22 23 113 Eligibility criteria for selecting studies – All RCTs that had been submitted and published by 24 25 114 The BMJ and PLOS Medicine subsequent to the adoption of data sharing policies by these 26 27 115 journals. 28 29 30 116 Results – Thirty-seven RCTs (21 from The BMJ and 16 from PLOS Medicine) published 31 32 117 between 2013 and 2016 met our eligibility criteria. 17/37 (46% (95% CI [30% - 62%]) satisfied 33 34 118 our definition of data availability and 14 of the 17 (82%, 95% CI [59% - 94%]) were fully 35 36 37 119 reproduced on all their primary outcomes. Of the remaining, we identified errors in two RCTs 38 39 120 but reached similar conclusions and one paper did not provide enough information in the 40 41 121 Methods section to reproduce the analyses. We identified difficulties, such as problems in 42 43 44 122 contacting corresponding authors and lack of resources on their behalf in preparing the datasets. 45 46 123 In addition, we found a range of different data sharing practices across study groups. 47 48 124 Conclusions - Data availability was not optimal even in journals with a strong policy for data 49 50 125 sharing. When investigators shared data, most re-analyses largely reproduced the original results. 51 52 53 126 Data sharing practices need to become more widespread and streamlined to allow meaningful re- 54 55 127 analyses and re-use of data. 56 57 58 59 4 60 https://mc.manuscriptcentral.com/bmj BMJ Page 36 of 63

1 2 3 128 Registration number - Open Science Framework (osf.io/c4zke) 4 5 6 129 7 8 130 9 10 11 Confidential: For Review Only 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 5 60 https://mc.manuscriptcentral.com/bmj Page 37 of 63 BMJ

1 2 3 131 What is already known on this subject 4 5 6 132 7 8 133 • In 2017, the International Committee of Medical Journal Editors mandate that a data 9 10 134 sharing plan to be included in each paper (and pre-specified in study registration). 11 Confidential: For Review Only 12 13 135 • Two leading general medical journals, The BMJ and PLOS Medicine, already have a 14 15 136 stronger policy expressly requiring data-sharing as a condition for publication of 16 17 137 randomised controlled trials (RCTs). 18 19 20 138 • Only a small number of re-analyses of RCTs have been published to date; of these, only a 21 22 139 minority was conducted by entirely independent authors. 23 24 140 25 26 141 What this study adds 27 28 29 142 30 31 143 • Data availability was not optimal even in journals with a stronger policy, but the 46% 32 33 144 data sharing rate we observed is much higher than elsewhere in the biomedical literature. 34 35 36 145 • When re-analyses are possible, these mostly yield similar results to the original analysis. 37 38 146 However, these re-analyses used data at a very mature analytical stage. Exploring the 39 40 147 impact of data processing, coding, cleaning and re-categorization of events on the results 41 42 43 148 requires further research. 44 45 149 • Problems in contacting corresponding authors, lack of resources in preparing the datasets 46 47 150 and important heterogeneity in data sharing practices are barriers to overcome. 48 49 50 151 51 52 152 53 153 54 154 55 56 57 58 59 6 60 https://mc.manuscriptcentral.com/bmj BMJ Page 38 of 63

1 2 3 155 INTRODUCTION 4 5 156 6 7 157 Ideally, patients, medical practitioners and health policy analysts feel more confident when the 8 158 results and conclusions of scientific studies can be verified. Value of medical care increases 9 10 159 when people waste less time pursuing wrong claims. However, for a long time, verifying the 11 Confidential: For Review Only 12 160 results of clinical trials was not possible, since the data on which their conclusions are based 13 14 161 were not available. 15 162 16 17 163 In 2016, the International Committee of Medical Journal Editors (ICMJE) published an editorial1 18 19 164 stating that “it is an ethical obligation to responsibly share data generated by interventional 20 21 165 clinical trials because participants have put themselves at risk.” The ICJME proposed to require 22 166 that de-identified individual-patient data (IPD) are made publicly available no later than 6 23 24 167 months after publication of the trial results. This proposal triggered debate.2-7 In June 2017, the 25 26 168 ICJME stepped back from their proposal. The new requirements do not mandate data sharing, 27 169 but only a data sharing plan to be included in each paper (and pre-specified in study 28 29 170 registration).8 30 31 171 32 33 172 Because of this trend toward a new norm where data sharing for randomized controlled trials 34 173 (RCTs) becomes a standard, it seems important to assess how accessible the data are in journals 35 36 174 with existing data-sharing policies. Two leading general medical journals, The BMJ9 10 and 37 11 38 175 PLOS Medicine, already have a policy expressly requiring data-sharing as a condition for 39 176 publication of clinical trials: data sharing became a requirement after January 2013 for RCTs on 40 41 177 drugs and devices9 and July 2015 for all therapeutics10 at The BMJ, and after March 2014 for all 42 43 178 types of interventions at PLOS Medicine. 44 45 179 46 180 The aim of our study was to explore the effectiveness of RCT data sharing in both journals in 47 48 181 terms of data availability, feasibility and accuracy of re-analyses, and to describe potential 49 50 182 difficulties encountered in the process of performing re-analyses of the primary outcome(s). We 51 52 183 focused on RCTs because they are considered to represent high quality evidence and because 53 184 availability of their data is crucial in the evaluation of health interventions. RCTs represent the 54 55 185 most firmly codified methodology, which also allows data to be most easily analysed. Moreover, 56 57 58 59 7 60 https://mc.manuscriptcentral.com/bmj Page 39 of 63 BMJ

1 2 3 186 RCTs have been the focus of transparency and data sharing initiatives for a longer time,12 due to 4 5 187 the importance of primary data availability in the evaluation of therapeutics (e.g. for individual 6 7 188 patient data meta-analyses). 8 189 9 10 190 METHODS 11 Confidential: For Review Only 12 191 13 14 192 Methods were specified in advance and documented in a protocol submitted for review on 2016- 15 193 11-12 and subsequently registered with the Open Science Framework on 2016-11-15 16 17 194 (registration number: osf.io/c4zke; see https://osf.io/u6hcv/register/565fb3678c5e4a66b5582f67). 18 19 195 20 21 196 Eligibility criteria 22 197 23 24 198 We surveyed publications of RCTs, including cluster trials and cross-over studies, non- 25 26 199 inferiority design and superiority designs, that had been submitted and published by The BMJ 27 200 and PLOS Medicine subsequently to the adoption of data sharing policies by these journals. Data 28 29 201 sharing became a requirement after January 2013 for RCTs on drugs and devices9 and July 2015 30 10 31 202 for all therapeutics at The BMJ, and after March 2014 for all types of interventions at PLOS 32 33 203 Medicine. 34 204 35 36 205 Search strategy 37 38 206 39 207 Eligible studies were identified from PubMed/Medline. For The BMJ the following search 40 41 208 strategy was used: "BMJ"[jour] AND ("2013/01/01"[PDAT]: "2017/01/01"[PDAT]) AND 42 43 209 Randomized Controlled Trial[ptyp]. For PLOS Medicine the following search strategy was used: 44 45 210 ""PLoS Med"[jour]) AND ("2014/03/01"[PDAT]: "2017/01/01"[PDAT]) AND Randomized 46 211 Controlled Trial[ptyp]. 47 48 212 49 50 213 Study selection 51 52 214 53 215 The eligibility assessment was performed independently by two reviewers (FN and PJ). 54 55 216 Disagreements were resolved by consensus or in consultation with a third reviewer (JPAI or 56 57 58 59 8 60 https://mc.manuscriptcentral.com/bmj BMJ Page 40 of 63

1 2 3 217 DM). More specifically, the eligibility assessment was based on the date of submission (and not 4 5 218 of publication). When these dates were not available they were obtained by contacting the 6 7 219 journal editors. 8 220 9 10 221 Data extraction and datasets retrieval 11 Confidential: For Review Only 12 222 13 14 223 A data extraction sheet was developed. For each included study, information was extracted on 15 224 study characteristics (country of the corresponding author, design, sample size, condition and 16 17 225 funding), type of intervention (drug, device, other) and procedure to gather the data. Two authors 18 19 226 (FN and PJ) independently extracted the data from the included studies. Disagreements were 20 21 227 resolved by consensus or in consultation with a third reviewer (JPAI). One reviewer (FN) was in 22 228 charge of retrieving the individual patient data for all included studies by following the 23 24 229 instructions found in the data-sharing statement of the included studies. More specifically, when 25 26 230 data were available upon request, a standardised e-mail was sent (see https://osf.io/h9cas/). All 27 231 initial emails were sent from a professional email address ([email protected]), and 3 28 29 232 additional reminders were sent to each author 2-3 weeks apart, in case of non-response. 30 31 233 32 33 234 Data availability 34 235 35 36 236 Our primary outcome was data availability, defined as the eventual receipt of data presented with 37 38 237 sufficient information to reproduce the analysis of the primary outcome(s) of the included RCTs 39 238 (i.e. complete data with clear labelling). Additional information was collected on type of data 40 41 239 sharing (upon request by e-mail, upon request on a specific website, upon request on a specific 42 43 240 register, available on a public register, other); time for collecting the data (in days, time between 44 45 241 first attempt to success of getting a database); de-identification of data (concerning name, 46 242 birthdate and address); type of data shared (from case report forms to directly analysable 47 48 243 datasets);13 sharing of analysis code; and reasons for non-availability in case data were not 49 50 244 shared. 51 52 245 53 246 Reproducibility 54 55 247 56 57 58 59 9 60 https://mc.manuscriptcentral.com/bmj Page 41 of 63 BMJ

1 2 3 248 When data were available, a re-analysis of the trial was performed by a single researcher (FN). 4 5 249 For each study, analyses were repeated exactly as described in the published report of the study. 6 7 250 Whenever insufficient details about the analysis was provided in the study report, we sought 8 251 clarifications from the trial investigators. Only analyses concerning the primary outcome (or 9 10 252 outcomes, if multiple primary outcomes existed) of each trial were considered. Any discrepancy 11 Confidential: For Review Only 12 253 between results obtained in the re-analysis and those reported in the publication was examined in 13 14 254 consultation with a statistician (CS). This examination aimed to determine if, based on both 15 255 quantitative (effect size, p-values) and qualitative (clinical judgment) consideration, the 16 17 256 discrepant results of the re-analysis entailed a different conclusion from the one reported in the 18 19 257 original publication. Any disagreement or uncertainty over such conclusions was resolved by 20 21 258 consulting a third co-author with expertise in both clinical medicine and statistical methodology 22 259 (JPAI). If, following this assessment process, results (and eventually conclusions) were still 23 24 260 determined to be not reproduced, CS independently re-analysed the data to confirm such 25 26 261 conclusion. Once the “not reproduced” status of a publication was confirmed, the authors of the 27 262 study were contacted by FN to discuss the source of the discrepancy. Following this assessment 28 29 263 procedure, studies were classified into 4 categories: fully reproduced; not fully reproduced but 30 31 264 same conclusion; not reproduced and different conclusion; and not reproduced (or partially 32 33 265 reproduced) because of missing information. 34 266 35 36 267 Difficulties encountered in getting data and/or code, using them and performing the re- 37 38 268 analyses 39 269 40 41 270 We noted whether the sharing of data and/or analytical code required clarifications for which 42 43 271 additional queries had to be made to the authors in order to obtain the relevant data and/or code, 44 45 272 clarify their labels and/or use, and reproduce the original analysis of the primary outcomes. A 46 273 catalogue of these queries was created and similar clarifications were grouped for descriptive 47 48 274 purposes in order to generate a list of some common challenges and to help address these 49 50 275 challenges pre-emptively in future published trials. 51 52 276 53 277 Statistical analyses 54 55 278 56 57 58 59 10 60 https://mc.manuscriptcentral.com/bmj BMJ Page 42 of 63

1 2 3 279 Percentages of data sharing or reproducibility were computed with 95% confidence intervals 4 5 280 based on binomial approximation or on Wilson score method without continuity correction if 6 14 7 281 necessary. For the purposes of registration, we hypothesized that, if effective, these data 8 282 sharing policies would lead to more than 80 % of studies sharing their data (i.e. the lower 9 10 283 boundary of its confidence interval had to be above 80 %). One should expect high rates of data 11 Confidential: For Review Only 12 284 sharing resulting from full data sharing policies. However, based on previous experience of 13 15 14 285 explicit data sharing policies in Psychological Science we knew that a rate of 100% was not 15 286 realistic and judged that an 80% rate could be a desirable outcome. 16 17 287 When data was available, re-analyses were performed, using the open source statistical software 18 19 288 R (R Development Core Team) for one researcher (FN) and SAS (SAS Institute Inc) for the 20 21 289 senior statistician (CS). In addition, when authors shared their codes (in R, SAS, STATA 22 290 (StataCorp. 2017) or other), these were checked and used. Estimates of effect sizes, 95% 23 24 291 confidence intervals and p-values were obtained for each re-analysis. 25 26 292 27 293 Changes from the initial protocol 28 29 294 30 31 295 Initially we planned to include all studies published after the data sharing policies were in place. 32 33 296 Nevertheless, some authors that we contacted suggested that their studies were not eligible 34 297 because the papers were submitted to the journal before the policy. We contacted editors who 35 36 298 confirmed that policies applied for papers submitted (and not published) after the policy was 37 38 299 adopted. In accordance, and to avoid any underestimation of data sharing rates, we changed our 39 300 selection criteria to those presented above. For a few additional studies submitted before the 40 41 301 policy, data were collected and re-analyzed but only described in the web-appendix. 42 43 302 44 45 303 Concerning the re-analysis, we initially planned to consider non-reproducibility as a 46 304 disagreement between re-analysed results and results reported in the original publication by more 47 48 305 than 2% in the point estimate or 95% confidence interval. After reanalysis of a couple of studies 49 50 306 we felt that such a definition was sometimes meaningless. Interpreting a RCT involves clinical 51 52 307 expertise, and cannot be reduced to solely quantitative factors. Accordingly, we changed our 53 308 definition as stated above and provided a detailed description of re-analyses and published 54 55 309 results, mentioning the nature of effect size and the type of outcome considered. 56 57 58 59 11 60 https://mc.manuscriptcentral.com/bmj Page 43 of 63 BMJ

1 2 3 310 4 5 311 Patient involvement 6 7 312 8 313 We had no established contacts with specific patient groups who might be involved in this 9 10 314 project. No patients were involved in setting the research question or the outcome measures, nor 11 Confidential: For Review Only 12 315 were they involved in the design and implementation of the study. There are no plans to involve 13 14 316 patients in the dissemination of results, nor will we disseminate results directly to patients. 15 317 16 17 318 RESULTS 18 19 319 20 21 320 Characteristics of included studies 22 321 23 24 322 A flow-chart detailing the study selection process is provided in Figure 1. The searches 25 26 323 performed on 2016-11-12 resulted in 159 citations. Of these, 134 full-texts were considered for 27 324 eligibility. Thirty-seven RCTs (21 from The BMJ and 16 from PLOS Medicine) published 28 29 325 between 2013 and 2016 met our eligibility criteria. Characteristics of these studies are presented 30 31 326 in Table 1. These RCTs had a median sample size of 432 participants (IQR = 213 – 1070), had 32 33 327 no industry funding in 26 cases (70 %) and were led by teams from Europe in 25 cases (67 %). 34 328 Twenty RCTs (54 %) evaluated pharmacological interventions, 9 (24 %) complex interventions 35 36 329 (e.g. psychotherapeutic program) and 8 (22 %) devices. 37 38 330 39 40 331 Data availability 41 332 42 43 333 We were able to access data for 19/37 studies (51%). Among these 19 studies, median time for 44 45 334 collecting the data was 4 days (range= [0 – 191]). Two of these studies, however, did not provide 46 335 sufficient information within the dataset to enable direct re-analysis (e.g., had unclear labels), 47 48 336 therefore 17 studies satisfied our definition of data availability. Data availability rate was thus 49 50 337 46% (95% CI [30% - 62%]). 51 52 338 Data was in principle available for two additional studies, not included in the count above and 53 339 both authored by the same research team. However, their authors asked us to cover the financial 54 55 340 costs of preparing the data for sharing (£607). Since other teams shared the data for free, we 56 57 58 59 12 60 https://mc.manuscriptcentral.com/bmj BMJ Page 44 of 63

1 2 3 341 considered that it would not have been fair to pay some and not others for similar work in the 4 5 342 context of our project and we classified these two studies as not sharing data. For a third study, 6 7 343 the authors were in correspondence with us and discussing conditions for sharing data, but we 8 344 did not receive their data by the time our data collection process was determined to be over (7 9 10 345 months). If these three studies were included, the proportion of data sharing would be 54% (95% 11 Confidential: For Review Only 12 346 CI [38% - 70%]). 13 14 15 347 For the remaining 15 studies classified as not sharing data, reasons for non-availability were: no 16 348 answer to the different e-mails (n=7); no answer after an initial agreement (n=2); and refusal to 17 18 349 share data (n=6). Explanations for refusal to share data included: lack of endorsement of the 19 20 350 objectives of our study (n=1); personal reason (e.g. sick leave, n=2); restrictions due to an 21 22 351 embargo on data sharing (n=1); no specific reason offered (n=2). The existence of possible 23 352 privacy concerns was never put forward as a reason for not sharing data. 24 25 353 26 27 354 Among the 19 studies sharing some data (analysable datasets and non-analysable datasets), 16 28 355 (84 %) datasets were totally de-identified. Birthdates were found in 3 datasets and geographical 29 30 356 information (country and postcode) in 1 of these 3. Most datasets were data ready for analysis 31 32 357 (n= 17), whereas 2 required some additional processing before the analysis could be repeated. In 33 34 358 these two cases, such processing was difficult to implement (even with the code being available) 35 359 and authors were then contacted to share analysable data. Statistical analysis code was available 36 37 360 for 7 studies (including 2 that were obtained after a second specific request). 38 39 361 40 41 362 Reproducibility 42 363 43 44 364 Among the 17 studies providing sufficient data for re-analysis of their primary outcome(s) 14 45 46 365 (82%, 95% CI [59% - 94%]) studies were fully reproduced on all their primary outcomes. One of 47 366 the 17 studies did not provide enough information in the Methods section to reproduce the 48 49 367 analyses (specifically, the methods used for adjustment were unclear). We contacted the authors 50 51 368 of this study to obtain clarifications but received no reply. Of the remaining 16 studies, we re- 52 53 369 analyzed 47 different primary analyses. Two of these studies were considered not fully 54 370 reproduced. For one study, we identified an error in the statistical code and for the second one 55 56 371 we found slightly different numerical values for the effect sizes measured as well as slight 57 58 59 13 60 https://mc.manuscriptcentral.com/bmj Page 45 of 63 BMJ

1 2 3 372 differences in terms of numbers of patients included in the analyses (a difference of 1 patient in 4 5 373 one of 4 analyses). Nevertheless, similar conclusions were reached in both cases (these two 6 7 374 studies were categorized as not fully reproduced but reaching the same conclusion). Therefore, 8 375 we found no results contradicting the initial publication, neither in terms of magnitude of the 9 10 376 effect (Table 2) nor in terms of statistical significance of the finding (Figure 2). 11 Confidential: For Review Only 12 377 We retrieved the data of 3 additional studies, published in The BMJ after its data sharing policy 13 14 378 was in place (but submitted before the policy). Although these studies were ineligible for our 15 379 main analysis, re-analyses were performed and also reached the same conclusions as the initial 16 17 380 study (e-Table 1). 18 19 381 20 21 382 Difficulties encountered in getting data and/or code, using them and performing the re- 22 383 analyses 23 24 384 25 26 385 Based on our correspondence with authors, we identified several difficulties in getting the data. 27 386 A frequent concern pertained to the costs involved in the data-sharing process, for example, the 28 29 387 costs of preparing the data or translating the database from one language to another. Some 30 31 388 authors wondered whether their team or our team should assume these costs. In addition, some of 32 33 389 the authors balanced these additional costs with their perceived benefits of sharing data for the 34 390 purpose of this study and seemed to rather value data sharing for the purpose of a meta-analysis 35 36 391 than for re-analyses, possibly because of the risk acknowledged by one investigator we contacted 37 38 392 about “naming and shaming” individual studies/investigators. 39 393 Getting prepared and preplanning for data sharing seems still a challenge for many trial groups; 40 41 394 data sharing proved to be novel for some authors who were unsure how to proceed. Indeed, there 42 43 395 was considerable heterogeneity between different procedures to share data: provided in an open 44 45 396 repository (n=5), downloadable on a secured website (n=1) after registration, included as 46 397 appendix of the published paper (n=3), or sent by e-mail (n=10). In 3 occasions, we signed a 47 48 398 data-sharing request/agreement. In these agreements, the sponsor and recipient parties specified 49 50 399 the terms that bound them in the data sharing process (e.g. concerning the use of data, 51 52 400 intellectual property, etc). In addition, typically there was no standard in what type of data were 53 401 shared (at what level of cleaning and processing). In one case, authors mentioned explicitly that 54 55 402 they followed standardized guidelines16 to prepare the dataset. 56 57 58 59 14 60 https://mc.manuscriptcentral.com/bmj BMJ Page 46 of 63

1 2 3 403 Some analyses were complex and it was sometimes challenging to re-analyze data from very 4 5 404 specific designs or when unusual measures were used (e.g. relative change in percentage). 6 7 405 Obtaining more information about the analysis by contacting authors was necessary for 6 of 17 8 406 studies to replicate the findings. In one case, specific exploration of the code revealed that, in a 9 10 407 survival analysis, authors treated repeated events in the same patient (multiple events) as distinct 11 Confidential: For Review Only 12 408 observations. This was in disagreement with the methods section describing usual survival 13 14 409 analysis (taking into account the only first event for each patient). However, alternative analyses 15 410 did not contradict the published results. 16 17 411 Three databases did not provide sufficient information to reproduce the analyses. Missing data 18 19 412 concerned variables used for adjustment, definition of the analysis population, and 20 21 413 randomization groups. Communication with authors was therefore necessary and was fruitful in 22 414 one these 3 cases. 23 24 415 25 26 416 DISCUSSION 27 417 28 29 418 Statement of principal findings 30 31 419 32 33 420 In two prominent medical journals with a strict data-sharing policy for RCTs, we found that for 34 421 46% (95% CI [30% - 62%]) of published articles the original investigators shared their data with 35 36 422 sufficient information to enable re-analyses. This rate was under the 80% boundary that we pre- 37 38 423 specified as an acceptable threshold for papers submitted under a policy that makes data sharing 39 424 an explicit condition for publication. However, despite being lower than might be expected, a 40 41 425 46% data sharing rate is much higher than the average rate of biomedical literature at large, in 42 17 43 426 which data sharing is almost non-existent (with few exceptions in some specific fields such as 44 18 45 427 genetics) . Moreover, our analyses focused on publications that were submitted right after the 46 428 implementation of new data-sharing policies, which might be expected to have practical and 47 48 429 cultural barriers to their full implementation. Indeed, our correspondence with the authors helped 49 50 430 identify several practical difficulties connected to data sharing, including difficulties in 51 52 431 contacting corresponding authors, and lack of time and financial resources on their behalf in 53 432 preparing the datasets for us. In addition, we found a wide variety of data sharing practices 54 55 433 between study groups (i.e. regarding the type of data that can be shared and the procedures that 56 57 58 59 15 60 https://mc.manuscriptcentral.com/bmj Page 47 of 63 BMJ

1 2 3 434 are necessary to follow in order to get the data). Data sharing practices could evolve in the future 4 5 435 to address these barriers to data sharing (Table 3). 6 7 8 436 For all results that we were able to re-analyze, we reached similar conclusions (despite 9 437 occasional slight differences in the numerical estimations) to those reported in the original 10 Confidential: For Review Only 11 438 publication and this result is reassuring that at least the available data shared do correspond 12 13 439 closely to the reported results. Of course, there is a large amount of diversity on what exactly 14 15 440 “raw data” mean and they can involve various transformations (from the case report forms to 16 441 coded and analyzable data).13 Here, we relied on late-stage, coded, and cleaned data and 17 18 442 therefore, the potential for leading to a different conclusion was probably small. Data processing, 19 20 443 coding, cleaning and re-categorization of events can have substantial impact on the results in 21 22 444 some trials. For example, SmithKline Beecham’s Study 329 was a well-known study on 23 445 paroxetine in adolescent depression presenting the drug as safe and effective19 while a re- 24 25 446 analysis starting from the case report forms demonstrated a lack of efficacy and some serious 26 20 27 447 safety issues. 28 448 29 30 449 Strengths and weaknesses of the study 31 32 450 33 34 451 Some leading general medical journals, New England Journal of Medicine, Lancet, JAMA and 35 452 JAMA Internal Medicine, have had no specific policy for data sharing in RCTs until recently. 36 37 453 Annals of Internal Medicine has encouraged (but not demanded) data-sharing since 2007.21 BMC 38 39 454 Medicine adopted a similar policy in 2015. The BMJ and PLOS Medicine have adopted stronger 40 455 policies, beyond the ICMJE policy, that mandate data sharing for RCTs. Our survey of RCTs 41 42 456 published in these two journals might therefore give a taste of the impact and caveats of such full 43 44 457 policies. However, one should be careful and not generalize these results to other journals. First, 45 46 458 we have a very selected sample of studies. Our sample includes studies (mostly from Europe) 47 459 that are larger and less likely to be funded by the industry than the average published RCT in the 48 49 460 medical literature.22 Most of these re-analyzed RCTs were large, and sometimes very large. 50 51 461 Several, especially those published in PLOS Medicine, were cluster randomized studies, and 52 53 462 many explored infectious disease or important public health issues, characteristics that are not 54 463 very common in RCTs overall. Similarly, few of the evaluated studies were sponsored by 55 56 464 industry. Some public funders (or charities) involved in their funding have already open access 57 58 59 16 60 https://mc.manuscriptcentral.com/bmj BMJ Page 48 of 63

1 2 3 465 policies such as the Bill and Melinda Gates foundation or the National Institute for Health 4 5 466 Research in the United Kingdom. 6 7 467 These reasons also explain why we did not compare journals. Qualitatively they don’t publish 8 468 the same kind of RCTs and quantitatively, The BMJ and PLOS Medicine publish few RCTs in 9 10 469 comparison to other leading journals, such as the New England Journal of Medicine and Lancet. 11 Confidential: For Review Only 12 470 Any comparisons might have been subject to confounding. Similarly, we believed that pre-post 13 14 471 studies at The BMJ and PLOS Medicine might have been subject to historical bias and 15 472 confounding factors since such policies might have changed the profile of submitting authors: 16 17 473 researchers with a specific interest in open science and reproducibility might have been attracted 18 19 474 and others might have opted for another journal. We think comparative studies will be easier to 20 21 475 conduct when all journals adopt data sharing standards. Even with the current proposed ICJME 22 476 requirements, The BMJ and PLOS Medicine will be journals with stronger policies. In addition, 23 24 477 The BMJ and PLOS Medicine are both major journals with many resources such as in-depth 25 26 478 discussion of papers in editorial meetings, and statistical peer review. Our findings concerning 27 479 reproducibility might not apply to smaller journals with more limited resources. We cannot 28 29 480 generalize this finding to studies that did not share their data: authors who are confident in their 30 31 481 results might more readily agree to share their data and this may lead to overestimation of 32 33 482 reproducibility. In addition, we might have missed a few references using the filter "Randomized 34 483 Controlled Trial[ptyp]", however, this is unlikely to have affected data sharing and 35 36 484 reproducibility rates. 37 38 485 Finally, in our study, the notion of data sharing was restricted to a request by a researcher group 39 486 while, in theory, other types of requestors (patients, clinicians, health and academic institutions, 40 41 487 etc.) may also be interested in the data. Moreover, we followed the strict procedure presented in 42 43 488 the paper and without sending any correspondence (e.g. rapid response) on the journals’ website. 44 45 489 In addition, our re-analyses were based on primary outcomes whereas secondary and safety 46 490 outcomes may be more problematic in their reproduction. These points need to be explored in 47 48 491 further studies. 49 50 492 51 52 493 Strengths and weaknesses in relation to other studies, discussing important differences in 53 494 results 54 55 495 56 57 58 59 17 60 https://mc.manuscriptcentral.com/bmj Page 49 of 63 BMJ

1 2 3 496 In a precedent survey of 160 randomly sampled BMJ research articles from 2009 to 2015, 4 5 497 excluding meta-analysis and systematic reviews,23 the authors found that only 5% shared their 6 7 498 data sets. Nevertheless, this survey assessed data sharing among all studies with original raw data 8 499 while The BMJ data sharing policy specifically applied to clinical trial data. When considering 9 10 500 clinical trials bound by The BMJ data sharing policy (n=21), the percent shared was 24% (8%- 11 Confidential: For Review Only 12 501 47%). Our study identified higher rates of shared datasets in accordance with an increase in the 13 14 502 rate of ‘data shared’ for every additional year between 2009 and 2015 found in the previous 15 503 survey. It is not known whether it results directly from the policy implementation or of a slow 16 17 504 and positive cultural swift of trialists. 18 19 505 20 21 506 Lack of response from authors was also identified as a caveat in the previous evaluation, which 22 507 suggested that the wording of The BMJ policy such as availability on ‘reasonable request’ might 23 24 508 be interpreted in different ways.23 Previous research across PLOS also suggests that data requests 25 24 26 509 by contacting authors might be sometimes ineffective. Despite writing a data sharing 27 510 agreement in their paper, corresponding authors are still free to decline data requests. Our study 28 29 511 might be perceived by some authors as imposing unreasonable requests on them. For example, 30 31 512 one may question whether sharing data for the purposes of this project constitutes a reasonable 32 33 513 request. One might consider that this meta-research project exploring the effectiveness of data 34 514 sharing policies lies outside the purpose for which the initial trial was done and for which the 35 36 515 participants gave consent. A survey found that authors are generally less willing to share their 37 25 38 516 data for the purpose of re-analyses than, for instance, for individual patient data meta-analyses. 39 517 Nevertheless, reproducibility checks, based on independent re-analysis are perfectly aligned with 40 41 518 the primary objective of clinical trials and indeed with patient interests. Conclusions that persist 42 43 519 through substantial re-analyses are becoming more credible. In order to explain and support the 44 45 520 interest of our study, we have registered a protocol and have transparently described our 46 521 intentions in our e-mail, but data sharing rates were still lower than we expected. Active auditing 47 48 522 of data sharing policies by journal editors may facilitate data sharing implementation. 49 50 523 51 52 524 Concerning reproducibility of clinical trials, an empirical analysis suggests that only a small 53 525 number of re-analyses of RCTs have been published to date; of these, only a minority was 54 55 526 conducted by entirely independent authors. In a previous empirical evaluation of published re- 56 57 58 59 18 60 https://mc.manuscriptcentral.com/bmj BMJ Page 50 of 63

1 2 3 527 analyses, thirty-five percent (13/37) of published re-analyses yielded changes in findings that 4 5 528 implied conclusions different from those of the original article as to whether patients should be 6 26 7 529 treated or not or about which patients should be treated. In our assessment, we found no 8 530 differences in conclusions pertaining to treatment decisions. The difference between the previous 9 10 531 empirical evaluation and the current one is probably due to many factors. Published re-analyses 11 Confidential: For Review Only 12 532 in the past would be unlikely to be published if they had found the very same results and had 13 14 533 reached the very same conclusions as the original analysis. Therefore, the set of published re- 15 534 analyses is enriched with discrepant results and conclusions. Moreover, past published re- 16 17 535 analyses addressed the same question on the same data but using typically different analytical 18 19 536 methods. Conversely, we used the very same analysis employed by the original paper. In 20 21 537 addition, the good computational reproducibility (replication of analyses) we found is only one 22 538 aspect of reproducibility27 and should not be over interpreted. For example, in psychology, 23 24 539 numerous labs have volunteered to re-run experiments (not solely analyses), with the methods 25 28 26 540 used by the original researchers. 39/100 studies were considered as successfully replicated. But 27 541 attempts to replicate psychological studies are often easier to implement than attempts to 28 29 542 replicate RCTs that are often costly and difficult to run. This is especially true for large RCTs. 30 31 543 32 33 544 Meaning of the study: possible explanations and implications for clinicians and 34 545 policymakers 35 36 546 37 38 547 The ICJME’s requirements adopted in 2017 mandate that a data sharing plan will have to be 39 548 included in each paper (and pre-specified in study registration).8 Our results suggest that this not 40 41 549 likely to be sufficient to achieve high rates of data sharing. One can imagine that individual 42 43 550 authors will agree to write such a statement in line with the promise of publication, but the time 44 45 551 and costs involved into such data preparation might lead them to be reluctant to answer data 46 552 sharing requests. Interestingly the ICJME also mandates that clinical trials that begin enrolling 47 48 553 participants on or after January 1, 2019, must include a data sharing plan in the trial’s 49 50 554 registration. An a priori data sharing plan might push more authors to pre-emptively address and 51 52 555 find funding for sharing, but its eventual impact on actual sharing is unknown. Funders are well 53 556 positioned to facilitate data sharing. Some such as the Wellcome Trust already allow 54 55 557 investigators to use funds towards open access article processing charges which is typically a 56 57 58 59 19 60 https://mc.manuscriptcentral.com/bmj Page 51 of 63 BMJ

1 2 3 558 very small fraction of awarded funding. Funders could extend their funding policy and also allow 4 5 559 investigators to use a similar small fraction of funding towards enabling data sharing. In addition, 6 5 7 560 patients can help promote a culture of data sharing for clinical trials. 8 9 561 In addition, reproducible research does not solely imply sharing data, but also reporting all steps 10 Confidential: For Review Only 11 562 of the statistical analysis. This is one core principle of the CONSORT statement “Describe 12 13 563 statistical methods with enough detail to enable a knowledgeable reader with access to the 14 29 15 564 original data to verify the reported results”. It is nonetheless sometimes difficult to provide 16 565 such detailed information in a restricted number of lines (i.e. a paper). We suggest that details of 17 18 566 the statistical analysis plan have to be provided as well as the detailed labels used in the table and 19 30 20 567 efficient analytical code sharing is also essential. 21 22 23 568 We suggest that if journals ask for data and code to be shared, they should ensure that this 24 569 material is also reviewed by editorial staff (and/or peer reviewers) or at a minimum checked for 25 26 570 completeness and basic usability.30 This could also be positively translated in specific incentives 27 15 28 571 for the paper; e.g. many psychology journals use badges as signs of good research practices 29 572 (including data sharing) adopted by papers. And, beyond incentivizing authors, journals adopting 30 31 573 such practices could also be incentivized by a gain in reputation and credibility. 32 33 34 574 Though ensuring patient privacy and lack of explicit consent for sharing are often cited as major 35 31 36 575 barriers to sharing RCT data (and generally accepted as valid exemptions), none of the 37 576 investigators we approached mentioned this reason. This could suggest that technical constraints, 38 39 577 lack of incentives and dedicated funding, and a general diffidence towards re-analyses might be 40 41 578 more germane obstacles to be addressed. 42 43 579 Finally, data sharing practices were very different from one team to another. There is thus a need 44 45 580 for standardization and for drafting specific guidelines for best practices in data sharing. 46 47 581 48 49 582 Unanswered questions and future research 50 583 51 52 584 We recommend some prospective monitoring of data sharing practices to ensure that the new 53 54 585 ICJME’s requirements are effective and useful. To this end, one also should keep in mind that 55 56 586 data-availability is a surrogate of the expected benefit of having open data. Proving that data- 57 58 59 20 60 https://mc.manuscriptcentral.com/bmj BMJ Page 52 of 63

1 2 3 587 sharing rates impacts re-use32 of the data is a further step. Proving that this reuse might translate 4 5 588 into discovery that can change care without generating false positive findings (e.g. in series of 6 7 589 unreliable a posteriori subgroup analyses) is even more challenging. 8 9 590 10 Confidential: For Review Only 11 591 REFERENCES 12 13 592 14 15 593 1. Taichman DB, Backus J, Baethge C, et al. Sharing Clinical Trial Data: A Proposal from the 16 594 International Committee of Medical Journal Editors. PLoS Med 2016;13(1):e1001950. doi: 17 595 10.1371/journal.pmed.1001950 [doi] 18 596 PMEDICINE-D-15-03676 [pii] [published Online First: 2016/01/21] 19 597 2. Platt R, Ramsberg J. Challenges for Sharing Data from Embedded Research. N Engl J Med 20 598 2016;374(19):1897. doi: 10.1056/NEJMc1602016 [doi] [published Online First: 2016/04/21] 21 22 599 3. Kalager M, Adami HO, Bretthauer M. Recognizing Data Generation. N Engl J Med 23 600 2016;374(19):1898. doi: 10.1056/NEJMc1603789 [doi] [published Online First: 2016/04/21] 24 601 4. Devereaux PJ, Guyatt G, Gerstein H, et al. Toward Fairness in Data Sharing. N Engl J Med 25 602 2016;375(5):405-7. doi: 10.1056/NEJMp1605654 [doi] [published Online First: 2016/08/16] 26 603 5. Haug CJ. Whose Data Are They Anyway? Can a Patient Perspective Advance the Data- 27 604 Sharing Debate? N Engl J Med 2017;376(23):2203-05. doi: 10.1056/NEJMp1704485 [published 28 605 Online First: 2017/04/27] 29 30 606 6. Rockhold F, Nisen P, Freeman A. Data Sharing at a Crossroads. N Engl J Med 31 607 2016;375(12):1115-7. doi: 10.1056/NEJMp1608086 [doi] [published Online First: 2016/09/23] 32 608 7. Rosenbaum L. Bridging the Data-Sharing Divide - Seeing the Devil in the Details, Not the 33 609 Other Camp. N Engl J Med 2017;376(23):2201-03. doi: 10.1056/NEJMp1704482 [doi] 34 610 [published Online First: 2017/04/27] 35 611 8. Taichman DB, Sahni P, Pinborg A, et al. Data sharing statements for clinical trials. BMJ 36 612 2017;357:j2372. [published Online First: 2017/06/07] 37 38 613 9. Godlee F, Groves T. The new BMJ policy on sharing data from drug and device trials. BMJ 39 614 2012;345:e7888. [published Online First: 2012/11/22] 40 615 10. Loder E, Groves T. The BMJ requires data sharing on request for all trials. BMJ 41 616 2015;350:h2373. [published Online First: 2015/05/09] 42 617 11. Bloom T, Ganley E, Winker M. Data access for the open access literature: PLOS's data 43 618 policy. PLoS Biology 2014;12(2):e1001797. 44 45 619 12. Mello MM, Francer JK, Wilenzick M, et al. Preparing for responsible sharing of clinical trial 46 620 data. N Engl J Med 2013;369(17):1651-8. doi: 10.1056/NEJMhle1309073 [doi] [published 47 621 Online First: 2013/10/23] 48 622 13. Zarin DA, Tse T. Sharing Individual Participant Data (IPD) within the Context of the Trial 49 623 Reporting System (TRS). PLoS Med 2016;13(1):e1001946. doi: 10.1371/journal.pmed.1001946 50 624 [doi] 51 625 PMEDICINE-D-15-03309 [pii] [published Online First: 2016/01/20] 52 53 626 14. Newcombe RG. Two-sided confidence intervals for the single proportion: comparison of 54 627 seven methods. Stat Med 1998;17(8):857-72. doi: 10.1002/(SICI)1097- 55 628 0258(19980430)17:8<857::AID-SIM777>3.0.CO;2-E [pii] [published Online First: 1998/05/22] 56 57 58 59 21 60 https://mc.manuscriptcentral.com/bmj Page 53 of 63 BMJ

1 2 3 629 15. Kidwell MC, Lazarevic LB, Baranski E, et al. Badges to Acknowledge Open Practices: A 4 630 Simple, Low-Cost, Effective Method for Increasing Transparency. PLoS Biol 5 6 631 2016;14(5):e1002456. doi: 10.1371/journal.pbio.1002456 [doi] 7 632 PBIOLOGY-D-16-00142 [pii] [published Online First: 2016/05/14] 8 633 16. Hrynaszkiewicz I, Norton ML, Vickers AJ, et al. Preparing raw clinical data for publication: 9 634 guidance for journal editors, authors, and peer reviewers. BMJ 2010;340:c181. [published Online 10 635 First: 2010/01/30]Confidential: For Review Only 11 636 17. Iqbal SA, Wallach JD, Khoury MJ, et al. Reproducible Research Practices and Transparency 12 637 across the Biomedical Literature. PLoS Biol 2016;14(1):e1002333. doi: 13 14 638 10.1371/journal.pbio.1002333 [doi] 15 639 PBIOLOGY-D-15-02900 [pii] [published Online First: 2016/01/05] 16 640 18. Mailman MD, Feolo M, Jin Y, et al. The NCBI dbGaP database of genotypes and 17 641 phenotypes. Nature genetics 2007;39(10):1181-6. doi: 10.1038/ng1007-1181 [published Online 18 642 First: 2007/09/28] 19 643 19. Keller MB, Ryan ND, Strober M, et al. Efficacy of paroxetine in the treatment of adolescent 20 21 644 major depression: a randomized, controlled trial. J Am Acad Child Adolesc Psychiatry 22 645 2001;40(7):762-72. doi: S0890-8567(09)60309-9 [pii] 23 646 10.1097/00004583-200107000-00010 [doi] [published Online First: 2001/07/05] 24 647 20. Le Noury J, Nardo JM, Healy D, et al. Restoring Study 329: efficacy and harms of paroxetine 25 648 and imipramine in treatment of major depression in adolescence. BMJ 2015;351:h4320. 26 649 [published Online First: 2015/09/18] 27 650 21. Laine C, Goodman SN, Griswold ME, et al. Reproducible research: moving toward research 28 29 651 the public can really trust. Ann Intern Med 2007;146(6):450-3. doi: 0000605-200703200-00154 30 652 [pii] [published Online First: 2007/03/07] 31 653 22. Chan AW, Altman DG. Epidemiology and reporting of randomised trials published in 32 654 PubMed journals. Lancet (London, England) 2005;365(9465):1159-62. doi: 10.1016/s0140- 33 655 6736(05)71879-1 [published Online First: 2005/03/30] 34 656 23. Rowhani-Farid A, Barnett AG. Has open data arrived at the British Medical Journal (BMJ)? 35 657 An observational study. BMJ Open 2016;6(10):e011784. doi: bmjopen-2016-011784 [pii] 36 37 658 10.1136/bmjopen-2016-011784 [doi] [published Online First: 2016/10/16] 38 659 24. Savage CJ, Vickers AJ. Empirical study of data sharing by authors publishing in PLoS 39 660 journals. PLoS One 2009;4(9):e7078. doi: 10.1371/journal.pone.0007078 [doi] [published Online 40 661 First: 2009/09/19] 41 662 25. Early Experiences With Journal Data Sharing Policies: A Survey of Published Clinical Trial 42 663 Investigators. International Congress on Peer Review and Scientific Publication; 2017; Chicago. 43 44 664 26. Ebrahim S, Sohani ZN, Montoya L, et al. Reanalyses of randomized clinical trial data. JAMA 45 665 2014;312(10):1024-32. doi: 1902230 [pii] 46 666 10.1001/jama.2014.9646 [doi] [published Online First: 2014/09/10] 47 667 27. Goodman SN, Fanelli D, Ioannidis JP. What does research reproducibility mean? Sci Transl 48 668 Med 2016;8(341):341ps12. doi: 8/341/341ps12 [pii] 49 669 10.1126/scitranslmed.aaf5027 [doi] [published Online First: 2016/06/03] 50 670 28. PSYCHOLOGY. Estimating the reproducibility of psychological science. Science 51 52 671 2015;349(6251):aac4716. doi: 349/6251/aac4716 [pii] 53 672 10.1126/science.aac4716 [doi] [published Online First: 2015/09/01] 54 55 56 57 58 59 22 60 https://mc.manuscriptcentral.com/bmj BMJ Page 54 of 63

1 2 3 673 29. Schulz KF, Altman DG, Moher D. CONSORT 2010 statement: updated guidelines for 4 674 reporting parallel group randomised trials. BMJ 2010;340:c332. [published Online First: 5 6 675 2010/03/25] 7 676 30. Stodden V, McNutt M, Bailey DH, et al. Enhancing reproducibility for computational 8 677 methods. Science 2016;354(6317):1240-41. doi: 10.1126/science.aah6168 [published Online 9 678 First: 2016/12/13] 10 679 31. LewandowskyConfidential: S, Bishop D. Research integrity:For Don'tReview let transparency Only damage science. 11 680 Nature 2016;529(7587):459-61. doi: 529459a [pii] 12 681 10.1038/529459a [doi] [published Online First: 2016/01/29] 13 14 682 32. Navar AM, Pencina MJ, Rymer JA, et al. Use of Open Access Platforms for Clinical Trial 15 683 Data. JAMA 2016;315(12):1283-4. doi: 2504799 [pii] 16 684 10.1001/jama.2016.2374 [doi] [published Online First: 2016/03/24] 17 685 18 19 686 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 23 60 https://mc.manuscriptcentral.com/bmj Page 55 of 63 BMJ

1 2 3 687 FIGURES 4 5 688 6 689 Figure 1: Study flow diagram 7 690 8 9 691 Figure 2: p-values in initial analyses and in re-analyses. The axes are on a log scale. Blue colour 10 692 indicates identical conclusion between the initial analysis and the re-analysis. 11 693 Confidential: For Review Only 12 694 13 14 695 15 696 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 24 60 https://mc.manuscriptcentral.com/bmj BMJ Page 56 of 63

1 2 3 697 TABLES 4 5 6 All BMJ PLOS Medicine

7 (37 studies) (21 studies) (16 studies) 8 Geographical area of the lead country 9 Europe 25 (67 %) 17 (80 %) 8 (50 %) 10 11 AustraliaConfidential: and New Zealand For4 (11 %) Review1 (5 %) Only 3 (19 %) 12 Northern America 3 (8 %) 1 (5 %) 2 (12.5 %) 13 Africa 3 (8 %) 1 (5 %) 2 (12.5 %) 14 East Asia 1 (3 %) 0 (0 %) 1 (6 %) 15 Middle East 1 (3 %) 1 (5 %) 0 (0 %) 16 Type of intervention 17 Drug 20 (54 %) 13 (62 %) 7 (44 %) 18 Device 8 (22 %) 8 (38 %) 0 (0 %) 19 Complex intervention 9 (24 %) 0 (0 %) 9 (56 %) 20 Medical specialty 21 Infectious disease 12 (33 %) 4 (19 %) 8 (50 %) 22 Rheumatology 5 (14 %) 5 (24 %) 0 (0 %) 23 24 Endocrinology/nutrition 4 (11 %) 1 (5 %) 3 (19 %) 25 Paediatrics 3 (8 %) 2 (9 %) 1 (6 %) 26 Mental health / addiction 2 (5 %) 1 (5 %) 1 (6 %) 27 Obstetrics 2 (5 %) 1 (5 %) 1 (6 %) 28 Emergency medicine 2 (5 %) 2 (9 %) 0 (0 %) 29 Geriatrics 2 (5 %) 0 (0 %) 2 (13 %) 30 Other 5 (14 %) 5 (24 %) 0 (0 %) 31 Designs 32 Superiority (Head to head) 18 (49 %) 15 (71 %) 3 (19 %) 33 Superiority (Factorial) 1 (3 %) 1 (5 %) 0 (0 %) 34 Superiority (Clusters) 8 (21 %) 1 (5 %) 7 (43 %) 35 Non-inferiority + Superiority (Head to head) 4 (11 %) 1 (5 %) 3 (19 %) 36 Non-inferiority (Head to head) 6 (16 %) 3 (14 %) 3 (19 %) 37 38 Sample size 432 (213 – 1070)† 221 (159 – 494) 1047 (433 – 2248)† 39 Private sponsorship 40 No 26 (70 %) 15 (71 %) 11 (69 %) 41 Provided the device 1 (3 %) 1 (5 %) 0 (0 %) 42 Provided the intervention 1 (3 %) 0 (0 %) 1 (6 %) 43 Provided the drug 5 (13 %) 1 (5 %) 4 (25 %) 44 Provided the drug and some financial support 2 (5 %) 2 (9 %) 0 (0 %) 45 Provided partial financial support 1 (3 %) 1 (5 %) 0 (0 %) 46 Provided total financial support 1 (3 %) 1 (5 %) 0 (0 %) 47 Statement of availability 48 Ask to contact by e-mail 23 (62 %) 17 (81 %) 6 (38 %) 49 Explain how to retrieve the data (e.g. platform) 9 (24 %) 0 (0 %) 9 (56 %) 50 51 State ‘no additional data available‘ 2 (5 %) 2 (9 %) 0 (0 %) 52 Ask to contact by mail 1 (3 %) 0 (0 %) 1 (6 %) 53 Embargo 1 (3 %) 1 (5 %) 0 (0 %) 54 No statement 1 (3 %) 1 (5 %) 0 (0 %) 55 698 56 57 58 59 25 60 https://mc.manuscriptcentral.com/bmj Page 57 of 63 BMJ

1 2 3 699 Table 1: Characteristics of the included studies. Numbers (and percentages) are presented (rounded 4 700 percentages add up to 100% for each variable). For sample size medians and interquartile ranges are 5 6 701 presented. † For one cluster trial in PLOS Medicine the exact sample size was not reported. 7 8 9 10 11 Confidential: For Review Only 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 26 60 https://mc.manuscriptcentral.com/bmj Page 58 of 63 27 p-value Re- analysis 0.26 0.45 0.095 0.047* 0.328 0.157* 0.239 0.339* 0.778 0.858* 0.170 0.253* 0.805 0.814* . . 0.001< 0.001< 0.777 0.777 p-value Published paper 0.25 0.43 0.075 0.024* 0.326 0.265* 0.219 0.216* 0.775 0.713* 0.199 0.187* 0.804 0.809* . . 0.001< 0.001< 0.759 0.777 Effectsizemeasure Re-analysis [-0.3, 0.4 1.2] [-0.5, 0.3 1.2] [-2.5, 10.2 22.9] [-5.3, 4.3 13.9] [-3.3, 3.6 0.1] [-7.6, 1.0 9.7] [-2.1, 4.0 10.2] [-8.4, 1.0 10.5] [-1.66, -0.18] -0.92 [-1.63, -0.15] -0.89 [0.13, 0.17 0.24] [0.11, 0.16 0.22] [0.942, 1.033 1.133] [0.921, 1.011 1.109] 0.17 [0.07, 0.17 0.46] [0.06, 0.17 0.49] 0.001† < 0.001 < 3 [-10.32, 3 6.73] [-1.73, 1 1.98] [-3.62, 5.19 14.00] [-2.45, 0.01 2.62] 0.58 [-1.99, -0.12 1.75] [-3.15, -0.49 2.18] 0.86 0.13 0.92 0.87 0.64 [1.52, 2.27 3.39] [1.52, 2.27 3.40] 0.001 < 0.001 < 1 [-8.69, 1 6.30] [-6.06, 1.42 8.90] 0.68 0.62 [-35, -58 -74] [-36, -58 -72] 0.001 < 0.001 < [1.52, 2.27 3.38] [1.51, 2.27 3.39] 0.001 < 0.001 < [513, 1037 1560] [513, 1037 1560] 0.001 < [1.52, 1.75 2.03] 0.001 < [1.52, 1.75 2.03] 0.001 < 0.001 < Effectsizemeasure Publishedpaper [-0.3, 0.4 1.2] [-0.5, 0.3 1.2] [-1.3, 10.2 21.7] [-5.1, 4.2 13.8] [-2.6, 3.6 9.7] [-7, 1.0 9.1] [-2.4, 3.8 10.0] [-8.3, 1.0 10.4] [-1.67, -0.16] -0.92 [-1.62, -0.13] -0.87 [0.14, 0.18 0.25] [0.12, 0.17 0.24] [0.942, 1.036 1.140] [0.919, 1.012 1.114] HR, HR, 95%CI MD, 99%CI MD, 99%CI MD, 99%CI IRR, 95 %CI MD, 99%CI RC%, 95%CI IRR, 95 %CI MD, 95%CI RR, 95%CI Effect size (type) MD, 95%CI MD, 95%CI MD, 95%CI MD, 95%CI MD, 95%CI MD, 95%CI MD, 95%CI MD, 95%CI MD, 95%CI MD, 95%CI HR, 95%CI HR, 95%CI RaR, 95%CI RaR, 95%CI BMJ https://mc.manuscriptcentral.com/bmj Primary outcomePrimary measures analogue Visual scale, 2 time Disability Oswestry Index, time 1 Disability Oswestry Index, time 2 leg Average pain, time 1 leg Average pain, time 2 parasite nPCR prevalence, time 1, zone1 parasite nPCR prevalence, time 2, 1 zone parasite nPCR prevalence, time1, 2 zone parasite nPCR prevalence, time 22,zone parasite nPCR prevalence, time 1, 3 zone parasite nPCR prevalence, time 2, 3 zone in Change (unadjusted) weight in Change (adjusted) weight Malaria,pneumonia, diarrhea or (PP) mortality P.vivaxinfection by qPCR, 1 time P.vivaxinfection by qPCR, 2 time attackVertigo attackVertigo Visual analogue Visual scale, 1 time of Burden and hypoxia hyperoxia Malaria,pneumonia, diarrhea or (ITT) mortality step count Daily ofUse antiplatelet, statin, 2≥ BP lowering drugs

Study Study ID Luedtke,2015† Cohen, 2015 Hyttel-SorensenS, 2015 Bousema, 2016 Gysin-Maillart, 2016†† BrieftherapyVS careusual Suicide / attempt Lombard, 2016 attempt Suicide 2016 Polyak, Robinson, 2015 2015 Harris, Adrion, 2016 Selak, 2014 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 28

0.33 0.33 0.08 0.36 0.57 1 0.339 0.31 0.31 0.08 0.36 0.57 1 0.339 -0.33 [-1.01, -0.33 0.34] [0.96, 1.38 1.99] [0.82, 1.24 1.89] [0.65, 1.27 2.45] [0.47, 1.05 2.32] [0.9, 1.0 1.2] -1.2 [-3.2, -1.2 0.8] [-0.17, -0.05 0.08] [-3.2, -1.2 0.8] [-0.17, -0.05 0.08] 0.46 0.22 0.46 0.22 1.64 [1.32, 1.64 2.05] [1.32, 1.65 2.05] 0.001 < 0.001 < [−0.20, 0.003 0.21] [−0.20, 0.003 0.21] 0.97 0.97 -2.2 [-5.6, -2.2 1.2] [-5.6, -2.2 1.2] 0.21 0.21 -0.33 [-0.96, -0.33 0.31] [0.96, 1.38 1.99] [0.82, 1.24 1.89] [0.65, 1.27 2.45] [0.47, 1.05 2.32] [0.9, 1.0 1.2] MD, 95%CI MD, 95%CI RR, 95%CI MD, 95%CI MD, 95%CI MD, 95%CI RR, 95%CI RR, 95%CI RR, 95%CI RR, 95%CI OR, 95%CI BMJ https://mc.manuscriptcentral.com/bmj Change in Change diastolic blood pressure in Change LDL-C elbowin Change intensity pain failure Treatment (ITT) failure Treatment (PP) cause All (ITT) mortality cause All (PP) mortality Neonatal mortality Bloodloss 500> ml of Speed six healingover weeks (cm2/day) Change in systolicin Change bloodpressure

Table 2: Results of the re-analyses re-analyses the of Results 2: Table Intention ITT: ratio; Odds OR: Risk; Relative RR: Ratio; Rate RaR: Ratio; HR: Hazard Percentage; in Change Relative RC%: Differences; Mean MD: Protocol. Per PP: Treat; To analyses. non-adjusted of complement in computed were analyses adjusted of p-values 2016, Bousema, * In Chesterton, 2013 Atukunda, 2014 2015 Paul, Thomas, 2015 2015Hanson, Page 59 of 63 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 Page 60 of 63 29 identified before before identified -

should systematically address data data address systematically should BMJ should provide guidance on the on the guidance provide should should favor data deposition when it is is it when deposition data favor should

should help to develop awareness of a a of awareness develop to help should

Identified issues Identified r the leaves policies sharing Data in still is sharing data for preplanning and prepared Getting of discretion exclusive the to leaves request upon sharing Data corresponding contacting in difficulties some be could There in befound can birth of as date such information identifying Some complete, (e.g. sharable effectively be to need databases Shared data sharing to limited not are practices research Reproducible money, (time, resources necessary the all obtain to researchers, andlegal ethical andservices, tools organizational and technical etc.). compliance, heterogeneity is considerable There units. trials in progress data of types the and data share to procedures different between shared. are that own their share to whether about decision the researchers the under and objectives which for groups, research other to data conditions. and terms which request. upon sharing data limiting authors databases. some labels, descriptive (e.g. meta-data including homogeneous), analysis). of methods tools, processing pre-analysis of description the and methods the of reporting complete code, and materials relevant. also are analyses statistical re-analyses and data-sharing for suggestions) (and challenges identified 3: Some Table 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 Page 61 of 63 BMJ

1 2 Records identified through database searching: 159 3 4 5 BMJ : 120 6 PLOS medicine: 39 Records excluded based on title and abstract: 25 7

Screening Screening 8 9 BMJ : 20 non RCTs 10 Full text considered for eligibility: 134 PLOS medicine: 5 non RCTs 11 12 13 BMJ : 100 14 PLOS medicine: 34 Record excluded based on full text: 72 15 16 Confidential: For ReviewBMJ : 55 no policy,Only 2 re-analyses, 11 secondary analyses 17 Full text meeting inclusion criteria published after the policy: 62 18 Eligibility PLOS medicine: 4 secondary analyses 19 20 21 BMJ : 32 22 PLOS medicine: 30 Record excluded because submitted before the policy: 25 23 24 25 BMJ : 11 26 Full text meeting inclusion criteria submitted after the policy: 37 PLOS medicine: 14 27 28 BMJ : 21

29 Inclusion 30 PLOS medicine: 16 Data not available: 20 31 32 BMJ : 13 33 34 Data available: 17 PLOS medicine: 7 35 36 BMJ : 8 37 38 PLOS medicine: 9 39 40 41 42 Analyses fully reproduced: 14 43 44 BMJ : 7 45 46 PLOS medicine: 7 47 48 49 Analyses not reproduced because of missing information : 1 50 Analysis 51 PLOS medicine: 1 52 53 54 Analyses not fully reproduced but same conclusion: 2 55 56 57 BMJ : 1 58 PLOS medicine: 1 59 60

https://mc.manuscriptcentral.com/bmj BMJ Page 62 of 63 1

1 2 3 0.500 4 Density 5 6 7 8 0.000 9 10 11 1 12 13 14 0.70 15 16 Confidential: For Review Only 17 18 0.40 19 20 21 22 23 24 25 26 27 28 29 0.10 30 31 32 33 34 0.05 35 36 37 38 39 40 41 42 43 (re−analysis) p−Value 44 45 46 0.01 47 48 49 50 51 52 53 54 55 56 57 58 59 60

0.001

https://mc.manuscriptcentral.com/bmj 0.001 0.01 0.05 0.10 0.40 0.70 1 0.000 0.500 1 p−Value (published paper) Density 0.46 0.46 0.95 0.69 0.80 0.77 0.004 p-value Re- analysis Effectsizemeasure Re-analysis [0.75, 1.19 1.88] [0.62, 0.98 1.56] [0.62, 0.91 1.34] [0.78, 1.04 1.37] [0.72, 0.96 1.27] [0.45, 0.63 0.87]

0.46 0.46 0.95 0.69 0.80 0.77 0.01 p-value Publishedpaper Effectsizemeasure Publishedpaper [0.75, 1.19 1.88] [0.62, 0.98 1.56] [0.62, 0.91 1.34] [0.78, 1.04 1.37] [0.72, 0.96 1.27] [0.45, 0.63 0.87]

Effectsize (type) OR, 95%CI OR, 95%CI OR, 95%CI RaR,95%CI RaR,95%CI RR,95%CI BMJ https://mc.manuscriptcentral.com/bmj Primary outcomePrimary measures Being depressed(PHQ-9) Being depressed(PHQ-9) Being depressed(PHQ-9) Falls injuryFalls Incidenceof depressive and/or anxiety disorders

Confidential: For Review Only Treatmentcomparison / condition Bluesthe Beating VS careusual Depression / MoodGYMVScare usual Depression / MoodGYMVS Beating Bluesthe Depression / Programto decreasefall injuries VS Acute careusual / hospitals Steppedcare VScare usual Visually / impaired older adults OR: Odds ratio; RR: Relative Risk; RaR: Rate Ratio; ITT: Intention To Treat; PP: Per Protocol. Per PP: Treat; To Intention ITT: Ratio; Rate RaR: Risk; Relative RR: ratio; Odds OR: Study Study ID Gilbody,2015 Barker,2016 Aa, van der 2015 before. submitted but policy the after BMJ in the published studies ineligible 3 for re-analyses the of 1: Results e-Table

Page 63 of 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60