For Peer Review Only Journal: BMJ Open

BMJ Open BMJ Open: first published as 10.1136/bmjopen-2014-006666 on 5 May 2015. Downloaded from

Evidence-informed recommendations to reduce dissemination bias in clinical research: conclusions from the OPEN (Overcome failure to Publish nEgative fiNdings) project based on an international consensus meeting

For peer review only Journal: BMJ Open

Manuscript ID: bmjopen-2014-006666.R2

Article Type: Research

Date Submitted by the Author: 15-Jan-2015

Complete List of Authors: Meerpohl, Joerg; Medical Center - University of Freiburg, German Cochrane Centre Schell, Lisa; Medical Center - University of Freiburg, German Cochrane Centre Bassler, Dirk; University Hospital Zurich, Department of Neonatology Gallus, Silvano; Istituto di Ricerche Farmacologiche Mario Negri, Department of Epidemiology Kleijnen, Jos; Kleijnen Systematic Reviews Ltd; Maastricht University, School for Public Health and Primary Care Kulig, Michael; Gemeinsamer Bundesausschuss, Medical Consultancy Department http://bmjopen.bmj.com/ La Vecchia, Carlo; University of Milan, Department of Clinical Sciences and Community Health Marusic, Ana; University of Split School of Medicine, Ravaud, Philippe; paris descartes University , Reis, Andreas; World Health Organization, Ethics, Equity, Trade & Human Rights Schmucker, Christine; Medical Center - University of Freiburg, German Cochrane Centre Strech, Daniel; Hannover Medical School Urrutia, Gerard; Iberoamerican Cochrane Centre, on September 23, 2021 by guest. Protected copyright. Wager, Elizabeth; Sideview, Antes, Gerd; Medical Center - University of Freiburg, German Cochrane Centre

Primary Subject Evidence based practice Heading:

Epidemiology, Ethics, Medical publishing and peer review, Research Secondary Subject Heading: methods

EPIDEMIOLOGY, Protocols & guidelines < HEALTH SERVICES Keywords: ADMINISTRATION & MANAGEMENT, JOURNALISM (see Medical Journalism)

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml Page 1 of 41 BMJ Open BMJ Open: first published as 10.1136/bmjopen-2014-006666 on 5 May 2015. Downloaded from 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 For peer review only 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32

33 http://bmjopen.bmj.com/ 34 35 36 37 38 39 40

8 9 Joerg J Meerpohl, deputy director1, Lisa K Schell, researcher1, Dirk Bassler, professor2,3, 10 Silvano Gallus, senior researcher4, Jos Kleijnen, professor5,6, Michael Kulig, researcher7, 11 12 Carlo La Vecchia, professor8, Ana Marušić, professor9, Philippe Ravaud, professor10, 13 11 1 14 Andreas Reis, technical officer , Christine Schmucker, researcher , Daniel Strech, 15 professor12, For Gerard Urrutia, peer researcher 13review, Elizabeth Wager, publicationsonly consultant14, Gerd 16 1 17 Antes, professor and the OPEN project consortium 18 19 20 Authors’ Affiliation: 21 1 German Cochrane Centre, Medical Center – University of Freiburg, Freiburg, Germany 22 2 23 Department of Neonatology, University Hospital Zurich, Zurich, Switzerland 24 3 Center for Pediatric Clinical Studies, University Children’s Hospital Tuebingen, Tuebingen, 25 26 Germany 27 4 Department of Epidemiology, IRCCS Istituto di Ricerche Farmacologiche “Mario Negri”, 28 29 Milan, Italy 30 5 Kleijnen Systematic Reviews Ltd, York, United Kingdom 31 32 6 School for Public Health and Primary Care, Maastricht University, Maastricht, The

33 http://bmjopen.bmj.com/ Netherlands 34 35 7 Medical Consultancy Department, Gemeinsamer Bundesausschuss (GBA), Berlin, 36 Germany 37 38 8 Department of Clinical Sciences and Community Health, University of Milan, Milan, Italy 39 9 40 Department of Research in Biomedicine and Health, University of Split School of Medicine,

41 Split, Croatia on September 23, 2021 by guest. Protected copyright. 42 10 43 INSERM U738 research unit, Paris Descartes University, Paris, France 44 11 Global Health Ethics, Department of Knowledge, Ethics and Research, World Health 45 46 Organization, Geneva, Switzerland 47 12 CELLS – Centre for Ethics and Law in the Life Sciences, Institute of History, Ethics and 48 49 Philosophy of Medicine, Hannover Medical School, Hannover, Germany 50 13 Centro Cochrane IberoamericanoServei d’Epidemiologia Clínica i Salut Pública, Institut 51 52 d’Investigació Biomèdica Sant Pau, Barcelona, Spain 53 14 Sideview, Princes Risborough, United Kingdom 54 55 56 Correspondence to: 57 58 Joerg J Meerpohl: [email protected] 59 60 1 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml Page 3 of 41 BMJ Open BMJ Open: first published as 10.1136/bmjopen-2014-006666 on 5 May 2015. Downloaded from 1 2 3 4 Word count: 5 6 Abstract: 299 7 Text: 3321 8 9 10 11 12 13 14 15 For peer review only 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32

33 http://bmjopen.bmj.com/ 34 35 36 37 38 39 40

41 on September 23, 2021 by guest. Protected copyright. 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 2 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml BMJ Open Page 4 of 41 BMJ Open: first published as 10.1136/bmjopen-2014-006666 on 5 May 2015. Downloaded from 1 2 3 Structured abstract 4 5 6 Background 7 Dissemination bias in clinical research severely impedes informed decisionmaking not only 8 9 for healthcare professionals and patients, but also for funders, research ethics committees, 10 regulatory bodies and other stakeholder groups that make healthrelated decisions. 11 12 Decisions based on incomplete and biased evidence can not only harm people, but may also 13 14 have huge financial implications by wasting resources on ineffective or harmful diagnostic 15 and therapeuticFor measures peer and unnecessary review research. Due only to involvement of multiple 16 17 stakeholders it remains easy for any single group to assign responsibility for resolving the 18 problem to others. 19 20 21 Objective 22 23 To develop evidenceinformed general and targeted recommendations addressing the 24 various stakeholders involved in knowledge generation and dissemination to help overcome 25 26 the problem of dissemination bias on the basis of previously collated evidence. 27 28 29 Methods 30 Based on findings from systematic reviews, document analyses and surveys, we developed 31 32 general and targeted draft recommendations. During a 2day workshop in summer 2013,

33 http://bmjopen.bmj.com/ these draft recommendations were discussed with external experts and key stakeholders 34 35 and refined following a rigorous and transparent methodological approach. 36

37 38 Results 39 40 Four general, overarching recommendations applicable to all or most stakeholder groups

41 were formulated, addressing (1) awareness raising, (2) implementation of targeted on September 23, 2021 by guest. Protected copyright. 42 43 recommendations, (3) trial registration and results posting, and (4) systematic approaches to 44 evidence synthesis. These general recommendations are complemented and specified by 47 45 46 targeted recommendations tailored towards funding agencies, pharmaceutical and device 47 companies, research institutions, researchers (systematic reviewers and trialists), research 48 49 ethics committees, trial registries, journal editors and publishers, regulatory agencies, benefit 50 (health technology) assessment institutions and legislators. 51 52 53 Conclusions 54 55 Despite various recent examples of dissemination bias and several initiatives to reduce it, the 56 problem of dissemination bias has not been resolved. Tailored recommendations based on a 57 58 59 60 3 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml Page 5 of 41 BMJ Open BMJ Open: first published as 10.1136/bmjopen-2014-006666 on 5 May 2015. Downloaded from 1 2 3 comprehensive approach will hopefully help increase transparency in biomedical research by 4 overcoming the failure to disseminate negative findings. 5 6 7 8 9 10 11 12 13 14 15 For peer review only 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32

33 http://bmjopen.bmj.com/ 34 35 36 37 38 39 40

41 on September 23, 2021 by guest. Protected copyright. 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 4 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml BMJ Open Page 6 of 41 BMJ Open: first published as 10.1136/bmjopen-2014-006666 on 5 May 2015. Downloaded from 1 2 3 Article Summary Section 4 5 6 7 Strengths and limitations of this study 8 9 Strengths: 10 11 • Comprehensive set of evidenceinformed recommendations aimed at reducing 12 dissemination bias based on work conducted as part of European Union FP7 project 13 (OPEN project; www.openproject.eu) 14 • Both general and specific recommendations targeted at various key stakeholder 15 For peer review only 16 groups involved in the knowledge generation and translation process 17 • Use of a rigorous and transparent methodology to develop recommendations 18 including at consensus meeting including external experts and stakeholders 19 20 Limitation: 21 22 • Iteration of some recommendations which are not new but have so far not been 23 broadly implemented 24 25 26 27 28 29 30 31 32

33 http://bmjopen.bmj.com/ 34 35 36 37 38 39 40

41 on September 23, 2021 by guest. Protected copyright. 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 5 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml Page 7 of 41 BMJ Open BMJ Open: first published as 10.1136/bmjopen-2014-006666 on 5 May 2015. Downloaded from 1 2 3 Introduction 4 In order to make informed decisions, healthcare practitioners, consumers, public health 5 6 professionals, policymakers, and health and research funding bodies, rely on evidence from 7 clinical research. The generation of evidence is made possible because patients participate 8 9 in clinical studies and accept any researchrelated risks and burdens. To inform medical 10 decisionmaking, demonstrate respect to trial participants, and maintain public trust in clinical 11 12 research it is important that such evidence is made available in an easily accessible and 13 1 14 unbiased way . However, many research findings are either not published at all (an 15 estimated 50%)For 2 or only peer selectively, i.e., review with a bias towards onlyspecific aspects or presenting 16 17 only partial information. Healthcare professionals and policymakers are therefore frequently 18 unable to make decisions based on the entire relevant research evidence 3. This problem 19 4 20 has been called ‘publication bias’ or – as more recently suggested ‘dissemination bias’ to 21 reflect the multiple facets of this problem 5 6. Dissemination bias has recently received 22 7 23 attention due to initiatives such as the AllTrials campaign (www.alltrials.net ) and the RIAT 24 proposal 8, the discussion of oseltamivir in the context of avian influenza 9, and companies’, 25 26 e.g. GlaxoSmithKline’s, announcement to grant access to patientlevel data from all company 27 trials 10. However, general awareness of this problem, and even more importantly, the 28 29 necessity to address and resolve it, has presumably not yet been fully recognized by many 30 stakeholders. 31 32

33 http://bmjopen.bmj.com/ Therefore, the European Commission called for projects to investigate the extent and impact 34 35 of dissemination bias and to develop recommendations to overcome incomplete or selective 36 access to trial results. The EU has committed funds to two projects, OPEN (www.open 37 38 project.eu) 11 and UNCOVER (www.ait.ac.at/uncover), supported within the 7th Framework 39 40 Program. The two projects used different methodologies but aimed to address similar

41 problems, namely the nonpublication – or in a broader sense the nondissemination – of the on September 23, 2021 by guest. Protected copyright. 42 43 complete data of all clinical studies. 44 45 46 We present the results of the 24months OPEN project that ran from November 2011 to 47 October 2013. As an interdisciplinary initiative, it brought together academics and 48 49 stakeholders from across Europe with the aim of developing evidenceinformed 50 recommendations and strategies for overcoming the failure to publish negative research 51 52 findings. 53 One activity of OPEN was to conduct a series of systematic reviews to assess the 54 55 occurrence of nonpublication of research findings and the resulting dissemination bias. 56 These reviews addressed aspects such as existing terminology to describe problems of 57 58 59 60 6 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml BMJ Open Page 8 of 41 BMJ Open: first published as 10.1136/bmjopen-2014-006666 on 5 May 2015. Downloaded from 1 2 3 publication and related biases 5, available methods to detect and measure dissemination bias 4 12, and the extent 13, and impact 14 of the problem of nonpublication of research findings. 5 6 7 Another focus of OPEN was to describe current practices by various players involved in 8 9 knowledge generation and knowledge translation to provide insights about how to avoid or 10 reduce bias due to nonpublication of research findings and to identify ways to change 11 12 practices that contribute to the problem. This has been addressed by assessing and 13 14 evaluating the policies and procedures in place for preventing dissemination bias by the main 15 parties involvedFor in approving, peer funding, conducting,review publishing, onlydisseminating, and assessing 16 17 clinical research. Different work packages of OPEN have surveyed representatives of 18 funding agencies, the pharmaceutical industry, research ethics committees, research 19 15 16 17 20 institutions, researchers , trial registers , medical journals , regulatory agencies, and 21 benefit (health technology) assessment agencies such as NICE in the UK (www.nice.org.uk) 22 23 and IQWiG in Germany (www.iqwig.de). 24 Findings from the OPEN work packages informed a 2day recommendations workshop in 25 26 May 2013 attended by the OPEN project partners, researchers from the UNCOVER project, 27 and selected key stakeholders from across the world. The workshop aimed at developing 28 29 and refining a set of general and targeted recommendations designed to specifically consider 30 the roles that the respective stakeholder groups should play in reducing the incomplete 31 32 dissemination of research findings.

33 http://bmjopen.bmj.com/

34 35 This article first describes the process used to reach a consensus on the recommendations 36 followed by a discussion of the general recommendations, which are aimed at the major 37 38 stakeholders in the knowledge translation process. It then presents 47 concrete 39 40 recommendations complementing the general recommendations aimed at the specific target

41 groups, namely funding agencies, pharmaceutical and device companies, research on September 23, 2021 by guest. Protected copyright. 42 43 institutions, researchers (systematic reviewers and trialists), research ethics committees, trial 44 registries, journal editors and publishers, regulatory agencies, benefit assessment institutions 45 46 and legislators. 47 48 49 50 51 52 53 54 55 56 57 58 59 60 7 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml Page 9 of 41 BMJ Open BMJ Open: first published as 10.1136/bmjopen-2014-006666 on 5 May 2015. Downloaded from 1 2 3 Methods 4 Based on the data gathered in each OPEN work package (i.e., conducted surveys, 5 6 qualitative studies and systematic reviews) (Supplementary Figure) each of these groups 7 formulated 4 to 7 recommendations on how to reduce dissemination bias tailored to their 8 9 respective key group in the knowledge translation process. These recommendations were 10 compiled, ordered, and harmonised to form draft recommendations used as the basis for 11 12 discussion at the OPEN workshop including further international key stakeholders in clinical 13 14 research. The focus of these recommendations was on nondissemination of full trials rather 15 than the selectiveFor reporting peer of outcomes review 2 and/or analyses 18only. This workshop was held in 16 17 Freiburg / Germany, 2324th May 2013, in order to reach a consensus and to finalise the 18 recommendations. The participants were the members of the OPEN Consortium (see 19 20 Appendix 1), members of the OPEN advisory board (see Appendix 2) and 11 stakeholders 21 from relevant key groups (see Appendix 3). Workshop participants discussed the draft 22 23 recommendations in a structured and transparent manner and decided on consistent 24 definitions to be used in the phrasing of its recommendations (e.g., what is meant by ‘clinical 25 26 trial’; see Box 1). 27 The recommendations were also discussed in small groups. To structure the discussion and 28 29 to facilitate the documentation of the judgements underlying the recommendations a decision 30 table was used for rating each individual recommendation (Figure 1). The decision tables 31 32 required each group to assess 5 criteria for each recommendation and to grade it as a

33 http://bmjopen.bmj.com/ “strong recommendation” or only a “recommendation” based on their judgements. The 34 35 following 5 explicit criteria, which are conceptually based on the GRADE approach for 36 developing recommendations 19 20, were assessed for each recommendation: 37 38 39 40 • Confidence in effectiveness – whether the group was confident that if this

41 recommendation were implemented it would effectively reduce dissemination bias. on September 23, 2021 by guest. Protected copyright. 42 43 • Balance of benefits and downsides – whether the group thought that the benefits of 44 this recommendation would clearly outweigh its potential downsides. 45 46 • Likelihood of opposition – whether the group thought that there would be no or only 47 minor opposition to the implementation of this recommendation by relevant key 48 49 groups. 50 • Resource use – whether the group thought that implementing this recommendation 51 52 would require only limited resources (both direct financial costs and indirect 53 54 resources). 55 • Implementability and feasibility – whether the group thought that this recommendation 56 57 could be easily implemented (within a reasonable timeframe) and easily sustained. 58 59 60 8 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml BMJ Open Page 10 of 41 BMJ Open: first published as 10.1136/bmjopen-2014-006666 on 5 May 2015. Downloaded from 1 2 3 For all 5 criteria, the groups were asked to provide a ‘yes or no’ answer and an explanation 4 for their judgement. Regarding the classification as either ‘strong recommendation’ or 5 6 ‘recommendation’, we decided not to use a strict rule (e.g. >= 3 saying yes results in strong 7 recommendation, < 2 saying yes results in recommendation), but rather apply the following 8 9 rule of thumb: if a recommendation was rated for each or most of the criteria as ‘yes’, the 10 recommendation was rated as a ‘strong recommendation'. If only one or a few judgements 11 12 strongly favoured a recommendation, it was classed simply as a recommendation. The 13 14 rationale for this flexible approach lies in the fact that the weight each of these criteria bears 15 on the final classificationFor peer can vary depending review on the recommendation only at hand and/or on the 16 17 extent to which a certain criterion speaks for or against a recommendation. In summary, a 18 judgement based on integration of transparent assessments for these criteria seemed more 19 20 20 appropriate than following a rigid and inflexible rule . 21 22 The outcomes of all small group deliberations were then discussed in the open plenary 23 24 sessions until a consensus on all recommendations was reached. The OPEN Consortium 25 and the other meeting participants then endorsed the full set of recommendations. 26 27 Following our workshop, the list of endorsed recommendations was circulated and refined by 28 members of the OPEN Consortium. As a last step, the final recommendations document was 29 30 sent to all participants of our recommendations workshop for final technical revisions and 31 approval. 32

33 http://bmjopen.bmj.com/ 34 35 36 37 38 39 40

41 on September 23, 2021 by guest. Protected copyright. 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 9 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml Page 11 of 41 BMJ Open BMJ Open: first published as 10.1136/bmjopen-2014-006666 on 5 May 2015. Downloaded from 1 2 3 Results 4 In this report, we first present four general overarching recommendations – aimed at major 5 6 stakeholders in the knowledge generation and translation process – followed by the main 7 results from the OPEN project, i.e. 47 targeted recommendations addressing 11 key 8 9 stakeholder groups. 10

11 12 General recommendations: 13 14 Each general recommendation is followed by an explanation of the rationale and reasons 15 why it was ratedFor as ‘strong’ peer or not based review on our decision table only(see Supplementary File 1 for 16 17 detailed decision tables). 18 19 20 1. All stakeholders should raise awareness about dissemination bias and measures to 21 reduce it (strong recommendation). 22 23 24 Awareness of the problem of dissemination bias is a prerequisite for change. However, some 25 26 of the responses to our surveys indicated that many people still deem this problem not 27 relevant 17. It needs to be emphasised among all stakeholder groups that dissemination bias 28 29 may have serious consequences on the health of people and trust in research. 30 We decided to make this a strong recommendation because we think that the benefits of this 31 32 recommendation clearly outweigh its potential drawbacks and that only limited resources are

33 7 http://bmjopen.bmj.com/ necessary for its implementation (as has recently been shown by the AllTrials campaign ). 34 35 Potential drawbacks of this strong recommendation are that it might not be very effective in 36 the short term and that it might face some opposition. But the Consortium deems the 37 38 recommendation to be highly effective in the long run and recognises that any opposition is 39 40 likely to be due to passivity on the part of the key stakeholder groups and the limited

41 resources available for actively engaging in raising awareness and emphasising its on September 23, 2021 by guest. Protected copyright. 42 43 relevance. Another important aspect that was identified is the need to extend awareness 44 raising activities to patients and their representatives. All patients participating in clinical trials 45 46 should be made aware of the fact that their participation only contributes to scientific 47 progress if the results are published, and that this is often not the case. 48 49 50 2. All stakeholders should disseminate and facilitate the implementation of targeted OPEN 51 52 recommendations as outlined in Table 1 (strong recommendation). 53 54 55 In order for our recommendations to have an impact, they need to be disseminated to as 56 many representatives of the key stakeholder groups as possible. Another prerequisite to 57 58 change is that the stakeholders make a collective effort. The problem of dissemination bias 59 60 10 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml BMJ Open Page 12 of 41 BMJ Open: first published as 10.1136/bmjopen-2014-006666 on 5 May 2015. Downloaded from 1 2 3 needs to be addressed through concerted activities that include several or all stakeholders in 4 order to prevent responsibilities from being shifted from one stakeholder to the other (our 5 6 research showed that many groups believe it to be ‘somebody else’s problem’). 7 This recommendation was rated a strong recommendation. Its drawbacks are the likelihood 8 9 of some opposition and the need for substantial resources for comprehensive dissemination 10 and implementation. While the dissemination of the recommendations alone will not 11 12 effectively reduce dissemination bias, their full or even partial implementation should help 13 14 reduce dissemination bias. In order to be fully effective, tailored dissemination and 15 implementationFor strategies peer will be necessary. review only 16 17 18 3. All stakeholders should promote trial registration and posting of results, and support 19 20 initiatives that facilitate searches across multiple trial registries (strong recommendation). 21 22 23 Clinical trial registers are important sources for identifying completed and ongoing trials. 24 Efforts to promote prospective registration of trials are therefore crucial. The usefulness of 25 26 registers, however, depends on the completeness and comprehensiveness of the included 27 information 21. Accordingly, adequate quality control measures need to be implemented. It 28 29 also became evident that metasearching, the possibility of searching across different 30 registries through a single interface, is key to their usefulness. This is highly relevant 31 32 considering existing language barriers and a lack of awareness of the existence of national

33 http://bmjopen.bmj.com/ registries among users which might result in relevant registers not being included in their 34 35 searches. A platform offering a metasearch for trials across multiple registries would help 36 identifying trials that would otherwise have been missed. 37 38 This recommendation was considered to have long term, rather than immediate, impact. 39 40 Together with the favourable balance of benefits and drawbacks, the knowledge of how

41 limited resources could hinder good implementation led us to make this a strong on September 23, 2021 by guest. Protected copyright. 42 43 recommendation. As a result of the US FDA Amendments Act Section 801 passed in 2007, 44 ClinicalTrials.gov has required (since September 2008) the posting of results of aggregate 45 22 46 data from applicable trials by researchers within 12 months of completion . Once more 47 registries adopt this approach and once platforms allowing metasearching are available, 48 49 clinical trial registers will become essential tools to reduce dissemination bias. 50 51 52 4. All stakeholders should support activities to systematically, with rigorous methodology, 53 synthesise information from studies (strong recommendation). 54 55 56 The work of identifying all studies that address a particular question undertaken by authors of 57 58 systematic reviews helps detect dissemination bias. These efforts include systematic 59 60 11 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml Page 13 of 41 BMJ Open BMJ Open: first published as 10.1136/bmjopen-2014-006666 on 5 May 2015. Downloaded from 1 2 3 literature searches across electronic databases, the searching of trial registries 23 and grey 4 literature as well as contacting experts in the field. Also helpful in detecting dissemination 5 6 bias are, for example, the exploration of selective reporting of outcomes as part of the 7 detailed risk of bias evaluation within systematic reviews 24 and the graphical (e.g., funnel 8 9 plots) and statistical assessment of potentially selective publication of whole studies (e.g., 10 Egger’s test) 25 26. Finally, users of systematic reviews will be made aware of the potential 11 12 impact of dissemination bias on the interpretation of the findings (if these are clearly 13 14 described by the review authors). 15 It was notedFor during the peerdiscussion that review the availability of systematic only reviews in languages 16 17 other than English needs to be increased to achieve wider dissemination and use of these 18 crucial resources for evidencebased decisionmaking 27. Broad acceptance and use of 19 20 systematic reviews will also help to increase awareness about the problem of dissemination 21 bias as suggested in recommendation #1. 22 23 24 25 26 Specific recommendations targeted at key stakeholder groups 27 The 47 specific recommendations targeted at funding agencies, pharmaceutical and device 28 29 industry, research institutions, researchers (systematic reviewers and trialists), research 30 ethics committees, trial registries, journal editors and publishers, regulatory agencies, benefit 31 32 assessment institutions and legislators are presented in Table 1.

33 http://bmjopen.bmj.com/ While they focus on broader concepts such as transparency (trial registration, access to 34 35 study protocols and results data), education and training, the tackling of language barriers, 36 and regulatory and legislative activities to counteract selective dissemination, they do provide 37 38 clear guidance on activities that could and should be undertaken by the respective key 39 40 stakeholder groups. It is obvious that some of these recommendations are also relevant for

41 other stakeholder groups; however, we decided to list them primarily amongst those key on September 23, 2021 by guest. Protected copyright. 42 43 groups where the largest impact can be expected. More detailed rationales as well as related 44 assessments based on our decision tables for these targeted recommendations will be 45 46 discussed more extensively in forthcoming articles and will be made available through the 47 OPEN website (www.openproject.eu). 48 49 50 51 52 53 54 55 56 57 58 59 60 12 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml BMJ Open Page 14 of 41 BMJ Open: first published as 10.1136/bmjopen-2014-006666 on 5 May 2015. Downloaded from 1 2 3 Discussion 4 The OPEN project included a broad range of key stakeholders across all relevant areas of 5 6 the knowledge generation and knowledge translation process. The research that was 7 undertaken within the separate work packages not only shed light on current measures in 8 9 place to limit dissemination bias but also gathered the views of key stakeholders on the 10 possible barriers to transparency in medical research. Thus, based on the evidence collated 11 12 throughout the first phase of OPEN, a draft set of preliminary recommendations for each 13 14 stakeholder group was developed. These were then discussed with the OPEN Consortium, 15 the advisoryFor board of OPEN,peer and with review selected external key only stakeholders to capture the 16 17 different perspectives. 18 Instead of solely describing aspects of the general problem of dissemination bias, in addition 19 20 to the general recommendations, we developed several targeted and concrete 21 recommendations, which specifically address each key stakeholder group and thus, as a 22 23 whole, offer a holistic approach to address the problem. Further, due to the discussions held 24 on topics such as possible opposition, resources required, and implementability, we were 25 26 able to go beyond generic recommendations. The detailed rationales and explanations for 27 these recommendations will be made available soon in separate articles and through the 28 29 OPEN website (www.openproject.eu). By means of this dissemination strategy, we hope to 30 expedite the widespread dissemination and uptake of OPEN recommendations by the 31 32 relevant key groups and institutions and to thus initiate the implementation of more adequate

33 http://bmjopen.bmj.com/ mechanisms to reduce and prevent dissemination bias in the long run. 34 35 36 Although we developed separate sets of recommendations for industry and public funding 37 38 agencies, it is important to highlight that we agreed that the same ethical considerations 39 40 apply to all research funders. After all, the interests of patients and the public in general 28 41 should always override commercial or other interests . During workshop discussions it was on September 23, 2021 by guest. Protected copyright. 42 43 noted that the tendency of forprofit companies and also (to some extent) other stakeholder 44 groups such as medical licensing bodies, research institutions or policymakers and 45 46 legislators to place the protection of commercial interest over the public interest in access to 47 data from clinical trials does not seem to be proportional 29 30. Despite the fact that companies 48 49 are investing large sums of money to develop new treatments, the group agreed that 50 intellectual and commercial property rights are overall less important than basic human rights 51 52 with regard to information and health. 53 54 55 As already raised, it is obvious that some of our recommendations are relevant for more than 56 one stakeholder group. The recommendations given for benefit assessment institutions, for 57 58 example, are also applicable for regulators; however, they are given specifically for benefit 59 60 13 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml Page 15 of 41 BMJ Open BMJ Open: first published as 10.1136/bmjopen-2014-006666 on 5 May 2015. Downloaded from 1 2 3 assessment agencies since work conducted within OPEN revealed that cooperation between 4 agencies, and the transparency of methodology used, could particularly be improved within 5 6 this key group. Similarly, recommendations with respect to data sharing, e.g. given for 7 research ethics committees and legislators, are again also relevant for other key groups. It 8 9 needs to be kept in mind, though, that data sharing was not the focus of the OPEN project. 10

11 12 We hope that the results of our project will complement and support ongoing and future 13 14 activities aiming for more transparency in clinical trial results dissemination. As outlined in the 15 OPEN paperFor on our conceptual peer approach review to the problem ofonly selective dissemination and 16 17 resulting dissemination bias (Bassler et al., submitted for publication), the OPEN project is 18 unique in that it aimed to determine which stakeholder groups should be held responsible 19 20 and for identifying what they can do to reduce and eliminate this multifaceted and 21 multidimensional problem. In addition, all stakeholders who are willing to address the 22 23 problem can now be provided with clear and concrete recommendations for action. 24 25 26 While the OPEN project aims mainly at supporting changes that will result in greater 27 transparency in the knowledge generation and knowledge translation process in the future, it 28 29 is clear that transparency and free access to data also need to be established for past trials 30 of current treatments. As recently raised by Doshi et al. 8 as well as by the AllTrials campaign 31 32 (www.alltrials.net), the complete and unbiased reporting of findings from past trials is equally

33 http://bmjopen.bmj.com/ important for securing access to the full evidence. The broader issue of waste in research, 34 35 first introduced by Chalmers & Glasziou in 2009 31, has recently been explored in a series of 36 articles including among others waste due to inaccessible research 32, and waste due to 37 38 incomplete or unusable research reports 33. 39 40

41 The recommendations from the OPEN project coincide with a number of important decisions on September 23, 2021 by guest. Protected copyright. 42 43 at the European Medicines Agency (EMA) and the European Parliament, namely the 44 finalisation and implementation of the EMA’s data sharing policy. EMA and the European 45 46 Parliament, which approved the new European clinical trials regulation on April 2nd, 2014, 47 have received praise for their leadership on facilitating access to clinical study reports 34. 48 49 However, the recently released draft terms for use of these reports included several 50 important restrictions such as strict confidentiality, access to registered users and onscreen 51 35 36 52 viewing only . Certainly, this would impede the widespread and most effective use of 53 these important data sources. Criticism was voiced by academia to which EMA responded 54 37 55 with a press release announcing more userfriendly amendments . As of now, it is unclear 56 what the final policy will look like. 57 58 59 60 14 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml BMJ Open Page 16 of 41 BMJ Open: first published as 10.1136/bmjopen-2014-006666 on 5 May 2015. Downloaded from 1 2 3 We hope that the findings from the OPEN project will contribute to these decisions and 4 implementation processes, emphasising once more the needs of academia and the public for 5 6 unrestricted access to an unbiased body of evidence of complete results of all clinical trials. 7

8 9 10 11 12 13 14 15 For peer review only 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32

33 http://bmjopen.bmj.com/ 34 35 36 37 38 39 40

41 on September 23, 2021 by guest. Protected copyright. 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 15 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml BMJ Open: first published as 10.1136/bmjopen-2014-006666 on 5 May 2015. Downloaded from 16 BMJ Open http://bmjopen.bmj.com/ on September 23, 2021 by guest. Protected copyright. For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml For peer review only part of the funding until a project’s results are adequately disseminated (recommendation). (recommendation). disseminated adequately are results project’s a until funding the of part project’s final report, but instead is followed up until all agreed data have been disseminated (recommendation). (recommendation). disseminated been have data agreed all up until followed is instead but report, final project’s disseminated, regardless of the nature of findings, in all funding contracts (strong recommendation). recommendation). (strong contracts funding all in findings, of nature the of regardless disseminated, participant (strong recommendation). recommendation). (strong participant clinical trials publicly accessible (strong recommendation). recommendation). (strong accessible trials publicly clinical order to keep an accurate record of funded projects and publication outcomes (recommendation). (recommendation). outcomes publication and projects funded of record accurate keepan to order proposals (strong recommendation). recommendation). (strong proposals all calls for proposals (strong recommendation). recommendation). (strong proposals for calls all committees) available upon the publication/dissemination of results (strong recommendation). recommendation). (strong results of publication/dissemination upon the available committees) 2.3 2.3 Pharmaceutical and devices companies should make their trial protocols + amendments (as submitted to research ethics 2.2 2.2 Pharmaceutical and devices companies should register all clinical trials in a public registry before the recruitment of the first 2.1 2.1 Pharmaceutical and medical device companies should make their policies concerning the dissemination of methods and results of Pharmaceutical and device companies companies device and Pharmaceutical 1.6 1.6 in database and on of disseminated their all accessible grants were create awarded a results should how publicly agencies Funding 1.5 1.5 Funding agencies should consider providing incentives for researchers who disseminate their results, or, alternatively, withhold a 1.4 1.4 Funding agencies should implement measures to ensure that the evaluation process of funded projects does not end with the 1.3 1.3 Funding agencies should include the requirement for grantees to explicitly declare that the results of funded research will be 1.2 1.2 Funding agencies should include the requirement for grantees to provide a dissemination plan for funded projects in all calls for 1.1 1.1 in results research of dissemination the for requirement and the bias dissemination a on statement include should agencies Funding Funding agencies agencies Funding Table 1: Targeted recommendations by key stakeholder stakeholder group key by recommendations Targeted 1: Table No. Recommendation Page 17 of 41 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 BMJ Open: first published as 10.1136/bmjopen-2014-006666 on 5 May 2015. Downloaded from 17 Page 18 of 41

Page 19 of 41 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 BMJ Open: first published as 10.1136/bmjopen-2014-006666 on 5 May 2015. Downloaded from 19 Page 20 of 41 BMJ Open http://bmjopen.bmj.com/ on September 23, 2021 by guest. Protected copyright. For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml For peer review only further (unpublished) data and to foster data sharing (strong recommendation). recommendation). (strong sharing data foster to and data (unpublished) further achieve better transparency and understanding (strong recommendation). recommendation). (strong understanding and transparency better achieve requirements are sanctioned (strong recommendation). recommendation). (strong sanctioned are requirements of of a trial is made in available the same registry that the trial was registered in, in a timely fashion (i.e.,1 year after study completion recommendation). (strong database EU the in registered forstudies all or inactivity) approved approved by the research ethics committee, including the potential protocol amendments, is submitted and made publicly available recommendation). (strong document as searchable a registered in an EU database which is publicly accessible (strong recommendation). recommendation). (strong accessible publicly is which database EU an in registered devices and biologicals (strong recommendation). recommendation). (strong biologicals and devices reporting the same findings (strong recommendation). recommendation). (strong findings same the reporting registration (recommendation). (recommendation). registration 10.3 10.3 Benefit assessment institutions should use the full evidence base available for an intervention for their assessments (strong 10.2 10.2 Benefit assessment institutions should aim for a higher degree of collaboration between institutions to facilitate the detection of 10.1 10.1 Benefit assessment institutions should make their methods and processes of benefit assessment publicly available in order to Benefit assessment institutions institutions assessment Benefit 9.5 9.5 Responsible authorities (such as EMA for drugs) should ensure that trial sponsors failing to comply with such result submission 9.4 9.4 Responsible authorities (such as EMA for drugs) should mandate that the full report including all results (e.g., clinical study report) 9.3 9.3 full an mandate in the a that, registration database, protocol fortrial’s upon EU (such should as authorities drugs) EMA Responsible 9.2 9.2 Responsible authorities (such as the EMA for drugs) should mandate that all clinical trials in humans falling under their remit are 9.1 9.1 Regulation of pharmaceutical products should be extended to cover other therapeutic and diagnostic agents such as medical Regulatory agencies agencies Regulatory 8.5 8.5 Journals should for check of publication redundant results by using software textmatching and peer asking reviewers about papers 8.4 8.4 Journal editors should publish editorials and commentaries about the problem of dissemination bias and the benefits of trial 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 BMJ Open: first published as 10.1136/bmjopen-2014-006666 on 5 May 2015. Downloaded from 20 BMJ Open http://bmjopen.bmj.com/ on September 23, 2021 by guest. Protected copyright. For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml For peer review only decisionmaking process (strong recommendation). recommendation). (strong process decisionmaking (registered in the EU database) (strong recommendation). recommendation). (strong database) EU the in (registered information (strong recommendation). recommendation). (strong information results (disclosure of full protocols and full clinical study reports): reports): study clinical full and protocols full of (disclosure results a. b. evidence requested all submit to manufacturers for obligation a legal c. databases to EMA access public recommendation) (strong reports full study and to protocols access public deemed incomplete (e.g., no adequate proof of benefit based on incomplete data set) (strong recommendation). recommendation). (strong set) data incomplete on based benefit of proof adequate no (e.g., incomplete deemed recommendation). recommendation). Organization WorldHealth WHO = Union; European = EU Agency; Medicines European = EMA 11.4 11.4 Legislators should ensure that the raw data (anonymized individual patient data) are made publicly available for all clinical studies 11.3 11.3 Legislators should institute a legal obligation for manufacturers to submit all data and other required information for the formal 11.2 11.2 proprietary confidential/ commercially NOT are public the and of patients tohealth the related data that all ensure should Legislators 11.1 11.1 recommendation). (strong mandatory humans in trials of clinical registration prospective make should Legislators Legislators Legislators 10.5 10.5 Benefit assessment institutions should request from legislators the following items which will allow the consideration of all study 10.4 10.4 Benefit assessment institutions should specify their course of action if they find that the evidence base for an assessment is

Page 21 of 41 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 BMJ Open Page 22 of 41 BMJ Open: first published as 10.1136/bmjopen-2014-006666 on 5 May 2015. Downloaded from 1 2 3 Box 1: Definition of a clinical trial 4 5 Clinical trial: 6 A clinical trial is any research study that prospectively assigns human participants or 7 groups of humans to one or more healthrelated interventions to evaluate the effects 8 on health outcomes. Clinical trials may also be referred to as interventional trials. 9 Interventions include but are not restricted to drugs, cells and other biological 10 products, surgical procedures, radiologic procedures, devices, behavioural 11 treatments, processofcare changes, preventive care, etc. This definition includes 12 39 13 Phase I to Phase IV trials . 14 15 For peer review only 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32

33 http://bmjopen.bmj.com/ 34 35 36 37 38 39 40

41 on September 23, 2021 by guest. Protected copyright. 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml 21 Page 23 of 41 BMJ Open BMJ Open: first published as 10.1136/bmjopen-2014-006666 on 5 May 2015. Downloaded from 1 2 3 Details of contributors: 4 JM and LS drafted this manuscript based on the discussions and results of the OPEN 5 6 recommendations workshop. All named authors were involved in the work packages that led 7 to the development of the draft recommendations. All had actively participated in the OPEN 8 9 workshop and repeatedly commented and revised the draft manuscript; they are listed in 10 alphabetical order. All authors approved the final manuscript. People listed as part of the 11 12 OPEN project consortium contributed to the various work packages of the OPEN project on 13 14 the basis of which the draft recommendations were developed. 15 For peer review only 16 17 Competing interest statement: 18 All authors have completed the ICMJE uniform disclosure form at 19 20 www.icmje.org/coi_disclosure.pdf and declare: the OPEN project has received funding from 21 the European Union under grant agreement n° 285453; Davina Ghersi reports being the 22 23 Chair of the WHO Advisory Panel on the International Clinical Trials Registry Platform; Jos 24 Kleijnen reports grants from Amgen, GSK, Sanofi, Biomarin, MSD, Roche and Bayer outside 25 26 the submitted work; Joerg Meerpohl reports being a member of the WHO Advisory Panel on 27 the International Clinical Trials Registry Platform; Andreas Reis is a staff member of the 28 29 World Health Organization. The author alone is responsible for the views expressed in this 30 article and they do not necessarily represent the decisions, policy or views of the World 31 32 Health Organization; Elizabeth Wager reports training and consultancy activities for various

33 http://bmjopen.bmj.com/ companies and publishers and training activities for various universities; Robert Wolff reports 34 35 grants from Amgen, GSK, Sanofi, Biomarin, MSD, Roche and Bayer outside the submitted 36 work; no financial relationships with any organisations that might have an interest in the 37 38 submitted work in the previous three years reported by other authors; no other relationships 39 40 or activities that could appear to have influenced the submitted work.

41 on September 23, 2021 by guest. Protected copyright. 42 43 Acknowledgements: 44 We would like to thank all participants of the OPEN workshop (Appendix 3) for the critical 45 46 discussion and great contributions to the final OPEN recommendations. Special thanks goes 47 to the OPEN Advisory Board (Appendix 2) for their input throughout the project. 48 49 50 Data sharing statement: 51 52 No additional data are available. 53 54 55 Copyright: 56 The Corresponding Author has the right to grant on behalf of all authors and does grant on 57 58 behalf of all authors, an exclusive licence (or non exclusive for government employees) on a 59 60

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml 22 BMJ Open Page 24 of 41 BMJ Open: first published as 10.1136/bmjopen-2014-006666 on 5 May 2015. Downloaded from 1 2 3 worldwide basis to the BMJ Publishing Group Ltd, and its Licensees to permit this article (if 4 accepted) to be published in BMJ editions and any other BMJPGL products and to exploit all 5 6 subsidiary rights, as set out in our licence (bmj.com/advice/copyright.shtml). 7

8 9 Transparency declaration: 10 The lead author (Joerg J Meerpohl) affirms that the manuscript is an honest, accurate, and 11 12 transparent account of the work being reported. No important aspects have been omitted. 13 14 15 Ethical approval:For peer review only 16 17 Not required. 18 19 20 Details of funding and statement of the independence of researchers from funders: 21 The research leading to these results has received funding from the European Union 22 23 Seventh Framework Programme (FP7/2007-2013) under grant agreement n° 285453. The 24 funder had no influence in the development of the recommendations and/or the writing of the 25 26 manuscript. 27 28 29 30 31 32

33 http://bmjopen.bmj.com/ 34 35 36 37 38 39 40

41 on September 23, 2021 by guest. Protected copyright. 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml 23 Page 25 of 41 BMJ Open BMJ Open: first published as 10.1136/bmjopen-2014-006666 on 5 May 2015. Downloaded from 1 2 3 References 4 5 6 1. Strech D. Normative arguments and new solutions for the unbiased registration and publication of 7 clinical trials. J Clin Epidemiol 2012;65(3):276-81. 8 2. Dwan K, Gamble C, Williamson PR, et al. Systematic review of the empirical evidence of study 9 publication bias and outcome reporting bias - an updated review. PLoS One 10 2013;8(7):e66844. 11 3. Antes G, Chalmers I. Under-reporting of clinical trials is unethical. Lancet 2003;361(9362):978-9. 12 4. Dickersin K. The existence of publication bias and risk factors for its occurrence. JAMA 13 1990;263(10):1385-89. 14 15 5. Muller KF, Briel M, D'Amario A, et al. Defining publication bias: protocol for a systematic review of For peer review only2 16 highly cited articles and proposal for a new framework. Syst Rev 2013; :34. 17 6. Song F, Parekh S, Hooper L, et al. Dissemination and publication of research findings: an updated 18 review of related biases. Health Technol Assess 2010;14(8):iii, ix-xi, 1-193. 19 7. Brown T. It's time for alltrials registered and reported. Cochrane Database Syst Rev 20 2013;4:ED000057. 21 8. Doshi P, Dickersin K, Healy D, et al. Restoring invisible and abandoned trials: a call for people to 22 publish the findings. BMJ 2013;346:f2865. 23 9. Groves T. What does oseltamivir do, and how will we know? BMJ 2013;347:f4687. 24 10. Nisen P, Rockhold F. Access to patient-level data from GlaxoSmithKline clinical trials. N Engl J Med 25 2013;369(5):475-8. 26 11. Littmann J, Antes G, Strech D, et al. To overcome failure to publish negative findings: The OPEN 27 28 project. Maturitas 2013. 29 12. Mueller KF, Meerpohl JJ, Briel M, et al. Detecting, quantifying and adjusting for publication bias in 30 meta-analyses: protocol of a systematic review on methods. Syst Rev 2013;2:60. 31 13. Schmucker C, Bluemle A, Briel M, et al. A protocol for a systematic review on the impact of 32 unpublished studies and studies published in the gray literature in meta-analyses. Syst Rev

33 2013;2:24. http://bmjopen.bmj.com/ 34 14. Portalupi S, von Elm E, Schmucker C, et al. Protocol for a systematic review on the extent of non- 35 publication of research studies and associated study characteristics. Syst Rev 2013;2:2. 36 15. Malički M, Marušić A. Is there a solution to publication bias? Researchers call for changes in 37 dissemination of clinical research results. Journal of Clinical Epidemiology 2014;67(10):1103- 38 10. 39 16. Viergever RF, Karam G, Reis A, et al. The quality of registration of clinical trials: still a problem. 40 PLoS One 2014;9(1):e84727.

41 on September 23, 2021 by guest. Protected copyright. 42 17. Wager E, Williams P, Project Overcome failure to Publish nEgative fiNdings C. "Hardly worth the 43 effort"? Medical journals' policies and their editors' and publishers' views on trial registration 44 and publication bias: quantitative and qualitative study. BMJ 2013;347:f5248. 45 18. Dwan K, Altman DG, Clarke M, et al. Evidence for the selective reporting of analyses and 46 discrepancies in clinical trials: a systematic review of cohort studies of clinical trials. PLoS 47 Med 2014;11(6):e1001666. 48 19. Andrews J, Guyatt G, Oxman AD, et al. GRADE guidelines: 14. Going from evidence to 49 recommendations: the significance and presentation of recommendations. J Clin Epidemiol 50 2013;66(7):719-25. 51 20. Andrews JC, Schunemann HJ, Oxman AD, et al. GRADE guidelines: 15. Going from evidence to 52 recommendation-determinants of a recommendation's direction and strength. J Clin 53 Epidemiol 2013;66(7):726-35. 54 55 21. Zarin DA, Tse T, Williams RJ, et al. The ClinicalTrials.gov results database--update and key issues. 364 56 N Engl J Med 2011; (9):852-60. 57 22. Hirsch L. Trial registration and results disclosure: impact of US legislation on sponsors, 58 investigators, and medical journal editors. Curr Med Res Opin 2008;24(6):1683-9. 59 60

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml 24 BMJ Open Page 26 of 41 BMJ Open: first published as 10.1136/bmjopen-2014-006666 on 5 May 2015. Downloaded from 1 2 3 23. Jones C, Keil L, Weaver M, et al. Clinical trials registries are under-utilized in the conduct of 4 systematic reviews: a cross-sectional analysis. Systematic Reviews 2014;3(1):126. 5 24. Higgins JPT, Altman DG, Sterne JAC. Chapter 8: Assessing risk of bias in included studies. In: 6 Higgins JPT, Green S, eds. Cochrane Handbook for Systematic Reviews of Interventions 7 Version 510 (updated March 2011), 2011. 8 25. Higgins JP, Green S. Cochrane Handbook for Systematic Reviews of Interventions; Version 5.1.0, 9 available from http://www.cochrane-handbook.org. Chichester, UK John Wiley & Sons, Ltd 10 2011. 11 26. Sterne JA, Sutton AJ, Ioannidis JP, et al. Recommendations for examining and interpreting funnel 12 343 13 plot asymmetry in meta-analyses of randomised controlled trials. BMJ 2011; :d4002. 14 27. Ofori-Adjei D, Antes G, Tharyan P, et al. Have online international medical journals made local 15 journalsFor obsolete? peerPLoS Med 2006; 3review(8):e359. only 16 28. Gotzsche PC. Why we need easy access to all data from all clinical trials and how to accomplish it. 17 Trials 2011;12:249. 18 29. Strech D, Littmann J. Lack of proportionality. Seven specifications of public interest that override 19 post-approval commercial interests on limited access to clinical data. Trials 2012;13(1):100. 20 30. Gotzsche PC, Jorgensen AW. Opening up data at the European Medicines Agency. BMJ 21 2011;342:d2686. 22 31. Chalmers I, Glasziou P. Avoidable waste in the production and reporting of research evidence. 23 Lancet 2009;374(9683):86-9. 24 32. Chan AW, Song F, Vickers A, et al. Increasing value and reducing waste: addressing inaccessible 25 383 26 research. Lancet 2014; (9913):257-66. 27 33. Glasziou P, Altman DG, Bossuyt P, et al. Reducing waste from incomplete or unusable reports of 28 biomedical research. Lancet 2014;383(9913):267-76. 29 34. Regulation of the European Parliament and of the Council on clinical trials on medicinal products 30 for human use, and repealing Directive 2001/20/EC: accessed on 05.06.2014 at: 31 http://www.europarl.europa.eu/sides/getDoc.do?pubRef=-//EP//NONSGML+AMD+A7-2013- 32 0208+291-291+DOC+PDF+V0//EN 2014. 33 35. Torjesen I. European drug agency backtracks on plan to give researchers access to clinical trial http://bmjopen.bmj.com/ 34 reports. BMJ 2014;348. 35 36. Goldacre B, Godlee F, Heneghan C, et al. Open letter: European Medicines Agency should remove 36 barriers to access clinical trial data. BMJ 2014;348:g3768. 37 37. EMA. European Medicines Agency agrees policy on publication of clinical trial data with more 38 39 user-friendly amendments: accessed on 04.08.2014 at: 40 http://www.ema.europa.eu/docs/en_GB/document_library/Press_release/2014/06/WC500 168342.pdf, 2014. 41 on September 23, 2021 by guest. Protected copyright. 42 38. Eden J, Levit L, Berg AO, et al. Finding what works in health care: Standards for systematic 43 reviews: National Academies Press, 2011. 44 39. World Health Organization. International Clinical Trials Registry Platform (ICTRP). Secondary 45 International Clinical Trials Registry Platform (ICTRP). http://www.who.int/ictrp/en/. 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml 25 Page 27 of 41 BMJ Open BMJ Open: first published as 10.1136/bmjopen-2014-006666 on 5 May 2015. Downloaded from 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 For peer review only 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32

33 http://bmjopen.bmj.com/ 34 35 36 37 38 39 40

41 on September 23, 2021 by guest. Protected copyright. 42 43 44 45 46 47 209x297mm (300 x 300 DPI) 48 49 50 51 52 53 54 55 56 57 58 59 60 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml BMJ Open Page 28 of 41

1 2 3 Appendix 1: OPEN Consortium 4 BMJ Open: first published as 10.1136/bmjopen-2014-006666 on 5 May 2015. Downloaded from 5 Contributor Participating Institution 6 7 8 Antes, Gerd German Cochrane Centre, Medical Center - University of Freiburg, 9 10 Freiburg, Germany 11 12 Bassler, Dirk Center for Pediatric Clinical Studies, University Medical Center 13 Tuebingen, Germany 14 15 Bertelè, Vittorio IRCCS – Istituto di Ricerche Farmacologiche “Mario Negri”, Milan, Italy 16 17 Bonfill, Xavier For peerThe Clinical Epidemiology review & Public Healthonly Department at the Hospital de 18 la Santa Creu i Sant Pau, Spain 19 20 Bouesseau, Marie- World Health Organization, Geneva, Switzerland 21 22 Charlotte 23 24 Boutron, Isabelle INSERM U738 research unit, Paris Descartes University, Paris, France 25 26 Briel, Matthias Center for Pediatric Clinical Studies, University Medical Center 27 Tuebingen, Germany & Basel Institute for Clinical Epidemiology and 28 Biostatistics, University Hospital of Basel, Switzerland 29 30 Gallus, Silvano Department of Epidemiology, IRCCS - Istituto di Ricerche Farmacologiche 31 “Mario Negri”, Milan, Italy 32 33 34 Garattini, Silvio IRCCS - Istituto di Ricerche Farmacologiche “Mario Negri”, Milan, Italy 35 36 Ghersi, Davina University of Sydney, Australia

37 http://bmjopen.bmj.com/ 38 Karam, Ghassan World Health Organization, Geneva, Switzerland 39 40 Kleijnen, Jos Kleijnen Systematic Reviews Ltd., York, United Kingdom & School for 41 Public Health and Primary Care, Maastricht University, Maastricht, The 42 Netherlands 43 44 Kulig, Michael Federal Joint Committee, Berlin, Germany 45 on September 23, 2021 by guest. Protected copyright. 46 La Vecchia, Carlo Department of Clinical Sciences and Community Health, University of 47 48 Milan, Milan Italy 49 50 Lang, Britta German Cochrane Centre, Medical Center - University of Freiburg, 51 Freiburg, Germany 52 53 Littmann, Jasper CELLS (Centre for Ethics and Law in Life Sciences), Hannover Medical 54 Scholl, Hannover, Germany 55 56 Malicki, Mario University of Split School of Medicine, Split, Croatia 57 58 Marusic, Ana University of Split School of Medicine, Split, Croatia 59 60 Meerpohl, Joerg German Cochrane Centre, Medical Center - University of Freiburg, Freiburg, Germany

1 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml Page 29 of 41 BMJ Open

1 2 3 Mueller, Katharina Center for Pediatric Clinical Studies, University Medical Center 4 BMJ Open: first published as 10.1136/bmjopen-2014-006666 on 5 May 2015. Downloaded from 5 Felicitas Tuebingen, Germany 6 7 Murisic, Bojana Department of Epidemiology, IRCCS - Istituto di Ricerche Farmacologiche 8 “Mario Negri”, Milan, Italy 9 10 Nolting, Alexandra Federal Joint Committee, Berlin, Germany 11 12 Pardo, Hector The Clinical Epidemiology & Public Health Department at the Hospital de 13 la Santa Creu i Sant Pau, Spain 14 15 Perleth, Matthias Federal Joint Committee, Berlin, Germany 16 17 For peer review only Ravaud, Philippe INSERM U738 research unit, Paris Descartes University, Paris, France 18 19 20 Reis, Andreas World Health Organization, Geneva, Switzerland 21 22 Schell, Lisa German Cochrane Centre, Medical Center - University of Freiburg, 23 Freiburg, Germany 24 25 Schmucker, Christine German Cochrane Centre, Medical Center - University of Freiburg, 26 Freiburg, Germany 27 28 Schwarzer, Guido Institute for Medical Biometry and Statistics, Medical Center – University 29 of Freiburg, Freiburg, Germany 30 31 32 Strech, Daniel CELLS (Centre for Ethics and Law in Life Sciences), Hannover Medical 33 Scholl, Hannover, Germany 34 35 Trinquart, Ludovic INSERM U738 research unit, Paris Descartes University, Paris, France 36

37 Urrútia, Gerard The Clinical Epidemiology & Public Health Department at the Hospital de http://bmjopen.bmj.com/ 38 la Santa Creu i Sant Pau, Spain 39 40 von Elm, Erik German Cochrane Centre, Medical Center - University of Freiburg, 41 Freiburg, Germany & Cochrane Switzerland, IUMSP, University Hospital 42 43 Lausanne, Lausanne, Switzerland 44 45 Wager, Elizabeth Sideview, Princes Risborough, United Kingdom on September 23, 2021 by guest. Protected copyright. 46 47 Wolff, Robert Kleijnen Systematic Reviews Ltd., York, United Kingdom 48 49 50 51 52 53 54 55 56 57 58 59 60

2 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml BMJ Open Page 30 of 41

1 2 3 Appendix 2: OPEN Advisory Board 4 BMJ Open: first published as 10.1136/bmjopen-2014-006666 on 5 May 2015. Downloaded from 5 Member Institution 6 7 8 Altman, Doug Centre for Statistics in Medicine, University of Oxford, United Kingdom 9 10 Chan, An-Wen Women's College Research Institute, University of Toronto, Canada 11 12 13 Dickersin, Kay Johns Hopkins Bloomberg School of Public Health, United States 14 15 Weber, Wim British Medical Journal 16 17 Song, Fujian For peerSchool of Medicine, review Health Policy and only Practice, University of East Anglia, 18 United Kingdom 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36

37 http://bmjopen.bmj.com/ 38 39 40 41 42 43 44 45 on September 23, 2021 by guest. Protected copyright. 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

1 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml Page 31 of 41 BMJ Open

1 2 3 Appendix 3: Participants at OPEN Meeting 4 BMJ Open: first published as 10.1136/bmjopen-2014-006666 on 5 May 2015. Downloaded from 5 Participant Institution 6 7 8 Antes, Gerd German Cochrane Centre, Institute of Medical Biometry and Medical 9 10 Informatics, University Medical Center, Freiburg, Germany 11 12 Bangdiwala, Shrikant UNC Gillings School of Global Public Health, Chapel Hill, United States 13 14 Bassler, Dirk Center for Pediatric Clinical Studies, University Medical Center 15 Tuebingen, Germany 16 17 Boutron, IsabelleFor peerINSERM U738 reviewresearch unit, Paris Descartesonly University, Paris, France 18 19 Collazo, Eduardo Lopez Instituto de Salud Carlos III, Barcelona, Spain 20 21 Doppelfeld, Elmar European Network of Research Ethics Committees 22 23 Dwan, Kerry University of Liverpool, United Kingdom 24 25 Frost, Robert GlaxoSmithKline plc, London, United Kingdom 26 27 28 Gallus, Silvano Department of Epidemiology, IRCCS - Istituto di Ricerche Farmacologiche 29 “Mario Negri”, Milan, Italy 30 31 Ghersi, Davina University of Sydney, Australia 32 33 Gøtzsche, Peter Nordic Cochrane Centre, Copenhagen, Denmark 34 35 Hrynaszkiewicz, Iain Faculty of 1000 Ltd, United Kingdom 36

37 Jena, Susanne German Register of Clinical Trials, Freiburg, Germany http://bmjopen.bmj.com/ 38 39 Karam, Ghassan World Health Organization, Geneva, Switzerland 40 41 Kienegger, Manuela Danube University Krems, Austria 42 43 Kirkham, Jamie University of Liverpool, United Kingdom 44 45 on September 23, 2021 by guest. Protected copyright. 46 Kleijnen, Jos Kleijnen Systematic Reviews Ltd., York, United Kingdom & School for 47 Public Health and Primary Care, Maastricht University, Maastricht, The 48 Netherlands 49 50 La Vecchia, Carlo Department of Clinical Sciences and Community Health, University of 51 Milan, Milan Italy 52 53 Malicki, Mario University of Split, School of Medicine, Split, Croatia 54 55 Marusic, Ana University of Split, School of Medicine, Split, Croatia 56 57 58 Meerpohl, Joerg German Cochrane Centre, Institute of Medical Biometry and Medical 59 Informatics, University Medical Center, Freiburg, Germany 60 Mueller, Katharina Center for Pediatric Clinical Studies, University Medical Center Felicitas Tuebingen, Germany

1 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml BMJ Open Page 32 of 41

1 2 3 Panteli, Dimitra Technical University of Berlin, Germany 4 BMJ Open: first published as 10.1136/bmjopen-2014-006666 on 5 May 2015. Downloaded from 5 Pardo, Hector The Clinical Epidemiology & Public Health Department at the Hospital de 6 7 la Santa Creu i Sant Pau, Spain 8 9 Schell, Lisa German Cochrane Centre, Institute of Medical Biometry and Medical 10 Informatics, University Medical Center, Freiburg, Germany 11 12 Schmucker, Christine German Cochrane Centre, Institute of Medical Biometry and Medical 13 Informatics, University Medical Center, Freiburg, Germany 14 15 Strech, Daniel CELLS (Centre for Ethics and Law in Life Sciences), Hannover Medical 16 17 For peerScholl, Hannover, review Germany only 18 19 Trinquart, Ludovic INSERM U738 research unit, Paris Descartes University, Paris, France 20 21 Urrútia, Gerard The Clinical Epidemiology & Public Health Department at the Hospital de 22 la Santa Creu i Sant Pau, Spain 23 24 Van Noord, Megan Danube University Krems, Austria 25 26 Wager, Elizabeth Sideview, Princes Risborough, United Kingdom 27 28 Wahler, Steffen Correvio International Sàrl, Geneva, Switzerland 29 30 Weber, Wim British Medical Journal 31 32 Wieseler, Beate Institute for Quality and Efficiency in Healthcare, Cologne, Germany 33 34 35 36

37 http://bmjopen.bmj.com/ 38 39 40 41 42 43 44 45 on September 23, 2021 by guest. Protected copyright. 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

2 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml Page 33 of 41 BMJ Open

1 2 3 Supplementary Figure: Work package (WP) structure of the OPEN project 4 BMJ Open: first published as 10.1136/bmjopen-2014-006666 on 5 May 2015. Downloaded from 5 6 7 8 9 WP!14*Project*management* 10 11 12 13 ! ! 14 ! ! ! ! ! ! ! ! 15 WP!1! WP!6! WP!11!! WP!2! WP!3!! WP4!! WP5!! WP!7!! WP!8!! WP!9!! WP!10!! Systema(c* Evaluate** Impact** 16 !Evaluate* Evaluate** Evaluate** Evaluate** Evaluate* Evaluate** Evaluate** Evaluate** *reviews* researchers,** of** funding** (pharma4* ethic** research** trial** regulatory* *to** For peer reviewbiomedical** only impact** 17 authors** publica(on** agencies* ceu(cal)** boards* ins(tu(ons* registers* journals* *agencies* on*beneﬁt** assess* and** bias*on** 18 industry* assessment** **publica4* reviewers* network* * process* 19 (on*bias* *metanalysis* 20 21 22 23 24 ! WP!12!! 25 Evalua(on*of*ﬁndings** 26 and*formula(on*of** recommenda(ons* 27 28 29 ! 30 WP!13!! 31 Dissemina(on** and*exploita(on*of*results* 32 33 34 35 36

37 http://bmjopen.bmj.com/ 38 39 40 41 42 43 44 45 on September 23, 2021 by guest. Protected copyright. 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml BMJ Open: first published as 10.1136/bmjopen-2014-006666 on 5 May 2015. Downloaded from

BMJ Open Page 34 of 41

1 Supplementary File: Decision tables for general recommendations 2 3 4 5 Recommendation #1: 6 7 We strongly recommend that 8 9 10 11 For peer review only 12 all stakeholders in the knowledge generation and translation process 13 14 15 16 raise awareness about dissemination bias (publication bias) and measures to reduce it. http://bmjopen.bmj.com/ 17 18 19 Category Judgment Remark 20 21 22 Likely limited effectiveness in the short run, but potentially high High confidence in effectiveness Yes Nox 23 effectiveness in the long run

24 on September 23, 2021 by guest. Protected copyright. 25 26 Benefits clearly outweigh downsides x Yes No No obvious downsides; potential to have important impact 27 28 Probably no strong opposition, but likelihood of “opposition” 29 Low likelihood of strong opposition x Yes No 30 due to passiveness/limited resources of key stakeholders x 31 32 At first only limited resources required; however, relevant 33 Limited resources required for 34 x Yes No resources would be required if this recommendation is implementation 35 implemented on a large scale 36 37 38 Easy to start raising awareness, but comprehensive strategy 39 Easily implementable Yes No 40 more difficult to implement 41 42 43 44 45 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 BMJ Open: first published as 10.1136/bmjopen-2014-006666 on 5 May 2015. Downloaded from

Page 35 of 41 BMJ Open

1 Strength of recommendation: x Strong recommendation 2 3

4 Recommendation 5 6 7 8 9 Remark: 10 11 For peerNot very concrete recommendation; room for interpretation. review only 12 13 Should also be extended to patients who should be made aware of dissemination bias. 14 15

16 http://bmjopen.bmj.com/ 17 18 19 20 21 22 23

24 on September 23, 2021 by guest. Protected copyright. 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 BMJ Open: first published as 10.1136/bmjopen-2014-006666 on 5 May 2015. Downloaded from

BMJ Open Page 36 of 41

1 2 3 Recommendation #2: 4 5 We strongly recommend that 6 7 8 9 10 all stakeholders in the knowledge generation and translation process 11 For peer review only 12 13 14 disseminate and facilitate implementation of targeted OPEN recommendations amongst key stakeholders of their respective group. 15

16 http://bmjopen.bmj.com/ 17 Category Judgment Remark 18 19 20 If OPEN recommendations were fully disseminated and 21 High confidence in effectiveness x Yes No implemented, dissemination bias would most likely be reduced 22 23 effectively.

24 on September 23, 2021 by guest. Protected copyright. 25 Although some specific recommendations might not have a 26 27 Benefits clearly outweigh downsides x Yes No unambiguous beneficial balance, overall benefits would clearly 28 outweigh downsides. 29 30 31 Likely that some key stakeholder groups or some key 32 stakeholders within a group are opposed and not going to 33 Low likelihood of strong opposition Yes Nox 34 disseminate and/or implement some or all OPEN 35 recommendations. 36 37 38 Limited resources required for A comprehensive dissemination and implementation plan would Yes No x 39 implementation likely require substantial resources. 40 41 42 Easily implementable Yes Nox Likely tailored dissemination and implementation strategies 43 44 45 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 BMJ Open: first published as 10.1136/bmjopen-2014-006666 on 5 May 2015. Downloaded from

Page 37 of 41 BMJ Open

1 would be required including monitoring efforts on 2 3 implementation and effectiveness. 4 5 Strength of recommendation: Strong recommendation 6 x 7

8 Recommendation 9 10 11 For peer review only 12 13 Remark: 14 15 Strong recommendation despite concerns around opposition, resources and 16 http://bmjopen.bmj.com/ 17 implementability; but obviously key recommendation amongst OPEN recommendations. 18 Without dissemination and implementation plans, OPEN recommendations will be not 19 helpful in reducing dissemination bias. 20 21 22 23

BMJ Open Page 38 of 41

1 2 3 Recommendation #3: 4 5 We strongly recommend that 6 7 8 9 10 all stakeholders in the knowledge generation and translation process 11 For peer review only 12 13 14 promote trial registration and support initiatives that facilitate searches across multiple trial registries. 15

16 http://bmjopen.bmj.com/ 17 Category Judgment Remark 18 19 20 Promotion and support in itself likely not yet highly effective in 21 reducing dissemination bias, but if respective initiatives are 22 High confidence in effectiveness x Yes No x 23 implemented and maintained relevant impact likely in the

24 longer run. on September 23, 2021 by guest. Protected copyright. 25 26 27 No obvious downsides in promoting trial registration and Benefits clearly outweigh downsides x Yes No 28 supporting such initiatives. 29 30 31 Low likelihood of strong opposition Yes Nox Some opposition by groups active in this field quite likely. 32 33 Promotion and political support possible without relevant 34 35 Limited resources required for resources. Technical consequences not fully clear; relevant x Yes No 36 implementation resources likely required to establish, promote, expand and 37 38 maintain such initiatives. 39 40 No serious barrier to both political and financial support was 41 Easily implementable x Yes No 42 identified. 43 44 45 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 BMJ Open: first published as 10.1136/bmjopen-2014-006666 on 5 May 2015. Downloaded from

Page 39 of 41 BMJ Open

1 Strength of recommendation: x Strong recommendation 2 3

4 Recommendation 5 6 7 8 9 Remark: 10 11 For peerWide range of support is possible. Although not very concrete, strong recommendation in review only 12 13 favour of supporting and thereby hopefully establishing/maintaining in the longer run 14 respective initiatives (one example of such a platform is the WHO International Clinical 15 Trials Registry Platform (ICRTP)). 16 http://bmjopen.bmj.com/ 17 18 Likely need for alternative initiatives to avoid monopoly?! 19 20 21 22 In the future need for similar initiatives that allow simultaneous access to results 23 databases.

BMJ Open Page 40 of 41

1 2 3 Recommendation #4: 4 5 We strongly recommend that 6 7 8 9 10 all stakeholders in the knowledge generation and translation process 11 For peer review only 12 13 14 support activities to systematically synthesize information from studies with rigorous methodology. 15

16 http://bmjopen.bmj.com/ 17 Category Judgment Remark 18 19 20 Support in itself likely not yet highly effective in reducing 21 dissemination bias, but if respective activities rigorously identify 22 High confidence in effectiveness Yes Nox 23 and integrate unpublished trials likely relevant impact in the

24 longer run. on September 23, 2021 by guest. Protected copyright. 25 26 27 Risk of spurious conclusions if activities based on very biased 28 subset of published trials; overall benefits seem to clearly 29 Benefits clearly outweigh downsides x Yes No 30 outweigh downsides, in particular if efforts are undertaken to 31 consider all conducted research for a specific question. 32 33 34 No strong opposition to general support likely; however, Low likelihood of strong opposition x Yes No 35 financial support more likely to be difficult to establish. 36 37 38 Political support possible without relevant resources. Relevant Limited resources required for 39 x Yes No resources likely required to increase and maintain efforts to 40 implementation 41 actually conduct and regularly update all these activities (SRs). 42 43 44 45 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 BMJ Open: first published as 10.1136/bmjopen-2014-006666 on 5 May 2015. Downloaded from

Page 41 of 41 BMJ Open

1 No serious barrier to both political and financial support was 2 Easily implementable x Yes No 3 identified. 4 5 Strength of recommendation: Strong recommendation 6 x 7

8 Recommendation 9 10 11 For peer review only 12 13 Remark: 14 15 Wide range of support is possible. Although not very concrete, strong recommendation in 16 http://bmjopen.bmj.com/ 17 favour of supporting and thereby increasing such activities. However, relevant resources 18 will be required in the long run to conduct and update regularly relevant systematic 19 reviews. 20 21 22 Dissemination strategies need to be developed for non-English speaking countries to 23 increase effectiveness. These will require further resources.

BMJ Open Page 42 of 41

1 2 3 4 5 6 7 8 9 10 11 For peer review only 12 13 14 15

16 http://bmjopen.bmj.com/ 17 18 19 20 21 22 23

24 on September 23, 2021 by guest. Protected copyright. 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 BMJ Open BMJ Open: first published as 10.1136/bmjopen-2014-006666 on 5 May 2015. Downloaded from

For peer review only

Journal: BMJ Open

Manuscript ID: bmjopen-2014-006666.R1

Article Type: Research

Date Submitted by the Author: 05-Jan-2015

Complete List of Authors: Meerpohl, Joerg; Medical Center - University of Freiburg, German Cochrane Centre Schell, Lisa; Medical Center - University of Freiburg, German Cochrane Centre Bassler, Dirk; University Hospital Zurich, Department of Neonatology Gallus, Silvano; Istituto di Ricerche Farmacologiche Mario Negri, Department of Epidemiology Kleijnen, Jos; Maastricht University, School for Public Health and Primary Care; Kleijnen Systematic Reviews Ltd Kulig, Michael; Gemeinsamer Bundesausschuss, Medical Consultancy Department http://bmjopen.bmj.com/ La Vecchia, Carlo; University of Milan, Department of Clinical Sciences and Community Health Marusic, Ana; University of Split School of Medicine, Ravaud, Philippe; paris descartes University , Reis, Andreas; World Health Organization, Ethics, Equity, Trade & Human Rights Schmucker, Christine; Medical Center - University of Freiburg, German Cochrane Centre Strech, Daniel; Hannover Medical School Urrutia, Gerard; Iberoamerican Cochrane Centre, on September 23, 2021 by guest. Protected copyright. Wager, Elizabeth; Sideview, Antes, Gerd; Medical Center - University of Freiburg, German Cochrane Centre

Primary Subject Evidence based practice Heading:

Epidemiology, Ethics, Medical publishing and peer review, Research Secondary Subject Heading: methods

EPIDEMIOLOGY, Protocols & guidelines < HEALTH SERVICES Keywords: ADMINISTRATION & MANAGEMENT, JOURNALISM (see Medical Journalism)

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml Page 1 of 40 BMJ Open BMJ Open: first published as 10.1136/bmjopen-2014-006666 on 5 May 2015. Downloaded from 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 For peer review only 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32

33 http://bmjopen.bmj.com/ 34 35 36 37 38 39 40

41 Split, Croatia on September 23, 2021 by guest. Protected copyright. 42 10 43 INSERM U738 research unit, Paris Descartes University, Paris, France 44 11 Global Health Ethics, Department of Knowledge, Ethics and Research, World Health 45 46 Organization, Geneva, Switzerland 47 12 CELLS – Centre for Ethics and Law in the Life Sciences, Institute of History, Ethics and 48 49 Philosophy of Medicine, Hannover Medical School, Hannover, Germany 50 13 Centro Cochrane IberoamericanoServei d’Epidemiologia Clínica i Salut Pública, Institut 51 52 d’Investigació Biomèdica Sant Pau, Barcelona, Spain 53 14 Sideview, Princes Risborough, United Kingdom 54 55 56 Correspondence to: 57 58 Joerg J Meerpohl: [email protected] 59 60 1 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml Page 3 of 40 BMJ Open BMJ Open: first published as 10.1136/bmjopen-2014-006666 on 5 May 2015. Downloaded from 1 2 3 4 Word count: 5 6 Abstract: 299 7 Text: 3321 8 9 10 11 12 13 14 15 For peer review only 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32

33 http://bmjopen.bmj.com/ 34 35 36 37 38 39 40

41 on September 23, 2021 by guest. Protected copyright. 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 2 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml BMJ Open Page 4 of 40 BMJ Open: first published as 10.1136/bmjopen-2014-006666 on 5 May 2015. Downloaded from 1 2 3 Structured abstract 4 5 6 Background 7 Dissemination bias in clinical research severely impedes informed decisionmaking not only 8 9 for healthcare professionals and patients, but also for funders, research ethics committees, 10 regulatory bodies and other stakeholder groups that make healthrelated decisions. 11 12 Decisions based on incomplete and biased evidence can not only harm people, but may also 13 14 have huge financial implications by wasting resources on ineffective or harmful diagnostic 15 and therapeuticFor measures peer and unnecessary review research. Due only to involvement of multiple 16 17 stakeholders it remains easy for any single group to assign responsibility for resolving the 18 problem to others. 19 20 21 Objective 22 23 To develop evidenceinformed general and targeted recommendations addressing the 24 various stakeholders involved in knowledge generation and dissemination to help overcome 25 26 the problem of dissemination bias on the basis of previously collated evidence. 27 28 29 Methods 30 Based on findings from systematic reviews, document analyses and surveys, we developed 31 32 general and targeted draft recommendations. During a 2day workshop in summer 2013,

33 http://bmjopen.bmj.com/ these draft recommendations were discussed with external experts and key stakeholders 34 35 and refined following a rigorous and transparent methodological approach. 36

37 38 Results 39 40 Four general, overarching recommendations applicable to all or most stakeholder groups

41 were formulated, addressing (1) awareness raising, (2) implementation of targeted on September 23, 2021 by guest. Protected copyright. 42 43 recommendations, (3) trial registration and results posting, and (4) systematic approaches to 44 evidence synthesis. These general recommendations are complemented and specified by 47 45 46 targeted recommendations tailored towards funding agencies, pharmaceutical and device 47 companies, research institutions, researchers (systematic reviewers and trialists), research 48 49 ethics committees, trial registries, journal editors and publishers, regulatory agencies, benefit 50 assessment institutions and legislators. 51 52 53 Conclusions 54 55 Despite various recent examples of dissemination bias and several initiatives to reduce it, the 56 problem of dissemination bias has not been resolved. Tailored recommendations based on a 57 58 59 60 3 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml Page 5 of 40 BMJ Open BMJ Open: first published as 10.1136/bmjopen-2014-006666 on 5 May 2015. Downloaded from 1 2 3 comprehensive approach will hopefully help increase transparency in biomedical research by 4 overcoming the failure to disseminate negative findings. 5 6 7 8 9 10 11 12 13 14 15 For peer review only 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32

33 http://bmjopen.bmj.com/ 34 35 36 37 38 39 40

41 on September 23, 2021 by guest. Protected copyright. 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 4 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml BMJ Open Page 6 of 40 BMJ Open: first published as 10.1136/bmjopen-2014-006666 on 5 May 2015. Downloaded from 1 2 3 Article Summary Section 4 5 6 7 Strengths and limitations of this study 8 9 Strengths: 10 11 • Comprehensive set of evidenceinformed recommendations aimed at reducing 12 dissemination bias based on work conducted as part of European Union FP7 project 13 (OPEN project; www.openproject.eu) 14 • Both general and specific recommendations targeted at various key stakeholder 15 For peer review only 16 groups involved in the knowledge generation and translation process 17 • Use of a rigorous and transparent methodology to develop recommendations 18 including at consensus meeting including external experts and stakeholders 19 20 Limitation: 21 22 • Iteration of some recommendations which are not new but have so far not been 23 broadly implemented 24 25 26 27 28 29 30 31 32

33 http://bmjopen.bmj.com/ 34 35 36 37 38 39 40

41 on September 23, 2021 by guest. Protected copyright. 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 5 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml Page 7 of 40 BMJ Open BMJ Open: first published as 10.1136/bmjopen-2014-006666 on 5 May 2015. Downloaded from 1 2 3 Introduction 4 In order to make informed decisions, healthcare practitioners, consumers, public health 5 6 professionals, policymakers, and health and research funding bodies, rely on evidence from 7 clinical research. The generation of evidence is made possible because patients participate 8 9 in clinical studies and accept any researchrelated risks and burdens. To inform medical 10 decisionmaking, demonstrate respect to trial participants, and maintain public trust in clinical 11 12 research it is important that such evidence is made available in an easily accessible and 13 1 14 unbiased way . However, many research findings are either not published at all (an 15 estimated 50%)For 2 or only peer selectively, i.e., review with a bias towards onlyspecific aspects or presenting 16 17 only partial information. Healthcare professionals and policymakers are therefore frequently 18 unable to make decisions based on the entire relevant research evidence 3. This problem 19 4 20 has been called ‘publication bias’ or – as more recently suggested ‘dissemination bias’ to 21 reflect the multiple facets of this problem 5 6. Dissemination bias has recently received 22 7 23 attention due to initiatives such as the AllTrials campaign (www.alltrials.net ) and the RIAT 24 proposal 8, the discussion of oseltamivir in the context of avian influenza 9, and companies’, 25 26 e.g. GlaxoSmithKline’s, announcement to grant access to patientlevel data from all company 27 trials 10. However, general awareness of this problem, and even more importantly, the 28 29 necessity to address and resolve it, has presumably not yet been fully recognized by many 30 stakeholders. 31 32

41 problems, namely the nonpublication – or in a broader sense the nondissemination – of the on September 23, 2021 by guest. Protected copyright. 42 43 complete data of all clinical studies. 44 45 46 We present the results of the 24months OPEN project that ran from November 2011 to 47 October 2013. As an interdisciplinary initiative, it brought together academics and 48 49 stakeholders from across Europe with the aim of developing evidenceinformed 50 recommendations and strategies for overcoming the failure to publish negative research 51 52 findings. 53 One activity of OPEN was to conduct a series of systematic reviews to assess the 54 55 occurrence of nonpublication of research findings and the resulting dissemination bias. 56 These reviews addressed aspects such as existing terminology to describe problems of 57 58 59 60 6 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml BMJ Open Page 8 of 40 BMJ Open: first published as 10.1136/bmjopen-2014-006666 on 5 May 2015. Downloaded from 1 2 3 publication and related biases 5, available methods to detect and measure dissemination bias 4 12, and the extent 13, and impact 14 of the problem of nonpublication of research findings. 5 6 7 Another focus of OPEN was to describe current practices by various players involved in 8 9 knowledge generation and knowledge translation to provide insights about how to avoid or 10 reduce bias due to nonpublication of research findings and to identify ways to change 11 12 practices that contribute to the problem. This has been addressed by assessing and 13 14 evaluating the policies and procedures in place for preventing dissemination bias by the main 15 parties involvedFor in approving, peer funding, conducting,review publishing, onlydisseminating, and assessing 16 17 clinical research. Different work packages of OPEN have surveyed representatives of 18 funding agencies, the pharmaceutical industry, research ethics committees, research 19 15 16 17 20 institutions, researchers , trial registers , medical journals , regulatory agencies, and 21 benefit (health technology) assessment agencies such as NICE in the UK (www.nice.org.uk) 22 23 and IQWiG in Germany (www.iqwig.de). 24 Findings from the OPEN work packages informed a 2day recommendations workshop in 25 26 May 2013 attended by the OPEN project partners, researchers from the UNCOVER project, 27 and selected key stakeholders from across the world. The workshop aimed at developing 28 29 and refining a set of general and targeted recommendations designed to specifically consider 30 the roles that the respective stakeholder groups should play in reducing the incomplete 31 32 dissemination of research findings.

33 http://bmjopen.bmj.com/

41 groups, namely funding agencies, pharmaceutical and device companies, research on September 23, 2021 by guest. Protected copyright. 42 43 institutions, researchers (systematic reviewers and trialists), research ethics committees, trial 44 registries, journal editors and publishers, regulatory agencies, benefit assessment institutions 45 46 and legislators. 47 48 49 50 51 52 53 54 55 56 57 58 59 60 7 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml Page 9 of 40 BMJ Open BMJ Open: first published as 10.1136/bmjopen-2014-006666 on 5 May 2015. Downloaded from 1 2 3 Methods 4 Based on the data gathered in each OPEN work package (i.e., conducted surveys, 5 6 qualitative studies and systematic reviews) (Figure S1) each of these groups formulated 4 to 7 7 recommendations on how to reduce dissemination bias tailored to their respective key 8 9 group in the knowledge translation process. These recommendations were compiled, 10 ordered, and harmonised to form draft recommendations used as the basis for discussion at 11 12 the OPEN workshop including further international key stakeholders in clinical research. The 13 14 focus of these recommendations was on nondissemination of full trials rather than the 15 selective reportingFor of outcomes peer 2 and/or review analyses 18. This workshop only was held in Freiburg / 16 17 Germany, 2324th May 2013, in order to reach a consensus and to finalise the 18 recommendations. The participants were the members of the OPEN Consortium (see 19 20 Appendix 1), members of the OPEN advisory board (see Appendix 2) and 11 stakeholders 21 from relevant key groups (see Appendix 3). Workshop participants discussed the draft 22 23 recommendations in a structured and transparent manner and decided on consistent 24 definitions to be used in the phrasing of its recommendations (e.g., what is meant by ‘clinical 25 26 trial’; see Box 1). 27 The recommendations were also discussed in small groups. To structure the discussion and 28 29 to facilitate the documentation of the judgements underlying the recommendations a decision 30 table was used for rating each individual recommendation (Figure 2). The decision tables 31 32 required each group to assess 5 criteria for each recommendation and to grade it as a

41 recommendation were implemented it would effectively reduce dissemination bias. on September 23, 2021 by guest. Protected copyright. 42 43 • Balance of benefits and downsides – whether the group thought that the benefits of 44 this recommendation would clearly outweigh its potential downsides. 45 46 • Likelihood of opposition – whether the group thought that there would be no or only 47 minor opposition to the implementation of this recommendation by relevant key 48 49 groups. 50 • Resource use – whether the group thought that implementing this recommendation 51 52 would require only limited resources (both direct financial costs and indirect 53 54 resources). 55 • Implementability and feasibility – whether the group thought that this recommendation 56 57 could be easily implemented (within a reasonable timeframe) and easily sustained. 58 59 60 8 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml BMJ Open Page 10 of 40 BMJ Open: first published as 10.1136/bmjopen-2014-006666 on 5 May 2015. Downloaded from 1 2 3 For all 5 criteria, the groups were asked to provide a ‘yes or no’ answer and an explanation 4 for their judgement. Regarding the classification as either ‘strong recommendation’ or 5 6 ‘recommendation’, we decided not to use a strict rule (e.g. >= 3 saying yes results in strong 7 recommendation, < 2 saying yes results in recommendation), but rather apply the following 8 9 rule of thumb: if a recommendation was rated for each or most of the criteria as ‘yes’, the 10 recommendation was rated as a ‘strong recommendation'. If only one or a few judgements 11 12 strongly favoured a recommendation, it was classed simply as a recommendation. The 13 14 rationale for this flexible approach lies in the fact that the weight each of these criteria bears 15 on the final classificationFor peer can vary depending review on the recommendation only at hand and/or on the 16 17 extent to which a certain criterion speaks for or against a recommendation. In summary, a 18 judgement based on integration of transparent assessments for these criteria seemed more 19 20 20 appropriate than following a rigid and inflexible rule . 21 22 The outcomes of all small group deliberations were then discussed in the open plenary 23 24 sessions until a consensus on all recommendations was reached. The OPEN Consortium 25 and the other meeting participants then endorsed the full set of recommendations. 26 27 Following our workshop, the list of endorsed recommendations was circulated and refined by 28 members of the OPEN Consortium. As a last step, the final recommendations document was 29 30 sent to all participants of our recommendations workshop for final technical revisions and 31 approval. 32

33 http://bmjopen.bmj.com/ 34 35 36 37 38 39 40

41 on September 23, 2021 by guest. Protected copyright. 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 9 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml Page 11 of 40 BMJ Open BMJ Open: first published as 10.1136/bmjopen-2014-006666 on 5 May 2015. Downloaded from 1 2 3 Results 4 In this report, we first present four general overarching recommendations – aimed at major 5 6 stakeholders in the knowledge generation and translation process – followed by the main 7 results from the OPEN project, i.e. 47 targeted recommendations addressing 11 key 8 9 stakeholder groups. 10

11 12 General recommendations: 13 14 Each general recommendation is followed by an explanation of the rationale and reasons 15 why it was ratedFor as ‘strong’ peer or not based review on our decision table only (see Appendix 4 for detailed 16 17 decision tables). 18 19 20 1. All stakeholders should raise awareness about dissemination bias and measures to 21 reduce it (strong recommendation). 22 23 24 Awareness of the problem of dissemination bias is a prerequisite for change. However, some 25 26 of the responses to our surveys indicated that many people still deem this problem not 27 relevant 17. It needs to be emphasised among all stakeholder groups that dissemination bias 28 29 may have serious consequences on the health of people and trust in research. 30 We decided to make this a strong recommendation because we think that the benefits of this 31 32 recommendation clearly outweigh its potential drawbacks and that only limited resources are

41 resources available for actively engaging in raising awareness and emphasising its on September 23, 2021 by guest. Protected copyright. 42 43 relevance. Another important aspect that was identified is the need to extend awareness 44 raising activities to patients and their representatives. All patients participating in clinical trials 45 46 should be made aware of the fact that their participation only contributes to scientific 47 progress if the results are published, and that this is often not the case. 48 49 50 2. All stakeholders should disseminate and facilitate the implementation of targeted OPEN 51 52 recommendations as outlined in Table 1 (strong recommendation). 53 54 55 In order for our recommendations to have an impact, they need to be disseminated to as 56 many representatives of the key stakeholder groups as possible. Another prerequisite to 57 58 change is that the stakeholders make a collective effort. The problem of dissemination bias 59 60 10 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml BMJ Open Page 12 of 40 BMJ Open: first published as 10.1136/bmjopen-2014-006666 on 5 May 2015. Downloaded from 1 2 3 needs to be addressed through concerted activities that include several or all stakeholders in 4 order to prevent responsibilities from being shifted from one stakeholder to the other (our 5 6 research showed that many groups believe it to be ‘somebody else’s problem’). 7 This recommendation was rated a strong recommendation. Its drawbacks are the likelihood 8 9 of some opposition and the need for substantial resources for comprehensive dissemination 10 and implementation. While the dissemination of the recommendations alone will not 11 12 effectively reduce dissemination bias, their full or even partial implementation should help 13 14 reduce dissemination bias. In order to be fully effective, tailored dissemination and 15 implementationFor strategies peer will be necessary. review only 16 17 18 3. All stakeholders should promote trial registration and posting of results, and support 19 20 initiatives that facilitate searches across multiple trial registries (strong recommendation). 21 22 23 Clinical trial registers are important sources for identifying completed and ongoing trials. 24 Efforts to promote prospective registration of trials are therefore crucial. The usefulness of 25 26 registers, however, depends on the completeness and comprehensiveness of the included 27 information 21. Accordingly, adequate quality control measures need to be implemented. It 28 29 also became evident that metasearching, the possibility of searching across different 30 registries through a single interface, is key to their usefulness. This is highly relevant 31 32 considering existing language barriers and a lack of awareness of the existence of national

41 limited resources could hinder good implementation led us to make this a strong on September 23, 2021 by guest. Protected copyright. 42 43 recommendation. As a result of the US FDA Amendments Act Section 801 passed in 2007, 44 ClinicalTrials.gov has required (since September 2008) the posting of results of aggregate 45 22 46 data from applicable trials by researchers within 12 months of completion . Once more 47 registries adopt this approach and once platforms allowing metasearching are available, 48 49 clinical trial registers will become essential tools to reduce dissemination bias. 50 51 52 4. All stakeholders should support activities to systematically, with rigorous methodology, 53 synthesise information from studies (strong recommendation). 54 55 56 The work of identifying all studies that address a particular question undertaken by authors of 57 58 systematic reviews helps detect dissemination bias. These efforts include systematic 59 60 11 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml Page 13 of 40 BMJ Open BMJ Open: first published as 10.1136/bmjopen-2014-006666 on 5 May 2015. Downloaded from 1 2 3 literature searches across electronic databases, the searching of trial registries 23 and grey 4 literature as well as contacting experts in the field. Also helpful in detecting dissemination 5 6 bias are, for example, the exploration of selective reporting of outcomes as part of the 7 detailed risk of bias evaluation within systematic reviews 24 and the graphical (e.g., funnel 8 9 plots) and statistical assessment of potentially selective publication of whole studies (e.g., 10 Egger’s test) 25 26. Finally, users of systematic reviews will be made aware of the potential 11 12 impact of dissemination bias on the interpretation of the findings (if these are clearly 13 14 described by the review authors). 15 It was notedFor during the peerdiscussion that review the availability of systematic only reviews in languages 16 17 other than English needs to be increased to achieve wider dissemination and use of these 18 crucial resources for evidencebased decisionmaking 27. Broad acceptance and use of 19 20 systematic reviews will also help to increase awareness about the problem of dissemination 21 bias as suggested in recommendation #1. 22 23 24 25 26 Specific recommendations targeted at key stakeholder groups 27 The 47 specific recommendations targeted at funding agencies, pharmaceutical and device 28 29 industry, research institutions, researchers (systematic reviewers and trialists), research 30 ethics committees, trial registries, journal editors and publishers, regulatory agencies, benefit 31 32 assessment institutions and legislators are presented in Table 1.

41 other stakeholder groups; however, we decided to list them primarily amongst those key on September 23, 2021 by guest. Protected copyright. 42 43 groups where the largest impact can be expected. More detailed rationales as well as related 44 assessments based on our decision tables for these targeted recommendations will be 45 46 discussed more extensively in forthcoming articles and will be made available through the 47 OPEN website (www.openproject.eu). 48 49 50 51 52 53 54 55 56 57 58 59 60 12 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml BMJ Open Page 14 of 40 BMJ Open: first published as 10.1136/bmjopen-2014-006666 on 5 May 2015. Downloaded from 1 2 3 Discussion 4 The OPEN project included a broad range of key stakeholders across all relevant areas of 5 6 the knowledge generation and knowledge translation process. The research that was 7 undertaken within the separate work packages not only shed light on current measures in 8 9 place to limit dissemination bias but also gathered the views of key stakeholders on the 10 possible barriers to transparency in medical research. Thus, based on the evidence collated 11 12 throughout the first phase of OPEN, a draft set of preliminary recommendations for each 13 14 stakeholder group was developed. These were then discussed with the OPEN Consortium, 15 the advisoryFor board of OPEN,peer and with review selected external key only stakeholders to capture the 16 17 different perspectives. 18 Instead of solely describing aspects of the general problem of dissemination bias, in addition 19 20 to the general recommendations, we developed several targeted and concrete 21 recommendations, which specifically address each key stakeholder group and thus, as a 22 23 whole, offer a holistic approach to address the problem. Further, due to the discussions held 24 on topics such as possible opposition, resources required, and implementability, we were 25 26 able to go beyond generic recommendations. The detailed rationales and explanations for 27 these recommendations will be made available soon in separate articles and through the 28 29 OPEN website (www.openproject.eu). By means of this dissemination strategy, we hope to 30 expedite the widespread dissemination and uptake of OPEN recommendations by the 31 32 relevant key groups and institutions and to thus initiate the implementation of more adequate

33 http://bmjopen.bmj.com/ mechanisms to reduce and prevent dissemination bias in the long run. 34 35 36 Although we developed separate sets of recommendations for industry and public funding 37 38 agencies, it is important to highlight that we agreed that the same ethical considerations 39 40 apply to all research funders. After all, the interests of patients and the public in general 28 41 should always override commercial or other interests . During workshop discussions it was on September 23, 2021 by guest. Protected copyright. 42 43 noted that the tendency of forprofit companies and also (to some extent) other stakeholder 44 groups such as medical licensing bodies, research institutions or policymakers and 45 46 legislators to place the protection of commercial interest over the public interest in access to 47 data from clinical trials does not seem to be proportional 29 30. Despite the fact that companies 48 49 are investing large sums of money to develop new treatments, the group agreed that 50 intellectual and commercial property rights are overall less important than basic human rights 51 52 with regard to information and health. 53 54 55 As already raised, it is obvious that some of our recommendations are relevant for more than 56 one stakeholder group. The recommendations given for benefit assessment institutions, for 57 58 example, are also applicable for regulators; however, they are given specifically for benefit 59 60 13 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml Page 15 of 40 BMJ Open BMJ Open: first published as 10.1136/bmjopen-2014-006666 on 5 May 2015. Downloaded from 1 2 3 assessment agencies since work conducted within OPEN revealed that cooperation between 4 agencies, and the transparency of methodology used, could particularly be improved within 5 6 this key group. Similarly, recommendations with respect to data sharing, e.g. given for 7 research ethics committees and legislators, are again also relevant for other key groups. It 8 9 needs to be kept in mind, though, that data sharing was not the focus of the OPEN project. 10

41 The recommendations from the OPEN project coincide with a number of important decisions on September 23, 2021 by guest. Protected copyright. 42 43 at the European Medicines Agency (EMA) and the European Parliament, namely the 44 finalisation and implementation of the EMA’s data sharing policy. EMA and the European 45 46 Parliament, which approved the new European clinical trials regulation on April 2nd, 2014, 47 have received praise for their leadership on facilitating access to clinical study reports 34. 48 49 However, the recently released draft terms for use of these reports included several 50 important restrictions such as strict confidentiality, access to registered users and onscreen 51 35 36 52 viewing only . Certainly, this would impede the widespread and most effective use of 53 these important data sources. Criticism was voiced by academia to which EMA responded 54 37 55 with a press release announcing more userfriendly amendments . As of now, it is unclear 56 what the final policy will look like. 57 58 59 60 14 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml BMJ Open Page 16 of 40 BMJ Open: first published as 10.1136/bmjopen-2014-006666 on 5 May 2015. Downloaded from 1 2 3 We hope that the findings from the OPEN project will contribute to these decisions and 4 implementation processes, emphasising once more the needs of academia and the public for 5 6 unrestricted access to an unbiased body of evidence of complete results of all clinical trials. 7

8 9 10 11 12 13 14 15 For peer review only 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32

33 http://bmjopen.bmj.com/ 34 35 36 37 38 39 40

41 on September 23, 2021 by guest. Protected copyright. 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 15 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml BMJ Open: first published as 10.1136/bmjopen-2014-006666 on 5 May 2015. Downloaded from 16 BMJ Open http://bmjopen.bmj.com/ on September 23, 2021 by guest. Protected copyright. For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml For peer review only part of the funding until a project’s results are adequately disseminated (recommendation). (recommendation). disseminated adequately are results project’s a until funding the of part project’s final report, but instead is followed up until all agreed data have been disseminated (recommendation). (recommendation). disseminated been have data agreed all up until followed is instead but report, final project’s disseminated, regardless of the nature of findings, in all funding contracts (strong recommendation). recommendation). (strong contracts funding all in findings, of nature the of regardless disseminated, participant (strong recommendation). recommendation). (strong participant clinical trials publicly accessible (strong recommendation). recommendation). (strong accessible trials publicly clinical order to keep an accurate record of funded projects and publication outcomes (recommendation). (recommendation). outcomes publication and projects funded of record accurate keepan to order proposals (strong recommendation). recommendation). (strong proposals all calls for proposals (strong recommendation). recommendation). (strong proposals for calls all committees) available upon the publication/dissemination of results (strong recommendation). recommendation). (strong results of publication/dissemination upon the available committees) 2.3 2.3 Pharmaceutical and devices companies should make their trial protocols + amendments (as submitted to research ethics 2.2 2.2 Pharmaceutical and devices companies should register all clinical trials in a public registry before the recruitment of the first 2.1 2.1 Pharmaceutical and medical device companies should make their policies concerning the dissemination of methods and results of Pharmaceutical and device companies companies device and Pharmaceutical 1.6 1.6 in database and on of disseminated their all accessible grants were create awarded a results should how publicly agencies Funding 1.5 1.5 Funding agencies should consider providing incentives for researchers who disseminate their results, or, alternatively, withhold a 1.4 1.4 Funding agencies should implement measures to ensure that the evaluation process of funded projects does not end with the 1.3 1.3 Funding agencies should include the requirement for grantees to explicitly declare that the results of funded research will be 1.2 1.2 Funding agencies should include the requirement for grantees to provide a dissemination plan for funded projects in all calls for 1.1 1.1 in results research of dissemination the for requirement and the bias dissemination a on statement include should agencies Funding Funding agencies agencies Funding Table 1: Targeted recommendations by key stakeholder stakeholder group key by recommendations Targeted 1: Table No. Recommendation Page 17 of 40 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 BMJ Open: first published as 10.1136/bmjopen-2014-006666 on 5 May 2015. Downloaded from 17 Page 18 of 40

Page 19 of 40 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 BMJ Open: first published as 10.1136/bmjopen-2014-006666 on 5 May 2015. Downloaded from 19 Page 20 of 40 BMJ Open http://bmjopen.bmj.com/ on September 23, 2021 by guest. Protected copyright. For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml For peer review only further (unpublished) data and to foster data sharing (strong recommendation). recommendation). (strong sharing data foster to and data (unpublished) further achieve better transparency and understanding (strong recommendation). recommendation). (strong understanding and transparency better achieve requirements are sanctioned (strong recommendation). recommendation). (strong sanctioned are requirements of of a trial is made in available the same registry that the trial was registered in, in a timely fashion (i.e.,1 year after study completion recommendation). (strong database EU the in registered forstudies all or inactivity) approved approved by the research ethics committee, including the potential protocol amendments, is submitted and made publicly available recommendation). (strong document as searchable a registered in an EU database which is publicly accessible (strong recommendation). recommendation). (strong accessible publicly is which database EU an in registered devices and biologicals (strong recommendation). recommendation). (strong biologicals and devices reporting the same findings (strong recommendation). recommendation). (strong findings same the reporting registration (recommendation). (recommendation). registration 10.3 10.3 Benefit assessment institutions should use the full evidence base available for an intervention for their assessments (strong 10.2 10.2 Benefit assessment institutions should aim for a higher degree of collaboration between institutions to facilitate the detection of 10.1 10.1 Benefit assessment institutions should make their methods and processes of benefit assessment publicly available in order to Benefit assessment institutions institutions assessment Benefit 9.5 9.5 Responsible authorities (such as EMA for drugs) should ensure that trial sponsors failing to comply with such result submission 9.4 9.4 Responsible authorities (such as EMA for drugs) should mandate that the full report including all results (e.g., clinical study report) 9.3 9.3 full an mandate in the a that, registration database, protocol fortrial’s upon EU (such should as authorities drugs) EMA Responsible 9.2 9.2 Responsible authorities (such as the EMA for drugs) should mandate that all clinical trials in humans falling under their remit are 9.1 9.1 Regulation of pharmaceutical products should be extended to cover other therapeutic and diagnostic agents such as medical Regulatory agencies agencies Regulatory 8.5 8.5 Journals should for check of publication redundant results by using software textmatching and peer asking reviewers about papers 8.4 8.4 Journal editors should publish editorials and commentaries about the problem of dissemination bias and the benefits of trial 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 BMJ Open: first published as 10.1136/bmjopen-2014-006666 on 5 May 2015. Downloaded from 20 BMJ Open http://bmjopen.bmj.com/ on September 23, 2021 by guest. Protected copyright. For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml For peer review only decisionmaking process (strong recommendation). recommendation). (strong process decisionmaking (registered in the EU database) (strong recommendation). recommendation). (strong database) EU the in (registered information (strong recommendation). recommendation). (strong information results (disclosure of full protocols and full clinical study reports): reports): study clinical full and protocols full of (disclosure results a. b. evidence requested all submit to manufacturers for obligation a legal c. databases to EMA access public recommendation) (strong reports full study and to protocols access public deemed incomplete (e.g., no adequate proof of benefit based on incomplete data set) (strong recommendation). recommendation). (strong set) data incomplete on based benefit of proof adequate no (e.g., incomplete deemed recommendation). recommendation). Organization WorldHealth WHO = Union; European = EU Agency; Medicines European = EMA 11.4 11.4 Legislators should ensure that the raw data (anonymized individual patient data) are made publicly available for all clinical studies 11.3 11.3 Legislators should institute a legal obligation for manufacturers to submit all data and other required information for the formal 11.2 11.2 proprietary confidential/ commercially NOT are public the and of patients tohealth the related data that all ensure should Legislators 11.1 11.1 recommendation). (strong mandatory humans in trials of clinical registration prospective make should Legislators Legislators Legislators 10.5 10.5 Benefit assessment institutions should request from legislators the following items which will allow the consideration of all study 10.4 10.4 Benefit assessment institutions should specify their course of action if they find that the evidence base for an assessment is

Page 21 of 40 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 BMJ Open Page 22 of 40 BMJ Open: first published as 10.1136/bmjopen-2014-006666 on 5 May 2015. Downloaded from 1 2 3 Box 1: Definition of a clinical trial 4 5 Clinical trial: 6 A clinical trial is any research study that prospectively assigns human participants or 7 groups of humans to one or more healthrelated interventions to evaluate the effects 8 on health outcomes. Clinical trials may also be referred to as interventional trials. 9 Interventions include but are not restricted to drugs, cells and other biological 10 products, surgical procedures, radiologic procedures, devices, behavioural 11 treatments, processofcare changes, preventive care, etc. This definition includes 12 39 13 Phase I to Phase IV trials . 14 15 For peer review only 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32

33 http://bmjopen.bmj.com/ 34 35 36 37 38 39 40

41 on September 23, 2021 by guest. Protected copyright. 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml 21 Page 23 of 40 BMJ Open BMJ Open: first published as 10.1136/bmjopen-2014-006666 on 5 May 2015. Downloaded from 1 2 3 Details of contributors: 4 JM and LS drafted this manuscript based on the discussions and results of the OPEN 5 6 recommendations workshop. All named authors were involved in the work packages that led 7 to the development of the draft recommendations. All had actively participated in the OPEN 8 9 workshop and repeatedly commented and revised the draft manuscript; they are listed in 10 alphabetical order. All authors approved the final manuscript. People listed as part of the 11 12 OPEN project consortium contributed to the various work packages of the OPEN project on 13 14 the basis of which the draft recommendations were developed. 15 For peer review only 16 17 Competing interest statement: 18 All authors have completed the ICMJE uniform disclosure form at 19 20 www.icmje.org/coi_disclosure.pdf and declare: the OPEN project has received funding from 21 the European Union under grant agreement n° 285453; Davina Ghersi reports being the 22 23 Chair of the WHO Advisory Panel on the International Clinical Trials Registry Platform; Jos 24 Kleijnen reports grants from Amgen, GSK, Sanofi, Biomarin, MSD, Roche and Bayer outside 25 26 the submitted work; Joerg Meerpohl reports being a member of the WHO Advisory Panel on 27 the International Clinical Trials Registry Platform; Andreas Reis is a staff member of the 28 29 World Health Organization. The author alone is responsible for the views expressed in this 30 article and they do not necessarily represent the decisions, policy or views of the World 31 32 Health Organization; Elizabeth Wager reports training and consultancy activities for various

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml 22 BMJ Open Page 24 of 40 BMJ Open: first published as 10.1136/bmjopen-2014-006666 on 5 May 2015. Downloaded from 1 2 3 worldwide basis to the BMJ Publishing Group Ltd, and its Licensees to permit this article (if 4 accepted) to be published in BMJ editions and any other BMJPGL products and to exploit all 5 6 subsidiary rights, as set out in our licence (bmj.com/advice/copyright.shtml). 7

33 http://bmjopen.bmj.com/ 34 35 36 37 38 39 40

41 on September 23, 2021 by guest. Protected copyright. 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml 23 Page 25 of 40 BMJ Open BMJ Open: first published as 10.1136/bmjopen-2014-006666 on 5 May 2015. Downloaded from 1 2 3 References 4 5 6 1. Strech D. Normative arguments and new solutions for the unbiased registration and publication of 7 clinical trials. J Clin Epidemiol 2012;65(3):276-81. 8 2. Dwan K, Gamble C, Williamson PR, et al. Systematic review of the empirical evidence of study 9 publication bias and outcome reporting bias - an updated review. PLoS One 10 2013;8(7):e66844. 11 3. Antes G, Chalmers I. Under-reporting of clinical trials is unethical. Lancet 2003;361(9362):978-9. 12 4. Dickersin K. The existence of publication bias and risk factors for its occurrence. JAMA 13 1990;263(10):1385-89. 14 15 5. Muller KF, Briel M, D'Amario A, et al. Defining publication bias: protocol for a systematic review of For peer review only2 16 highly cited articles and proposal for a new framework. Syst Rev 2013; :34. 17 6. Song F, Parekh S, Hooper L, et al. Dissemination and publication of research findings: an updated 18 review of related biases. Health Technol Assess 2010;14(8):iii, ix-xi, 1-193. 19 7. Brown T. It's time for alltrials registered and reported. Cochrane Database Syst Rev 20 2013;4:ED000057. 21 8. Doshi P, Dickersin K, Healy D, et al. Restoring invisible and abandoned trials: a call for people to 22 publish the findings. BMJ 2013;346:f2865. 23 9. Groves T. What does oseltamivir do, and how will we know? BMJ 2013;347:f4687. 24 10. Nisen P, Rockhold F. Access to patient-level data from GlaxoSmithKline clinical trials. N Engl J Med 25 2013;369(5):475-8. 26 11. Littmann J, Antes G, Strech D, et al. To overcome failure to publish negative findings: The OPEN 27 28 project. Maturitas 2013. 29 12. Mueller KF, Meerpohl JJ, Briel M, et al. Detecting, quantifying and adjusting for publication bias in 30 meta-analyses: protocol of a systematic review on methods. Syst Rev 2013;2:60. 31 13. Schmucker C, Bluemle A, Briel M, et al. A protocol for a systematic review on the impact of 32 unpublished studies and studies published in the gray literature in meta-analyses. Syst Rev

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml 24 BMJ Open Page 26 of 40 BMJ Open: first published as 10.1136/bmjopen-2014-006666 on 5 May 2015. Downloaded from 1 2 3 23. Jones C, Keil L, Weaver M, et al. Clinical trials registries are under-utilized in the conduct of 4 systematic reviews: a cross-sectional analysis. Systematic Reviews 2014;3(1):126. 5 24. Higgins JPT, Altman DG, Sterne JAC. Chapter 8: Assessing risk of bias in included studies. In: 6 Higgins JPT, Green S, eds. Cochrane Handbook for Systematic Reviews of Interventions 7 Version 510 (updated March 2011), 2011. 8 25. Higgins JP, Green S. Cochrane Handbook for Systematic Reviews of Interventions; Version 5.1.0, 9 available from http://www.cochrane-handbook.org. Chichester, UK John Wiley & Sons, Ltd 10 2011. 11 26. Sterne JA, Sutton AJ, Ioannidis JP, et al. Recommendations for examining and interpreting funnel 12 343 13 plot asymmetry in meta-analyses of randomised controlled trials. BMJ 2011; :d4002. 14 27. Ofori-Adjei D, Antes G, Tharyan P, et al. Have online international medical journals made local 15 journalsFor obsolete? peerPLoS Med 2006; 3review(8):e359. only 16 28. Gotzsche PC. Why we need easy access to all data from all clinical trials and how to accomplish it. 17 Trials 2011;12:249. 18 29. Strech D, Littmann J. Lack of proportionality. Seven specifications of public interest that override 19 post-approval commercial interests on limited access to clinical data. Trials 2012;13(1):100. 20 30. Gotzsche PC, Jorgensen AW. Opening up data at the European Medicines Agency. BMJ 21 2011;342:d2686. 22 31. Chalmers I, Glasziou P. Avoidable waste in the production and reporting of research evidence. 23 Lancet 2009;374(9683):86-9. 24 32. Chan AW, Song F, Vickers A, et al. Increasing value and reducing waste: addressing inaccessible 25 383 26 research. Lancet 2014; (9913):257-66. 27 33. Glasziou P, Altman DG, Bossuyt P, et al. Reducing waste from incomplete or unusable reports of 28 biomedical research. Lancet 2014;383(9913):267-76. 29 34. Regulation of the European Parliament and of the Council on clinical trials on medicinal products 30 for human use, and repealing Directive 2001/20/EC: accessed on 05.06.2014 at: 31 http://www.europarl.europa.eu/sides/getDoc.do?pubRef=-//EP//NONSGML+AMD+A7-2013- 32 0208+291-291+DOC+PDF+V0//EN 2014. 33 35. Torjesen I. European drug agency backtracks on plan to give researchers access to clinical trial http://bmjopen.bmj.com/ 34 reports. BMJ 2014;348. 35 36. Goldacre B, Godlee F, Heneghan C, et al. Open letter: European Medicines Agency should remove 36 barriers to access clinical trial data. BMJ 2014;348:g3768. 37 37. EMA. European Medicines Agency agrees policy on publication of clinical trial data with more 38 39 user-friendly amendments: accessed on 04.08.2014 at: 40 http://www.ema.europa.eu/docs/en_GB/document_library/Press_release/2014/06/WC500 168342.pdf, 2014. 41 on September 23, 2021 by guest. Protected copyright. 42 38. Eden J, Levit L, Berg AO, et al. Finding what works in health care: Standards for systematic 43 reviews: National Academies Press, 2011. 44 39. World Health Organization. International Clinical Trials Registry Platform (ICTRP). Secondary 45 International Clinical Trials Registry Platform (ICTRP). http://www.who.int/ictrp/en/. 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml 25 Page 27 of 40 BMJ Open BMJ Open: first published as 10.1136/bmjopen-2014-006666 on 5 May 2015. Downloaded from 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 For peer review only 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32

33 http://bmjopen.bmj.com/ 34 35 36 37 38 39 40

1 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml Page 29 of 40 BMJ Open

2 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml BMJ Open Page 30 of 40

37 http://bmjopen.bmj.com/ 38 39 40 41 42 43 44 45 on September 23, 2021 by guest. Protected copyright. 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

1 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml Page 31 of 40 BMJ Open

1 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml BMJ Open Page 32 of 40

37 http://bmjopen.bmj.com/ 38 39 40 41 42 43 44 45 on September 23, 2021 by guest. Protected copyright. 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

2 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml Page 33 of 40 BMJ Open

1 2 3 Supplementary Figure 1: Work package (WP) structure of the OPEN project 4 BMJ Open: first published as 10.1136/bmjopen-2014-006666 on 5 May 2015. Downloaded from 5 6 7 8 9 WP!14*Project*management* 10 11 12 13 ! ! 14 ! ! ! ! ! ! ! ! 15 WP!1! WP!6! WP!11!! WP!2! WP!3!! WP4!! WP5!! WP!7!! WP!8!! WP!9!! WP!10!! Systema(c* Evaluate** Impact** 16 !Evaluate* Evaluate** Evaluate** Evaluate** Evaluate* Evaluate** Evaluate** Evaluate** *reviews* researchers,** of** funding** (pharma4* ethic** research** trial** regulatory* *to** For peer reviewbiomedical** only impact** 17 authors** publica(on** agencies* ceu(cal)** boards* ins(tu(ons* registers* journals* *agencies* on*beneﬁt** assess* and** bias*on** 18 industry* assessment** **publica4* reviewers* network* * process* 19 (on*bias* *metanalysis* 20 21 22 23 24 ! WP!12!! 25 Evalua(on*of*ﬁndings** 26 and*formula(on*of** recommenda(ons* 27 28 29 ! 30 WP!13!! 31 Dissemina(on** and*exploita(on*of*results* 32 33 34 35 36

37 http://bmjopen.bmj.com/ 38 39 40 41 42 43 44 45 on September 23, 2021 by guest. Protected copyright. 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml BMJ Open: first published as 10.1136/bmjopen-2014-006666 on 5 May 2015. Downloaded from

BMJ Open Page 34 of 40

1 Supplementary File 1: Decision tables for general recommendations 2 3 4 5 Recommendation #1: 6 7 We strongly recommend that 8 9 10 11 For peer review only 12 all stakeholders in the knowledge generation and translation process 13 14 15 16 raise awareness about dissemination bias (publication bias) and measures to reduce it. http://bmjopen.bmj.com/ 17 18 19 Category Judgment Remark 20 21 22 Likely limited effectiveness in the short run, but potentially high High confidence in effectiveness Yes Nox 23 effectiveness in the long run

Page 35 of 40 BMJ Open

1 Strength of recommendation: x Strong recommendation 2 3

4 Recommendation 5 6 7 8 9 Remark: 10 11 For peerNot very concrete reviewrecommendation; room for interpretation. only 12 13 Should also be extended to patients who should be made aware of dissemination bias. 14 15

16 http://bmjopen.bmj.com/ 17 18 19 20 21 22 23

BMJ Open Page 36 of 40

Page 37 of 40 BMJ Open

1 would be required including monitoring efforts on 2 3 implementation and effectiveness. 4 5 Strength of recommendation: Strong recommendation 6 x 7

BMJ Open Page 38 of 40

Page 39 of 40 BMJ Open

1 Strength of recommendation: x Strong recommendation 2 3

BMJ Open Page 40 of 40

16 http://bmjopen.bmj.com/ 17 Category Judgment Remark 18 19 20 Support in itself likely not yet highly effective in reducing 21 dissemination bias, but if respective activities rigorously identify 22 High confidence in effectiveness Yes x No 23 and integrate unpublished trials likely relevant impact in the

Page 41 of 40 BMJ Open

1 No serious barrier to both political and financial support was 2 Easily implementable x Yes No 3 identified. 4 5 Strength of recommendation: Strong recommendation 6 x 7

24 on September 23, 2021 by guest. Protected copyright. 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 BMJ Open BMJ Open: first published as 10.1136/bmjopen-2014-006666 on 5 May 2015. Downloaded from

Evidence-informed recommendations to reduce dissemination bias in clinical research: conclusions from the OPEN (Overcome failure to Publish nEgative fiNdings) project

For peer review only Journal: BMJ Open

Manuscript ID: bmjopen-2014-006666

Article Type: Research

Date Submitted by the Author: 17-Sep-2014

Primary Subject Evidence based practice Heading:

Epidemiology, Ethics, Medical publishing and peer review, Research Secondary Subject Heading: methods

EPIDEMIOLOGY, Protocols & guidelines < HEALTH SERVICES Keywords: ADMINISTRATION & MANAGEMENT, JOURNALISM (see Medical Journalism)

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml Page 1 of 55 BMJ Open BMJ Open: first published as 10.1136/bmjopen-2014-006666 on 5 May 2015. Downloaded from 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 For peer review only 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32

33 http://bmjopen.bmj.com/ 34 35 36 37 38 39 40

8 9 Joerg J Meerpohl, deputy director1, Lisa K Schell, researcher1, Dirk Bassler, professor2,3, 10 Silvano Gallus, senior researcher4, Jos Kleijnen, professor5,6, Michael Kulig, researcher7, 11 12 Carlo La Vecchia, professor8, Ana Marušić, professor9, Philippe Ravaud, professor10, 13 11 1 12 14 Andreas Reis, researcher , Christine Schmucker, researcher , Daniel Strech, professor , 15 Gerard Urrutia,For researcher peer13, Elizabeth review Wager, publications only consultant14, Gerd Antes, 16 1 17 professor and the OPEN project consortium 18 19 20 Authors’ Affiliation: 21 1 German Cochrane Centre, Medical Center – University of Freiburg, Freiburg, Germany 22 2 23 Department of Neonatology, University Hospital Zurich, Zurich, Switzerland 24 3 Center for Pediatric Clinical Studies, University Children’s Hospital Tuebingen, Tuebingen, 25 26 Germany 27 4 Department of Epidemiology, IRCCS Istituto di Ricerche Farmacologiche “Mario Negri”, 28 29 Milan, Italy 30 5 Kleijnen Systematic Reviews Ltd, York, United Kingdom 31 32 6 School for Public Health and Primary Care, Maastricht University, Maastricht, The

41 Split, Croatia on September 23, 2021 by guest. Protected copyright. 42 10 43 INSERM U738 research unit, Paris Descartes University, Paris, France 44 11 Ethics and Health, Department of Ethics, Trade, Human Rights and Health Law, World 45 46 Health Organization, Geneva, Switzerland 47 12 CELLS – Centre for Ethics and Law in the Life Sciences, Institute of History, Ethics and 48 49 Philosophy of Medicine, Hannover Medical School, Hannover, Germany 50 13 Centro Cochrane IberoamericanoServei d’Epidemiologia Clínica i Salut Pública, Institut 51 52 d’Investigació Biomèdica Sant Pau, Barcelona, Spain 53 14 Sideview, Princes Risborough, United Kingdom 54 55 56 Correspondence to: 57 58 Joerg J Meerpohl: [email protected] 59 60 1 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml Page 3 of 55 BMJ Open BMJ Open: first published as 10.1136/bmjopen-2014-006666 on 5 May 2015. Downloaded from 1 2 3 4 5 6 Word count: 7 Abstract: 299 8 9 Text: 3321 10 11 12 13 14 15 For peer review only 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32

33 http://bmjopen.bmj.com/ 34 35 36 37 38 39 40

41 on September 23, 2021 by guest. Protected copyright. 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 2 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml BMJ Open Page 4 of 55 BMJ Open: first published as 10.1136/bmjopen-2014-006666 on 5 May 2015. Downloaded from 1 2 3 Structured abstract 4 5 6 Introduction 7 Dissemination bias in clinical research severely impedes informed decisionmaking not only 8 9 for healthcare professionals and patients, but also for funders, research ethics committees, 10 regulatory bodies and other stakeholder groups that make healthrelated decisions. 11 12 Decisions based on incomplete and biased evidence can not only harm people, but may also 13 14 have huge financial implications by wasting resources on ineffective or harmful diagnostic 15 and therapeuticFor measures peer and unnecessary review research. Due only to involvement of multiple 16 17 stakeholders it remains easy for any single group to assign responsibility for resolving the 18 problem to others. 19 20 21 Objective 22 23 To collate current available evidence on the scope and extent of the problem of 24 dissemination bias, and to develop evidenceinformed general and targeted 25 26 recommendations addressing the various stakeholders involved in knowledge generation 27 and dissemination to help overcome the problem of dissemination bias. 28 29 30 Methods 31 32 Based on findings from systematic reviews, document analyses and surveys, we developed

33 http://bmjopen.bmj.com/ general and targeted draft recommendations. During a 2day workshop in summer 2013, 34 35 these draft recommendations were discussed with external experts and key stakeholders 36 and refined following a rigorous and transparent methodological approach. 37 38 39 40 Results

41 Four general, overarching recommendations applicable to all or most stakeholder groups on September 23, 2021 by guest. Protected copyright. 42 43 were formulated, addressing (1) awareness raising, (2) implementation of targeted 44 recommendations, (3) trial registration and results posting, and (4) systematic approaches to 45 46 evidence synthesis. These general recommendations are complemented by 47 targeted 47 recommendations tailored towards funding agencies, pharmaceutical and device companies, 48 49 research institutions, researchers (systematic reviewers and trialists), research ethics 50 committees, trial registries, journal editors and publishers, regulatory agencies, benefit 51 52 assessment institutions and legislators. 53 54 55 Conclusions 56 Despite various recent examples of dissemination bias and several initiatives to reduce it, the 57 58 problem of dissemination bias has not been resolved. Tailored recommendations based on a 59 60 3 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml Page 5 of 55 BMJ Open BMJ Open: first published as 10.1136/bmjopen-2014-006666 on 5 May 2015. Downloaded from 1 2 3 comprehensive approach will hopefully help increase transparency in biomedical research by 4 overcoming the failure to disseminate negative findings. 5 6 7 8 9 10 11 12 13 14 15 For peer review only 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32

33 http://bmjopen.bmj.com/ 34 35 36 37 38 39 40

41 on September 23, 2021 by guest. Protected copyright. 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 4 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml BMJ Open Page 6 of 55 BMJ Open: first published as 10.1136/bmjopen-2014-006666 on 5 May 2015. Downloaded from 1 2 3 Article Summary Section 4 5 6 7 Strengths and limitations of this study 8 9 Strengths: 10 11 • Comprehensive set of evidenceinformed recommendations aimed at reducing 12 dissemination bias based on work conducted as part of European Union FP7 project 13 (OPEN project; www.openproject.eu) 14 • Both general and specific recommendations targeted at various key stakeholder 15 For peer review only 16 groups involved in the knowledge generation and translation process 17 • Use of a rigorous and transparent methodology to develop recommendations 18 including at consensus meeting including external experts and stakeholders 19 20 Limitation: 21 22 • Iteration of some recommendations which are not new but have so far not been 23 broadly implemented 24 25 26 27 28 29 30 31 32

33 http://bmjopen.bmj.com/ 34 35 36 37 38 39 40

41 on September 23, 2021 by guest. Protected copyright. 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 5 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml Page 7 of 55 BMJ Open BMJ Open: first published as 10.1136/bmjopen-2014-006666 on 5 May 2015. Downloaded from 1 2 3 Introduction 4 In order to make informed decisions, healthcare practitioners, consumers, public health 5 6 professionals, policymakers, and health and research funding bodies, rely on evidence from 7 clinical research. The generation of evidence is made possible because patients participate 8 9 in clinical studies and accept any researchrelated risks and burdens. To inform medical 10 decisionmaking, demonstrate respect to trial participants, and maintain public trust in clinical 11 12 research it is important that such evidence is made available in an easily accessible and 13 1 14 unbiased way . However, many research findings are either not published at all (an 15 estimated 50%)For 2 or only peer selectively, i.e., review with a bias towards onlyspecific aspects or presenting 16 17 only partial information. Healthcare professionals and policymakers are therefore frequently 18 unable to make decisions based on the entire relevant research evidence 3. This problem 19 4 20 has been called ‘publication bias’ or – as more recently suggested ‘dissemination bias’ to 21 reflect the multiple facets of this problem 5 6. Dissemination bias has recently received 22 7 23 attention due to initiatives such as the AllTrials campaign (www.alltrials.net ) and the RIAT 24 proposal 8, the discussion of oseltamivir in the context of avian influenza 9, and companies’, 25 26 e.g. GlaxoSmithKline’s, announcement to grant access to patientlevel data from all company 27 trials 10. However, general awareness of this problem, and even more importantly, the 28 29 necessity to address and resolve it, has presumably not yet been fully recognized by many 30 stakeholders. 31 32

41 problems, namely the nonpublication – or in a broader sense the nondissemination – of the on September 23, 2021 by guest. Protected copyright. 42 43 complete data of all clinical studies. 44 45 46 We present the results of the 24months OPEN project that ran from November 2011 to 47 October 2013. As an interdisciplinary initiative, it brought together academics and 48 49 stakeholders from across Europe with the aim of developing evidenceinformed 50 recommendations and strategies for overcoming the failure to publish negative research 51 52 findings. 53 One activity of OPEN was to conduct a series of systematic reviews to assess the 54 55 occurrence of nonpublication of research findings and the resulting dissemination bias. 56 These reviews addressed aspects such as existing terminology to describe problems of 57 58 59 60 6 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml BMJ Open Page 8 of 55 BMJ Open: first published as 10.1136/bmjopen-2014-006666 on 5 May 2015. Downloaded from 1 2 3 publication and related biases 5, available methods to detect and measure dissemination bias 4 12, and the extent 13, and impact 14 of the problem of nonpublication of research findings. 5 6 7 Another focus of OPEN was to describe current practices by various players involved in 8 9 knowledge generation and knowledge translation to provide insights about how to avoid or 10 reduce bias due to nonpublication of research findings and to identify ways to change 11 12 practices that contribute to the problem. This has been addressed by assessing and 13 14 evaluating the policies and procedures in place for preventing dissemination bias by the main 15 parties involvedFor in approving, peer funding, conducting,review publishing, onlydisseminating, and assessing 16 17 clinical research. Different work packages of OPEN have surveyed representatives of 18 funding agencies, the pharmaceutical industry, research ethics committees, research 19 15 20 institutions, researchers, trial registers, medical journals , regulatory agencies, and benefit 21 assessment agencies. 22 23 Findings from the OPEN work packages informed a 2day recommendations workshop in 24 May 2013 attended by the OPEN project partners, researchers from the UNCOVER project, 25 26 and selected key stakeholders from across the world. The workshop aimed at developing 27 and refining a set of general and targeted recommendations designed to specifically consider 28 29 the roles that the respective stakeholder groups should play in reducing the incomplete 30 dissemination of research findings. 31 32

33 http://bmjopen.bmj.com/ This article first describes the process used to reach a consensus on the recommendations 34 35 followed by a discussion of the general recommendations, which are aimed at the major 36 stakeholders in the knowledge translation process. It then presents 47 recommendations 37 38 aimed at the specific target groups, namely funding agencies, pharmaceutical and device 39 40 companies, research institutions, researchers (systematic reviewers and trialists), research

41 ethics committees, trial registries, journal editors and publishers, regulatory agencies, benefit on September 23, 2021 by guest. Protected copyright. 42 43 assessment institutions and legislators. 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 7 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml Page 9 of 55 BMJ Open BMJ Open: first published as 10.1136/bmjopen-2014-006666 on 5 May 2015. Downloaded from 1 2 3 Methods 4 Based on the data gathered in each OPEN work package (i.e., conducted surveys, 5 6 qualitative studies and systematic reviews) (Figure S1) each of these groups formulated 4 to 7 7 recommendations on how to reduce dissemination bias tailored to their respective key 8 9 group in the knowledge translation process. These recommendations were compiled, 10 ordered, and harmonised to form draft recommendations used as the basis for discussion at 11 12 the OPEN workshop including further international key stakeholders in clinical research. This 13 14 workshop was held in Freiburg / Germany, 2324th May 2013, in order to reach a consensus 15 and to finaliseFor the recommendations. peer review The participants were only the members of the OPEN 16 17 Consortium (see Appendix 1), members of the OPEN advisory board (see Appendix 2) and 18 11 stakeholders from relevant key groups (see Appendix 3). Workshop participants 19 20 discussed the draft recommendations in a structured and transparent manner and decided 21 on consistent definitions to be used in the phrasing of its recommendations (e.g., what is 22 23 meant by ‘clinical trial’) and defined which different types of data corresponded to the 24 individual recommendations. The definitions are listed in Boxes 13. 25 26 The recommendations were also discussed in small groups. To structure the discussion and 27 to facilitate the documentation of the judgements underlying the recommendations a decision 28 29 table was used for rating each individual recommendation (Figure 2). The decision tables 30 required each group to assess 5 criteria for each recommendation and to grade it as a 31 32 “strong recommendation” or only a “recommendation” based on their judgements. The

33 http://bmjopen.bmj.com/ following 5 explicit criteria, which are conceptually based on the GRADE approach for 34 35 developing recommendations 16 17, were assessed for each recommendation: 36

37 38 • Confidence in effectiveness – whether the group was confident that if this 39 40 recommendation were implemented it would effectively reduce dissemination bias.

41 • Balance of benefits and downsides – whether the group thought that the benefits of on September 23, 2021 by guest. Protected copyright. 42 43 this recommendation would clearly outweigh its potential downsides. 44 • Likelihood of opposition – whether the group thought that there would be no or only 45 46 minor opposition to the implementation of this recommendation by relevant key 47 groups. 48 49 • Resource use – whether the group thought that implementing this recommendation 50 would require only limited resources (both direct financial costs and indirect 51 52 resources). 53 54 • Implementability and feasibility – whether the group thought that this recommendation 55 could be easily implemented (within a reasonable timeframe) and easily sustained. 56 57 58 59 60 8 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml BMJ Open Page 10 of 55 BMJ Open: first published as 10.1136/bmjopen-2014-006666 on 5 May 2015. Downloaded from 1 2 3 For all 5 criteria, the groups were asked to provide a ‘yes or no’ answer and an explanation 4 for their judgement. Regarding the classification as either ‘strong recommendation’ or 5 6 ‘recommendation’, we decided not to use a strict rule (e.g. >= 3 saying yes results in strong 7 recommendation, < 2 saying yes results in recommendation), but rather apply the following 8 9 rule of thumb: if a recommendation was rated for each or most of the criteria as ‘yes’, the 10 recommendation was rated as a ‘strong recommendation'. If only one or a few judgements 11 12 strongly favoured a recommendation, it was classed simply as a recommendation. The 13 14 rationale for this flexible approach lies in the fact that the weight each of these criteria bears 15 on the final classificationFor peer can vary depending review on the recommendation only at hand and/or on the 16 17 extent to which a certain criterion speaks for or against a recommendation. In summary, a 18 judgement based on integration of transparent assessments for these criteria seemed more 19 17 20 appropriate than following a rigid and inflexible rule . 21 22 The outcomes of all small group deliberations were then discussed in the open plenary 23 24 sessions until a consensus on all recommendations was reached. The OPEN Consortium 25 and the other meeting participants then endorsed the full set of recommendations. 26 27 Following our workshop, the list of endorsed recommendations was circulated and refined by 28 members of the OPEN Consortium. As a last step, the final recommendations document was 29 30 sent to all participants of our recommendations workshop for final technical revisions and 31 approval. 32

33 http://bmjopen.bmj.com/ 34 35 36 37 38 39 40

41 on September 23, 2021 by guest. Protected copyright. 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 9 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml Page 11 of 55 BMJ Open BMJ Open: first published as 10.1136/bmjopen-2014-006666 on 5 May 2015. Downloaded from 1 2 3 Results 4 In this report, we first present four general overarching recommendations – aimed at major 5 6 stakeholders in the knowledge generation and translation process – followed by the main 7 results from the OPEN project, i.e. 47 targeted recommendations addressing 11 key 8 9 stakeholder groups. 10

11 12 General recommendations: 13 14 Each general recommendation is followed by an explanation of the rationale and reasons 15 why it was ratedFor as ‘strong’ peer or not based review on our decision table only (see Appendix 4 for detailed 16 17 decision tables). 18 19 20 1. All stakeholders should raise awareness about dissemination bias and measures to 21 reduce it (strong recommendation). 22 23 24 Awareness of the problem of dissemination bias is a prerequisite for change. However, some 25 26 of the responses to our surveys indicated that many people still deem this problem not 27 relevant 15. It needs to be emphasised among all stakeholder groups that dissemination bias 28 29 may have serious consequences on the health of people and trust in research. 30 We decided to make this a strong recommendation because we think that the benefits of this 31 32 recommendation clearly outweigh its potential drawbacks and that only limited resources are

41 resources available for actively engaging in raising awareness and emphasising its on September 23, 2021 by guest. Protected copyright. 42 43 relevance. Another important aspect that was identified is the need to extend awareness 44 raising activities to patients and their representatives. All patients participating in clinical trials 45 46 should be made aware of the fact that their participation only contributes to scientific 47 progress if the results are published, and that this is often not the case. 48 49 50 2. All stakeholders should disseminate and facilitate the implementation of targeted OPEN 51 52 recommendations as outlined in Table 1 (strong recommendation). 53 54 55 In order for our recommendations to have an impact, they need to be disseminated to as 56 many representatives of the key stakeholder groups as possible. Another prerequisite to 57 58 change is that the stakeholders make a collective effort. The problem of dissemination bias 59 60 10 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml BMJ Open Page 12 of 55 BMJ Open: first published as 10.1136/bmjopen-2014-006666 on 5 May 2015. Downloaded from 1 2 3 needs to be addressed through concerted activities that include several or all stakeholders in 4 order to prevent responsibilities from being shifted from one stakeholder to the other (our 5 6 research showed that many groups believe it to be ‘somebody else’s problem’). 7 This recommendation was rated a strong recommendation. Its drawbacks are the likelihood 8 9 of some opposition and the need for substantial resources for comprehensive dissemination 10 and implementation. While the dissemination of the recommendations alone will not 11 12 effectively reduce dissemination bias, their full or even partial implementation should help 13 14 reduce dissemination bias. In order to be fully effective, tailored dissemination and 15 implementationFor strategies peer will be necessary. review only 16 17 18 3. All stakeholders should promote trial registration and posting of results, and support 19 20 initiatives that facilitate searches across multiple trial registries (strong recommendation). 21 22 23 Clinical trial registers are important sources for identifying completed and ongoing trials. 24 Efforts to promote prospective registration of trials are therefore crucial. The usefulness of 25 26 registers, however, depends on the completeness and comprehensiveness of the included 27 information. Accordingly, adequate quality control measures need to be implemented. It also 28 29 became evident that metasearching, the possibility of searching across different registries 30 through a single interface, is key to their usefulness. This is highly relevant considering 31 32 existing language barriers and a lack of awareness of the existence of national registries

33 http://bmjopen.bmj.com/ among users which might result in relevant registers not being included in their searches. A 34 35 platform offering a metasearch for trials across multiple registries would help identifying 36 trials that would otherwise have been missed. 37 38 This recommendation was considered to have long term, rather than immediate, impact. 39 40 Together with the favourable balance of benefits and drawbacks, the knowledge of how

41 limited resources could hinder good implementation led us to make this a strong on September 23, 2021 by guest. Protected copyright. 42 43 recommendation. As a result of the US FDA Amendments Act Section 801 passed in 2007, 44 ClinicalTrials.gov has required (since September 2008) the posting of results of aggregate 45 18 46 data from applicable trials by researchers within 12 months of completion . Once more 47 registries adopt this approach and once platforms allowing metasearching are available, 48 49 clinical trial registers will become essential tools to reduce dissemination bias. 50 51 52 4. All stakeholders should support activities to systematically, with rigorous methodology, 53 synthesise information from studies (strong recommendation). 54 55 56 The work of identifying all studies that address a particular question undertaken by authors of 57 58 systematic reviews helps detect dissemination bias. These efforts include systematic 59 60 11 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml Page 13 of 55 BMJ Open BMJ Open: first published as 10.1136/bmjopen-2014-006666 on 5 May 2015. Downloaded from 1 2 3 literature searches across electronic databases, the searching of trial registries and grey 4 literature as well as contacting experts in the field. Also helpful in detecting dissemination 5 6 bias are, for example, the exploration of selective reporting of outcomes as part of the 7 detailed risk of bias evaluation within systematic reviews 19 and the graphical (e.g., funnel 8 9 plots) and statistical assessment of potentially selective publication of whole studies (e.g., 10 Egger’s test) 20 21. Finally, users of systematic reviews will be made aware of the potential 11 12 impact of dissemination bias on the interpretation of the findings (if these are clearly 13 14 described by the review authors). 15 It was notedFor during the peerdiscussion that review the availability of systematic only reviews in languages 16 17 other than English needs to be increased to achieve wider dissemination and use of these 18 crucial resources for evidencebased decisionmaking 22. Broad acceptance and use of 19 20 systematic reviews will also help to increase awareness about the problem of dissemination 21 bias as suggested in recommendation #1. 22 23 24 25 26 Specific recommendations targeted at key stakeholder groups 27 The 47 specific recommendations targeted at funding agencies, pharmaceutical and device 28 29 industry, research institutions, researchers (systematic reviewers and trialists), research 30 ethics committees, trial registries, journal editors and publishers, regulatory agencies, benefit 31 32 assessment institutions and legislators are presented in Table 1.

41 other stakeholder groups; however, we decided to list them primarily amongst those key on September 23, 2021 by guest. Protected copyright. 42 43 groups where the largest impact can be expected. More detailed rationales as well as related 44 assessments based on our decision tables for these targeted recommendations will be 45 46 discussed more extensively in forthcoming articles and will be made available through the 47 OPEN website (www.openproject.eu). 48 49 50 51 52 53 54 55 56 57 58 59 60 12 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml BMJ Open Page 14 of 55 BMJ Open: first published as 10.1136/bmjopen-2014-006666 on 5 May 2015. Downloaded from 1 2 3 Discussion 4 The OPEN project included a broad range of key stakeholders across all relevant areas of 5 6 the knowledge generation and knowledge translation process. The research that was 7 undertaken within the separate work packages not only shed light on current measures in 8 9 place to limit dissemination bias but also gathered the views of key stakeholders on the 10 possible barriers to transparency in medical research. Thus, based on the evidence collated 11 12 throughout the first phase of OPEN, a draft set of preliminary recommendations for each 13 14 stakeholder group was developed. These were then discussed with the OPEN Consortium, 15 the advisoryFor board of OPEN,peer and with review selected external key only stakeholders to capture the 16 17 different perspectives. 18 Instead of solely describing aspects of the general problem of dissemination bias, in addition 19 20 to the general recommendations, we developed several targeted and concrete 21 recommendations, which specifically address each key stakeholder group and thus, as a 22 23 whole, offer a holistic approach to address the problem. Further, due to the discussions held 24 on topics such as possible opposition, resources required, and implementability, we were 25 26 able to go beyond generic recommendations. The detailed rationales and explanations for 27 these recommendations will be made available soon in separate articles and through the 28 29 OPEN website (www.openproject.eu). By means of this dissemination strategy, we hope to 30 expedite the widespread dissemination and uptake of OPEN recommendations by the 31 32 relevant key groups and institutions and to thus initiate the implementation of more adequate

33 http://bmjopen.bmj.com/ mechanisms to reduce and prevent dissemination bias in the long run. 34 35 36 Although we developed separate sets of recommendations for industry and public funding 37 38 agencies, it is important to highlight that we agreed that the same ethical considerations 39 40 apply to all research funders. After all, the interests of patients and the public in general 23 41 should always override commercial or other interests . During workshop discussions it was on September 23, 2021 by guest. Protected copyright. 42 43 noted that the tendency of forprofit companies and also (to some extent) other stakeholder 44 groups such as medical licensing bodies, research institutions or policymakers and 45 46 legislators to place the protection of commercial interest over the public interest in access to 47 data from clinical trials does not seem to be proportional 24 25. Despite the fact that companies 48 49 are investing large sums of money to develop new treatments, the group agreed that 50 intellectual and commercial property rights are overall less important than basic human rights 51 52 with regard to information and health. 53 54 55 As already raised, it is obvious that some of our recommendations are relevant for more than 56 one stakeholder group. The recommendations given for benefit assessment institutions, for 57 58 example, are also applicable for regulators; however, they are given specifically for benefit 59 60 13 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml Page 15 of 55 BMJ Open BMJ Open: first published as 10.1136/bmjopen-2014-006666 on 5 May 2015. Downloaded from 1 2 3 assessment agencies since work conducted within OPEN revealed that cooperation between 4 agencies, and the transparency of methodology used, could particularly be improved within 5 6 this key group. Similarly, recommendations with respect to data sharing, e.g. given for 7 research ethics committees and legislators, are again also relevant for other key groups. It 8 9 needs to be kept in mind, though, that data sharing was not the focus of the OPEN project. 10

33 http://bmjopen.bmj.com/ important for securing access to the full evidence. The broader issue of waste in research, 34 35 first introduced by Chalmers & Glasziou in 2009 26, has recently been explored in a series of 36 articles including among others waste due to inaccessible research 27, and waste due to 37 38 incomplete or unusable research reports 28. 39 40

41 The recommendations from the OPEN project coincide with a number of important decisions on September 23, 2021 by guest. Protected copyright. 42 43 at the European Medicines Agency (EMA) and the European Parliament, namely the 44 finalisation and implementation of the EMA’s data sharing policy. EMA and the European 45 46 Parliament, which approved the new European clinical trials regulation on April 2nd, 2014, 47 have received praise for their leadership on facilitating access to clinical study reports 29. 48 49 However, the recently released draft terms for use of these reports included several 50 important restrictions such as strict confidentiality, access to registered users and onscreen 51 30 31 52 viewing only . Certainly, this would impede the widespread and most effective use of 53 these important data sources. Criticism was voiced by academia to which EMA responded 54 32 55 with a press release announcing more userfriendly amendments . As of now, it is unclear 56 what the final policy will look like. 57 58 59 60 14 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml BMJ Open Page 16 of 55 BMJ Open: first published as 10.1136/bmjopen-2014-006666 on 5 May 2015. Downloaded from 1 2 3 We hope that the findings from the OPEN project will contribute to these decisions and 4 implementation processes, emphasising once more the needs of academia and the public for 5 6 unrestricted access to an unbiased body of evidence of complete results of all clinical trials. 7

8 9 Copyright: 10 The Corresponding Author has the right to grant on behalf of all authors and does grant on 11 12 behalf of all authors, an exclusive licence (or non exclusive for government employees) on a 13 14 worldwide basis to the BMJ Publishing Group Ltd, and its Licensees to permit this article (if 15 accepted) toFor be published peer in BMJ editions review and any other BMJPGL only products and to exploit all 16 17 subsidiary rights, as set out in our licence (bmj.com/advice/copyright.shtml). 18 19 20 Competing interest statement: 21 All authors have completed the ICMJE uniform disclosure form at 22 23 www.icmje.org/coi_disclosure.pdf and declare: the OPEN project has received funding from 24 the European Union under grant agreement n° 285453; Davina Ghersi reports being the 25 26 Chair of the WHO Advisory Panel on the International Clinical Trials Registry Platform; Jos 27 Kleijnen reports grants from Amgen, GSK, Sanofi, Biomarin, MSD, Roche and Bayer outside 28 29 the submitted work; Joerg Meerpohl reports being a member of the WHO Advisory Panel on 30 the International Clinical Trials Registry Platform; Elizabeth Wager reports training and 31 32 consultancy activities for various companies and publishers and training activities for various

33 http://bmjopen.bmj.com/ universities; Robert Wolff reports grants from Amgen, GSK, Sanofi, Biomarin, MSD, Roche 34 35 and Bayer outside the submitted work; no financial relationships with any organisations that 36 might have an interest in the submitted work in the previous three years reported by other 37 38 authors; no other relationships or activities that could appear to have influenced the 39 40 submitted work.

41 on September 23, 2021 by guest. Protected copyright. 42 43 Details of contributors: 44 JM and LS drafted this manuscript based on the discussions and results of the OPEN 45 46 recommendations workshop. All named authors were involved in the work packages that led 47 to the development of the draft recommendations. All had actively participated in the OPEN 48 49 workshop and repeatedly commented and revised the draft manuscript; they are listed in 50 alphabetical order. All authors approved the final manuscript. People listed as part of the 51 52 OPEN project consortium contributed to the various work packages of the OPEN project on 53 the basis of which the draft recommendations were developed. 54 55 56 Transparency declaration: 57 58 59 60 15 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml Page 17 of 55 BMJ Open BMJ Open: first published as 10.1136/bmjopen-2014-006666 on 5 May 2015. Downloaded from 1 2 3 The lead author (Joerg J Meerpohl) affirms that the manuscript is an honest, accurate, and 4 transparent account of the work being reported. No important aspects have been omitted. 5 6 7 Data sharing statement: 8 9 No additional data are available. 10

11 12 Ethical approval: 13 14 Not required. 15 For peer review only 16 17 Details of funding and statement of the independence of researchers from funders: 18 The research leading to these results has received funding from the European Union 19 20 Seventh Framework Programme (FP7/2007-2013) under grant agreement n° 285453. The 21 funder had no influence in the development of the recommendations and/or the writing of the 22 23 manuscript. 24 25 26 27 28 29 30 31 32

33 http://bmjopen.bmj.com/ 34 35 36 37 38 39 40

41 on September 23, 2021 by guest. Protected copyright. 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 16 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml BMJ Open: first published as 10.1136/bmjopen-2014-006666 on 5 May 2015. Downloaded from 17 Page 18 of 55 BMJ Open http://bmjopen.bmj.com/ on September 23, 2021 by guest. Protected copyright. For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml For peer review only part of the funding until a project’s results are adequately disseminated (recommendation). (recommendation). disseminated adequately are results project’s a until funding the of part project’s final report, but instead is followed up until all agreed data have been disseminated (recommendation). (recommendation). disseminated been have data agreed all up until followed is instead but report, final project’s disseminated, regardless of the nature of findings, in all funding contracts (strong recommendation). recommendation). (strong contracts funding all in findings, of nature the of regardless disseminated, participant (strong recommendation). recommendation). (strong participant clinical trials publicly accessible (strong recommendation). recommendation). (strong accessible trials publicly clinical order to keep an accurate record of funded projects and publication outcomes (recommendation). (recommendation). outcomes publication and projects funded of record accurate keepan to order proposals (strong recommendation). recommendation). (strong proposals all calls for proposals (strong recommendation). recommendation). (strong proposals for calls all committees) available upon the publication/dissemination of results (strong recommendation). recommendation). (strong results of publication/dissemination upon the available committees) 2.3 2.3 Pharmaceutical and devices companies should make their trial protocols + amendments (as submitted to research ethics 2.2 2.2 Pharmaceutical and devices companies should register all clinical trials in a public registry before the recruitment of the first 2.1 2.1 Pharmaceutical and medical device companies should make their policies concerning the dissemination of methods and results of Pharmaceutical and device companies companies device and Pharmaceutical 1.6 1.6 in database and on of disseminated their all accessible grants were create awarded a results should how publicly agencies Funding 1.5 1.5 Funding agencies should consider providing incentives for researchers who disseminate their results, or, alternatively, withhold a 1.4 1.4 Funding agencies should implement measures to ensure that the evaluation process of funded projects does not end with the 1.3 1.3 Funding agencies should include the requirement for grantees to explicitly declare that the results of funded research will be 1.2 1.2 Funding agencies should include the requirement for grantees to provide a dissemination plan for funded projects in all calls for 1.1 1.1 in results research of dissemination the for requirement and the bias dissemination a on statement include should agencies Funding Funding agencies agencies Funding Table 1: Targeted recommendations by key stakeholder stakeholder group key by recommendations Targeted 1: Table No. Recommendation 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 BMJ Open: first published as 10.1136/bmjopen-2014-006666 on 5 May 2015. Downloaded from 18

BMJ Open http://bmjopen.bmj.com/ on September 23, 2021 by guest. Protected copyright. ) (strong recommendation). recommendation). (strong ) For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml 21 For peer impact of for the the theassessment and search trials practices those concerning (especially SRs for performing review only 20 20 33 rialists should disseminate complete summary results (as soon as possible, but no later than 12 months) from all clinical trials they clinical from but all later months) than no 12 resultspossible, soon as (as summary complete disseminate should rialists recommendation). recommendation). conducted and provide access to their clinical study reports (CSR) (for clinical trials) upon request (recommendation). (recommendation). request trials) upon clinical (for (CSR) reports study clinical their to access provide and conducted of dissemination bias bias of dissemination recommendation). recommendation). recommendation). recommendation). conduct, i.e, through journal publications and results posting of results in registers (strong recommendation). (strong registers in results of posting results and publications journal through i.e, conduct, Trialists Trialists should make trial recommendation). (strong publication journal the with material appendix/supporting protocols publicly available both within the register where the trial is registered and as best practices bestpractices dissemination bias (strong recommendation). recommendation). (strong bias dissemination 6.1 6.1 Research ethics committees should require the registration of all clinical trials before the recruitment of the first participant (strong Research ethics committees (RECs) (RECs) committees ethics Research 5.3 5.3 5.2 5.2 T 5.1 5.1 recommendation). (strong first participant the of recruitment the before to conduct plan trial they every register should Trialists Researchers II: Trialists Trialists II: Researchers 4.2 4.2 Systematic reviewers should make SR protocols and the results of SRs informing clinical care publicly available (strong 4.1. 4.1. Researchers conducting systematic reviews (SRs), metaanalyses (MAs) and network metaanalyses (NMAs) should follow the Researchers I: Systematic reviewers reviewers Systematic I: Researchers 3.3 3.3 recommendation). (strong trials clinical all of results summary complete of the dissemination mandate should institutions Research 3.2 3.2 Research institutions should not accept any funding that includes clauses that prevent the dissemination of data (strong 3.1 3.1 Research institutions should provide guidance and training about the implications of and possible measures for avoiding Research institutions institutions Research 2.4 2.4 Pharmaceutical and devices companies should publish/disseminate complete summary results (aggregate data) of all trials Page 19 of 55 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 BMJ Open: first published as 10.1136/bmjopen-2014-006666 on 5 May 2015. Downloaded from 19 Page 20 of 55 BMJ Open http://bmjopen.bmj.com/ on September 23, 2021 by guest. Protected copyright. For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml For peer review only publication regardless of the direction of their findings or their sources of funding (strong recommendation). recommendation). (strong funding of sources their or findings their of the direction of regardless publication (strong recommendation). recommendation). (strong other bibliographical databases, data repositories) in trial registry entries (strong recommendation). recommendation). (strong entries registry trial in repositories) data databases, bibliographical other (recommendation). (recommendation). recommendation). recommendation). (strong recommendation). recommendation). (strong recommendation). recommendation). (recommendation). (recommendation). 8.4 8.4 Journal editors should publish editorials and commentaries about the problem of dissemination bias and the benefits of trial 8.3 8.3 Journals should check all submitted manuscripts against study protocols and/or trial registry entries to detect selective reporting 8.2 8.2 recommendation). (strong forpublication requirement a registration trial make journals should All 8.1 8.1 Journal editors and publishers should remove all barriers to publishing negative or inconclusive studies and consider studies for Journal Editors / Publishers /Publishers Editors Journal 7.6 7.6 recommendation). (strong Registries Trials Clinical for Standards WHOInternational the with comply should trial registries All 7.5 7.5 of theand registries enable (e.g.,encourage should and Trial data inclusion tosources PubMed, links other permanent publications 7.4 7.4 Initiatives should be developed that enable nonEnglish speakers to use the information contained in trial registers 7.3 7.3 Trial registries outside Englishspeaking should countries facilitate the registration for process speaking nonEnglish trialists (strong 7.2 7.2 (recommendation). studies human observational of registration the encourage and enable should registries Trial 7.1 7.1 recommendation). (strong results summary aggregate of reporting the enable should registries Trial Trial registries registries Trial 6.4 6.4 results their ofstudy the dissemination describing reports annual provide applicants that require should committees ethics Research 6.3 6.3 Research ethics committees should encourage applicants to share anonymized individual patient level data on request (strong 6.2 6.2 Research ethics committees should require that applicants commit to making complete summary results publicly available 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 BMJ Open: first published as 10.1136/bmjopen-2014-006666 on 5 May 2015. Downloaded from 0 2 BMJ Open http://bmjopen.bmj.com/ on September 23, 2021 by guest. Protected copyright. For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml For peer review only recommendation). recommendation). further (unpublished) data and to foster data sharing (strong recommendation). recommendation). (strong sharing data foster to and data (unpublished) further achieve better transparency and understanding (strong recommendation). recommendation). (strong understanding and transparency better achieve requirements are sanctioned (strong recommendation). recommendation). (strong sanctioned are requirements of of a trial is made in available the same registry that the trial was registered in, in a timely fashion (i.e.,1 year after study completion recommendation). (strong database EU the in registered forstudies all or inactivity) approved approved by the research ethics committee, including the potential protocol amendments, is submitted and made publicly available recommendation). (strong document as searchable a registered in an EU database which is publicly accessible (strong recommendation). recommendation). (strong accessible publicly is which database EU an in registered devices and biologicals (strong recommendation). recommendation). (strong biologicals and devices reporting the same findings (strong recommendation). recommendation). (strong findings same the reporting registration (recommendation). (recommendation). registration 10.3 10.3 Benefit assessment institutions should use the full evidence base available for an intervention for their assessments (strong 10.2 10.2 Benefit assessment institutions should aim for a higher degree of collaboration between institutions to facilitate the detection of 10.1 10.1 Benefit assessment institutions should make their methods and processes of benefit assessment publicly available in order to Benefit assessment institutions institutions assessment Benefit 9.5 9.5 Responsible authorities (such as EMA for drugs) should ensure that trial sponsors failing to comply with such result submission 9.4 9.4 Responsible authorities (such as EMA for drugs) should mandate that the full report including all results (e.g., clinical study report) 9.3 9.3 full an mandate in the a that, registration database, protocol fortrial’s upon EU (such should as authorities drugs) EMA Responsible 9.2 9.2 Responsible authorities (such as the EMA for drugs) should mandate that all clinical trials in humans falling under their remit are 9.1 9.1 Regulation of pharmaceutical products should be extended to cover other therapeutic and diagnostic agents such as medical Regulatory agencies agencies Regulatory 8.5 8.5 Journals should for check of publication redundant results by using software textmatching and peer asking reviewers about papers

Page 21 of 55 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 BMJ Open: first published as 10.1136/bmjopen-2014-006666 on 5 May 2015. Downloaded from 21 Page 22 of 55 BMJ Open http://bmjopen.bmj.com/ on September 23, 2021 by guest. Protected copyright. For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml For peer review only decisionmaking process (strong recommendation). recommendation). (strong process decisionmaking (registered in the EU database) (strong recommendation). recommendation). (strong database) EU the in (registered information (strong recommendation). recommendation). (strong information results (disclosure of full protocols and full clinical study reports): reports): study clinical full and protocols full of (disclosure results a. b. evidence requested all submit to manufacturers for obligation a legal c. databases to EMA access public recommendation) (strong reports full study and to protocols access public deemed incomplete (e.g., no adequate proof of benefit based on incomplete data set) (strong recommendation). recommendation). (strong set) data incomplete on based benefit of proof adequate no (e.g., incomplete deemed Organization WorldHealth WHO = Union; European = EU Agency; Medicines European = EMA 11.4 11.4 Legislators should ensure that the raw data (anonymized individual patient data) are made publicly available for all clinical studies 11.3 11.3 Legislators should institute a legal obligation for manufacturers to submit all data and other required information for the formal 11.2 11.2 proprietary confidential/ commercially NOT are public the and of patients tohealth the related data that all ensure should Legislators 11.1 11.1 recommendation). (strong mandatory humans in trials of clinical registration prospective make should Legislators Legislators Legislators 10.5 10.5 Benefit assessment institutions should request from legislators the following items which will allow the consideration of all study 10.4 10.4 Benefit assessment institutions should specify their course of action if they find that the evidence base for an assessment is 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 Page 23 of 55 BMJ Open BMJ Open: first published as 10.1136/bmjopen-2014-006666 on 5 May 2015. Downloaded from 1 2 3 Box 1: Definition of a clinical trial 4 5 Clinical trial: 6 A clinical trial is any research study that prospectively assigns human participants or 7 groups of humans to one or more healthrelated interventions to evaluate the effects 8 on health outcomes. Clinical trials may also be referred to as interventional trials. 9 Interventions include but are not restricted to drugs, cells and other biological 10 products, surgical procedures, radiologic procedures, devices, behavioural 11 treatments, processofcare changes, preventive care, etc. This definition includes 12 34 13 Phase I to Phase IV trials . 14 15 For peer review only 16 17 18 19 Box 2: Information on trial methodology required to comprehensively assess a trial 20 21 Design, Conduct and Analysis: 22 study protocol (+ all amendments) 23 statistical analysis plan (+ all amendments) 24 25 26 27 28 29 Box 3: Levels of results data and formats/products that allow access 30 31 Data Format / Product 32

33 Aggregate summary data • Grey literature http://bmjopen.bmj.com/ 34 • Incomplete • Scientific abstract published in journal 35 • Complete • Full article published in journal 36 Individual participant data • Trial registry and results database 37 • Incomplete • Clinical study reports 38 • Complete 39 • Case report forms

40 • Internal communication (company or

41 nonprofit organization) on September 23, 2021 by guest. Protected copyright. 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml 22 BMJ Open Page 24 of 55 BMJ Open: first published as 10.1136/bmjopen-2014-006666 on 5 May 2015. Downloaded from 1 2 3 Figure 1: Example of framework worksheet used for discussion of recommendations 4 5 6 7 8 9 10 11 12 13 14 15 For peer review only 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32

33 http://bmjopen.bmj.com/ 34 35 36 37 38 39 40

41 on September 23, 2021 by guest. Protected copyright. 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml 24 BMJ Open Page 26 of 55 BMJ Open: first published as 10.1136/bmjopen-2014-006666 on 5 May 2015. Downloaded from 1 2 3 Appendices: 4 5 6 7 Appendix 1: OPEN Consortium 8 9 Contributor Participating Institution 10 11 Antes, Gerd German Cochrane Centre, Medical Center University of Freiburg, 12 Freiburg, Germany 13 Bassler, Dirk Center for Pediatric Clinical Studies, University Medical Center 14 Tuebingen, Germany 15 Bertelè, VittorioFor peerIRCCS – Istituto review di Ricerche Farmacologiche only “Mario Negri”, Milan, Italy 16 17 Bonfill, Xavier The Clinical Epidemiology & Public Health Department at the Hospital de 18 la Santa Creu i Sant Pau, Spain 19 Bouesseau, Marie World Health Organization, Geneva, Switzerland 20 Charlotte 21 Boutron, Isabelle INSERM U738 research unit, Paris Descartes University, Paris, France 22 Briel, Matthias Center for Pediatric Clinical Studies, University Medical Center 23 Tuebingen, Germany & Basel Institute for Clinical Epidemiology and 24 25 Biostatistics, University Hospital of Basel, Switzerland 26 Gallus, Silvano Department of Epidemiology, IRCCS Istituto di Ricerche Farmacologiche 27 “Mario Negri”, Milan, Italy 28 Garattini, Silvio IRCCS Istituto di Ricerche Farmacologiche “Mario Negri”, Milan, Italy 29 Ghersi, Davina University of Sydney, Australia 30 Karam, Ghassan World Health Organization, Geneva, Switzerland 31 32 Kleijnen, Jos Kleijnen Systematic Reviews Ltd., York, United Kingdom & School for

33 Public Health and Primary Care, Maastricht University, Maastricht, The http://bmjopen.bmj.com/ 34 Netherlands 35 Kulig, Michael Federal Joint Committee, Berlin, Germany 36 La Vecchia, Carlo Department of Clinical Sciences and Community Health, University of 37 Milan, Milan Italy 38 Lang, Britta German Cochrane Centre, Medical Center University of Freiburg, 39 40 Freiburg, Germany Littmann, Jasper CELLS (Centre for Ethics and Law in Life Sciences), Hannover Medical 41 on September 23, 2021 by guest. Protected copyright. 42 Scholl, Hannover, Germany 43 Malicki, Mario University of Split School of Medicine, Split, Croatia 44 Marusic, Ana University of Split School of Medicine, Split, Croatia 45 Meerpohl, Joerg German Cochrane Centre, Medical Center University of Freiburg, 46 Freiburg, Germany 47 48 Mueller, Katharina Center for Pediatric Clinical Studies, University Medical Center 49 Felicitas Tuebingen, Germany 50 Murisic, Bojana Department of Epidemiology, IRCCS Istituto di Ricerche Farmacologiche 51 “Mario Negri”, Milan, Italy 52 Nolting, Alexandra Federal Joint Committee, Berlin, Germany 53 Pardo, Hector The Clinical Epidemiology & Public Health Department at the Hospital de 54 55 la Santa Creu i Sant Pau, Spain 56 Perleth, Matthias Federal Joint Committee, Berlin, Germany 57 Ravaud, Philippe INSERM U738 research unit, Paris Descartes University, Paris, France 58 Reis, Andreas World Health Organization, Geneva, Switzerland 59 Schell, Lisa German Cochrane Centre, Medical Center University of Freiburg, 60

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml 25 Page 27 of 55 BMJ Open BMJ Open: first published as 10.1136/bmjopen-2014-006666 on 5 May 2015. Downloaded from 1 2 3 Freiburg, Germany 4 Schmucker, Christine German Cochrane Centre, Medical Center University of Freiburg, 5 Freiburg, Germany 6 Schwarzer, Guido Institute for Medical Biometry and Statistics, Medical Center – University 7 8 of Freiburg, Freiburg, Germany 9 Strech, Daniel CELLS (Centre for Ethics and Law in Life Sciences), Hannover Medical 10 Scholl, Hannover, Germany 11 Trinquart, Ludovic INSERM U738 research unit, Paris Descartes University, Paris, France 12 Urrútia, Gerard The Clinical Epidemiology & Public Health Department at the Hospital de 13 la Santa Creu i Sant Pau, Spain 14 von Elm, Erik German Cochrane Centre, Medical Center University of Freiburg, 15 For peer review only 16 Freiburg, Germany & Cochrane Switzerland, IUMSP, University Hospital 17 Lausanne, Lausanne, Switzerland 18 Wager, Elizabeth Sideview, Princes Risborough, United Kingdom 19 Wolff, Robert Kleijnen Systematic Reviews Ltd., York, United Kingdom 20 21 22 23 24 25 26 27 28 29 30 31 32

33 http://bmjopen.bmj.com/ 34 35 36 37 38 39 40

41 on September 23, 2021 by guest. Protected copyright. 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml 26 BMJ Open Page 28 of 55 BMJ Open: first published as 10.1136/bmjopen-2014-006666 on 5 May 2015. Downloaded from 1 2 3 Appendix 2: OPEN Advisory Board 4 5 Contributor Participating Institution 6 7 Altman, Doug Centre for Statistics in Medicine, University of Oxford, United Kingdom 8 Chan, AnWen Women's College Research Institute, University of Toronto, Canada 9 Dickersin, Kay Johns Hopkins Bloomberg School of Public Health, United States 10 Weber, Wim British Medical Journal 11 12 Song, Fujian School of Medicine, Health Policy and Practice, University of East Anglia, 13 United Kingdom 14 15 For peer review only 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32

33 http://bmjopen.bmj.com/ 34 35 36 37 38 39 40

41 on September 23, 2021 by guest. Protected copyright. 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml 27 Page 29 of 55 BMJ Open BMJ Open: first published as 10.1136/bmjopen-2014-006666 on 5 May 2015. Downloaded from 1 2 3 Appendix 3: Participants at OPEN Meeting 4 5 Contributor Participating Institution 6 7 Antes, Gerd German Cochrane Centre, Institute of Medical Biometry and Medical 8 Informatics, University Medical Center, Freiburg, Germany 9 Bangdiwala, Shrikant UNC Gillings School of Global Public Health, Chapel Hill, United States 10 Bassler, Dirk Center for Pediatric Clinical Studies, University Medical Center 11 Tuebingen, Germany 12 13 Boutron, Isabelle INSERM U738 research unit, Paris Descartes University, Paris, France 14 Collazo, Eduardo Lopez Instituto de Salud Carlos III, Barcelona, Spain 15 Doppelfeld, ElmarFor peerEuropean Network review of Research Ethics onlyCommittees 16 Dwan, Kerry University of Liverpool, United Kingdom 17 Frost, Robert GlaxoSmithKline plc, London, United Kingdom 18 Gallus, Silvano Department of Epidemiology, IRCCS Istituto di Ricerche Farmacologiche 19 20 “Mario Negri”, Milan, Italy 21 Ghersi, Davina University of Sydney, Australia 22 Gøtzsche, Peter Nordic Cochrane Centre, Copenhagen, Denmark 23 Hrynaszkiewicz, Iain Faculty of 1000 Ltd, United Kingdom 24 Jena, Susanne German Register of Clinical Trials, Freiburg, Germany 25 Karam, Ghassan World Health Organization, Geneva, Switzerland 26 27 Kienegger, Manuela Danube University Krems, Austria 28 Kirkham, Jamie University of Liverpool, United Kingdom 29 Kleijnen, Jos Kleijnen Systematic Reviews Ltd., York, United Kingdom & School for 30 Public Health and Primary Care, Maastricht University, Maastricht, The 31 Netherlands 32 La Vecchia, Carlo Department of Clinical Sciences and Community Health, University of 33 http://bmjopen.bmj.com/ Milan, Milan Italy 34 35 Malicki, Mario University of Split, School of Medicine, Split, Croatia 36 Marusic, Ana University of Split, School of Medicine, Split, Croatia 37 Meerpohl, Joerg German Cochrane Centre, Institute of Medical Biometry and Medical 38 Informatics, University Medical Center, Freiburg, Germany 39 Mueller, Katharina Center for Pediatric Clinical Studies, University Medical Center 40 Felicitas Tuebingen, Germany

41 on September 23, 2021 by guest. Protected copyright. 42 Panteli, Dimitra Technical University of Berlin, Germany 43 Pardo, Hector The Clinical Epidemiology & Public Health Department at the Hospital de 44 la Santa Creu i Sant Pau, Spain 45 Schell, Lisa German Cochrane Centre, Institute of Medical Biometry and Medical 46 Informatics, University Medical Center, Freiburg, Germany 47 Schmucker, Christine German Cochrane Centre, Institute of Medical Biometry and Medical 48 Informatics, University Medical Center, Freiburg, Germany 49 50 Strech, Daniel CELLS (Centre for Ethics and Law in Life Sciences), Hannover Medical 51 Scholl, Hannover, Germany 52 Trinquart, Ludovic INSERM U738 research unit, Paris Descartes University, Paris, France 53 Urrútia, Gerard The Clinical Epidemiology & Public Health Department at the Hospital de 54 la Santa Creu i Sant Pau, Spain 55 Van Noord, Megan Danube University Krems, Austria 56 57 Wager, Elizabeth Sideview, Princes Risborough, United Kingdom 58 Wahler, Steffen Correvio International Sàrl, Geneva, Switzerland 59 Weber, Wim British Medical Journal 60

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml 28 BMJ Open Page 30 of 55 BMJ Open: first published as 10.1136/bmjopen-2014-006666 on 5 May 2015. Downloaded from 1 2 3 Wieseler, Beate Institute for Quality and Efficiency in Healthcare, Cologne, Germany 4 5 6 7 8 9 10 11 12 13 14 15 For peer review only 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32

33 http://bmjopen.bmj.com/ 34 35 36 37 38 39 40

41 on September 23, 2021 by guest. Protected copyright. 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml 29 BMJ Open: first published as 10.1136/bmjopen-2014-006666 on 5 May 2015. Downloaded from 30

implemented on a large onscale a implemented resources would be required if this recommendation is thisrecommendation required if be would resources At first only limited resources required; however, relevant however, required; resourcesonly limited first At No obvious downsides; potential to have important impact to have potential downsides; Noobvious Remark Easy to start raising awareness, but comprehensive strategy more strategy comprehensive but awareness, raising start Easyto implement to difficult Probably no strong opposition, but likelihood of “opposition” due to to due of “opposition” likelihood opposition, but strong no Probably of resources key stakeholders passiveness/limited Likely limited effectiveness in the short run, but potentially high short potentially run, in but the effectiveness limited Likely long the in run effectiveness BMJ Open http://bmjopen.bmj.com/

x x x

x Yes No Yes Yes No Yes Yes No Yes Yes No Yes Yes No Yes Judgment Judgment

on September 23, 2021 by guest. Protected copyright. For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml For peer review only Easily implementable implementable Easily implementation implementation Limited resources required for required resources Limited Low likelihood of strong opposition opposition of strong likelihood Low Benefits clearly outweigh clearly downsides outweigh Benefits High confidence in effectiveness effectiveness in confidence High Category Category We strongly recommend that recommend strongly We processtranslation and theknowledgegeneration in stakeholders all it. reduce to measures (publicationand bias aboutbias) dissemination awareness raise Supplementary File 1: Decision tables for general recommendations recommendations File general for Decision tables 1: Supplementary #1: Recommendation Page 31 of 55 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 BMJ Open: first published as 10.1136/bmjopen-2014-006666 on 5 May 2015. Downloaded from Page 32 of 55 31

BMJ Open http://bmjopen.bmj.com/ Recommendation Recommendation

Should also be extended to patients who should be made aware of dissemination bias. of dissemination beaware made should patients who extended to also be Should Not very concrete recommendation; room for interpretation. interpretation. for room veryrecommendation; concrete Not Remark: Remark:

Strong recommendation Strong recommendation

on September 23, 2021 by guest. Protected copyright. For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml For peer review only

Strength of recommendation:of Strength

likely require substantial resources. substantialresources. require likely A comprehensive dissemination and implementation plan would implementation and comprehensive dissemination A outweigh downsides. downsides. outweigh unambiguous beneficial balance, overall benefits would clearly overall benefits would balance, beneficial unambiguous Although some specific recommendations might not have ahave not might some recommendations specific Although Remark Likely tailored dissemination and implementation strategies wouldstrategies and implementation dissemination tailored Likely andimplementation on monitoring efforts including required be effectiveness. Likely that some key stakeholder groups or some key stakeholders or stakeholders groups some key key stakeholder some that Likely and/or disseminate to notgoing and opposed aregroup a within OPEN all recommendations. someor implement If OPEN recommendations were fully disseminated and disseminated fully were Ifrecommendations OPEN likely reducedmost be would bias dissemination implemented, effectively. BMJ Open http://bmjopen.bmj.com/

x x x

x x

Yes No Yes Yes No Yes Yes No Yes Yes No Yes Yes No Yes Judgment Judgment

on September 23, 2021 by guest. Protected copyright. For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml For peer review only Easily implementable implementable Easily implementation implementation Limited resources required for required resources Limited Low likelihood of strong opposition opposition of strong likelihood Low Benefits clearly outweigh clearly downsides outweigh Benefits High confidence in effectiveness effectiveness in confidence High Category Category We strongly recommend that recommend strongly We processtranslation and theknowledgegeneration in stakeholders all respectivegroup. their of amongst key stakeholders recommendations of OPEN targeted facilitateimplementation and disseminate Recommendation #2: Recommendation Page 33 of 55 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 BMJ Open: first published as 10.1136/bmjopen-2014-006666 on 5 May 2015. Downloaded from Page 34 of 55 33 BMJ Open http://bmjopen.bmj.com/ Recommendation Recommendation

dissemination bias. dissemination and implementation plans, OPEN recommendations will be not helpful in reducing inreducing will not helpful recommendations be OPEN plans, implementation and but obviously key recommendation amongst OPEN recommendations. Without dissemination dissemination Without amongst recommendations. OPEN obviouslykey recommendation but Strong recommendation despite concerns around opposition, resources and implementability; and resourcesopposition,around concerns despite recommendation Strong Remark: Remark:

Strong recommendation Strong recommendation

on September 23, 2021 by guest. Protected copyright. For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml For peer review only

Strength of recommendation:of Strength

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 BMJ Open: first published as 10.1136/bmjopen-2014-006666 on 5 May 2015. Downloaded from 34 maintain such initiatives. suchinitiatives. maintain resources likely required to establish, promote, expand and and expand promote, establish, to required likelyresources resources. Technical consequences not fully clear; relevant clear; fully relevant not consequences Technicalresources. Promotion and political support possible without possible relevant politicalsupport and Promotion supporting such initiatives. such supporting No obvious downsides in promoting trial registration and and registration trial promoting indownsides Noobvious Remark No serious barrier to both political and financial support was was financial and tosupport both barrier political Noserious identified. Promotion and support in itself likely not yet highly effective in yethighlyeffectiveitself in in likely support not and Promotion areinitiatives respective if bias, but dissemination reducing longerthe likelyimpact in relevant maintained and implemented run. BMJ Open http://bmjopen.bmj.com/

x x

Yes No Yes likely. quitefieldin groupsactive by this opposition Some x Yes No x Yes Yes No Yes Yes No Yes Yes No Yes Judgment Judgment on September 23, 2021 by guest. Protected copyright. For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml For peer review only Easily implementable implementable Easily implementation implementation Limited resources required for required resources Limited Low likelihood of strong opposition opposition of strong likelihood Low Benefits clearly outweigh clearly downsides outweigh Benefits High confidence in effectiveness effectiveness in confidence High Category Category We strongly recommend that recommend strongly We processtranslation and theknowledgegeneration in stakeholders all registries. trial searchesacross multiple facilitate initiatives that and registrationsupport trial promote Recommendation #3: Recommendation Page 35 of 55 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 BMJ Open: first published as 10.1136/bmjopen-2014-006666 on 5 May 2015. Downloaded from Page 36 of 55 35 BMJ Open http://bmjopen.bmj.com/ r of supporting and thereby hopefully establishing/maintaining in the longer runlonger in establishing/maintaining the therebyhopefully and supporting ofr u Recommendation Recommendation

In the future need for similar initiatives that allow simultaneous access to results databases. databases. accessto results simultaneousallow that for initiatives similar future need the In

Likely need for alternative initiatives to avoidmonopoly?! to forinitiatives alternative need Likely Registry Platform (ICRTP)). (ICRTP)). Platform Registry respective initiatives (one example of such a platform is the WHO International Clinical Trials Clinical International WHO the a is platform(one such ofexample initiatives respective favo Wide range of support is possible. Although not very concrete, strong recommendation in in recommendation strongvery not concrete, Although is possible. of support range Wide Remark: Remark:

Strong recommendation Strong recommendation

on September 23, 2021 by guest. Protected copyright. For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml For peer review only

Strength of recommendation:of Strength

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 BMJ Open: first published as 10.1136/bmjopen-2014-006666 on 5 May 2015. Downloaded from 36 conduct and regularly update all these activities (SRs).activitiesallthese andupdate regularly conduct resources likely required to increase and maintain efforts to actually effortsmaintainto increase and to required likelyresources Political support possible without relevant resources. Relevant Relevant resources.relevant without possible Politicalsupport conducted research for aresearchforquestion.specific conducted downsides, in particular if efforts are undertaken to consider all consider undertaken to if efforts are in particular downsides, of published trials; overall benefits seem to clearly outweigh to clearly seem outweigh benefits overall trials; ofpublished Risk of spurious conclusions if activities based on very biased subset subset on based ifbiased very activities conclusions ofspurious Risk Remark No serious barrier to both political and financial support was was financial and tosupport both barrier political Noserious identified. No strong opposition to general support likely; however, financial likely; however, support opposition general to Nostrong establish. to difficult morelikelybe to support Support in itself likely not yet highly effective in reducing effectivein yet likelyhighly itselfnot in Support activitiesidentify rigorously if respective but bias, dissemination longerthe impact in likely relevant trials unpublished integrate and run. BMJ Open http://bmjopen.bmj.com/

x x

Yes No Yes Yes No Yes Yes No Yes Yes No Yes Yes No Yes Judgment Judgment on September 23, 2021 by guest. Protected copyright. For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml For peer review only Easily implementable implementable Easily implementation implementation Limited resources required for required resources Limited Low likelihood of strong opposition opposition of strong likelihood Low Benefits clearly outweigh clearly downsides outweigh Benefits High confidence in effectiveness effectiveness in confidence High Category Category We strongly recommend that recommend strongly We processtranslation and theknowledgegeneration in stakeholders all methodology. rigorous with informationfromstudies synthesize activitiessystematically to support Recommendation #4: Recommendation Page 37 of 55 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 BMJ Open: first published as 10.1136/bmjopen-2014-006666 on 5 May 2015. Downloaded from Page 38 of 55 37

BMJ Open http://bmjopen.bmj.com/ r of supporting and thereby increasing such activities. However, relevant resources will be be will resources relevant such However, activities. therebyincreasing and supporting ofr u Recommendation Recommendation

effectiveness. These will require further resources. further require will These effectiveness. Dissemination strategies need to be developed for non-English speaking countries to increase countries to for non-Englishspeaking be developed to strategies need Dissemination required in the long run to conduct and update regularly relevant systematicreviews. regularly and relevant update torun longconduct in the required favo Wide range of support is possible. Although not very concrete, strong recommendation in in recommendation strongvery not concrete, Although is possible. of support range Wide Remark: Remark:

Strong recommendation Strong recommendation

on September 23, 2021 by guest. Protected copyright. For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml For peer review only

Strength of recommendation:of Strength

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 Page 39 of 55 BMJ Open BMJ Open: first published as 10.1136/bmjopen-2014-006666 on 5 May 2015. Downloaded from 1 2 3 References 4 5 6 1. Strech D. Normative arguments and new solutions for the unbiased registration and publication of 7 clinical trials. J Clin Epidemiol 2012;65(3):276-81. 8 2. Dwan K, Gamble C, Williamson PR, et al. Systematic review of the empirical evidence of study 9 publication bias and outcome reporting bias - an updated review. PLoS One 10 2013;8(7):e66844. 11 3. Antes G, Chalmers I. Under-reporting of clinical trials is unethical. Lancet 2003;361(9362):978-9. 12 4. Dickersin K. The existence of publication bias and risk factors for its occurrence. JAMA 13 1990;263(10):1385-89. 14 15 5. Muller KF, BrielFor M, D'Amario peer A, et al. Defining review publication bias: protocolonly for a systematic review of 16 highly cited articles and proposal for a new framework. Syst Rev 2013;2:34. 17 6. Song F, Parekh S, Hooper L, et al. Dissemination and publication of research findings: an updated 18 review of related biases. Health Technol Assess 2010;14(8):iii, ix-xi, 1-193. 19 7. Brown T. It's time for alltrials registered and reported. Cochrane Database Syst Rev 20 2013;4:ED000057. 21 8. Doshi P, Dickersin K, Healy D, et al. Restoring invisible and abandoned trials: a call for people to 22 publish the findings. BMJ 2013;346:f2865. 23 9. Groves T. What does oseltamivir do, and how will we know? BMJ 2013;347:f4687. 24 10. Nisen P, Rockhold F. Access to patient-level data from GlaxoSmithKline clinical trials. N Engl J Med 25 2013;369(5):475-8. 26 11. Littmann J, Antes G, Strech D, et al. To overcome failure to publish negative findings: The OPEN 27 28 project. Maturitas 2013. 29 12. Mueller KF, Meerpohl JJ, Briel M, et al. Detecting, quantifying and adjusting for publication bias in 30 meta-analyses: protocol of a systematic review on methods. Syst Rev 2013;2:60. 31 13. Schmucker C, Bluemle A, Briel M, et al. A protocol for a systematic review on the impact of 32 unpublished studies and studies published in the gray literature in meta-analyses. Syst Rev

33 2013;2:24. http://bmjopen.bmj.com/ 34 14. Portalupi S, von Elm E, Schmucker C, et al. Protocol for a systematic review on the extent of non- 35 publication of research studies and associated study characteristics. Syst Rev 2013;2:2. 36 15. Wager E, Williams P, Project Overcome failure to Publish nEgative fiNdings C. "Hardly worth the 37 effort"? Medical journals' policies and their editors' and publishers' views on trial registration 38 and publication bias: quantitative and qualitative study. BMJ 2013;347:f5248. 39 16. Andrews J, Guyatt G, Oxman AD, et al. GRADE guidelines: 14. Going from evidence to 40 recommendations: the significance and presentation of recommendations. J Clin Epidemiol

41 on September 23, 2021 by guest. Protected copyright. 42 2013;66(7):719-25. 43 17. Andrews JC, Schunemann HJ, Oxman AD, et al. GRADE guidelines: 15. Going from evidence to 44 recommendation-determinants of a recommendation's direction and strength. J Clin 45 Epidemiol 2013;66(7):726-35. 46 18. Hirsch L. Trial registration and results disclosure: impact of US legislation on sponsors, 47 investigators, and medical journal editors. Curr Med Res Opin 2008;24(6):1683-9. 48 19. Higgins JPT, Altman DG, Sterne JAC. Chapter 8: Assessing risk of bias in included studies. In: 49 Higgins JPT, Green S, eds. Cochrane Handbook for Systematic Reviews of Interventions 50 Version 510 (updated March 2011), 2011. 51 20. Higgins JP, Green S. Cochrane Handbook for Systematic Reviews of Interventions; Version 5.1.0, 52 available from http://www.cochrane-handbook.org. Chichester, UK John Wiley & Sons, Ltd 53 2011. 54 55 21. Sterne JA, Sutton AJ, Ioannidis JP, et al. Recommendations for examining and interpreting funnel 56 plot asymmetry in meta-analyses of randomised controlled trials. BMJ 2011;343:d4002. 57 22. Ofori-Adjei D, Antes G, Tharyan P, et al. Have online international medical journals made local 58 journals obsolete? PLoS Med 2006;3(8):e359. 59 60

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml 38 BMJ Open Page 40 of 55 BMJ Open: first published as 10.1136/bmjopen-2014-006666 on 5 May 2015. Downloaded from 1 2 3 23. Gotzsche PC. Why we need easy access to all data from all clinical trials and how to accomplish it. 4 Trials 2011;12:249. 5 24. Strech D, Littmann J. Lack of proportionality. Seven specifications of public interest that override 6 post-approval commercial interests on limited access to clinical data. Trials 2012;13(1):100. 7 25. Gotzsche PC, Jorgensen AW. Opening up data at the European Medicines Agency. BMJ 8 2011;342:d2686. 9 26. Chalmers I, Glasziou P. Avoidable waste in the production and reporting of research evidence. 10 Lancet 2009;374(9683):86-9. 11 27. Chan AW, Song F, Vickers A, et al. Increasing value and reducing waste: addressing inaccessible 12 13 research. Lancet 2014;383(9913):257-66. 14 28. Glasziou P, Altman DG, Bossuyt P, et al. Reducing waste from incomplete or unusable reports of 15 biomedicalFor research. peer Lancet 2014; 383review(9913):267-76. only 16 29. Regulation of the European Parliament and of the Council on clinical trials on medicinal products 17 for human use, and repealing Directive 2001/20/EC: accessed on 05.06.2014 at: 18 http://www.europarl.europa.eu/sides/getDoc.do?pubRef=-//EP//NONSGML+AMD+A7-2013- 19 0208+291-291+DOC+PDF+V0//EN 2014. 20 30. Torjesen I. European drug agency backtracks on plan to give researchers access to clinical trial 21 reports. BMJ 2014;348. 22 31. Goldacre B, Godlee F, Heneghan C, et al. Open letter: European Medicines Agency should remove 23 barriers to access clinical trial data. BMJ 2014;348:g3768. 24 32. EMA. European Medicines Agency agrees policy on publication of clinical trial data with more 25 26 user-friendly amendments: accessed on 04.08.2014 at: 27 http://www.ema.europa.eu/docs/en_GB/document_library/Press_release/2014/06/WC500 28 168342.pdf, 2014. 29 33. Eden J, Levit L, Berg AO, et al. Finding what works in health care: Standards for systematic 30 reviews: National Academies Press, 2011. 31 34. World Health Organization. International Clinical Trials Registry Platform (ICTRP). Secondary 32 International Clinical Trials Registry Platform (ICTRP). http://www.who.int/ictrp/en/.

33 http://bmjopen.bmj.com/ 34 35 36 37 38 39 40

41 on September 23, 2021 by guest. Protected copyright. 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml 39 Page 41 of 55 BMJ Open BMJ Open: first published as 10.1136/bmjopen-2014-006666 on 5 May 2015. Downloaded from 1 2 3 Appendices: 4 5 6 7 Appendix 1: OPEN Consortium 8 9 Contributor Participating Institution 10 11 12 Antes, Gerd German Cochrane Centre, Medical Center - University of Freiburg, 13 Freiburg, Germany 14 15 Bassler, DirkFor peerCenter for Pediatric review Clinical Studies, Universityonly Medical Center 16 Tuebingen, Germany 17 18 Bertelè, Vittorio IRCCS – Istituto di Ricerche Farmacologiche “Mario Negri”, Milan, Italy 19 20 Bonfill, Xavier The Clinical Epidemiology & Public Health Department at the Hospital de 21 la Santa Creu i Sant Pau, Spain 22 23 Bouesseau, Marie- World Health Organization, Geneva, Switzerland 24 Charlotte 25 26 Boutron, Isabelle INSERM U738 research unit, Paris Descartes University, Paris, France 27 28 Briel, Matthias Center for Pediatric Clinical Studies, University Medical Center 29 Tuebingen, Germany & Basel Institute for Clinical Epidemiology and 30 Biostatistics, University Hospital of Basel, Switzerland 31 32 Gallus, Silvano Department of Epidemiology, IRCCS - Istituto di Ricerche Farmacologiche 33 “Mario Negri”, Milan, Italy http://bmjopen.bmj.com/ 34 35 Garattini, Silvio IRCCS - Istituto di Ricerche Farmacologiche “Mario Negri”, Milan, Italy 36 37 Ghersi, Davina University of Sydney, Australia 38 39 Karam, Ghassan World Health Organization, Geneva, Switzerland 40

41 Kleijnen, Jos Kleijnen Systematic Reviews Ltd., York, United Kingdom & School for on September 23, 2021 by guest. Protected copyright. 42 Public Health and Primary Care, Maastricht University, Maastricht, The 43 Netherlands 44 45 Kulig, Michael Federal Joint Committee, Berlin, Germany 46 47 La Vecchia, Carlo Department of Clinical Sciences and Community Health, University of 48 49 Milan, Milan Italy 50 51 Lang, Britta German Cochrane Centre, Medical Center - University of Freiburg, 52 Freiburg, Germany 53 54 Littmann, Jasper CELLS (Centre for Ethics and Law in Life Sciences), Hannover Medical 55 Scholl, Hannover, Germany 56 57 Malicki, Mario University of Split School of Medicine, Split, Croatia 58 59 60 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml BMJ Open Page 42 of 55 BMJ Open: first published as 10.1136/bmjopen-2014-006666 on 5 May 2015. Downloaded from 1 2 3 Marusic, Ana University of Split School of Medicine, Split, Croatia 4 5 Meerpohl, Joerg German Cochrane Centre, Medical Center - University of Freiburg, 6 Freiburg, Germany 7 8 Mueller, Katharina Center for Pediatric Clinical Studies, University Medical Center 9 Felicitas Tuebingen, Germany 10 11 Murisic, Bojana Department of Epidemiology, IRCCS - Istituto di Ricerche Farmacologiche 12 “Mario Negri”, Milan, Italy 13 14 Nolting, Alexandra Federal Joint Committee, Berlin, Germany 15 For peer review only 16 Pardo, Hector The Clinical Epidemiology & Public Health Department at the Hospital de 17 la Santa Creu i Sant Pau, Spain 18 19 Perleth, Matthias Federal Joint Committee, Berlin, Germany 20 21 Ravaud, Philippe INSERM U738 research unit, Paris Descartes University, Paris, France 22 23 Reis, Andreas World Health Organization, Geneva, Switzerland 24 25 Schell, Lisa German Cochrane Centre, Medical Center - University of Freiburg, 26 Freiburg, Germany 27 28 Schmucker, Christine German Cochrane Centre, Medical Center - University of Freiburg, 29 Freiburg, Germany 30 31 Schwarzer, Guido Institute for Medical Biometry and Statistics, Medical Center – University 32

33 of Freiburg, Freiburg, Germany http://bmjopen.bmj.com/ 34 35 Strech, Daniel CELLS (Centre for Ethics and Law in Life Sciences), Hannover Medical 36 Scholl, Hannover, Germany 37 38 Trinquart, Ludovic INSERM U738 research unit, Paris Descartes University, Paris, France 39 40 Urrútia, Gerard The Clinical Epidemiology & Public Health Department at the Hospital de la Santa Creu i Sant Pau, Spain 41 on September 23, 2021 by guest. Protected copyright. 42 43 von Elm, Erik German Cochrane Centre, Medical Center - University of Freiburg, 44 Freiburg, Germany & Cochrane Switzerland, IUMSP, University Hospital 45 Lausanne, Lausanne, Switzerland 46 47 Wager, Elizabeth Sideview, Princes Risborough, United Kingdom 48 49 Wolff, Robert Kleijnen Systematic Reviews Ltd., York, United Kingdom 50 51 52 53 54 55 56 57 58 59 60 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml Page 43 of 55 BMJ Open BMJ Open: first published as 10.1136/bmjopen-2014-006666 on 5 May 2015. Downloaded from 1 2 3 Appendix 2: OPEN Advisory Board 4 5 Contributor Participating Institution 6 7 8 Altman, Doug Centre for Statistics in Medicine, University of Oxford, United Kingdom 9 10 Chan, An-Wen Women's College Research Institute, University of Toronto, Canada 11 12 Dickersin, Kay Johns Hopkins Bloomberg School of Public Health, United States 13 14 Weber, Wim British Medical Journal 15 For peer review only 16 Song, Fujian School of Medicine, Health Policy and Practice, University of East Anglia, 17 United Kingdom 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32

33 http://bmjopen.bmj.com/ 34 35 36 37 38 39 40

41 on September 23, 2021 by guest. Protected copyright. 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml BMJ Open Page 44 of 55 BMJ Open: first published as 10.1136/bmjopen-2014-006666 on 5 May 2015. Downloaded from 1 2 3 Appendix 3: Participants at OPEN Meeting 4 5 Contributor Participating Institution 6 7 8 Antes, Gerd German Cochrane Centre, Institute of Medical Biometry and Medical 9 Informatics, University Medical Center, Freiburg, Germany 10 11 Bangdiwala, Shrikant UNC Gillings School of Global Public Health, Chapel Hill, United States 12 13 Bassler, Dirk Center for Pediatric Clinical Studies, University Medical Center 14 Tuebingen, Germany 15 For peer review only 16 Boutron, Isabelle INSERM U738 research unit, Paris Descartes University, Paris, France 17 18 Collazo, Eduardo Lopez Instituto de Salud Carlos III, Barcelona, Spain 19 20 Doppelfeld, Elmar European Network of Research Ethics Committees 21 22 Dwan, Kerry University of Liverpool, United Kingdom 23 24 Frost, Robert GlaxoSmithKline plc, London, United Kingdom 25 26 Gallus, Silvano Department of Epidemiology, IRCCS - Istituto di Ricerche Farmacologiche 27 “Mario Negri”, Milan, Italy 28 29 Ghersi, Davina University of Sydney, Australia 30 31 Gøtzsche, Peter Nordic Cochrane Centre, Copenhagen, Denmark 32

33 Hrynaszkiewicz, Iain Faculty of 1000 Ltd, United Kingdom http://bmjopen.bmj.com/ 34 35 Jena, Susanne German Register of Clinical Trials, Freiburg, Germany 36 37 Karam, Ghassan World Health Organization, Geneva, Switzerland 38 39 Kienegger, Manuela Danube University Krems, Austria 40 Kirkham, Jamie University of Liverpool, United Kingdom 41 on September 23, 2021 by guest. Protected copyright. 42 43 Kleijnen, Jos Kleijnen Systematic Reviews Ltd., York, United Kingdom & School for 44 Public Health and Primary Care, Maastricht University, Maastricht, The 45 Netherlands 46 47 La Vecchia, Carlo Department of Clinical Sciences and Community Health, University of 48 Milan, Milan Italy 49 50 Malicki, Mario University of Split, School of Medicine, Split, Croatia 51 52 Marusic, Ana University of Split, School of Medicine, Split, Croatia 53 54 Meerpohl, Joerg German Cochrane Centre, Institute of Medical Biometry and Medical 55 Informatics, University Medical Center, Freiburg, Germany 56 57 Mueller, Katharina Center for Pediatric Clinical Studies, University Medical Center 58 Felicitas Tuebingen, Germany 59 60 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml Page 45 of 55 BMJ Open BMJ Open: first published as 10.1136/bmjopen-2014-006666 on 5 May 2015. Downloaded from 1 2 3 Panteli, Dimitra Technical University of Berlin, Germany 4 5 Pardo, Hector The Clinical Epidemiology & Public Health Department at the Hospital de 6 la Santa Creu i Sant Pau, Spain 7 8 Schell, Lisa German Cochrane Centre, Institute of Medical Biometry and Medical 9 Informatics, University Medical Center, Freiburg, Germany 10 11 Schmucker, Christine German Cochrane Centre, Institute of Medical Biometry and Medical 12 Informatics, University Medical Center, Freiburg, Germany 13 14 Strech, Daniel CELLS (Centre for Ethics and Law in Life Sciences), Hannover Medical 15 For peerScholl, Hannover, review Germany only 16 17 Trinquart, Ludovic INSERM U738 research unit, Paris Descartes University, Paris, France 18 19 Urrútia, Gerard The Clinical Epidemiology & Public Health Department at the Hospital de 20 la Santa Creu i Sant Pau, Spain 21 22 Van Noord, Megan Danube University Krems, Austria 23 24 Wager, Elizabeth Sideview, Princes Risborough, United Kingdom 25 26 Wahler, Steffen Correvio International Sàrl, Geneva, Switzerland 27 28 Weber, Wim British Medical Journal 29 30 Wieseler, Beate Institute for Quality and Efficiency in Healthcare, Cologne, Germany 31 32 33 http://bmjopen.bmj.com/ 34 35 36 37 38 39 40

33 http://bmjopen.bmj.com/ 34 35 36 37 38 39 40

Easy to start raising awareness, but comprehensive strategy strategy comprehensive but awareness, start raising to Easy more difficult to difficult implement more implemented on a largescale on implemented At first only limited resources required; however, relevant relevant however, required; resources limited Atonly first is recommendation thisif required be would resources No obvious downsides; potential to have important impact important to have downsides; potential No obvious Likely limited effectiveness in the short run, but potentially high shortrun, high potentially but the in limited Likely effectiveness effectiveness in therun long in effectiveness Remark Remark Probably no strong opposition, but likelihood of “opposition” likelihood “opposition” of opposition, but strong no Probably stakeholders key of resources to passiveness/limited due BMJ Open http://bmjopen.bmj.com/

x x x

x Yes No No Yes Yes No No Yes Yes No Yes Yes No Yes Judgment Judgment

on September 23, 2021 by guest. Protected copyright. For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml For peer review only Easily implementable implementable Easily implementation implementation Limited resources required for for required resources Limited Low likelihood of strong opposition strong of likelihood Low No Yes Benefits clearly outweigh downsides downsides clearly outweigh Benefits High confidence in effectiveness confidence High Category Category We strongly recommend that recommend We strongly process translation and generation knowledge the instakeholders all it. reduce measures to and bias) (publication bias dissemination about awareness raise Supplementary File 1: Decision tables for general recommendations recommendations general for tables Decision File 1: Supplementary #1: Recommendation 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 BMJ Open: first published as 10.1136/bmjopen-2014-006666 on 5 May 2015. Downloaded from

BMJ Open http://bmjopen.bmj.com/ Recommendation Recommendation

Should also be extended to patients who should be made aware of dissemination bias. madedissemination of aware should be who patients to also extended Should be Not very concrete recommendation; room for interpretation. interpretation. for room recommendation; concrete very Not Remark: Remark:

Strong recommendation recommendation Strong

on September 23, 2021 by guest. Protected copyright. For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml For peer review only

Strength of recommendation: recommendation: of Strength Page 49 of 55 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 BMJ Open: first published as 10.1136/bmjopen-2014-006666 on 5 May 2015. Downloaded from Page 50 of 55

omprehensive dissemination and implementation plan would implementation and plan dissemination omprehensive Ac substantial resources. likely require Likely that some key stakeholder groups or some key somekey or groups stakeholder thatkey some Likely stakeholders within a group are opposed and not going to and going notare opposed a group within stakeholders allor some implement OPEN and/or disseminate recommendations. outweigh downsides. downsides. outweigh unambiguous beneficial balance, overall benefits would benefits clearly overallwould beneficial balance, unambiguous Although some specific recommendations might not have a mighthave not recommendations specific some Although Remark Remark Likely tailored dissemination and implementation strategies implementation and tailored dissemination Likely If OPEN recommendations were fully disseminated and and were fully disseminated recommendations OPEN If reduced be would likely most bias dissemination implemented, effectively. BMJ Open http://bmjopen.bmj.com/

x x x

x x

Yes No No Yes Yes No No Yes Yes No Yes Yes No Yes Judgment Judgment

on September 23, 2021 by guest. Protected copyright. For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml For peer review only Easily implementable implementable Easily Limited resources required for for required resources Limited implementation Low likelihood of strong opposition strong of likelihood Low No Yes Benefits clearly outweigh downsides downsides clearly outweigh Benefits High confidence in effectiveness confidence High Category Category We strongly recommend that recommend strongly We processtranslation and theknowledgegeneration in stakeholders all respectivegroup. their of amongst key stakeholders recommendations of OPEN targeted facilitateimplementation and disseminate Recommendation #2: Recommendation 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 BMJ Open: first published as 10.1136/bmjopen-2014-006666 on 5 May 2015. Downloaded from would be required including monitoring efforts on efforts on monitoring including required would be effectiveness. and implementation BMJ Open http://bmjopen.bmj.com/ Recommendation Recommendation

helpful in reducing dissemination dissemination bias. reducing in helpful Without dissemination and implementation plans, OPEN recommendations will be not will recommendations not be OPEN plans, implementation and dissemination Without implementability; but obviously key recommendation amongst OPEN recommendations. recommendations. OPEN amongst recommendation key obviously but implementability; Strong recommendation despite concerns around opposition, resources and and resources opposition, around despite concerns recommendation Strong Remark: Remark:

Strong recommendation recommendation Strong

on September 23, 2021 by guest. Protected copyright. For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml For peer review only

Strength of recommendation: recommendation: of Strength Page 51 of 55 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 BMJ Open: first published as 10.1136/bmjopen-2014-006666 on 5 May 2015. Downloaded from Page 52 of 55 maintain such initiatives. such maintain resources likely required to establish, promote, expand and and expand promote, establish, to likelyrequired resources resources. Technical consequences not fully clear; relevant relevant fully clear; not Technical consequences resources. Promotion and political support possible without relevant without possible support and political Promotion No obvious downsides in promoting trial registration and and trialregistration promoting in downsides No obvious initiatives. such supporting Promotion and support in itself likely not yet highly effective in in effective highly yet not itselfsupport in likely and Promotion reducing dissemination bias, but if respective initiatives are initiatives respective if dissemination but bias, reducing likelythe relevant impact in maintained and implemented run. longer Remark Remark No serious barrier to both political and financial support was was financial support politicaland to both barrier No serious identified. BMJ Open http://bmjopen.bmj.com/

x x

x Yes No Yes x Yes No No Yes Yes No Yes Yes No Yes Judgment Judgment on September 23, 2021 by guest. Protected copyright. For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml For peer review only Easily implementable implementable Easily implementation implementation Limited resources required for for required resources Limited Low likelihood of strong opposition strong of likelihood Low No Yes this likely. quite field active in groups by Some opposition Benefits clearly outweigh downsides downsides clearly outweigh Benefits High confidence in effectiveness confidence High Category Category We strongly recommend that recommend strongly We processtranslation and theknowledgegeneration in stakeholders all registries. multipleacross trial initiatives that support searches facilitate and trialregistration promote Recommendation #3: Recommendation 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 BMJ Open: first published as 10.1136/bmjopen-2014-006666 on 5 May 2015. Downloaded from BMJ Open http://bmjopen.bmj.com/ r of supporting and thereby hopefully establishing/maintaining in the longer run run longer the in establishing/maintaining andhopefully thereby supporting of r u Recommendation Recommendation

databases. databases. In the future need for similar initiatives that allow simultaneous access to results access similarsimultaneous initiativesallow that thefor need future In

Likely need for alternative initiatives to avoid monopoly?! monopoly?! to avoid initiatives alternative for need Likely Trials Registry Platform (ICRTP)). (ICRTP)). Platform Trials Registry respective initiatives (one example of such a platform is the WHO International Clinical Clinical WHO International the a is platform such of example (one initiatives respective favo Wide range of support is possible. Although not very concrete, strong recommendation in in recommendation strong concrete, very not Although is possible. support of Wide range Remark: Remark:

Strong recommendation recommendation Strong

on September 23, 2021 by guest. Protected copyright. For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml For peer review only

Strength of recommendation: recommendation: of Strength Page 53 of 55 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 BMJ Open: first published as 10.1136/bmjopen-2014-006666 on 5 May 2015. Downloaded from Page 54 of 55 actually conduct and regularly update all these activities (SRs). all activities (SRs). these update and regularly actually conduct Political support possible without relevant resources. Relevant resources. without relevant possible Political support to efforts and maintain increase to likelyrequired resources No strong opposition to general support likely; however, likely;however, tosupport general opposition No strong financial support more likely to be difficult to establish. establish. moreto difficult likely to be financial support consider all conducted research for a specific question. question. a specific for research allconducted consider outweigh downsides, in particular if efforts are undertaken to are undertaken inefforts if particular downsides, outweigh Risk of spurious conclusions if activities based on very biased very biased ifbased on activities conclusions spurious of Risk to seemclearly trials; of benefits overall published subset Support in itself likely not yet highly effective in reducing in effective reducing highly yet itselfin likely not Support dissemination bias, but if respective activities rigorously identify identify activities rigorously respective if but bias, dissemination the in impact trialslikelyrelevant integrate unpublished and run. longer Remark Remark BMJ Open http://bmjopen.bmj.com/

x x

Yes No No Yes Yes No Yes Yes No Yes Judgment Judgment on September 23, 2021 by guest. Protected copyright. For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml For peer review only implementation implementation Limited resources required for for required resources Limited Low likelihood of strong opposition strong of likelihood Low No Yes Benefits clearly outweigh downsides downsides clearly outweigh Benefits High confidence in effectiveness confidence High Category Category We strongly recommend that recommend strongly We processtranslation and theknowledgegeneration in stakeholders all methodology. studieswith rigorous from information synthesize activitiessystematically to support Recommendation #4: Recommendation 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 BMJ Open: first published as 10.1136/bmjopen-2014-006666 on 5 May 2015. Downloaded from No serious barrier to both political and financial support was was financial support politicaland to both barrier No serious identified. BMJ Open http://bmjopen.bmj.com/

r of supporting and thereby increasing such activities. However, relevant resources resources relevant However, such activities. andincreasing thereby supporting of r u Recommendation Recommendation

x x

Yes No No Yes

increase effectiveness. These will require further resources. resources. further These willeffectiveness. require increase Dissemination strategies need to be developed for non-English speaking countries to to countries speaking non-English for to developed be need strategies Dissemination reviews. will be required in the long run to conduct and update regularly relevant systematic regularly and conduct update relevant to therun long in required will be favo Wide range of support is possible. Although not very concrete, strong recommendation in recommendation strong concrete, very not Although is possible. support of Wide range Remark: Remark:

Strong recommendation recommendation Strong

on September 23, 2021 by guest. Protected copyright. For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml For peer review only

Strength of recommendation: recommendation: of Strength Easily implementable implementable Easily Page 55 of 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 BMJ Open: first published as 10.1136/bmjopen-2014-006666 on 5 May 2015. Downloaded from Page 56 of 55 BMJ Open http://bmjopen.bmj.com/ on September 23, 2021 by guest. Protected copyright. For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml For peer review only

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60