Tools to assess the risk of due to inaccessible data in evidence syntheses: protocol for a

Correspondence to : Dr. Matthew Page, School of Public Health and Preventive Medicine,

Monash University, 553 St Kilda Road, Melbourne, Victoria, 3004, Australia. Email address: [email protected]

1

BACKGROUND

The credibility of evidence syntheses can be compromised when some outcome data are inaccessible to users because of the nature of the results (e.g. statistical significance, magnitude or direction of effect estimates) (1). Examples include when a study with null findings or results favouring the comparator is not published (“” (2)), or when outcomes that are statistically non-significant are not reported or are only partially reported in a journal article (“outcome reporting bias” (3)). Such biased reporting of research is common.

Syntheses of cohorts of trials followed from protocol/registration to publication suggest that half of all trials are not published (2, 4), that trials with statistically significant results are twice as likely to be published (4), and that 13% to 50% of pre-specified outcomes are not reported, or only partially reported, in trial publications (5).

Several approaches to address the risk of such bias due to inaccessible data have been advocated. Review authors are encouraged to perform comprehensive and sensitive searches of various sources (e.g. bibliographic databases such as MEDLINE®, and “grey literature” databases such as OpenSIGLE). Trial registries such as ClinicalTrials.gov can be used to identify completed but unpublished trials, pre-specified but non-reported outcomes, and aggregated data posted prior to a trial’s publication (1). Clinical study reports prepared by drug/device manufacturers for regulators (such as the European Medicines Agency) provide more extensive data from a trial than that presented in journal articles (6). Funnel plots and tests for funnel plot asymmetry may be used in some circumstances to infer publication bias, by examining whether smaller trials are asymmetrically distributed around the meta-analytic estimate or the larger trials (7). And the Cochrane risk of bias tool for randomized trials

(“selective reporting” domain) (8), or the “outcome reporting bias in trials” (ORBIT) classification system (3), can be used to record non-/partially reported outcomes in trials.

2

Despite the availability of several approaches, most systematic reviewers do not adequately address the risk of bias due to inaccessible data (9-11). In a cross-sectional study of 300 systematic reviews indexed in MEDLINE® in February 2014 (9), an assessment of publication bias was considered infeasible by 56% of authors. Authors nearly always claimed this was because the small number of studies or inability to perform a meta-analysis precluded the use of funnel plots. Funnel plots and associated statistical tests were used in

31% of reviews. However, 43% of these reviews included fewer than 10 studies of varying size, meaning the plots were difficult to interpret and tests had low statistical power (7). Only

19% of reviews included a search of a trial registry, and only 7% included a search of another source of data disseminated outside of journal articles. The risk of outcome reporting bias in the included studies was assessed in only 24% of reviews (9). Another study showed that even when outcome reporting bias was detected in trials, few authors acknowledged that the synthesis was missing data that were not/partially reported (12).

Possibly the risk of bias due to inaccessible data is addressed inadequately because all the factors that must be considered are fragmented. A similar problem occurred a decade ago with the assessment of risk of bias in randomized trials. Some systematic reviewers might have assessed problems with randomization, while others might have assessed problems with blinding or attrition (13). It was not until all the important bias domains were brought together into a single tool – the Cochrane risk of bias tool for randomized trials (8) - that systematic reviewers started to assess risk of bias in trials comprehensively (14). Linking all the components needed to judge the risk of bias due to inaccessible data into a single, novel tool, may have a similar impact on the conduct of evidence syntheses. Doing so should allow systematic reviewers to reach conclusions that are more trustworthy for decision makers.

3

When developing a new measurement tool, its structure and content should be informed by a review of existing tools (15). Several reviews have examined the properties of tools designed to assess the methodological quality/risk of bias in randomized trials (16, 17), non- randomized studies of interventions (18), diagnostic test accuracy studies (19), and systematic reviews (20, 21)). However, to our knowledge, no prior review has focused on tools to assess risk of bias due to inaccessible data. Therefore, the aim of this systematic review is to summarise the properties of existing tools designed to assess risk of bias due to inaccessible data in evidence syntheses.

METHODS

Eligibility criteria

We will include any paper reporting a tool designed to assess the risk of bias due to unpublished studies (“publication bias”), the risk of bias due to non/partial reporting of outcomes (“outcome reporting bias”), or both sources of bias. By “tool”, we mean a structured approach that requires users to identify potential problems in the studies, and to make a judgement about the corresponding risk of bias in the results. Eligible tools can take any form, including scales, checklists, or domain-based tools. To be considered a scale, each item has to have a numeric score attached to it, and an overall summary score can be calculated (16). Checklists include multiple questions, but the developers’ intention is not to attach a numerical score to each response, nor to calculate an overall score (17). Domain- based tools require users to judge quality/risk of bias within specific bias domains, and to record the information on which each judgement is based (8).

4

We will only include tools that were designed for completion by authors conducting a synthesis of evidence (e.g. meta-analysis, narrative synthesis). Tools with a broad scope (e.g. to assess the overall quality/risk of bias in the evidence) will be eligible if one of the mandatory components focuses on risk of bias due to inaccessible data. We will include multi-dimensional tools with a statistical component (e.g. those that require users to answer questions as well as perform statistical tests for funnel plot asymmetry). We will also include any studies that have evaluated the properties of existing tools (e.g. construct validity, inter- rater reliability, time taken to complete assessments). Papers will be eligible regardless of the date, language, or format of publication.

We will exclude papers describing guidelines to address bias due to inaccessible data (e.g. the

Cochrane Handbook chapter on reporting bias (22)), and general lists of items to consider rather than structured tools (which cannot be used for evaluative application). Tools will be ineligible if they were developed for one specific systematic review, since such tools are unlikely to have been developed systematically. We will exclude tools developed for users to appraise published systematic reviews, such as the ROBIS tool (23) or AMSTAR (24). We

will also exclude papers that only describe the development or evaluation of statistical

methods to assess or adjust for the risk of bias due to inaccessible data, as these have been

reviewed extensively elsewhere (7, 25, 26).

Search methods

We will search Ovid MEDLINE (January 1946 to February 2017), Ovid EMBASE (January

1980 to February 2017), and Ovid PsycINFO (January 1806 to February 2017). We have

developed search strategies in conjunction with an information specialist, using search terms

5 adopted by Whiting et al. to identify quality assessment tools (21) (see full Boolean search strategies in Appendix 1).

To capture any tools posted on the Internet or not published by formal academic publishers

(i.e. grey literature), we will search Google Scholar using the phrase “reporting bias tool”.

We will screen the titles of the first 300 results, as recommended by Haddaway et al. (27). To capture any papers that may be missed by all searches, we will screen the references of included articles.

Study selection and data collection

One author will screen all titles and abstracts retrieved by the searches. The same author will screen any full text articles retrieved, and a second author will verify all inclusion decisions independently. Disagreements in the latter stage will be resolved via discussion until consensus is reached, or referral to a third author where necessary.

One author will extract data from included papers using a standardised data collection form.

Another author will verify the accuracy of the data extracted. We will extract the following data:

• type of tool (scale, checklist, domain-based or other);

• types of bias due to inaccessible data addressed by the tool (e.g. unpublished studies,

non-/partially reported outcomes);

• level of assessment (i.e. whether users direct assessments at the meta-analysis or at

the individual studies included in the meta-analysis);

• whether the tool is designed for general use (generic) or targets syntheses of specific

study designs or topic areas (specific);

6

• items included in the tool;

• how items within the tool are rated;

• how items were selected for inclusion;

• methods used to develop the tool (e.g. Delphi study, expert consensus meeting);

• availability of guidance to assist with completion of the tool (e.g. guidance manual);

• any psychometric properties recorded for the tool (e.g. inter-rater reliability via

Cohen’s kappa coefficient (28));

• time taken to complete the tool.

Data analysis

We will summarise the characteristics of included tools in tables. We will sum the number of items per tool and calculate the median (interquartile range) number of items across all tools.

We will tabulate the number of tools assessing each item, and provide a narrative summary of the methods used to develop and evaluate the tools.

7

References

1. Song F, Parekh S, Hooper L, Loke YK, Ryder J, Sutton AJ, et al. Dissemination and

publication of research findings: an updated review of related . Health Technol

Assess. 2010;14:8.

2. Chan A-W, Song F, Vickers A, Jefferson T, Dickersin K, Gøtzsche PC, et al.

Increasing value and reducing waste: addressing inaccessible research. The Lancet.

2014;383(9913):257-66.

3. Kirkham JJ, Dwan KM, Altman DG, Gamble C, Dodd S, Smyth R, et al. The impact

of outcome reporting bias in randomised controlled trials on a cohort of systematic

reviews. BMJ. 2010;340:c365.

4. Schmucker C, Schell LK, Portalupi S, Oeller P, Cabrera L, Bassler D, et al. Extent of

non-publication in cohorts of studies approved by research ethics committees or

included in trial registries. PLoS One. 2014;9(12):e114023.

5. Dwan K, Gamble C, Williamson PR, Kirkham JJ. Systematic review of the empirical

evidence of study publication bias and outcome reporting bias - an updated review.

PLoS One. 2013;8(7):e66844.

6. Doshi P, Jefferson T. Open data 5 years on: a case series of 12 freedom of information

requests for regulatory data to the European Medicines Agency. Trials.

2016;17(1):78.

7. Sterne JAC, Sutton AJ, Ioannidis JPA, Terrin N, Jones DR, Lau J, et al.

Recommendations for examining and interpreting funnel plot asymmetry in meta-

analyses of randomised controlled trials. BMJ. 2011;343:d4002.

8. Higgins JPT, Altman DG, Gøtzsche PC, Jüni P, Moher D, Oxman AD, et al. The

Cochrane Collaboration’s tool for assessing risk of bias in randomised trials. BMJ.

2011;343:d5928.

8

9. Page MJ, Shamseer L, Altman DG, Tetzlaff J, Sampson M, Tricco AC, et al.

Epidemiology and reporting characteristics of systematic reviews of biomedical

research: a cross-sectional study. PLoS Med. 2016;13(5):e1002028.

10. Koletsi D, Valla K, Fleming PS, Chaimani A, Pandis N. Assessment of publication

bias required improvement in oral health systematic reviews. J Clin Epidemiol.

2016;76:118-24

11. Hedin RJ, Umberham BA, Detweiler BN, Kollmorgen L, Vassar M. Publication Bias

and Nonreporting Found in Majority of Systematic Reviews and Meta-analyses in

Anesthesiology Journals. Anesth Analg. 2016;123(4):1018-25.

12. Page MJ, Higgins JPT. Rethinking the assessment of risk of bias due to selective

reporting: a cross-sectional study. Systematic reviews. 2016;5(1):108.

13. Lundh A, Gotzsche PC. Recommendations by Cochrane Review Groups for

assessment of the risk of bias in studies. BMC Med Res Methodol. 2008;8:22.

14. Jorgensen L, Paludan-Muller AS, Laursen DR, Savovic J, Boutron I, Sterne JA, et al.

Evaluation of the Cochrane tool for assessing risk of bias in randomized clinical trials:

overview of published comments and analysis of user practice in Cochrane and non-

Cochrane reviews. Systematic reviews. 2016;5(1):80.

15. Streiner DL, Norman GR, Cairney J. Health measurement scales: a practical guide for

their development and use. Fifth ed. Oxford, UK: Oxford University Press; 2008.

16. Moher D, Jadad AR, Nichol G, Penman M, Tugwell P, Walsh S. Assessing the quality

of randomized controlled trials: an annotated bibliography of scales and checklists.

Control Clin Trials. 1995;16(1):62-73.

17. Armijo Olivo S, Macedo LG, Gadotti IC, Fuentes J, Stanton T, Magee DJ. Scales to

assess the quality of randomized controlled trials: a systematic review. Phys Ther.

2008;88(2):156-75.

9

18. Sanderson S, Tatt ID, Higgins JP. Tools for assessing quality and susceptibility to

bias in observational studies in : a systematic review and annotated

bibliography. Int J Epidemiol. 2007;36(3):666-76.

19. Whiting P, Rutjes AW, Dinnes J, Reitsma JB, Bossuyt PM, Kleijnen J. A systematic

review finds that diagnostic reviews fail to incorporate quality despite available tools.

J Clin Epidemiol. 2005;58(1):1-12.

20. Bai A, Shukla VK, Bak G, Wells G. Quality Assessment Tools Project Report.

Ottawa: Canadian Agency for Drugs and Technologies in Health; 2012.

21. Whiting P, Davies P, Savovic J, Caldwell D, Churchill R. Evidence to inform the

development of ROBIS, a new tool to assess the risk of bias in systematic reviews,

September 2013. Available from http://www.robis-tool.info [accessed 8 February

2017].

22. Sterne JAC, Egger M, Moher D. Chapter 10: Addressing reporting biases. In: Higgins

JPT, Green S, editors. Cochrane handbook for systematic reviews of interventions

Version 510 [updated March 2011] 2011.

23. Whiting P, Savovic J, Higgins JP, Caldwell DM, Reeves BC, Shea B, et al. ROBIS: A

new tool to assess risk of bias in systematic reviews was developed. J Clin Epidemiol.

2016;69:225-34.

24. Shea BJ, Grimshaw JM, Wells GA, Boers M, Andersson N, Hamel C, et al.

Development of AMSTAR: a measurement tool to assess the methodological quality

of systematic reviews. BMC Med Res Methodol. 2007;7:10.

25. Mueller KF, Meerpohl JJ, Briel M, Antes G, von Elm E, Lang B, et al. Methods for

detecting, quantifying and adjusting for dissemination bias in meta-analysis are

described. J Clin Epidemiol. 2016;80:25-33.

10

26. Jin ZC, Zhou XH, He J. Statistical methods for dealing with publication bias in meta-

analysis. Stat Med. 2015;34(2):343-60.

27. Haddaway NR, Collins AM, Coughlin D, Kirk S. The Role of Google Scholar in

Evidence Reviews and Its Applicability to Grey Literature Searching. PLoS One.

2015;10(9):e0138237.

28. Cohen J. A coefficient of agreement for nominal scales. Educ Psychol Meas.

1960;20(1):37-46.

11

Appendix 1

Database: Ovid MEDLINE(R) <1946 to present> Search Strategy: ------1 "Review Literature as Topic"/ 2 meta-analysis/ 3 meta-analysis as topic/ 4 systematic review$.tw. 5 (meta-analys$ or metaanalys$).tw. 6 or/1-5 7 ((tool or tools or instrument$ or checklist$ or check list$ or scale or scales) and (quality or methodolog$ or method or methods)).ti. 8 (quality adj10 (score or scores or scoring or rating or rate) adj5 (methodolog$ or method or methods)).tw. 9 (guideline$ and (quality or methodolog$ or method or methods)).ti. 10 ((assess$ or apprais$ or critical$) adj3 (systematic review$ or meta-analys$ or metaanalys$)).ti. 11 ((score or scores or scoring or rating or rate) and (quality or methodolog$ or method or methods)).ti. 12 ((quality or methodology) adj3 (review or meta-analys$ or metaanalys$) adj3 (assess$ or method$)).tw. 13 (quality adj3 article$).tw. 14 (critical$ adj2 (apprais$ or evaluat$)).tw. 15 ((apprais$ or evaluat$) adj3 (systematic review$ or meta-analys$ or metaanalys$)).tw. 16 (guideline$ adj3 (systematic review$ or meta-analys$ or metaanalys$)).tw. 17 or/7-16 18 Checklist/ 19 17 or 18 20 Publication Bias/ 21 exp "bias (epidemiology)"/ 22 (bias adj3 (systematic review$ or meta-analys$ or metaanalys$)).tw. 23 ((quality or bias or methodolog$) adj3 (systematic review$ or meta-analys$ or metaanalys$)).tw. 24 (bias$ adj3 (publication$ or disseminat$ or language$ or reporting or grey or gray or citation$ or time delay or time lag or conference or abstract)).tw. 25 or/20-24 26 6 and 19 and 25

Database: Embase <1980 to 2017 Week 06> Search Strategy: ------1 "Review Literature as Topic"/ 2 "meta analysis (topic)"/ 3 meta analysis/ 4 "systematic review (topic)"/ 5 systematic review/ 6 systematic review$.tw. 7 (meta-analys$ or metaanalys$).tw.

12

8 or/1-7 9 (bias adj3 (systematic review$ or meta-analys$ or metaanalys$)).tw. 10 ((quality or bias or methodolog$) adj3 (systematic review$ or meta-analys$ or metaanalys$)).tw. 11 (bias$ adj3 (publication$ or disseminat$ or language$ or reporting or grey or gray or citation$ or time delay or time lag or conference or abstract)).tw. 12 "internal validity"/ 13 publishing/ 14 or/9-13 15 ((tool or tools or instrument$ or checklist$ or check list$ or scale or scales) and (quality or methodolog$ or method or methods)).ti. 16 (quality adj10 (score or scores or scoring or rating or rate) adj5 (methodolog$ or method or methods)).tw. 17 (guideline$ and (quality or methodolog$ or method or methods)).ti. 18 ((assess$ or apprais$ or critical$) adj3 (systematic review$ or meta-analys$ or metaanalys$)).ti. 19 ((score or scores or scoring or rating or rate) and (quality or methodolog$ or method or methods)).ti. 20 ((quality or methodology) adj3 (review or meta-analys$ or metaanalys$) adj3 (assess$ or method$)).tw. 21 (quality adj3 article$).tw. 22 (critical$ adj2 (apprais$ or evaluat$)).tw. 23 ((apprais$ or evaluat$) adj3 (systematic review$ or meta-analys$ or metaanalys$)).tw. 24 (guideline$ adj3 (systematic review$ or meta-analys$ or metaanalys$)).tw. 25 or/15-24 26 checklist/ 27 25 or 26 28 8 and 14 and 27 29 limit 28 to embase

Database: PsycINFO <1806 to February Week 1 2017> Search Strategy: ------1 meta-analysis/ 2 systematic review$.tw. 3 (meta-analys$ or metaanalys$).tw. 4 or/1-3 5 ((tool or tools or instrument$ or checklist$ or check list$ or scale or scales) and (quality or methodolog$ or method or methods)).ti. 6 (quality adj10 (score or scores or scoring or rating or rate) adj5 (methodolog$ or method or methods)).tw. 7 (guideline$ and (quality or methodolog$ or method or methods)).ti. 8 ((assess$ or apprais$ or critical$) adj3 (systematic review$ or meta-analys$ or metaanalys$)).ti. 9 ((score or scores or scoring or rating or rate) and (quality or methodolog$ or method or 1. methods)).ti. 10 ((quality or methodology) adj3 (review or meta-analys$ or metaanalys$) adj3 (assess$ or method$)).tw. 11 (quality adj3 article$).tw.

13

12 (critical$ adj2 (apprais$ or evaluat$)).tw. 13 ((apprais$ or evaluat$) adj3 (systematic review$ or meta-analys$ or metaanalys$)).tw. 14 (guideline$ adj3 (systematic review$ or meta-analys$ or metaanalys$)).tw. 15 checklist/ 16 or/5-15 17 (bias adj3 (systematic review$ or meta-analys$ or metaanalys$)).tw. 18 ((quality or bias or methodolog$) adj3 (systematic review$ or meta-analys$ or metaanalys$)).tw. 19 (bias$ adj3 (publication$ or disseminat$ or language$ or reporting or grey or gray or citation$ or time delay or time lag or conference or abstract)).tw. 20 bias.mp. 21 or/17-20 22 4 and 16 and 21

14