Estimating the of transparency and -related practices in (2014-2017)

Tom E. Hardwicke1,2, Robert T. Thibault3,4, Jessica E. Kosie5,6, Joshua D. Wallach7,8, Mallory C. Kidwell9, & John P. A. Ioannidis2,10,11

1 Department of Psychology, University of Amsterdam

2 Meta-Research Innovation Center Berlin (METRIC-B), QUEST Center for Transforming Biomedical Research, Berlin Institute of Health, Charité – Universitätsmedizin Berlin

3 School of Psychological , University of Bristol

4 MRC Integrative Unit at the University of Bristol

5 Department of Psychology, University of Oregon

6 Department of Psychology, Princeton University

7 Department of Environmental Health , Yale School of Public Health

8 Collaboration for Research Integrity and Transparency, Yale School of

9 Department of Psychology,

10 Departments of Medicine, of Health Research and Policy, of Biomedical Data Science, and of , Stanford University

11 Meta-Research Innovation Center at Stanford (METRICS), Stanford University

Author note

Tom Hardwicke was based at Stanford University when the study began and is now based at the University of Amsterdam. Jessica Kosie was based at University of Oregon when the study began and is now based at Princeton University. Correspondence concerning this article should be addressed to Tom Hardwicke, Nieuwe Achtergracht 129B, Department of Psychology, University of Amsterdam, 1018 WT Amsterdam, The Netherlands. E-mail: [email protected]

1 Abstract

Psychologists are navigating an unprecedented period of about the credibility and utility of their discipline. Reform initiatives have emphasized the benefits of several transparency and reproducibility-related research practices; however, their adoption across the psychology literature is unknown. To estimate their prevalence, we manually examined a random sample of

250 psychology articles published between 2014-2017. Over half of the articles were publicly available (154/237, 65% [95% confidence interval, 59%-71%]); however, sharing of research materials (26/183, 14% [10%-19%]), study protocols (0/188, 0% [0%-1%]), raw data (4/188, 2%

[1%-4%]), and analysis scripts (1/188, 1% [0%-1%]) was rare. Pre-registration was also uncommon (5/188, 3% [1%-5%]). Many articles included a funding disclosure statement

(142/228, 62% [56%-69%]), but conflict of interest statements were less common (88/228, 39%

[32%-45%]). Replication studies were rare (10/188, 5% [3%-8%]) and few studies were included in systematic reviews (21/183, 11% [8%-16%]) or meta-analyses (12/183, 7% [4%-10%]).

Overall, the results suggest that transparency and reproducibility-related research practices were far from routine. These findings establish a baseline which can be used to assess future progress towards increasing the credibility and utility of psychology research.

Keywords: transparency, reproducibility, meta-research, psychology,

2 Introduction

Serious concerns about the credibility and utility of some scientific literature (Ioannidis, 2005;

Ioannidis et al., 2014) have prompted calls for increased adoption of research practices that enhance reproducibility and transparency (Miguel et al., 2014; Munafò et al., 2017; Nosek et al.,

2015; Wallach et al., 2018). Close scrutiny of psychology in particular has suggested that standard research and publication practices have rendered the discipline highly exposed to bias, potentially resulting in a large volume of exaggerated and misleading results (Ioannidis et al.,

2014; John et al., 2012; Open Science Collaboration, 2015; Pashler & Wagenmakers, 2012;

Simmons et al., 2011; Szucs & Ioannidis, 2017). This realization within the field has also led to a number of reform efforts (Hardwicke & Ioannidis, 2018a; Hardwicke et al., 2019; Nelson et al.,

2018; Vazire, 2018), which have the potential to improve efficiency (Chalmers & Glasziou,

2009), facilitate self-correction (Ioannidis, 2012), and enhance credibility (Vazire, 2017).

A central focus of reform initiatives has been to encourage scientists to share more information about the studies they perform. Journal articles are only the most visible facade of deeper layers of scholarship that may include protocols, original research materials, raw data, and analysis scripts – resources that are not necessarily shared with other scientists (Buckheit &

Donoho, 1995; Klein et al., 2018). Even journal articles themselves may only be accessible to those with institutional access or the ability to pay a fee (Piwowar et al., 2018). Additionally, potential sources of bias, such as conflicts of interest and funding sources, may not be disclosed

(Bekelman et al., 2003; Cristea & Ioannidis, 2018). However, there is a growing (or re-emerging,

David, 2008) appreciation that the scientific community needs to be able to access all of this information in order to comprehensively evaluate, interpret, and independently verify scientific claims (Vazire, 2017; Munafò et al., 2017). Furthermore, access to this information enables

3 replication, evidence synthesis, and discovery activities that may ultimately accelerate scientific progress (Ioannidis, 2012; Klein et al., 2018).

The burgeoning discipline of meta-research (‘research on research’) has already begun to evaluate the impact of various reform initiatives (Hardwicke et al., 2020). For example, journal policies have been associated with moderate to substantial increases in data sharing

(Hardwicke et al., 2018; Kidwell et al., 2016; Nuijten et al., 2018; Rowhani-Farid & Barnett,

2016; Naudet et al., 2018). However, the prevalence of transparency and reproducibility-related research practices across the psychology literature is largely unknown. Based on previous investigations in biomedicine (Iqbal et al., 2016; Wallach et al., 2018) and the social sciences

(Hardwicke et al., 2019), we manually examined a random sample of 250 articles to estimate the prevalence of several transparency and reproducibility-related indicators in psychology articles published between 2014-2017. The indicators were open access to published articles; availability of study materials, study protocols, raw data, and analysis scripts; pre-registration; disclosure of funding sources and conflicts of interest; conduct of replication studies; and cumulative synthesis of evidence in meta-analyses and systematic reviews.

Methods

Design

This was a retrospective with a cross-sectional design. Sampling units were individual articles. Measured variables are shown in Table 1.

4 Table 1. Measured variables. The variables measured for an individual article depended on the study design classification. For articles that were not available (the full text could not be retrieved) or non-English language, only article characteristics were obtained. The exact operational definitions and procedures for data extraction/coding are available in the structured form here: https://osf.io/x9rmy/

Applicable study designs

Article characteristics

Subject area, year of publication, study design, All country of origin (based on corresponding author's affiliation), human/animal subjects, 2017 journal impact factor (according to Thomson Reuters Journal Citation Reports)

Articles

Accessibility and Retrieval Method (can the All article be accessed, is there a public version or is paywall access required?)

Protocols

Availability statement (is availability, or lack of, Study designs involving primary data#, study explicitly declared?) designs involving secondary data (commentaries with analysis and meta-analyses). Content (what aspects of the study are included in the ?)

Materials

Availability statement (is availability, or lack of, Study designs involving primary data# explicitly declared?)

Retrieval Method (e.g., upon request or via online repository)

Accessibility (can the materials be accessed?)

Raw data

Availability statement (is availability, or lack of, explicitly declared?)

5 Retrieval Method (e.g., upon request or via online Study designs involving primary data#, study repository) designs involving secondary data (commentaries with analysis and meta-analyses). Accessibility (can the data be accessed?)

Content (has all relevant data been shared?)

Documentation (are the data understandable?)

Analysis scripts

Availability statement (is availability, or lack of, Study designs involving primary data#, study explicitly declared?) designs involving secondary data (commentaries with analysis and meta-analyses). Retrieval Method (e.g., upon request or via online repository)

Accessibility (can the scripts be accessed?)

Pre-registration

Availability statement (is availability, or lack of, Study designs involving primary data#, study explicitly declared?) designs involving secondary data (commentaries with analysis and meta-analyses). Retrieval Method (which registry was used?)

Accessibility (can the pre-registration be accessed?)

Content (what was pre-registered?)

Funding

Disclosure statement (are funding sources, or lack All of, explicitly declared?)

Conflicts of interest

Disclosure statement (are conflicts of interest, or All lack of, explicitly declared?)

Replication

Statement (does the article claim to report a All replication?)

6 Citation history (has the article been cited by a Study designs involving primary data# study that claims to be a replication?)

Evidence synthesis

Meta-analysis citation history* (has the article Study designs involving primary data# been cited by, and included in the evidence- synthesis component of, a meta-analysis)

Systematic review citation history* (has the article Study designs involving primary data# been cited by, and included in the evidence- synthesis component of, a )

# ‘Study designs involving primary data’ encompasses the following study design classifications: field studies, laboratory studies, surveys, case studies, multiple types, , and ‘other’ designs. * Meta-analysis and systematic review variables were coded as exclusive variables.

Sample

We obtained a random sample of 250 psychology articles published between 2014 and 2017.

Specifically, we used a random number generator to sample 250 articles from all 224,556 documents in the Scopus database (as of 22nd September, 2018) designated with the document type “article” or “review”, published between 2014 and 2017, and with an All Science Journal

Classification (ASJC) code related to psychology (i.e., ASJC codes 3200, 3201, 3202, 3203,

3204, 3205, 3206, 3207). This classification included 1323 journals and all of the subject areas shown in Table 2. The sample size was based on our judgement of what was both informative and tractable.

Procedure

Data collection took place between September 27th, 2018 and September 23rd, 2019. Data extraction for the measured variables shown in Table 1 involved close manual examination of the articles supplemented with keyword searching guided by a Google form (https://osf.io/x9rmy/) based on previous investigations in biomedicine (Iqbal et al., 2016; Wallach et al., 2018) and the

7 social sciences (Hardwicke et al., 2019). As indicated in the second column of the table, the exact variables measured for a given article depended on study design. Attempts to access each article were made by searching with the Open Access Button (https://openaccessbutton.org/), Google

(https://www.google.com/), and if necessary, at least two of the online libraries of the coders’ affiliated universities. For articles that were not available (the full text could not be retrieved) or non-English language, only article characteristics were obtained.

Three investigators performed initial extraction and coding; articles were assigned to each investigator on demand by randomly selecting from remaining articles (JDW, n = 32; JK, n = 93;

RT, n = 125). A single investigator obtained subject area, year of publication, and citation histories from Scopus and 2017 journal impact factors from Thomson Reuters Journal Citation

Reports. All other variables in Table 1 were second coded by one of two other investigators

(TEH, n = 198; MCK, n = 52) and any coding differences were resolved through discussion.

Inter-reliability assessments are shown in Supplementary Table 1. Assuming all articles were published on 1st July of their respective publication year (i.e., half way through the year), the time between publication and recording citation information ranged from 448 to 1544 (median = 996) days for number of citations and 456 to 1900 (median = 1178) days for number of citing articles that were replications, systematic reviews, and/or meta-analyses. Conflict of interest statements

(Table 3) were categorized by two investigators (TEH and JDW) in an exploratory manner (i.e., not pre-registered).

Analysis

Results are descriptive and focus on the proportion of articles that fulfil each of the evaluated indicators, using as a denominator the number of articles where each indicator is applicable (see

8 Table 1). We also report 95% confidence intervals (in square brackets) based on the Sison-Glaz method for multinomial proportions (Sison & Glaz, 1995).

Results

Sample characteristics

Sample characteristics for all 250 articles and for the 228 articles that were eligible for in-depth data extraction (i.e., English-language and accessible) are displayed in Table 2.

Table 2. Sample characteristics for the 250 randomly sampled articles and the 228 English language and accessible articles that were eligible for in depth data extraction. Information on study design and subjects was not available for 2 articles which could not be accessed and did not have abstracts.

Eligible All articles articles

n n

Subject area

Clinical Psychology 57 65

General Psychology 41 43

Developmental and Educational Psychology 40 44

Applied Psychology 34 38

Social Psychology 29 33

Experimental and Cognitive Psychology 16 16

Neuropsychology and Physiological Psychology 10 10

Psychology (miscellaneous) 1 1

Year of publication

2014 54 58

9 2015 60 68

2016 49 54

2017 65 70

Study design

Survey/interview 73 77

Laboratory study 52 53

No empirical data 40 48

Observational study 33 40

Clinical trial 15 15

Multiple study types 6 6

Case study 4 4

Commentary with analysis 3 3

Meta-analysis 2 2

Country of origin

United States of America 99 99

United Kingdom 20 20

Canada 12 13

Germany 11 15

The Netherlands 11 11

28 other countries*(accounting for <10 per country) 75 92

Subjects

10 Human 174 184

Animal 7 7

Both 1 1

Neither human nor animal subjects involved 46 56

Min-max Min-max (median) (median)

Journal impact factor#

2017 0.33-15.07 0.22-15.07 (2.23) (2.06)

Publication year 0.23-20.77 0.22-20.77 (2.09) (2.00) *For all countries see https://osf.io/kg7j5/ # For 111 articles (97 of eligible) no 2017 journal impact factor was available and for 113 articles (99 of eligible) no publication year impact factor was available.

Article availability (‘open access’)

Among the 237 English-language articles, we were able to obtain a publicly available version for

154 (65% [59% to 71%]; Fig. 1A), whereas 74 (31% [25% to 38%]) were only accessible to us through a paywall and 9 (4% [0% to 10%]) were not available to us at all (Fig. 1A).

Materials and protocol availability

Of the 183 articles that involved primary data (see Table 1), 26 contained a statement regarding availability of original research materials such as survey instruments, software, or stimuli (14%

[10% to 19%]; Fig. 1B). Of the 188 articles involving primary or secondary data, zero reported availability of a study protocol (0% [0% to 1%]; Fig. 1C).

11 For the 26 articles where materials were reportedly available, the materials were not actually available for 7 articles due to broken links. For the 19 articles for which we could access the materials, they were made available in the article itself (e.g., in a table or appendix; n = 8), in a journal hosted supplement (n = 6), on a personal or institutionally-hosted (non-repository) webpage (n = 3), or in an online third-party repository (n = 2).

A Article availability B Materials availability

Status: Statement reports: No access Paywall only Publicly available No statement Not available Available N = 237 N = 183

9 74 154 157 26

0% 25% 50% 75% 100% 0% 25% 50% 75% 100% C Protocol availability D Data availability

Statement reports: Statement reports: No statement Available No statement Not available Available N = 188 N = 188

188 184 4

0% 25% 50% 75% 100% 0% 25% 50% 75% 100% E Analysis script availability F Pre−registration

Statement reports: Statement reports: No statement Not available Available No statement Not pre−registered Pre−registered N = 188 N = 188

187 1 183 5

0% 25% 50% 75% 100% 0% 25% 50% 75% 100% G Funding H Conflicts of interest

Statement reports: Statement reports: No statement Unclear Private Public & private Public No funding No statement No conflicts Conflicts N = 228 N = 228

86 8 10 10 96 18 140 76 12 0% 25% 50% 75% 100% 0% 25% 50% 75% 100%

Figure 1. Assessment of transparency and reproducibility-related research practices in psychology. X-axis shows percentage of ‘N’ total articles assessed for a given indicator (which was contingent on study design, see Table 1). Raw counts are shown inside bars.

Data availability

Of the 188 articles that involved primary or secondary data, 4 contained data availability statements (2% [1% to 4%]; Fig. 1D). For one dataset, a fee was required (which we did not pay) to obtain access. Of the 3 accessible datasets, 2 were available via an online third-party

12 repository, and 1 was available in journal-hosted supplementary materials. One dataset was incomplete whereas the remaining 2 appeared complete and clearly documented.

Analysis script availability

Of the 188 articles that involved primary or secondary data, an analysis script was shared for one article (1/188, 1% [0% to 1%]; Fig. 1E), via a third-party repository.

Pre-registration

Of the 188 articles involving primary or secondary data, 5 included a statement regarding pre- registration (3% [1% to 5%]; Fig. 1F). One pre-registration at EUDRA-CT was not accessible to us. The accessible pre-registrations were from ClinicalTrials.gov (n = 2) and the Netherlands

Trials Register (n = 2), and pertained to 3 clinical trials and 1 observational study. All accessible pre-registrations contained information about hypotheses and methods but did not contain analysis plans.

Conflicts of interest and funding statements

Of the 228 English-language and accessible articles, 142 included a statement about funding sources (62% [56% to 69%]; Fig. 1G). Most articles disclosed public funding only (n = 96, 42%),

10 (4%) disclosed private funding only, 10 (4%) disclosed a combination of public and private funding, and 18 (8%) disclosed that the work had received no funding. For 8 articles, we were unable to clearly determine the status of the funding.

Eighty-eight of the 228 articles included a conflicts of interest statement (39% [32% to

45%]; Fig. 1H). Of these 88 articles, most reported that there were no conflicts of interest (n = 76,

86%) and 12 (14%) reported that there was at least one conflict of interest. The content of these

12 statements is summarized in Table 3.

13 Table 3. Frequency of different types of conflict of interest reported in the 12 statements reporting one or more conflicts of interest.

Conflict of interest type Frequency (appearing in n statements)*

Industry related

Authorship/editorship royalties 4

Research funding from industry 4

Served on industry advisory board 4

Consultancy for industry 3

Ownership of relevant commercial products, patents, or 3 procedures

Speaking fees from industry 3

Employed by industry 2

Honoraria from industry 2

Industry equity holder 2

Travel or hospitality awards from industry 2

Other undefined payments from industry 1

Non-industry related

Research funding from government 4

Research funding from foundations, charities, and/or NGOs# 2

14 Consultancy for foundations, charities, and/or NGOs# 1

Honoraria from foundations, charities, and/or NGOs# 1

*Note: as each of the 12 relevant conflict of interest statements may contain more than one type of conflict of interest, the frequency column sums to greater than 12. #NGOs = “Non-governmental organisations”.

Replication and evidence synthesis Of the 188 articles involving primary or secondary data, 10 (5% [3% to 8%]) claimed to include a replication study. Of the 183 articles involving primary data, 1 article (1% [0% to 1%]) was cited by another article that claimed to be a replication. Of the 183 articles involving primary data, 21 were formally included in systematic reviews (11% [8% to 16%]). An additional 8 articles were cited incidentally (there was no intention to include in the review), but not formally included, and a further 1 article was explicitly excluded. 12 were formally included in meta-analyses (7% [4% to 10%]). An additional 2 articles were cited incidentally, but not formally included, and a further

1 article was excluded. Meta-analyses and systematic reviews were coded as exclusive variables

(such that if a systematic review included a meta-analysis, we coded it as a meta-analysis and not a systematic review), thus the 12 meta-analyses and 21 systematic reviews sum to a total of 33 evidence synthesis articles. Overall, the 228 English language and accessible articles tended to be infrequently cited (median = 3, min = 0, max = 74).

Discussion

Our evaluation of transparency and reproducibility-related research practices in a random sample of 250 psychology articles published between 2014-2017 highlights that whilst many articles were publicly available, crucial components of research – protocols, materials, raw data, and

15 analysis scripts – were rarely made publicly available alongside them. Pre-registration remained a nascent proposition with minimal adoption. Disclosure of funding sources and conflicts of interest was modest. Replication or evidence synthesis via meta-analysis or systematic review was relatively infrequent. Although there is evidence that some recent methodological reform initiatives have been effective (Hardwicke et al., 2019; Nelson et al., 2018; Vazire, 2018), the findings of the current study imply that their impact on the psychology literature during the examined period was fairly limited in scope.

For most of the articles (65%) we examined, we were able to access a publicly available version (“open access”). This is higher than recent open access estimates obtained for biomedicine (25%; Wallach et al., 2018) and the social sciences (40%; Hardwicke et al., 2019), as well as a large-scale automated analysis which suggested that 45% of the scientific literature published in 2015 was publicly available (Piwowar et al., 2018). Limiting access to academic publications reduces opportunities for researchers, policy-makers, practitioners, and the general public, to evaluate and make use of scientific evidence. One step psychologists can take to improve public availability of their articles is to upload them to the free preprint server PsyArXiv

(https://psyarxiv.com/). Uploading a preprint does not preclude publication at most journals

(Bourne et al., 2017), though specific policies regarding open access can be checked on the

Sherpa/Romeo database (http://sherpa.ac.uk/romeo/index.php).

Reported availability of research materials was modest in the articles we examined (14%), comparable with recent estimates in the social sciences (11%; Hardwicke et al., 2019), and lower than in biomedicine (33%; Wallach et al., 2018). Several reportedly available sets of materials were in fact not available due to broken links, an example of the ‘link rot’ phenomenon that has been observed by others trying to access research resources (Evangelou et al., 2005; Rowhani-

16 Farid & Barnett, 2018). Availability of original research materials (e.g., survey instruments, stimuli, software, videos) enables comprehensive evaluation of research (during traditional peer review and beyond; Vazire, 2017) and high-fidelity independent replication attempts (Open

Science Collaboration, 2015; Simons, 2014), both of which are important for the verification and systematic accumulation of scientific knowledge (Ioannidis, 2012). Furthermore, re-use of materials reduces waste and enhances efficiency (Chalmers & Glasziou, 2009). Psychologists can share their materials online in various third-party repositories that use stable permalinks, such as the Open Science Framework (OSF; see Klein et al., 2018).

Data availability statements in the articles we examined were extremely uncommon. This is consistent with accumulating evidence which suggests that the data underlying scientific claims is rarely immediately available (Alsheikh-Ali et al., 2011; Iqbal et al., 2016), though some modest improvement has been observed in recent years in biomedicine (Wallach et al., 2018).

Although we did not request data from authors directly, such requests to psychology researchers typically have modest yield (Vanpaemel et al., 2015; Wicherts et al., 2006). Most data appear to be effectively lost, including for some of the most influential psychology and psychiatry studies

(Hardwicke & Ioannidis, 2018b). Vanpaemel et al. (2015), for example, were unable to obtain

62% of the 394 datasets they requested from authors of papers published in four APA journals in

2012. Sharing of raw data, which is the evidence upon which scientists base their claims, enables verification through independent assessment of analytic or computational reproducibility

(Hardwicke et al., 2018; LeBel et al., 2018) and analytic robustness (Steegen et al., 2016). Data sharing also enhances evidence synthesis, such as through individual participant-level meta- analysis (Tierney et al., 2015), and can facilitate discovery, through merging of datasets and re- analysis with novel techniques (Voytek, 2016). Psychologists can improve data availability by

17 uploading raw data to third-party repositories such as the OSF (Klein et al., 2018). Data sharing must be managed with caution if there are ethical concerns, but such concerns do not always preclude all forms of sharing, nor necessarily negate ethical motivations for sharing (Meyer,

2017). Furthermore, when data cannot be made available it is always possible to explicitly declare this in research articles and explain the rationale for not sharing (Morey et al., 2016).

Of the articles we examined, only one shared an analysis script, a dearth consistent with assessments in biomedicine (Wallach et al., 2018), the social sciences (Hardwicke et al., 2019), and biostatistics (Rowhani-Farid & Barnett, 2018). Analysis scripts (a step-by-step description of the analysis in the form of computer code or instructions for re-creating the analysis in point-and- click software) provide the most veridical documentation of how the raw data were filtered, summarized, and analyzed. Verbal descriptions of analysis procedures are often ambiguous, contain errors, or do not adequately capture sufficient detail to enable analytic reproducibility

(Hardwicke et al., 2018; Stodden et al., 2018). Psychologists can share their analysis scripts on a third-party repository, such as the OSF (Klein et al., 2018), and educational resources are available to help researchers improve the quality of their analysis code (Wilson et al., 2017).

Sharing the computational environment in which analysis code successfully runs may also help to promote its longevity and trouble-free transfer to other researchers’ computers (Clyburne-Sherin et al., 2018).

Pre-registration, which involves making a time-stamped, read-only, record of a study’s rationale, hypotheses, methods, and analysis plan on an independent online repository, was rare in the articles we examined. Pre-registration fulfils a number of potential functions (Nosek et al.,

2019), including clarifying the distinction between exploratory and confirmatory aspects of research (Kimmelman et al., 2014; Wagenmakers et al., 2012) and enabling the detection and

18 mitigation of questionable research practices like selective outcome reporting (Franco et al.,

2016; John et al., 2012; Simmons et al., 2011). Pre-registration is relatively new to psychology

(Nosek et al., 2018; Nosek et al. 2019), but similar concepts of registration have a longer history in the context of clinical trials in biomedicine (Dickersin & Rennie, 2012) where they have become the expected norm (Zarin et al., 2017). However, clinical trials represent only a minority of biomedical research, and estimates suggest that pre-registration is overall rare in biomedicine

(Iqbal et al., 2016; Wallach et al., 2018). Pre-registration is also rare in the social sciences

(Hardwicke et al., 2019). There is no doubt that the number of pre-registrations (and the related

‘Registered Reports’ article format) is increasing in psychology (Hardwicke & Ioannidis, 2018a;

Nosek et al., 2018); however, the findings of the present study suggest that efforts to promote pre-registration may not yet have had widespread impact on routine practice. It is important to note that because there is a time-lag between registration and study publication, our measures may underestimate adoption. Whilst norms and standards for pre-registration in psychology are still evolving (Nosek et al., 2019), several dedicated registries, like the OSF, will pre- registrations and detailed guidance is available (Klein et al., 2018).

The current findings suggest that psychology articles were more likely to include funding statements (62%) and conflict of interest statements (39%) than social science articles in general

(31% and 15% respectively; Hardwicke et al., 2019), but less likely than biomedical articles

(69% and 65% respectively; Wallach et al., 2018). It is possible that these disclosure statements are more common than most other practices we examined because they are often mandated by journals (Nutu et al., 2019). Disclosing funding sources and potential conflicts of interest in research articles helps readers to make an informed judgement about risk of bias (Bekelman et al., 2003; Cristea & Ioannidis, 2018). In the absence of established norms or journal mandates,

19 authors may often assume that such statements are not relevant to them (Chivers, 2019).

However, because the absence of a statement is ambiguous, researchers should ideally always include one, even if it is to explicitly declare that there were no funding sources and no potential conflicts.

Of the articles we examined, 5% claimed to be a replication study - slightly higher than a previous estimate in psychology of 1% (Makel et al., 2012) and a similar estimate of 1% in the social sciences (Hardwicke et al., 2019), but comparable with a 5% estimate in biomedicine

(Wallach et al. 2018). Only 1% of the articles we examined were cited by another article that claimed to be a replication attempt. 11% had been included in a systematic review and 7% had been included in a meta-analysis. Replication and evidence synthesis through systematic reviews and meta-analyses help to verify and build upon the existing evidence base. Although the current findings suggest that routine replication and evidence synthesis is relatively rare in psychology, many high-profile replication attempts have been conducted in recent years (Open Science

Collaboration, 2015; Pashler & Wagenmakers, 2012). Additionally, because the articles we examined were published relatively recently, there may be some time lag before relevant replication and evidence synthesis studies emerge. For example, in biomedicine at least, there is a geometric growth in the number of meta-analyses and in many fields multiple meta-analyses are often conducted once several studies appear on the same research question (Ioannidis, 2016).

The current study has several caveats and limitations. Firstly, our findings are based on a random sample of 250 articles, and the obtained estimates may not necessarily generalize to specific contexts, such as other disciplines, sub-fields of psychology, or articles published in particular journals. However, this target sample size was selected to balance informativeness with tractability, and the observed estimates have reasonable precision. Secondly, although the focus

20 of this study was transparency and reproducibility-related practices, this does not imply that adoption of these practices is sufficient to promote the goals they are intended to achieve. For example, poorly documented data may not enable analytic reproducibility (Hardwicke et al.,

2018) and inadequately specified pre-registrations may not sufficiently constrain researcher degrees of freedom (Claesen et al., 2019; Veldkamp et al., 2018). Thirdly, we relied only on published information. Direct requests to authors may have yielded additional information, however, such requests to psychology researchers are often unsuccessful (Hardwicke &

Ioannidis, 2018a; Vanpaemel et al., 2015; Wicherts et al., 2006). Finally, a lack of transparency may have been justified in some cases if there were overriding practical, legal, or ethical concerns

(Meyer, 2017). However, no constraints of this kind were declared in any of the articles we examined.

The present findings imply minimal adoption of transparency and reproducibility-related practices in psychology during the examined time period. Although researchers appear to recognize the problems of low credibility and reproducibility (Baker, 2016) and endorse the values of transparency and reproducibility in principle (Anderson et al., 2010), they are often wary of change (Fuchs et al., 2012; Houtkoop et al., 2018) and routinely neglect these principles in practice (Hardwicke et al., 2019; Iqbal et al., 2016; Wallach et al., 2018). There is unlikely to be a single remedy to this situation. A multifaceted approach will likely be required, with iterative evaluation and careful scrutiny of reform initiatives (Hardwicke et al., 2020). At the educational level, guidance and resources are available to aid researchers (Crüwell et al., 2018;

Klein et al., 2018). At the institutional level, there is evidence that funder and journal policies can be effective at fomenting change (Hardwicke et al., 2018; Nuijten et al., 2018) and these initiatives should be translated and disseminated where relevant. Currently, heterogeneous

21 journal policies (Nutu et al., 2019) may be disrupting efforts to establish norms and promote better standards in routine practice. The Transparency and Openness Promotion (TOP) initiative promises to encourage adoption and standardization of journal policies related to transparency and reproducibility (Nosek et al., 2015), but it remains to be seen how effective this initiative will be in practice. Aligning academic rewards and incentives (e.g., funding awards, publication acceptance, promotion and tenure) with better research practices may also be instrumental in encouraging wider adoption of these practices (Moher et al., 2018).

The present study is one of several efforts to examine the prevalence of transparency and reproducibility-related research practices across scientific disciplines (Hardwicke et al., 2019;

Iqbal et al., 2016; Wallach et al., 2018). Here, we have sketched out some of the topography of psychological territory. Additional studies will be required to fill in as-yet-unexplored areas of the map, and increase the resolution in specific areas (for example, sub-fields of psychology).

Future studies can also add a temporal dimension by comparing to the baseline established here, allowing us to explore the evolution of this landscape over time.

Funding statement

The authors received no specific funding for this work. The Meta-Research Innovation Center at

Stanford (METRICS) is supported by a grant from the Laura and John Arnold Foundation. The

Meta-Research Innovation Center Berlin (METRIC-B) is supported by a grant from the Einstein

Foundation and Stiftung Charité. Robert T. Thibault is supported by a postdoctoral fellowship from the Fonds de la recherche en santé du Québec.

22 Competing interests statement

In the past 36 months, JDW received research support through the Meta Research Innovation Center at

Stanford (METRICS) and the Collaboration for Research Integrity and Transparency (CRIT) at Yale, from the Laura and John Arnold Foundation, and through the Yale-Male Clinic Center for Excellence in

Regulatory Science and Innovation (CERSI) (U01FD005938). All other authors declare no competing interests.

Open practices statement

The study protocol was pre-registered on September 26, 2018, and is available at https://osf.io/q96eh/. All deviations from this protocol are explicitly acknowledged in Supplementary Information A. We report how we determined our sample size, all data exclusions, all manipulations, and all measures in the study. All data (https://osf.io/5qmz7/), materials (https://osf.io/c89sy/), and analysis scripts (https://osf.io/gfjtq/) related to this study are publicly available. To facilitate reproducibility this manuscript was written by interleaving regular prose with analysis code and is available in a Code Ocean container (https://doi.org/10.24433/CO.1618143.v2) which re-creates the software environment in which the original analyses were performed.

Acknowledgements

We thank Kevin Boyack for assistance obtaining the sample of articles.

Author contributions

T.E.H., J.D.W., and J.P.A.I. designed the study. T.E.H., J.D.W., M.C.K., R.T.T., and J.E.K. conducted the data collection. T.E.H. performed the data analysis. T.E.H., J.D.W., R.T.T., and

J.P.A.I. wrote the manuscript. All the authors gave their final approval for publication.

23 References

Alsheikh-Ali, A. A., Qureshi, W., Al-Mallah, M. H., & Ioannidis, J. P. A. (2011). Public

availability of published research data in high-impact journals. PLoS ONE, 6(9), e24357 4.

https://doi.org/10.1371/journal.pone.0024357

Anderson, M. S., Ronning, E. A., Devries, R., & Martinson, B. C. (2010). Extending the

Mertonian Norms: Scientists’ Subscription to Norms of Research. The Journal of Higher

Education, 81(3), 366 393. https://doi.org/10.1353/jhe.0.0095

Baker, M. (2016). Is there a reproducibility crisis? A Nature survey lifts the lid on how

researchers view the 'crisis' rocking science and what they think will help. Nature, 533(7552),

452 454. https://doi.org/10.1038/521274a

Bekelman, J. E., Li, Y., & Gross, C. P. (2003). Scope and Impact of Financial Conflicts of

Interest in Biomedical Research: A Systematic Review. JAMA, 289(4), 454.

https://doi.org/10.1001/jama.289.4.454

Bourne, P. E., Polka, J. K., Vale, R. D., & Kiley, R. (2017). Ten simple rules to consider

regarding preprint submission. PLoS , 13(5), e1005473 6.

https://doi.org/10.1371/journal.pcbi.1005473

Buckheit, J. B., & Donoho, D. L. (1995). WaveLab and Reproducible Research. In P. Bickel, P.

Diggle, S. Fienberg, K. Krickeberg, I. Olkin, N. Wermuth, S. Zeger, A. Antoniadis, & G.

Oppenheim (Eds.), Wavelets and Statistics (Vol. 103, pp. 55–81). Springer New York.

https://doi.org/10.1007/978-1-4612-2544-7_5

24 Chalmers, I., & Glasziou, P. (2009). Viewpoint Avoidable waste in the production and reporting

of research evidence. , 374(9683), 86 89. https://doi.org/10.1016/s0140-

6736(09)60329-9

Claesen, A., Gomes, S. L. B. T., Tuerlinckx, F., & vanpaemel. (2019). Preregistration:

Comparing Dream to Reality [Preprint]. PsyArXiv. https://doi.org/10.31234/osf.io/d8wex

Chivers, T. (2019). Does psychology have a conflict-of-interest problem? Nature, 571 (7763),

20–23. https://doi.org/10.1038/d41586-019-02041-5

Clyburne-Sherin, A., Fei, X., & Green, S. A. (2018). Computational Reproducibility via

Containers in Psychology [Preprint]. PsyArXiv. https://doi.org/10.31234/osf.io/mf82t

Cristea, I.-A., & Ioannidis, J. P. A. (2018). Improving Disclosure of Financial Conflicts of

Interest for Research on Psychosocial Interventions. JAMA Psychiatry.

https://doi.org/10.1001/jamapsychiatry.2018.0382

Crüwell, S., van Doorn, J., Etz, A., Makel, M. C., Moshontz, H., Niebaum, J., Orben, A.,

Parsons, S., & Schulte-Mecklenbeck, M. (2018). 7 Easy Steps to Open Science: An Annotated

Reading List [Preprint]. PsyArXiv. https://doi.org/10.31234/osf.io/cfzyx

David, P. A. (2008). The Historical Origins of ’Open Science’: An Essay on Patronage,

Reputation and Common Agency Contracting in the Scientific Revolution. Capitalism and

Society, 3(2). https://doi.org/10.2202/1932-0213.1040

Dickersin, K., & Rennie, D. (2012). The Evolution of Trial Registries and Their Use to Assess

the Clinical Trial Enterprise. JAMA, 307(17), 1861–1864.

https://doi.org/10.1001/jama.2012.4230

25 Evangelou, E., Trikalinos, T. A., & Ioannidis, J. P. A. (2005). Unavailability of online

supplementary scientific information from articles published in major journals. The FASEB

Journal, 19(14), 1943–1944. https://doi.org/10.1096/fj.05-4784lsf

Franco, A., Malhotra, N., & Simonovits, G. (2016). Underreporting in Psychology .

Social Psychological and Personality Science, 7(1), 8–12.

https://doi.org/10.1177/1948550615598377

Fuchs, H. M., Jenny, M., & Fiedler, S. (2012). Psychologists Are Open to Change, yet Wary of

Rules. Perspectives on Psychological Science, 7(6), 639–642.

https://doi.org/10.1177/1745691612459521

Hardwicke, T. E., & Ioannidis, J. P. A. (2018a). Mapping the universe of registered reports.

Nature Human Behaviour, 2(11), 793–796. https://doi.org/10.1038/s41562-018-0444-y

Hardwicke, T. E., & Ioannidis, J. P. A. (2018b). Populating the Data Ark: An attempt to retrieve,

preserve, and liberate data from the most highly-cited psychology and psychiatry articles.

PLOS ONE, 13(8), e0201856. https://doi.org/10.1371/journal.pone.0201856

Hardwicke, T. E., Mathur, M. B., MacDonald, K., Nilsonne, G., Banks, G. C., Kidwell, M. C.,

Mohr, A. H., Clayton, E., Yoon, E. J., Tessler, M. H., Lenne, R. L., Altman, S., Long, B., &

Frank, M. C. (2018). Data availability, reusability, and analytic reproducibility: Evaluating

the impact of a mandatory open data policy at the journal Cognition. Royal Society Open

Science, 5(8), 180448. https://doi.org/10.1098/rsos.180448

Hardwicke, T. E., Serghiou, S., Janiaud, P., Danchev, V., Crüwell, S., Goodman, S., & Ioannidis,

J. P. A. (2020). Calibrating the scientific ecosystem through meta-research. Annual Review of

26 Statistics and Its Application, 7, 11-37. https://doi.org/10.1146/annurev-statistics-031219-

041104

Hardwicke, T. E., Wallach, J. D., Kidwell, M., Bendixen, T., Crüwell, S., & Ioannidis, J. P. A.

(2019). An empirical assessment of transparency and reproducibility-related research

practices in the social sciences (2014-2017). Royal Society Open Science, 7: 190806.

https://doi.org/10.1098/rsos.190806

Houtkoop, B. L., Chambers, C., Macleod, M., Bishop, D. V. M., Nichols, T. E., & Wagenmakers,

E.-J. (2018). Data Sharing in Psychology: A Survey on Barriers and Preconditions. Advances

in Methods and Practices in Psychological Science, 1(1), 70–85.

https://doi.org/10.1177/2515245917751886

Ioannidis, J. P. (2016). The mass production of redundant, misleading, and conflicted systematic

reviews and meta-analyses: mass production of systematic reviews and meta-analyses. The

Milbank Quarterly, 94(3), 485–514. https://doi.org/10.1111/1468-0009.12210

Ioannidis, J. P. A. (2005). Why most published research findings are false. PLoS Medicine, 2(8),

e124. https://doi.org/10.1371/journal.pmed.0020124

Ioannidis, J. P. A. (2012). Why Science Is Not Necessarily Self-Correcting. Perspectives on

Psychological Science, 7(6), 645–654. https://doi.org/10.1177/1745691612464056

Ioannidis, J. P. A., Greenland, S., Hlatky, M. A., Khoury, M. J., Macleod, M. R., Moher, D.,

Schulz, K. F., & Tibshirani, R. (2014). Increasing value and reducing waste in research

design, conduct, and analysis. The Lancet, 383(9912), 166 175.

https://doi.org/10.1016/s0140-6736(13)62227-8

27 Ioannidis, J. P. A., Munafò, M. R., Fusar-Poli, P., Nosek, B. A., & David, S. P. (2014).

Publication and other reporting biases in cognitive sciences: Detection, prevalence, and

prevention. Trends in Cognitive Sciences, 18(5), 235 241.

https://doi.org/10.1016/j.tics.2014.02.010

Iqbal, S. A., Wallach, J. D., Khoury, M. J., Schully, S. D., & Ioannidis, J. P. A. (2016).

Reproducible research practices and transparency across the biomedical literature. PLoS

Biology, 14(1), e1002333 13. https://doi.org/10.1371/journal.pbio.1002333

John, L. K., Loewenstein, G., & Prelec, D. (2012). Measuring the prevalence of questionable

research practices with incentives for truth telling. Psychological Science, 23(5), 524 532.

https://doi.org/10.1177/0956797611430953

Kidwell, M. C., Lazarevic, L. B., Baranski, E., Hardwicke, T. E., Piechowski, S., Falkenberg, L.-

S., Kennett, C., Slowik, A., Sonnleitner, C., Hess-Holden, C., Errington, T. M., Fiedler, S., &

Nosek, B. A. (2016). Badges to Acknowledge Open Practices: A Simple, Low-Cost, Effective

Method for Increasing Transparency. PLOS Biology, 14(5), e1002456.

https://doi.org/10.1371/journal.pbio.1002456

Kimmelman, J., Mogil, J. S., & Dirnagl, U. (2014). Distinguishing between Exploratory and

Confirmatory Preclinical Research Will Improve Translation. PLoS Biology, 12(5), e1001863

4. https://doi.org/10.1371/journal.pbio.1001863

Klein, O., Hardwicke, T. E., Aust, F., Breuer, J., Danielsson, H., Mohr, A. H., Ijzerman, H.,

Nilsonne, G., Vanpaemel, W., & Frank, M. C. (2018). A Practical Guide for Transparency in

Psychological Science. Collabra: Psychology, 4(1), 20. https://doi.org/10.1525/collabra.158

28 LeBel, E. P., McCarthy, R. J., Earp, B. D., Elson, M., & Vanpaemel, W. (2018). A Unified

Framework to Quantify the Credibility of Scientific Findings. Advances in Methods and

Practices in Psychological Science, 1(3), 389–402.

https://doi.org/10.1177/2515245918787489

Makel, M. C., Plucker, J. A., & Hegarty, B. (2012). Replications in Psychology Research.

Perspectives on Psychological Science, 7(6), 537–542.

https://doi.org/10.1177/1745691612460688

Meyer, M. N. (2017). Practical Tips for Ethical Data Sharing. Advances in Methods and

Practices in Psychological Science, 1(1), 131–144.

https://doi.org/10.1177/2515245917747656

Miguel, E., Camerer, C., Casey, K., Cohen, J., Esterling, K. M., Gerber, A., Glennerster, R.,

Green, D. P., Humphreys, M., Imbens, G., Laitin, D., Madon, T., Nelson, L., Nosek, B. A.,

Petersen, M., Sedlmayr, R., Simmons, J. P., Simonsohn, U., & der Laan, M. V. (2014).

Promoting Transparency in Social Science Research. Science, 343(6166), 30–31.

https://doi.org/10.1126/science.1245317

Morey, R. D., Chambers, C. D., Etchells, P. J., Harris, C. R., Hoekstra, R., Lakens, D.,

Lewandowsky, S., Morey, C. C., Newman, D. P., Schönbrodt, F. D., Vanpaemel, W.,

Wagenmakers, E.-J., & Zwaan, R. A. (2016). The Peer Reviewers’ Openness Initiative:

Incentivizing open research practices through peer review. Royal Society Open Science, 3(1),

150547. https://doi.org/10.1098/rsos.150547

29 Munafò, M. R., Nosek, B. A., Bishop, D. V. M., Button, K. S., Chambers, C. D., du Sert, N. P.,

Simonsohn, U., Wagenmakers, E.-J., Ware, J. J., & Ioannidis, J. P. A. (2017). A manifesto for

reproducible science. Nature Human Behaviour, 1(1), 1 9. https://doi.org/10.1038/s41562-

016-0021

Naudet, F., Sakarovitch, C., Janiaud, P., Cristea, I., Fanelli, D., Moher, D., & Ioannidis, J. P. A.

(2018). Data sharing and reanalysis of randomized controlled trials in leading biomedical

journals with a full data sharing policy: Survey of studies published in The BMJ and PLOS

Medicine. BMJ, 360, k400. https://doi.org/10.1136/bmj.k400

Nelson, L. D., Simmons, J. P., & Simonsohn, U. (2018). Psychology’s Renaissance. Annual

Review of Psychology, 69(1), 511 534. https://doi.org/10.1146/annurev-psych-122216-

011836

Nosek, B. A., Alter, G., Banks, G. C., Borsboom, D., Bowman, S. D., Breckler, S. J., Buck, S.,

Chambers, C. D., Chin, G., Christensen, G., Contestabile, M., Dafoe, A., Eich, E., Freese, J.,

Glennerster, R., Goroff, D., Green, D. P., Hesse, B., Humphreys, M., … Yarkoni, T. (2015).

Promoting an open research culture. Science, 348(6242), 1422 1425.

https://doi.org/10.1126/science.aab2374

Nosek, B. A., Beck, E. D., Campbell, L., Flake, J. K., Hardwicke, T. E., Mellor, D. T., van ’t

Veer, A. E., & Vazire, S. (2019). Preregistration Is Hard, And Worthwhile. Trends in

Cognitive Sciences. https://doi.org/10.1016/j.tics.2019.07.009

30 Nosek, B. A., Ebersole, C. R., DeHaven, A. C., & Mellor, D. T. (2018). The preregistration

revolution. Proceedings of the National Academy of Sciences of the United States of America,

115(11), 2600 2606. https://doi.org/10.1073/pnas.1708274114

Nuijten, M. B., Borghuis, J., Veldkamp, C. L. S., Dominguez-Alvarez, L., Assen, M. A. L. M. V.,

& Wicherts, J. M. (2018). Journal Data Sharing Policies and Statistical Reporting

Inconsistencies in Psychology. Collabra: Psychology, 3(1), 31.

https://doi.org/10.1525/collabra.102

Nutu, D., Gentili, C., Naudet, F., & Cristea, I. A. (2019). Open science practices in clinical

psychology journals: An audit study. Journal of Abnormal Psychology, 128(6), 510–516.

https://doi.org/10.1037/abn0000414

Open Science Collaboration. (2015). Estimating the reproducibility of psychological science.

Science, 349(6251), aac4716. https://doi.org/10.1126/science.aac4716

Pashler, H., & Wagenmakers, E.-J. (2012). Editors’ Introduction to the Special Section on

Replicability in Psychological Science. Perspectives on Psychological Science, 7(6), 528–

530. https://doi.org/10.1177/1745691612465253

Piwowar, H., Priem, J., Lariviere, V., Alperin, J. P., Matthias, L., Norlander, B., Farley, A., West,

J., & Haustein, S. (2018). The state of OA: A large-scale analysis of the prevalence and

impact of Open Access articles. PeerJ, 6, e4375. https://doi.org/10.7717/peerj.4375

Rowhani-Farid, A., & Barnett, A. G. (2018). Badges for sharing data and code at Biostatistics:

An observational study. F1000Research, 7, 90.

https://doi.org/10.12688/f1000research.13477.2

31 Rowhani-Farid, A., & Barnett, A. G. (2016). Has open data arrived at the British Medical Journal

(BMJ)? An observational study. BMJ Open, 6(10), e011784 9.

https://doi.org/10.1136/bmjopen-2016-011784

Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology: Undisclosed

flexibility in data collection and analysis allows presenting anything as significant.

Psychological Science, 22(11), 1359 1366. https://doi.org/10.1177/0956797611417632

Simons, D. J. (2014). The Value of Direct Replication. Perspectives on Psychological Science,

9(1), 76–80. https://doi.org/10.1177/1745691613514755

Sison, C. P., & Glaz, J. (1995). Simultaneous confidence intervals and sample size determination

for multinomial proportions. Journal of the American Statistical Association, 90(429), 366

369. https://doi.org/10.1080/01621459.1995.10476521

Steegen, S., Tuerlinckx, F., Gelman, A., & Vanpaemel, W. (2016). Increasing Transparency

Through a Multiverse Analysis. Perspectives on Psychological Science, 11(5), 702 712.

https://doi.org/10.1177/1745691616658637

Stodden, V., Seiler, J., & Ma, Z. (2018). An empirical analysis of journal policy effectiveness for

computational reproducibility. Proceedings of the National Academy of Sciences, 115(11),

201708290. https://doi.org/10.1073/pnas.1708290115

Szucs, D., & Ioannidis, J. P. A. (2017). Empirical assessment of published effect sizes and power

in the recent cognitive neuroscience and psychology literature. PLOS Biology, 15(3),

e2000797. https://doi.org/10.1371/journal.pbio.2000797

32 Tierney, J. F., Vale, C., Riley, R., Smith, C. T., Stewart, L., Clarke, M., & Rovers, M. (2015).

Individual Participant Data (IPD) Meta-analyses of Randomised Controlled Trials: Guidance

on Their Use. PLoS Medicine, 12(7), e1001855 16.

https://doi.org/10.1371/journal.pmed.1001855

Vanpaemel, W., Vermorgen, M., Deriemaecker, L., & Storms, G. (2015). Are we wasting a good

crisis? The availability of psychological research data after the storm. Collabra, 1(1), 1 5.

https://doi.org/10.1525/collabra.13

Vazire, S. (2017). Quality Uncertainty Erodes Trust in Science. Collabra: Psychology, 3(1), 1.

https://doi.org/10.1525/collabra.74

Vazire, S. (2018). Implications of the Credibility Revolution for Productivity, Creativity, and

Progress. Perspectives on Psychological Science, 13(4), 411–417.

https://doi.org/10.1177/1745691617751884

Veldkamp, C. L. S., Bakker, M., van Assen, M. A. L. M., Crompvoets, E. A. V., Ong, H. H.,

Nosek, B. A., Soderberg, C. K., Mellor, D. T., & Wicherts, J. M. (2018). Ensuring the quality

and specificity of preregistrations [Preprint]. PsyArXiv. https://doi.org/10.31234/osf.io/cdgyh

Voytek, B. (2016). The Virtuous Cycle of a Data Ecosystem. PLoS Computational Biology,

12(8), e1005037 6. https://doi.org/10.1371/journal.pcbi.1005037

Wagenmakers, E.-J., Wetzels, R., Borsboom, D., van der Maas, H. L. J., & Kievit, R. A. (2012).

An Agenda for Purely Confirmatory Research. Perspectives on Psychological Science, 7(6),

632–638. https://doi.org/10.1177/1745691612463078

33 Wallach, J. D., Boyack, K. W., & Ioannidis, J. P. A. (2018). Reproducible research practices,

transparency, and open access data in the biomedical literature, 20152017. PLOS Biology,

16(11), e2006930. https://doi.org/10.1371/journal.pbio.2006930

Wallach, J. D., Gonsalves, G. S., & Ross, J. S. (2018). Research, regulatory, and clinical

decision-making: The importance of scientific integrity. Journal of Clinical Epidemiology,

93, 88–93. https://doi.org/10.1016/j.jclinepi.2017.08.021

Wicherts, J. M., Borsboom, D., Kats, J., & Molenaar, D. (2006). The poor availability of

psychological research data for reanalysis. American Psychologist, 61(7), 726 728.

https://doi.org/10.1037/0003-066x.61.7.726

Wilson, G., Bryan, J., Cranston, K., Kitzes, J., Nederbragt, L., & Teal, T. K. (2017). Good

enough practices in scientific computing. PLOS Computational Biology, 13(6), e1005510.

https://doi.org/10.1371/journal.pcbi.1005510

Zarin, D. A., Tse, T., Williams, R. J., & Rajakannan, T. (2017). Update on trial registration 11

Years after the ICMJE policy was established. The New Journal of Medicine, 4, 383

391. https://doi.org/10.1056/nejmsr1601330

SUPPLEMENTARY INFORMATION

Supplementary Information A. Pre-registered protocol updates

During the course of the study, we made several minor adjustments to the pre-registered protocol. These are listed below.

34 (1) For the questions “How does the statement indicate the materials are available?” and “How does the statement indicate the data are available?” we added an explicit response option: “Appendix within the present article”. (2) For the questions “Does the article state whether or not the materials are available?” and “Does the article state whether or not the data are available?” we added an explicit response option “Source of data/materials provided but no explicit availability statement”. However, because this was introduced late in the data collection process it was not applied consistently and we have elected to re-code this response option as “No - there is no data availability statement”. (3) For the questions “What type of study is being reported?” we changed the response option ‘survey’ to ‘survey/interview’ and renamed ‘field study’ to ‘observational study’. We did this to better match our coding decisions. (4) For the citation evaluation, we added two questions: “How many times was the article cited incidentally in a meta-analysis (i.e., no intention to include in data synthesis)?” and “How many times was the article cited incidentally in a systematic review (i.e., no intention to include in formal review)?” (5) For the questions “Does the article include a statement indicating whether there were funding sources?” we added the response option “Yes - the statement identifies funding sources but the private/public status is unclear” (6) To concisely present the content of conflict of interest statements we categorized them into the types shown in Table 3. This analysis was not pre-registered. (7) To align the extraction form with previous studies (e.g., Wallach et al., 2019) we included an additional response option (“Empirical data - cost effectiveness and/or decision analysis”) for the question “What type of study is being reported?”

Supplementary Information B. Inter-rater reliability

We computed inter-rater reliability using Fleiss’ Kappa (see Supplementary Table 1) for article access and availability statements for key research resources (as shown in Figure 1). All coding differences were resolved through discussion.

Supplementary Table 1. Inter-rater reliability (Fleiss’ Kappa) for article access and availability statements for key research resources.

Article Materials Protocol Data Analysis Pre-registration Conflicts of Funding scripts interest

0.86 0.78 0.71 0.61 0.75 0.66 0.96 0.92

35

36