Estimating the prevalence of transparency and reproducibility-related research practices in psychology (2014-2017)
Tom E. Hardwicke1,2, Robert T. Thibault3,4, Jessica E. Kosie5,6, Joshua D. Wallach7,8, Mallory C. Kidwell9, & John P. A. Ioannidis2,10,11
1 Department of Psychology, University of Amsterdam
2 Meta-Research Innovation Center Berlin (METRIC-B), QUEST Center for Transforming Biomedical Research, Berlin Institute of Health, Charité – Universitätsmedizin Berlin
3 School of Psychological Science, University of Bristol
4 MRC Integrative Epidemiology Unit at the University of Bristol
5 Department of Psychology, University of Oregon
6 Department of Psychology, Princeton University
7 Department of Environmental Health Sciences, Yale School of Public Health
8 Collaboration for Research Integrity and Transparency, Yale School of Medicine
9 Department of Psychology, University of Utah
10 Departments of Medicine, of Health Research and Policy, of Biomedical Data Science, and of Statistics, Stanford University
11 Meta-Research Innovation Center at Stanford (METRICS), Stanford University
Author note
Tom Hardwicke was based at Stanford University when the study began and is now based at the University of Amsterdam. Jessica Kosie was based at University of Oregon when the study began and is now based at Princeton University. Correspondence concerning this article should be addressed to Tom Hardwicke, Nieuwe Achtergracht 129B, Department of Psychology, University of Amsterdam, 1018 WT Amsterdam, The Netherlands. E-mail: [email protected]
1 Abstract
Psychologists are navigating an unprecedented period of introspection about the credibility and utility of their discipline. Reform initiatives have emphasized the benefits of several transparency and reproducibility-related research practices; however, their adoption across the psychology literature is unknown. To estimate their prevalence, we manually examined a random sample of
250 psychology articles published between 2014-2017. Over half of the articles were publicly available (154/237, 65% [95% confidence interval, 59%-71%]); however, sharing of research materials (26/183, 14% [10%-19%]), study protocols (0/188, 0% [0%-1%]), raw data (4/188, 2%
[1%-4%]), and analysis scripts (1/188, 1% [0%-1%]) was rare. Pre-registration was also uncommon (5/188, 3% [1%-5%]). Many articles included a funding disclosure statement
(142/228, 62% [56%-69%]), but conflict of interest statements were less common (88/228, 39%
[32%-45%]). Replication studies were rare (10/188, 5% [3%-8%]) and few studies were included in systematic reviews (21/183, 11% [8%-16%]) or meta-analyses (12/183, 7% [4%-10%]).
Overall, the results suggest that transparency and reproducibility-related research practices were far from routine. These findings establish a baseline which can be used to assess future progress towards increasing the credibility and utility of psychology research.
Keywords: transparency, reproducibility, meta-research, psychology, open science
2 Introduction
Serious concerns about the credibility and utility of some scientific literature (Ioannidis, 2005;
Ioannidis et al., 2014) have prompted calls for increased adoption of research practices that enhance reproducibility and transparency (Miguel et al., 2014; Munafò et al., 2017; Nosek et al.,
2015; Wallach et al., 2018). Close scrutiny of psychology in particular has suggested that standard research and publication practices have rendered the discipline highly exposed to bias, potentially resulting in a large volume of exaggerated and misleading results (Ioannidis et al.,
2014; John et al., 2012; Open Science Collaboration, 2015; Pashler & Wagenmakers, 2012;
Simmons et al., 2011; Szucs & Ioannidis, 2017). This realization within the field has also led to a number of reform efforts (Hardwicke & Ioannidis, 2018a; Hardwicke et al., 2019; Nelson et al.,
2018; Vazire, 2018), which have the potential to improve efficiency (Chalmers & Glasziou,
2009), facilitate self-correction (Ioannidis, 2012), and enhance credibility (Vazire, 2017).
A central focus of reform initiatives has been to encourage scientists to share more information about the studies they perform. Journal articles are only the most visible facade of deeper layers of scholarship that may include protocols, original research materials, raw data, and analysis scripts – resources that are not necessarily shared with other scientists (Buckheit &
Donoho, 1995; Klein et al., 2018). Even journal articles themselves may only be accessible to those with institutional access or the ability to pay a fee (Piwowar et al., 2018). Additionally, potential sources of bias, such as conflicts of interest and funding sources, may not be disclosed
(Bekelman et al., 2003; Cristea & Ioannidis, 2018). However, there is a growing (or re-emerging,
David, 2008) appreciation that the scientific community needs to be able to access all of this information in order to comprehensively evaluate, interpret, and independently verify scientific claims (Vazire, 2017; Munafò et al., 2017). Furthermore, access to this information enables
3 replication, evidence synthesis, and discovery activities that may ultimately accelerate scientific progress (Ioannidis, 2012; Klein et al., 2018).
The burgeoning discipline of meta-research (‘research on research’) has already begun to evaluate the impact of various reform initiatives (Hardwicke et al., 2020). For example, journal data sharing policies have been associated with moderate to substantial increases in data sharing
(Hardwicke et al., 2018; Kidwell et al., 2016; Nuijten et al., 2018; Rowhani-Farid & Barnett,
2016; Naudet et al., 2018). However, the prevalence of transparency and reproducibility-related research practices across the psychology literature is largely unknown. Based on previous investigations in biomedicine (Iqbal et al., 2016; Wallach et al., 2018) and the social sciences
(Hardwicke et al., 2019), we manually examined a random sample of 250 articles to estimate the prevalence of several transparency and reproducibility-related indicators in psychology articles published between 2014-2017. The indicators were open access to published articles; availability of study materials, study protocols, raw data, and analysis scripts; pre-registration; disclosure of funding sources and conflicts of interest; conduct of replication studies; and cumulative synthesis of evidence in meta-analyses and systematic reviews.
Methods
Design
This was a retrospective observational study with a cross-sectional design. Sampling units were individual articles. Measured variables are shown in Table 1.
4 Table 1. Measured variables. The variables measured for an individual article depended on the study design classification. For articles that were not available (the full text could not be retrieved) or non-English language, only article characteristics were obtained. The exact operational definitions and procedures for data extraction/coding are available in the structured form here: https://osf.io/x9rmy/
Applicable study designs
Article characteristics
Subject area, year of publication, study design, All country of origin (based on corresponding author's affiliation), human/animal subjects, 2017 journal impact factor (according to Thomson Reuters Journal Citation Reports)
Articles
Accessibility and Retrieval Method (can the All article be accessed, is there a public version or is paywall access required?)
Protocols
Availability statement (is availability, or lack of, Study designs involving primary data#, study explicitly declared?) designs involving secondary data (commentaries with analysis and meta-analyses). Content (what aspects of the study are included in the protocol?)
Materials
Availability statement (is availability, or lack of, Study designs involving primary data# explicitly declared?)
Retrieval Method (e.g., upon request or via online repository)
Accessibility (can the materials be accessed?)
Raw data
Availability statement (is availability, or lack of, explicitly declared?)
5 Retrieval Method (e.g., upon request or via online Study designs involving primary data#, study repository) designs involving secondary data (commentaries with analysis and meta-analyses). Accessibility (can the data be accessed?)
Content (has all relevant data been shared?)
Documentation (are the data understandable?)
Analysis scripts
Availability statement (is availability, or lack of, Study designs involving primary data#, study explicitly declared?) designs involving secondary data (commentaries with analysis and meta-analyses). Retrieval Method (e.g., upon request or via online repository)
Accessibility (can the scripts be accessed?)
Pre-registration
Availability statement (is availability, or lack of, Study designs involving primary data#, study explicitly declared?) designs involving secondary data (commentaries with analysis and meta-analyses). Retrieval Method (which registry was used?)
Accessibility (can the pre-registration be accessed?)
Content (what was pre-registered?)
Funding
Disclosure statement (are funding sources, or lack All of, explicitly declared?)
Conflicts of interest
Disclosure statement (are conflicts of interest, or All lack of, explicitly declared?)
Replication
Statement (does the article claim to report a All replication?)
6 Citation history (has the article been cited by a Study designs involving primary data# study that claims to be a replication?)
Evidence synthesis
Meta-analysis citation history* (has the article Study designs involving primary data# been cited by, and included in the evidence- synthesis component of, a meta-analysis)
Systematic review citation history* (has the article Study designs involving primary data# been cited by, and included in the evidence- synthesis component of, a systematic review)
# ‘Study designs involving primary data’ encompasses the following study design classifications: field studies, laboratory studies, surveys, case studies, multiple types, clinical trial, and ‘other’ designs. * Meta-analysis and systematic review variables were coded as exclusive variables.
Sample
We obtained a random sample of 250 psychology articles published between 2014 and 2017.
Specifically, we used a random number generator to sample 250 articles from all 224,556 documents in the Scopus database (as of 22nd September, 2018) designated with the document type “article” or “review”, published between 2014 and 2017, and with an All Science Journal
Classification (ASJC) code related to psychology (i.e., ASJC codes 3200, 3201, 3202, 3203,
3204, 3205, 3206, 3207). This classification included 1323 journals and all of the subject areas shown in Table 2. The sample size was based on our judgement of what was both informative and tractable.
Procedure
Data collection took place between September 27th, 2018 and September 23rd, 2019. Data extraction for the measured variables shown in Table 1 involved close manual examination of the articles supplemented with keyword searching guided by a Google form (https://osf.io/x9rmy/) based on previous investigations in biomedicine (Iqbal et al., 2016; Wallach et al., 2018) and the
7 social sciences (Hardwicke et al., 2019). As indicated in the second column of the table, the exact variables measured for a given article depended on study design. Attempts to access each article were made by searching with the Open Access Button (https://openaccessbutton.org/), Google
(https://www.google.com/), and if necessary, at least two of the online libraries of the coders’ affiliated universities. For articles that were not available (the full text could not be retrieved) or non-English language, only article characteristics were obtained.
Three investigators performed initial extraction and coding; articles were assigned to each investigator on demand by randomly selecting from remaining articles (JDW, n = 32; JK, n = 93;
RT, n = 125). A single investigator obtained subject area, year of publication, and citation histories from Scopus and 2017 journal impact factors from Thomson Reuters Journal Citation
Reports. All other variables in Table 1 were second coded by one of two other investigators
(TEH, n = 198; MCK, n = 52) and any coding differences were resolved through discussion.
Inter-reliability assessments are shown in Supplementary Table 1. Assuming all articles were published on 1st July of their respective publication year (i.e., half way through the year), the time between publication and recording citation information ranged from 448 to 1544 (median = 996) days for number of citations and 456 to 1900 (median = 1178) days for number of citing articles that were replications, systematic reviews, and/or meta-analyses. Conflict of interest statements
(Table 3) were categorized by two investigators (TEH and JDW) in an exploratory manner (i.e., not pre-registered).
Analysis
Results are descriptive and focus on the proportion of articles that fulfil each of the evaluated indicators, using as a denominator the number of articles where each indicator is applicable (see
8 Table 1). We also report 95% confidence intervals (in square brackets) based on the Sison-Glaz method for multinomial proportions (Sison & Glaz, 1995).
Results
Sample characteristics
Sample characteristics for all 250 articles and for the 228 articles that were eligible for in-depth data extraction (i.e., English-language and accessible) are displayed in Table 2.
Table 2. Sample characteristics for the 250 randomly sampled articles and the 228 English language and accessible articles that were eligible for in depth data extraction. Information on study design and subjects was not available for 2 articles which could not be accessed and did not have abstracts.
Eligible All articles articles
n n
Subject area
Clinical Psychology 57 65
General Psychology 41 43
Developmental and Educational Psychology 40 44
Applied Psychology 34 38
Social Psychology 29 33
Experimental and Cognitive Psychology 16 16
Neuropsychology and Physiological Psychology 10 10
Psychology (miscellaneous) 1 1
Year of publication
2014 54 58
9 2015 60 68
2016 49 54
2017 65 70
Study design
Survey/interview 73 77
Laboratory study 52 53
No empirical data 40 48
Observational study 33 40
Clinical trial 15 15
Multiple study types 6 6
Case study 4 4
Commentary with analysis 3 3
Meta-analysis 2 2
Country of origin
United States of America 99 99
United Kingdom 20 20
Canada 12 13
Germany 11 15
The Netherlands 11 11
28 other countries*(accounting for <10 per country) 75 92
Subjects
10 Human 174 184
Animal 7 7
Both 1 1
Neither human nor animal subjects involved 46 56
Min-max Min-max (median) (median)
Journal impact factor#
2017 0.33-15.07 0.22-15.07 (2.23) (2.06)
Publication year 0.23-20.77 0.22-20.77 (2.09) (2.00) *For all countries see https://osf.io/kg7j5/ # For 111 articles (97 of eligible) no 2017 journal impact factor was available and for 113 articles (99 of eligible) no publication year impact factor was available.
Article availability (‘open access’)
Among the 237 English-language articles, we were able to obtain a publicly available version for
154 (65% [59% to 71%]; Fig. 1A), whereas 74 (31% [25% to 38%]) were only accessible to us through a paywall and 9 (4% [0% to 10%]) were not available to us at all (Fig. 1A).
Materials and protocol availability
Of the 183 articles that involved primary data (see Table 1), 26 contained a statement regarding availability of original research materials such as survey instruments, software, or stimuli (14%
[10% to 19%]; Fig. 1B). Of the 188 articles involving primary or secondary data, zero reported availability of a study protocol (0% [0% to 1%]; Fig. 1C).
11 For the 26 articles where materials were reportedly available, the materials were not actually available for 7 articles due to broken links. For the 19 articles for which we could access the materials, they were made available in the article itself (e.g., in a table or appendix; n = 8), in a journal hosted supplement (n = 6), on a personal or institutionally-hosted (non-repository) webpage (n = 3), or in an online third-party repository (n = 2).
A Article availability B Materials availability
Status: Statement reports: No access Paywall only Publicly available No statement Not available Available N = 237 N = 183
9 74 154 157 26
0% 25% 50% 75% 100% 0% 25% 50% 75% 100% C Protocol availability D Data availability
Statement reports: Statement reports: No statement Available No statement Not available Available N = 188 N = 188
188 184 4
0% 25% 50% 75% 100% 0% 25% 50% 75% 100% E Analysis script availability F Pre−registration
Statement reports: Statement reports: No statement Not available Available No statement Not pre−registered Pre−registered N = 188 N = 188
187 1 183 5
0% 25% 50% 75% 100% 0% 25% 50% 75% 100% G Funding H Conflicts of interest
Statement reports: Statement reports: No statement Unclear Private Public & private Public No funding No statement No conflicts Conflicts N = 228 N = 228
86 8 10 10 96 18 140 76 12 0% 25% 50% 75% 100% 0% 25% 50% 75% 100%
Figure 1. Assessment of transparency and reproducibility-related research practices in psychology. X-axis shows percentage of ‘N’ total articles assessed for a given indicator (which was contingent on study design, see Table 1). Raw counts are shown inside bars.
Data availability
Of the 188 articles that involved primary or secondary data, 4 contained data availability statements (2% [1% to 4%]; Fig. 1D). For one dataset, a fee was required (which we did not pay) to obtain access. Of the 3 accessible datasets, 2 were available via an online third-party
12 repository, and 1 was available in journal-hosted supplementary materials. One dataset was incomplete whereas the remaining 2 appeared complete and clearly documented.
Analysis script availability
Of the 188 articles that involved primary or secondary data, an analysis script was shared for one article (1/188, 1% [0% to 1%]; Fig. 1E), via a third-party repository.
Pre-registration
Of the 188 articles involving primary or secondary data, 5 included a statement regarding pre- registration (3% [1% to 5%]; Fig. 1F). One pre-registration at EUDRA-CT was not accessible to us. The accessible pre-registrations were from ClinicalTrials.gov (n = 2) and the Netherlands
Trials Register (n = 2), and pertained to 3 clinical trials and 1 observational study. All accessible pre-registrations contained information about hypotheses and methods but did not contain analysis plans.
Conflicts of interest and funding statements
Of the 228 English-language and accessible articles, 142 included a statement about funding sources (62% [56% to 69%]; Fig. 1G). Most articles disclosed public funding only (n = 96, 42%),
10 (4%) disclosed private funding only, 10 (4%) disclosed a combination of public and private funding, and 18 (8%) disclosed that the work had received no funding. For 8 articles, we were unable to clearly determine the status of the funding.
Eighty-eight of the 228 articles included a conflicts of interest statement (39% [32% to
45%]; Fig. 1H). Of these 88 articles, most reported that there were no conflicts of interest (n = 76,
86%) and 12 (14%) reported that there was at least one conflict of interest. The content of these
12 statements is summarized in Table 3.
13 Table 3. Frequency of different types of conflict of interest reported in the 12 statements reporting one or more conflicts of interest.
Conflict of interest type Frequency (appearing in n statements)*
Industry related
Authorship/editorship royalties 4
Research funding from industry 4
Served on industry advisory board 4
Consultancy for industry 3
Ownership of relevant commercial products, patents, or 3 procedures
Speaking fees from industry 3
Employed by industry 2
Honoraria from industry 2
Industry equity holder 2
Travel or hospitality awards from industry 2
Other undefined payments from industry 1
Non-industry related
Research funding from government 4
Research funding from foundations, charities, and/or NGOs# 2
14 Consultancy for foundations, charities, and/or NGOs# 1
Honoraria from foundations, charities, and/or NGOs# 1
*Note: as each of the 12 relevant conflict of interest statements may contain more than one type of conflict of interest, the frequency column sums to greater than 12. #NGOs = “Non-governmental organisations”.
Replication and evidence synthesis Of the 188 articles involving primary or secondary data, 10 (5% [3% to 8%]) claimed to include a replication study. Of the 183 articles involving primary data, 1 article (1% [0% to 1%]) was cited by another article that claimed to be a replication. Of the 183 articles involving primary data, 21 were formally included in systematic reviews (11% [8% to 16%]). An additional 8 articles were cited incidentally (there was no intention to include in the review), but not formally included, and a further 1 article was explicitly excluded. 12 were formally included in meta-analyses (7% [4% to 10%]). An additional 2 articles were cited incidentally, but not formally included, and a further
1 article was excluded. Meta-analyses and systematic reviews were coded as exclusive variables
(such that if a systematic review included a meta-analysis, we coded it as a meta-analysis and not a systematic review), thus the 12 meta-analyses and 21 systematic reviews sum to a total of 33 evidence synthesis articles. Overall, the 228 English language and accessible articles tended to be infrequently cited (median = 3, min = 0, max = 74).
Discussion
Our evaluation of transparency and reproducibility-related research practices in a random sample of 250 psychology articles published between 2014-2017 highlights that whilst many articles were publicly available, crucial components of research – protocols, materials, raw data, and
15 analysis scripts – were rarely made publicly available alongside them. Pre-registration remained a nascent proposition with minimal adoption. Disclosure of funding sources and conflicts of interest was modest. Replication or evidence synthesis via meta-analysis or systematic review was relatively infrequent. Although there is evidence that some recent methodological reform initiatives have been effective (Hardwicke et al., 2019; Nelson et al., 2018; Vazire, 2018), the findings of the current study imply that their impact on the psychology literature during the examined period was fairly limited in scope.
For most of the articles (65%) we examined, we were able to access a publicly available version (“open access”). This is higher than recent open access estimates obtained for biomedicine (25%; Wallach et al., 2018) and the social sciences (40%; Hardwicke et al., 2019), as well as a large-scale automated analysis which suggested that 45% of the scientific literature published in 2015 was publicly available (Piwowar et al., 2018). Limiting access to academic publications reduces opportunities for researchers, policy-makers, practitioners, and the general public, to evaluate and make use of scientific evidence. One step psychologists can take to improve public availability of their articles is to upload them to the free preprint server PsyArXiv
(https://psyarxiv.com/). Uploading a preprint does not preclude publication at most journals
(Bourne et al., 2017), though specific policies regarding open access can be checked on the
Sherpa/Romeo database (http://sherpa.ac.uk/romeo/index.php).
Reported availability of research materials was modest in the articles we examined (14%), comparable with recent estimates in the social sciences (11%; Hardwicke et al., 2019), and lower than in biomedicine (33%; Wallach et al., 2018). Several reportedly available sets of materials were in fact not available due to broken links, an example of the ‘link rot’ phenomenon that has been observed by others trying to access research resources (Evangelou et al., 2005; Rowhani-
16 Farid & Barnett, 2018). Availability of original research materials (e.g., survey instruments, stimuli, software, videos) enables comprehensive evaluation of research (during traditional peer review and beyond; Vazire, 2017) and high-fidelity independent replication attempts (Open
Science Collaboration, 2015; Simons, 2014), both of which are important for the verification and systematic accumulation of scientific knowledge (Ioannidis, 2012). Furthermore, re-use of materials reduces waste and enhances efficiency (Chalmers & Glasziou, 2009). Psychologists can share their materials online in various third-party repositories that use stable permalinks, such as the Open Science Framework (OSF; see Klein et al., 2018).
Data availability statements in the articles we examined were extremely uncommon. This is consistent with accumulating evidence which suggests that the data underlying scientific claims is rarely immediately available (Alsheikh-Ali et al., 2011; Iqbal et al., 2016), though some modest improvement has been observed in recent years in biomedicine (Wallach et al., 2018).
Although we did not request data from authors directly, such requests to psychology researchers typically have modest yield (Vanpaemel et al., 2015; Wicherts et al., 2006). Most data appear to be effectively lost, including for some of the most influential psychology and psychiatry studies
(Hardwicke & Ioannidis, 2018b). Vanpaemel et al. (2015), for example, were unable to obtain
62% of the 394 datasets they requested from authors of papers published in four APA journals in
2012. Sharing of raw data, which is the evidence upon which scientists base their claims, enables verification through independent assessment of analytic or computational reproducibility
(Hardwicke et al., 2018; LeBel et al., 2018) and analytic robustness (Steegen et al., 2016). Data sharing also enhances evidence synthesis, such as through individual participant-level meta- analysis (Tierney et al., 2015), and can facilitate discovery, through merging of datasets and re- analysis with novel techniques (Voytek, 2016). Psychologists can improve data availability by
17 uploading raw data to third-party repositories such as the OSF (Klein et al., 2018). Data sharing must be managed with caution if there are ethical concerns, but such concerns do not always preclude all forms of sharing, nor necessarily negate ethical motivations for sharing (Meyer,
2017). Furthermore, when data cannot be made available it is always possible to explicitly declare this in research articles and explain the rationale for not sharing (Morey et al., 2016).
Of the articles we examined, only one shared an analysis script, a dearth consistent with assessments in biomedicine (Wallach et al., 2018), the social sciences (Hardwicke et al., 2019), and biostatistics (Rowhani-Farid & Barnett, 2018). Analysis scripts (a step-by-step description of the analysis in the form of computer code or instructions for re-creating the analysis in point-and- click software) provide the most veridical documentation of how the raw data were filtered, summarized, and analyzed. Verbal descriptions of analysis procedures are often ambiguous, contain errors, or do not adequately capture sufficient detail to enable analytic reproducibility
(Hardwicke et al., 2018; Stodden et al., 2018). Psychologists can share their analysis scripts on a third-party repository, such as the OSF (Klein et al., 2018), and educational resources are available to help researchers improve the quality of their analysis code (Wilson et al., 2017).
Sharing the computational environment in which analysis code successfully runs may also help to promote its longevity and trouble-free transfer to other researchers’ computers (Clyburne-Sherin et al., 2018).
Pre-registration, which involves making a time-stamped, read-only, record of a study’s rationale, hypotheses, methods, and analysis plan on an independent online repository, was rare in the articles we examined. Pre-registration fulfils a number of potential functions (Nosek et al.,
2019), including clarifying the distinction between exploratory and confirmatory aspects of research (Kimmelman et al., 2014; Wagenmakers et al., 2012) and enabling the detection and
18 mitigation of questionable research practices like selective outcome reporting (Franco et al.,
2016; John et al., 2012; Simmons et al., 2011). Pre-registration is relatively new to psychology
(Nosek et al., 2018; Nosek et al. 2019), but similar concepts of registration have a longer history in the context of clinical trials in biomedicine (Dickersin & Rennie, 2012) where they have become the expected norm (Zarin et al., 2017). However, clinical trials represent only a minority of biomedical research, and estimates suggest that pre-registration is overall rare in biomedicine
(Iqbal et al., 2016; Wallach et al., 2018). Pre-registration is also rare in the social sciences
(Hardwicke et al., 2019). There is no doubt that the number of pre-registrations (and the related
‘Registered Reports’ article format) is increasing in psychology (Hardwicke & Ioannidis, 2018a;
Nosek et al., 2018); however, the findings of the present study suggest that efforts to promote pre-registration may not yet have had widespread impact on routine practice. It is important to note that because there is a time-lag between registration and study publication, our measures may underestimate adoption. Whilst norms and standards for pre-registration in psychology are still evolving (Nosek et al., 2019), several dedicated registries, like the OSF, will host pre- registrations and detailed guidance is available (Klein et al., 2018).
The current findings suggest that psychology articles were more likely to include funding statements (62%) and conflict of interest statements (39%) than social science articles in general
(31% and 15% respectively; Hardwicke et al., 2019), but less likely than biomedical articles
(69% and 65% respectively; Wallach et al., 2018). It is possible that these disclosure statements are more common than most other practices we examined because they are often mandated by journals (Nutu et al., 2019). Disclosing funding sources and potential conflicts of interest in research articles helps readers to make an informed judgement about risk of bias (Bekelman et al., 2003; Cristea & Ioannidis, 2018). In the absence of established norms or journal mandates,
19 authors may often assume that such statements are not relevant to them (Chivers, 2019).
However, because the absence of a statement is ambiguous, researchers should ideally always include one, even if it is to explicitly declare that there were no funding sources and no potential conflicts.
Of the articles we examined, 5% claimed to be a replication study - slightly higher than a previous estimate in psychology of 1% (Makel et al., 2012) and a similar estimate of 1% in the social sciences (Hardwicke et al., 2019), but comparable with a 5% estimate in biomedicine
(Wallach et al. 2018). Only 1% of the articles we examined were cited by another article that claimed to be a replication attempt. 11% had been included in a systematic review and 7% had been included in a meta-analysis. Replication and evidence synthesis through systematic reviews and meta-analyses help to verify and build upon the existing evidence base. Although the current findings suggest that routine replication and evidence synthesis is relatively rare in psychology, many high-profile replication attempts have been conducted in recent years (Open Science
Collaboration, 2015; Pashler & Wagenmakers, 2012). Additionally, because the articles we examined were published relatively recently, there may be some time lag before relevant replication and evidence synthesis studies emerge. For example, in biomedicine at least, there is a geometric growth in the number of meta-analyses and in many fields multiple meta-analyses are often conducted once several studies appear on the same research question (Ioannidis, 2016).
The current study has several caveats and limitations. Firstly, our findings are based on a random sample of 250 articles, and the obtained estimates may not necessarily generalize to specific contexts, such as other disciplines, sub-fields of psychology, or articles published in particular journals. However, this target sample size was selected to balance informativeness with tractability, and the observed estimates have reasonable precision. Secondly, although the focus
20 of this study was transparency and reproducibility-related practices, this does not imply that adoption of these practices is sufficient to promote the goals they are intended to achieve. For example, poorly documented data may not enable analytic reproducibility (Hardwicke et al.,
2018) and inadequately specified pre-registrations may not sufficiently constrain researcher degrees of freedom (Claesen et al., 2019; Veldkamp et al., 2018). Thirdly, we relied only on published information. Direct requests to authors may have yielded additional information, however, such requests to psychology researchers are often unsuccessful (Hardwicke &
Ioannidis, 2018a; Vanpaemel et al., 2015; Wicherts et al., 2006). Finally, a lack of transparency may have been justified in some cases if there were overriding practical, legal, or ethical concerns
(Meyer, 2017). However, no constraints of this kind were declared in any of the articles we examined.
The present findings imply minimal adoption of transparency and reproducibility-related practices in psychology during the examined time period. Although researchers appear to recognize the problems of low credibility and reproducibility (Baker, 2016) and endorse the values of transparency and reproducibility in principle (Anderson et al., 2010), they are often wary of change (Fuchs et al., 2012; Houtkoop et al., 2018) and routinely neglect these principles in practice (Hardwicke et al., 2019; Iqbal et al., 2016; Wallach et al., 2018). There is unlikely to be a single remedy to this situation. A multifaceted approach will likely be required, with iterative evaluation and careful scrutiny of reform initiatives (Hardwicke et al., 2020). At the educational level, guidance and resources are available to aid researchers (Crüwell et al., 2018;
Klein et al., 2018). At the institutional level, there is evidence that funder and journal policies can be effective at fomenting change (Hardwicke et al., 2018; Nuijten et al., 2018) and these initiatives should be translated and disseminated where relevant. Currently, heterogeneous
21 journal policies (Nutu et al., 2019) may be disrupting efforts to establish norms and promote better standards in routine practice. The Transparency and Openness Promotion (TOP) initiative promises to encourage adoption and standardization of journal policies related to transparency and reproducibility (Nosek et al., 2015), but it remains to be seen how effective this initiative will be in practice. Aligning academic rewards and incentives (e.g., funding awards, publication acceptance, promotion and tenure) with better research practices may also be instrumental in encouraging wider adoption of these practices (Moher et al., 2018).
The present study is one of several efforts to examine the prevalence of transparency and reproducibility-related research practices across scientific disciplines (Hardwicke et al., 2019;
Iqbal et al., 2016; Wallach et al., 2018). Here, we have sketched out some of the topography of psychological territory. Additional studies will be required to fill in as-yet-unexplored areas of the map, and increase the resolution in specific areas (for example, sub-fields of psychology).
Future studies can also add a temporal dimension by comparing to the baseline established here, allowing us to explore the evolution of this landscape over time.
Funding statement
The authors received no specific funding for this work. The Meta-Research Innovation Center at
Stanford (METRICS) is supported by a grant from the Laura and John Arnold Foundation. The
Meta-Research Innovation Center Berlin (METRIC-B) is supported by a grant from the Einstein
Foundation and Stiftung Charité. Robert T. Thibault is supported by a postdoctoral fellowship from the Fonds de la recherche en santé du Québec.
22 Competing interests statement
In the past 36 months, JDW received research support through the Meta Research Innovation Center at
Stanford (METRICS) and the Collaboration for Research Integrity and Transparency (CRIT) at Yale, from the Laura and John Arnold Foundation, and through the Yale-Male Clinic Center for Excellence in
Regulatory Science and Innovation (CERSI) (U01FD005938). All other authors declare no competing interests.
Open practices statement
The study protocol was pre-registered on September 26, 2018, and is available at https://osf.io/q96eh/. All deviations from this protocol are explicitly acknowledged in Supplementary Information A. We report how we determined our sample size, all data exclusions, all manipulations, and all measures in the study. All data (https://osf.io/5qmz7/), materials (https://osf.io/c89sy/), and analysis scripts (https://osf.io/gfjtq/) related to this study are publicly available. To facilitate reproducibility this manuscript was written by interleaving regular prose with analysis code and is available in a Code Ocean container (https://doi.org/10.24433/CO.1618143.v2) which re-creates the software environment in which the original analyses were performed.
Acknowledgements
We thank Kevin Boyack for assistance obtaining the sample of articles.
Author contributions
T.E.H., J.D.W., and J.P.A.I. designed the study. T.E.H., J.D.W., M.C.K., R.T.T., and J.E.K. conducted the data collection. T.E.H. performed the data analysis. T.E.H., J.D.W., R.T.T., and
J.P.A.I. wrote the manuscript. All the authors gave their final approval for publication.
23 References
Alsheikh-Ali, A. A., Qureshi, W., Al-Mallah, M. H., & Ioannidis, J. P. A. (2011). Public
availability of published research data in high-impact journals. PLoS ONE, 6(9), e24357 4.
https://doi.org/10.1371/journal.pone.0024357
Anderson, M. S., Ronning, E. A., Devries, R., & Martinson, B. C. (2010). Extending the
Mertonian Norms: Scientists’ Subscription to Norms of Research. The Journal of Higher
Education, 81(3), 366 393. https://doi.org/10.1353/jhe.0.0095
Baker, M. (2016). Is there a reproducibility crisis? A Nature survey lifts the lid on how
researchers view the 'crisis' rocking science and what they think will help. Nature, 533(7552),
452 454. https://doi.org/10.1038/521274a
Bekelman, J. E., Li, Y., & Gross, C. P. (2003). Scope and Impact of Financial Conflicts of
Interest in Biomedical Research: A Systematic Review. JAMA, 289(4), 454.
https://doi.org/10.1001/jama.289.4.454
Bourne, P. E., Polka, J. K., Vale, R. D., & Kiley, R. (2017). Ten simple rules to consider
regarding preprint submission. PLoS Computational Biology, 13(5), e1005473 6.
https://doi.org/10.1371/journal.pcbi.1005473
Buckheit, J. B., & Donoho, D. L. (1995). WaveLab and Reproducible Research. In P. Bickel, P.
Diggle, S. Fienberg, K. Krickeberg, I. Olkin, N. Wermuth, S. Zeger, A. Antoniadis, & G.
Oppenheim (Eds.), Wavelets and Statistics (Vol. 103, pp. 55–81). Springer New York.
https://doi.org/10.1007/978-1-4612-2544-7_5
24 Chalmers, I., & Glasziou, P. (2009). Viewpoint Avoidable waste in the production and reporting
of research evidence. The Lancet, 374(9683), 86 89. https://doi.org/10.1016/s0140-
6736(09)60329-9
Claesen, A., Gomes, S. L. B. T., Tuerlinckx, F., & vanpaemel. (2019). Preregistration:
Comparing Dream to Reality [Preprint]. PsyArXiv. https://doi.org/10.31234/osf.io/d8wex
Chivers, T. (2019). Does psychology have a conflict-of-interest problem? Nature, 571 (7763),
20–23. https://doi.org/10.1038/d41586-019-02041-5
Clyburne-Sherin, A., Fei, X., & Green, S. A. (2018). Computational Reproducibility via
Containers in Psychology [Preprint]. PsyArXiv. https://doi.org/10.31234/osf.io/mf82t
Cristea, I.-A., & Ioannidis, J. P. A. (2018). Improving Disclosure of Financial Conflicts of
Interest for Research on Psychosocial Interventions. JAMA Psychiatry.
https://doi.org/10.1001/jamapsychiatry.2018.0382
Crüwell, S., van Doorn, J., Etz, A., Makel, M. C., Moshontz, H., Niebaum, J., Orben, A.,
Parsons, S., & Schulte-Mecklenbeck, M. (2018). 7 Easy Steps to Open Science: An Annotated
Reading List [Preprint]. PsyArXiv. https://doi.org/10.31234/osf.io/cfzyx
David, P. A. (2008). The Historical Origins of ’Open Science’: An Essay on Patronage,
Reputation and Common Agency Contracting in the Scientific Revolution. Capitalism and
Society, 3(2). https://doi.org/10.2202/1932-0213.1040
Dickersin, K., & Rennie, D. (2012). The Evolution of Trial Registries and Their Use to Assess
the Clinical Trial Enterprise. JAMA, 307(17), 1861–1864.
https://doi.org/10.1001/jama.2012.4230
25 Evangelou, E., Trikalinos, T. A., & Ioannidis, J. P. A. (2005). Unavailability of online
supplementary scientific information from articles published in major journals. The FASEB
Journal, 19(14), 1943–1944. https://doi.org/10.1096/fj.05-4784lsf
Franco, A., Malhotra, N., & Simonovits, G. (2016). Underreporting in Psychology Experiments.
Social Psychological and Personality Science, 7(1), 8–12.
https://doi.org/10.1177/1948550615598377
Fuchs, H. M., Jenny, M., & Fiedler, S. (2012). Psychologists Are Open to Change, yet Wary of
Rules. Perspectives on Psychological Science, 7(6), 639–642.
https://doi.org/10.1177/1745691612459521
Hardwicke, T. E., & Ioannidis, J. P. A. (2018a). Mapping the universe of registered reports.
Nature Human Behaviour, 2(11), 793–796. https://doi.org/10.1038/s41562-018-0444-y
Hardwicke, T. E., & Ioannidis, J. P. A. (2018b). Populating the Data Ark: An attempt to retrieve,
preserve, and liberate data from the most highly-cited psychology and psychiatry articles.
PLOS ONE, 13(8), e0201856. https://doi.org/10.1371/journal.pone.0201856
Hardwicke, T. E., Mathur, M. B., MacDonald, K., Nilsonne, G., Banks, G. C., Kidwell, M. C.,
Mohr, A. H., Clayton, E., Yoon, E. J., Tessler, M. H., Lenne, R. L., Altman, S., Long, B., &
Frank, M. C. (2018). Data availability, reusability, and analytic reproducibility: Evaluating
the impact of a mandatory open data policy at the journal Cognition. Royal Society Open
Science, 5(8), 180448. https://doi.org/10.1098/rsos.180448
Hardwicke, T. E., Serghiou, S., Janiaud, P., Danchev, V., Crüwell, S., Goodman, S., & Ioannidis,
J. P. A. (2020). Calibrating the scientific ecosystem through meta-research. Annual Review of
26 Statistics and Its Application, 7, 11-37. https://doi.org/10.1146/annurev-statistics-031219-
041104
Hardwicke, T. E., Wallach, J. D., Kidwell, M., Bendixen, T., Crüwell, S., & Ioannidis, J. P. A.
(2019). An empirical assessment of transparency and reproducibility-related research
practices in the social sciences (2014-2017). Royal Society Open Science, 7: 190806.
https://doi.org/10.1098/rsos.190806
Houtkoop, B. L., Chambers, C., Macleod, M., Bishop, D. V. M., Nichols, T. E., & Wagenmakers,
E.-J. (2018). Data Sharing in Psychology: A Survey on Barriers and Preconditions. Advances
in Methods and Practices in Psychological Science, 1(1), 70–85.
https://doi.org/10.1177/2515245917751886
Ioannidis, J. P. (2016). The mass production of redundant, misleading, and conflicted systematic
reviews and meta-analyses: mass production of systematic reviews and meta-analyses. The
Milbank Quarterly, 94(3), 485–514. https://doi.org/10.1111/1468-0009.12210
Ioannidis, J. P. A. (2005). Why most published research findings are false. PLoS Medicine, 2(8),
e124. https://doi.org/10.1371/journal.pmed.0020124
Ioannidis, J. P. A. (2012). Why Science Is Not Necessarily Self-Correcting. Perspectives on
Psychological Science, 7(6), 645–654. https://doi.org/10.1177/1745691612464056
Ioannidis, J. P. A., Greenland, S., Hlatky, M. A., Khoury, M. J., Macleod, M. R., Moher, D.,
Schulz, K. F., & Tibshirani, R. (2014). Increasing value and reducing waste in research
design, conduct, and analysis. The Lancet, 383(9912), 166 175.
https://doi.org/10.1016/s0140-6736(13)62227-8
27 Ioannidis, J. P. A., Munafò, M. R., Fusar-Poli, P., Nosek, B. A., & David, S. P. (2014).
Publication and other reporting biases in cognitive sciences: Detection, prevalence, and
prevention. Trends in Cognitive Sciences, 18(5), 235 241.
https://doi.org/10.1016/j.tics.2014.02.010
Iqbal, S. A., Wallach, J. D., Khoury, M. J., Schully, S. D., & Ioannidis, J. P. A. (2016).
Reproducible research practices and transparency across the biomedical literature. PLoS
Biology, 14(1), e1002333 13. https://doi.org/10.1371/journal.pbio.1002333
John, L. K., Loewenstein, G., & Prelec, D. (2012). Measuring the prevalence of questionable
research practices with incentives for truth telling. Psychological Science, 23(5), 524 532.
https://doi.org/10.1177/0956797611430953
Kidwell, M. C., Lazarevic, L. B., Baranski, E., Hardwicke, T. E., Piechowski, S., Falkenberg, L.-
S., Kennett, C., Slowik, A., Sonnleitner, C., Hess-Holden, C., Errington, T. M., Fiedler, S., &
Nosek, B. A. (2016). Badges to Acknowledge Open Practices: A Simple, Low-Cost, Effective
Method for Increasing Transparency. PLOS Biology, 14(5), e1002456.
https://doi.org/10.1371/journal.pbio.1002456
Kimmelman, J., Mogil, J. S., & Dirnagl, U. (2014). Distinguishing between Exploratory and
Confirmatory Preclinical Research Will Improve Translation. PLoS Biology, 12(5), e1001863
4. https://doi.org/10.1371/journal.pbio.1001863
Klein, O., Hardwicke, T. E., Aust, F., Breuer, J., Danielsson, H., Mohr, A. H., Ijzerman, H.,
Nilsonne, G., Vanpaemel, W., & Frank, M. C. (2018). A Practical Guide for Transparency in
Psychological Science. Collabra: Psychology, 4(1), 20. https://doi.org/10.1525/collabra.158
28 LeBel, E. P., McCarthy, R. J., Earp, B. D., Elson, M., & Vanpaemel, W. (2018). A Unified
Framework to Quantify the Credibility of Scientific Findings. Advances in Methods and
Practices in Psychological Science, 1(3), 389–402.
https://doi.org/10.1177/2515245918787489
Makel, M. C., Plucker, J. A., & Hegarty, B. (2012). Replications in Psychology Research.
Perspectives on Psychological Science, 7(6), 537–542.
https://doi.org/10.1177/1745691612460688
Meyer, M. N. (2017). Practical Tips for Ethical Data Sharing. Advances in Methods and
Practices in Psychological Science, 1(1), 131–144.
https://doi.org/10.1177/2515245917747656
Miguel, E., Camerer, C., Casey, K., Cohen, J., Esterling, K. M., Gerber, A., Glennerster, R.,
Green, D. P., Humphreys, M., Imbens, G., Laitin, D., Madon, T., Nelson, L., Nosek, B. A.,
Petersen, M., Sedlmayr, R., Simmons, J. P., Simonsohn, U., & der Laan, M. V. (2014).
Promoting Transparency in Social Science Research. Science, 343(6166), 30–31.
https://doi.org/10.1126/science.1245317
Morey, R. D., Chambers, C. D., Etchells, P. J., Harris, C. R., Hoekstra, R., Lakens, D.,
Lewandowsky, S., Morey, C. C., Newman, D. P., Schönbrodt, F. D., Vanpaemel, W.,
Wagenmakers, E.-J., & Zwaan, R. A. (2016). The Peer Reviewers’ Openness Initiative:
Incentivizing open research practices through peer review. Royal Society Open Science, 3(1),
150547. https://doi.org/10.1098/rsos.150547
29 Munafò, M. R., Nosek, B. A., Bishop, D. V. M., Button, K. S., Chambers, C. D., du Sert, N. P.,
Simonsohn, U., Wagenmakers, E.-J., Ware, J. J., & Ioannidis, J. P. A. (2017). A manifesto for
reproducible science. Nature Human Behaviour, 1(1), 1 9. https://doi.org/10.1038/s41562-
016-0021
Naudet, F., Sakarovitch, C., Janiaud, P., Cristea, I., Fanelli, D., Moher, D., & Ioannidis, J. P. A.
(2018). Data sharing and reanalysis of randomized controlled trials in leading biomedical
journals with a full data sharing policy: Survey of studies published in The BMJ and PLOS
Medicine. BMJ, 360, k400. https://doi.org/10.1136/bmj.k400
Nelson, L. D., Simmons, J. P., & Simonsohn, U. (2018). Psychology’s Renaissance. Annual
Review of Psychology, 69(1), 511 534. https://doi.org/10.1146/annurev-psych-122216-
011836
Nosek, B. A., Alter, G., Banks, G. C., Borsboom, D., Bowman, S. D., Breckler, S. J., Buck, S.,
Chambers, C. D., Chin, G., Christensen, G., Contestabile, M., Dafoe, A., Eich, E., Freese, J.,
Glennerster, R., Goroff, D., Green, D. P., Hesse, B., Humphreys, M., … Yarkoni, T. (2015).
Promoting an open research culture. Science, 348(6242), 1422 1425.
https://doi.org/10.1126/science.aab2374
Nosek, B. A., Beck, E. D., Campbell, L., Flake, J. K., Hardwicke, T. E., Mellor, D. T., van ’t
Veer, A. E., & Vazire, S. (2019). Preregistration Is Hard, And Worthwhile. Trends in
Cognitive Sciences. https://doi.org/10.1016/j.tics.2019.07.009
30 Nosek, B. A., Ebersole, C. R., DeHaven, A. C., & Mellor, D. T. (2018). The preregistration
revolution. Proceedings of the National Academy of Sciences of the United States of America,
115(11), 2600 2606. https://doi.org/10.1073/pnas.1708274114
Nuijten, M. B., Borghuis, J., Veldkamp, C. L. S., Dominguez-Alvarez, L., Assen, M. A. L. M. V.,
& Wicherts, J. M. (2018). Journal Data Sharing Policies and Statistical Reporting
Inconsistencies in Psychology. Collabra: Psychology, 3(1), 31.
https://doi.org/10.1525/collabra.102
Nutu, D., Gentili, C., Naudet, F., & Cristea, I. A. (2019). Open science practices in clinical
psychology journals: An audit study. Journal of Abnormal Psychology, 128(6), 510–516.
https://doi.org/10.1037/abn0000414
Open Science Collaboration. (2015). Estimating the reproducibility of psychological science.
Science, 349(6251), aac4716. https://doi.org/10.1126/science.aac4716
Pashler, H., & Wagenmakers, E.-J. (2012). Editors’ Introduction to the Special Section on
Replicability in Psychological Science. Perspectives on Psychological Science, 7(6), 528–
530. https://doi.org/10.1177/1745691612465253
Piwowar, H., Priem, J., Lariviere, V., Alperin, J. P., Matthias, L., Norlander, B., Farley, A., West,
J., & Haustein, S. (2018). The state of OA: A large-scale analysis of the prevalence and
impact of Open Access articles. PeerJ, 6, e4375. https://doi.org/10.7717/peerj.4375
Rowhani-Farid, A., & Barnett, A. G. (2018). Badges for sharing data and code at Biostatistics:
An observational study. F1000Research, 7, 90.
https://doi.org/10.12688/f1000research.13477.2
31 Rowhani-Farid, A., & Barnett, A. G. (2016). Has open data arrived at the British Medical Journal
(BMJ)? An observational study. BMJ Open, 6(10), e011784 9.
https://doi.org/10.1136/bmjopen-2016-011784
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology: Undisclosed
flexibility in data collection and analysis allows presenting anything as significant.
Psychological Science, 22(11), 1359 1366. https://doi.org/10.1177/0956797611417632
Simons, D. J. (2014). The Value of Direct Replication. Perspectives on Psychological Science,
9(1), 76–80. https://doi.org/10.1177/1745691613514755
Sison, C. P., & Glaz, J. (1995). Simultaneous confidence intervals and sample size determination
for multinomial proportions. Journal of the American Statistical Association, 90(429), 366
369. https://doi.org/10.1080/01621459.1995.10476521
Steegen, S., Tuerlinckx, F., Gelman, A., & Vanpaemel, W. (2016). Increasing Transparency
Through a Multiverse Analysis. Perspectives on Psychological Science, 11(5), 702 712.
https://doi.org/10.1177/1745691616658637
Stodden, V., Seiler, J., & Ma, Z. (2018). An empirical analysis of journal policy effectiveness for
computational reproducibility. Proceedings of the National Academy of Sciences, 115(11),
201708290. https://doi.org/10.1073/pnas.1708290115
Szucs, D., & Ioannidis, J. P. A. (2017). Empirical assessment of published effect sizes and power
in the recent cognitive neuroscience and psychology literature. PLOS Biology, 15(3),
e2000797. https://doi.org/10.1371/journal.pbio.2000797
32 Tierney, J. F., Vale, C., Riley, R., Smith, C. T., Stewart, L., Clarke, M., & Rovers, M. (2015).
Individual Participant Data (IPD) Meta-analyses of Randomised Controlled Trials: Guidance
on Their Use. PLoS Medicine, 12(7), e1001855 16.
https://doi.org/10.1371/journal.pmed.1001855
Vanpaemel, W., Vermorgen, M., Deriemaecker, L., & Storms, G. (2015). Are we wasting a good
crisis? The availability of psychological research data after the storm. Collabra, 1(1), 1 5.
https://doi.org/10.1525/collabra.13
Vazire, S. (2017). Quality Uncertainty Erodes Trust in Science. Collabra: Psychology, 3(1), 1.
https://doi.org/10.1525/collabra.74
Vazire, S. (2018). Implications of the Credibility Revolution for Productivity, Creativity, and
Progress. Perspectives on Psychological Science, 13(4), 411–417.
https://doi.org/10.1177/1745691617751884
Veldkamp, C. L. S., Bakker, M., van Assen, M. A. L. M., Crompvoets, E. A. V., Ong, H. H.,
Nosek, B. A., Soderberg, C. K., Mellor, D. T., & Wicherts, J. M. (2018). Ensuring the quality
and specificity of preregistrations [Preprint]. PsyArXiv. https://doi.org/10.31234/osf.io/cdgyh
Voytek, B. (2016). The Virtuous Cycle of a Data Ecosystem. PLoS Computational Biology,
12(8), e1005037 6. https://doi.org/10.1371/journal.pcbi.1005037
Wagenmakers, E.-J., Wetzels, R., Borsboom, D., van der Maas, H. L. J., & Kievit, R. A. (2012).
An Agenda for Purely Confirmatory Research. Perspectives on Psychological Science, 7(6),
632–638. https://doi.org/10.1177/1745691612463078
33 Wallach, J. D., Boyack, K. W., & Ioannidis, J. P. A. (2018). Reproducible research practices,
transparency, and open access data in the biomedical literature, 20152017. PLOS Biology,
16(11), e2006930. https://doi.org/10.1371/journal.pbio.2006930
Wallach, J. D., Gonsalves, G. S., & Ross, J. S. (2018). Research, regulatory, and clinical
decision-making: The importance of scientific integrity. Journal of Clinical Epidemiology,
93, 88–93. https://doi.org/10.1016/j.jclinepi.2017.08.021
Wicherts, J. M., Borsboom, D., Kats, J., & Molenaar, D. (2006). The poor availability of
psychological research data for reanalysis. American Psychologist, 61(7), 726 728.
https://doi.org/10.1037/0003-066x.61.7.726
Wilson, G., Bryan, J., Cranston, K., Kitzes, J., Nederbragt, L., & Teal, T. K. (2017). Good
enough practices in scientific computing. PLOS Computational Biology, 13(6), e1005510.
https://doi.org/10.1371/journal.pcbi.1005510
Zarin, D. A., Tse, T., Williams, R. J., & Rajakannan, T. (2017). Update on trial registration 11
Years after the ICMJE policy was established. The New England Journal of Medicine, 4, 383
391. https://doi.org/10.1056/nejmsr1601330
SUPPLEMENTARY INFORMATION
Supplementary Information A. Pre-registered protocol updates
During the course of the study, we made several minor adjustments to the pre-registered protocol. These are listed below.
34 (1) For the questions “How does the statement indicate the materials are available?” and “How does the statement indicate the data are available?” we added an explicit response option: “Appendix within the present article”. (2) For the questions “Does the article state whether or not the materials are available?” and “Does the article state whether or not the data are available?” we added an explicit response option “Source of data/materials provided but no explicit availability statement”. However, because this was introduced late in the data collection process it was not applied consistently and we have elected to re-code this response option as “No - there is no data availability statement”. (3) For the questions “What type of study is being reported?” we changed the response option ‘survey’ to ‘survey/interview’ and renamed ‘field study’ to ‘observational study’. We did this to better match our coding decisions. (4) For the citation evaluation, we added two questions: “How many times was the article cited incidentally in a meta-analysis (i.e., no intention to include in data synthesis)?” and “How many times was the article cited incidentally in a systematic review (i.e., no intention to include in formal review)?” (5) For the questions “Does the article include a statement indicating whether there were funding sources?” we added the response option “Yes - the statement identifies funding sources but the private/public status is unclear” (6) To concisely present the content of conflict of interest statements we categorized them into the types shown in Table 3. This analysis was not pre-registered. (7) To align the extraction form with previous studies (e.g., Wallach et al., 2019) we included an additional response option (“Empirical data - cost effectiveness and/or decision analysis”) for the question “What type of study is being reported?”
Supplementary Information B. Inter-rater reliability
We computed inter-rater reliability using Fleiss’ Kappa (see Supplementary Table 1) for article access and availability statements for key research resources (as shown in Figure 1). All coding differences were resolved through discussion.
Supplementary Table 1. Inter-rater reliability (Fleiss’ Kappa) for article access and availability statements for key research resources.
Article Materials Protocol Data Analysis Pre-registration Conflicts of Funding scripts interest
0.86 0.78 0.71 0.61 0.75 0.66 0.96 0.92
35
36