TICS-1311; No. of Pages 7 Review Publication and other reporting biases in cognitive sciences: detection, prevalence, and prevention 1,2,3,4 5,6 7 8 John P.A. Ioannidis , Marcus R. Munafo` , Paolo Fusar-Poli , Brian A. Nosek , 1 and Sean P. David 1 Department of Medicine, Stanford University School of Medicine, Stanford, CA 94305, USA 2 Department of Health Research and Policy, Stanford University School of Medicine, Stanford, CA 94305, USA 3 Department of Statistics, Stanford University School of Humanities and Sciences, Stanford, CA 94305, USA 4 Meta-Research Innovation Center at Stanford (METRICS), Stanford, CA 94305, USA 5 MRC Integrative Epidemiology Unit, UK Centre for Tobacco and Alcohol Studies, University of Bristol, Bristol, UK 6 School of Experimental Psychology, University of Bristol, Bristol, UK 7 Department of Psychosis Studies, Institute of Psychiatry, King’s College London, London, UK 8 Center for Open Science, and Department of Psychology, University of Virginia, VA, USA Recent systematic reviews and empirical evaluations of Definitions of biases and relevance for cognitive the cognitive sciences literature suggest that publica- sciences tion and other reporting biases are prevalent across The terms ‘publication bias’ and ‘selective reporting bias’ diverse domains of cognitive science. In this review, refer to the differential choice to publish studies or report we summarize the various forms of publication and particular results, respectively, depending on the nature reporting biases and other questionable research prac- or directionality of findings [1]. There are several forms of tices, and overview the available methods for probing such biases in the literature [2], including: (i) study publi- into their existence. We discuss the available empirical cation bias, where studies are less likely to be published evidence for the presence of such biases across the when they reach nonstatistically significant findings; (ii) neuroimaging, animal, other preclinical, psychological, selective outcome reporting bias, where multiple out- clinical trials, and genetics literature in the cognitive comes are evaluated in a study and outcomes found to sciences. We also highlight emerging solutions (from be nonsignificant are less likely to be published than are study design to data analyses and reporting) to prevent statistically significant ones; and (iii) selective analysis bias and improve the fidelity in the field of cognitive reporting bias, where certain data are analyzed using science research. different analytical options and publication favors the more impressive, statistically significant variants of the Introduction results (Box 1). The promise of science is that the generation and evalua- Cognitive science uses a range of methods producing a tion of evidence is done transparently to promote an effi- multitude of measures and rich data sets. Given this cient, self-correcting process. However, multiple biases quantity of data, the potential analyses available and may produce inefficiency in knowledge building. In this pressures to publish, the temptation to use arbitrary (in- review, we discuss the importance of publication and other stead of planned) statistical analyses, and to mine data are reporting biases, and present some potential correctives substantial and may lead to questionable research prac- that may reduce bias without disrupting innovation. Con- tices. Even extreme cases of fraud in the cognitive sciences, sideration of publication and other reporting biases is such as reporting of experiments that never took place, particularly timely for cognitive sciences because it is a falsifying results, and confabulating data, have been field that is expanding rapidly. As such, preventing or recently reported [3–5]. Overt fraud is probably rare, but remedying these biases will have substantial impact on evidence of reporting biases in multiple domains of the the efficient development of a credible corpus of published cognitive sciences (discussed below) raise concern about research. the prevalence of questionable practices that occur via motivated reasoning without any intent to be fraudulent. Corresponding author: Ioannidis, J.P.A. ([email protected]). Keywords: publication bias; reporting bias; cognitive sciences; neuroscience; bias. Tests for publication and other reporting biases 1364-6613/$ – see front matter Tests for single studies, specific topics, and wider ß 2014 Published by Elsevier Ltd. http://dx.doi.org/10.1016/j.tics.2014.02.010 disciplines Explicit documentation of publication and other reporting biases requires availability of protocols, data, and results of primary analyses from conducted studies so that these Trends in Cognitive Sciences xx (2014) 1–7 1 TICS-1311; No. of Pages 7 Review Trends in Cognitive Sciences xxx xxxx, Vol. xxx, No. x Box 1. Various publication and reporting biases compared with large studies, this may reflect publication or selective reporting biases, but alternative explanations Study publication bias can distort meta-analyses and systematic exist (as reviewed elsewhere [10]). Sensitivity and specific- reviews of a large number of published studies when authors are more likely to submit and/or editors more likely to publish studies ity of these tests in real life is unknown, but simulation with ‘positive’ (i.e., statistically significant) results or suppress studies have evaluated the performance in different set- ‘negative’ (i.e., nonsignificant) results. tings. Published recommendations suggest cautious use of Study publication bias is also called the ‘file drawer problem’ [88], such tests [10]. In brief, visual evaluations of inverted whereby most studies, with null effects, are never submitted and funnel plots without statistical testing are precarious thereby left in the file drawer. However, study publication bias is probably also accentuated by peer reviewers and editors, even for [11]; some test variants have better type I and type II ‘negative’ studies that are submitted by their authors for publication. error properties than others [12,13]; and, for most meta- Sometimes, studies with ‘negative’ results may be published, but analyses where there are a limited number of studies, the with greater delay, than studies with ‘positive’ results, causing the power of these tests is low [10]. so-called ‘time-lag bias’ [89]. In the ‘Proteus phenomenon’, ‘positive’ results are published quickly and then there is a window of opportunity for the Selection models publication also of ‘negative’ results contradicting the original Selection model approaches evaluate whether the pattern observation that may be tantalizing and interesting to editors and of results that have been accumulated from a number of investigators. The net effect is rapidly alternating extreme results [17,90]. studies suggests an underlying filtering process, such as Selective outcome or analyses reporting biases can occur in many the nonpublication of a given percentage of results that had forms, including selective reporting of some of the analyzed not reached formal statistical significance [14–16]. These outcomes in a study when others exist; selective reporting of a methods have been less widely applied than small-study specific outcome; and incomplete reporting of an outcome [91]. effect tests, even though they may be more promising. One Examples of selective analyses reporting include selective report- application of a selection model approach examined an ing of subgroup analyses [92], reporting of per protocol instead of intention-to-treat analyses [93] and relegation of primary out- entire discipline (Alzheimer’s disease genetics) and found comes to secondary outcomes [94] so that nonsignificant results that selection forces may be different for first and/or dis- for the pre-specified primary outcomes are subordinated to covery results versus subsequent replication and/or refu- nonprimary positive results [8]. tation results or late replication efforts [17]. Another As a result of all these selective reporting biases, the published method that probes for data ‘fiddling’ has been proposed literature is likely to have an excess of statistically significant results. Even when associations and effects are genuinely non- [18] to identify selective reporting of extremely promising null, the published results are likely to overestimate their P values in a body of published results. magnitude [95,96]. Excess significance can be compared against the published literature. Howev- Excess significance testing evaluates whether the number er, these are not often available. A few empirical studies of statistically significant results in a corpus of studies is have retrieved study protocols from authors or trial data too high, under some plausible assumptions about the from submissions to the US Food and Drug Administration magnitude of the true effect size [19]. The Ioannidis test (FDA) [6–9]. These studies have shown that deviations in [19] can be applied to meta-analyses of multiple studies the analysis plan between protocols and published papers and also to larger fields and disciplines where many meta- are common [6], and effect sizes of drug interventions are analyses are compiled. The number of expected studies larger in the published literature compared with the cor- with nominally statistically significant results is estimated responding data from the same trials submitted to the FDA by summing the calculated power of all the considered [7,8]. It is challenging, but more common, to detect these studies. Simulations