The Statistical Crisis in Science: How Is It Relevant to Clinical Neuropsychology?
Total Page:16
File Type:pdf, Size:1020Kb
THE CLINICAL NEUROPSYCHOLOGIST, 2017 VOL. 31, NOS. 67, 10001014 https://doi.org/10.1080/13854046.2016.1277557 The statistical crisis in science: how is it relevant to clinical neuropsychology? Andrew Gelmana,b and Hilde M. Geurtsc,d aDepartment of Statistics, Columbia University, New York, NY, USA; bDepartment of Political Science, Columbia University, New York, NY, USA; cDutch ADHD and Autism Research Center, Department of Psychology, Brain and Cognition Section, University of Amsterdam, Amsterdam, The Netherlands; dDr. Leo Kannerhuis, Department of Research, Development, and Innovation, Doorwerth, The Netherlands ABSTRACT ARTICLE HISTORY There is currently increased attention to the statistical (and replication) Received 9 June 2016 crisis in science. Biomedicine and social psychology have been at the Accepted 22 December 2016 heart of this crisis, but similar problems are evident in a wide range of KEYWORDS felds. We discuss three examples of replication challenges from the Replication crisis; statistics; feld of social psychology and some proposed solutions, and then sociology of science consider the applicability of these ideas to clinical neuropsychology. In addition to procedural developments such as preregistration and open data and criticism, we recommend that data be collected and analyzed with more recognition that each new study is a part of a learning process. The goal of improving neuropsychological assessment, care, and cure is too important to not take good scientifc practice seriously. 1. Introduction In the last few years psychology and other experimental sciences have been rocked with what has been called a replication crisis. Recent theoretical and experimental results have cast doubt on what previously had been believed to be solid fndings in various felds (e.g. see Baker, 2016; Begley & Ellis, 2012; Button et al., 2013; Dumas-Mallet, Button, Boraud, Munafo, & Gonon, 2016; Ioannidis & Panagiotou, 2011). Here we focus on psychology, a feld in which big steps have been made to try to address the crisis. Downloaded by [Columbia University Libraries] at 15:00 13 September 2017 Where do we refer to when we speak of the statistical crisis in science? From the theoretical direction a close examination of the properties of hypothesis testing has made it clear that conventional ‘statistical signifcance’ does not necessarily provide a strong signal in favor of scientifc claims (Button et al., 2013; Cohen, 1994; Francis, 2013; Gelman & Carlin, 2014; Ioannidis, 2005; Simmons, Nelson, & Simonsohn, 2011). From the empirical direction, several high-profle studies have been subject to preregistered replications that have failed (see e.g. Doyen, Klein, Pichon, & Cleeremans, 2012; Open Science Collaboration, 2015). This all creates challenges for drawing conclusions regarding psychological theories, and in mapping research fndings to clinical practice. CONTACT Andrew Gelman [email protected] © 2017 Informa UK Limited, trading as Taylor & Francis Group THE CLINICAL NEUROPSYCHOLOGIST 1001 In the present paper we shall frst review the crisis within social psychology and then pose the question of its relevance to clinical neuropsychology and clinical practice. We do not attempt to estimate the extent of replication problems within the feld of clinical neuropsy- chology; rather, we argue that from the general crisis in the social and biological sciences, lessons can be learned that are of importance to all research felds that are subject to high levels of variation. Proposed solutions are relevant for clinical neuropsychological research, but for some specifc challenges there are not yet any clear remedies. In line with The Clinical Neuropsychologist’s record of scrutiny (e.g. Schoenberg, 2014), we hope to keep clinicians and researchers alert when reading papers and designing new studies. 2. The replication crisis in social psychology We shall use three diferent examples from social psychology to illustrate diferent aspects of the replication crisis. The frst example is a claim promoted by certain research groups, and widely promulgated in the news media, that holding an open posture – the ‘power pose’ – can ‘cause neuroen- docrine and behavioral changes’ so that you ‘embody power and instantly become more powerful.’ These and related assertions have been questioned both by statistical arguments detailing how the published fndings could have occurred by chance alone (data exclusion, interactions, choices in regressions, miscalculation of p-values) and by a failed replication that was larger in size than the original study (Ranehill et al., 2015; Simmons & Simonsohn, 2015). From a science-communication perspective, a major challenge is that the researchers on the original study (Carney, Cuddy, & Yap, 2010) seem to refuse to accept that their pub- lished fndings may be spurious, despite the failed replications, the statistical theory that explains how they could have obtained low p-values by chance alone (Gelman & Loken, 2014), and the revelations of errors in their calculations (e.g. Gelman, 2016a). Of course in science mistakes can be made, but the crucial aspect of science is that we recognize these mistakes and learn from them so we can increase the body of knowledge regarding a specifc topic and know which fndings are actually false. Replication – and learning from replications both successful and failed – is a crucial aspect of research. As our second example, we use a clear demonstration of the value of replication focusing on the phenomenon of social priming, where a subtle cognitive or emotional manipulation infuences overt behavior. The prototypical example for social priming is the elderly walking study from Bargh, Chen, and Burrows (1996). Students were either confronted with neutral words or with words that are related to the concept of the elderly (e.g. ‘bingo’). This is the Downloaded by [Columbia University Libraries] at 15:00 13 September 2017 so-called priming phase which seemed to be efective as students’ walking speed was slower after having been primed with the elderly-related words. Wagenmakers, Wetzels, Borsboom, Kievit, and van der Maas (2015) critically discuss these fndings and quote cognitive psy- chologist Kahneman who wrote of such studies: The reaction is often disbelief … The idea you should focus on, however, is that disbelief is not an option. The results are not made up, nor are they statistical fukes. You have no choice but to accept that the major conclusions of these studies are true. Maybe not. Wagenmakers et al. continue: At the 2014 APS annual meeting in San Francisco, however, Hal Pashler presented a long series of failed replications of social priming studies, conducted together with Christine Harris, the upshot of which was that disbelief does in fact remain an option. 1002 A. GELMAN AND H. M. GEURTS The quote from Kahneman is, in retrospect, ironic given his celebrated work such as reported in Tversky and Kahneman (1971), fnding that research psychologists were consistently over- stating the statistical evidence from small samples. Being convinced that something is a genuine efect is not sufcient, and can sometimes even trick thoughtful researchers into believing in a false conclusion. Criticism and replication are essential steps in the scientifc feedback loop. There has, of course also been push back at revelations of statistical problems in published work. For example, in his response to the failed replications of his classic study, Bargh (2012) wrote: There are already at least two successful replications of that particular study by other, independ- ent labs, published in a mainstream social psychology journal. … Both appeared in the Journal of Personality and Social Psychology, the top and most rigorously reviewed journal in the feld. Both articles found the efect but with moderation by a second factor: Hull et al. 2002 showed the efect mainly for individuals high in self-consciousness, and Cesario et al. 2006 showed the efect mainly for individuals who like (vs. dislike) the elderly. … Moreover, at least two television science programs have successfully replicated the elderly-walking-slow efect as well, (South) Korean national television, and Great Britain’s BBC1. The BBC feld study is available on YouTube. This response inadvertently demonstrates a key part of the replication crisis in science: the researcher degrees of freedom that allows a researcher to declare victory in so many ways. To stay with this example: if the papers by Hull and Cesario had found main efects, Bargh could have considered them to be successful replications of his earlier work. Instead he considered the interactions to represent success – but there are many number of possible interactions or subsets where an efect could have been found. Realistically there was no way for Bargh to lose: this shows how, once an idea lodges in the literature, its proponents can keep it alive forever. This works against the ideally self-correcting nature of science. We are not saying here that replications should only concern main efects; rather, the purported replications reported by Bargh are problematic in that they looked at diferent interactions in diferent experiments justifed with post hoc explanations. Our third example comes from Nosek, Spies, and Motyl (2012) who performed an exper- iment focusing on how political ideology related to performance on a perceptual judgment task. Their initial fndings were, to use their own words, ‘stunning.’ Based on the results the conclusion could have been that political