The Alleged Crisis and the Illusion of Exact Replication
Total Page:16
File Type:pdf, Size:1020Kb
PPSXXX10.1177/1745691613514450Stroebe and StrackAlleged Crisis and Illusion of Exact Replication 514450research-article2013 Perspectives on Psychological Science 2014, Vol 9(1) 59 –71 The Alleged Crisis and the Illusion of Exact © The Author(s) 2013 Reprints and permissions: sagepub.com/journalsPermissions.nav Replication DOI: 10.1177/1745691613514450 pps.sagepub.com Wolfgang Stroebe1,2 and Fritz Strack3 1Department of Psychology, Utrecht University, the Netherlands; 2Department of Social and Organizational Psychology, University of Groningen, the Netherlands; and 3Department of Psychology, University of Würzburg, Germany Abstract There has been increasing criticism of the way psychologists conduct and analyze studies. These critiques as well as failures to replicate several high-profile studies have been used as justification to proclaim a “replication crisis” in psychology. Psychologists are encouraged to conduct more “exact” replications of published studies to assess the reproducibility of psychological research. This article argues that the alleged “crisis of replicability” is primarily due to an epistemological misunderstanding that emphasizes the phenomenon instead of its underlying mechanisms. As a consequence, a replicated phenomenon may not serve as a rigorous test of a theoretical hypothesis because identical operationalizations of variables in studies conducted at different times and with different subject populations might test different theoretical constructs. Therefore, we propose that for meaningful replications, attempts at reinstating the original circumstances are not sufficient. Instead, replicators must ascertain that conditions are realized that reflect the theoretical variable(s) manipulated (and/or measured) in the original study. Keywords replication, replicability crisis, null findings, scientific fraud, priming, epistemology, critical rationalism At a time when social psychologists believed that they openscienceframework.org). In contrast to the prevalent had every reason to be proud of their discipline came the sentiment, we will argue that the claim of a replicability shattering news that Diederik Stapel, a prominent crisis is greatly exaggerated and that the hope that such researcher in social psychology, had committed scientific a crisis (if it ever existed) could be solved by increasing fraud on a major scale. Social psychologists had hardly the number of exact replications is misplaced. recovered from this shock when two more colleagues were accused of fraud and resigned from their positions. Is the Claim of a Replicability Crisis These events were particularly damaging, because they coincided with the start of a discussion of trust in psycho- Exaggerated? logical data (see Special Section on Replicability in There seems to have been two sets of events that fueled Psychological Science: A Crisis of Confidence? Perspectives the crisis perception. First, there have been claims that on Psychological Science, 2012). Even though this discus- some psychological researchers engage in “questionable sion focused on methodological issues that were unre- research practices” that result in “false positive” findings lated to fraud, this distinction was not always maintained (e.g., Bakker, van Dijk, & Wicherts, 2012; John, Loewenstein, by the popular press. According to the introduction to & Prelec, 2012; LeBel et al., 2013; Simmons, Nelson, & the special section, there is “currently a crisis of confi- Simonsohn, 2011; Vul, Harris, Winkielman, & Pashler, dence in psychological science reflecting an unprece- 2009). Simmons et al. (2011) reported simulations that dented level of doubt among practitioners about the showed that a number of research practices (e.g., stopping reliability of research findings in the field” (Pashler & data collection on the basis of interim data analysis; Wagenmakers, 2012, p. 528). Nosek and colleagues started the “Reproducibility Project,” a large-scale, collab- Corresponding Author: orative effort to estimate the reproducibility of psycho- Wolfgang Stroebe, Department of Psychology, Utrecht University, P.O. logical science, which involves replicating all studies Box 80140, 3508 TC, Utrecht, the Netherlands published in three psychology journals in 2008 (http:// E-mail: [email protected] Downloaded from pps.sagepub.com by Gerald Haeffel on January 8, 2015 60 Stroebe and Strack dropping experimental conditions from published reports) memory (e.g., Higgins, Rholes, & Jones, 1977; Meyer & can result in an erroneous rejection of the null hypothesis. Schvaneveldt, 1971; Tulving & Schacter, 1990). In this tra- To be sure, methodological discussions are important for dition, its impact on behavior was studied in a minor any discipline, and both fraud and dubious research pro- subarea of research, whereas most social psychological cedures are damaging to the image of any field and poten- priming studies investigated the impact of subliminal or tially undermine confidence in the validity of social supraliminal primes on judgment. A meta-analysis of psychological research findings. Thus far, however, no studies that investigated how trait primes influence solid data exist on the prevalence of such research prac- impression formation identified 47 articles based on tices in either social or any other area of psychology. 6,833 participants and found overall effects to be statisti- In fact, the discipline still needs to reach an agreement cally highly significant (DeCoster & Claypool, 2004). about the conditions under which these practices are unacceptable. Are Exact Replications the Answer? Second, there have been a number of failures to repli- cate high-profile experiments on social priming (Doyen, The claim of a replicability crisis in psychology is based Klein, Pichon, & Cleeremans, 2012; Pashler, Coburn, & on a major misunderstanding. Particularly, the myopic Harris, 2012; Shanks et al., 2013). For example, Doyen focus on “exact” replications neglects basic epistemologi- et al. (2012) reported an exact replication of a study by cal principles. Exact replications are replications of an Bargh, Chen, and Burrows (1996; Experiment 2a, 2b) that experiment that operationalize both the independent and failed to reproduce the original results. Bargh et al. had the dependent variable in exactly the same way as the repeatedly found that students who had been primed original study. (In contrast, conceptual replications try to with words that triggered the stereotype of elderly walked operationalize the underlying theoretical variables using more slowly down a corridor than students primed with different manipulations and/or different measures.) words unrelated to the elderly stereotype. The Doyen In evaluating the usefulness of exact replications, one et al. (2012) failure was soon followed by a report by has to distinguish between applied and basic research. A Shanks et al. (2013) of a series of nine studies that failed scientist who wants to establish the efficiency of a spe- to replicate the findings of another iconic experiment, the cific treatment or intervention is well advised to repeat- “professor study” of Dijksterhuis and van Knippenberg edly apply exactly the same procedure. This is particularly (1998). These authors had found that participants who relevant for clinical trials where a lack of reliability may were primed with a category of persons who are consid- have fatal consequences. However, matters are different ered highly intelligent (e.g., professors) performed better in basic research where empirical outcomes are mean- on a task of trivial pursuit than did participants primed ingful only with respect to the theory being tested. In the with a category of persons who are considered less intel- postbehaviorist era, psychological theories are based on ligent (e.g., hooligans). These replication failures were internal mechanisms such that replications must be the more astounding because of earlier publications of directed at the internal antecedents of such theories. successful replications of both of these findings (summa- Although reproducibility of scientific findings is one of rized in the Appendix). science’s defining features, the ultimate issue is the extent Do these failures to replicate amount to a crisis of rep- to which a theory has undergone strict tests and has been licability, as Pashler and Harris (2012) claimed in their supported by empirical findings. It would be a mistake to contribution to the special section of Perspectives on assume that estimates of the reproducibility of empirical Psychological Science on replications? We would argue findings are the same as estimates of the validity of a that such a conclusion is premature. Failures to replicate specific theory. A finding may be eminently reproducible are puzzling, but in social psychology, as in most sci- and yet constitute a poor test of a theory. ences, empirical findings cannot always be replicated The fact that good experimental research is typically (this was one of the reasons for the development of conducted with the aim to test theories throws a different meta-analytic methods). It is therefore surprising that the light on the discussion of replicability. We will therefore failures to replicate some social priming studies received briefly discuss the notion of theory and what researchers such disproportionate attention. Furthermore, one must do when they test hypotheses derived from a theory. wonder whether the response by Kahneman, who in a Theories consist of a set of abstract constructs and