Download Preprint

Psychological Science replicates just fine, thanks Psychological Science replicates just fine, thanks Abstract This manuscript aims to make two things clear. First, perhaps (to improve matters) Psychological Science should provide pupils (and others) proper tools, and a basis. Perhaps Psychological Science should be more aware of, acknowledge, focus on, train, and try and optimize what possibly underlies, permeates, and/or influences nearly everything mentioned in this manuscript: reasoning, logic, and argumentation. Secondly, it is hoped that this manuscript makes clear that there might be a distinct possibility that seen from several different perspectives, in several different ways, and on several different levels Psychological Science replicates just fine. Keywords: reasoning, logic, argumentation, replication, open science, crisis of confidence, replication crisis, education, academia, university A brief scientific introduction In 2012, Doyen et al. (2012) published a paper concerning a “failed” replication of a study by Bargh et al. (1996). Later that year Bargh (2012) wrote a blog post titled “Priming effects replicate just fine, thanks” responding to an article (and more general concerns) regarding the replicability of so-called “priming” effects. In the blog post, Bargh (2012) mentions several “successful” replication attempts. These included “direct” replications like the Doyen et al. (2012) study which can be viewed as being a repetition of an experimental procedure (cf. Schmidt, 2009, p. 91). And these included “conceptual” replications, which can be viewed as being a repetition of a test of a hypothesis or a result of earlier research work with different methods (cf. Schmidt, 2009, p. 91). The publishing of Doyen et al.’s (2012) “failed” replication, and Bargh’s (2012) reaction in the form of his blog post, can be placed in a larger series of events that all happened around 2011-2012, and caused quite a stir in (parts of) the world of Psychological Science. Around that time period, a paper was published that presented evidence, based on studies that involved commonly used research practices, for what can be considered to be “impossible” (Bem, 2011). As LeBel & Peters (2011) state: “By using accepted standards for experimental, analytic, and data reporting practices, yet arriving at a fantastic conclusion, Bem has put empirical psychologists in a difficult position (…)” (p. 371). Additionally, there was a paper published that made clear that certain research practices can lead to finding “statistically significant” results for just about anything (Simmons et al., 2011), and there was a paper published that presented findings that indicate that (some of) these “questionable research practices” may be widespread, and commonly used (John et al., 2012). On top of this all, it became known in 2011 that a certain Psychological scientist had engaged in fraudulent activities for years on end, leading to a subsequent investigation and report concerning the matter (Levelt et al., 2012). This report included several findings and/or statements that could further lead one to question the validity of certain findings, the capability and/or intention of certain scientists, and the general research culture of certain parts of Psychological Science. Finally, at the end of 2012 a special replication-issue (also mentioned in the Levelt et al. report) was published that included papers regarding replication, the possible “replication crisis”, more general problematic issues in Psychological Science, and the possibility of the existence of “a crisis of confidence” regarding the reliability and validity of research findings in the field (cf. Pashler & Wagenmakers, 2012). “Psychological Science replicates just fine, thanks”: Version and/or level I A brief personal introduction In 2011-2012 I graduated from university (“Research Master in Behavioural Science”), and subsequently encountered information (e.g. from papers in the special replication-issue mentioned above) that I thought should have been part of my education but wasn’t. Mostly from outside academia and not employed in science, I started trying to help improve matters by engaging in online discussions, posting ideas and manuscripts regarding possible improvements, joining efforts that could possibly help better things, etc. It’s now about 10 years later and at this point in time, from a certain perspective, I would say that my initial enthusiastic, and hopeful outlook regarding the attempts to try and improve matters has been replaced by an increasingly more frustrated, and cynical one. Concerning the latter, perhaps these current gloomy thoughts replicate the thoughts I had when first encountering information about the possibly problematic issues in Psychological Science some 10 years ago. During these 10 years or so, I increasingly began to notice and/or worry that some of the possibly problematic issues, and processes might still be ignored and/or neglected. I also increasingly began to notice and/or worry that the possibly problematic issues, and processes might be more or less “conceptually” replicated in the form of “new” proposals, and changes. At a certain point in time the idea emerged that it might be useful and/or amusing in some way, shape, or form to write a manuscript about the things I noticed and worried about. I also had the title of this possible manuscript in mind (“Psychological Science replicates just fine, thanks”) from remembering the title of the blog post by Bargh from 2012 (“Priming effects replicate just fine, thanks”). I hereby reasoned that this title hopefully, and perhaps even likely, would fit the approach, content, and style of writing that seems to come most naturally to me, is most enjoyable for me, and might therefore be most useful. On with the list then. Replication of the presence of “conceptual” replications? There may have been a sub-optimal emphasis on “conceptual” replications compared to “direct” replications in Psychological Science, although it may be hard to sometimes determine if something is a conceptual or direct replication (cf. Schmidt, 2009). In what follows, I will list some of the things, issues, and processes that I have noticed in the last 10 years or so that are possibly replications of other things, issues, and processes. Some of these things, issues, and processes seem to me to be problematic. Please note that the term “replication” is used very loosely in this context, and will mostly refer to a more or less “conceptual” replication where (viewed from a certain perspective) some (possibly crucial) part of something is possibly being replicated in some way, shape, or form. The following (“conceptual”) replications can perhaps be seen as being in line with the possibility that “Psychological Science replicates just fine”, at least seen from a certain perspective. Replication of the “crisis of confidence”? The accumulation, and combination, of the events in 2011-2012 mentioned in the introduction above may have contributed to the declaration of a “replication crisis” or a “crisis of confidence” (cf. Pashler & Wagenmakers, 2012) regarding the reliability, replicability, and/or validity of research findings in (parts of) Psychological Science. It might however be noteworthy that this may not be the first and/or only (type of) crisis Psychological Science has faced. The following two introductory sentences from two specific papers might be seen as especially illustrative in this regard. A paper by Lewin (1977) states the following: “This paper examines the crisis in social psychology in the light of attempts to solve the same problems fifty years ago. The fundamental question is, are the phenomena of social psychology lawful?” (p. 159). Sturm & Mülberger (2012) start their introduction with: “This special issue is devoted to the analysis of discussions of the crisis in psychology that took place from the 1890s through to the mid-1970s.” (p. 425). Replication of “mindless statistics”? Possibly encouraged and/or facilitated by the Simmons et al. (2011) paper and the declaration of a “replication crisis” or a “crisis of confidence” mentioned in the introduction, renewed attention for the possible problems of null-hypothesis significance testing (NHST) has since emerged. For instance, Gigerenzer (2018) writes in the abstract of his paper: “Here, I want to draw attention to a complementary internal factor, namely, researchers’ widespread faith in a statistical ritual and associated delusions (the statistical-ritual hypothesis).” (p. 198). The statistical ritual Gigerenzer (2018) refers to is “(…) an incoherent mishmash of ideas from Fisher on the one hand and Neyman and Pearson on the other, spiked with a characteristically novel contribution: the elimination of the researchers’ judgement.” (p. 202). Gigerenzer (2004) makes clear that null hypothesis testing received criticism decades ago: “The clinical psychologist Paul Meehl (1978, p. 817) called routine null hypothesis testing “one of the worst things that ever happened in the history of psychology” (…)” (p. 591). And as Cumming (2014) states in his paper: “(…) for more than half a century, scholars have been publishing cogent critiques of NHST, documenting the damage it does, and urging change. There have been very few replies, but also little reduction in reliance on NHST.” (p. 26). The recent papers that have outlined problematic issues regarding NHST have also come with several proposals to attempt to improve matters. For instance, Cumming (2014) mentions the following

Load more