March April 13 *_SI new design masters 1/31/13 9:52 AM Page 5 [ NEWS AND COMMENT Failure to Replicate Results of Bem Parapsychology “Correcting the Past: Failures to Repli- Experiments Published by Same Journal cate Psi.” The article is lengthy, but the central KENDRICK FRAZIER conclusion is succinctly stated: “Across seven experiments (N= 3,289), we replicate the procedure of Experiments Two years ago the prepublication re lease 8 and 9 from Bem (2011), which had orig- of a research paper by psychologist inally demonstrated retro active facilitation Daryl Bem claiming experimental evi- of recall. We failed to replicate that find- dence for precognition created a world- ing.” They further conducted a meta- wide media stir and intense controversy analysis of all replication attempts of the within the scientific and skeptical com- Bem experiments “and find that the aver- munities. age effect size (d = 0.04) is not different Bem, of Cornell University, claimed from 0.” that through nine experiments he had To put it even more directly (from the demonstrated the existence of precogni- beginning of their conclusions section): tion, specifically the existence of “con- “We conducted seven experiments test- scious cognitive awareness ...of a future ing for precognition and found no evi- event that could not otherwise be antic- dence supporting its existence.” ipated through any known inferential How can their results be reconciled process.” Essentially, he had claimed to Journal of Personality and Social Psychology © 2012 American Psychological Association 2012, Vol. 103, No. 6, 933–948 0022-3514/12/$12.00 DOI: 10.1037/a0029709 with Bem’s? “It is unclear how Bem have produced evidence that psychic Correcting the Past: Failures to Replicate Psi could find significant support for a hy- abilities not only exist but can transcend Jeff Galak Robyn A. LeBoeuf Carnegie Mellon University University of Florida pothesis that appears to be untrue,” the time and allow the future to reach back- Leif D. Nelson Joseph P. Simmons University of California, Berkeley University of Pennsylvania authors say. They suggest such possibil- Across 7 experiments (N ϭ 3,289), we replicate the procedure of Experiments 8 and 9 from Bem (2011), ward to change the past. which had originally demonstrated retroactive facilitation of recall. We failed to replicate that finding. We further conduct a meta-analysis of all replication attempts of these experiments and find that the ities as what psychologists call Type I average effect size (d ϭ 0.04) is no different from 0. We discuss some reasons for differences between Informed critics of parapsychology the results in this article and those presented in Bem (2011). Keywords: psi, precognition, ESP, researcher degrees of freedom, meta-analysis error—a false rejection of the null hy- were almost uniformly incredulous. Al- Recently, Bem (2011) published an extremely thought- provide the necessary tightly controlled experiments, the purpose provoking article demonstrating the existence of precognition, a of the current article is to conduct and to synthesize replications by pothesis. They also point to concerns “conscious cognitive awareness . of a future event that could not independent investigators. otherwise be anticipated through any known inferential process” though Bem is a respected psychologist, (p. 407). Through nine experiments, Bem found consistent support Psi Phenomena for the idea that people have such precognitive abilities. He sug- about researcher degrees of freedom, gested that these findings present examples of retroactive influ- The precognitive abilities reported by Bem (2011) emerged ence, through which future events influence people’s current re- across a range of tasks. As one example, in Experiment 1, Bem they found so many flaws in the research sponses and that more broadly these findings are instances of psi (2011) asked participants to select whether a picture would appear phenomena, or “anomalous processes of information or energy on the left side of the screen or the right side of the screen. which also raise the likelihood of falsely transfer that are currently unexplained in terms of known physical Participants’ selections were accurate more often than chance protocols and methods that in their view or biological mechanisms” (Bem, 2011, p. 407). would predict when the picture in question was an erotic one (but In his article, Bem (2011) acknowledged that psi is a contro- not a neutral, positive, or negative one), suggesting that people versial topic. He reported data suggesting that many, if not most, rejecting the null hy pothesis. While have precognitive abilities to detect where erotic stimuli will academic psychologists do not believe that psi phenomena exist. appear. the conclusions had no validity. One of Indeed, the publication of Bem’s research met with a wide variety Precognitive abilities also manifested on more complicated of reactions in the academic and popular media alike, and although tasks. For example, in Experiment 5, participants were asked to many experimental decisions are defen- some reactions were supportive, many were skeptical (Carey, choose which of two negatively arousing pictures they liked better. 2011a; Carey, 2011b; Wagenmakers, Wetzels, Borsboom, & van After this choice, the computer randomly selected one of the the most stinging re bukes came in the der Maas, 2011). In light of the skepticism surrounding psi and in pictures to serve as the target picture, which then flashed sublim- anticipation of the reaction to his article, Bem suggested that psi inally on the screen from 4 to 10 times. Research on the mere- sible, “because their application is at the researchers must conduct tightly controlled experiments that dem- exposure effect suggests that subliminal exposure to a negative onstrate psi and “that can be replicated by independent investiga- target increases liking of that target (i.e., causes habituation; form of an ex tended, in-depth critique of tors” (Bem, 2011, p. 407). Whereas Bem’s article may indeed Kunst-Wilson & Zajonc, 1980). Bem (2011) suggested that if people have precognitive abilities, their current liking of a negative discretion of the researcher examining picture would be enhanced by the fact that they will see that This article was published Online First August 27, 2012. picture several times in the future (even though they have no all nine experiments by York University Jeff Galak, Tepper School of Business, Carnegie Mellon University; known way of knowing that they will see it). Bem’s results Robyn A. LeBoeuf, Department of Marketing, University of Florida; Leif supported this prediction: When participants chose between neg- data after the completion of the exper- D. Nelson, Department of Marketing, University of California, Berkeley; ative picture pairs, they were more likely to prefer the one that Joseph P. Simmons, Operations and Information Management Department, would later be selected to be the subliminally presented target. University of Pennsylvania. Perhaps the most straightforward and impressive demonstration psychologist and CSI Executive Council We would like to thank Eduardo Andrade and Justin Kruger for crucial of precognition emerged in Bem’s (2011) Experiments 8 and 9, assistance with the project and Gia Nardini for her help in collecting data iment, they can make a true effect more for this article. which documented “retroactive facilitation of recall” (p. 419). In KEPTICAL Correspondence concerning this article should be addressed to Jeff these studies, participants saw 48 words and then were asked to member James Alcock in the S Galak, Tepper School of Business, Carnegie Mellon University, 5000 recall as many of those words as possible. Next, participants were Forbes Avenue, Office 381-D, Pittsburgh, PA 15213. E-mail: jgalak@ given a chance to practice a randomly chosen subset of the 48 difficult to discern. ... Researcher de- cmu.edu words by, for example, retyping them and recategorizing them. In INQUIRER (“Back from the Future: 933 grees of freedom do not make a finding Parapsy chology and the Bem Affair,” SI, false . but they do make it much March/April 2011; see also editorial various universities, but none were ac- harder to distinguish between truth and “Why the Bem Experiments are Not cepted for publication by JPSP. In fact, falseness in reported data.” Parapsychology’s Next Big Thing” in the it said it would not consider publishing They end by quoting philosopher of same issue). replication failures. This fact raised more science Karl Popper: “An effect is not an Alcock also concluded that the jour- controversy and concern. effect unless it is replicable, and a sci- nal that published Bem’s study, the Jour- Now the journal has had an apparent ence is not a science unless it conducts nal of Personality and Social Psy chology change of heart. It has finally published (and values) attempted replications.” (JPSP), had done everyone a disservice a set of experiments that attempted (and They do compliment Bem for encour- by publishing this “badly flawed research failed) to replicate Bem’s results. Seven aging the independent replication of his article.” Parapsy chology and the journal’s experiments conducted by Jeff Galak of experiments. own reputation, he wrote, had been dam- Carnegie Mellon University, Robyn A. Alcock, who wrote SI’s 2011 critique, aged, and the article’s publication dis- LeBoeuf of the University of Florida, has mixed feelings about the Galak et served the public as well, “for it only adds Leif D. Nelson of the Uni versity of Cal- al. paper: to [public] confusion about the existence ifornia at Berkeley, and Joseph P. Sim- While I am happy that they carried of psi.” mons of the University of Pennsylvania out this research and that it was pub- Experiments attempting to replicate have been published in JPSP’s final issue lished, I find it very odd that they fo- Bem’s results were quickly conducted at of 2012 (Vol. 103, No. 6) under the title cused on Bem’s studies as though Skeptical Inquirer | March/April 2013 5 March April 13 *_SI new design masters 1/31/13 9:53 AM Page 6 they had been carefully de signed and Here is his initial comment: al.