Archived Discussions Following to Doyen Et Al Failure to Replicate Bargh Chen & Burrows 1996 in Our Annual R

Home , Ed Yong, John Bargh, Norbert Schwarz

Archived discussions following to Doyen et al failure to replicate Bargh Chen & Burrows 1996

Supplementary material for Nelson et al., “Psychology’s Renaissance”, in Annual Review of Psychology. This copy: March 27, 2017

In our Annual Review article on “Psychology’s Renaissance” we refer to an exchange triggered by Doyen et al’s failure to replicate the original article by Bargh et al. Below we have compiled some key exchanges that ensued among the original first author, John Bargh, a science journalist Ed Yong, Nobel Laurette, Danny Kahneman, and social psychologist Norbert Schwarz.

In this first page we include an link to the original URLs (when available), a link to the web‐archived copy, and to a .pdf printout of those sites stored in the OSF. All those .pdf files were combined into a single file and presented in the pages that follow.

– 0 –

1) John Bargh’s. Psychology Today Blogpost, March 5th, 2012, “Nothing in Their Heads” Note: this blogpost was subsequently deleted, but archived copies are available: Archived copy: http://web.archive.org/web/*/http:/www.psychologytoday.com/blog/the‐natural‐unconscious/201203/nothing‐in‐their‐heads 2nd copy (.pdf): https://osf.io/bgb3c/

2) Ed Yong. Blogpost in Discover, March 10th, 2012 “A failed replication draws a scathing personal attack from a psychology professor” Original URL: http://blogs.discovermagazine.com/notrocketscience/2012/03/10/failed‐replication‐bargh‐psychology‐study‐doyen Archived copy: http://web.archive.org/web/*/http://blogs.discovermagazine.com/notrocketscience/2012/03/10/failed‐replication‐bargh‐psychology‐study‐doyen 2nd copy (.pdf): https://osf.io/esgdh/

3) Danny Kahneman’s open letter, September 26 2012, as made available in Decision Science News (a blog hosted by Dan Goldstein) Original: http://www.decisionsciencenews.com/2012/10/05/kahneman‐on‐the‐storm‐of‐doubts‐surrounding‐social‐priming‐research

Archived copy: http://web.archive.org/web/*/http://www.decisionsciencenews.com/2012/10/05/kahneman‐on‐the‐storm‐of‐doubts‐surrounding‐social‐priming‐ research/ 2nd copy:

4) Ed Yong. Nature News. October 3rd, 2012. “Nobel laureate challenges psychologists to clean up their act” Original url: http://www.nature.com/news/nobel‐laureate‐challenges‐psychologists‐to‐clean‐up‐their‐act‐1.11535

Archived copy: http://web.archive.org/web/*/http://www.nature.com/news/nobel‐laureate‐challenges‐psychologists‐to‐clean‐up‐their‐act‐1.11535 2nd copy (.pdf) https://osf.io/gcske/

5) Norbert Schwarz. Response to Ed Young’s questions, October 2nd 2012. Posted on Google Docs. Original url: https://docs.google.com/document/pub?id=1KKN5Gz35CPz6QmtpJ7‐cIwdLG_zeceuu67WRmlE6U8A Archived copy (.pdf): https://osf.io/v3435/

Nothing in Their Heads | Psychology Today http://web.archive.org/web/20120307100648/http://www.psychologytoda...

A narcissist will almost certainly look good in public but infuriate you in private. Mark Banschick, M.D. Home Find a Therapist Topic Streams Get Help Magazine Tests Psych Basics Experts

Previous Post Automaticity in cognition, motivation, and emotion. Does Kanye West by John A. Bargh, Ph.D. believe in free will? Yes and no... Nothing in Their Heads Debunking the Doyen et al. claims regarding the elderly-priming study Subscribe to The Natural Unconscious Published on March 5, 2012 by John A. Bargh, Ph.D. in The Natural Unconscious Subscribe via RSS

Welcome back to "The Natural Unconscious," which returns after a hiatus of a couple years. Why we went underground for awhile is a topic for a later post; we'll get to that soon. John Bargh is Professor of Psychology and Cognitive Science at Yale University. Scientific integrity in the era of pay-as-you-go publications and superficial online science more... journalism. What prompts the return of the blog is a recent article titled "Behavioral Priming: It's All in the Mind, but Whose Mind?" by Stéphane Doyen, Olivier Klein, Cora-Lise Pichon, and Axel Cleeremans. The researchers reported that they could not replicate our lab's 1996 finding that priming (subtly activating in the minds of our college-age experimental participants, without their awareness) the stereotype of the elderly caused participants to walk more slowly when leaving the experiment. We had predicted this effect based on emerging theory and evidence that perceptual mental representations were intimately linked with behavioral representations, a finding that is very well The Natural Unconscious established now in the field (see below). Following their failure to replicate, Doyen et al. went on to show that if the experimenter knew the hypothesis of the study, they were able to then find the effect. Their conclusion was that experimenter expectancies or awareness of the research hypotheses had Debunking the Doyen et al. claims regarding the elderly-priming therefore produced the effect in our original 1996 study as well—in other words, that there was no study actual unconscious stereotype effect on the participants' behavior. One's belief in free will goes missing The Doyen et al. article appeared in an online journal, PLoS ONE, which in the face of others' disapproval Related Articles quite obviously does not receive the usual high scientific journal standards Experimental of peer-review scrutiny (keep reading for the evidence of this); instead, the One reason the public needs to know Philosophy Month journal follows a "business model" in which authors pay to have their about limits to their free will To Ignore or Confront? Dealing with Racially articles published (at a hefty $1,350 per article). The journal promises a Stereotyping "rigorous peer review" for technical soundness but not as to the What I think about what he thinks I Comments importance of the finding. On their website PLoS dismisses the use of think The Greater Good: Psychology and Social knowledgable editors to oversee what gets published and what does not, Policy claiming this adds only a subjective element to the acceptance decision The debate itself, the newsletter articles, and now the blogs Sneaky Commercials: that can be biased against new research directions. But knowledgeable The Unconscious Way editors also can prevent articles from being published based on faulty peer TV Makes You Eat Why Personal Science reviews, such as by inexpert, lazy, or biased reviewers. Expert editors also is a Good Idea know the relevant theory and past research in a given domain, and also know of common methodological pitfalls that inexpert researchers in the domain—such as, apparently, Doyen et al. (keep reading)—can fall prey to. Most Read Most Emailed Find a Therapist 1 Teasing and Bullying, Boys and The lack of rigorous expert editorial scrutiny by PLoS in the Doyen et al. Girls Search for a mental case means that I must supply it here, only after it has been published. If by Nancy Darling, Ph.D. health professional near you. I'd been asked to review it (oddly for an article that purported to fail to replicate one of my past studies, I wasn't) I could have pointed out at that 2 How Your Greatest Insecurities time the technical flaws, though these might not have mattered to PLoS Reveal Your Deepest Gifts by Ken Page, L.C.S.W. ONE—as a for-profit enterprise, PLoS published 14,000 articles in the year 2011 alone. Fourteen thousand. Something tells me they don't turn down 3 What Makes A True Friend many $1,350 checks... by Alex Lickerman, M.D.

Although the essentially self-published nature of the Doyen et al. article is bad enough, the misleading conclusions it drew were made even worse by 4 The How and Why of 100 Years of the publicity given to them by some online science-journalism blogs, one of Happiness Find Local: which posted about the Doyen et al. failure with the title "Primed by by Howard S. Friedman, Ph.D. Acupuncturists expectations: Why a classic psychology experiment isn't what it seemed". Chiropractors So as you have read this far, and are therefore a reader interested in 5 What Nonbelievers Believe Massage Therapists by David Niose Dentists psychological science, I'd like (and apparently need) to set the record and more! straight.

There are two main reasons why the Doyen et al. conclusions, and those

1 of 3 3/27/2017 11:33 AM Nothing in Their Heads | Psychology Today http://web.archive.org/web/20120307100648/http://www.psychologytoda...

in science journalism blogs that swallowed their conclusions whole, are Clos 36 captumisleading and false. The first is that in our original 1996 research the 7 7 Mar 12 -experim 28 Dec 16 e 201120122013 Help experimental condition of the participant and did not even collect the main dependent variable of walking time down the hall. Thus there is no way in our original 1996 studies that the effects could have been produced by Are You with the Right Mate? It is natural to wonder if your partner experimenter expectations. The second is that Doyen et al. did not follow is the right one for you. our original procedure (which is what one must do when attempting to MORE FROM THIS ISSUE replicate a study) but instead made critical changes that are known (in social psychology, which is not Doyen et al's field of specialization) to ISSUE ARCHIVES eliminate the stereotype-behavior effect. I'll take these points one at a time. SUBSCRIBE

1. The experimenter was entirely blind to hypotheses and participants' experimental conditions in the 1996 elderly-priming study. As recounted elsewhere, when we designed and ran the elderly stereotype priming studies (actually, back in 1991), we had solid theoretical reasons to hypothesize the effects we did obtain. But we were very careful to make sure that experimenter bias could not cause the effect. The experimenter who ran the participants in the study was blind to the study hypotheses, and all he did was to greet the participants and give them an envelope. He told the participants to go into the experimental room, open the envelope and follow the instructions inside. Moreover, he told the participants to leave the material inside the room. Thus the person who had actual contact with the participants in the elderly priming study never saw the priming manipulation (the scrambled sentence test), and certainly did not know whether the participant was in the elderly priming or the control condition.

The experimenter also did not collect the dependent measure of the study, which was how long the participant took to walk down the hallway leaving the experimental room. This measure was collected by a different person, a graduate student, who was posing as the next participant and sat waiting with her coat in her lap in a chair outside the experimental room. She had a stopwatch under the coat and surreptitiously started it when the participant came out of the experimental room, and stopped it when the participant disappeared around the corner at the end of the hallway.

Thus there is no possible way that experimenter expectations or any other form of experimenter bias could have produced our results in the original 1996 elderly priming study. We were fully aware of this potential problem at the time and took every precaution to make sure it did not contaminate or spuriously produce our results.

2. Doyen et al.'s replication of our 1996 study had many important differences from our procedure, all of which worked to eliminate the effect. When researchers attempt (in good faith) to replicate another lab's findings, they are supposed to follow the original procedure as closely as possible. But Doyen et al.'s procedure differed in many important respects from our original study, and three of their departures from our procedures were especially likely to mask or eliminate the stereotype priming effect, and so cause them not to find the effect.

1 2 3 next › last »

Like 0 Tweet

1 Reader comments join the discussion here!

Tags: business model, cora, doyen, expectancies, experimenter, experimenter bias, hiatus, hypothesis, mental representations, participants, peer review, pichon, plos one, priming, recent article, research hypotheses, science journalism, science reporting, scrutiny, stereotype, stereotypes, technical soundness, unconscious

Follow Psychology Today: Twitter FaceBook

About/Contact Therapy Directory Index Privacy Policy Healthprofs Index

2 of 3 3/27/2017 11:33 AM Nothing in Their Heads | Psychology Today http://web.archive.org/web/20120307100648/http://www.psychologytoda...

© Copyright 1991-2012 © Copyright 2002-2012 Sussex Directories, Inc. Site Help/Customer Buildingpros Index Clos Sussex Publishers, LLC Service 36 captu Terms of Service 7 7 Mar 12 - 28 Dec 16 201120122013 Help

3 of 3 3/27/2017 11:33 AM A failed replication draws a scathing personal attack from a psychology ... http://blogs.discovermagazine.com/notrocketscience/2012/03/10/failed-re...

« A shiny dinosaur –four-winged Microraptor gets colour and gloss I've got your missing links right here (10 March 2012) »

By Ed Yong | March 10, 2012 12:00 pm

John Bargh, a psychologist at Yale University, has published a scathing attack on a paper that failed to replicate one of his most famous studies. His post, written on his own blog on Psychology Today, is a mixture of critiques of the science within the paper, and personal attacks against the researchers, PLOS ONE, the journal that published it, and me, who covered it. I’m going to take a closer look at Bargh’s many objections.

The background

First, a recap. The original study, published in 1996, is indeed a classic. According to Google Scholar, it has been cited almost 2,000 times. Here’s how I described it in my post:

John Bargh and his colleagues found that infusing people’s minds with the concept of age could slow their movements (PDF). The volunteers in the study had to create a sentence from scrambled words pick the odd word from a group of scrambled ones. When this word related to being old, the volunteers walked more slowly when they left the laboratory. They apparently didn’t notice anything untoward about the words, but their behaviour changed nonetheless.

1 of 32 3/27/2017 11:45 AM A failed replication draws a scathing personal attack from a psychology ... http://blogs.discovermagazine.com/notrocketscience/2012/03/10/failed-re...

Surprisingly, this prominent result has seldom been replicated. There have been two attempts but neither stuck closely to the original experiment. This prompted Stephane Doyen and colleagues to try and repeat Bargh’s study. They tried to match the original set-up, but they made some tweaks: they timed volunteers with infrared sensors rather than a stopwatch; they doubled the number of volunteers; and they recruited four experimenters who carried out the study, but didn’t know what the point of it was. As I wrote:

This time, the priming words had no impact on the volunteers’ walking speed. They left the test room neither more slowly nor more quickly than when they arrived. Doyen suspected that Bargh’s research team could have unwittingly told their volunteers how they were meant to behave… Perhaps they themselves moved more slowly if they expected the volunteer to do so. Maybe they spoke more languidly, or shook hands more leisurely… Maybe they were responsible for creating the very behaviour they expected to see.

To test that idea, Doyen repeated his experiment with 50 fresh volunteers and 10 fresh experimenters. The experimenters always stuck to the same script, but they knew whether each volunteer had been primed or not. Doyen told half of them that people would walk more slowly thanks to the power of priming, but he told the other half to expect faster walks.

…He found that the volunteers moved more slowly only when they were tested by experimenters who expected them to move slowly… Let that sink in: the only way Doyen could repeat Bargh’s results was to deliberately tell the experimenters to expect those results.

Was this possible? In Bargh’s study, an experimenter had packed envelopes with one of two different word tasks (either elderly-related or neutral words). When each volunteer arrived, the experimenter chose an envelope at random, led the volunteer into a test room, briefed them, and then left them to finish the task.

Doyen thinks that, during this time, the experimenter could have seen which set of tests the volunteer received, and tuned their behaviour accordingly. This was not a deliberate act of manipulation, but could easily have been an unconscious one. He wrote, “This possibility was in fact confirmed informally in our own study, as we found that it was very easy, even unintentionally, to discover the condition in which a particular participant takes part by giving a simple glimpse to the priming material.”

In his new post, Bargh dismisses Doyen’s experiments on two technical points, and other personal ones. Let’s consider each in turn.

Bargh’s objections – blinding

First, he says that “there is no possible way” that the experimenter in his study could have primed the volunteers with his own expectations. He says that the experimenter “was blind to the study hypotheses” (meaning that he didn’t know what the point of the experiment was). Bargh adds, “The person who had actual contact with the participants in the elderly priming study never saw the priming manipulation… and certainly did not know whether the participant was in the elderly priming or the control condition.”

Could the experimenter have known what the experiment was about, even though Bargh asserts that they were blind? In the comments section of Bargh’s post, another psychologist, Matt Craddock, notes that the experimenter was also responsible for pre-packaging the various tasks in their packets, and so

2 of 32 3/27/2017 11:45 AM A failed replication draws a scathing personal attack from a psychology ... http://blogs.discovermagazine.com/notrocketscience/2012/03/10/failed-re...

had ample time to study the materials. [This is the first of several inconsistencies in Bargh’s interpretation of his own study – more on that later.)

Could the experimenter have primed the volunteers? It’s not clear. This hinges on what actually happened in the test room, and we only have Bargh’s word on this. There is precious little in the way of description in the actual paper (here it is as a PDF; let me know if I’ve missed something). As such, the paper does not seem to be at odds with Doyen’s vision of what happened, although it does not provide evidence for it either.

Bargh’s objections – differences between the two studies

Bargh’s second objection (in many parts) is that Doyen’s study had differences from his own, which would have eliminated the elderly-priming effect. However, in all of these cases, Craddock and other commenters have pointed out inaccuracies in his statements.

For example, he says that after the test, Doyen instructed his volunteers to “go straight down the hall when leaving” (his quotes), while he “let the participant leave in the most natural way”. This is important because drawing someone’s attention to an automatic process tends to eliminate that effect. But Doyen did nothing of the sort, and his paper never contains the words that Bargh quoted. Instead, Doyen wrote, “Participants were clearly directed to the end of the corridor”. It is not clear how this differs from Bargh’s own study where “the experimenter told the participant that the elevator was down the hall”.

Bargh also says that Doyen used too many age-related words in his word task. The volunteers might have noticed, cancelling out the effect of the priming. But this contradicts what Bargh says in his own methods paper, where he says that if there are too many primes, volunteers would be more likely to perform as expected. By that reasoning, Doyen’s volunteers should have showed an even stronger effect.

Bargh says that priming depends on there being something to prime. Volunteers would only walk more slowly if they associated old age with infirmity. He says, “Doyen et al. apparently did not check to make sure their participants possessed the same stereotype of the elderly as our participants did.” However, neither did Bargh. His original study says nothing about assessing stereotypes. [Update: actually, I note that Doyen et al chose their priming words by using the most common answers in an online student survey where people reported adjectives related to old age; that’s at least a tangential way of assessing stereotypes.]

“To adapt the items, we conducted an online survey (80 participants) in which participants had to report 10 adjectives related to the concept of old age. Only the most frequent responses were used as replacement words.” (i.e., as primes) Bargh says that Doyen used the same experimenter who administered the test to time how slowly the volunteers walked down the hall. This is also false – they used infrared sensors.

What do Doyen’s team have to say about Bargh’s criticisms? They support Craddock’s analysis. And one of the authors, Axel Cleeremans says:

“The fact is that we failed to replicate this experiment, despite having twice as many participants and using objective timing methods. Regardless of the arguments one may come up with that explain why his study worked and ours did not, this suggests that unconscious behavioural priming is not as strong as it is cast to be. If the effect were truly robust, it shouldn’t depend on minute differences. The fact that we did manage to replicate the original results when both

3 of 32 3/27/2017 11:45 AM A failed replication draws a scathing personal attack from a psychology ... http://blogs.discovermagazine.com/notrocketscience/2012/03/10/failed-re...

experimenters and participants were appropriately primed suggests interesting avenues for further research and should be taken as an opportunity to better delineate the conditions under which the effect is observed.”

Bargh’s objections – er, the other stuff

As stated before, Bargh also directs personal attacks at the authors of the paper (“incompetent or ill-informed”), at PLoS (“does not receive the usual high scientific journal standards of peer-review scrutiny”), and at me (“superficial online science journalism”). The entire post is entitled “Nothing in their heads”.

Yes, well.

I’ve dealt with the scientific aspects of the critique; I think we’re all a bit too old to respond to playground tactics with further puerility. The authors certainly aren’t rising to it. In an email to me, Doyen wrote, “This entire discussion should be about the reasons that best explain the differences between his findings and ours, but has somehow turned into something else that unhelpfully confuses personal attacks with scientific disagreement as well as scientific integrity with publishing politics.” And PLoS publisher Peter Binfield has already corrected Bargh’s “several factual errors” about their journals.

For my part, I’m always happy to correct myself when I’ve screwed up in my reporting. Here, I believe I did my due diligence. Contrary to accusations at the time, I read both the Bargh and Doyen papers. I contacted other psychologists for their view, and none of them spotted egregious technical flaws. More importantly, I sent the paper to Bargh five days before the embargo lifted and asked for a comment. He said, “There are many reasons for a study not to work, and as I had no control over your [sic] attempt, there’s not much I can say.” The two-page piece he has now posted would seem to falsify that statement.

After some reflection, I largely stand by what I wrote. I can’t see much in the original study or in Bargh’s critique that would have caused me to decide not to cover it, or to radically change my approach. There is one thing, though. Someone (on Twitter; sorry, I can’t find the link) noted that a single failure to replicate doesn’t invalidate the original finding, and this is certainly true. That’s something I could have made more explicit in the original post, maybe somewhere in the fourth paragraph. Mea culpa.

Replicate, good times, come on! (It’s a replication… it’s a replicatio-o-on)

There is a wider issue here. A lack of replication is a large problem in psychology (and arguably in science, full stop). Without it, science has lost a limb. Results need to be checked, and they gain strength through repetition. On the other hand, if someone cannot repeat another person’s experiments, that raises some serious question marks.

Scientists get criticised for not carrying out enough replications – there is little glory, after all, in merely duplicating old ground rather than forging new ones. Science journals get criticised for not publishing these attempts. Science journalists get criticised for not covering them. This is partly why I covered Doyen’s study in the first place.

In light of this “file drawer problem”, you might have thought that replication attempts would be welcome. Instead, we get an aggressive and frequently ill-founded attack at everyone involved in such an attempt. Daniel Simons, another noted psychologist, says, “[Bargh’s] post is a case study of what

4 of 32 3/27/2017 11:45 AM A failed replication draws a scathing personal attack from a psychology ... http://blogs.discovermagazine.com/notrocketscience/2012/03/10/failed-re...

NOT to do when someone fails to replicate one of your findings.”

Others have suggested that the Bargh study has many positive replications, but this is in question. In his post, Bargh speaks of “dozens if not hundreds of other conceptual replications”. He says that the “stereotype priming of behavior effect has been widely replicated”, and cites the well-established “stereotype threat” effect (which I have also written about). He implores responsible scientists and science journalists to not “rush to judgment and make claims that the entire phenomenon in question is illusory”.

I’m not sure which scientists or science journalists he is referring to. Neither Doyen nor I implicated that the entire concept of priming was illusory. I specifically said the opposite, and quoted two other psychologists who did the same. The issue at stake is whether Bargh’s results from that one specific experiment could be replicated. They could not.

Notably, one site – PsychFileDrawer – is trying to rectify the file drawer problem by providing psychologists with a “quick and easy way” to post the results of replication attempts, whether positive or negative. Hal Pashler, who created the site, has also reportedly tried to replicate Bargh’s study and failed.

If there’s an element to this farrago that heartens me, it’s that the comments in Bargh’s piece allowed various parties to set the record straight. In concluding his piece, Bargh says, “I’m worried about your ability to trust supposedly reputable online media sources for accurate information on psychological science.” Well, dear professor, this is the era of post-publication peer review. I’m not that worried.

Image source

CATEGORIZED UNDER: NEUROSCIENCE AND PSYCHOLOGY ADVERTISEMENT Comments (60)

You handled that a lot better than I would have. I can’t really believe that Blargh would stoop so low. Shockingly poor form.

March 10, 2012 at 12:33 pm

Stuart Coleman

Honestly I feel the very thing Bargh criticized when he attacked you and your lack of credentials is a boon to science. Finally there is someone outside of the old boys network who polices the results of the peer review process with all its biases and problems. Qui custodiet ipsos custodes? Ed Young on a good day. Rock on!

jakob March 10, 2012 at 12:39 pm

It’s quite sad that a distinguished scientist resorts to personal attacks to cover potentials flaws in his work…

IMarch think 10, you’ve 2012 at 12:47hit on pm a serious issue with the lack of replication in science. Researchers at all levels, especially PhDs and post-docs (i.e. when most of the

5 of 32 3/27/2017 11:45 AM http://www.decisionsciencenews.com/2012/10/05/kahneman-on-the-stor...

[ View menu ]

Main

Filed in Gossip ,Ideas ,Research News Subscribe to Decision Science News by Email (one email per week, easy unsubscribe)

OPEN LETTER URGES LABS TO REPLICATE RESULTS TO AVOID A LOOMING “TRAIN WRECK”

Daniel Kahneman issued an open letter to researchers doing social priming research, which has become the subject of skepticism after some studies were found to be fabricated and others were not able to be independently replicated. His letter offers advice to scholars about how to address the situation: Find out the truth through extensive replication and announce it.

The text of the letter is below:

From: Daniel Kahneman Sent: Wednesday, September 26, 2012 9:32 AM Subject: A proposal to deal with questions about priming effects

Dear colleagues,

I write this letter to a collection of people who were described to me (mostly by John Bargh) as students of social priming. There were names on the list that I could not match to an email. Please pass it on to anyone else you think might be relevant.

1 of 6 3/27/2017 12:10 PM http://www.decisionsciencenews.com/2012/10/05/kahneman-on-the-stor...

As all of you know, of course, questions have been raised about the robustness of priming results. The storm of doubts is fed by several sources, including the recent exposure of fraudulent researchers, general concerns with replicability that affect many disciplines, multiple reported failures to replicate salient results in the priming literature, and the growing belief in the existence of a pervasive file drawer problem that undermines two methodological pillars of your field: the preference for conceptual over literal replication and the use of meta-analysis. Objective observers will point out that the problem could well be more severe in your field than in other branches of experimental psychology, because every priming study involves the invention of a new experimental situation.

For all these reasons, right or wrong, your field is now the poster child for doubts about the integrity of psychological research. Your problem is not with the few people who have actively challenged the validity of some priming results. It is with the much larger population of colleagues who in the past accepted your surprising results as facts when they were published. These people have now attached a question mark to the field, and it is your responsibility to remove it.

I am not a member of your community, and all I have personally at stake is that I recently wrote a book that emphasizes priming research as a new approach to the study of associative memory – the core of what dual-system theorists call System 1. Count me as a general believer. I also believe in a point that John Bargh made in his response to Cleeremans, that priming effects are subtle and that their design requires high-level skills. I am skeptical about replications by investigators new to priming research, who may not be attuned to the subtlety of the conditions under which priming effects are observed, or to the ease with which these effects can be undermined.

My reason for writing this letter is that I see a train wreck looming. I expect the first victims to be young people on the job market. Being associated with a controversial and suspicious field will put them at a severe disadvantage in the competition for positions. Because of the high visibility of the issue, you may already expect the coming crop of graduates to encounter problems. Another reason for writing is that I am old enough to remember two fields that went into a prolonged eclipse after similar outsider attacks on the replicability of findings: subliminal perception and dissonance reduction.

I believe that you should collectively do something about this mess. To deal effectively with the doubts you should acknowledge their existence and confront them straight on, because a posture of defiant denial is self-defeating. Specifically, I believe that you should have an association, with a board that might include prominent social psychologists from other field. The first mission of the board would be to organize an effort to examine the replicability of priming results, following a protocol that avoids the questions that have been raised and guarantees credibility among colleagues outside the field.

The following is just an example of such a protocol: * Assemble a group of five labs, where the leading investigators have an established reputation (tenure should perhaps be a requirement). Substantial labs with several students are the most desirable participants. * Each lab selects a recent demonstration of a priming effect, which they consider robust and most likely to replicate. * The board makes a public commitment to these five specific effects

2 of 6 3/27/2017 12:10 PM http://www.decisionsciencenews.com/2012/10/05/kahneman-on-the-stor...

* Set up a daisy chain of labs A-B-C-D-E-A, where each lab will replicate the study selected by its neighbor: B replicates A, C replicates B etc. * Have the replicating lab send someone to see how subjects are run (hence the emphasis on recency – the experiments should be in the active repertoire of the original lab, so that additional subjects can be run with confidence that the same procedure is followed). * Have the replicated lab send someone to vet the procedure of the replicating lab as it starts its work * Run enough subjects to guarantee power (probably more than in the original study) * Use technology (e.g. video) to ensure that every detail of the method is documented and can be copied by others. * Pre-commit to publish the results, letting the chips fall where they may, and make all data available for analysis by others.

This is something you could do quickly, and relatively cheaply. The main costs are 10 trips, and funds to cover these costs would be easy to get (I have checked). You would have to be careful in selecting laboratories and results to maximize credibility, and every step of the procedure should be open and documented. The unusually high openness to scrutiny may be annoying and even offensive, but it is a small price to pay for the big prize of restored credibility.

Success (say, replication of four of the five positive priming results) would immediately rehabilitate the field. Importantly, success would also provide an effective challenge to the adequacy of outsiders’ replications. A publicly announced and open effort would be credible among colleagues at large, because it would show that you are sufficiently confident in your results to take a risk.

More ambiguous results would be painful, of course, but they would still protect the reputations of scholars who sincerely believe in their work – even if they are sometimes wrong.

The protocol I outlined is just an example of something you might do. The main point of my letter is that you should do something, and that you must do it collectively. No single individual will be able to overcome the doubts, but if you act as a group and avoid defensiveness you will be credible.

In response to Ed Yong’s article in Nature in which Kahneman’s letter is published, Norbert Schwarz, has written a response, saying “There is no empirical evidence that work in this area is more or less replicable than work in other areas.”

REFERENCE Nobel laureate challenges psychologists to clean up their act

Related stories and links

From nature.com Uncertainty shrouds psychologist's resignation 12 July 2012 The data detective 03 July 2012 Replication studies: Bad copy 16 May 2012 Report finds massive fraud at Dutch universities 01 November 2011

Supplementary information

PDF files 1. Kahneman's letter (156K)

For the best commenting experience, please login or register as a user and agree to our Community Guidelines. You will be re-directed back to this page where you will see comments updating in real-time and have the ability to recommend comments to other users.

Comments for this thread are now closed.

2 of 4 3/27/2017 11:50 AM Nobel laureate challenges psychologists to clean up their act : Nature Ne... http://web.archive.org/web/20160201091357/http://www.nature.com/ne...

6 comments Go Subscribe to commentsClose 78 captures 1 Help Peter S • 2013-03-114 Oct 12 - 2210:32 Oct 16 AM 2015 2016 2017 The field of psychologists is rather hard to validate, isn't it? Scientific researches and tests have to be validated by other universities and institutes. What Daniel Kahneman suggests seems to make very much sense indeed. Open letters usually have a good effect and I certainly hope this one will have a big impact on the psychologists community. If only a few institutes/universities adopt and stick to the daisy chain, it makes less sense, but if the majority adopts it, it should be a great step forward! I plan to publish the open letter for the German community to spread the word in hope that other will find this suggestion useful to adopt.

Barlocia Marline • 2013-01-17 11:15 PM It's gene expression in gonadotropin releasing hormone neurosecretory cells of brain tissue that links sensory stimuli directly to changes in nutrient chemical-dependent species-specific social priming and behaviors. Ben LSW

Richard Plant • 2012-11-14 04:51 PM Readers may be unaware that we have published numerous articles over the last decade that would suggest that to large degree equipment error might be responsible for many unstable findings across the field (Plant et al 2002 onwards).

This is not a new problem in computer-based studies and counter intuitively one which is actually getting worse year-on-year. We hope shortly to be summarising this decade worth of work in a CABN special issue paper which should be due out next month. In the meantime readers are strongly advised to visit our website for an overview of the issues: http://www.blackboxtoolkit.com.

We will be presenting at SCiP and Psychonomics in Minneapolis in the next few days if readers would like to talk to us about the issues in person.

Hana Johnson • 2012-10-12 09:52 AM I remember an article in Science or Nature saying that the Chinese government started to clamp down on the hundreds of bogus "stem cell clinics" that offer whatever therapies they can convince people to pay good cash for. Not too different from placenta supplements or Mamushi Powder.

Brian Owens • 2012-10-05 12:19 PM Posted on behalf of Daniel Kahneman:

I write to complain about the irresponsible and damaging title that was affixed to Ed Yongâ€™s piece on October 3. The headline asserts that Nobel laureates challenges â€¦ to clean up their act. There is no challenge in my letter, and certainly not a challenge for anyone to â€ৌlean up their act.â€ Instead, I offered friendly advice to colleagues whose work I respect, about an image problem they face and how they might deal with it. The misleading title outraged many of my friends, and probably caused real damage by making it harder for priming researchers to address my suggestion. I would not have expected misleading headlines from â€œNature,â€ and hope you will be kind enough to publish this correction.

Regards, Daniel Kahneman

3 of 4 3/27/2017 11:50 AM Nobel laureate challenges psychologists to clean up their act : Nature Ne... http://web.archive.org/web/20160201091357/http://www.nature.com/ne...

Go Close

78 captures 1 4 Oct 12 - 22 Oct 16 2015 2016 2017 Help James Vaughn Kohl • 2012-10-04 03:10 AM From behavioral epigenetics it is now clearer that an environmental drive evolved from that of nutrient chemical ingestion in unicellular organisms to that of socialization in insects. The honeybee model organism exemplifies that fact. What the queen eats determines her pheromone production and everything else about the interactions in the colony. It is also clear that, in mammals, food odors and pheromones cause changes in hormones that have developmental affects on behavior in nutrient-dependent reproductively fit individuals.

Social scientists may want to include knowledge of the basic principles of biology and levels of biological organization in their studies of social priming. In all vertebrates, for example, the effects of social odors on hormones cause unconscious affects on the development of behavior. A sensory stimulus-> effect-> hormone-> affect approach could incorporate what is known about the requirement for gene, cell, tissue, organ, organ system reciprocity in adaptive evolution, and help to ensure that no missing steps in this critical path lead to results that cannot be replicated because the requirements for adaptive evolution of behavior via ecological, social, neurogenic, and socio-cognitive niche construction were not considered.

For example, social priming doesn't begin with a visual stimulus or a hormone like oxytocin in mammals. It begins with gene activation by a sensory stimulus that epigenetically effects intracellular signaling and stochastic gene expression. It's gene expression in gonadotropin releasing hormone neurosecretory cells of brain tissue that links sensory stimuli directly to changes in nutrient chemical- dependent species-specific social priming and behaviors — via downstream effects on other hormones in all vertebrates. Starting with something that affects behavior in individuals or groups from one species and expecting to find reproducible results at different developmental stages in another species virtually ensures missing the involvement of genetic predispositions and experience-dependent epigenetic effects.

Replication attempts are doomed to fail if only for the fact that each individual of each group tested has slightly different genetic predispositions and experiences that epigenetically effect hormones that affect behavior during development. No matter what measure of behavior is used, if a model for behavioral development is not used there's likely to be a problem with replication.

See other News & Comment articles from Nature

Nature ISSN 0028-0836 EISSN 1476-4687

4 of 4 3/27/2017 11:50 AM Response to Ed Yong https://docs.google.com/document/pub?id=1KKN5Gz35CPz6QmtpJ7-c...

Response to Ed Yong

Response to Ed Yong’s Questions Norbert Schwarz http://sitemaker.umich.edu/norbert.schwarz/home [email protected]

2 Oct 2012

Background

Ed Yong, a science journalist (http://notexactlyrocketscience.wordpress.com/), emailed questions in response to a call from Danny Kahneman that priming researchers should engage in a concerted effort to replicate findings. Ed’s questions are in italics. I share the answers in form of a public Google Document because they are longer than Ed can possibly use and I anticipate that there will be questions down the road about what I may or may not have said once parts of my response might be quoted.

Questions & Answers

EY: Do you think that suspicions about social priming are as strong as he suggests?

NS: Experiments are conducted to test theoretical predictions. No theoretical proposal stands or falls on the basis of a single, isolated finding. Instead, theoretical proposals are evaluated on the basis of a body of convergent findings and their compatibility with what else we know. Individual findings can provoke a rethinking of assumptions, but they are just one building block in a research program.

In his book “Thinking, fast and slow” Danny Kahneman has done a masterful job of reviewing and integrating the diverse findings that some people loosely refer to as “priming research” (knowledge accessibility effects, automaticity, fluency, and so on). As his book shows, there is a large body of converging findings from labs around the world, accumulated over almost four decades of peer-reviewed research published in an array of different journals. This work paints a coherent picture of the underlying processes that does not ride on any single individual finding. Researchers familiar with this literature are also familiar with the large number of conceptual, and sometimes exact, replications and the convergence documented in meta-analyses.

There is no empirical evidence that work in this area is more or less replicable than work in other areas. What distinguishes this work from other areas is solely that some of the findings are more surprising to lay people than findings in other domains. Unfortunately, the surprise value of the findings has sometimes been in the foreground of the publications (and has always been in the foreground of popular reports). This gave some particularly surprising individual findings an iconic status that far exceeds their empirical contribution to theory testing. It also focused the popular discussion on individual results and away from the convergence of a large body of evidence, including many findings that are not eye-catching, and the rather straightforward processes that underlie the surprising effects.

This created a context in which the concerns of a few sceptics, focused on one or two iconic findings, received more attention than either the critics’ slim empirical evidence or the relevance of the iconic findings warrants. You can think of this as psychology’s version of the climate change debate: Much as the consensus among the vast majority of climate researchers gets drowned out by a debate created by poorly supported and narrowly focused claims of a few persistent climate sceptics, the consensus of the vast majority of psychologists closely familiar with work in this area gets drowned out by claims of a few persistent “priming” sceptics. Their scepticism is based on isolated nonreplications of individual findings combined with a refusal to acknowledge the results of meta-analyses that count as conclusive evidence in any other area. Their critiques find attention because the findings they doubt are counterintuitive and of interest to a wide audience -- a failure to replicate a ten millisecond difference in a standard attention experiment would never be covered by you, Ed, or your colleagues. Hence, nonreplications in other domains of psychology rarely become the topic of public debate -- that people care in the case of “priming” studies is a tribute to those who put these phenomena on the map in the first place. While much remains to be learned about these phenomena, a response of broad doubt is incompatible with the available body of consistent evidence and its compatibility with related domains of knowledge (as Kahneman’s

1 of 2 3/27/2017 11:53 AM Response to Ed Yong https://docs.google.com/document/pub?id=1KKN5Gz35CPz6QmtpJ7-c...

“Thinking, fast and slow” documents).

EY: Would you agree with him that there is a "train wreck looming" and that priming researchers must take action to address the suspicions?

NS: If there is a “train wreck” looming, it is one of public perception, not one of the quality of the vast majority of the scientific work. The perceived “suspicion” far exceeds what critics’ supporting evidence might warrant. But as the climate change debate illustrates, the perceptions created by such debates are difficult to change through scientific evidence. Obviously, Danny Kahneman is more optimistic on this count than I am -- he thinks that the “suspicions” are unwarranted and that the perceptions can be corrected by the daisy-chain replications he suggests.

EY: Does his suggestion of a daisy-chain of labs carrying out replications make sense? Are you willing to take up the suggestion, and would others in the field do the same? If so/not, why?

NS: A daisy-chain of replications is an interesting idea and could provide information about the reliability of new results that is more quickly available than the results of meta-analyses. I will participate in such a daisy-chain if the field decides that it is something that should be implemented in a broader way. I will not participate in it when it is merely directed at one single area of research that happens to be the target of poorly supported “suspicions” voiced by critics who find a few isolated individual results implausible and ignore the majority of the available research. (Independent of this, I will obviously provide what is needed for others to replicate findings from my lab, but that’s not the point of the Kahneman proposal.)

Published by Google Drive – Report Abuse

2 of 2 3/27/2017 11:53 AM