In press, Psychological Inquiry 1

There’s Nothing Social about Social Priming: Derailing the “Train Wreck”

Jeffrey W. Sherman Andrew M. Rivers University of California, Davis University of British Columbia

Failures to replicate high-profile priming effects have raised questions about the reliability of priming phenomena. Studies at the center of the controversy have been labeled “social priming,” and have been viewed as a specific indictment of priming that is social in nature. However, an examination of this research shows that the relevant studies are not particularly social in nature. Moreover, other robust priming effects that are clearly social in nature do not “count” as social priming. Most importantly, use of the term “social priming” has obscured factors that do help to account for the relative reliability of priming effects. Here, we examine the construct of social priming, evaluate different possible characterizations of the research, describe some simple demonstrations that help clarify the nature of the controversy, and discuss future avenues for building a better understanding of priming. We conclude that the term “social priming” offers nothing in the way of understanding priming phenomena and should be laid to rest. We argue that it is time to move past arguments about the reliability of specific effects and shift our energy to building theories that help us better understand the mechanisms underlying priming effects and help to shed light on when, how, and why priming occurs.

In this paper, we examine the controversy challenging common critiques of social priming surrounding the replicability of “social priming” are obvious and that everyone already research. 1 We place the controversy in a understands them. Indeed, the issues are not historical context, review the construct of social complicated, and it is unlikely that critics of priming, evaluate alternative characterizations of social priming will believe they have learned the relevant literature, describe some simple anything from reading this article. Nevertheless, demonstrations that help clarify the nature of the in 2017, at the same time we were being controversy, and discuss future avenues of reassured, a number of analyses appeared in blogs research for better understanding priming and mainstream media that repeated the standard phenomena. criticisms without acknowledging these issues One could reasonably ask whether another (e.g., Engber, 2017; McCook, 2017; Schimmack, examination of social priming is really necessary. Heene, & Kesavan, 2017). Nor has the term Indeed, our own interest in publishing our disappeared from empirical academic research analysis had waned. Social priming has been (e.g., Gilder & Heerey, 2018; Meyer, 2019). In discussed widely, including in a special issue December 2019, Nature published a news feature wholly devoted to it in the journal Social on social priming (Chivers, 2019; Sherman & Cognition in 2014. Keith Payne and his Rivers, 2020), with an associated podcast that, colleagues made some of our central points in an once again, promoted the same flawed narrative excellent paper in 2016 (Payne, Brown-Iannuzzi, and commentary. That article included the & Loersch, 2016). Moreover, we have been greatest hits from the onset of the controversy, assured repeatedly that our main arguments in with a re-telling of Bem, Stapel, failed

1 We use the term “social priming” in recognition of the has no meaning. Wherever we include the term, readers popular usage of the term. We put quotation marks around should assume the quotation marks if they are not provided. the term to indicate our belief that the construct, as used, In press, Psychological Inquiry 2 replications, and Kahneman’s infamous letter. All of this contributed to the onset of an intense There was still no acknowledgement of simple period of reexamination of research practices definitional and methodological problems with among experimental psychologists, commonly the critique. This misrepresentation is not referred to as a “crisis of confidence” or a inconsequential. Beyond simply being wrong, it “supposed crisis of confidence,” depending on draws toward a handful of controversial one’s perspective (Spellman, 2015). A effects and away from developing a deeper particularly influential voice in instigating this theoretical understanding of priming. movement belonged to , the The Looming Train Wreck winning cognitive psychologist. Kahneman had described a number of priming So, how did we get here? How did social studies in his best-selling book Thinking Fast and priming become such a lightning rod of Slow (again, 2011), and was embarrassed that controversy? The year 2011 was eventful for researchers were having trouble replicating some scientific , particularly social of them. In 2012, he published an open letter to psychology. That year saw the publication in researchers working in what he referred to as the Journal of Personality and of “field” of social priming, warning of a “train Bem’s article claiming to show scientific support wreck looming,” and that the field had become for the operation of extrasensory the “poster child for doubts about the integrity of (Bem, 2011). Later that same year, the social psychological research” (Kahneman, 2012). This psychologist Diederik Stapel was found to have letter attracted considerable professional and engaged in research fraud on a massive scale. media attention, much of which consisted of Some of Stapel’s fraudulent publications reported critiques of social priming. A Google search of results from social priming studies. Also in 2011, “Kahneman Train Wreck” yielded more than Simmons, Nelson, and Simonsohn (2011) 350,000 results at the time of this writing, and the published their paper on the use of researcher letter has been cited more than 100 times in degrees of freedom to produce statistically academic journal articles. significant research results at will, highlighted by the absurd example of causing people to report a Aside from accelerating introspection about later birthdate after having been primed with the research practices, Kahneman’s letter had two Beatles song “When I’m 64.” This amusing other significant impacts. First, it identified social demonstration bore superficial resemblance to psychological research as a particularly troubled social priming studies. field. Given his stature, it is not surprising that Kahneman’s prophecy about the field becoming The following year, Doyen, Klein, Pichon, and a poster child was self-fulfilling. Second, in Cleeremans (2012) reported a failure to replicate calling into question the integrity of the research, a priming study published by social psychologists the letter contributed to the still-ongoing Bargh, Chen, and Burrows (1996), in which conflation of research fraud with reproducibility people primed with words associated with aging issues, particularly as it relates to social subsequently walked more slowly. The details psychology (e.g., Neuroskeptic, 2017; surrounding this replication effort received Wagenmakers, 2014). This conflation was considerable attention in an article by Ed Yong magnified by the Levelt Report (2012), which (Yong, 2012) that had been shared over 900 times summarized the official investigation into on social media by August, 2017 (Discover Stapel’s fraudulent activities. The Report stated magazine subsequently stopped publishing that “far more than was originally assumed, there figures on social media sharing) and has been are certain aspects of the discipline itself that cited in academic literature more than 30 times. should be deemed undesirable or even incorrect Subsequent failures to replicate other social from the perspective of academic standards and priming effects also were widely discussed (e.g., scientific integrity.” This elicited much protest Harris, Coburn, Rohrer, & Pashler, 2013; Shanks among social psychologists, including a rebuke et al., 2013). from the European of Social Psychology. In press, Psychological Inquiry 3

All of these events help to explain the fixation that created it. In 2018, Neuroskeptic on social priming, specifically, and social (Neuroskeptic, 2018) conducted a deep dive into psychology, more generally, as the Problem the origins of the term “social priming.” They Child of psychological science—the appearance identified chapters in an edited volume (Bargh, of Bem’s publication in a social psychological 2005; Trope and Fishbach, 2005) as the first journal, Stapel’s status as a social psychologist, “modern” uses of the term. However, in both Simmons et al.’s parody of social priming, the cases, the “social” in social priming was an commotion surrounding Doyen et al’s (2012) adjective describing the type of content that was failure to replicate Bargh et al. (1996), being primed. “Social” modified the noun Kahneman’s letter (2012), and the Levelt Report “priming,” but “social priming” was not its own (2012). Thing. In Neuroskeptic’s timeline, the first piece The Invention of Social Priming to name social priming as a Thing was a review article published in 2012 by Prabhakaran and Broadly, priming refers to the phenomenon Gray (2012), neither of whom are social whereby exposure to a influences psychologists (the scientists presumably subsequent behavior without conscious guidance conducting social priming research). This review or intention. Frequently, priming effects are described “the social priming paradigm,” but thought to result from the activation of mental failed to include some of the earliest studies and representations that facilitate or interfere with most widely used priming measures that might related subsequent behavior (Molden, 2014). have been labeled social priming. This was a Research traditions have utilized priming tasks to troubling omen of things to come. investigate many different theoretical questions of interest. Cognitive psychologists initially used Though Neuroskeptic’s etymological effort priming tasks to probe the organization of mental was constructive and informative, it failed to representations (e.g., Neely, 1991). Social identify what we (and we believe most social psychologists initially adapted priming psychologists) consider to be the true source of methodologies to understand how activated the term “social priming,” at least in terms of knowledge and evaluations influence perception introducing it to a broad audience. The source for and behavior (e.g., Fazio, Sanbonmatsu, Powell, most of us was the open letter Kahneman wrote & Kardes, 1986). At a more granular level, in the fall of 2012 that identified social priming different areas of investigation have developed as a “field” that was particularly problematic. At unique paradigms to elicit priming effects and the time, there was considerable puzzlement have generated a plethora of theoretical models to among social psychologists, including those explain those effects. conducting the so-called social priming research, as to the meaning and origin of the term “social Across research traditions, priming involves priming.” The senior author of the present paper exposing participants to stimuli–or ‘primes’–that had been Editor at the journal are incidental to, and yet still influence, for 7 years at that time, and had never subsequent behavior. For example, work with the encountered the term. Likewise, many social (Meyer & Schvaneveldt, psychologists working in the broad field of social 1971) demonstrates that people are faster to cognition, who were quite familiar with the correctly identify words (e.g., ‘doctor’) following relevant studies, shrugged confusedly. Whatever the presentation of a related prime word (e.g., the exact timeline of who said what and when, a ‘nurse’) compared to an unrelated prime word few things are clear. First, at the time of (e.g., ‘paper’). In other words, the target behavior Kahneman’s letter, social priming was not a – correctly indicating that ‘doctor’ is a word – recognized specific methodology or paradigm, occurs more quickly when related, incidental never mind a field, as he had declared. Second, stimuli are observed than when unrelated stimuli the idea of social priming as a Thing was not are observed. invented by the social psychologists largely doing What, then, is social priming? The truth is that and reading the work. Third, though the idea it did not exist as a Thing until the controversy remains popular (the online title of Chivers’ 2019 In press, Psychological Inquiry 4 piece in Nature refers to the “field of social these measures produce robust results that have priming”), the term still has not been defined to been widely replicated (Cameron, Brown- anyone’s satisfaction. Iannuzzi, & Payne, 2012). Upon being What Counts as Social Priming? confronted with this large body of robust priming effects in the social psychological literature, A major impediment to evaluating social many critics of social priming declared that, for a priming is that there is no clear definition of what variety of reasons, these effects did not count as it is. Most obviously, it would seem that social social priming. This makes clear that social priming should involve priming studies that priming is not simply priming research using include stimuli and/or responses related to social social content or priming research conducted by cognition and behavior. That is the spirit in which social psychologists. Obviously, this Bargh and Trope/Fishbach had used “social” as a contradiction introduces a major complication for modifier of “priming.” However, it was a broad critique of a so-called paradigm or field immediately apparent that the inclusion of social of social priming. content was neither necessary nor sufficient for acquiring the label of social priming as Behavioral Priming? envisioned by Kahneman and other critics of the The most commonly stated basis for excluding research. Even the archetypal social priming these effects from social priming is that they do paper (Bargh et al., 1996) fails to meet this not examine the influence of primes on behavior. criterion. There is nothing particularly social In this view, there is an important qualitative about the knowledge of a relationship between difference between the button presses in these aging and walking speed or the notion that the paradigms and the behaviors reflected in social salience of that knowledge could affect actual priming studies (e.g., Pashler, Rohrer, & Harris, walking speed. 2013; Weingarten et al., 2016). To reflect this Perhaps more revealing than what is counted distinction, many researchers adopted the term as social priming is what isn’t counted. There are “behavioral priming” to refer to the studies huge bodies of research on the social psychology identified as social priming (e.g., Doyen et al., of attitudes, prejudice, and stereotyping that rely 2012; Wagenmakers, 2014). Though this tacit on priming paradigms. The Evaluative Priming recognition of robust priming effects in the Task (Fazio, Jackson, Dunton, & Williams, 1995) broader social psychological literature was and the Weapons Identification Task (Payne, welcome, the conversation surrounding the 2001) rely on standard sequential priming supposed problems with “social priming” designs. Other frequently used indirect measures continues unabated (e.g., Chivers, 2019; Engber, of attitudes and intergroup bias, such as the 2017; McCook, 2017; Schimmack, et al., 2017). Implicit Association Test (Greenwald McGhee, Often, the terms “social priming” and “behavioral & Schwartz, 1998), the Go/No-Go Association priming” are used interchangeably, particularly Task (Nosek & Banaji, 2001), the First Person outside of academic publishing, among authors Shooter Task (Correll, Park, Judd, & who have accurately detected that the two terms Wittenbrink, 2002), the Affect Misattribution refer to the same body of research (e.g., Chivers, Procedure (Payne, Cheng, Govorun, & Stewart, 2019). 2005), and the Misperception Task Rather than clarifying the contradictions (Krieglmeyer & Sherman, 2012) rely on the same inherent in the usage of the term “social priming,” underlying logic of priming information in the term “behavioral priming” introduces a new prior to measuring some associated of conceptual problems. What constitutes behavioral response. All of these measures are “behavior?” Which behaviors count? Pressing clearly priming tasks, they were created by social buttons on computer keyboards to judge the psychologists, and they were first reported in identity or valence of a target word or image is social psychology journals. In fact, there are certainly behavior. What if the buttons being orders of magnitude more published articles pushed are meant to represent shooting a gun using these measures than the sorts described and (e.g., Correll et al., 2002)? What about using critiqued by Kahneman and others. Moreover, mouse clicks to choose gambles (Payne et al., In press, Psychological Inquiry 5

2016)? Judging the pleasantness of Chinese research designated as social priming/not social symbols (Payne et al., 2005) or the threat level of priming. In particular, there are many examples ambiguous faces (Krieglmeyer & Sherman, of long-term priming that are not classified as 2012)? Are these behaviors qualitatively different social priming (e.g., Becker, Moscovitch, kinds of behaviors than circling a number on a Behrmann, & Joordens, 1997; Hughes & rating scale (Srull & Wyer, 1979), filling in Whittlesea, 2003; Sherman & Jordan, 2011; Tse responses on a multiple-choice test of general & Neely, 2005; 2007; Woltz & Was, 2006; 2007). knowledge (Dijksterhuis & van Knippenberg, The Role of Experimental Design 1998), rating the humor of cartoons (Strack, Martin, & Stepper, 1988) or completing word To this point, we have determined that social fragments (Zhong & Liljenquist, 2006), all of priming is not defined by the presence of social which are measures in studies commonly content in a priming task, the home discipline of recognized as bone fide social priming? The point the researchers, or the length of time between is that, just as no one ever provided a clear prime and target. We also have argued that type definition of social priming, no one has provided of behavior does not provide a coherent basis a coherent definition of behavioral priming. upon which to distinguish studies that have been There is no theoretical rationale for including declared social priming/not social priming. What, some behaviors and excluding others. Words then, best characterizes the distinction between such as “broad,” “complex,” and “higher-order” studies that have been included or excluded as are invoked, but they do not seem apt for the examples of social priming? behavioral measures from social priming studies In our reading of the literature, the feature that described above. We have yet to see any seems to most closely covary with the social/non- semblance of a cogent explanation of exactly social priming distinction is the status of a study when, why, and how a behavior crosses the as using a within- versus between-subjects threshold from being just a behavior to being a design. Specifically, the studies identified as Behavior. As such, the behavior/non-behavior social priming, whether they include social divide cannot capture the distinction between content or not and regardless of the gap between social psychological priming research that has prime and target, almost uniformly use between- been admitted and excluded from the category of subjects designs, whereas studies considered to social priming. We will return to this issue below. exemplify non-social priming almost exclusively 2 Long-term versus Short-term Priming use within-subjects designs. Wentura and Rothermund (2014) recognized In between-subjects designs, each participant an important distinction between priming receives a single type of prime, whereas in paradigms that separate prime and target by within-subjects designs, each participant milliseconds versus paradigms that separate them responds to all prime-target combinations, by multiple seconds or longer. This may serve as usually over hundreds of trials. All else being another basis for excluding robust priming equal, between-subjects designs offer paradigms in social psychology from counting as substantially less statistical power than within- social priming (Wentura and Rothermund do not subject designs to detect the same effect. During advocate this use of the distinction). Many the period in which much of the initial social paradigms that employ very brief gaps between priming research was conducted, little attention prime and target have been demonstrated to be was paid to selecting sample sizes appropriate to robust and replicable, whereas the controversial study design. As such, the between-subjects paradigms labeled as social priming generally social priming studies were frequently severely impose much longer gaps between prime and under-powered. For example, Bargh et al.’s target. Yet, the long-term/short-term distinction (1996) elderly-priming studies used a fully also does not capture the separation between between-subjects design with two conditions and

2 We use the term “almost exclusively” to acknowledge the consensually regarded as social priming that use a within- fact that we are not familiar with every relevant published subjects design. study. However, we are not aware of any studies In press, Psychological Inquiry 6 one critical measurement per participant. Those type of priming effects he later labeled as “social studies sampled from 15 participants per priming” and lamented as unreliable. Not all of condition. This would require an assumed effect the 29 social priming effects described in that size of ds = 1.06 to achieve Cohen’s (1988) chapter (see Schimmack et al., 2017) are recommended level of power, 1-β = 0.8. particularly social in nature. However, all 29 of According to Cohen’s descriptions, this effect them used a fully between-subjects design. would be ‘grossly perceptible.’ Given the subtlety of Bargh et al.’s (1996) manipulation, such an effect size seems unlikely. Illustrative Demonstrations Underpowered studies result in lower positive That between-subjects designs offer less predictive value (PPV); that is, the probability statistical power than within-subjects designs is a that a 'positive' research finding reflects a true fact that is taught in introductory research effect (Button et al., 2013). Moreover, positive methods classes. One might think that the role of results from low PPV studies are likely to provide this factor in the reproducibility of priming exaggerated estimates of effect size. Thus, studies effects would be readily recognized. That has not with low PPV are less likely to subsequently been the case, from Kahneman’s initial failure to replicate. In turn, selective publication policies recognize this feature to recent commentary. favor the publication of positive research Shanks (2017) argued that consideration of the findings. Thus, to the extent that published social within- versus between-subject nature of priming studies were relatively more likely to be different priming studies “sheds minimal light on under-powered compared to other published the priming controversy” (p. 1221). priming studies, they also would be relatively less To make the influence of design type on likely to subsequently replicate. It is not priming results more concrete, we conducted surprising that the most controversial and some simple illustrative studies. We selected two difficult to replicate priming effects have used robust priming paradigms that employ within- between-subjects designs and were inadequately- subjects designs from both the cognitive and powered (e.g., Bargh et al., 1996; Dijksterhuis & social psychology literatures, and showed that van Knippenberg, 1998). In contrast, the highly those robust effects were much more difficult to replicable priming effects in social psychology obtain and yielded less precise effect size described above employ within-subjects designs estimates in a between- than within-subjects in studies that are well-powered. Clearly, it is not analysis. From the cognitive literature, we chose the social versus non-social nature of the priming the Lexical Decision Task (LDT; Meyer & or the behavioral versus non-behavioral nature of Schvaneveldt, 1971) and the Stroop Task (Stroop, the measures that is relevant in this comparison. 1935).3 From the social literature, we chose the Simply, adequately powered and published Weapons Identification Task (WIT; Payne, 2001) priming results are more likely to replicate than and the Stereotype Misperception Task (SMT; inadequately powered and published priming Krieglmeyer & Sherman, 2012). For each effect, results. we analyzed full data sets in two different ways. Let us return to the origin of the term “social First, we conducted the standard within-subjects priming:” Kahneman’s embarrassment at having analysis, based on participants’ responses on all described priming studies that were difficult to trial types (and, thus, all prime-target replicate. In the fourth chapter of Thinking Fast combinations). In the second approach, each and Slow (Kahneman, 2011), he describes the participant was assigned to one level of the

3 We recognize that not everyone considers the Stroop sometimes implemented with an SOA of 0-ms (e.g., effect to be a priming effect. We believe that it meets the Gawronski & Ye, 2013; Musch & Klauer, 1997). As well, definition in that it captures the unintended influence of a researchers have implemented SOAs greater than 0-ms stimulus on associated behavior (i.e., the word and its ink (essentially turning Stroop into a sequential priming task) color). Some have argued that the Stroop Task is not a to examine Stroop effects (Glaser and Glaser, 1982; Logan priming task because there is no gap between prime (word 1980). In any case, whether you agree that Stroop is meaning) and target (color naming). That is, there is an priming or not, the methodological point is unchanged. SOA of 0-ms. However, other priming effects are In press, Psychological Inquiry 7 experimental design (and, thus, one prime-target What We are not Saying combination) based on the first experimental trial Here, we address some potential sources of to which they had been randomly exposed. In misunderstanding about exactly what we are and essence, this analysis treated the task as a single are not claiming. First, we are not claiming that trial between-subjects design (for details, see between-subjects designs necessarily have less Rivers & Sherman, 2018). power than within-subjects designs. Of course, it The results were clear. As expected, in the full is possible to adequately power between-subjects within-subjects analyses, all four tasks replicated designs (e.g., by collecting additional previous work, demonstrating robust and observations). The simple reality is that, at the statistically significant priming effects. In time of the publication of the most noteworthy contrast, in the between-subjects analysis, though social priming studies, sample size norms often all results were in the expected direction, only the left between-subjects designs under-powered. All WIT attained (barely) traditional statistical else being equal, the effects demonstrated in significance (p = .048). those studies would be subsequently less likely to The confidence intervals for the effect sizes replicate than priming studies published with also differed substantially across the analyses. within-subjects designs. The average span of 95% confidence intervals for One might fairly wonder why we focus on the the Hedges g effect sizes obtained using the variable of design type rather than power, per se, within- and between-subjects analyses as it is the direct cause of effect size mis- respectively were .22 versus .75. Simply put, estimation and publication of false positives. The within-subjects designs yielded confidence answer is that it is a simple matter to classify intervals that spanned a range that was one-third priming studies in terms of research design. It is of the size of those from between-subjects not such a simple matter to classify them in terms designs. of power. We know that social priming studies Based on the effect size estimates from the full differ from other priming studies in their almost within-subjects analyses, we estimated how many exclusive use of between-subjects designs. We participants would be needed to achieve power = cannot definitively state that power covaries .8 in both within- and between-subjects designs. perfectly with the distinction between social and non-social priming. It is safe to assume that, in For the LDT (dz = .649), a within-subjects design would require 17 participants and a between- general, between-subjects priming studies were subjects design would require 602 participants. less likely to be adequately powered than within- subjects priming studies prior to the “crisis of For the Stroop Task (dz = .565), a within-subjects design would require 21 participants and a confidence,” which moved the power issue to the between-subjects design would require 404 fore. And, in many cases, it can be shown that between-subjects priming effects would have had participants. For the WIT (dz = .309), a within- subjects design would require 67 participants and to assume unrealistically large effect sizes to have a between-subjects design would require 102 been adequately powered (e.g., Bargh et al., 1996). participants. Finally, for the SMT (dz = .292), a within-subjects design would require 74 The results of our demonstration studies do not participants and a between-subjects design would show that failures to replicate social priming require 792 participants. effects can be solely attributed to research design, Rudimentary facts about the fixed relationship and we make no such claim. Likewise, we make between design type and power apply in the no claim that social priming effects would all context of priming research just as they do for replicate in appropriately powered studies, other research domains. With equal sample sizes, whether between- or within-subjects. There are between-subjects designs yield relatively noisy several examples of research teams carrying out effect size estimates, are less likely to detect true high-powered replications of between-subjects effects, and are more likely to yield false positive social priming effects that failed to find evidence results. for the effects. For example, Gomes and McCullough (2015) were unable to replicate the In press, Psychological Inquiry 8 effect of religious priming on decisions in an of within-subjects approaches in naturalistic economic game despite running 455 participants experiments. Many social issues of critical across two critical between-subjects conditions. importance are likely only possible to study using Similarly, Shanks and colleagues (2015), across between-subjects designs and analyses. We do 9 separate experiments (N = 1,325), failed to find not wish to suggest that psychologists abandon evidence consistent with previous work showing the study of important psychological principles or that priming mating motives affects people’s of naturalistic behavior simply because within- spending behavior. Shanks et al. (2015) subjects approaches are not possible. Instead, specifically had at least 80% power to detect a researchers employing between-subjects designs between-subjects effect as small as ds = .14. This should seek to maximize statistical power highly powered failure to replicate in tandem through all means available (e.g., Chartier et al., with demonstrable evidence of 2017; Judd, Westfall, & Kenny, 2017; Wang, indicates that the reported effect – priming Sparks, Gonzalez, Hess, & Ledgerwood, 2017). mating motives influences consumer behavior – The Fallout is most likely a Type I error. Had Kahneman and subsequent critics We do not assume that social priming effects concluded that underpowered published research that have not been tested with appropriately is unreliable, regardless of its field of origin, very powered designs are “real” effects. Absent such few people would have noticed or cared. tests, the existing data are simply uninformative. However, in associating such work with social Even in the case of Bargh et al. (1996), the data psychology and in carelessly tying the work to are not conclusive. If we assume that the effect questions about research integrity, a firestorm size of Bargh et al.’s (1996) elderly study is ensued. He framed his letter as an attempt to comparable to a simple average of the four protect young researchers from the harm of being within-subject priming effect sizes we observed associated with a “controversial and suspicious” in our illustrative research (a very optimistic field. In truth, the letter did much more damage estimate in our view), then one would require at than any failed replication. Social priming least 388 participants over 2 between-subjects researchers and social psychologists, more conditions to achieve the recommended .80 level broadly, were hardly alone in paying insufficient of power. If we combine all reported data attention to statistical power in those days. It had available on Curate Science (Bargh et al., 1996; been a long-standing epidemic across much of Cesario, Plaks, & Higgins, 2006; Doyen et al., scientific psychology (see Cohen, 1962; 2012; Hull, Slone, Meteyer, & Matthews, 2002; Gigerenzer, 2018). While failing to recognize this Pashler, Harris, & Coburn, 2008), the resulting fundamental problem, Kahneman saddled the sample (N = 447) would provide the entire field with doubts about integrity. recommended level of power for a single Administrations, departments, and individual experimental test of the hypothesis. Given the scientists who knew little or nothing about social available evidence from the initially reported psychology, not to mention priming research in effect and subsequent failures to replicate, it social psychology, suddenly had strong doubts seems that, at best, the effect must be about the field and its researchers. It is difficult to substantially smaller than initially reported. Still, quantify the damage, but few would argue that the the exact status of this particular finding is not yet episode didn’t impact publishing, grant funding, entirely clear. and employment for social psychologists. Finally, we want to be clear that we are not Perhaps Kahneman’s initial oversight of suggesting that statistical power should be research design issues was innocent enough, but prioritized above developing psychological it is difficult to extend the same good will to some theory or above the study of behavior in natural of the subsequent critiques. Responses to contexts. There are a multitude of important Kahneman’s letter had brought the design issue theoretical questions that cannot be answered to the attention of interested parties, many of using within-subjects approaches and, similarly, there are many difficulties that preclude the use In press, Psychological Inquiry 9 whom proceeded to studiously ignore it. 4 It is broad theoretical models to account for when and clear that some of what followed reflected the how priming works (e.g., Cesario, Plaks, settling of long-standing grievances about the Hagiwara, Navarrete, & Higgins, 2010; Loersch attention paid to some of the social priming & Payne, 2011; Schröder & Thagard, 2013; effects (e.g., Dominus, 2017; Zwaan, 2013; see Wheeler, Demarree, & Petty, 2007). This was Funder, 2020 for a candid discussion). The exactly the sort of work that critics of social schadenfreude following replication failures of priming, both within and outside of social the research was (and is) thick on social media psychology, believed should be getting more and in conference hotel bars. attention. However, the impact of this work was There are a number of ironies in all of this. The blunted as it was crowded out by controversy social priming work that has received the most over a small handful of effects. attention reports effects that are eye-catching and The Future: Moderation of Priming counter-intuitive. It is the kind of sexy research Moving forward, we concur with many others that popular science writers love to describe, and that the focus should shift from effect-driven they were increasingly doing so in the years debates to theory building and testing (e.g., preceding 2011. Likely for the same reasons, Doyen, Klein, Simons, & Cleeremans, 2014; these are the studies that Kahneman (2011) chose Higgins & Eitam, 2014; Molden, 2014). One to share with his readers in his chapter about aspect of such an effort would be to search for priming. The work was attaining great notoriety. important moderators of priming effects. There This led to some professional resentment, are many differences among priming studies that particularly among researchers who felt that there may influence their reproducibility. We are not was an unhealthy focus on flashy effects to the aware of systematic attempts to identify and detriment of theoretical acuity. One ironic aspect characterize these moderators. Certainly, no one of this is that the same concerns were every bit as has systematically studied the impact of content prevalent among social psychologists as they type or researcher home field (e.g., social vs. non- were in other fields. There was no shortage of social) while controlling for other variables, such researchers who questioned the reliability of the as statistical power. We see no a priori theoretical findings and frowned upon the attention given to basis for predicting such moderation effects. In those effects. However, the experience of these contrast, length of delay between prime and target skeptics changed dramatically when Kahneman is a potentially important moderator that can be threw the whole field of social psychology under readily examined. Every theory of priming the bus, not only with respect to reproducibility, predicts that the effects weaken as the gap but also with insinuations of outright fraud. between prime and target increases. However, we Another ironic aspect of the situation is that the are not aware of systematic attempts to specify criticism only increased the focus on the flashy these relationships (for exceptions, see Hermans, findings at the center of the controversy. De Houwer, & Eelen, 2001; Rivers & Sherman, Tremendous amounts of effort, ink, and bile were 2020). spilled fighting about the reproducibility of a Dividing studies into behavioral and non- small handful of effects that were not and are still behavioral research is conceptually problematic, not representative of the research (priming or as described above. To begin, someone would otherwise) being conducted by social need to develop a coherent and empirically psychologists. In the grand scheme of things, validated taxonomy of types of behavior. One whether those specific effects replicate or not is possibly important distinction concerns the extent not of great consequence to the development of of constraint on the measured behavior. The psychological science. In the midst of the robust priming effects typically examined in publication of and furor over these studies, a within-subjects design largely involve two- number of researchers were busy developing alternative forced choice tasks. For example,

4 Kahneman eventually came to recognize the problem as the public record with respect to the “social” component of one of statistical power (2017). However, he did not correct “social priming.” In press, Psychological Inquiry 10 words must be judged as new or old in memory We don’t know if behavioral constraint will tasks; strings of letters must be judged as words ultimately prove to be an important determinant or not in the LDT; words must be evaluated as of priming, but it represents the kind of moderator good or bad in evaluative priming; objects must that should be the focus of future research in that be judged as a gun or tool in the WIT, and so it offers a theoretical basis for expecting forth. In contrast, the measures in studies moderation. Moderators that are post-hoc or that identified as social priming are not constrained to are merely descriptive offer little in the way of two options. Walking speed, for example, is a theoretical advance. continuous behavior. Trait judgments along Mechanisms of Priming scales offer more than two options (e.g., Srull & Wyer, 1979). Even performance on a multiple Future research should also focus on the choice knowledge test offers more than two mechanisms that produce priming effects. This responses (e.g., Dijksterhuis & van Knippenberg, has been the primary aim of recent theoretical 1998). There certainly are reliable priming effects work (e.g., Cesario et al., 2010; Loersch & Payne, with more than two behavioral options (e.g., 2011; Schröder & Thagard, 2013; Wheeler et al., Coles, Larsen, & Lench, 2019; Strack et al., 2007). One increasingly common means of 1988), but it may be that the robustness of developing and testing theoretical models of priming effects is inversely related to the priming is via the use of formal mathematical constraints on the measured behavior. If the models of behavior. Many such models have been behavior requires a choice between only two proposed to identify and quantify the processes options, the influence of a prime may be that account for priming effects on memory and sufficient to tip the behavior one way or the other. judgment. These models also specify the ways in In contrast, with less constrained behavior, the which the component processes interact and influence of a prime may be insufficient to guide constrain one another in producing outcomes. In the behavior among multiple or continuous this way, the models are, in fact, specified options. Perhaps this is akin to the fan effect in theories of the mechanisms underlying priming memory, in which the likelihood of accurately phenomena. The most commonly used types of recognizing any particular item is inversely models in social (and non-social) psychological related to the number of items learned (Anderson, research are signal detection (e.g., Correll et al., 1974). As the number of items learned increases, 2002; Green & Swets, 1966), process dissociation the full extent of activation from a recognition (e.g., Jacoby, 1991; Payne, 2001), mutinomial cue must be spread among increasing numbers of processing trees (e.g., Krieglmeyer & Sherman, items, leading each item to receive less total 2012; Meissner & Rothermund, 2013; Payne, activation (and likelihood of being accurately Hall, Cameron, & Bishara, 2010; Riefer & recognized). Batchelder, 1988; Sherman et al., 2008), and drift diffusion models (e.g., Klauer, Voss, Schmitz, & Alternatively, one could make a sensible Teige-Mocigemba, 2007; Pleskac, Cesario, & prediction of exactly the opposite relationship Johnson, 2018; Ratcliff, 1978). between priming strength and behavioral constraint. Specifically, one might argue that the A comprehensive review of the use of more constrained the behavior, the less room mathematical models to study priming is beyond there is for the behavior to be pushed around by a the scope of this article (see Calanchini, Rivers, prime. Deciding whether a stimulus is a Klauer, & Sherman, 2018; Sherman, Klauer, & word/non-word, positive/negative, etc. is easy. Allen, 2010), but we do wish to offer an example Generally, few errors are made on two-alternative from our own research. Recently, we have been forced choice tasks. If a stimulus is obviously a using multinomial models to investigate both the word or is obviously negative, there may not be moderators and mechanisms of priming effects. sufficient opportunity for the influence of a This particular work has been conducted in the prime. However, less constrained behaviors, such context of the Stereotype Misperception Task as walking or making judgments along a 7-point (SMT; Krieglmeyer & Sherman, 2012). In this scale, may be more open to external influence. task, participants are asked to form impressions of people whose faces are shown in blurred In press, Psychological Inquiry 11 drawings. They are asked to judge the faces as performance, whereas the AMP’s misattribution high or low on a chosen target trait. Immediately model reliably offers a better account of AMP preceding the presentation of the target faces, performance. There are a number of differences participants are exposed to prime faces. They are between the tasks, and why different processes instructed to pay attention to the prime faces, but seem to account for performance on the two tasks not respond to them. Any combination of prime will be an interesting question for future research. types and target traits may be paired. In our In the meantime, we wondered if observed studies, the primes are pictures of Black or White priming could be driven by different processes males and targets are to be judged as high or low depending on variations in the implementation of in threateningness. Though the target faces are the priming task. Specifically, Loersch and Payne blurry and ambiguous, there are, in fact, (2011) argued that any variable that increases objectively high and low threat versions of the confusion between reactions to a prime and same targets. As such, on each trial there is a reactions to a target should increase the correct and incorrect response. The standard likelihood that reactions to the prime will be SMT priming effect is that the targets are more misattributed as coming from the target. For likely to be judged as threatening when they are example, the shorter the delay between prime and preceded by Black face primes than White face target, the more likely a respondent is to confuse primes. reactions to the prime as having come from the As part of the initial development and target. Source misattribution should also increase validation of the task, Krieglmeyer and Sherman to the extent that the signal coming from the (2012) developed a multinomial model to account target is ambiguous. The more ambiguous the for performance and to measure the processes target, the more readily reactions to it may be that contribute to it. Without going into great affected by misattributed reactions to the prime. detail, the model proposes that the primes may or In a series of studies, we tested both of these may not activate that may or may not potential moderators of priming. be corrected in judging the targets. When In one study, we manipulated the time that stereotypes are not activated, responses are separated the offset of prime images and the onset determined by the ability to detect the correct of target images. We developed three different responses or response bias. When the activated conditions; one in which the target appeared stereotype (i.e., Black men are threatening) is directly after prime-offset, 50-ms after prime- inconsistent with judging a non-threatening offset, or 175-ms after prime-offset. We target, the conflicting responses must be resolved. replicated the basic priming effect in all cases. One of the interesting aspects of the SMT is Model fitting showed that the SMT response that it bears close resemblance to the Affect conflict model and the AMP misattribution model Misattribution Procedure (AMP) developed by were indistinguishable in terms of model fit Payne et al. (2005), in which participants judge indices when prime and target appeared with no the pleasantness of Chinese pictographs time separation. However, with each increase in following the presentation of Black and White separation, the advantage of the SMT model prime faces. Payne et al. proposed that the AMP increased. At both 50- and 175-ms delay, the effect occurs when positive/negative evaluations SMT model provided significantly better fit to the of the prime pictures are misattributed to feelings data than the AMP model. Thus, as prime-target about pictographs and developed a multinomial separation increased, so too did the extent to model to measure this process (Payne et al. 2010). which the response conflict process model better Thus, despite the structural similarities between accounted for priming than did the misattribution the SMT and AMP, responses on the tasks are process model. If the mechanisms underlying proposed to arise from different processes; priming effects shift within a few hundred specifically, response conflict versus milliseconds, it is a safe bet that the longer gaps misattribution. And, in fact, our own research has frequently used in social priming studies will also shown that the SMT’s response-conflict model involve a variety of different psychological reliably offers a better account of SMT mechanisms. In press, Psychological Inquiry 12

In another study, we manipulated the level of As it stands, the term “social priming” offers ambiguity in the target faces. In one condition, nothing in the way of explaining the relative the high and low threat targets were 3 SDs above robustness of different priming effects. and below a mean level of threat. In the other Moreover, use of that term has obscured condition, the high and low threat targets were, in important features that do help to account for the fact, equal in objective level of threat. Once relative robustness of priming effects. The studies again, we replicated the basic priming effect both discussed and examined as cases of social when targets were unambiguous and when targets priming are best described as priming studies that were fully ambiguous. Whereas the SMT process employ between-subjects designs. The content of model provided significantly better fit when the the primes and targets is not necessarily social. targets were unambiguous, the fits of the two As well, priming research with clearly social models were indistinguishable when the targets content is not counted as social priming when it were ambiguous. Thus, increasing target uses within-subjects designs. Likewise, priming ambiguity reduced the extent to which response research conducted by social psychologists is not conflict processes better accounted for priming described as social priming research when it uses than misattribution. within-subjects designs. There are reasons why We provide these as examples of the kinds of social psychological priming research may be questions about priming moderation and more likely to rely on between-subjects designs mechanism that we view as useful for developing than priming research in other fields. In a deeper theoretical understanding of when, how, particular, concerns about carryover effects in and why priming effects occur. Modeling is within-subjects designs may be relatively more certainly not the only way to conduct such common in social psychological research (see investigations, and other strategies may be Greenwald, 1976). Nevertheless, a large majority equally or more effective. of priming research in social psychology has used within-subjects designs that have yielded reliable Conclusion results. Does it matter how we characterize the It is well past time to finally put to rest the label research that has been labeled “social priming?” “social priming.” It also is past time to move We think that it does. First and foremost, if we beyond debates about specific findings. We want to improve our research practices so that our should have more ambitious goals than merely work may be more reliable, we need to accurately testing the reliability of particular effects. We identify the reasons why research is more or less should aim for a broader theoretical robust. To the extent that we misidentify and understanding of the moderators and mechanisms misunderstand the causes of unreliable research, underlying priming in order to better predict scientific progress will slow. In learning the when, why, and how priming occurs. wrong lessons, we also may fail to learn the right ones. References Anderson, J. R. (1974). Retrieval of propositional Beyond the issue of accuracy, how we account information from long-term memory. Cognitive for variations in the reproducibility of research Psychology, 5, 451–474. has real implications for real people and the work Bargh, J. A. (2005). Bypassing the will: Toward that they do. As a rule, we would suggest that demystifying the nonconscious control of social commenters should be cautious about ascribing behavior. In R. R. Hassin, J. S. Uleman, & J. A. Bargh (Eds.), The New Unconscious (pp. 37-60). Oxford: specific systematic causes for replication failure Oxford University Press. in the absence of thorough empirical Bargh, J. A., Chen, M., & Burrows, L. (1996). examination. Certainly, unsubstantiated of social behavior: Direct effects of trait construct and insinuations of unethical behavior are uncalled stereotype activation on action. Journal of Personality and Social Psychology, 71, 230-244. for, particularly when there are more mundane Becker, S., Moscovitch, M., Behrmann, M., & Joordens, S. explanations for failures to replicate. Such (1997). Long-term semantic priming: A computational insinuations are also unethical. account and empirical evidence. Journal of Experimental Psychology: Learning, Memory, and Cognition, 23, 1059–1082. In press, Psychological Inquiry 13

Bem, D. J. (2011). Feeling the future: Experimental evidence https://slate.com/health-and-science/2017/06/daryl-bem- for anomalous retroactive influences on cognition and proved-esp-is-real-showed-science-is-broken.html affect. Journal of Personality and Social Psychology, Fazio, R. H., Jackson, J. R., Dunton, B. C., & Williams, C. 100, 407–425. J. (1995). Variability in automatic activation as an Button, K., Ioannidis, J., Mokrysz, C., Nosek, B. A., Flint, unobtrusive measure of racial attitudes: A bona fide J., Robinson, E. S. J., & Munafo, M. R. (2013). Power pipeline? Journal of Personality and Social Psychology, failure: Why small sample size undermines the reliability 69, 1013–1027. of neuroscience. Nature Reviews Neuroscience, 14, 365– Fazio, R. H., Sanbonmatsu, D. M., Powell, M. C., & Kardes, 376. F. R. (1986). On the automatic activation of attitudes. Calanchini, J., Rivers, A. M., Klauer, K. C., & Sherman, J. Journal of Personality and Social Psychology, 50, 229- W. (2018). Multinomial processing trees as theoretical 238. bridges between cognitive and social psychology. Funder, D. C. (2020, May 8). Thoughts on “Ego Depletion” Psychology of Learning and , 69, 39-65. and some related issues concerning replication. Cameron, C. D., Brown-Iannuzzi, J. L., & Payne, B. K. Funderstorms. Accessed at: (2012). Sequential priming measures of implicit social https://funderstorms.wordpress.com/2020/05/08/thought cognition: A meta-analysis of associations with behavior s-on-ego-depletion-and-some-related-issues-concerning- and explicit attitudes. Personality and Social Psychology replication/ Review, 16, 330–350. Gawronski, B., & Ye, Y. (2014). What drives priming Cesario, J., Plaks, J. E., Hagiwara, N., Navarrete, C. D., & effects in the affect misattribution procedure? Personality Higgins, E. T. (2010). The Ecology of Automaticity: and Social Psychology Bulletin, 40, 3-15. How Situational Contingencies Shape Action Gigerenzer, G. (2018). Statistical rituals: The replication and Social Behavior. Psychological Science, 21, 1311– delusion and how we got there. Advances in Methods and 1317. Practices in Psychological Science, 1, 198-218. Cesario J, Plaks, J. E., & Higgins, E. T. (2006). Automatic Gilder, T. S. E., & Heerey, E. A. (2018). The role of social behavior as motivated preparation to interact. experimenter belief in social priming. Psychological Journal of Personality and Social Psychology, 90, 893‐ Science, 29, 403–417. 910. Glaser M. O., & Glaser, W. R. (1982). Time course analysis Chartier, C. R., McCarthy, R. J., Williams, S. R., Ebersole, of the Stroop phenomenon. Journal of Experimental C. R., Hamlin, K., Lucas, R. E., … Lenne, R. L. (2017). Psychology: Human Perception and Performance, 8, StudySwap: A platform for interlab replication, 875-894. collaboration, and research resource exchanged. Gomes, C. M., & McCullough, M. E. (2015). The effects of Retrieved from osf.io/9aj5g implicit religious primes on dictator game allocations: A Chivers, T. (2019). A theory in crisis. Nature, 576, 200-202. preregistered replication experiment. Journal of Cohen, J. (1988). Statistical power analysis for the Experimental Psychology: General, 144, 94-104. behavioral sciences. Hillsdale, NJ: Lawrence Erlbaum. Green, D. M., & Swets, J. A. (1966). Signal detection theory Cohen, J. (1962). The statistical power of abnormal-social and psychophysics. New York: Wiley. psychological research: A review. Journal of Abnormal Greenwald, A. G. (1976). Within-subjects designs: To use or and Social Psychology, 65, 145-153. not to use. Psychological Bulletin, 83, 314-320. Coles, N. A., Larsen, J. T., & Lench, H. C. (2019). A meta- Greenwald, A. G., McGhee, D. E., & Schwartz, J. L. K. analysis of the facial feedback literature: Effects of facial (1998). Measuring individual differences in implicit feedback on emotional experience are small and variable. cognition: The implicit association test. Journal of Psychological Bulletin, 145, 610–651. Personality and Social Psychology, 74, 1464–1480. Correll, J., Park, B., Judd, C. M., & Wittenbrink, B. (2002). Harris, C. R., Coburn, N., Rohrer, D., & Pashler, H. (2013). The police officer's dilemma: Using ethnicity to Two failures to replicate high-performance-goal priming disambiguate potentially threatening individuals. Journal effects. PloS One, 8, e72467. of Personality and Social Psychology, 83, 1314–1329. https://doi.org/10.1371/journal.pone.0072467 Dijksterhuis, A., & van Knippenberg, A. (1998). The Hermans, D., De Houwer, J., & Eelen, P. (2001). A time relation between perception and behavior, or how to win course analysis of the affective priming effect. Cognition at a game of Trivial Pursuit. Journal of Personality and and Emotion, 15, 143–165. Social Psychology, 74, 865-877. Higgins, E. T., & Eitam, B. (2014). Priming…shmiming: It’s Dominus, S. (2017, October). When the revolution came for about knowing when and why stimulated memory Amy Cuddy. The New York Times Magazine. representations become active. Social Cognition, 32, Doyen, S., Klein, O., Pichon, C., & Cleeremans, A. (2012). 225-242. Behavioral priming: It’s all in the mind, but whose mind? Hughes, A. D., & Whittlesea, B. W. A. (2003). Long-term PLoS One, 7, e29081. semantic transfer: An overlapping-operations account. https://doi.org/10.1371/journal.pone.0029081 Memory & Cognition, 31, 401–411. Doyen, S., Klein, O., Simons, D. J., & Cleeremans, A. Hull, J. G., Slone, L. B., Meteyer, K. B., & Matthews, A. R. (2014). On the other side of the mirror: Priming in (2002). The nonconsciousness of self-. cognitive and social psychology. Social Cognition, 32, Journal of Personality and Social Psychology, 83, 406- 12-32. 424. Engber, D. (2017, June 7). Daryl Bem proved ESP is real: Which means science is broken. Slate. Accessed at: In press, Psychological Inquiry 14

Jacoby, L. L. (1991). A process dissociation framework: Neuroskeptic (2018, January 26). What is “social priming”? Separating automatic from intentional uses of memory. Discover Blogs. Accessed at: Journal of Memory and Language, 30, 513–541. https://www.discovermagazine.com/the-sciences/what- Judd, C. M., Westfall, J., & Kenny, D. A. (2017). is-social-priming Experiments with more than one random factor: Designs, Nosek, B. A., & Banaji, M. R. (2001). The Go/No-go analytic methods, and statistical power. Annual Review of Association Task. Social Cognition, 19, 625–666. Psychology, 68, 601-625. Pashler, H., Harris, C., & Coburn, N. (2008). Elderly-related Kahneman, D. (2011). Thinking, Fast and Slow. New York: words prime slow walking. Accessed at: Farrar, Straus, and Giroux. http://www.PsychFileDrawer.org/replication.php?attem Kahneman, D. (2012). A proposal to deal with questions pt=MTU%3D about priming effects. Pashler, H., Rohrer, D., & Harris, C. (2013). Can the goal of Kahneman, D. (2017, February 14). Re: Reconstruction of a honesty be primed? Journal of Experimental Social train wreck: How priming research went off the rails. Psychology, 14, 959-964. Retrieved from: Payne, B. K. (2001). Prejudice and perception: The role of https://replicationindex.com/2017/02/02/reconstruction- automatic and controlled processes in misperceiving a of-a-train-wreck-how-priming-research-went-of-the- weapon. Journal of Personality and Social Psychology, rails/comment-page-1/#comments 81, 181-192. Klauer, K. C., Voss, A., Schmitz, F., & Teige-Mocigemba, Payne, B. K., Brown-Iannuzzi, J. L., & Loersch, C. (2016). S. (2007). Process components of the Implicit Replicable effects of primes on human behavior. Journal Association Test: A diffusion-model analysis. Journal of of Experimental Psychology: General, 145, 1269-1279. Personality and Social Psychology, 93, 353–368. Payne, B. K., Cheng, C. M., Govorun, O., & Stewart, B. D. Krieglmeyer, R., & Sherman, J. W. (2012). Disentangling (2005). An inkblot for attitudes: Affect misattribution as stereotype activation and stereotype application in the implicit measurement. Journal of Personality and Social Stereotype Misperception Task. Journal Personality and Psychology, 89, 277–293. Social Psychology, 103, 205-224. Payne, B. K., Hall, D., Cameron, C. D., & Bishara, A. J. Loersch, C., & Payne, B. K. (2011). The Situated Inference (2010). A process model of affect misattribution. Model: An Integrative Account of the Effects of Primes Personality and Social Psychological Bulletin, 36, 1397- on Perception, Behavior, and Motivation. Perspectives 1408. on Psychological Science, 6, 234–252. Pleskac, T.J., Cesario, J. & Johnson, D.J. (2018). How race Logan, G. D. (1980). Attention and automaticity in the affects evidence accumulation during the decision to Stroop and priming tasks: Theory and data. Cognitive shoot. Psychonomic Bulletin and Review, 25, 1301–1330. Psychology, 12, 523-553. Prabhakaran, R., & Gray, J. R. (2012). The pervasive nature McCook, A. (2017, February 20). ‘I placed too much faith of unconscious social information processing in in underpowered studies:’ Nobel Prize winner admits executive control. Frontiers in human neuroscience, 6, mistakes. Retraction Watch. 105. Meissner, F., & Rothermund, K. (2013). Estimating the https://doi.org/10.3389/fnhum.2012.00105 contributions of associations and recoding in the Implicit Ratcliff, R. (1978). A theory of memory retrieval. Association Test: The ReAL model for the IAT. Journal Psychological Review, 85, 59-108. of Personality and Social Psychology, 104, 45–69. Riefer, D. M., & Batchelder, W. H. (1988). Multinomial Meyer, D. E., & Schvaneveldt, R. W. (1971). Facilitation in modeling and the measurement of cognitive processes. recognizing pairs of words: Evidence of a dependence Psychological Review, 95, 318–339. between retrieval operations. Journal of Experimental Rivers, A. M., & Sherman, J. W. (2018, January 19). Psychology: General, 90, 227-234. Experimental Design and the Reliability of Priming Meyer, M. L. (2019). Social by default: Characterizing the Effects: Reconsidering the "Train Wreck". social functions of the resting brain. Current Directions https://doi.org/10.31234/osf.io/r7pd3 in Psychological Science, 28, 380-86. Rivers, A. M., & Sherman, J. W. (2020). Same task, different Molden, D. (2014). Understanding priming effects in social processes: Different implementations of an implicit psychology: What is “social priming” and how does it measure invoke differing underlying mechanisms of bias. occur? Social Cognition, 32, 1-11. Unpublished manuscript. Musch, J., & Klauer, K. C. (1997). The proportion affect in Schimmack, U., Heene, M., Kesavan, K. (2017, February 2). affective priming: Replication and evaluation of a Reconstruction of a train wreck: How priming research theoretical explanation. Zeitschrift Experimental went off the rails. Replicability Index. Accessed at: Psychology, 44, 266-292. https://replicationindex.wordpress.com/2017/02/02/reco Neely, J. H. (1991). Semantic priming effects in visual word nstruction-of-a-train-wreck-how-priming-research-went- recognition: A selective review of current findings and of-the-rails theories. Basic Processes in Reading: Visual Word Schröder, T., & Thagard, P. (2014). Priming: Constraint Recognition, 11, 264-336. satisfaction and interactive competition. Social Neuroskeptic (2017, August 16). More on ‘behavior Cognition, 32, 157–172. priming’ and unconscious influences. Discover Blogs. Shanks, D. R. (2017). Misunderstanding the behavior Accessed at: priming controversy: Comment on Payne, Brown- http://blogs.discovermagazine.com/neuroskeptic/2017/0 Iannuzzi, and Loersch (2016). Journal of Experimental 8/16/behavior-priming-controversy/ Psychology: General, 146, 1216‐1222. In press, Psychological Inquiry 15

Shanks, D. R., Newell, B. R., Lee, E. H., Balakrishnan, D., Wang, Y. A., Sparks, J., Gonzalez, J. E., Hess, Y. D., & Ekelund, L., Cenac, Z., … Moore, C. (2013). Priming Ledgerwood, A. (2017). Using independent covariates in intelligent behavior: An elusive phenomenon. PloS One, experimental designs: Quantifying the trade-off between 8, e56515. https://doi.org/10.1371/journal.pone.0056515 power boost and Type I error inflation. Journal of Shanks, D. R., Vadillo, M. A., Riedel, B., Clymo, A., Experimental Social Psychology, 72, 118-124. Govind, S., Hickin, N., … Puhlmann, L. M. (2015). Weingarten, E., Chen, Q., McAdams, M., Yi, J., Helper, J., Romance, risk, and replication: Can consumer choices & Albarracín (2016). From primed concepts to action: A and risk-taking be primed by mating motives? Journal of meta-analysis of the behavioral effects of incidentally Experimental Psychology: General, 144, 142-158. presented words. Psychological Bulletin, 142, 472-497. Sherman, J. W., Gawronski, B., Gonsalkorale, K., Wentura, D., & Rothermund, K. (2014). Priming is not Hugenberg, K., Allen, T. J., & Groom, C. J. (2008). The priming is not priming. Social Cognition, 32, 47-67. self-regulation of automatic associations and behavioral Wheeler, S. C., DeMarree, K. G., & Petty, R. E. (2007). impulses. Psychological Review, 115, 314-335. Understanding the role of the self in prime-to-behavior Sherman, J. W., Klauer, K. C., & Allen, T. J. (2010). effects: The active-self account. Personality and Social Mathematical modeling of implicit social cognition: The Psychology Review, 11, 234-261. machine in the ghost. In B. Gawronski & B. K. Payne Woltz, D. J., & Was, C. A. (2006). Availability of related (Eds.), Handbook of implicit social cognition: long-term memory during and after attention focus in Measurement, theory, and applications (pp. 156-175). . Memory & Cognition, 34, 668–684. New York: Guilford Press. Woltz, D. J., & Was, C. A. (2007). Available but unattended Sherman, J. W., & Rivers, A. M. (2020). Social priming: A conceptual information in working memory: Temporarily dubious term. Nature, 579, 29. active semantic content or persistent memory for prior Sherman, S. M., & Jordan, T. R. (2011). Word-frequency operations? Journal of Experimental Psychology: effects in long-term semantic priming and . Learning, Memory, and British Journal of Psychology, 102, 559‐568. Cognition, 33, 155–168. Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). Yong, E. (2012, March 10). A failed replication draws a False-positive psychology: Undisclosed flexibility in scathing personal attack from a psychology professor. data collection and analysis allows presenting anything as Discover blogs: Not Exactly Rocket Science. Accessed at: significant. Psychological Science, 22, 1359–1366. http://blogs.discovermagazine.com/notrocketscience/20 Spellman, B. A. (2015). A short (personal) future history of 12/03/10/failed-replication-bargh-psychology-study- revolution 2.0. Perspectives on Psychological Science, doyen/ 10, 886-899. Zhong, C.-B., & Liljenquist, K. (2006). Washing away your Srull, T. K., & Wyer, R. S. (1979). The role of category sins: Threatened morality and physical cleansing. accessibility in the interpretation of information about Science, 313, 1451–1452. persons: Some determinants and implications. Journal of Zwaan, R. (2013, September 17). How to cook up your own Personality and Social Psychology, 37, 1660–1672. social priming article. Zeistgeist: Psychological Strack, F., Martin, L. L., & Stepper, S. (1988). Inhibiting and Experimentation, Cognition, Language, and Academia. facilitating conditions of the human smile: A Accessed at: nonobtrusive test of the facial feedback hypothesis. https://rolfzwaan.blogspot.com/2013/09/how-to-cook- Journal of Personality and Social Psychology, 54, 768– up-your-own-social-priming.html 777. Stroop, J. R. (1935). Studies of interference in serial verbal reactions. Journal of Experimental Psychology, 18, 643– 662. Trope, Y., & Fishbach, A. (2005). Going beyond the motivation given: Self-control and situational control over behavior. In R. R. Hassin, J. S. Uleman, & J. A. Bargh (Eds.), The New Unconscious (pp. 537-566). Oxford: Oxford University Press. Tse, C.-S., & Neely, J. H. (2005). Assessing activation without source monitoring in the DRM false memory paradigm. Journal of Memory and Language, 53, 532– 550. Tse, C.-S., & Neely, J. H. (2007). Semantic priming from letter-searched primes occurs for low- but not high- frequency targets: Automatic semantic access may not be a myth. Journal of Experimental Psychology: Learning, Memory, and Cognition, 33, 1143–1161. Wagenmakers, E.-J. (2014, March 26). Behavioral priming: Time to nut up or shut up. Open Science Collaboration. Accessed at: http://osc.centerforopenscience.org/2014/03/26/behavior al-priming/