Experimental Design and the Reliability of Priming Effects: Examining the “Train Wreck” Through the Lens of Statistical Power
Total Page:16
File Type:pdf, Size:1020Kb
1 Experimental Design and the Reliability of Priming Effects: Examining the “Train Wreck” through the Lens of Statistical Power Andrew M. Rivers Jeffrey W. Sherman University of British Columbia University of California-Davis Failures to replicate high-profile priming effects have raised questions about the reliability of priming phenomena. Studies at the discussion’s center, labeled “social priming,” have been interpreted as a specific indictment of priming that is social in nature. However, “social priming” differs from other priming effects in multiple ways. The present research examines one important difference: whether effects have been demonstrated with within- or between- subjects experimental designs. To examine the significance of this feature, we assess the reliability of four well-known priming effects from the cognitive and social psychological literatures using both between- and within-subjects designs and analyses. All four priming effects are reliable when tested using a within-subjects approach. In contrast, only one priming effect reaches that statistical threshold when using a between-subjects approach. This demonstration serves as a salient illustration of the underappreciated importance of experimental design for statistical power, generally, and for the reliability of priming effects, specifically. Keywords: priming; replication; statistical power; experimental design hope to advance and sharpen the discussion 2010 kicked off a volatile decade in surrounding the reliability of priming psychological science. Researchers have begun phenomena. questioning the current research paradigm in which the field operates, leading to an What is priming? introspective period commonly referred to as a Most broadly, priming refers to the incidental “crisis of confidence” or a “supposed crisis of influence of environmental context on cognition confidence,” depending on one’s perspective and behavior (Logan, 1980). Frequently, priming (Spellman, 2015). The phenomenon of priming, effects are thought to result from the activation of specifically, has proven to be one of the most mental representations that facilitate or interfere polarizing topics of discussion. with related subsequent behavior (Molden, In the present article we argue that discussions 2014). Research traditions have utilized priming about the replicability of priming effects have tasks to investigate many different theoretical overlooked the importance of research design in questions of interest. Cognitive psychologists contributing to successful replication of priming initially utilized priming tasks to probe the effects. The topics discussed within this organization of mental representations (e.g., manuscript should not be surprising, or even Neely, 1991). Social psychologists initially novel, to most readers. In fact, our central adapted priming methodologies to understand argument—that within-subjects designs are more how activated knowledge and evaluations powerful than between-subjects designs—is influence perception and behavior (e.g., Fazio, taught in introductory research methods classes at Sanbonmatsu, Powell, & Kardes, 1986). At a most universities. Despite having learned the more granular level, different areas of importance of research design in statistical power investigation have developed unique paradigms and replication, we believe that its importance to elicit priming effects and have generated a may continue to be overlooked because scientists plethora of theoretical models to explain those lack concrete examples that tangibly illustrate the effects. importance of design factors. To remedy this, we Across research traditions, priming involves seek to provide concrete examples using priming exposing subjects to different environmental effects that are familiar to researchers within contexts–or ‘primes’–that are incidental to, and many domains of psychology. In doing so, we yet still influence, subsequent behavior. For 2 example, work from the Lexical Decision Task Kahneman (2012) wrote an open letter to (Schvaneveldt & Meyer, 1973) demonstrates that researchers working in the field of ‘social people are faster to correctly identify words (e.g., priming’ that described a “train wreck looming,” ‘doctor’) following the presentation of a related in which he describes the field as the “poster child prime word (e.g., ‘nurse’) compared to an for doubts about the integrity of psychological unrelated prime word (e.g., ‘paper’). In other research.” This letter attracted considerable words, the target behavior – correctly indicating professional and media attention, much of which that ‘doctor’ is a word – occurs more quickly in consisted of critiques of ‘social priming.’ A contexts in which related, incidental stimuli are Google search of “Kahneman Train Wreck” observed than in contexts in which unrelated yielded almost 7000 results, and the letter has stimuli are observed. been cited almost 60 times in academic journal articles. Controversies in Priming The implication of Kahneman’s letter was that Several recent replication studies have failed there was something particularly amiss among to find evidence for priming phenomena, priming studies with a social component. spawning sometimes heated debates regarding However, exactly what constituted “social the replicability of priming effects. A few, high- priming” was then and remains ambiguous and profile effects have been at the center of this controversial. There is no consensually agreed debate, and have stayed in the spotlight for a upon definition of the term. Nevertheless, it number of years. continues to be a lightning rod for replication Arguably the most prominent single effect in controversy, and the narrative of social priming the discussion is reported in Experiments 2a and as a train wreck continues to hold sway (Engber, 2b from Bargh, Chen, and Burrows (1996). In 2017; McCook, 2017; Meyer & Chabris, 2014; these experiments, participants unscramble Poole, 2016; Schimmack, Heene, & Kesavan, sentences that either contained words 2017). stereotypically associated with elderly Americans An Alternative View of the Train Wreck: (e.g., often too early retired they) or not (e.g., they Research Design and Statistical Power her see outside normally). The dependent measure was the amount of time taken to walk The proposed dichotomy between social and from the experimental room to a nearby elevator. non-social priming has obscured important The key observation was that participants methodological differences among the identified exposed to the elderly primes took longer to walk priming effects and fails to account for to the elevator than did control participants. This differences in the relative reliability of various result was taken to demonstrate that incidentally priming effects (Molden, 2014; Payne, Brown- activating the elderly stereotype produced Iannuzzi, & Loersch, 2016). Those effects behavior related to the stereotyped group (i.e., the identified as “social priming” differed from other elderly walk more slowly). priming effects in a variety of ways beyond their status as social versus non-social. For example, Two independent research teams sought to the types of behaviors measured and the time lags reproduce this finding using similar priming between primes and targets differed between manipulations and speed of walking as the studies identified as social and non-social dependent measure (Doyen, Klein, Pichon, & priming. In the present research, we examine Cleeremans, 2012; Pashler, Coburn, & Harris, another important methodological difference 2008). Neither research team detected a priming between studies identified as social versus non- effect similar to that reported in Bargh et al.’s social priming; the status of the study as a within- (1996) Experiment 2. Doyen and colleagues’ versus between-subjects design. Specifically, the (2012) failure to replicate received considerable studies identified as “social priming” most press in Discover magazine, which has been frequently use between-subjects designs, whereas shared over 900 times on social media and has studies considered to exemplify non-social been cited in academic literature more than 20 priming most frequently use within-subjects times (Yong, 2012). A few months later, Daniel 3 designs. Thus, the dichotomy between social and large (ds = 1.06 and ds = .82) was viewed non-social priming largely overlaps with the skeptically. Both of the teams that failed to dichotomy between different experimental replicate Bargh et al (1996) also used between designs. Between-subjects designs most often subjects designs. Pashler et al. (2008) sampled require larger samples of subjects to achieve the from 66 subjects, which requires an effect size of same statistical power as within-subjects designs ds = .70 to achieve 80% power. Doyen et al. investigating similar effects. If researchers using (2012) recruited a sample of 120 participants, between-subjects designs do not obtain larger which requires an effect size of ds = .52 to achieve samples (as was the case in Bargh et al. 1996), 80% power. Underpowered research, in any their observations will be underpowered and domain, not only reduces the ability to detect results reported in the literature will be more effects but also increases the rate of false difficult to replicate. This in turn suggests a new, discoveries in the literature through the selective and we argue more useful, dichotomy through publication of results (e.g., Simonsohn, 2015; which we can view the train wreck—a dichotomy