Using replication projects in teaching methods Lionel G. Standing, Manuel Grenier, Erica A. Lane, Meigan S. Roberts & Sarah J. Sykes

It is suggested that replication projects may be valuable in teaching research methods, and also address the current need in for more independent verification of published studies. Their use in an undergraduate methods course is described, involving student teams who performed direct replications of four well-known , yielding results which were subsequently published online. Illustrative data are given for the one successful replication and three failures obtained, and practical suggestions are given for incorporating replication projects into a methods course as an alternative to the usual term project. It is also noted that the published success rates of replication attempts appear to be higher for those studies that were performed as class projects. Keywords: Replication; teaching; research methods.

HIS PAPER points to a pressing need in Only 13 successful replications are scientific psychology today for increased reported among the 44 attempts that are Treplication of earlier studies, and suggests posted currently on the PsychFileDrawer.org a practical solution to this problem which also website, a success rate of 30 per cent overall; carries potential benefits for both students the targets are 30 prominent articles in and teachers in research methods courses. human psychology. The most common area represented is , with 10 target studies The replication crisis today and 18 replication attempts (sometimes Rosenthal’s (1979) seminal paper first more than one per target article) that pointed out the ‘file drawer problem’: yielded four successful replications and 14 journals print papers that are often based on failures, a success rate of only 22 per cent. It Type 1 errors while the file drawers of has also been noted that only about one per researchers contain unpublished studies of cent of psychology articles in psychology the same topics showing null outcomes. journals clearly involve replication attempts, Many commentators today believe that there and of these possibly 93 per cent succeed is a ‘crisis’ within the discipline of psychology when published by the original authors, but (Laws, 2013; Neuliep, 1990; Pashler & only 69 per cent when published by new Wagenmakers, 2012; Ritchie, Wiseman & authors (Makel, Plucker & Hegarty, 2012). French, 2012; Yong, 2012). Very few reports However, the pervasive file drawer problem of replication studies appear in psychology means that the true situation is worse, since journals, despite a growing realisation that journals in most disciplines, including even major studies appearing in leading psychology, generally publish only positive journals often fail to replicate successfully results (Fanelli, 2011), resulting in the when this is attempted by independent inves - ‘…promulgation of numerous undead theo - tigators (Schmidt, 2009). The recent special ries that are ideologically popular but have issue of Perspectives on Psychological little basis in fact’ (Ferguson & Heene, 2012, confirms the seriousness of this problem p.555). (Pashler & Harris, 2012), which also occurs Students in research methods courses are in other scientific fields, for example, taught that replicability is a cardinal feature cancer research (Begley & Ellis, 2012). of the (e.g. McBurney &

96 Psychology Teaching Review Vol. 20 No. 1, Spring 2014 © The British Psychological Society Using replication projects

White, 2010, pp.208 –210), but without Science has announced that its journal demonstrable repeatability of findings the Perspectives on Psychological Science is now scientific enterprise is invalidated on logical running a replication project with a stan - grounds alone. It seems hypocritical to dardised protocol (Gage, 2013; ‘Registered present this idea as a guarantee of method - replication reports’, 2013), and PsychDisclo - ological rigour when it is being violated in sure.org has appeared as a public database many cases today. Ioannidis (2012) argues for recently published articles, to provide that the estimate of 53 per cent for fallacies additional methodological details (Lebel et that are perpetuated in the literature which al., 2013). Finally, occasional multi-experi - is proposed by Pashler and Harris (2012) ment studies are published which exhaus - may be too low, and that the true figure tively examine the replicability of a given could reach 95 per cent in certain fields of effect (e.g. Shanks et al., 2013, present nine psychology. At the least, numerical analyses studies, with 475 participants, which all suggest that a majority of published research failed to replicate the intelligence priming findings are false (Ioannidis, 2005). effect). If a replication crisis exists, then However, approximately 123,000 new constructive solutions are needed. The few entries for peer-reviewed journal articles are replications that are published today lead to added each year to PsycINFO, which already exchanges that are often protracted, incon - includes the abstracts of over 2,400,000 arti - clusive, or abrasive (e.g. Byrne et al., 1966; cles (only 32,000 of these, or 1.3 per cent, Chabris et al., 1999; Dijksterhuis, 2013; include the stem replicat* in the abstract). Doyen et al., 2012; Shanks et al., 2013; With so many unverified research findings, Wagenmakers et al., 2011). the most plausible way to reduce the problem may be to employ students as Students as a strategic resource collaborators, as suggested by Grahe et al. Many suggestions have been advanced which (2012), and by Frank and Saxe (2012). As ideally might overcome the replication these authors argue, this would be good for problem (Asendorpf et al., 2013), for the public scientific accountability of example, reward replication research as psychology, as well as for students and much as original work, require external teachers. Students may represent an abun - replication of results before publication, or dant and underutilised resource, assuming require studies to be registered in advance they may be relied upon to collect valid data, with a central registry. Brandt et al. (2014) and in comparison with this possible have elaborated these ideas into a Replica - problem, the likely payoffs appear consider - tion Recipe with a checklist of 36 questions able: enhanced of the to guarantee a ‘convincing’ replication, but field, a more manageable term project for the likelihood of its general implementation the student, and easier classwork for a appears minimal, due to reviewer teacher to set up. Furthermore the new against replications (Neuliep & Crandall, online posting forum PsychFileDrawer.org 1993), scarce journal space, the low prestige provides a simple, cost-efficient way to build attached to replications, time constraints on up over time a public database of replication researchers, and the restriction of funding to attempts and outcomes. new work. Some initiatives are evident here, since Typical problems with the traditional the online journal BioMed Central Psychology independent term project has pledged that it will ‘put less emphasis on Although a few individual student projects interest levels’, and will publish repeat are published, on rare occasions becoming studies and negative results (Laws, 2013). well-known (e.g. Pheterson, Kiesler & Gold - Also the Association for Psychological berg, 1971; ‘This week’s citation classic’,

Psychology Teaching Review Vol. 20 No. 1, Spring 2014 97 Lionel G. Standing, Manuel Grenier, Erica A. Lane, Meigan S. Roberts & Sarah J. Sykes

1983), and an individual project might credit for a basic research methods course appear to be a desirable adjunct to standard and two courses, all taken in first lab exercises that are set up by the instructor, year, with an average grade of at least 75 per our experience over the years has been that cent. It is not likely that undergraduates most students find it very difficult within a without a prior background in methods or single term or semester to meet the require - statistics would be suitable for this approach. ments for a meaningful independent The instructor and class first chose four project: an original hypothesis based on rele - target articles from the current listing of vant literature, operationalisation of the key papers given on PsychFileDrawer.org, with idea, preparation of materials, ethics the results of at least one prior replication approval, pilot testing and , to attempt posted, whether successful or not. be followed by data analysis and a written (This is not essential, but provides a way to report. compare the class results not only with the For students who are fairly new to target article but also with other replication psychology, with other work to perform in attempts.) the methods course, this is a major chal - To do this, the original papers listed in lenge, with time pressure often leading to PsychFileDrawer.org were all scrutinised, experiments that have been described as and eliminated as possible targets if they ‘…silly, poorly designed, and unlikely to involved practical difficulties re apparatus or connect to current issues in psychological materials, a very specialised or technical science’ (Frank & Saxe, 2012, p.601). focus, lengthy or individualised testing Proposed projects may show little theoretical requirements, or a complicated design. understanding, knowledge of the literature, Instances were also noted of marginal statis - or scientific awareness. Too often the exper - tical significance, low , or large imental hypothesis is unclear, or has been numbers of subjects (as we wished at least to created by the teacher, with methodological match the N of each target study). When two flaws sometimes evident, and samples are experiments within a given study both usually so small that statistical power is low. appeared to be suitable, the one reporting By way of contrast, a two-term honours the larger effect size was chosen so as to dissertation project involves a manageable maximise statistical power. Each of the schedule, generally leading to a much better chosen papers included a multiple set of study and possibly a joint publication with experiments, from which only one was the supervisor. selected for replication. The selection stage For the above reasons, the senior author provides a useful opportunity to discuss with decided to offer students the option of the class the issues of power, significance, performing a replication project in place of and effect size by examining the original an independent term project. (All chose this study and the number of subjects needed for more structured option.) Direct rather than the replication, facilitated by computational conceptual replications were used, since the software (e.g. Allen & Hannent, 2013). latter are logically subordinate to studies This process occupied about two weeks which first test for the existence of an effect and led to the selection of the four papers before attempts are made to generalise it. posted in PsychFileDrawer.org which best met the above criteria. Three of these were Implementing replication projects priming studies (Dijksterhuis & van Knip - This approach was developed in a three- penberg, 1998, 4; Vohs, Mead & credit advanced research methods class at a Goode, 2006, Experiment 3; Williams & small university. This course is taken in the Bargh, 2008, Study 3). These studies second year of a three-year psychology reported that primes related to intelligence, honours BA programme, and requires prior money, or distance, respectively, raise cogni -

98 Psychology Teaching Review Vol. 20 No. 1, Spring 2014 Using replication projects tive performance, reduce helpfulness, and paper, and an outline template for the cause lower caloric estimates for unhealthy results section, as well as standard forms for foods. Despite recent controversy con- informed consent and debriefing. The cerning the validity of some priming experi - instructor handled the paperwork to obtain ments (Bartlett, 2013), the selection of these approval from the research ethics board for three priming studies as targets was fortu - each project. itous and due solely to the above-mentioned The instructor then met with each team, criteria. The fourth study selected (Gailliot and discussed the details of their project to et al., 2007, Study 8) reported that ingesting clarify details of its methods and procedure, a glucose drink enhances a subject’s self- for example, if the time allowed for a certain control when it has been threatened by test was not specified in the report, a deci - thoughts of mortality. sion was made as to how to standardise it. Every attempt was made to follow the Organising the teams specifics of the target article closely, and in Although a replication may be performed by some cases details were obtained from the a pair of lab partners (Frank & Saxe, 2012), original authors by email. The numbers of this would entail heavy time demands on the subjects ultimately tested in the four studies student, so we had the 24 members of the were similar to those in the target papers, class sign up into four teams of six each, to with respective values as follows (target spread the time load. This team size proved article Ns are given in parentheses): 48 (43), satisfactory, although the final choice is arbi - 58 (73), 40 (39), and 60 (59) participants. trary. One member of each team was desig - Testing participants nated as a co-ordinator, on the basis of a The instructor met the co-ordinators individ - group vote, with responsibilities for ually to run through a trial testing session, preparing materials, consulting regularly taking the role of the subject. It is important with the instructor, organising team first to create a standardised written protocol members to carry out the testing, and storing for all student testers to follow, once the the data. Co-ordinators were required to test details of materials and apparatus have been only a handful of subjects, for the experi - finalised. Each team thereafter largely ran ence, so as to equalise the total time load for their own project, performing their assigned them and their team members. There is no testing in a departmental lab. When pilot need to make all teams the same size, and if testing and data collection had been a given project requires many subjects to completed, each co-ordinator collated their match the original sample size (or involves a team’s data sheets and prepared a composite lengthy testing procedure, or individual as SPSS data file in conjunction with the opposed to group testing), the sizes of the instructor, who cross-checked their statistical teams may be adjusted so as to balance out analysis. This data file was sent to the team the workload per student. Teams were members for use in writing up their reports, encouraged to take personal responsibility which were graded individually. Although for their projects. participation in fact proved to be excellent, Each student next wrote a graded this was vouchsafed by explaining in the first proposal of about 10 pages in APA format, class meeting that each person’s contribu - which they had already encountered in class, tion would be rated at the end by the other to ensure that they would be familiar with members of their team, using the Peer Eval - their target study. This proposal included a uation Form of Herreid (2001). No objec - short literature review, details of the method tions were raised. and procedure gleaned from the target

Psychology Teaching Review Vol. 20 No. 1, Spring 2014 99 Lionel G. Standing, Manuel Grenier, Erica A. Lane, Meigan S. Roberts & Sarah J. Sykes

Results these three failures yielded non-significant With some monitoring by the instructor, the differences in the opposite direction to the experiments all proceeded smoothly, with target article, and the composite effect size, less stress evident over time pressures than is averaged over the four experiments, was usual with term projects, and few logistic effectively zero. The data, given in Table 1, issues arising. are quite typical for PsychFileDrawer.org, At the end of each project, each team which lists six other failures to replicate member submitted a full report in APA these three priming studies, and no format, which enlarged upon their proposal successes. and included data, analysis, and conclusions. It may be noted that even though we The report by each co-ordinator, suitably were able to replicate Study 8 of the study by edited, also served as the preliminary basis Gaillot et al., PsychFileDrawer.com lists an for a composite final report, with their team unsuccessful attempt at replicating Study 7 members and the instructor listed as addi - of that paper (Cesario & Corker, 2010), and tional authors, which was posted on the the theory of glucose as an aid to self-control public archive of replication attempts main - has elsewhere been challenged (Kurzban, tained at PsychFileDrawer.org (Grenier et 2011; Molden et al., 2012). Conversely, al., 2012; Lane et al., 2012; Roberts et al., although we failed to replicate Vohs et al. 2012; Sykes et al., 2012). (2006), other studies have found a link The four projects overall yielded one between money-priming and reduced help - successful replication (of Gailliot et al., fulness (e.g. Chatterjee, Rose & Sinha, 2013, 2007), which showed almost the same effect Gasiorowska & Helka, 2012; Roberts & size as the original study, and three failures Roberts, 2012). A failure to replicate a given (involving the three priming studies). Two of study does not prove that it was faulty.

Table 1: Comparison of target study and replication results.

Target Target study finding Target Replication Did the study study result replication result succeed? dpdp 1 Verbal priming raises intelligence 0.85 <.02 -0.29 .84 No 2 Glucose aids self-control processes .65 .05 0.30 .03 Yes 3 Money priming reduces helpfulness 0.66 <.05 0.08 .42 No 4 Distance prime alters calorie estimates 0.90 .04 -0.023 .70 No Mean d= 0.69 0.017

Note: d=effect size Target study 1=Dijksterhuis & van Knippenberg (1998), Experiment 4. Target study 2=Gailliot et al. (2007), Study 8. Target study 3=Vohs, Mead & Goode (2006), Experiment 3. Target study 4=Williams & Bargh (2008), Study 3.

100 Psychology Teaching Review Vol. 20 No. 1, Spring 2014 Using replication projects

Conclusions and discussion cate the same target article, to explore a We believe that the approach outlined here further dimension of replicability. Or can bridge the gap between the classroom different teams could attempt to repeat the and the world of psychological research, various experiments comprising a given yielding much-needed scientific information paper. Repeating a replication attempt in as well as meaningful project work that successive years would provide another incorporates diverse active/reflective and index of reliability. In the case of a between- abstract/concrete aspects of learning (Kolb, subjects design, extra groups can be added 1984). Students encountered replication as a to create a replication-extension study where requirement for scientific progress, exam - both an exact and a conceptual replication ined issues of power and effect size, and are attempted concurrently, to test the relia - engaged in some critical thinking. They also bility of the original finding and explore improved their report writing, computing, moderator effects which might explain it and statistical skills, and learned to work (Bonett, 2012). This approach might be suit - collaboratively. In comparison with previous able for an honours dissertation (e.g. Carlin years, it was easier to organise the class work, & Standing, 2013), or a graduate thesis. and students were less anxious over their Some studies are beyond the capacities of projects, working in teams within a pre-estab - undergraduate students, but may be suited lished template. This team interaction may to graduate students (Frank & Saxe, 2012). help with the problem of statistics-anxiety It has been argued that failures to repli - (Williams, 2010). They showed good motiva - cate occur because psychology students do tion, contributing suggestions and raising not have the necessary methodological skills points for clarification, and seemed pleased to perform valid studies (Dijksterhuis, 2013), when they finally achieved an online publi - and if there is any increased noise in a system cation. due to inexperienced testers then this might A further benefit of this approach is that seem more likely to produce a Type II than a it involves many fewer different projects, Type I error. But the evidence to date using pre-established , which suggests otherwise. Fourteen of the 44 may facilitate early approval from an ethics studies listed by PsychFileDrawer.com board, whereas with individual projects a (at 6 January 2014) are listed as having been month’s delay can occur, seriously performed as class projects. These class pro- hampering the collection of data. jects actually showed a higher probability of The ultimate benefit of replication achieving replication (seven successes in 14 studies hopefully may spread throughout the attempts), than the remaining studies (6 field of psychology as the reliability of major successes out of 30). This difference is reli - studies is clarified through further internet able (Fisher’s exact test, p=.049; Cramér’s V, postings. This approach appears to produce p=.042), a result which directly contradicts a win-win-win situation. We encountered no Dijksterhuis’s hypothesis, perhaps because real problems, and can recommend this idea the file drawer effect operates more strongly for adoption by instructors who are teaching when student investigators are involved. a methods or lab course with students who We suggest that for simple studies, already know some basic statistics and performed with faculty monitoring, a satis - methodology. factory level of reliability can probably be The responsibility for writing the achieved. We note that the target papers for proposal and the final report may be replication are themselves often based on assigned to individuals, pairs of lab partners, data that were collected by students under or teams. The basic framework is flexible, supervision. Should this view seem too opti - and could be adapted so that, for example, mistic, then a control test would resolve the some teams independently attempt to repli - issue. If it is possible to identify some experi -

Psychology Teaching Review Vol. 20 No. 1, Spring 2014 101 Lionel G. Standing, Manuel Grenier, Erica A. Lane, Meigan S. Roberts & Sarah J. Sykes mental effect that unquestionably does exist, The Authors then we could determine how many out of Lionel G. Standing 100 student experimenters succeed in repro - Manuel Grenier ducing it. A validity check of this type would Erica A. Lane complement the current approach and assess Meigan S. Roberts whether serious measurement error is intro - Sarah J. Sykes duced by the use of student investigators. Department of Psychology, Bishop’s University. Authors’ Note This research was supported by a grant from Correspondence the Senate Research Committee of Bishop’s Lionel G. Standing University. The members of the PSY313A Department of Psychology, class at Bishop’s University are thanked for Bishop’s University, their assistance in collecting data. The 2600 College Street, helpful advice of Kathleen Vohs is gratefully Sherbrooke QC, acknowledged. Canada J1M 1Z7. Email: [email protected]

References Allen, R. & Hannent, I. (2013). A new computerised Chabris, C.F., Steele, K.M., Bella, S.D., Peretz, I., power calculator with graphical interface. Available at: Dunlop, T., Dawe, L.A. & Rauscher, F.H. (1999, http://homepages.gold.ac.uk/ps-tec/ 26 August). Prelude or requiem for the ‘Mozart powercalculator/ effect’? Science, 400 (6747), 826–827. Asendorpf, J.B., Conner, M., De Fruyt, F., De Houwer, Chatterjee, P., Rose, R.L. & Sinha, J. (2013). Why J., Denissen, J.J., Fiedler, K. & Wicherts, J.M. money meanings matter in decisions to donate (2013). Recommendations for increasing time and money. Marketing Letters, 24 , 109–118. replicability in psychology. European Journal of Dijksterhuis, A. (2013, 25 April). Replication crisis or Personality, 27 ,108–119. crisis in replication? A reinterpretation of Shanks Bartlett, T. (2013). Power of suggestion. Chronicle of et al. PLoS ONE Reader comments. Available at: Higher Education. Available at: http://www.plosone.org/annotation/list http://chronicle.com/article/Power-of- Thread.action?root=64751 Suggestion/136907/ Dijksterhuis, A. & van Knippenberg, A. (1998). The Begley, C.G. & Ellis, L.M. (2012). Drug development: relation between perception and behavior, or how Raise standards for pre-clinical cancer research. to win a game of Trivial Pursuit. Journal of , 483 (7391), 531–533. Personality and , 74 , 865–877. Bonett, D.G. (2012). Replication-extension studies. Doyen, S., Klein, O., Pichon, C.-L. & Cleeremans, A. Current Directions in Psychological Science, 21 , (2012). Behavioral priming: It’s all in the mind, 409–412. but whose mind? PLoS ONE, 7 (1), e29081. Brandt, M.J., IJzerman, H., Dijksterhuis, A., Farach, Fanelli, D. (2011). Negative results are disappearing F.J., Geller, J., Giner-Sorolla, R. & van’t Veer, A. from most disciplines and countries. , (2014). The replication recipe: What makes for a 90 , 891. convincing replication? Journal of Experimental Ferguson, C.J. & Heene, M. (2012). A vast graveyard of Social Psychology, 50 , 217–224. undead theories: and psycho- Byrne, W.L., Samuel, D., Bennett, E.L., Rosenzweig, logical science’s aversion to the null. Perspectives on M.R, Wasserman, E., Wagner, A.R. & Carbon, P.L. Psychological Science, 7 , 555–561. (1966). Memory transfer. Science, 153 , 658–659. Frank, M.C. & Saxe, R. (2012). Teaching replication. Carlin, S.P. & Standing, L.G. (2013). Is intelligence Perspectives on Psychological Science, 7 , 600–604. enhanced by letter priming? A failure to replicate Gage, S. (2013, May 15). Psychology uses ‘registered the results of Ciani and Sheldon (2010). replication reports’ to improve reliability. Psychological Reports, 112 , 533–544. Guardian Blog. Available at: Cesario, J. & Corker, K. (2010). Glucose drinks and self- http://www.guardian.co.uk/science/sifting-the- control (Stroop) . Available at: evidence/2013/may/15/psychology-registered- http://www. psychfiledrawer.org/ replication-reports-reliability replication.php?attempt=MTIw

102 Psychology Teaching Review Vol. 20 No. 1, Spring 2014 Using replication projects

Gailliot, M.T., Baumeister, R.F., DeWall, C.N., Maner, Molden, D.C., Hui, C.M., Scholer, A.A., Meier, B.P., J.K., Plant, E.A., Tice, D.M., Brewer, L.E. & Noreen, E.E., D’Agostino, P.R. & Martin, V. Schmeichel, B.J. (2007). Self-control relies on (2012). Motivational versus metabolic effects of glucose as a limited energy source: Willpower is carbohydrates on self-control. Psychological Science, more than a metaphor. Journal of Personality and 23 , 1137–1144. Social Psychology, 92 , 325–336. Neuliep, J.W. (Ed.). (1990). Handbook of replication Gasiorowska, A. & Helka, A.M. (2012). Psychological research in the behavioral and social . consequences of money and money attitudes in Corte Madera, CA: Select Press. dictator game. Polish Psychological Bulletin, 43 , Neuliep, J.W. & Crandall, R. (1993). Reviewer bias 20–26. against replication research. Journal of Social Grahe, J.E., Reifman, A., Hermann, A.D., Walker, M., Behavior & Personality, 8 , 1–8. Oleson, K.C., Nario-Redmond, M. & Wiebe, R.P. Pashler, H. & Harris, C.R. (2012). Is the replicability (2012). Harnessing the undiscovered resource of crisis overblown? Three arguments examined. student research projects. Perspectives on Perspectives on Psychological Science, 7 , 531–536. Psychological Science, 7 , 605–607. Pashler, H. & Wagenmakers, E. (2012). Editors’ Grenier, M., Bertrand, T., Dupuis Laflamme, P., Pepin, introduction to the Special Section on E., Webster, L.E., Wheeler, L.E. & Standing, L.G. replicability in psychological science: A crisis of (2012). A replication of Vohs, Mead and Goode confidence? Perspectives on Psychological Science, 7 , (2006). Available at: 528–530. http://www.PsychFileDrawer.org/chart.php? Pheterson, G.I., Kiesler, S.B. & Goldberg, P.A. (1971). target_article=32 Evaluation of the performance of women as a Herreid, C.F. (2001). When justice peeks: Evaluating function of their sex, achievement, and personal students in case study teaching. Journal of College history. Journal of Personality and Social Psychology, Science Teaching, 30 , 430–433. 19 , 114–118. Ioannidis, J.P. (2005). Why most published research Registered Replication Reports (2013). Association for findings are false. PLoS Med, 2 (8): e124. Psychological Science. Available at: Ioannidis, J.P. (2012). Why science is not necessarily http://www.psychologicalscience.org/ self-correcting. Perspectives on Psychological Science, index.php/replication 7, 645–654. Ritchie, S.J., Wiseman, R. & French, C.C. (2012). Kolb, D.A. (1984). Experiential learning: Experience as the Replication, replication, replication. The Psycho- source of learning and development. Englewood Cliffs, logist, 25 , 346–357. NJ: Prentice-Hall. Roberts, J.A. & Roberts, C.R. (2012). Money matters: Kurzban, R. (2011). Glucose is not willpower fuel. Does the symbolic presence of money affect Psychology Today . Available at: charitable giving and attitudes among http://www.psychologytoday.com/blog/mind- adolescents? Young Consumers, 13 , 329–336. design/201108/glucose-is-not-willpower-fuel Roberts, M.S., Crooks, W., Kolody, T.J., Pavlovic, T., Lane, E.A., Beadman, K.S., Bélisle, A.M., Campbell, Rombola, K.J. & Standing, L.G. (2012). No effect R.M., Cournoyer Lemaire, E., Stewart, D. & on intelligence from priming: A replication of Standing, L.G. (2012). Mortality salience improves Dijksterhuis and van Knippenberg (1998). Available rather than impairs self-control: A replication of Gailliot, at: Baumeister et al. (2007). Available at: http://www.PsychFileDrawer.org/ http://www.PsychFileDrawer.org/ chart.php?target_article=33 chart.php?target_ article=29 Rosenthal, R. (1979). The file drawer problem and Laws, K. (2013, 27 February). It’s time for tolerance for null results. Psychological Bulletin, 86 , psychologists to put their house in order. Guardian 638–641. Science blog . Available at: Schmidt, S. (2009). Shall we really do it again? The http://www.guardian.co.uk/science/blog/2013/ powerful concept of replication is neglected in feb/27/ psychologists-bmc-psychology the social sciences. Review of General Psychology, 13 , Lebel, E.P., Borsboom, D., Giner-Sorolla, R., 90–100. Hasselman, F., Peters, K.R., Ratliff, K.A. & Smith, Shanks, D.R., Newell, B.R., Lee, E.H., Balakrishnan, C.T. (2013). Grassroots support for reforming D., Ekelund, L., Cenac, Z., Kavvadia, F. & Moore, reporting standards in psychology. Perspectives on C. (2012). Priming intelligent behavior: An Psychological Science, 8 , 424–432. elusive phenomenon. PLoS ONE, 8 (4): e56515. Makel, M.C., Plucker, J.A. & Hegarty, B. (2012). Sykes, S.J., Carr Kinnear, C.D., Carrière, J.S., Replications in psychology research: How often Chatelain, M.C., Chieco, A., Santini, S.E. & do they really occur? Perspectives on Psychological Standing, L.G. (2012). A replication of Williams and Science, 7 , 537–542. Bargh (2008). Available at: McBurney, D. & White, T. (2010). Research methods http://www.psychfiledrawer.org/chart.php? (8th ed.). Belmont, CA: Wadsworth. target_article=2

Psychology Teaching Review Vol. 20 No. 1, Spring 2014 103 Lionel G. Standing, Manuel Grenier, Erica A. Lane, Meigan S. Roberts & Sarah J. Sykes

This Week’s Citation Classic (1983, 17 October). Williams, A.S. (2010). Statistics anxiety and instructor ISI Current Contents, 42, 20. immediacy. Journal of Statistics Education, 18. Vohs, K.D., Mead, N.L. & Goode, M.R. (2006). The Available at: psychological consequences of money. Science, http://www.amstat.org/publications/ 314(5802), 1154–1156. jse/v18n2/williams.pdf. Wagenmakers, E.J., Wetzels, R., Borsboom, D. & van Williams, L.E. & Bargh, J.A. (2008). Keeping one’s der Maas, H.L. (2011). Why psychologists must distance: The influence of spatial distance cues on change the way they analyse their data: The case affect and evaluation. Psychological Science, 19, of psi: Comment on Bem (2011). Journal of 302–308. Personality and Social Psychology, 100, 426–432. Yong, E. (2012, 17 May). Replication studies: Bad copy. Nature, 485(7398), 279–410.

104 Psychology Teaching Review Vol. 20 No. 1, Spring 2014