1 Taking stock of the credibility revolution: Scientific reform 2011-now Coordinated by Gilad Feldman ([email protected]) at University of Hong Kong psychology department. This is shared under Attribution-NonCommercial-ShareAlike 4.0 International (CC BY-NC-SA 4.0) This is an open collaborative book. If you wish to help develop this book, visit: https://tinyurl.com/replicationcrisissummary2019 or contact the coordinator. 2 Team The book was written by the PSYC2020 Fundamentals of Social Psychology course undergraduate students. Each of the chapters details contributors for that chapter. Coordinator: Gilad Feldman [email protected], University of Hong Kong, http://giladfeldman.org, @giladfeldman Advisors: Boley Cheng [email protected] Myrte Louise Druyvesteyn [email protected] Bill Jiaxin Shi [email protected] Mira Ziqing Yao [email protected] Reviewers: Yeung Siu Kit (Wilson) Email: [email protected] Xiao Qinyu Email: [email protected] 3 Contents 1 - Realization of a problem .......................................................................................................................... 8 Team names and contribution .................................................................................................................. 8 Managerial Summary ................................................................................................................................ 9 In Depth Report....................................................................................................................................... 10 Science as a Game ............................................................................................................................... 10 Data Fabrication in Replication ........................................................................................................... 10 P hacking/ScienceMedia ..................................................................................................................... 13 Mainstreaming Scientific Transparency .............................................................................................. 14 Modifying and Omitting Data ............................................................................................................. 16 False Positives ..................................................................................................................................... 16 Questionable Research Validity in Psychology ................................................................................... 18 Revelation of Mass Replication Studies .............................................................................................. 19 Quiz ......................................................................................................................................................... 21 References .............................................................................................................................................. 23 2 - Identifying the problems........................................................................................................................ 25 Team names and contribution ................................................................................................................ 25 Managerial summary .............................................................................................................................. 26 In depth report ........................................................................................................................................ 26 Misunderstanding of statistics ............................................................................................................ 26 Misuse of statistics: Questionable Research Practices ....................................................................... 28 Credibility: Open Science On The Rise ................................................................................................ 29 Publication system: aversion to null findings, no replications, wow findings, closed networks, closed review system, no accountability, huge file-drawer ........................................................................... 30 Bad incentives system: number of papers and impact factor/rankings, rather than solid science. .. 31 Reputation and Prestige ..................................................................................................................... 32 Conclusion ........................................................................................................................................... 34 Quiz ......................................................................................................................................................... 35 References .............................................................................................................................................. 37 3 - Mass collaboration in psychology and initial findings about low replicability ...................................... 39 Team names and contribution ................................................................................................................ 39 Managerial summary .............................................................................................................................. 40 4 In depth report ........................................................................................................................................ 41 Introduction ........................................................................................................................................ 41 What are mass collaborations and where did they come from ......................................................... 41 Mass collaboration: the Reproducibility Project................................................................................. 43 Consequences for subsequent replications ........................................................................................ 45 Quiz ......................................................................................................................................................... 46 References .............................................................................................................................................. 48 4 - Reactions/debate about low replicability or the “replication crisis” - is there a problem? .................. 49 Team names and contribution ................................................................................................................ 49 Managerial summary .............................................................................................................................. 50 In depth report ........................................................................................................................................ 51 Psychology in the eye of the hurricane: An open letter and serious debate ..................................... 51 Views from the Opposite Sides: Is there really a Replication Crisis? .................................................. 54 Beyond the Debate —— Improvements, Reflections and Getting Closer to the Truth ..................... 60 Last but not Least——Treatments towards Criticism ......................................................................... 62 Quiz ......................................................................................................................................................... 64 References .............................................................................................................................................. 66 5 - Beyond Psychology: Replications in other fields ................................................................................... 69 Team names and contribution ................................................................................................................ 69 Managerial summary .............................................................................................................................. 70 In depth report ........................................................................................................................................ 71 Introduction ........................................................................................................................................ 71 Cancer Research .................................................................................................................................. 71 Neuroscience ...................................................................................................................................... 72 Microarray gene expression ............................................................................................................... 73 Biomedicine ........................................................................................................................................ 74 Economics ........................................................................................................................................... 75 Experimental philosophy .................................................................................................................... 76 Conclusion ........................................................................................................................................... 77 Quiz: .......................................................................................................................................................