DOCSLIB.ORG

Taking Stock of the Credibility Revolution: Scientific Reform 2011-Now

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Taking Stock of the Credibility Revolution: Scientific Reform 2011-Now 1 Taking stock of the credibility revolution: Scientific reform 2011-now Coordinated by Gilad Feldman ([email protected]) at University of Hong Kong psychology department. This is shared under Attribution-NonCommercial-ShareAlike 4.0 International (CC BY-NC-SA 4.0) This is an open collaborative book. If you wish to help develop this book, visit: https://tinyurl.com/replicationcrisissummary2019 or contact the coordinator. 2 Team The book was written by the PSYC2020 Fundamentals of Social Psychology course undergraduate students. Each of the chapters details contributors for that chapter. Coordinator: Gilad Feldman [email protected], University of Hong Kong, http://giladfeldman.org, @giladfeldman Advisors: Boley Cheng [email protected] Myrte Louise Druyvesteyn [email protected] Bill Jiaxin Shi [email protected] Mira Ziqing Yao [email protected] Reviewers: Yeung Siu Kit (Wilson) Email: [email protected] Xiao Qinyu Email: [email protected] 3 Contents 1 - Realization of a problem .......................................................................................................................... 8 Team names and contribution .................................................................................................................. 8 Managerial Summary ................................................................................................................................ 9 In Depth Report....................................................................................................................................... 10 Science as a Game ............................................................................................................................... 10 Data Fabrication in Replication ........................................................................................................... 10 P hacking/ScienceMedia ..................................................................................................................... 13 Mainstreaming Scientific Transparency .............................................................................................. 14 Modifying and Omitting Data ............................................................................................................. 16 False Positives ..................................................................................................................................... 16 Questionable Research Validity in Psychology ................................................................................... 18 Revelation of Mass Replication Studies .............................................................................................. 19 Quiz ......................................................................................................................................................... 21 References .............................................................................................................................................. 23 2 - Identifying the problems........................................................................................................................ 25 Team names and contribution ................................................................................................................ 25 Managerial summary .............................................................................................................................. 26 In depth report ........................................................................................................................................ 26 Misunderstanding of statistics ............................................................................................................ 26 Misuse of statistics: Questionable Research Practices ....................................................................... 28 Credibility: Open Science On The Rise ................................................................................................ 29 Publication system: aversion to null findings, no replications, wow findings, closed networks, closed review system, no accountability, huge file-drawer ........................................................................... 30 Bad incentives system: number of papers and impact factor/rankings, rather than solid science. .. 31 Reputation and Prestige ..................................................................................................................... 32 Conclusion ........................................................................................................................................... 34 Quiz ......................................................................................................................................................... 35 References .............................................................................................................................................. 37 3 - Mass collaboration in psychology and initial findings about low replicability ...................................... 39 Team names and contribution ................................................................................................................ 39 Managerial summary .............................................................................................................................. 40 4 In depth report ........................................................................................................................................ 41 Introduction ........................................................................................................................................ 41 What are mass collaborations and where did they come from ......................................................... 41 Mass collaboration: the Reproducibility Project................................................................................. 43 Consequences for subsequent replications ........................................................................................ 45 Quiz ......................................................................................................................................................... 46 References .............................................................................................................................................. 48 4 - Reactions/debate about low replicability or the “replication crisis” - is there a problem? .................. 49 Team names and contribution ................................................................................................................ 49 Managerial summary .............................................................................................................................. 50 In depth report ........................................................................................................................................ 51 Psychology in the eye of the hurricane: An open letter and serious debate ..................................... 51 Views from the Opposite Sides: Is there really a Replication Crisis? .................................................. 54 Beyond the Debate —— Improvements, Reflections and Getting Closer to the Truth ..................... 60 Last but not Least——Treatments towards Criticism ......................................................................... 62 Quiz ......................................................................................................................................................... 64 References .............................................................................................................................................. 66 5 - Beyond Psychology: Replications in other fields ................................................................................... 69 Team names and contribution ................................................................................................................ 69 Managerial summary .............................................................................................................................. 70 In depth report ........................................................................................................................................ 71 Introduction ........................................................................................................................................ 71 Cancer Research .................................................................................................................................. 71 Neuroscience ...................................................................................................................................... 72 Microarray gene expression ............................................................................................................... 73 Biomedicine ........................................................................................................................................ 74 Economics ........................................................................................................................................... 75 Experimental philosophy .................................................................................................................... 76 Conclusion ........................................................................................................................................... 77 Quiz: .......................................................................................................................................................
Load more

Recommended publications
  • Reproducibility and Replicability in Science (2019)
    THE NATIONAL ACADEMIES PRESS This PDF is available at http://nap.edu/25303 SHARE Reproducibility and Replicability in Science (2019) DETAILS 256 pages | 6 x 9 | PAPERBACK ISBN 978-0-309-48616-3 | DOI 10.17226/25303 CONTRIBUTORS GET THIS BOOK Committee on Reproducibility and Replicability in Science; Board on Behavioral, Cognitive, and Sensory Sciences; Committee on National Statistics; Division of Behavioral and Social Sciences and Education; Nuclear and Radiation Studies FIND RELATED TITLES Board; Division on Earth and Life Studies; Board on Mathematical Sciences and Analytics; Committee on Applied and Theoretical Statistics; Division on SUGGESTEDEngineering an CITATIONd Physical Sciences; Board on Research Data and Information; NCaotmiomnaittl eAec oand eSmciiesn coef ,S Ecniegninceeesr, inEgn,g Mineedeircininge,, aanndd MPeudbilcicin Peo 2li0c1y;9 P. Roleicpyr oadnudcibility aGnlodb Rael Aplfifcaairbsi;li tNy ainti oSncaiel Ancea.d Wemaisehsi nogf tSonc,ie DnCce: sT,h Een Ngianteioenrainl gA, caandde Mmeiedsi cPinress. https://doi.org/10.17226/25303. Visit the National Academies Press at NAP.edu and login or register to get: – Access to free PDF downloads of thousands of scientific reports – 10% off the price of print titles – Email or social media notifications of new titles related to your interests – Special offers and discounts Distribution, posting, or copying of this PDF is strictly prohibited without written permission of the National Academies Press. (Request Permission) Unless otherwise indicated, all materials in this
    [Show full text]
  • Statistical Fallacy: a Menace to the Field of Science
    International Journal of Scientific and Research Publications, Volume 9, Issue 6, June 2019 297 ISSN 2250-3153 Statistical Fallacy: A Menace to the Field of Science Kalu Emmanuel Ogbonnaya*, Benard Chibuike Okechi**, Benedict Chimezie Nwankwo*** * Department of Ind. Maths and Applied Statistics, Ebonyi State University, Abakaliki ** Department of Psychology, University of Nigeria, Nsukka. *** Department of Psychology and Sociological Studies, Ebonyi State University, Abakaliki DOI: 10.29322/IJSRP.9.06.2019.p9048 http://dx.doi.org/10.29322/IJSRP.9.06.2019.p9048 Abstract- Statistical fallacy has been a menace in the field of easier to understand but when used in a fallacious approach can sciences. This is mostly contributed by the misconception of trick the casual observer into believing something other than analysts and thereby led to the distrust in statistics. This research what the data shows. In some cases, statistical fallacy may be investigated the conception of students from selected accidental or unintentional. In others, it is purposeful and for the departments on statistical concepts as it relates statistical fallacy. benefits of the analyst. Students in Statistics, Economics, Psychology, and The fallacies committed intentionally refer to abuse of Banking/Finance department were randomly sampled with a statistics and the fallacy committed unintentionally refers to sample size of 36, 43, 41 and 38 respectively. A Statistical test misuse of statistics. A misuse occurs when the data or the results was conducted to obtain their conception score about statistical of analysis are unintentionally misinterpreted due to lack of concepts. A null hypothesis which states that there will be no comprehension. The fault cannot be ascribed to statistics; it lies significant difference between the students’ conception of with the user (Indrayan, 2007).
    [Show full text]
  • The R and Omegahat Projects in Statistical Computing
    The R and Omegahat Projects in Statistical Computing Brian D. Ripley [email protected] http://www.stats.ox.ac.uk/∼ripley Outline • Statistical Computing – History – S – R • Application and Comparisons – Web servers – Embedding — Medieval Chant – R vs S-PLUS • Omegahat and Component Systems – The Omegahat project – Components — GGobi – The future? Statistical Computing and S Scene-setting: Statistical Computing 1980 Mainly Fortran programming, or PL/I (SAS). Batch computing (SAS, BMDP, SPSS, Genstat) with restricted range of platforms. Some small interactive systems (GLIM 3.77, Minitab). Very poor interactive graphics (2400 baud to a Tektronix storage tube if you were lucky). Flatbed and drum plotters, microfilm for publication-quality output off-line. Mainly home-brew solutions in research. (GLIM macros?) 1990 PCs become widespread, but FPUs still uncommon. Sun etc workstations available for researchers, and for teaching in a few places. Graphics could be pretty good (postscript printers, ca 1000 × 1000 pixel screens), but often was not, and mono text terminals were still widespread. C was beginning to be used, as more portable than Fortran. (Few PC Fortran compilers then and now.) Still SAS, SPSS etc as batch programs. S beginning to be make an impact on research and teaching. 2001 Little spread in machine speed (min 500MHz, max 1.5GHz), fast FPUs are universal. Colour everywhere, usually 24-bit colour. The video-games generation is now at university. Few people would dream of writing a complete program for a research idea: prototype and distribute in a higher-level language such as S or Matlab or Gauss or Ox or ...
    [Show full text]
  • Misuse of Statistics in Surgical Literature
    Statistics Corner Misuse of statistics in surgical literature Matthew S. Thiese1, Brenden Ronna1, Riann B. Robbins2 1Rocky Mountain Center for Occupational & Environment Health, Department of Family and Preventive Medicine, 2Department of Surgery, School of Medicine, University of Utah, Salt Lake City, Utah, USA Correspondence to: Matthew S. Thiese, PhD, MSPH. Rocky Mountain Center for Occupational & Environment Health, Department of Family and Preventive Medicine, School of Medicine, University of Utah, 391 Chipeta Way, Suite C, Salt Lake City, UT 84108, USA. Email: [email protected]. Abstract: Statistical analyses are a key part of biomedical research. Traditionally surgical research has relied upon a few statistical methods for evaluation and interpretation of data to improve clinical practice. As research methods have increased in both rigor and complexity, statistical analyses and interpretation have fallen behind. Some evidence suggests that surgical research studies are being designed and analyzed improperly given the specific study question. The goal of this article is to discuss the complexities of surgical research analyses and interpretation, and provide some resources to aid in these processes. Keywords: Statistical analysis; bias; error; study design Submitted May 03, 2016. Accepted for publication May 19, 2016. doi: 10.21037/jtd.2016.06.46 View this article at: http://dx.doi.org/10.21037/jtd.2016.06.46 Introduction the most commonly used statistical tests of the time (6,7). Statistical methods have since become more complex Research in surgical literature is essential for furthering with a variety of tests and sub-analyses that can be used to knowledge, understanding new clinical questions, as well as interpret, understand and analyze data.
    [Show full text]
  • Supplementary Materials
    Tomic et al, SIMON, an automated machine learning system reveals immune signatures of influenza vaccine responses 1 Supplementary Materials: 2 3 Figure S1. Staining profiles and gating scheme of immune cell subsets analyzed using mass 4 cytometry. Representative gating strategy for phenotype analysis of different blood- 5 derived immune cell subsets analyzed using mass cytometry in the sample from one donor 6 acquired before vaccination. In total PBMC from healthy 187 donors were analyzed using 7 same gating scheme. Text above plots indicates parent population, while arrows show 8 gating strategy defining major immune cell subsets (CD4+ T cells, CD8+ T cells, B cells, 9 NK cells, Tregs, NKT cells, etc.). 10 2 11 12 Figure S2. Distribution of high and low responders included in the initial dataset. Distribution 13 of individuals in groups of high (red, n=64) and low (grey, n=123) responders regarding the 14 (A) CMV status, gender and study year. (B) Age distribution between high and low 15 responders. Age is indicated in years. 16 3 17 18 Figure S3. Assays performed across different clinical studies and study years. Data from 5 19 different clinical studies (Study 15, 17, 18, 21 and 29) were included in the analysis. Flow 20 cytometry was performed only in year 2009, in other years phenotype of immune cells was 21 determined by mass cytometry. Luminex (either 51/63-plex) was performed from 2008 to 22 2014. Finally, signaling capacity of immune cells was analyzed by phosphorylation 23 cytometry (PhosphoFlow) on mass cytometer in 2013 and flow cytometer in all other years.
    [Show full text]
  • United Nations Fundamental Principles of Official Statistics
    UNITED NATIONS United Nations Fundamental Principles of Official Statistics Implementation Guidelines United Nations Fundamental Principles of Official Statistics Implementation guidelines (Final draft, subject to editing) (January 2015) Table of contents Foreword 3 Introduction 4 PART I: Implementation guidelines for the Fundamental Principles 8 RELEVANCE, IMPARTIALITY AND EQUAL ACCESS 9 PROFESSIONAL STANDARDS, SCIENTIFIC PRINCIPLES, AND PROFESSIONAL ETHICS 22 ACCOUNTABILITY AND TRANSPARENCY 31 PREVENTION OF MISUSE 38 SOURCES OF OFFICIAL STATISTICS 43 CONFIDENTIALITY 51 LEGISLATION 62 NATIONAL COORDINATION 68 USE OF INTERNATIONAL STANDARDS 80 INTERNATIONAL COOPERATION 91 ANNEX 98 Part II: Implementation guidelines on how to ensure independence 99 HOW TO ENSURE INDEPENDENCE 100 UN Fundamental Principles of Official Statistics – Implementation guidelines, 2015 2 Foreword The Fundamental Principles of Official Statistics (FPOS) are a pillar of the Global Statistical System. By enshrining our profound conviction and commitment that offi- cial statistics have to adhere to well-defined professional and scientific standards, they define us as a professional community, reaching across political, economic and cultural borders. They have stood the test of time and remain as relevant today as they were when they were first adopted over twenty years ago. In an appropriate recognition of their significance for all societies, who aspire to shape their own fates in an informed manner, the Fundamental Principles of Official Statistics were adopted on 29 January 2014 at the highest political level as a General Assembly resolution (A/RES/68/261). This is, for us, a moment of great pride, but also of great responsibility and opportunity. In order for the Principles to be more than just a statement of noble intentions, we need to renew our efforts, individually and collectively, to make them the basis of our day-to-day statistical work.
    [Show full text]
  • The Split-Apply-Combine Strategy for Data Analysis
    JSS Journal of Statistical Software April 2011, Volume 40, Issue 1. http://www.jstatsoft.org/ The Split-Apply-Combine Strategy for Data Analysis Hadley Wickham Rice University Abstract Many data analysis problems involve the application of a split-apply-combine strategy, where you break up a big problem into manageable pieces, operate on each piece inde- pendently and then put all the pieces back together. This insight gives rise to a new R package that allows you to smoothly apply this strategy, without having to worry about the type of structure in which your data is stored. The paper includes two case studies showing how these insights make it easier to work with batting records for veteran baseball players and a large 3d array of spatio-temporal ozone measurements. Keywords: R, apply, split, data analysis. 1. Introduction What do we do when we analyze data? What are common actions and what are common mistakes? Given the importance of this activity in statistics, there is remarkably little research on how data analysis happens. This paper attempts to remedy a very small part of that lack by describing one common data analysis pattern: Split-apply-combine. You see the split-apply- combine strategy whenever you break up a big problem into manageable pieces, operate on each piece independently and then put all the pieces back together. This crops up in all stages of an analysis: During data preparation, when performing group-wise ranking, standardization, or nor- malization, or in general when creating new variables that are most easily calculated on a per-group basis.
    [Show full text]
  • Hadley Wickham, the Man Who Revolutionized R Hadley Wickham, the Man Who Revolutionized R · 51,321 Views · More Stats
    12/15/2017 Hadley Wickham, the Man Who Revolutionized R Hadley Wickham, the Man Who Revolutionized R · 51,321 views · More stats Share https://priceonomics.com/hadley-wickham-the-man-who-revolutionized-r/ 1/10 12/15/2017 Hadley Wickham, the Man Who Revolutionized R “Fundamentally learning about the world through data is really, really cool.” ~ Hadley Wickham, prolific R developer *** If you don’t spend much of your time coding in the open-source statistical programming language R, his name is likely not familiar to you -- but the statistician Hadley Wickham is, in his own words, “nerd famous.” The kind of famous where people at statistics conferences line up for selfies, ask him for autographs, and are generally in awe of him. “It’s utterly utterly bizarre,” he admits. “To be famous for writing R programs? It’s just crazy.” Wickham earned his renown as the preeminent developer of packages for R, a programming language developed for data analysis. Packages are programming tools that simplify the code necessary to complete common tasks such as aggregating and plotting data. He has helped millions of people become more efficient at their jobs -- something for which they are often grateful, and sometimes rapturous. The packages he has developed are used by tech behemoths like Google, Facebook and Twitter, journalism heavyweights like the New York Times and FiveThirtyEight, and government agencies like the Food and Drug Administration (FDA) and Drug Enforcement Administration (DEA). Truly, he is a giant among data nerds. *** Born in Hamilton, New Zealand, statistics is the Wickham family business: His father, Brian Wickham, did his PhD in the statistics heavy discipline of Animal Breeding at Cornell University and his sister has a PhD in Statistics from UC Berkeley.
    [Show full text]
  • The Numbers Game - the Use and Misuse of Statistics in Civil Rights Litigation
    Volume 23 Issue 1 Article 2 1977 The Numbers Game - The Use and Misuse of Statistics in Civil Rights Litigation Marcy M. Hallock Follow this and additional works at: https://digitalcommons.law.villanova.edu/vlr Part of the Civil Procedure Commons, Civil Rights and Discrimination Commons, Evidence Commons, and the Labor and Employment Law Commons Recommended Citation Marcy M. Hallock, The Numbers Game - The Use and Misuse of Statistics in Civil Rights Litigation, 23 Vill. L. Rev. 5 (1977). Available at: https://digitalcommons.law.villanova.edu/vlr/vol23/iss1/2 This Article is brought to you for free and open access by Villanova University Charles Widger School of Law Digital Repository. It has been accepted for inclusion in Villanova Law Review by an authorized editor of Villanova University Charles Widger School of Law Digital Repository. Hallock: The Numbers Game - The Use and Misuse of Statistics in Civil Righ 1977-19781 THE NUMBERS GAME - THE USE AND MISUSE OF STATISTICS IN CIVIL RIGHTS LITIGATION MARCY M. HALLOCKt I. INTRODUCTION "In the problem of racial discrimination, statistics often tell much, and Courts listen."' "We believe it evident that if the statistics in the instant matter represent less than a shout, they certainly constitute '2 far more than a mere whisper." T HE PARTIES TO ACTIONS BROUGHT UNDER THE CIVIL RIGHTS LAWS3 have relied increasingly upon statistical 4 analyses to establish or rebut cases of unlawful discrimination. Although statistical evidence has been considered significant in actions brought to redress racial discrimination in jury selection,5 it has been used most frequently in cases of allegedly discriminatory t B.A., University of Pennsylvania, 1972; J.D., Georgetown University Law Center, 1975.
    [Show full text]
  • Sources of False Positives and False Negatives in the STATCHECK 1 Algorithm: Reply to Nuijten Et Al
    Cite as: Schmidt, T. (2017). Sources of false positives and false negatives in the STATCHECK 1 algorithm: Reply to Nuijten et al. (2016). arXiv:1610.01010v8 [q-bio.NC] Sources of false positives and false negatives in the STATCHECK algorithm: Reply to Nuijten et al. (2016) arXiv:1610.01010v8 [q-bio.NC] Thomas Schmidt University of Kaiserslautern, Germany Experimental Psychology Unit www.sowi.uni-kl.de/psychologie [email protected] Abstract STATCHECK is an R algorithm designed to scan papers automatically for inconsistencies between test statistics and their associated p values (Nuijten et al., 2016). The goal of this comment is to point out an important and well-documented flaw in this busily applied algorithm: It cannot handle corrected p values. As a result, statistical tests applying appropriate corrections to the p value (e.g., for multiple tests, post-hoc tests, violations of assumptions, etc.) are likely to be flagged as reporting inconsistent statistics, whereas papers omitting necessary corrections are certified as correct. The STATCHECK algorithm is thus valid for only a subset of scientific papers, and conclusions about the quality or integrity of statistical reports should never be based solely on this program. STATCHECK is an R algorithm designed to scan papers automatically for inconsistencies between test statistics and their reported p values (Nuijten, Hartgerink, van Assen, Epskamp, and Wicherts, 2016). The algorithm is currently part of these authors’ project to automatically scan thousands of scientific psychology papers and post the results on a website when the program detects some discrepancy, essentially flagging papers for reporting inconsistent test statistics.
    [Show full text]
  • Misuse of Statistics
    MISUSE OF STATISTICS Author: Rahul Dodhia Posted: May 25, 2007 Last Modified: October 15, 2007 This article is continuously updated. For the latest version, please go to www.RavenAnalytics.com/articles.php INTRODUCTION Percent Return on Investment 40 Did you know that 54% of all statistics are made up on the 30 spot? 20 Okay, you may not have fallen for that one, but there are 10 plenty of real-life examples that bait the mind. For 0 example, data from a 1988 census suggest that there is a high correlation between the number of churches and the year1 year2 number of violent crimes in US counties. The implied year3 Group B year4 Group A message from this correlation is that religion and crime are linked, and some would even use this to support the preposterous sounding hypothesis that religion causes FIGURE 1 crimes, or there is something in the nature of people that makes the two go together. That would be quite shocking, Here is the same data in more conventional but less pretty but alert statisticians would immediately point out that it format. Now it is clear that Fund A outperformed Fund B is a spurious correlation. Counties with a large number of in 3 out of 4 years, not the other way around. churches are likely to have large populations. And the larger the population, the larger the number of crimes.1 40 Percent Return on Investment Statistical literacy is not a skill that is widely accepted as Group A Group B necessary in education. Therefore a lot of misuse of 30 statistics is not intentional, just uninformed.
    [Show full text]
  • R Generation [1] 25
    IN DETAIL > y <- 25 > y R generation [1] 25 14 SIGNIFICANCE August 2018 The story of a statistical programming they shared an interest in what Ihaka calls “playing academic fun language that became a subcultural and games” with statistical computing languages. phenomenon. By Nick Thieme Each had questions about programming languages they wanted to answer. In particular, both Ihaka and Gentleman shared a common knowledge of the language called eyond the age of 5, very few people would profess “Scheme”, and both found the language useful in a variety to have a favourite letter. But if you have ever been of ways. Scheme, however, was unwieldy to type and lacked to a statistics or data science conference, you may desired functionality. Again, convenience brought good have seen more than a few grown adults wearing fortune. Each was familiar with another language, called “S”, Bbadges or stickers with the phrase “I love R!”. and S provided the kind of syntax they wanted. With no blend To these proud badge-wearers, R is much more than the of the two languages commercially available, Gentleman eighteenth letter of the modern English alphabet. The R suggested building something themselves. they love is a programming language that provides a robust Around that time, the University of Auckland needed environment for tabulating, analysing and visualising data, one a programming language to use in its undergraduate statistics powered by a community of millions of users collaborating courses as the school’s current tool had reached the end of its in ways large and small to make statistical computing more useful life.
    [Show full text]