SUPPLEMENTARY INFORMATION COMMENT Supplementary information to: Retire statistical significance Valentin Amrhein et al. Supplementary text to a Comment published in Nature 567, 305–307 (2019) https://doi.org/10.1038/d41586-019-00857-9 SUPPLEMENTARY INFORMATION | NATURE | 1 51% (402/791) articles from five journals erroneously interpret statistically non-significant results as indicating “no effect” Most recent Number of Number of Survey* time articles with articles con- Error criterion period** errors sidered*** Schatz P, Jay KA, McComb J, McLaughlin JR (2005). “using statistical tests to confirm the null, that Misuse of statistical tests in Archives of Clinical Neu- 81 170 there is no difference between groups.” 2001-2004 ropsychology publications. Archives of Clinical Neu- (Page 1057) (Page 1057) (Page 1054) ropsychology 20:1053-1059 Fidler F, Burgman MA, Cumming G, Buttrose R, Thomason N. (2006). Impact of criticism of null- 42 100 “statistically nonsignificant results were interpreted 2005 hypothesis significance testing on statistical report- (Table 1) (Table 1) as evidence of ‘no effect’ or ‘no relationship’ “ ing practices in conservation biology. Conservation (Page 1542) Biology 20:1539-1544 “Phrases such as ‘there is no effect,’ ‘there was no evidence for’ (combined with an effect in the Hoekstra R, Finch S, Kiers HAL, Johnson A. 145 expected direction), ‘the nonexistence of the effect,’ (2006). Probability as certainty: dichotomous think- 259 2002-2004 (Table 1; ‘no effect was found,’ ‘are equally affected,’ ‘there ing and the misuse of p values. Psychonomic Bulletin (Table 1) 60% of 242) was no main effect,’ ‘A and B did not differ,’ or ‘the & Review 13:1033-1037 significance test reveals that there is no difference’ “ (Pages 1034-1035) Bernardi F, Chakhaia L, Leopold L. (2017). ‘Sing me 134 a song with social significance’: the (mis)use of sta- (Table 2; 262 “authors mechanically equate a statistically tistical significance testing in European sociological 2010-2014 100%-49% (Table 1) insignificant effect with a zero effect.” research. European Sociological Review 33:1-15 = 51% of 262) (Page 2) * All four surveys examined the journal in which they were published. Fidler et al. 2006 also examined Biological Conservation. ** All four surveys compared two or more distinct time periods; we provide data for only the most recent time period. *** Schatz et al. 2005 provide only the total number of articles considered. Fidler et al. 2006 and Hoekstra et al. 2006 provide the total number of articles considered and the number of articles that contained a statistically non-significant esultr and were thus eligible to make the error. Bernardi et al. 2017 provide only the total number of “articles qualifying for the review.” For consistency across all four surveys, we use the total number of articles considered as the denominator when computing the percentage making an error; this is a conservatively low estimate of the proportion of articles that misinterpret statistically non-significant results because it includes articles ineligible to make the error (i.e., because they do not contain a statistically non-significant result) in the denominator. Supplementary information to: Retire statistical significance (Comment in Nature 567, 305–307; 2019) Full list of co-signatories 854 scientists from 52 countries are signatories to "Retire statistical significance" Compiled by Lilla Lovász, Zoological Institute, University of Basel, Switzerland Peter Aaby [email protected] Bandim Health Project, Bissau, Guinea-Bissau Kevin Aagaard [email protected] Colorado Parks and Wildlife, Fort Collins, CO, USA Preben Aavitsland [email protected] Division of Infection Control and Environmental Health, Norwegian Institute of Public Health, Oslo, Norway Paul Acker [email protected] School of Biological Sciences, University of Aberdeen, Aberdeen, UK Mohd Bakri Adam [email protected] Institute for Mathematical Research, University Putra Malaysia, Selangor, Malaysia Amin Adibi [email protected] Faculty of Pharmaceutical Sciences, University of British Columbia, Vancouver, BC, Canada Matthew Agler [email protected] Department of Microbiology, Friedrich-Schiller Univsrity Jena, Jena, Germany Daniel Aguirre-Acevedo [email protected] Medical Research Institute, Medicine School, Unversidad de AntioQuia, Medellín, Colombia Thomas P. Ahern [email protected] Department of Surgery, Larner College of Medicine, University of Vermont, Burlington, VT, USA Saiam Ahmed [email protected] MRC Clinical Trials Unit, Institute of Clinical Trials and Methodology, University College London, London, UK Jeffrey Akiki [email protected] School of Professional Studies, Columbia University, New York, NY, USA Teddy J. Akiki [email protected] Department of Psychiatry, Yale University School of Medicine, New Haven, CT, USA Yasser Albogami [email protected] Clinical Pharmacy Department, King Saud University, Riyadh, Saudi Arabia Robert W Aldridge [email protected] Institute of Health Informatics, University College London, London, UK Ayesha S. Ali [email protected] Cancer Research UK Clinical Trials Unit, University of Birmingham, Edgbaston, Birmingham, UK Anna Alińska [email protected] Department of Psychology, University of Warsaw, Warsaw, Poland Nico Alioravainen [email protected] Department of Environmental and Biological Sciences, Faculty of Science and Forestry, University of Eastern Finland, Kuopio, Finland Christian L. Althaus [email protected] Institute of Social and Preventive Medicine, University of Bern, Bern, Switzerland Jonathan Amburgey [email protected] Department of Psychology, Westminster College, Salt Lake City, UT, USA Avnika B. Amin [email protected] Department of Epidemiology, Emory University, Atlanta, GA, USA Shobeir AmirneQuiee [email protected] Ivey Business School, University of Western Ontario, London, ON, Canada Rune Martens Andersen [email protected] Centre for Cancer and Organ Diseases, Rigshospitalet, Copenhagen, Denmark Elizabeth B. Andrews [email protected] Research Triangle Institute, Research Triangle Park, NC, USA Peter D. Angevine [email protected] Department of Neurological Surgery, Vagelos College of Physicians and Surgeons, Columbia University, New York, NY, USA Nils Anthes [email protected] Institute of Evolution and Ecology, University of Tuebingen, Tuebingen, Germany Onyebuchi A. Arah [email protected] Department of Epidemiology, University of California - Los Angeles, Los Angeles, CA, USA Gianluigi Ardissino [email protected] Department of Pediatrics, Fondazione IRCCS Ca' Granda Ospedale Maggiore Policlinico, Milano, Italy Cristina Ardura-Garcia [email protected] Institute of Social and Preventive Medicine (ISPM), University of Bern, Bern, Switzerland Corson N. Areshenkoff [email protected] Centre for Neuroscience Studies, Queens University, Kingston, ON, Canada Cono Ariti [email protected] Cardiff University Medical School, Cardiff, UK Kellyn F. Arnold [email protected] Leeds Institute for Data Analytics, University of Leeds, Leeds, UK Jaan Aru [email protected] Institute of Biology, Humboldt University of Berlin, Berlin, Germany Ann Aschengrau [email protected] Department of Epidemiology, Boston University School of Public Health, Boston, MA, USA Peter H. Asdahl [email protected] Department of Hematology, Aarhus University Hospital, Aarhus, Denmark Deborah Ashby [email protected] School of Public Health, Imperial College London, London, UK Fortune Atri [email protected] Kaiser Permanente Medical Group, Dept of Psychiatry, Los Angeles, CA, USA Reto Auer [email protected] Institute of Primary Health Care (BIHAM), Universtiy of Bern, Bern, Switzerland Matthieu Authier [email protected] Observatoire PELAGIS - UMS 3462, CNRS-LRU, La Rochelle Université, La Rochelle, France Ignacio Avellino [email protected] Sorbonne Université, Institut des Systèmes Intelligents et de RobotiQue (ISIR), CNRS, Paris, France Marc T. Avey [email protected] Public Health Agency of Canada, Ottawa, ON, Canada Eli Awtrey [email protected] Department of Management, Carl H. Lindner College of Business, University of Cincinnati, Cincinnati, OH, USA Flavio Azevedo [email protected] Cologne University, Cologne, Germany; Social Justice Lab, New York University, New York, NY, USA Rasmus Bååth [email protected] Lund University, Lund, Sweden Marko Bachl [email protected] University of Hohenheim, Stuttgart, Germany Lance Bachmeier [email protected] Department of Economics, Kansas State University, Manhattan, KS, USA Kathleen E. Bachynski [email protected] Department of Medicine, NYU Langone Health, New York, NY, USA J. Michael Bailey [email protected] Department of Psychology, Northwestern University, Evanston, IL, USA Emmanuel S. Baja [email protected] Institute of Clinical Epidemiology, NIH University of the Philippines-Manila, Manila, Philippines Andrew M. Baker [email protected] Marketing Department, Fowler College of Business, San Diego State University, San Diego, CA, USA Daniel Hart Baker [email protected] Department of Psychology, University of York, York, UK Nekane Balluerka [email protected] University of the Basque Country, Donostia, Spain Layla J. Barkal [email protected] Department of Medicine, Stanford University School of Medicine, Palo Alto, CA, USA Carlos Javier Barrera-Causil [email protected] Faculty of Applied and