THE NATIONAL ACADEMIES PRESS This PDF is available at http://nap.edu/25303 SHARE Reproducibility and Replicability in Science (2019) DETAILS 218 pages | 6 x 9 | PAPERBACK ISBN 978-0-309-48616-3 | DOI 10.17226/25303 CONTRIBUTORS GET THIS BOOK Committee on Reproducibility and Replicability in Science; Committee on National Statistics; Board on Behavioral, Cognitive, and Sensory Sciences; Division of Behavioral and Social Sciences and Education; Nuclear and Radiation Studies FIND RELATED TITLES Board; Division on Earth and Life Studies; Committee on Applied and Theoretical Statistics; Board on Mathematical Sciences and Analytics; Division on SUGGESTEDEngineering an CITATIONd Physical Sciences; Committee on Science, Engineering, Medicine, Nanadti oPnuabl lAicc Paodleicmyi;e Bso oafr Sd coienn Rcese, aErncghi nDeaetrain agn, da nIndf oMrmedaitcioinne; P20o1lic9y. Ranedp rGodloubcaibl ility aAnffda iRrse; pNliactaiobniliatyl Ainc aSdceiemnicees. oWf aSschieinngcteosn,, EDnCg:i nTeheer inNga,t iaonnda lM Aecdaidceinmeies Press. https://doi.org/10.17226/25303. Visit the National Academies Press at NAP.edu and login or register to get: – Access to free PDF downloads of thousands of scientific reports – 10% off the price of print titles – Email or social media notifications of new titles related to your interests – Special offers and discounts Distribution, posting, or copying of this PDF is strictly prohibited without written permission of the National Academies Press. (Request Permission) Unless otherwise indicated, all materials in this PDF are copyrighted by the National Academy of Sciences. Copyright © National Academy of Sciences. All rights reserved. Reproducibility and Replicability in Science Prepublication copy, uncorrected proofs Reproducibility and Replicability in Science Committee on Reproducibility and Replicability in Science Division of Behavioral and Social Sciences and Education Board on Behavioral, Cognitive, and Sensory Sciences Committee on National Statistics Division on Earth and Life Studies Nuclear and Radiation Studies Board Division on Engineering and Physical Sciences Board on Mathematical Sciences and Analytics Committee on Applied and Theoretical Statistics Policy and Global Affairs Board on Research Data and Information Committee on Science, Engineering, Medicine, and Public Policy A Consensus Study Report of Copyright National Academy of Sciences. All rights reserved. Reproducibility and Replicability in Science Prepublication copy, uncorrected proofs THE NATIONAL ACADEMIES PRESS 500 Fifth Street, NW Washington, DC 20001 This activity was supported by contracts between the National Academy of Sciences and Alfred P. Sloan Foundation (G-2018-10102), National Science Foundation (1743856). Support for the work of the Board on Behavioral, Cognitive, and Sensory Sciences is provided primarily by a grant from the National Science Foundation (Award No. BCS-1729167). Any opinions, findings, conclusions, or recommendations expressed in this publication do not necessarily reflect the views of any organization or agency that provided support for the project. Any opinions, findings, conclusions, or recommendations expressed in this publication do not necessarily reflect the views of any organization or agency that provided support for the project. International Standard Book Number-13: 978-0-309-XXXXX-X International Standard Book Number-10: 0-309-XXXXX-X Digital Object Identifier: https://doi.org/10.17226/25303 Additional copies of this publication are available for sale from the National Academies Press, 500 Fifth Street, NW, Keck 360, Washington, DC 20001; (800) 624-6242 or (202) 334- 3313; http://www.nap.edu. Copyright 2019 by the National Academy of Sciences. All rights reserved. Printed in the United States of America Suggested citation: National Academies of Sciences, Engineering, and Medicine. 2019. Reproducibility and Replicability in Science. Washington, DC: The National Academies Press. doi: https://doi.org/10.17226/25303. Copyright National Academy of Sciences. All rights reserved. Reproducibility and Replicability in Science Prepublication copy, uncorrected proofs The National Academy of Sciences was established in 1863 by an Act of Congress, signed by President Lincoln, as a private, nongovernmental institution to advise the nation on issues related to science and technology. Members are elected by their peers for outstanding contributions to research. Dr. Marcia McNutt is president. The National Academy of Engineering was established in 1964 under the charter of the National Academy of Sciences to bring the practices of engineering to advising the nation. Members are elected by their peers for extraordinary contributions to engineering. Dr. C. D. Mote, Jr., is president. The National Academy of Medicine (formerly the Institute of Medicine) was established in 1970 under the charter of the National Academy of Sciences to advise the nation on medical and health issues. Members are elected by their peers for distinguished contributions to medicine and health. Dr. Victor J. Dzau is president. The three Academies work together as the National Academies of Sciences, Engineering, and Medicine to provide independent, objective analysis and advice to the nation and conduct other activities to solve complex problems and inform public policy decisions. The National Academies also encourage education and research, recognize outstanding contributions to knowledge, and increase public understanding in matters of science, engineering, and medicine. Learn more about the National Academies of Sciences, Engineering, and Medicine at www.nationalacademies.org. Copyright National Academy of Sciences. All rights reserved. Reproducibility and Replicability in Science Prepublication copy, uncorrected proofs Consensus Study Reports published by the National Academies of Sciences, Engineering, and Medicine document the evidence-based consensus on the study’s statement of task by an authoring committee of experts. Reports typically include findings, conclusions, and recommendations based on information gathered by the committee and the committee’s deliberations. Each report has been subjected to a rigorous and independent peer-review process and it represents the position of the National Academies on the statement of task. Proceedings published by the National Academies of Sciences, Engineering, and Medicine chronicle the presentations and discussions at a workshop, symposium, or other event convened by the National Academies. The statements and opinions contained in proceedings are those of the participants and are not endorsed by other participants, the planning committee, or the National Academies. For information about other products and activities of the National Academies, please visit www.nationalacademies.org/about/whatwedo. Copyright National Academy of Sciences. All rights reserved. Reproducibility and Replicability in Science Prepublication copy, uncorrected proofs COMMITTEE ON REPRODUCIBILITY AND REPLICABILITY IN SCIENCE HARVEY V. FINEBERG,1 (Chair), Gordon and Betty Moore Foundation DAVID B. ALLISON,1 School of Public Health-Bloomington, Indiana University LORENA A. BARBA, School of Engineering and Applied Science, George Washington University DIANNE CHONG,2 Boeing Research and Technology (Retired) DAVID DONOHO, 3, 4 Department of Statistics, Stanford University JULIANA FREIRE, Tandon School of Engineering, New York University GERALD GABRIELSE,3 Department of Physics, Northwestern University CONSTANTINE GATSONIS, Center for Statistical Sciences, Brown University EDWARD HALL, Department of Philosophy, Harvard University THOMAS H. JORDAN,3 Department of Earth Sciences, University of Southern California DIETRAM A. SCHEUFELE, Madison and Morgridge Institute for Research, University of Wisconsin-Madison VICTORIA STODDEN, Institute for Data Sciences and Engineering, University of Illinois at Urbana-Champaign SIMINE VAZIRE,5 Department of Psychology, University of California, Davis TIMOTHY D. WILSON, Department of Psychology, University of Virginia WENDY WOOD, Department of Psychology, University of Southern California and INSEAD- Sorbonne University Study Staff JENNIFER HEIMBERG, Study Director THOMAS ARRISON, Program Director MICHAEL COHEN, Senior Program Officer MICHELLE SCHWALBE, Director, Board on Mathematical Sciences and Analytics ADRIENNE STITH BUTLER, Associate Board Director BARBARA A. WANCHISEN, Director, Board on Behavioral, Cognitive, and Sensory Sciences TINA WINTERS, Associate Program Officer REBECCA MORGAN, Senior Librarian THELMA COX, Program Coordinator (beginning January 2019) LESLEY WEBB, Program Assistant (September 2018 through January 2019) GARRET TYSON, Program Assistant (September 2017 through August 2018) ERIN HAMMERS FORSTAG, Consultant Writer 1 Member of the National Academy of Medicine 2 Member of the National Academy of Engineering 3 Member of the National Academy of Sciences 4 Resigned from the committee on July 24, 2018 5 Resigned from the committee on October 11, 2018 v Copyright National Academy of Sciences. All rights reserved. Reproducibility and Replicability in Science Prepublication copy, uncorrected proofs Division of Behavioral and Social Sciences and Education BOARD ON BEHAVIORAL, COGNITIVE, AND SENSORY SCIENCES SUSAN FISKE,1 (Chair), Princeton University JOHN BAUGH, Washington University in St. Louis LAURA CARTENSEN,2 Stanford University JUDY DUBNO, Medical University of South Carolina JENNIFER EBERHARDT,1 Stanford University WILSON S. GEISLER, 1 The University of Texas