Research Development & Support Series: Rigor and Reproducibility
Total Page:16
File Type:pdf, Size:1020Kb
3/10/2020 Office of Research Research Development and Support Series Rigor and Reproducibility: Focus on the NAE Report Tuesday, February 25, 2020 University Hall, Room 454 Overview Introductions 1 3/10/2020 Our Experts • Jacinda Dariotis, Ph.D., Professor of Education and Director of Evaluation Services Center • Tiffany Grant, Ph.D., Assistant Director for Research and Informatics and Associate Librarian • Ken Greis, Ph.D., Professor of Cancer and Cell Biology and Associate Dean • Jane Strasser, Ph.D., Sr. AVP, Office of Research Integrity Just Released 2 3/10/2020 Rigor and Reproducibility 25 February 2020 Ken Greis, Ph.D. Jacinda Dariotis, Ph.D. Tiffany Grant, Ph.D. Jane Strasser, Ph.D. http://nap.edu/21896 2017 http://nap.edu/25303 2017 2017 2019 3 3/10/2020 Rigor & Reproducibility An Overview 4 3/10/2020 Some practical insight/impact into Rigor and A few short commentaries worth reading Reproducibility in Research Reporting in Biomedical Begley and Ellis. Raise standards for preclinical cancer research (2012) Nature, 483, 531–533, doi:10.1038/483531a Sciences. Collins and Tabak. NIH plans to enhance reproducibility (2014) Nature, 505, 612–613, doi:10.1038/505612a Learning objectives: Babic et al. eLife 2019; 8:e41676. DOI: https://doi.org/10.7554/eLife.41676 1. Review some of the basic concepts Responsible Conduct in Research (RCR) and Rigor and Reproducibility Other Reference Material: 2. Understand the research landscape that has led to a re‐evaluation of Enhancing Research Reproducibility: Recommendations from the Federation of American Societies for Experimental Biology Effective January 14, 2016. reporting standards. https://www.faseb.org/Portals/2/PDFs/opa/2016/FASEB_Enhancing%20Research%20Re 3. Become familiar with the new and emerging standards of research producibility.pdf reporting as adopted by the NIH, publishing groups, and a variety of Scientific Utopia II. Restructuring Incentives and Practices to Promote Truth Over scientific societies (e.g. FASEB). Publishability. Nosek, Spies and Motyl Outline: doi: 10.1177/1745691612459058. Perspectives on Psychological Science November 1. Review of RCR principles with some examples of common challenges 2012 vol. 7 no. 6 615‐631 in reporting of data NIH policy: Rigor and Transparency Module 1 2. The “New” Problem—Rigor and Reproducibility in Research Reporting https://grants.nih.gov/reproducibility/module_1/presentation.html. 30 min video 3. Academic pressures that may have contributed to The Problem outlining NIH policy and Practice 4. Introduce new FASEB recommendation and NIH guidelines Case study on Research Integrity—Michael C. Lauer, NIH Deputy director for extramural 5. Real situations: How would you respond? How to minimize problems. research May 22, 2019 (https://youtu.be/ZKwpe77iZws) Responsible conduct of research (RCR) review* Research Practices can all be lumped into the Definition: “…the practice of scientific investigation with integrity. It involves concept that someone skilled in the field the awareness and application of established professional norms and ethical should be able to accurately reproduce principles in the performance of all activities related to scientific research.” another person’s research results. This implies that: [underline added for emphasis]. Regulatory Accurate record keeping of experimental Subject Matter: rules • conflict of interest –personal, professional, and financial design, execution, collection of experimental • policies regarding humans and animals subjects in research, and safe results, data analysis and reporting of the laboratory practices method and results is essential. • mentor/mentee responsibilities and relationships Research • collaborative research including collaborations with industry Practice • peer review • data acquisition and laboratory tools; management, sharing and ownership • research misconduct and policies for handling misconduct • responsible authorship and publication • the scientist as a responsible member of society, contemporary ethical *https://grants.nih.gov/grants/guide/notice‐ issues in biomedical research, and the environmental and societal impacts files/NOT‐OD‐10‐019.html 10 of scientific research 5 3/10/2020 One of the alarms: The Saga of Serum Biomarkers of Ovarian Cancer 1. Petricoin et al. Use of proteomic patterns in serum to identify ovarian cancer. (2002) THE LANCET 359, 572‐577. “Findings: [Comparative Mass Spectral profiles of serum samples]…identified a cluster pattern that, in the training set, completely segregated cancer from non‐cancer… [with] a sensitivity of 100% (95% CI 93–100), specificity of 95% (87–99), and positive predictive value of 94% (84–99).” Correlogic, LLC in 2003 licenses OvaCheck to LabCorp and Quest Diagnostic to screen and diagnose Ovarian Cancer from serum samples (even at early stages). 2. Baggerly KA, et al. Reproducibility of SELDI‐TOF protein patterns in serum: comparing datasets from different experiments. Bioinformatics. 2004 Mar 22;20(5):777‐85. Rigorous biostatistics models applied to the above data set found: 1. the results were generally non‐reproducible 2. There was evidence of a major shift in protocol mid‐experiment that seems to bias the work. 3. feature solely associate with the noise region of the spectra were responsible for the correlation. 3. FDA does not approve OvaCheck as a diagnostic test in 2004. 4. Later studies showed that much of the distinguishing noise was related to uncontrolled factors that biased the results: the serum collection tubes, spectral collection variability, and analysis at a single location. 5. Early Detection Research Network (EDRN) established at NCI to address best practices to avoid data bias in biomarker discovery and validation studies.—Robust Biostats and analysis of potential biases are a must! 11 The “New” Problem: • More than 50% of published studies from academic laboratories cannot be replicated according to venture capital firms for biomedical research • From: Scientific Utopia II. Restructuring Incentives and Practices to Promote Truth Over Publishability Brian A. Nosek, Jeffrey R. Spies and Matt Motyl. doi: 10.1177/1745691612459058. Perspectives on Psychological Science November 2012 vol. 7 no. 6 615‐631 • Only about 25% of published preclinical studies based on literature reports could be confirmed by Bayer scientist prior to moving forward with drug studies. • Prinz, F., Schlange, T. & Asadullah, K. Nature Rev. Drug Discov. 10, 712 (2011) • Only 6 of 53 (11%) of preclinical cancer studies that were published as landmark finding could be reproduced by scientist at Amgen. • Begley and Ellis, (2012) Nature 483, 531. Questions: On the laboratory scale, how many of you have tried to reproduce a finding and have been unsuccessful? Why do you think this is the case? 12 6 3/10/2020 The Problem continued Two‐thirds of clinical trials are not shared publicly within two years of completion. Study results are neither reported on the government website dedicated to that purpose, clinicaltrials.gov, nor published in a medical journal. Academic medicine has fostered a culture in which the sharing of results is considered discretionary, rather than mandatory. Question: Would you agree to be in a study with the knowledge that the results wouldn't be shared? http://www.npr.org/sections/health‐shots/2016/02/23/467712481/academic‐medical‐centers‐get‐an‐f‐in‐ sharing‐research‐results 13 What factors may contribute to The Problem? Group 1: There is a Disconnect Between What Is Good for Scientists and what Is Good for Science Are there any disincentives to producing high quality unbiased science? Group 2: Novelty and Positive Results Are Vital for Publishability, but Not always for the full Truth What do you think is meant by this statement? Is it valid? Group 3: Science is “self‐correcting” over the long term: Is this valid? What are some of the barriers to this concept? Scientific Utopia II. Restructuring Incentives and Practices to Promote Truth Over Publishability. Nosek, Spies and Motyl 14 doi: 10.1177/1745691612459058. Perspectives on Psychological Science November 2012 vol. 7 no. 6 615‐631 7 3/10/2020 15 Enhancing Research Reproducibility: Recommendations from the Federation of American Societies for Experimental Biology (FASEB). Background: Science advances through the publication of novel results, followed by efforts to reproduce them. Such replication of experimental findings distinguishes science from other forms of intellectual inquiry. Today, as we learn more about the complexity of living organisms, both successful and failed attempts to replicate a given study can provide valuable insights into biological processes. We are also gaining greater understanding of the many factors that can affect the outcomes of experiments. What are some of the factors that effect Research Reproducibility? Enhancing Research Reproducibility: Recommendations from the Federation of American Societies for Experimental Biology Effective January 16 14, 2016. https://www.faseb.org/Portals/2/PDFs/opa/2016/FASEB_Enhancing%20Research%20Reproducibility.pdf 8 3/10/2020 Enhancing Research Reproducibility: Recommendations from the Federation of American Societies for Experimental Biology (FASEB) Background: Variability of reagents, model systems, methods, and resources is difficult to avoid—particularly in biology—and can have a sizeable effect on experimental outcomes. Conclusions: 12 recommendations were put forth that address common terminology, details of materials and methods, training on keeping proper records, experimental