DOCSLIB.ORG

1 Estimating the Prevalence of Transparency and Reproducibility

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
1 Estimating the Prevalence of Transparency and Reproducibility Estimating the prevalence of transparency and reproducibility-related research practices in psychology (2014-2017) Tom E. Hardwicke1,2, Robert T. Thibault3,4, Jessica E. Kosie5,6, Joshua D. Wallach7,8, Mallory C. Kidwell9, & John P. A. Ioannidis2,10,11 1 Department of Psychology, University of Amsterdam 2 Meta-Research Innovation Center Berlin (METRIC-B), QUEST Center for Transforming Biomedical Research, Berlin Institute of Health, Charité – Universitätsmedizin Berlin 3 School of Psychological Science, University of Bristol 4 MRC Integrative Epidemiology Unit at the University of Bristol 5 Department of Psychology, University of Oregon 6 Department of Psychology, Princeton University 7 Department of Environmental Health Sciences, Yale School of Public Health 8 Collaboration for Research Integrity and Transparency, Yale School of Medicine 9 Department of Psychology, University of Utah 10 Departments of Medicine, of Health Research and Policy, of Biomedical Data Science, and of Statistics, Stanford University 11 Meta-Research Innovation Center at Stanford (METRICS), Stanford University Author note Tom Hardwicke was based at Stanford University when the study began and is now based at the University of Amsterdam. Jessica Kosie was based at University of Oregon when the study began and is now based at Princeton University. Correspondence concerning this article should be addressed to Tom Hardwicke, Nieuwe Achtergracht 129B, Department of Psychology, University of Amsterdam, 1018 WT Amsterdam, The Netherlands. E-mail: [email protected] 1 Abstract Psychologists are navigating an unprecedented period of introspection about the credibility and utility of their discipline. Reform initiatives have emphasized the benefits of several transparency and reproducibility-related research practices; however, their adoption across the psychology literature is unknown. To estimate their prevalence, we manually examined a random sample of 250 psychology articles published between 2014-2017. Over half of the articles were publicly available (154/237, 65% [95% confidence interval, 59%-71%]); however, sharing of research materials (26/183, 14% [10%-19%]), study protocols (0/188, 0% [0%-1%]), raw data (4/188, 2% [1%-4%]), and analysis scripts (1/188, 1% [0%-1%]) was rare. Pre-registration was also uncommon (5/188, 3% [1%-5%]). Many articles included a funding disclosure statement (142/228, 62% [56%-69%]), but conflict of interest statements were less common (88/228, 39% [32%-45%]). Replication studies were rare (10/188, 5% [3%-8%]) and few studies were included in systematic reviews (21/183, 11% [8%-16%]) or meta-analyses (12/183, 7% [4%-10%]). Overall, the results suggest that transparency and reproducibility-related research practices were far from routine. These findings establish a baseline which can be used to assess future progress towards increasing the credibility and utility of psychology research. Keywords: transparency, reproducibility, meta-research, psychology, open science 2 Introduction Serious concerns about the credibility and utility of some scientific literature (Ioannidis, 2005; Ioannidis et al., 2014) have prompted calls for increased adoption of research practices that enhance reproducibility and transparency (Miguel et al., 2014; Munafò et al., 2017; Nosek et al., 2015; Wallach et al., 2018). Close scrutiny of psychology in particular has suggested that standard research and publication practices have rendered the discipline highly exposed to bias, potentially resulting in a large volume of exaggerated and misleading results (Ioannidis et al., 2014; John et al., 2012; Open Science Collaboration, 2015; Pashler & Wagenmakers, 2012; Simmons et al., 2011; Szucs & Ioannidis, 2017). This realization within the field has also led to a number of reform efforts (Hardwicke & Ioannidis, 2018a; Hardwicke et al., 2019; Nelson et al., 2018; Vazire, 2018), which have the potential to improve efficiency (Chalmers & Glasziou, 2009), facilitate self-correction (Ioannidis, 2012), and enhance credibility (Vazire, 2017). A central focus of reform initiatives has been to encourage scientists to share more information about the studies they perform. Journal articles are only the most visible facade of deeper layers of scholarship that may include protocols, original research materials, raw data, and analysis scripts – resources that are not necessarily shared with other scientists (Buckheit & Donoho, 1995; Klein et al., 2018). Even journal articles themselves may only be accessible to those with institutional access or the ability to pay a fee (Piwowar et al., 2018). Additionally, potential sources of bias, such as conflicts of interest and funding sources, may not be disclosed (Bekelman et al., 2003; Cristea & Ioannidis, 2018). However, there is a growing (or re-emerging, David, 2008) appreciation that the scientific community needs to be able to access all of this information in order to comprehensively evaluate, interpret, and independently verify scientific claims (Vazire, 2017; Munafò et al., 2017). Furthermore, access to this information enables 3 replication, evidence synthesis, and discovery activities that may ultimately accelerate scientific progress (Ioannidis, 2012; Klein et al., 2018). The burgeoning discipline of meta-research (‘research on research’) has already begun to evaluate the impact of various reform initiatives (Hardwicke et al., 2020). For example, journal data sharing policies have been associated with moderate to substantial increases in data sharing (Hardwicke et al., 2018; Kidwell et al., 2016; Nuijten et al., 2018; Rowhani-Farid & Barnett, 2016; Naudet et al., 2018). However, the prevalence of transparency and reproducibility-related research practices across the psychology literature is largely unknown. Based on previous investigations in biomedicine (Iqbal et al., 2016; Wallach et al., 2018) and the social sciences (Hardwicke et al., 2019), we manually examined a random sample of 250 articles to estimate the prevalence of several transparency and reproducibility-related indicators in psychology articles published between 2014-2017. The indicators were open access to published articles; availability of study materials, study protocols, raw data, and analysis scripts; pre-registration; disclosure of funding sources and conflicts of interest; conduct of replication studies; and cumulative synthesis of evidence in meta-analyses and systematic reviews. Methods Design This was a retrospective observational study with a cross-sectional design. Sampling units were individual articles. Measured variables are shown in Table 1. 4 Table 1. Measured variables. The variables measured for an individual article depended on the study design classification. For articles that were not available (the full text could not be retrieved) or non-English language, only article characteristics were obtained. The exact operational definitions and procedures for data extraction/coding are available in the structured form here: https://osf.io/x9rmy/ Applicable study designs Article characteristics Subject area, year of publication, study design, All country of origin (based on corresponding author's affiliation), human/animal subjects, 2017 journal impact factor (according to Thomson Reuters Journal Citation Reports) Articles Accessibility and Retrieval Method (can the All article be accessed, is there a public version or is paywall access required?) Protocols Availability statement (is availability, or lack of, Study designs involving primary data#, study explicitly declared?) designs involving secondary data (commentaries with analysis and meta-analyses). Content (what aspects of the study are included in the protocol?) Materials Availability statement (is availability, or lack of, Study designs involving primary data# explicitly declared?) Retrieval Method (e.g., upon request or via online repository) Accessibility (can the materials be accessed?) Raw data Availability statement (is availability, or lack of, explicitly declared?) 5 Retrieval Method (e.g., upon request or via online Study designs involving primary data#, study repository) designs involving secondary data (commentaries with analysis and meta-analyses). Accessibility (can the data be accessed?) Content (has all relevant data been shared?) Documentation (are the data understandable?) Analysis scripts Availability statement (is availability, or lack of, Study designs involving primary data#, study explicitly declared?) designs involving secondary data (commentaries with analysis and meta-analyses). Retrieval Method (e.g., upon request or via online repository) Accessibility (can the scripts be accessed?) Pre-registration Availability statement (is availability, or lack of, Study designs involving primary data#, study explicitly declared?) designs involving secondary data (commentaries with analysis and meta-analyses). Retrieval Method (which registry was used?) Accessibility (can the pre-registration be accessed?) Content (what was pre-registered?) Funding Disclosure statement (are funding sources, or lack All of, explicitly declared?) Conflicts of interest Disclosure statement (are conflicts of interest, or All lack of, explicitly declared?) Replication Statement (does the article claim to report a All replication?) 6 Citation history (has the article been cited by a Study designs involving primary data# study that claims to be a replication?) Evidence synthesis Meta-analysis citation history* (has the article Study designs involving primary data# been cited by, and included
Load more

Recommended publications
  • Reproducibility, Replicability, and Generalization in the Social, Behavioral, and Economic Sciences
    REPRODUCIBILITY, REPLICABILITY, AND GENERALIZATION IN THE SOCIAL, BEHAVIORAL, AND ECONOMIC SCIENCES Report of the Subcommittee on Replicability in Science of the SBE Advisory Committee to the National Science Foundation 13 May 2015 Presentation at SBE AC Spring Meeting by K. Bollen. Report of the Subcommittee on Replicability in Science Advisory Committee to the NSF SBE Directorate Subcommittee Members: Kenneth Bollen (University of North Carolina at Chapel Hill, Cochair) John T. Cacioppo (University of Chicago, Cochair) Robert M. Kaplan (Agency for Healthcare Research and Quality) Jon A. Krosnick (Stanford University) James L. Olds (George Mason University) Staff Assistance Heather Dean (National Science Foundation) TOPICS I. Background II. Subcommittee Report III. Definitions IV. Recommendations V. Final Comments Background • Spring, 2013 o NSF SBE Advisory Committee establishes subcommittee on how SBE can promote robust research practices • Summer & Fall 2013 o Subcommittee proposal for workshop on “Robust Research in the Social, Behavioral, and Economic Sciences.” • February 20-21, 2014 o Workshop convened o Participants drawn from variety of universities, funding agencies, scientific associations, and journals o Cover broad range of issues from extent and cause of problems to possible solutions o Details are in the appendix of the circulated report • Post-workshop period o Document & digest workshop content o Discuss and propose recommendations o Complete report Subcommittee Report • DEFINITIONS o No consensus in science on the meanings
    [Show full text]
  • FORMULAS from EPIDEMIOLOGY KEPT SIMPLE (3E) Chapter 3: Epidemiologic Measures
    FORMULAS FROM EPIDEMIOLOGY KEPT SIMPLE (3e) Chapter 3: Epidemiologic Measures Basic epidemiologic measures used to quantify: • frequency of occurrence • the effect of an exposure • the potential impact of an intervention. Epidemiologic Measures Measures of disease Measures of Measures of potential frequency association impact (“Measures of Effect”) Incidence Prevalence Absolute measures of Relative measures of Attributable Fraction Attributable Fraction effect effect in exposed cases in the Population Incidence proportion Incidence rate Risk difference Risk Ratio (Cumulative (incidence density, (Incidence proportion (Incidence Incidence, Incidence hazard rate, person- difference) Proportion Ratio) Risk) time rate) Incidence odds Rate Difference Rate Ratio (Incidence density (Incidence density difference) ratio) Prevalence Odds Ratio Difference Macintosh HD:Users:buddygerstman:Dropbox:eks:formula_sheet.doc Page 1 of 7 3.1 Measures of Disease Frequency No. of onsets Incidence Proportion = No. at risk at beginning of follow-up • Also called risk, average risk, and cumulative incidence. • Can be measured in cohorts (closed populations) only. • Requires follow-up of individuals. No. of onsets Incidence Rate = ∑person-time • Also called incidence density and average hazard. • When disease is rare (incidence proportion < 5%), incidence rate ≈ incidence proportion. • In cohorts (closed populations), it is best to sum individual person-time longitudinally. It can also be estimated as Σperson-time ≈ (average population size) × (duration of follow-up). Actuarial adjustments may be needed when the disease outcome is not rare. • In an open populations, Σperson-time ≈ (average population size) × (duration of follow-up). Examples of incidence rates in open populations include: births Crude birth rate (per m) = × m mid-year population size deaths Crude mortality rate (per m) = × m mid-year population size deaths < 1 year of age Infant mortality rate (per m) = × m live births No.
    [Show full text]
  • Statistical Guidance on Reporting Results from Studies Evaluating Diagnostic Tests Document Issued On: March 13, 2007
    Guidance for Industry and FDA Staff Statistical Guidance on Reporting Results from Studies Evaluating Diagnostic Tests Document issued on: March 13, 2007 The draft of this document was issued on March 12, 2003. For questions regarding this document, contact Kristen Meier at 240-276-3060, or send an e-mail to [email protected]. U.S. Department of Health and Human Services Food and Drug Administration Center for Devices and Radiological Health Diagnostic Devices Branch Division of Biostatistics Office of Surveillance and Biometrics Contains Nonbinding Recommendations Preface Public Comment Written comments and suggestions may be submitted at any time for Agency consideration to the Division of Dockets Management, Food and Drug Administration, 5630 Fishers Lane, Room 1061, (HFA-305), Rockville, MD, 20852. Alternatively, electronic comments may be submitted to http://www.fda.gov/dockets/ecomments. When submitting comments, please refer to Docket No. 2003D-0044. Comments may not be acted upon by the Agency until the document is next revised or updated. Additional Copies Additional copies are available from the Internet at: http://www.fda.gov/cdrh/osb/guidance/1620.pdf. You may also send an e-mail request to [email protected] to receive an electronic copy of the guidance or send a fax request to 240-276-3151 to receive a hard copy. Please use the document number 1620 to identify the guidance you are requesting. Contains Nonbinding Recommendations Table of Contents 1. Background....................................................................................................4
    [Show full text]
  • Estimated HIV Incidence and Prevalence in the United States
    Volume 26, Number 1 Estimated HIV Incidence and Prevalence in the United States, 2015–2019 This issue of the HIV Surveillance Supplemental Report is published by the Division of HIV/AIDS Prevention, National Center for HIV/AIDS, Viral Hepatitis, STD, and TB Prevention, Centers for Disease Control and Prevention (CDC), U.S. Department of Health and Human Services, Atlanta, Georgia. Estimates are presented for the incidence and prevalence of HIV infection among adults and adolescents (aged 13 years and older) based on data reported to CDC through December 2020. The HIV Surveillance Supplemental Report is not copyrighted and may be used and reproduced without permission. Citation of the source is, however, appreciated. Suggested citation Centers for Disease Control and Prevention. Estimated HIV incidence and prevalence in the United States, 2015–2019. HIV Surveillance Supplemental Report 2021;26(No. 1). http://www.cdc.gov/ hiv/library/reports/hiv-surveillance.html. Published May 2021. Accessed [date]. On the Web: http://www.cdc.gov/hiv/library/reports/hiv-surveillance.html Confidential information, referrals, and educational material on HIV infection CDC-INFO 1-800-232-4636 (in English, en Español) 1-888-232-6348 (TTY) http://wwwn.cdc.gov/dcs/ContactUs/Form Acknowledgments Publication of this report was made possible by the contributions of the state and territorial health departments and the HIV surveillance programs that provided surveillance data to CDC. This report was prepared by the following staff and contractors of the Division of HIV/AIDS Prevention, National Center for HIV/AIDS, Viral Hepatitis, STD, and TB Prevention, CDC: Laurie Linley, Anna Satcher Johnson, Ruiguang Song, Sherry Hu, Baohua Wu, H.
    [Show full text]
  • Understanding Relative Risk, Odds Ratio, and Related Terms: As Simple As It Can Get Chittaranjan Andrade, MD
    Understanding Relative Risk, Odds Ratio, and Related Terms: As Simple as It Can Get Chittaranjan Andrade, MD Each month in his online Introduction column, Dr Andrade Many research papers present findings as odds ratios (ORs) and considers theoretical and relative risks (RRs) as measures of effect size for categorical outcomes. practical ideas in clinical Whereas these and related terms have been well explained in many psychopharmacology articles,1–5 this article presents a version, with examples, that is meant with a view to update the knowledge and skills to be both simple and practical. Readers may note that the explanations of medical practitioners and examples provided apply mostly to randomized controlled trials who treat patients with (RCTs), cohort studies, and case-control studies. Nevertheless, similar psychiatric conditions. principles operate when these concepts are applied in epidemiologic Department of Psychopharmacology, National Institute research. Whereas the terms may be applied slightly differently in of Mental Health and Neurosciences, Bangalore, India different explanatory texts, the general principles are the same. ([email protected]). ABSTRACT Clinical Situation Risk, and related measures of effect size (for Consider a hypothetical RCT in which 76 depressed patients were categorical outcomes) such as relative risks and randomly assigned to receive either venlafaxine (n = 40) or placebo odds ratios, are frequently presented in research (n = 36) for 8 weeks. During the trial, new-onset sexual dysfunction articles. Not all readers know how these statistics was identified in 8 patients treated with venlafaxine and in 3 patients are derived and interpreted, nor are all readers treated with placebo. These results are presented in Table 1.
    [Show full text]
  • Ethics of Vaccine Research
    COMMENTARY Ethics of vaccine research Christine Grady Vaccination has attracted controversy at every stage of its development and use. Ethical debates should consider its basic goal, which is to benefit the community at large rather than the individual. accines truly represent one of the mira- include value, validity, fair subject selection, the context in which it will be used and Vcles of modern science. Responsible for favorable risk/benefit ratio, independent acceptable to those who will use it. This reducing morbidity and mortality from sev- review, informed consent and respect for assessment considers details about the pub- eral formidable diseases, vaccines have made enrolled participants. Applying these princi- lic health need (such as the prevalence, bur- substantial contributions to global public ples to vaccine research allows consideration den and natural history of the disease, as health. Generally very safe and effective, vac- of some of the particular challenges inherent well as existing strategies to prevent or con- cines are also an efficient and cost-effective in testing vaccines (Box 1). trol it), the scientific data and possibilities way of preventing disease. Yet, despite their Ethically salient features of clinical vac- (preclinical and clinical data, expected brilliant successes, vaccines have always been cine research include the fact that it involves mechanism of action and immune corre- controversial. Concerns about the safety and healthy subjects, often (or ultimately) chil- lates) and the likely use of the vaccine (who untoward effects of vaccines, about disturb- dren and usually (at least when testing effi- will use and benefit from it, safety, cost, dis- ing the natural order, about compelling indi- cacy) in very large numbers.
    [Show full text]
  • Disease Incidence, Prevalence and Disability
    Part 3 Disease incidence, prevalence and disability 9. How many people become sick each year? 28 10. Cancer incidence by site and region 29 11. How many people are sick at any given time? 31 12. Prevalence of moderate and severe disability 31 13. Leading causes of years lost due to disability in 2004 36 World Health Organization 9. How many people become sick each such as diarrhoeal disease or malaria, it is common year? for individuals to be infected repeatedly and have several episodes. For such conditions, the number The “incidence” of a condition is the number of new given in the table is the number of disease episodes, cases in a period of time – usually one year (Table 5). rather than the number of individuals affected. For most conditions in this table, the figure given is It is important to remember that the incidence of the number of individuals who developed the illness a disease or condition measures how many people or problem in 2004. However, for some conditions, are affected by it for the first time over a period of Table 5: Incidence (millions) of selected conditions by WHO region, 2004 Eastern The Mediter- South- Western World Africa Americas ranean Europe East Asia Pacific Tuberculosisa 7.8 1.4 0.4 0.6 0.6 2.8 2.1 HIV infectiona 2.8 1.9 0.2 0.1 0.2 0.2 0.1 Diarrhoeal diseaseb 4 620.4 912.9 543.1 424.9 207.1 1 276.5 1 255.9 Pertussisb 18.4 5.2 1.2 1.6 0.7 7.5 2.1 Measlesa 27.1 5.3 0.0e 1.0 0.2 17.4 3.3 Tetanusa 0.3 0.1 0.0 0.1 0.0 0.1 0.0 Meningitisb 0.7 0.3 0.1 0.1 0.0 0.2 0.1 Malariab 241.3 203.9 2.9 8.6 0.0 23.3 2.7
    [Show full text]
  • Metabolic Sensor Governing Bacterial Virulence in Staphylococcus Aureus
    Metabolic sensor governing bacterial virulence in PNAS PLUS Staphylococcus aureus Yue Dinga, Xing Liub, Feifei Chena, Hongxia Dia, Bin Xua, Lu Zhouc, Xin Dengd,e, Min Wuf, Cai-Guang Yangb,1, and Lefu Lana,1 aDepartment of Molecular Pharmacology and bChinese Academy of Sciences Key Laboratory of Receptor Research, Shanghai Institute of Materia Medica, Chinese Academy of Sciences, Shanghai 201203, China; cDepartment of Medicinal Chemistry, School of Pharmacy, Fudan University, Shanghai 201203, China; dDepartment of Chemistry and eInstitute for Biophysical Dynamics, The University of Chicago, Chicago, IL 60637; and fDepartment of Basic Sciences University of North Dakota School of Medicine and Health Sciences, Grand Forks, ND 58203 Edited by Richard P. Novick, New York University School of Medicine, New York, NY, and approved October 14, 2014 (received for review June 13, 2014) An effective metabolism is essential to all living organisms, in- To survive and replicate efficiently in the host, S. aureus has cluding the important human pathogen Staphylococcus aureus.To developed exquisite mechanisms for scavenging nutrients and establish successful infection, S. aureus must scavenge nutrients adjusting its metabolism to maintain growth while also coping with and coordinate its metabolism for proliferation. Meanwhile, it also stress (6, 11). On the other hand, S. aureus produces a wide array must produce an array of virulence factors to interfere with host of virulence factors to evade host immune defenses and to derive defenses. However, the ways in which S. aureus ties its metabolic nutrition either parasitically or destructively from the host during state to its virulence regulation remain largely unknown. Here we infections (6).
    [Show full text]
  • Prevalence and Determinants of Vaccine Hesitancy and Vaccines Recommendation Discrepancies Among General Practitioners in French-Speaking Parts of Belgium
    Article Prevalence and Determinants of Vaccine Hesitancy and Vaccines Recommendation Discrepancies among General Practitioners in French-Speaking Parts of Belgium Cathy Gobert 1, Pascal Semaille 2, Thierry Van der Schueren 3 , Pierre Verger 4 and Nicolas Dauby 1,5,6,* 1 Department of Infectious Diseases, CHU Saint-Pierre, Université Libre de Bruxelles (ULB), 1000 Bruxelles, Belgium; [email protected] 2 Department of General Medicine, Université Libre de Bruxelles (ULB), 1070 Bruxelles, Belgium; [email protected] 3 Scientific Society of General Practice, 1060 Bruxelles, Belgium; [email protected] 4 Southeastern Health Regional Observatory (ORS PACA), 13005 Marseille, France; [email protected] 5 School of Public Health, Université Libre de Bruxelles (ULB), 1070 Bruxelles, Belgium 6 Institute for Medical Immunology, Université Libre de Bruxelles (ULB), 1070 Bruxelles, Belgium * Correspondence: [email protected] Abstract: General practitioners (GPs) play a critical role in patient acceptance of vaccination. Vaccine hesitancy (VH) is a growing phenomenon in the general population but also affects GPs. Few data exist on VH among GPs. The objectives of this analysis of a population of GPs in the Belgian Wallonia-Brussels Federation (WBF) were to: (1) determine the prevalence and the features of VH, (2) identify the correlates, and (3) estimate the discrepancy in vaccination’s behaviors between the GPs’ children and the recommendations made to their patients. An online survey was carried out Citation: Gobert, C.; Semaille, P.; Van among the population of general practitioners practicing in the WBF between 7 January and 18 der Schueren, T.; Verger, P.; Dauby, N. March 2020. A hierarchical cluster analysis was carried out based on various dimensions of vaccine Prevalence and Determinants of hesitancy: perception of the risks and the usefulness of vaccines as well as vaccine recommendations Vaccine Hesitancy and Vaccines for their patients.
    [Show full text]
  • Principles of Validation of Diagnostic Assays for Infectious Diseases1
    PRINCIPLES OF VALIDATION OF DIAGNOSTIC ASSAYS FOR INFECTIOUS DISEASES1 R. H. JACOBSON, New York State College of Veterinary Medicine, Cornell University, Ithaca, New York, USA Abstract PRINCIPLES OF VALIDATION OF DIAGNOSTIC ASSAYS FOR INFECTIOUS DISEASES. Assay validation requires a series of inter-related processes. Assay validation is an experimental process: reagents and protocols are optimized by experimentation to detect the analyte with accuracy and precision. Assay validation is a relative process: its diagnostic sensitivity and diagnostic specificity are calculated relative to test results obtained from reference animal populations of known infection/exposure status. Assay validation is a conditional process: classification of animals in the target population as infected or uninfected is conditional upon how well the reference animal population used to validate the assay represents the target population; accurate predictions of the infection status of animals from test results (PV+ and PV−) are conditional upon the estimated prevalence of disease/infection in the target population. Assay validation is an incremental process: confidence in the validity of an assay increases over time when use confirms that it is robust as demonstrated by accurate and precise results; the assay may also achieve increasing levels of validity as it is upgraded and extended by adding reference populations of known infection status. Assay validation is a continuous process: the assay remains valid only insofar as it continues to provide accurate and precise results as proven through statistical verification. Therefore, the work required for validation of diagnostic assays for infectious diseases does not end with a time-limited series of experiments based on a few reference samples rather, to assure valid test results from an assay requires constant vigilance and maintenance of the assay, along with reassessment of its performance characteristics for each unique population of animals to which it is applied.
    [Show full text]
  • New NIH Guidelines to Improve Experimental Rigor and Reproducibility
    New NIH guidelines to improve Experimental Rigor and Reproducibility Oswald Steward Professor, Department of Anatomy & Neurobiology Senior Associate Dean for Research University of California School of Medicine What I’ll tell you 1) Background: Increasing reports of failures to replicate 2) Previous actions by NIH to enhance rigor 3) New requirements for grants submitted after January 2016 4) What NIH will be looking for in new required sections 5) Some suggested best practices to meet new standards for scientific rigor • The Problem: Numerous recent reports of findings that can’t be replicated. Failure to replicate key results from preclinical cancer studies: 1) Prinz et al. (2011) Believe it or not: How much can we rely on published data on potential drug targets? Nature Reviews Drug Discovery. Inconsistencies were found in 2/3 studies 2) Begley and Ellis (2012) Raise standards for preclinical cancer research. Nature. Only 6/53 “landmark” papers replicated 3) Begley (2013) Six red flags for suspect work. Nature. -Were experiments performed blinded? -Were basic experiments repeated? -Were all the results presented? -Were there positive and negative controls? -Were reagents validated? -Were statistical tests appropriate? • In ALS field, Scott et al., (2008) re-assessed 70 drugs that had been reported to increase lifespan in an animal model of ALS (SOD1-G93A knockout mice). • 221 separate studies were carried out over a five year period that involved over 18,000 mice. • Some drugs were already in clinical trials. • Outcome: There was no statistically significant positive effect for any of the 70 compounds tested. • Conclusion: Previously published effects were measurements of noise in the distribution of survival means as opposed to an actual drug effect.
    [Show full text]
  • Studies of Staphylococcal Infections. I. Virulence of Staphy- Lococci and Characteristics of Infections in Embryonated Eggs * WILLIAM R
    Journal of Clinical Investigation Vol. 43, No. 11, 1964 Studies of Staphylococcal Infections. I. Virulence of Staphy- lococci and Characteristics of Infections in Embryonated Eggs * WILLIAM R. MCCABE t (From the Research Laboratory, West Side Veterans Administration Hospital, and the Department of Medicine, Research and Educational Hospitals, University of Illinois College of Medicine, Chicago, Ill.) Many of the determinants of the pathogenesis niques still require relatively large numbers of and course of staphylococcal infections remain staphylococci to produce infection (19). Fatal imprecisely defined (1, 2) despite their increas- systemic infections have been equally difficult to ing importance (3-10). Experimental infections produce in animals and have necessitated the in- in suitable laboratory animals have been of con- jection of 107 to 109 bacteria (20-23). A few siderable assistance in clarifying the role of host strains of staphylococci have been found that are defense mechanisms and specific bacterial virulence capable of producing lethal systemic infections factors with a variety of other infectious agents. with inocula of from 102 to 103 bacteria (24) and A sensitive experimental model would be of value have excited considerable interest (25-27). The in defining the importance of these factors in virulence of these strains apparently results from staphylococcal infections, but both humans and an unusual antigenic variation (27, 28) which, the usual laboratory animals are relatively re- despite its interest, is of doubtful significance in sistant. Extremely large numbers of staphylo- human staphylococcal infection, since such strains cocci are required to produce either local or sys- have been isolated only rarely from clinical in- temic infections experimentally.
    [Show full text]