on

Anita Bandrowski, Sci Cr unch / NI F / RRI D [email protected] Is Reproducibility really a Problem? Growing Challenge: Ensuring the Rigor and Reproducibility of Science Growing Challenge: Ensuring the Rigor and Reproducibility of Science

4 Rigor and Transparency in Research *New Grant Review Criteria*

To support the highest quality science, public accountability, and social responsibility in the conduct of science, NIH’s Rigor and Transparency efforts are intended to clarify expectations and highlight attention to four areas that may need more explicit attention by applicants and reviewers:

• Scientific premise • Scientific rigor • Consideration of relevant biological variables, such as sex • Authentication of key biological and/or chemical resources Rigor and Transparency in Research *New Grant Review Criteria*

To support the highest quality science, public accountability, and social responsibility in the conduct of science, NIH’s Rigor and Transparency efforts are intended to clarify expectations and highlight attention to four areas that may need more explicit attention by applicants and reviewers:

• Scientific premise • Scientific rigor • Consideration of relevant biological variables, such as sex • Authentication of key biological and/or chemical resources What is a Key Biological Resource?

• Antibodies • Cell Lines • Organisms (transgenic)

Reagents that are the most common failure point in experiments What does it cost to have Key Biological Resource fail sometimes? *

Freeman et al, 2017. Reproducibility2020: Progress and priorities Do we know which key biological resources are being used?

Papers are currently poor at identifying the simplest part of the paper, the materials used

Vasilevsky 2013

Do we publish papers with antibodies that don’t work? How to authenticate antibodies; Validation pillars How to authenticate antibodies; RRID • The authority is antibodyregistry.org • Caveats for antibodies: antibodies may be the same between vendors; check to see if duplicates are reported (catalog number and clone ID may be different, if RRID is the same = same product) • Chose antibodies validated for your application One 3 • Chose antibodies with independent validation: RRID companies • ENCODE ISO 9001: 2008 “discontinued 2016 due to • RRID publications and ISO 13485: animal welfare concerns” • Original manufacturer tag 2003 “Originating Manufacturer of Test each lot for your application this product; Tested • no validation applications: WB/ ELISA” data or MDS “ ENCODE PROJECT External available from Validated by validation DATA SET is released” vendor S cienceExchange Do we publish papers with contaminated cell lines?

“Not one of my published papers has led to a retraction by a journal or scientist. Less than 10 corrections have been issued, when each false line I discovered affects the conclusions of hundreds or thousands of papers.” How to authenticate a cell line; What to do

ICLAC.org – the International Cell Line Authentication Committee gives some useful guidelines and also provides a list of questionable cell lines. ICLAC suggests: • Verification using sequencing of the cell line as it comes into the lab • Spot checking the cell line on some schedule • Verifying cell line at the end of the experiments • Check with known contaminated cell lines How to authenticate a cell line: RRID Is the cell line, already in my lab, a known problematic cell line? • Check database early and often • http://rrid.site search for your cell line, look for comment “Problematic cell line” • Data is derived from ICLAC list of contaminated cell lines, and it is curated by Cellosaurus • Contamination noted in Not all vendors report 730+ cell lines contamination on their website, note ATCC & ECACC do How can we get better data into papers?

• 2009: LAMHDI meeting – project hatched Journals key solution • 2011: Meeting with Society for Neuroscience, Journal of Neuroscience full editorial board presenting the problem and results of text mining study Journals will not change • 2012: Society for Neuroscience – defined the problem for Editors of top Neuroscience journals; sponsored by INCF Journals in aggregate • 2013: NIH Meeting - brought the editors back to define the solution; 2 day workshop sponsored by NIDA and INCF, several IC directors in attendance Funders role • 2013: Society for Neuroscience – mainly publishers, defined the timeline of starting the project • 2014: Neuroscience Information Framework – built scicrunch.org/resources based on NIF technologies and members of the OHSU team populated web pages / instructions etc. RRID based tracking • 2014: Project starts with Journal of Neuroscience, Neuroinformatics, F1000, Brain and Behavior and Journal of Comparative Neurology taking a strong lead Project Management Key • 2015: Paper describing how RRIDs are used by authors of the first 100 papers is co-published in 4 journals • 2016: integration with Hypothes.is tool gives curators an easy way to verify RRIDs, sci-score gives authors an easier way to detect what is a resource Technology innovation

RRID Author’s Workflow Journal directs author http://scicrunch.org/resources to RRID portal

Author searches for an antibody

Paper Author copies becomes ”Cite This” text data into manuscript

RRID portal includes: Antibodies 2.5M Paper is Organisms 500K (25 stock center/MODs) published Cell lines 80K Software projects 14K Resources are identifiable!

Bandrowski et al, 2015 SciScore, an RRID Shortcut Journal workflow directs methods section to SciScore

Report is generated for author Paper becomes Author follows data links adds RRIDs

Paper is published SciBot reads a paper and brings back information as an annotation, display is in Hypothes.is https://blog.box.com/b log/introducing- developer-tokens/

Annotated data is used in several applications

Data about RRID data fills in the this paper is in data in the Hypothes.is SciCrunch.org/resolver

Which other papers used this antibody? RRIDs have been used in ~3000 papers (data is from 1900)

15K RRIDs, 8K are unique so each RRID used avg. twice What are authors saying? July 2016 Who does this?

Aug 2016

Sept/Oct 2016

Jan 2016 *Reproducibility of science is critical* *Transparency is achievable*

Number of journals where RRIDs are found: 265

Number of journals asking effectively: 15

Number of journals enforcing RRIDs: 10

Next Steps ….we are at 1%, how to get to 100?

To ask for RRIDs To bring your journals

To help raise awareness of RRIDs (blogs, webinar, newsletters, twitter #RRID)

Comments: [email protected]