Springernature Experiments R

springernature.com

White Paper REPRODUCIBILITY IN THE LIFE SCIENCES: the role of protocols and methods 2 Reproducibility in the life sciences: the role of protocols and methods springernature.com

Reproducibilit y is one of the foundations on which scientific discover y is built. The idea that scientific research is reproducible is central to science in both its theor y and practice. However, concerns around reproducibilit y have been gaining at tention over recent years especially in pre-clinical research in the life sciences, where concerns have frequently centred on issues with protocol s, experimental design, reference material s and data collection and analysis. Protocols and methods are a major factor in the ability to reproduce experiments and can be defined as the procedural method used in the design and implementation of an experiment. (1) Failure in the recording or execution of a protocol can be the prime cause of an inabilit y to replicate the results of published biological and clinical research, with reproducibility issues frequently cited as major causal factors leading to high failure rates in clinical trial s. Such evidence has led to scientific methodology including protocol s rising in impor tance in both experimental practice, research curation and publication. (3).

Shining a spotlight on irreproducibility

A narrative around trends towards irreproducibility of research has unfolded in schol arly paper s during the l ast ten year s. Bet ween 2016 and 2017, there was a three-fold increase noted in papers listed by World of Science citing a ‘crisis’ in reproducibilit y science(4). In 2016, Nature published findings from the research communit y that asked researcher s if they saw a crisis in reproducibilit y develo- ping. 52% agreed that there was a significant crisis with over 70% reporting they had tried and f ailed to reproduce another scientist ’s experiment s.

On the BBC’s Today programme in 2017, Marcus Munafo, now professor of biological psychology at Bristol Universit y, admit ted he almost gave up on a career in science as a PhD student when he failed to reproduce a textbook study. It was an experience that “I had a crisis of confidence. I (6) gave him a career-long interest in making science more reproducible. thought maybe I didn’t run my Inter views conducted with librarians on behalf of the Springer Nature Experiments study well, maybe I’m not cut out team in 2019 show strong parallel s to the points made by researchers. Stacey Wahl, a to be a scientist.” Librarian at Virginia Commonwealth Universit y, shared her own experience as a Marcus Munafo Professor of biological psychology at researcher before she became a librarian: “I did experience this as a researcher. I think Bristol University it’s difficult to write down all the detail s in a method, so it’s not about saying you can’t reproduce something, it’s that something may be missing”. So where is the debate today ? How are scientists, librarians, institutes and publishers responding to irreproducibilit y and how are they tackling it ?

Reproducibility in the life sciences: the role of protocols and methods springernature.com 3

Tackling reproducibility in high-impact cancer biology research

Cancer studies are one of the highest profile and well-funded areas of the biological sciences. They too have been dogged by problems with reproducibilit y. With controver sial studies in the field estimating that as much as $26bn of research in basic cancer biology could be irreproducible, researcher s have set out to test this hypothesis for themselves (3).

Six years ago, a US-funded project devoted $1.6million to the objective of how reproducible high-impact research papers in basic biology of cancer were. Called the Reproducibility Project: Cancer Biology and led by Tim Errington at the Centre for Open Science, its remit was to at tempt to replicate the results from 50 high-impact cancer biology papers. The work proved complex and by 2018 this number of studies that the project was tackling had shrunk to 18. “I wish we could have done more. Tim Errington, who runs the project in Charlottesville, understands how the project has One reason for the reduction is that highlighted the real challenges of reproducibilit y. One of the findings of the project was it was taking too long to that increasing the level of detail in the published protocol and making a wider bank of troubleshoot or optimise results and material available through open science initiatives, should make it far easier experiments to get meaningful results. This work could have for scientists to reproduce and build on the work of others. proceeded faster if methodological The Reproducibilit y Project is still ongoing. Replication results have been published so details had been included in the far for ten of the 18 studies. Out of the 10 studies investigated, reviewed and published, original papers”. only five so far were mostly repeatable.(7) Tim Errington 4 Reproducibility in the life sciences: the role of protocols and methods springernature.com

Actions to drive better reproducible science

Researcher s see the difficulties in ensuring reproducibilit y as a potential challenge to the integrit y and reput ation of science, and are coming together to solve the problem from the ground up. Reproducibility conferences, workshops and formal coll abor ations are t aking pl ace to debate the best ways for ward and considering a broad scope of issues.

The initiatives are wide ranging, as can be seen in the SciDataCentre workshops run by a Swedish consortia of KTH Royal Institute of Technology, the Karolinska Institutet and the Universities of Stockholm and Uppsala (9) focusing on best practice in data collection and data methods, The Sackler Colloquium on Reproducibility of Research(2) a conference with a full set of proceedings and papers that covered a wide spectrum of topics across reproducibilit y, and NIST’s collaboration with the NPA (10) with its focus on instrumentation and data collection and their role in better reproducible outcomes. More training is also being put in place to support scientists in working practices designed to increase and support reproducibility. This can be transparency and best practice in experimental design, choice and recording of methods and protocols or data methods, statistical analysis and the selection of results. Many life sciences institutes are regularly training researchers in methods and protocol s and have launched reproducibilit y training courses. The European Molecular Biology Laborator y has courses across life sciences methods and protocols(11). The Universit y of Cambridge through the ELIXIR net work has an active programme in both training and training trainers in bioinformatics to drive best practices in the area (12). The role and integrity of data is as important as methods and protocols for reproducibility. Recent initiatives such as FAIR (findable, accessible, interoperable and reusable) data are a major step in making improved data reproducibility a reality in life sciences research. Librarians and publishers alongside researchers play a central role in underpinning and improving these practices to ensure bet ter reproducibilit y. This can be by providing the most comprehensive material to suppor t experimental work or supporting high standards in research practice and research, data curation or paper submission. Reproducibility in the life sciences: the role of protocols and methods springernature.com 5

Ensuring reproducibility through the publishing process Initiatives like editorial checklists - one was launched by Nature(13) in 2013 - have succeeded in raising the bar for rigour in submissions to journals(14). Editorial checklists have sought to establish guidance for researchers in terms of methods and data checks to help ensure published research meets standards for reproducibility and for peer-reviewed papers to be more easily reproduced by the communit y. The checklists are also living documents; the Nature checklist, for example, was updated in 2017 to include additional detail s on experimental design, reagents and analysis as well as more detailed summaries for specific areas such as flow cy tometr y and magnetic resonance imaging (MRI). The checklists have proved only a first step with cross-publisher collaboration to drive reproducibility standards upwards. Recent moves in 2018 have included publisher’s working with journal editors and exper ts in transparency and reproducibilit y to define a framework across publishing, looking at the core areas of Material s, Design, Analysis and Reporting (MDAR)(15). This next stage shows how publishers are commit ted to drive more robust capture of methods and results for published papers. Nature branded journal s are al so leading the way in carr ying out peer review of code, given the impor tance of computational work across the sciences, moves like this are essential to ensure reproducibility across the widest span of experimental methods and analysis. (16)

Changing roles for librarians

Libr arians too have engaged positively in the need to suppor t reproducible scientific outcomes. In 2018, t wo libr arian conferences, the ML A and the International Congress on Medical Librarianship had reproducibility conference tr ack s to help libr arians bet ter under st and these challenges. O ther organisations, such as EQUATOR, have pioneered reproducibilit y guidelines. EQUATOR has put in pl ace a series of initiatives and resources to suppor t medical libr arians including set s of best pr ac tice, guidance and tr aining to help clinical libr arians t ackle issues with reproducibility(17).

Other practical examples are emerging of Head Librarians and Librar y organisations taking a lead in improving reproducibilit y. Transparency and reproducibilit y of reseref arch findings were central to the Open Science Symposium hosted by Carnegie Mellon Universit y (CMU) Librar y. Aimed at librarians and researchers across CMU, the symposium launched the concept of liaison librarianship, which sees facult y working more closely with librarians to encourage researchers to collaborate, share, document and register their research through new library platforms and initiatives and improve the research record and archives. (18) Librarians, especially, not only understand the impor tance that reproducibilit y has for researchers in their institutions but are finding themselves right at the heart of the process. This may be assisting the capture of the primary research information or helping researchers understand new tools and products in research methods or dealing with information suppor t, especially in evidence-based disciplines, such as biomedical or clinical subjects. The librarian’s role is becoming essential in supporting study design through sourcing and investing in the right methods content. 6 Reproducibility in the life sciences: the role of protocols and methods springernature.com

The reproducibility librarian Reproducibility was headline news in the librarian community when NYU appointed the world’s first Research Data Management and Reproducibilit y Librarian, Vick y Steeves. Since her appointment in 2015, Vick y has spent time informing fellow librarians and scientists about why this role is impor tant. She’s passionate about her ground-breaking role as a Reproducibilit y Librarian and why the creation of this new role is key for the scientific community(19). Vick y defines reproducibilit y as “when someone independently confirms results with the same data and computer code or tool s.” Working par ticularly on computational reproducibilit y, Vick y ensures that tool s or coding work across computational environments, which is critical in both biological and information science fields. Reproducibilit y practice is now par t of the new skill set needed in librarianship and one of the growing new directions in the field of information science. Reproducibilit y takes its place with open science, open data, new data repositories and greater curation of the earliest stages of the research process among new responsibilities for librarians. Since Vick y’s appointment as an NYU first, Universities across the US including the Universities of Utah, Oregon and Florida have followed NYU’s lead and created Reproducibilit y Librarianship posts, with Utah hosting a reproducibilit y symposium in 2016 helping position its library to lead change at the institution. (20)

How librarians see protocols in supporting reproducibility

What is cer t ain is that researcher s are at different st ages in this process. While some are at the cutting edge with the use of collaboration tools and computational notebooks, many researchers are still using protocols in print. Former researcher and libr arian at Virginia Commonwealth Univer sit y, St acey Wahl , noticed that during her research career.

“The labs have stacks of print At Virginia Tech, librarian Inga Haugen sees a more mixed pat tern in her institute with protocols which have been its active research programme in agriculture and plant biology: “Protocols are optimised and further developed by requested across formats; some in published books on standards, and some in the researchers. Protocols tend not ar ticle-associated protocol s. I need to have protocol s available to all my researchers, to be used online but are printed and the labs use the print standard. many who are out-stationed or field researchers, and the online access needs to be This can present a problem, if this robust.” is lost or changed, and there is only Practice may differ substantially in the corporate lab environment where standard use a print copy available”. of electronic lab notebooks and workflow management tool s sees protocol s uploaded Stacey Wahl and shared across research teams. The demands on commercial researchers to meet Researcher and librarian at Virginia industry and government regulatory requirements such as in the pharma industry Commonwealth University means that systematic storage and curation of methods and data are an essential part of the approval process, much more so than in the academic research environment. Practices such as these will become more common in the academic environment. Reproducibility in the life sciences: the role of protocols and methods springernature.com 7

Where are we now?

The past eight to ten years have seen a growing number of responses from researcher s, univer sities, libr arians and publisher s to the challenges of reproducibilit y. One thing is cer t ain: all sec tor s of the research communit y t ake the challenge seriously and are working to maintain the reputation and integrity of peer-reviewed scientific literature.

Reproducibilit y practices are increasing as the researchers, publishers and librarians tackle the issue head on. Researchers are focusing on improved methods, data curation and better lab practices and recording of experiments. There is more work being done to test how reproducible research is by specialist projects. Institutes, societies and professional bodies are creating suppor t net works and channel s for best practice for researchers through symposiums, conferences, new and enhanced professional training and additional resources and guidelines. Librarians are reskilling and new roles such as reproducibilit y librarians have been created to help institutes capture research output through all the stages of the research process and provide more suppor t to researchers with curation. Publishers too, from their journals submissions’ process to their published output of specialist journal s, databases and protocol s have both the processes and information products to suppor t the drive for improvements to help researchers and librarians alike.

The role of role of Springer Nature to support reproducibility across the life sciences

Dat abases such as Springer Nature E xperiment s can pl ay a centr al role in these initiatives to suppor t reproducibilit y. Providing researcher s with a source of both cut ting-edge and cl assical protocol content , Springer Nature E xperiment s provides a gateway for researchers to access content across methods and protocols. From the latest novel methods in Nature Methods to tried-and-tested protocols in Nature Protocols and Springer Protocols, users can search the entire Springer Nature por t folio of over 60,000 editorially assessed or peer-reviewed protocols and methods papers.

Springer was one of the fir st publisher s to recognise the impor t ance of protocol s. The publication of Springer Protocols, a pioneering series, set out best pr ac tice and standards in the life sciences long before the current debate. Seeing a need for protocol s in what was then a ver y new and cut ting-edge par t of the life sciences, Springer published "Methods in Molecular Biology" series in 1980, as the first dedicated modern lab protocol publication. It is still included as core content in the current Springer Protocols with subsequent additions that have built on the strength of it s original classics. Today, Springer Protocols deliver s complete online access to over 58,000 protocols across the life sciences. Nature Methods and Nature Protocols were launched later (in 2004 and 2006) and have smaller archives of around 5,000 methods paper s and 3,000 protocol s, respec tively. These journal s are highly selec tive, publishing only the most exciting new methods and gold-standard protocols, with a rigorous peer review process and in-house editing to the highest standards.

At both Nature Protocols and Springer Protocols, protocol s are kept up to date with either versioning or Protocol Updates. Protocols are constantly being developed, so it’s impor t ant that researcher s can always find the l atest ver sion, but the retention of older ver sions is an impor t ant consider ation for l abs that might not have the l atest equipment . All of the protocol s at these journal s cont ain a comprehensive troubleshooting guide, health and safety considerations, and a complete catalogue of conditions and reagents. 8 Reproducibility in the life sciences: the role of protocols and methods springernature.com

Protocols published by Nature Protocols al so cont ain det ailed experiment al design and anticipated results sections. With their comprehensive and in-depth coverage, all of the protocol s published by Springer Nature provide researcher s with solutions to many of the f ac tor s that have been cited as major issues in irreproducibilit y. In Summary With more guidelines, understanding and new professional practice, the curation and handling of the whole research process is moving to be more transparent and rigorous. With increasing publication and sharing through open science, more transparency should lead to a greater use of scientific data and results. Recording and publishing protocols play a vital part in this progress with more detail and earlier records in the research process essential in producing more meticulous scientific methods. The good news is that bet ter experimental design, improved data collection and handling, and the capture of detailed and robust experimental protocols will drive improvements in reproducibilit y. This will benefit researchers by leading to more solid scientific results, increased numbers of breakthroughs and ultimately bet ter outcomes for us all. Reproducibility in the life sciences: the role of protocols and methods springernature.com 9

References 1 https://www.cancer.gov/publications/dictionaries/cancer-terms/def/protocol 2 All is on, D, Shiff rin, R & Stodden, V, PNA S, 115, 11, (2018), Reproducibil it y of research: issues and proposed remedies. 3 Freedman, L & Inglese, J, Cancer Res, 74, 15 (2014) The Increasing Urgency for Standards in Basic Biological Research 4 Fanielli, Daniele, PNAS, 115, 11, (2018), Is science really facing a reproducibility crisis and do we need it to? 5 Baker, M, Nature, 533, 452-454, (2016), Is There a Reproducibilit y Crisis? 6 BBC News, https://www.bbc.co.uk/news/science-environment-39054778 (2017) Most scientists can’t replicate studies by their peers. 7 Kaiser, J, https://www.sciencemag.org/news/2018/07/plan-replicate-50-high-impact- cancer-papers-shrinks-just-18 (2018) Plan to replicate 50 high-impact cancer papers shrinks to just 18. 8 Stark, P, Nature, 557, 613 (2018), Before reproducibilit y must come preproducibilit y 9 https://www.scilifelab.se/events/reproducibility 10 Hanisch, R, Gilmore, I and Plant, A, Journal of Research of the National Institute of Standards and Technology, 124, 124024 (2019), Improving reproducibility in research: The role of measurement science 11 https://www.embl.de/training/index.php 12 https://bioinfotraining.bio.cam.ac.uk 13 Editorial, Nature, 556, 273-274 (2018), Checklist checked 14 https://openscience.bmj.com/content/3/1/e000035 15 K o u s t a , S , S c i e n c e E d i t o r , 4 2 , 7 7 - 8 2 ( 2 0 1 9 ) , T h r e e A p p r o a c h e s t o S u p p o r t R e p r o d u c ibility, https://www.csescienceeditor.org/article/three-approaches-to-support-repr ducible-research/ 16 https://bioinfotraining.bio.cam.ac.uk 17 https://www.equator-network.org/librarians/ 1 8 https://kilthub.cmu.edu/articles/Building_Community_and_Support_for_Open_ Science_at_Carnegie_Mellon_University_A_Conference_Report/7455197 19 Steeves, V, Collaborative Librarianship, Reproducibility Librarianship https://osf.io/ preprints/lissa/vzuj3/. 20 Sayre, F, College and Research Libraries, 79, 1 (2018), The Reproducibilit y Crisis and Academic Librarie springernature.com

Springer Nature advances discover y by publishing robust and insightful research, suppor ting the development of new areas of knowledge and making ideas and information accessible around the world.

Want to know more about how Springer Nature E xperiment s can help you? Get in touch by emailing us at [email protected]