sign in sign up
<<
Home , Clinical research, In vivo, Medical writing, Reproducibility

The crisis in preclinical – lessons to learn from

Laia Pedro-Roig 1 and crisis. 1 The published literature is a common promotion, or tenure). Journal editors, referees, Christoph H. Emmerich 2 source for potential new drug targets used by the and grant reviewers look for the perfect story: 1 Trilogy Writing & Consulting GmbH, , and publication results simple, clear, and complete. 3 These demands Frankfurt am Main, Germany are routinely validated in-house to ensure tempt investigators to cherry pick 2 PAASP GmbH, Heidelberg, Germany reproducibility. According to Prinz et al., almost for publishing, develop hypotheses to fit the data, two-thirds of the validation projects conducted or keep collecting data until the desired at Bayer from 2007 to 2010 showed significance level is reached (p-hacking). Correspondence to: inconsistencies in results (including some from Competition among laboratories and pressure to Laia Pedro-Roig prestigious journals). 2 Begley and Ellis reported publish among scientists may result in negligent Trilogy Writing & Consulting GmbH that researchers at Amgen could only confirm the controlling or reporting of experimental Falkensteiner Strasse 77 scientific findings of six out of 53 (11%) conditions. 2 Another issue is the bias towards 60322 Frankfurt am Main, Germany landmark studies. 3 In addition, more than 70% of publishing positive results and the difficulties in [email protected] the researchers who participated in the Nature publishing results that contradict data in high- +49 69 138 2528-44 survey have failed to reproduce another scientist’s impact journals or currently established opinion results, and more than 50% admitted to having (publication bias). 2 This leads to strengthening Christoph Emmerich failed to reproduce their own. 1 certain hypotheses, even if there is a body of PAASP GmbH The reproducibility crisis is a quantifiable unpublished evidence against them. Am Aukopf 14-1 economic problem. Venture capital firms Published preclinical research often lacks 69118 Heidelberg, Germany consider that, when repeated by an independent proper quality standards in study [email protected] laboratory, the experiments in at least 50% of design (e.g., blinding and Low +49 696 777 8485 published studies do not provide the same randomisation) and valida - quality results. 4 In the US alone, US $28 billion per year tion of research tools that standards are spent on preclinical research that is not ensure the data obtained can make drug Abstract reproducible. 5 More importantly, the lack of is meaningful and un - candidates In recent years, the robustness and reproducibility has a negative impact on the biased. Begley and Ellis look more reproducibility of preclinical data have been bench-to-bedside time for new , observed that authors of promising a topic for discussion. Quality standards and and increases drug development costs, as each reproducible preclinical than they good practices are often not well defined for study replication conducted by the pharma - cancer studies had paid actually different methods and models, ceutical industry to validate academic research close attention to controls, are. and not harmonised amongst preclinical findings requires 3 to 24 months of work and reagents, and description of the research laboratories. This results in poorly US $500,000 to $2 million. 5 complete dataset, while in studies that could not reliable literature, has a negative impact on the Data robustness becomes even more be reproduced, data were not routinely analysed bench-to-bedside time for new drugs, and important at later stages of preclinical research, by blinded investigators and often results from increases the resources needed for clinical when results determine “go/no-go” decisions for only one were pre sent ed. 3 Accord - development. Clinical research, on the other drug candidates to enter clinical testing. A meta- ing to Freedman et al., errors leading to irrepro - hand, is tightly regulated and has high quality analysis identified higher effect sizes in animal ducibility of preclinical data can be due to study standards in place. Although improvements models of in studies with low quality design, biological reagents and reference are slowly introduced, preclinical develop - standards. 6 This implies how low quality research materials, laboratory protocols, and data analysis ment (especially in its confirmatory phases) standards can make drug candidates look more and reporting. 5 would benefit from taking a closer look and promising than they actually are. An enduring challenge in drug development adapting more of the internationally accepted is the erroneous use and misinterpretation of principles used in clinical research. What is behind the preclinical data from cell lines and animal reproducibility crisis? models. In vitro cell culture systems are crucial Participants in the Nature survey consider that research tools for analysing complex mechanisms Reproducibility issues are gaining awareness the main reasons are pressure to publish and regulating cell . However, over 480 amongst preclinical scientists: in a Nature survey, selective reporting. 1 For academics, publishing is misidentified cell lines (as of November 2017) 52% of researchers state that there is a significant a career essential (e.g., for research funding, job routinely used in published studies are

28 | December 2017 | Volume 26 Number 4 Pedro-Roig and Emmerich – The reproducibility crisis in preclinical research

the existing GxP standards (Good Laboratory Practice [GLP], [GCP], Good Manufacturing Practice [GMP], etc.) can be used to ensure high quality preclinical research outcomes. Whether conducted in an academic or industrial laboratory, this non-regulated research is, however, essential to identify and validate novel drug targets and to build the basis for successful translation of preclinical data into clinically meaningful efficacy. Thus, there is a need for new specialised Good Research Practice (GRP) guidelines that focus on study design, unbiased conduct, statistical analysis, and transparent reporting. Clinical research, on the other hand, is highly regulated and adherence to quality standards is routinely monitored. Human experimentation has strong ethical restrictions that require researchers to comply with higher research standards to avoid submitting study participants to unnecessary risks. There are several lessons that preclinical research could learn from clinical research regarding quality standards.

Lessons to learn from clinical research Clinical research is not perfect: A recent analysis of more than 5,000 papers in eight leading medical journals showed that roughly 2% of randomised controlled clinical trials may include fabricated data or lack adequate ethical app - roval. 13 However, clinical research is supported by strong standards and well-established proce- dures, as the following examples demonstrate, which could be used in preclinical research. contaminated, very frequently with HeLa cells identified as candidates for in an ALS (list available from the International Cell Line mouse model. 8 Most of these compounds failed Authentication Committee [ICLAC]). 7 to slow the in animals (including eight As the cornerstone document of clinical research The causes behind the reproducibility crisis drugs that had previously looked promising, ethics , the Declaration of Helsinki helps ensure are not limited to a specific field (in vitro or in proceeded to clinical trials, and ultimately failed). that the risk to trial subjects is proportionate to vivo) of preclinical research or therapeutic area. These discrepancies are likely due to the low the benefit expected to society. Similar codes of The limitations of preclinical cancer models quality standards of the original publications, as practice would help preclinical researchers to include (i) the use of a small number of poorly most did not include statistical models to realise that there is an implicit responsibility in characterised tumour cell lines that inadequately minimise experimental noise or implement all their activities. Currently, a similar concept recapitulate human disease, (ii) the inability to blinding and randomisation procedures. exists only for animal research: the 3Rs capture the human tumour environment, (iii) the (Replacement, Reduction, and Refinement) are lack of consideration for pharmacodynamics and What is the current situation considered in the US Guide for the Care and Use pharmacokinetics, (iv) the use of problematic and what could be done? of Laboratory Animals and the European endpoints and testing strategies, and (v) the There are no commonly accepted and followed Directive 2010/63/EU. 14,15 These guidelines regular exclusion of predictive biomarkers guidelines and quality standards for encourage finding alternatives to the use of for efficacy. 3 In the amyotrophic lateral The preclinical research outside those animals, using the right number of animals, sclerosis (ALS) field, Steve Perrin reproducibility intended for studies that directly refining breeding, accommodation and care, and and his team re-examined 100 crisis is a quantifiable support drug marketing autho - minimising distress. Of note, “reduction” means compounds that had been economic problem. risations. 9-12 Indeed, none of using the minimum number of animals required

www.emwa.org Volume 26 Number 4 | Medical Writing December 2017 | 29 The reproducibility crisis in preclinical research – Pedro-Roig and Emmerich

to obtain statistically significant There are several biostatisticians to have a more translational predictability. They also address the results based on power lessons that relevant position in the reaction norm issue (whether response of an calculations (and not less than preclinical research experimental planning and organism to an experimental treatment can be those); the same principle is could learn from analysis of preclinical data. In affected by environmental factors such as food applied for sample size addition, medical writers as and housing conditions). 19 Trials combining calculations in clinical trials. clinical research specialists in guidelines and good data from different centres with slightly different regarding quality reporting practices have become environmental conditions are well suited to ICH E6 (GCP) standards. essen tial in clinical research, and analyse the robustness of effects and the The International Conference for could equally contribute to reproducibility of in vivo experiments. Harmonisation (ICH) guideline E6 enhance quality standards in covers ethical and scientific quality preclinical research reporting. Transparency standards for designing, conducting, recording, G Audits serve to evaluate trial conduct and The EMA Policy 70 is an attempt to enhance and reporting clinical trials, and enhances data compliance with the , standard transparency by publishing clinical data for credibility. Amongst other, ICH E6 includes the operating procedures (SOPs), GCP, and medicines once the decision making process on following concepts: regulatory require ments. In non-regulated an application for an EU-wide marketing G The Independent Ethics Com mittee (IEC) or pre clinical research, monitoring of compli - authorisation is complete. 20 This implies having Institutional Review Board (IRB) are ance with quality requirements that ensure open access to full datasets from those trials. In independent bodies consti tuted of medical, un biased conduct of research is increasingly similar ways, some journals publishing scientific, and non-scientific members who becoming the focus of discussion, and has biomedical preclinical research have now ensure protection of the rights, safety, and already been performed at contract research implemented “open data” policies: publications well-being of the participants of a organisations (CROs) offering preclinical need to include full datasets, biological properties by reviewing and approving essential trial services. Furthermore, these routine audits of all samples, and complete methodology. The aspects such as the protocol and its amend- could be interesting for agencies funding Transparency and Openness Promotion (TOP) ments, or the suitability of investigators and preclinical research. guidelines advise journals and funding agencies facilities. The Association for Assessment and As mentioned, preclinical research studies on how to incentivise transparency in planning Accreditation of Laboratory Animal Care intended to support drug marketing applications and reporting preclinical research. 21 (AAALAC) 16 and the International Council are governed by strict regulations set by GxP. 9-12 for Laboratory Animal (ICLAS) 17 However, these standards are not suitable for Registration of clinical trials carry out a similar function in animal non-regulated, preclinical biomedical research Clinical trials need to be registered, as this avoids experimentation: promot ing proper treat - and there is a need for the specialised set of GRP reporting bias, a common problem in preclinical ment and ethical use of animals in science. guidelines already discussed. Regarding in vitro research. 22 Notably, an increasing number of Other areas of preclinical research, such as cell cell culture, Good Cell Culture Practice (GCCP) journals in preclinical research now offer the line or in vitro work, still lack a mechanism to (principles for standardisation, rationalisation, “Registered Reports” publishing format, in which obtain feedback on quality and relevance. and international harmonisation of cell and tissue peer review is conducted prior to data collection, G Adequate and accurate source documents and culture laboratory practices) has already been based on the importance of the research question trial records must be maintained. Source data defined. 18 Nevertheless, consensus procedures and the quality of the methodology. Article should be Attributable, Legible, Contempo- for unambiguous authentication and identifi - acceptance for publication is ensured unless raneous, Original, and Accurate (ALCOA cation of cell lines are still missing, and cell line quality assurance or unresolvable reporting principles to ensure data integrity), as well as misidentification, contamination, and genotypic problems arise. complete. Furthermore, any changes to and phenotypic instability remain issues. source data should be traceable and not Reporting guidelines obscure the original entry (i.e., audit trail ICH E8 and ICH E9 Many journals require that authors follow the should be maintained). In preclinical The ICH E8 guideline (“General Considerations Consolidated Standards of Reporting Trials research, there is still no clear consensus on for Clinical Trials”) provides recommendations (CONSORT) statement, an evidence-based, which is the full set of essential parameters to for the design, methodology, and analysis of mini mum set of recommendations for complete be recorded in a specific experiment. clinical trials, and ICH E9 (“Statistical Principles and transparent reporting of randomised Furthermore, the use of lab notebooks as for Clinical Trials”) attempts to harmonise the controlled trials. 23 Several reporting guidelines tools to record research results is not principles of statistical methodology applied to have been developed for preclinical research, standardised, and practices such as data them. Recently, international research consortia including National Institutes of Health’s (NIH) witnessing are often not implemented. started to conduct so-called preclinical Phase III Principles and Guidelines for Reporting G GCP states that qualified individuals should trials (i.e., multicentre, randomised, blinded Preclinical Research, Nature’s checklist, the be involved in the conduct of a trial. animal studies) to test drug efficacy. T hese Animal Research: Reporting of In Vivo Experi- Frequently, errors in preclinical research result preclinical trials allow larger sample sizes and ments (ARRIVE) guidelines, 24 and the Cell from the incorrect analysis of data, suggesting reduce bias, thus improving robustness and Press’s Structured Transparent Accessible

30 | December 2017 Medical Writing | Volume 26 Number 4 Pedro-Roig and Emmerich – The reproducibility crisis in preclinical research

Reporting (STAR) Methods. 25 They are intended to prompt authors to disclose technical and statistical information and reviewers to consider relevant aspects for research reproducibi lity. However, the huge variability of experimental designs and analytical techniques needs to be accounted for. The community- driven approach to this situation was the definition of “minimum information” checklists. The Minimum Information About a Microarray Experiment (MIAME), developed in 2001, was the first of such guidelines, and details which information needs to be provided to ensure reproducibility and unambiguous interpretation of microarray-based data. 26 Similar guidelines for other preclinical research techniques are described at the Minimum Information about Biological and Biomedical Investigations (MIBBI) portal, although only a few methods are covered so far. 27

A word of caution Clinical and preclinical research are not directly comparable. In basic and preclinical research, scientists require enough freedom to use their creativity, which is key to the advancement of science and thus the development of novel drug candidates and innovative medicines. However, science progresses by building on existing knowledge, making rigorous, reproducible, high quality studies crucial. The importance of finding a compromise between the need to trust conclusions of published research findings and the freedom for scientists to explore and innovate, has led to the concept of exploratory and confirmatory preclinical studies: at the exploratory stage, statistical testing and low quality standards should be acceptable as long as the experimental setting, and various approaches to enhance the Disclaimers procedure is transparently described. However, robustness of preclinical data are being The opinions expressed in this article are the for confirmatory studies (aimed to demonstrate considered (strict adherence to quality standards, authors’ own and not necessarily shared by their robust and reproducible treatment effects), multicentre collaborations, , etc.). It employers or EMWA. proper study design and implementation of the seems worth noting that clinical research has highest quality standards are essential, even if gone a long way to improve its quality standards. Conflicts of interest time- and resource-consuming. 28 Preclinical These developments may also illuminate the path The authors are employed by Trilogy Writing & studies supporting decision making processes for preclinical research. Consulting (LPR) and PAASP GmbH (CE). (e.g., whether to advance to animal studies or to first-in-human trials) should, therefore, be Acknowledgements References designed and treated as carefully as any clinical The authors would like to thank James Visanji, 1. Baker M. Is there a reproducibility crisis? trial. manager at Trilogy Writing & Consulting Nature. 2016;533(7604):452–4. GmbH, and Anton Bespalov, managing partner 2. Prinz F, Schlange T, Asadullah K. Believe it Conclusion at PAASP GmbH, for useful advice and or not: How much can we rely on Some of the concepts from clinical research are discussion. published data on potential drug targets? already starting to be applied in the preclinical Nat Rev Drug Discov. 2011;10(9):712–3.

www.emwa.org Volume 26 Number 4 | Medical Writing December 2017 | 31 The reproducibility crisis in preclinical research – Pedro-Roig and Emmerich

3. Begley CG, Ellis LM. Drug development: http://eur-lex.europa.eu/legal-content/ preclinical studies. Med Writ. Raise standards for preclinical cancer EN/TXT/?uri=OJ:L:1989:315:TOC . 2013;22(2):131 –3. research. Nature. 2012;483(7391):531–3. 13. Carlisle JB. Data fabrication and other 23. Consolidated Standards of Reporting Trials 4. Osherovich L. Hedging against academic reasons for non-random sampling in 5087 (CONSORT) statement. [cited 2017 risk. SciBX 2011;4. randomised, controlled trials in anaesthetic November 13]. Available from: 5. Freedman LP, Cockburn IM, Simcoe TS. and general medical journals. Anaesthesia. http://www.consort-statement.org/ . The economics of reproducibility in 2017;72(8):944–52. 24. Animal Research: Reporting of In Vivo preclinical research. PLoS Biol. 14. National Research Council (US) Experiments (ARRIVE) guidelines. [cited 2015;13(6):e1002165. Committee for the Update of the Guide for 2017 November 13]. Available from: 6. Macleod MR, van der Worp HB, Sena ES, the Care and Use of Laboratory Animals. https://www.nc3rs.org.uk/arrive- Howells DW, Dirnagl U, Donnan GA. Guide for the care and use of laboratory guidelines . Evidence for the efficacy of NXY-059 in animals. 8th edition. Washington (DC): 25. Structured Transparent Accessible experimental focal cerebral ischaemia is National Academies Press (US); 2011. Reporting (STAR) Methods. [cited 2017 confounded by study quality. Stroke. 15. Directive 2010/63/EU of the European November 13]. Available from: 2008;39(10):2824–9. Parliament and of the Council of 22 http://www.cell.com/star-methods . 7. International Cell Line Authentication September 2010 on the protection of 26. Brazma A, Hingamp P, Quackenbush J, Committee (ICLAC). [cited 2017 animals used for scientific purposes. Sherlock G, Spellman P, Stoeckert C, et al. November 13]. Available from: Official J Eur Comm. 2010;L276/33. [cited Minimum information about a microarray http://iclac.org/. 2017 November 13]. Available from: experiment (MIAME) toward standards 8. Perrin S. Preclinical research: Make mouse http://eur-lex.europa.eu/legal-content/ for microarray data. Nat Genet. studies work. Nature 2014;507(7493): EN/TXT/?uri=celex%3A32010L0063 . 2001;29(4):365 –71. 423 –5. 16. Association for Assessment and 27. Fairsharing.org collections. Minimum 9. Good Laboratory Practices (GLP) for Accreditation of Laboratory Animal Care Information for Biological and Biomedical Non-Clinical Laboratory Studies, 21 C.F.R. (AAALAC). [cited 2017 November 13]. Investigations (MIBBI). [cited 2017 Part 58. [cited 2017 November 13]. Available from: https://www.aaalac.org/ . November 13]. Available from: Available from: https://www.fda.gov/ 17. International Council for Laboratory https://fairsharing.org/collection/MIBBI . 1ohrms/ dockets/98fr/980335s1.pdf . Animal Science (ICLAS). [cited 2017 28. Kimmelman J, Mogil JS, Dirnagl U. 10. Directive 2004/10/EC of the European November 13]. Available from: Distinguishing between exploratory and Parliament and of the Council of 11 http://iclas.org/ . confirmatory preclinical research will February 2004 on the harmonisation of 18. Coecke S, Balls M, Bowe G, Davis J, improve translation. PLOS Biol. laws, regulations and administrative Gstraunthaler G, Hartung T, et al. 2014;12(5):e1001863. provisions relating to the application of the Guidance on good cell culture practice. principles of good laboratory practice and A report of the second ECVAM task force the verification of their applications for on good cell culture practice. Altern Lab tests on chemical substances. Official J Eur Anim. 2005;33(3):261–87. Comm. 2004;L50/44. [cited 2017 19. Voelkl B, Würbel H. Reproducibility crisis: November 13]. Available from: Are we ignoring reaction norms? Trends http://www.bfr.bund.de/cm/349/ Pharmacol Sci. 2016;37(7):509–10. Author information directive _2004 _10 _ec.pdf . 20. EMA/240810/2013, POLICY/0070 of Laia Pedro-Roig, PhD, has 10 years of 11. Directive 2004/9/EC of the European 2 October 2014. European Medicines research experience in cell signalling, Parliament and of the Council of 11 Agency policy on publication of clinical including in February 2004 on the inspection and data for medicinal products for human use. Parkinson’s disease. S he is currently a medical verification of good laboratory practice [cited 2017 November 13]. Available from: writer at Trilogy Writing & Consulting. (GLP). Official J Eur Comm. http://www.ema.europa.eu/docs/en _GB/ Christoph Emmerich, PhD, is a trained 2004;L50/28. [cited 2017 November 13]. document_library/Other/2014/10/ biochemist with over 12 years of experience Available from: http://www.bfr.bund.de/ WC500174796.pdf . in preclinical drug development. He is co- cm/349/directive _2004 _9_ec.pdf . 21. Nosek BA, Alter G, Banks GC, Borsboom founder of PAASP GmbH, a Heidelberg- 12. 89/569/EEC Council Decision of 28 July D, Bowman SD, Breckler SJ, et al. based company that focuses on establishing 1989 on the acceptance by the European Promoting an open research culture: quality standards in preclinical research and Economic Community of an OECD Author guidelines for journals could help provides assessment of operational risks to decision/recommendation on compliance to promote transparency, openness, and facilitate decision making in early-phase drug with principles of good laboratory practice. reproducibility. Science. 2015;348(6242): discovery projects. Official J Eur Comm. 1989;L315/32. [cited 1422 –5. 2017 November 13]. Available from: 22. Pavlica S. The need for registration of

32 | December 2017 Medical Writing | Volume 26 Number 4