Practical Resources for Enhancing the Reproducibility of Mechanistic Modeling in

Michael L. Blinova,g, John H. Gennarib,g, Jonathan R. Karrc,d,g, Ion I. Morarua,g, David P. Nickersone,g, Herbert M. Saurof,g,h

aCenter for Cell Analysis and Modeling, UConn Health, Farmington, CT, US bDepartment of Biomedical Informatics and Medical Education, University of Washington, Seattle, WA, US cIcahn Institute for Data Science and Genomic Technology, Icahn School of Medicine at Mount Sinai, New York, NY, US dDepartment of and Genomic Sciences, Icahn School of Medicine at Mount Sinai, New York, NY, US eAuckland Bioengineering Institute, University of Auckland, Auckland, NZ fDepartment of Bioengineering, University of Washington, Seattle, WA, US gThese authors contributed equally to this work hCorrespondence: [email protected]

Abstract

Although reproducibility is a core tenet of the scientific method, it remains challenging to re- produce many results. Surprisingly, this also holds true for computational results in domains such as systems biology where there have been extensive standardization efforts. For exam- ple, Tiwari et al. recently found that they could only repeat 50% of published simulation results in systems biology. Toward improving the reproducibility of computational systems research, we identified several resources that investigators can leverage to make their research more accessible, executable, and comprehensible by others. In particular, we identified sev- eral domain standards and curation services, as well as powerful approaches pioneered by the software engineering industry that we believe many investigators could adopt. Together, we believe these approaches could substantially enhance the reproducibility of systems biology research. In turn, we believe enhanced reproducibility would accelerate the development of more sophisticated models that could inform precision medicine and .

arXiv:2104.04604v1 [q-bio.QM] 9 Apr 2021 Keywords: systems biology, mechanistic modeling, repeatability, reproducibility, reusability

1. Introduction

One of the central pillars of the scientific method is reproducibility [1], the ability of indepen- dent researchers to reproduce results de novo without the aid of their original investigators New Assumptions Biological Knowledge

Biological Models Simulation Validation Publication Knowledge Results

Figure 1: Typical steps for generating simulation results in systems biology. For results to be repeatable, each of these steps must be repeatable. or their hardware and software. We believe such reproducibility would accelerate computa- tional science by making it easier for peer reviewers to quality control reported results and by making it easier for investigators build upon reported results. In particular, integrating models of multiple biological subsystems into more comprehensive models of entire cells and organisms will only be feasible if there are high-quality components that are accessible and reusable. While attaining this degree of reproducibility is challenging, we believe that computational scientists have access to all of the raw ingredients needed to report repeatable results that other researchers can recreate using the same data files, software tools, and a similar com- putational environment. For example, tools such as Java can execute code across multiple platforms, and repositories such as Docker Hub, GitHub, and Zenodo [2] make it easy to share data, code, and computational environments. Despite the importance of repeatability, many computational results are not repeatable. For example, Tiwari et al. recently found that they could only repeat 50% of simulation results reported in systems biology Tiwari et al. [3]. Similar concerns have been reported in [4] and other computational domains [5–7]. One artificial intelligence researcher was so concerned that they created a web site called ‘Papers Without Code’ (https://www.paperswithoutcode.com/) to enable investigators to report irrepeatable pa- pers. (The same researcher also created a complimentary site called ‘Papers With Code’ (https://paperswithcode.com/), to highlight repeatable research.) Taken together, poor repeatability is currently an endemic problem to computational research. In order for a computational experiment to be repeatable, each of its steps must be repeat- able. Figure 1 illustrates the typical steps of a simulation experiment in systems biology. We and others have extensively reviewed the technical and cultural challenges to repeating each of these steps [8, 9]. Here, we outline the key bottlenecks to the repeatability of simulations of mechanistic models in systems biology and provide practical recommendations for increasing their repeatability. In many cases, we recommend that investigators adopt practices and resources that are already widely used by the software engineering industry [10, 11].

2 2. Toward Repeatability and Reproducibility

To repeat a study, a researcher first needs access to the same data and software that was used in the original study, ideally in popular formats that many researchers can utilize, and with detailed metadata about how to use it, its meaning, and its provenance [12]. Access to identical hardware is typically not necessary because most researchers use similar hardware thanks to substantial standardization among computer hardware manufacturers. Access to the same operating system is also typically not needed due to extensive efforts to standardize the execution of most popular programming languages across platforms. When a specific operating system is needed, we encourage investigators to use containerization and virtualization technologies.

2.1. Formats for Experimental Data and Knowledge Because computational models in systems biology are typically based on experimental data and knowledge, the ability to repeat simulation results requires this information to be pub- lished, ideally in a form that is both readily understandable by other investigators and machine-readable. In our opinion, there are many good formats for sharing different types of data and knowledge. For example, the XLSX format is often a good choice for small data sets because most researchers are familiar with the format and XLSX files are readily read- able by both humans and machines. The HDF5 format [13] can be a good choice for larger data sets because it can capture multidimensional data and metadata and is supported by most popular programming languages. Efforts to develop more expressive and higher per- formance formats for data include xarray [14] and Zarr (https://zarr.readthedocs.io/). BioPAX is an excellent format for capturing knowledge about biochemical pathways [15]. Whichever format investigators choose, it is critical to include metadata about the semantic meaning and provenance of this information, including information about how the data was processed and the assumptions the processing employed.

2.2. Formats for Models Currently, models in systems biology are described using a broad range of formats including textual descriptions of equations in articles and supplementary materials; code for pro- gramming languages such as C/C++, Java, MATLAB, Python, and R; and domain-specific formats such as CellML [16] and the Systems Biology Markup Language (SBML) [17]. While pioneering studies may need to use custom formats to describe innovative experiments, this broad range of formats frustrates peer reviewers’ abilities to review models and compli- cates investigators’ abilities to reuse models. To facilitate quality control and collaboration, where possible, we recommend that investigators embrace commonly used domain-specific formats such as CellML and SBML. The benefits of using domain-specific abstractions such as CellML and SBML are exemplified by the TEX/LATEX document preparation system initially developed by Knuth in the late 1970’s [18] . Although Knuth’s original implementation of TEX is no longer used, TEX files 3 created over 40 years ago remain recompilable and editable today in large part because Knuth abstracted the description of TEX files from the software tools used to render them into documents and because Knuth developed standards both for TEX files and the documents they specify [19]. These abstractions have enabled developers to create and maintain newer TEX compilers. Some of the benefits of domain-specific formats are the abilities to develop domain-specific tools for validating models and to exchange models among software tools. For example, libSBML [20] provides methods for validating models. Importantly, the ability to exchange models among tools ensures that modelers will be able to reuse models into the future, even if the original software tool used to develop the model is no longer available. For example, models developed in the 2000’s with tools such as Gepasi [21] and Jarnac [22] that were exported to SBML can still be executed with the modern tools catalogued at http://sbml.org such as COPASI [23], Tellurium [24], and Virtual Cell [25]. Irrespective of which formats investigators choose to use, it is important to include metadata that enables others to evaluate and reuse models. At a minimum, models should include information about how to execute the model, the meaning of each variable and equation, and how the model was constructed and validated [26–28]. Where possible, this information should be provided in a structured form that is understandable by machines. For example, the Simulation Experiment Description Markup Language (SED-ML) [29] and the Kinetic Simulation Algorithm Ontology (KiSAO) [30] can often be used to describe simulations of models. When SED-ML cannot be used, instructions for simulating models can be provided as scripts or workflows. PEtab [31] is an emerging standard for capturing how models are calibrated. In many cases, modelers can use tools such as SBMLsqueezer [32] and SemGen [33] and ontologies such as the Systems Biology Ontology (SBO) [30] and the Ontology of Physics for Biology (OPB) [34] to concretely describe the meaning of the biology represented by models. The Systems Biology Graphical Notation (SBGN) [35, 36] can be used to describe diagrammatic depictions of models. New formats and ontologies must be developed to help modelers better capture the data, assumptions, and design decisions used to construct models.

2.3. Tools for Verifying Models Before data and models are shared with the community, it is important for authors to verify their findings to help ensure that other investigators focus their efforts on building upon correct results. For the most part, model verification in systems biology is still ad hoc and piecemeal. We believe that many investigators could enhance the quality of their data and models by adopting processes that are widely used among the software engineering industry. First, we encourage modelers to develop a list of ‘unit’ tests that evaluate whether their simulations produce accurate results, ideally across all of the outputs of their simulations and across all of the conditions that can be captured by their simulations. Such unit tests can be organized using domain-independent frameworks such as Python’s unittest module or domain-specific frameworks such as MEMOTE [37] and SciUnit [38]. 4 Second, we encourage investigators to use continuous integration (CI) systems to automate the execution of their tests each time they revise their data, models, or code [39–42]. Used effectively, such CI systems can help investigators identify problems quickly, at a stage when they are easy to fix. Over the past five years, cloud CI systems such as CircleCI and GitHub Actions have made it easier to continuously integrate data, models, and code organized into GitHub repositories. In fact, GitHub provides Actions free for most academic projects.

2.4. Interfaces for Repeating Simulations Because modern scientific data and model files are usually complex, graphical user interfaces (GUIs) are often essential to making scientific data and models reusable by investigators other than their authors. Historically, developed GUIs primarily by implementing custom desktop and web-based software tools. While this approach enables developers to create custom GUIs, the amount of work required is often impractical. Over the past decade, Jupyter notebooks [43–46] have made it easier for authors to provide simple interactive, graphical windows into their data and models. These notebooks also make it easy for authors to bundle simulation experiments together with snapshots of their results and textual descriptions of their interpretation. For example, Stencila [47] enables authors to use Jupyter-like documents to publish ‘live’ articles [48] that have executable computational experiments embedded directly within them.

2.5. Formats for Packaging Simulation Studies Because multiple files are often required to repeat a simulation result, it is important for these files to be distributed together in a format that preserves their links. We encourage investigators to use the COMBINE archive format [49] to bundle such files. COMBINE archives are zip files that include manifest files that describe their contents. The COMBINE archive format is supported by a growing number of modeling software tools and model repositories.

2.6. Distributing Data and Models Over the past decade, it has become easier to publish data, models, and the software needed to reuse them. Popular avenues for sharing data and models include supplementary materials to papers; code repositories such as GitHub; domain-independent data repositories such as figshare [50], Dryad [51]), Harvard Dataverse (https://dataverse.harvard.edu/), and Zenodo [2]; and domain-specific repositories such as BioModels [52], OpenSeek [53, 54], and the Physiome Model Repository (PMR) [55, 56]. We recommend that investigators utilize archives that provide permanent storage with persistent identifiers such as BioModels and the PMR. Popular avenues for sharing the software needed to reuse data and models include code repositories such as GitHub and software package management systems such as CRAN and PyPI. Avenues for sharing Jupyter notebooks include Binder [57] and CoLab [58]. Recently, Docker images and image repositories such as BioContainers [59] and Docker Hub have 5 become a popular way to share software [60]. Such images make it easier for authors to share the often complex computational environments needed to reuse scientific data and models, including the software that works directly with scientific data and models, as well as all of its dependent libraries and even the operating system that it requires.

2.7. Curation Services While we believe that authors should bear most of the responsibility for reporting repeatable results, we also believe that independent curation services and journals should also play a stronger role in helping and ensuring that authors publish repeatable results. Ultimately, we aim to encourage journals to only accept articles that meet a minimum threshold, such as providing a publicly accessible version of each simulation experiment. Due the complexity of modern computational research and the limited resources that inves- tigators have for peer review, we believe dedicated curation services are needed both to help investigators prepare their work for dissemination and help journals rigorously evaluate both the validity and the reusability of submitted work. For example, our Center for Reproducible Biomedical Modeling (https://reproduciblebiomodels.org/) has begun to provide PLoS [61] reports of the reproducibility of computational results submitted to the journal. These reports outline whether the results reported by the authors can be recreated and whether the authors provide sufficient instructions for others to utilize their work. Others have recently launched similar efforts with the American Journal of Political Science [62], [63], and Physiome [64]. Over the past few years, the Journal of Open Source Software [65] has also begun to provide a similar service for scientific software tools.

3. Conclusion

Adoption of domain-independent tools such as Docker and Jupyter notebooks and domain- specific formats such as CellML, SBML, and SED-ML over the past 20 years has advanced the repeatability of computational systems biology studies. At present, many models are publicly accessible from repositories such as BioModels and PMR, the models available from these repositories often include basic metadata about model elements, and these models can often be reused with multiple simulation software tools. AS a result, by some estimates approximately 50% of published results can be repeated with reasonable amount of effort. Despite this progress, computational systems biology still has a long way to go to make most models reproducible and reusable. Key gaps in our ability to produce reusable models include limited tools for capturing the data and assumptions used to build models; limited adoption of newer formats such as PEtab, SED-ML, and the COMBINE archive format for describing model calibrations, specifying simulations, and bundling entire studies; and limited adoption of structured approaches to verify models such as unit testing and con- tinuous integration. Furthermore, our scientific culture continues to under-prioritize and under-reward reproducible research [66–71].

6 We believe that many of these issues can be tackled by increased adoption of community standards by existing tools and the development of new tools that fill the gaps in between them. In addition, independent curation services and public model repositories could play a vital role in teaching investigators how to conduct research more reproducibly, helping researchers prepare their work for dissemination, and helping authors and journals evaluate the reproducibility of results submitted for publication. Finally, we feel that it is critically important to shift the culture of science to more strongly value reproducibility and reusability. Because computational systems biology is not an isolated field, this issue must be addressed systemically across science and throughout the world. This will likely require reflecting these values in higher level drivers of science such as faculty hiring and promotion decisions. Together, increased adoption of existing domain resources, targeted development of new tools, expanded curation services, and a cultural shift toward reproducibility could substan- tially enhance the reproducibility and reusability of computational systems biology. In turn, more reusable scientific building blocks could dramatically accelerate systems biology and the attainment of ambitious goals such as comprehensive computational models of cells and organisms that should underpin personalized medicine in the future.

Funding

This work was supported by National Institute for Biomedical Imaging and Bioengineering award P41GM109824 and National Science Foundation award 1933453. The content ex- pressed here is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health, the National Science Foundation, the Icahn School of Medicine at Mount Sinai, the University of Auckland, the University of Connecticut, or the University of Washington.

Acknowledgements

We thank the COMBINE community [72] for many fruitful discussions that helped spur the ideas expressed here. All authors contributed equally to this work.

Declaration of competing interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported here.

References

[1] B. Roger, Opus Majus, 1267. Translation: Bacon, R., 2016, Part 1. Opus Majus, Volumes 1 and 2. University of Pennsylvania Press, SN - 9781512814064.

7 [2] I. Peters, P. Kraker, E. Lex, C. Gumpenberger, J. I. Gorraiz, Zenodo in the spotlight of traditional and new metrics, Front Res Metr Anal 2 (2017) 13. [3] K. Tiwari, S. Kananathan, M. G. Roberts, J. P. Meyer, M. U. Sharif Shohan, A. Xavier, M. Maire, A. Zyoud, J. Men, S. Ng, et al., Reproducibility in systems biology modelling, Mol Syst Biol 17 (2021) e9982. ** One of the first papers that systematically analyzed the irreproducibility of computational modeling in systems biology. [4] T. Hothorn, F. Leisch, Case studies in reproducibility, Brief Bioinform 12 (2011) 288–300. [5] National Academies of Sciences, Engineering, and Medicine and others, Reproducibility and replicability in science, National Academies Press, 2019. [6] B. Haibe-Kains, G. A. Adam, A. Hosny, F. Khodakarami, L. Waldron, B. Wang, C. McIntosh, A. Gold- enberg, A. Kundaje, C. S. Greene, et al., Transparency and reproducibility in artificial intelligence, Nature 586 (2020) E14–E16. [7] S. M. McKinney, A. Karthikesalingam, D. Tse, C. J. Kelly, Y. Liu, G. S. Corrado, S. Shetty, Reply to: Transparency and reproducibility in artificial intelligence, Nature 586 (2020) E17–E18. [8] V. L. Porubsky, A. P. Goldberg, A. K. Rampadarath, D. P. Nickerson, J. R. Karr, H. M. Sauro, Best practices for making reproducible biochemical models, Cell Syst 11 (2020) 109–120. [9] V. Porubsky, L. Smith, H. M. Sauro, Publishing reproducible dynamic kinetic models, Brief Bioinform (2020). [10] J. L. Hellerstein, S. Gu, K. Choi, H. M. Sauro, Recent advances in biomedical simulations: a manifesto for model engineering, F1000Res 8 (2019). [11] C. Sch¨olzel,V. Blesius, G. Ernst, A. Dominik, The impact of mathematical modeling languages on model quality in systems biology: A software engineering perspective, bioRxiv (2020). [12] M. D. Wilkinson, M. Dumontier, I. J. Aalbersberg, G. Appleton, M. Axton, A. Baak, N. Blomberg, J.-W. Boiten, L. B. da Silva Santos, P. E. Bourne, et al., The FAIR Guiding Principles for scientific data management and stewardship, Scientific Data 3 (2016). [13] M. Folk, G. Heber, Q. Koziol, E. Pourmal, D. Robinson, An overview of the HDF5 technology suite and its applications, in: Proceedings of the EDBT/ICDT 2011 Workshop on Array Databases, pp. 36–47. [14] S. Hoyer, J. Hamman, xarray: ND labeled arrays and datasets in Python, J Open Res Softw 5 (2017). [15] E. Demir, M. P. Cary, S. Paley, K. Fukuda, C. Lemer, I. Vastrik, G. Wu, P. D’eustachio, C. Schaefer, J. Luciano, et al., The BioPAX community standard for pathway data sharing, Nature Biotechnol 28 (2010) 935–942. [16] A. A. Cuellar, C. M. Lloyd, P. F. Nielsen, D. P. Bullivant, D. P. Nickerson, P. J. Hunter, An overview of CellML 1.1, a biological model description language, Simulation 79 (2003) 740–747. ** CellML is a one of the most commonly used format for describing models in systems biology. CellML is primarily used for models of animal . [17] S. M. Keating, D. Waltemath, M. K¨onig,F. Zhang, A. Dr¨ager,C. Chaouiya, F. T. Bergmann, A. Finney, C. S. Gillespie, T. Helikar, et al., SBML Level 3: an extensible format for the exchange and reuse of biological models, Mol Syst Biol 16 (2020) e9110. ** SBML is the most widely used format for describing models in systems biology. SBML’s abstractions facilitate the interpretation and reuse of models. [18] D. Knuth, TeXbook, Addison-Wesley Reading, 1984. [19] The TUG DVI Driver Standards Committee, The DVI Driver Standard, Level 0, 1992. [20] B. J. Bornstein, S. M. Keating, A. Jouraku, M. Hucka, LibSBML: an API library for SBML, Bioinfor- matics 24 (2008) 880–881. [21] P. Mendes, GEPASI: a software package for modelling the dynamics, steady states and control of biochemical and other systems., Comput Appl Biosci 9 (1993) 563–71. [22] S. H. M, Jarnac: a system for interactive metabolic analysis, in: S. J. Hofmeyr J-HS, Rohwer JM (Ed.), 9th Int BioThermoKinetics Meeting, Stellenbosch University Press, 2000, pp. 221–228. [23] F. T. Bergmann, S. Hoops, B. Klahn, U. Kummer, P. Mendes, J. Pahle, S. Sahle, COPASI and its

8 applications in , J Biotechnol 261 (2017) 215–220. [24] K. Choi, J. K. Medley, M. K¨onig,K. Stocking, L. Smith, S. Gu, H. M. Sauro, Tellurium: An extensible Python-based modeling environment for systems and synthetic biology, Biosystems 171 (2018) 74–79. [25] L. M. Loew, J. C. Schaff, The Virtual Cell: a software environment for computational , Trends Biotechnol 19 (2001) 401–406. [26] N. Le Nov`ere,A. Finney, M. Hucka, U. S. Bhalla, F. Campagne, J. Collado-Vides, E. J. Crampin, M. Halstead, E. Klipp, P. Mendes, et al., Minimum information requested in the annotation of bio- chemical models (MIRIAM), Nature Biotechnol 23 (2005) 1509–1515. [27] L. Mulugeta, A. Drach, A. Erdemir, C. A. Hunt, M. Horner, J. P. Ku, J. G. Myers Jr, R. Vadigepalli, W. W. Lytton, Credibility, replicability, and reproducibility in simulation for biomedicine and clinical applications in , Front Neuroinform 12 (2018) 18. [28] A. Erdemir, L. Mulugeta, J. P. Ku, A. Drach, M. Horner, T. M. Morrison, G. C. Peng, R. Vadigepalli, W. W. Lytton, J. G. Myers, Credible practice of modeling and simulation in healthcare: ten rules from a multidisciplinary perspective, J Transl Med 18 (2020) 1–18. * Guidelines for good practices in multi-scale modeling from the Interagency Modeling and Analysis Group. [29] D. Waltemath, R. Adams, F. T. Bergmann, M. Hucka, F. Kolpakov, A. K. Miller, I. I. Moraru, D. Nickerson, S. Sahle, J. L. Snoep, N. Le Nov`ere, Reproducible computational biology experiments with SED-ML - The Simulation Experiment Description Markup Language, BMC Syst Biol 5 (2011) 198. ** SED-ML is the main standard for describing simulations in computational systems biology. SED-ML can be used with any model language such as CellML and SBML. [30] M. Courtot, N. Juty, C. Kn¨upfer, D. Waltemath, A. Zhukova, A. Dr¨ager,M. Dumontier, A. Finney, M. Golebiewski, J. Hastings, et al., Controlled vocabularies and semantics in systems biology, Mol Syst Biol 7 (2011) 543. [31] L. Schmiester, Y. Sch¨alte,F. T. Bergmann, T. Camba, E. Dudkin, J. Egert, F. Fr¨ohlich, L. Fuhrmann, A. L. Hauber, S. Kemmer, et al., PEtab–interoperable specification of parameter estimation problems in systems biology, PLoS Comput Biol 17 (2021) e1008646. [32] A. Dr¨ager,D. C. Zielinski, R. Keller, M. Rall, J. Eichner, B. O. Palsson, A. Zell, SBMLsqueezer 2: context-sensitive creation of kinetic equations in biochemical networks, BMC Syst Biol 9 (2015) 1–17. [33] M. L. Neal, C. T. Thompson, K. G. Kim, R. C. James, D. L. Cook, B. E. Carlson, J. H. Gennari, Sem- Gen: a tool for semantics-based annotation and composition of biosimulation models, Bioinformatics 35 (2019) 1600–1602. [34] D. L. Cook, M. L. Neal, F. L. Bookstein, J. H. Gennari, Ontology of Physics for Biology: representing physical dependencies as a basis for biological processes, J Biomed Semant 4 (2013) 1–8. [35] N. L. Nov`ere,M. Hucka, H. Mi, S. Moodie, F. Schreiber, A. Sorokin, E. Demir, K. Wegner, M. I. Aladjem, S. M. Wimalaratne, F. T. Bergman, R. Gauges, P. Ghazal, H. Kawaji, L. Li, Y. Matsuoka, A. Vill´eger,S. E. Boyd, L. Calzone, M. Courtot, U. Dogrusoz, T. C. Freeman, A. Funahashi, S. Ghosh, A. Jouraku, S. Kim, F. Kolpakov, A. Luna, S. Sahle, E. Schmidt, S. Watterson, G. Wu, I. Goryanin, D. B. Kell, C. Sander, H. Sauro, J. L. Snoep, K. Kohn, H. Kitano, The Systems Biology Graphical Notation, Nature Biotechnol 27 (2009) 735–741. [36] A. Rougny, sbgntikz–a TikZ library to draw SBGN maps, Bioinformatics 35 (2019) 4499–4500. [37] C. Lieven, M. E. Beber, B. G. Olivier, F. T. Bergmann, M. Ataman, P. Babaei, J. A. Bartell, L. M. Blank, S. Chauhan, K. Correia, et al., MEMOTE for standardized genome-scale metabolic model testing, Nature Biotechnol 38 (2020) 272–276. [38] C. Omar, J. Aldrich, R. C. Gerkin, Collaborative infrastructure for test-driven scientific model valida- tion, in: Proc 36th Int Conf Softw Eng, pp. 524–527. [39] M. Meyer, Continuous integration and its tools, IEEE Softw 31 (2014) 14–16. [40] C. F. Lopez, J. L. Muhlich, J. A. Bachman, P. K. Sorger, Programming biological models in Python using PySB, Mol Syst Biol 9 (2013) 646. [41] B. K. Beaulieu-Jones, C. S. Greene, Reproducibility of computational workflows is automated using

9 continuous analysis, Nature Biotechnol 35 (2017) 342–346. [42] M. Krafczyk, A. Shi, A. Bhaskar, D. Marinov, V. Stodden, Scientific tests and continuous integration strategies to enhance reproducibility in the scientific software context, in: Proc 2nd Int Workshop Pract Reproducible Eval Comput Syst, pp. 23–28. [43] J. M. Perkel, Why Jupyter is data scientists’ computational notebook of choice., Nature 563 (2018) 145–147. [44] J. K. Medley, K. Choi, M. K¨onig,L. Smith, S. Gu, J. Hellerstein, S. C. Sealfon, H. M. Sauro, Tellurium notebooks–An environment for reproducible dynamical modeling in systems biology, PLoS Comput Biol 14 (2018) e1006220. [45] M. Badenhorst, C. J. Barry, C. J. Swanepoel, C. T. Van Staden, J. Wissing, J. M. Rohwer, Workflow for data analysis in experimental and computational systems biology: Using Python as ‘glue’, Processes 7 (2019) 460. [46] K. Hinsen, ActivePapers: a platform for publishing and archiving computer-aided research, F1000Res 3 (2014). [47] M. Aufreiter, A. Pawlik, N. Bentley, Stencila–an office suite for reproducible research, eLife Labs 2 (2018). [48] L. M. Lewis, M. C. Edwards, Z. R. Meyers, C. C. Talbot Jr, H. Hao, D. Blum, et al., Replication study: Transcriptional amplification in tumor cells with elevated c-myc, eLife 7 (2018) e30274. [49] F. T. Bergmann, R. Adams, S. Moodie, J. Cooper, M. Glont, M. Golebiewski, M. Hucka, C. Laibe, A. K. Miller, D. P. Nickerson, B. G. Olivier, N. Rodriguez, H. M. Sauro, M. Scharm, S. Soiland-Reyes, D. Waltemath, F. Yvon, N. Le Nov`ere, COMBINE archive and OMEX format: one file to share all information to reproduce a modeling project, BMC Bioinform 15 (2014) 369. [50] J. Singh, FigShare, J Pharmacol Pharmacother 2 (2011) 138–139. [51] H. White, S. Carrier, A. Thompson, J. Greenberg, R. Scherle, The Dryad Data Repository: A singapore framework metadata architecture in a DSpace environment., in: Dublin core conference, pp. 157–162. [52] R. S. Malik-Sheriff, M. Glont, T. V. Nguyen, K. Tiwari, M. G. Roberts, A. Xavier, M. T. Vu, J. Men, M. Maire, S. Kananathan, et al., BioModels–15 years of sharing computational models in life science, Nucleic Acids Res 48 (2020) D407–D415. * BioModels is one of the largest repositories of systems biology models. BioModels is primarily used to publish models that are encoded in SBML. Importantly, BioModels’ Team work to ensure that each curated model can reproduce at least one reported result. [53] U. Wittig, M. Rey, A. Weidemann, W. M¨uller, Data management and data enrichment for systems biology projects, J Biotechnol 261 (2017) 229–237. [54] K. Wolstencroft, S. Owen, O. Krebs, Q. Nguyen, N. J. Stanford, M. Golebiewski, A. Weidemann, M. Bittkowski, L. An, D. Shockley, et al., SEEK: a systems biology data and model management platform, BMC Syst Biol 9 (2015) 1–12. [55] C. M. Lloyd, J. R. Lawson, P. J. Hunter, P. F. Nielsen, The CellML model repository, Bioinformatics 24 (2008) 2122–2123. [56] T. Yu, C. M. Lloyd, D. P. Nickerson, M. T. Cooling, A. K. Miller, A. Garny, J. R. Terkildsen, J. Lawson, R. D. Britten, P. J. Hunter, et al., The Physiome Model Repository 2, Bioinformatics 27 (2011) 743– 744. ** PMR is one of the largest repositories of systems biology models. PMR is primarily used to publish models described using CellML. [57] B. Ragan-Kelley, C. Willing, Binder 2.0-reproducible, interactive, sharable environments for science at scale, in: Proc 17th Python Sci Conf, pp. 113–120. [58] J. Jeon, H. U. Kim, Setup of a scientific computing environment for computational biology: Simulation of a genome-scale metabolic model of Escherichia coli as an example, J Microbiol 58 (2020) 227–234. [59] F. da Veiga Leprevost, B. A. Gr¨uning,S. Alves Aflitos, H. L. R¨ost,J. Uszkoreit, H. Barsnes, M. Vaudel, P. Moreno, L. Gatto, J. Weber, et al., BioContainers: an open-source and community-driven framework for software standardization, Bioinformatics 33 (2017) 2580–2582. [60] D. N¨ust,V. Sochat, B. Marwick, S. J. Eglen, T. Head, T. Hirst, B. D. Evans, Ten simple rules for

10 writing Dockerfiles for reproducible data science, PLoS Comput Biol (2020). [61] J. A. Papin, F. Mac Gabhann, H. M. Sauro, D. Nickerson, A. Rampadarath, Improving reproducibility in computational biology research, PLoS Comput Biol (2020). * This paper describes one of the first pilot studies to use outside curation to improve modeling papers in systems biology. [62] Editors of American Political Science Review, Notes from the editors, Am Pol Sci Rev 113 (2019) iii–vii. [63] R. D. Peng, Reproducible research and biostatistics, Biostatistics 10 (2009) 405–408. [64] P. Hunter, The Virtual Physiological Human: the Physiome project aims to develop reproducible, multiscale models for clinical practice, IEEE Pulse 7 (2016) 36–42. [65] A. M. Smith, K. E. Niemeyer, D. S. Katz, L. A. Barba, G. Githinji, M. Gymrek, K. D. Huff, C. R. Madan, A. C. Mayes, K. M. Moerman, et al., Journal of Open Source Software (JOSS): design and first-year review, PeerJ Comput Sci 4 (2018) e147. [66] S. Schnell, “reproducible” research in mathematical sciences requires changes in our peer review culture and modernization of our current publication approach, Bull Math Biol 80 (2018) 3095–3105. [67] E. K. Samota, R. P. Davey, Knowledge and attitudes among life scientists towards reproducibility within journal articles, BioRxiv (2019) 581033. [68] I. Hrynaszkiewicz, Publishers’ responsibilities in promoting data quality and reproducibility, in: Good Research Practice in Non-Clinical and Biomedicine, Springer, Cham, 2019, pp. 319–348. [69] R. Heesen, Why the reward structure of science makes reproducibility problems inevitable, J Philos 115 (2018) 661–674. [70] W. Quan, B. Chen, F. Shu, Publish or impoverish: An investigation of the monetary reward system of science in China (1999-2016), Aslib J Inform Manag (2017). [71] A. W. Cowley, Jr, A journalistic break through, Function 1 (2020). [72] D. Waltemath, M. Golebiewski, M. L. Blinov, P. Gleeson, H. Hermjakob, M. Hucka, E. T. Inau, S. M. Keating, M. K¨onig,O. Krebs, et al., The first 10 years of the international coordination network for standards in systems and synthetic biology (COMBINE), J Integr Bioinform 17 (2020).

11