Towards Improved Sharing of Model Code in Computational Cognitive Neuroscience

Martin Wiener ([email protected]) Department of Psychology, George Mason University 4400 University Drive Fairfax, VA 22207 USA

Abstract:

Computational models of cognitive processes are a pillar of data sharing is more common. Here, I suggest challenges for cognitive neuroscience. Recently, the number and diversity of the sharing of model code and how they can be overcome. these models has increased, and are being increasingly applied to neuroscience datasets. Yet, the sharing of the code underlying these models has not kept pace with their Challenges for Sharing development. With the advance of open science frameworks for the neuroscience community, the sharing of model code is Recently, the challenges for greater sharing and transparency imperative for new discoveries and model comparison. Here, I in neuroscience were outlined (Wiener, et al. 2016). Chief advocate for renewed efforts to share the code for our models, among these challenges are ones of Incentivization (why and chart a path for greater transparency within the growing computational cognitive neuroscience community. should researchers share?), Discoverability (how will they find shared data/code?), and Sustainability (how will repositories be maintained?). The sharing of model code also Keywords: Data Sharing; Code Sharing; Computational carries unique challenges for computational cognitive Modeling; Cognitive Models; Open Science Framework neuroscience; although the size of any library of code for a computational model is much smaller than the size of a Introduction neuroscience dataset, model code may be more complicated. One reason for this is the heterogeneity of programming Computational modeling of cognitive processes has been a languages in which model code exists (Addyman & French, pillar of cognitive science since the advent of the field, over 2012). Many researchers design model code in their favorite 50 years ago. These models of cognition share an origin with programming languages, which may differ even within the high-level computing languages from which a multitude of same lab, depending on how comfortable the researcher is model types have been developed to understand and predict with that language. These programs are typically written not the performance of cognitive agents. Similarly, models of with the intention that they will be shared, but that they will neuroscientific processes also form a cornerstone of their work. This heterogeneity presents a particular challenge of field, in order to implement and understand the level at which interpretability for sharing, as another researcher must be neural effects occur. In the combinatory field of cognitive familiar with the language that the code is written in to neuroscience, computational models are increasingly interpret how it works. pervasive, with an exponential increase in the number of papers discussing or implementing them over the past two Recent large-scale community engagement has led to a decades (Palminteri, et al. 2017). proliferation of online tools for open science and sharing (e.g. Open Science Framework1; Neuroscience Information With the increase in model proliferation for cognitive Framework3). For the sharing of model code, online science, a new framework is needed for the dissemination, repositories such as https://sourceforge.net and review, and replication of model findings. The reason is that, https://github.com allow for researchers to easily and although modelling is more common, the sharing of code adaptively upload and maintain online directories of code underlying these models is not (Addyman & French, 2012). projects, with added features such as version control and With new initiatives in scientific data-sharing, such as the unique identifiers (DOIs). Other recent repositories, such as Open Science Framework1 and Center for Open Science2, the https://figshare.com, allow researchers to easily reproduce sharing of model and analysis code should similarly become key findings from research papers. Yet, all this openness still easier and more common. Yet, greater transparency and requires cognitive neuroscientists to hunt for the code they openness for model code is not a fait accompli, just because

1 https://osf.io/ 3 https://neuinfo.org/ 2 https://cos.io/ are after, leaving open the possibility that particular projects Discrepancies in model implementation can muddy their or repositories will be undiscovered. interpretation when the same model disagrees between labs.

How to Improve Sharing of Model Code Yet, the opportunity here is tremendous. Armed with a resource for cognitive models, researchers may identify, In order to improve model sharing in cognitive neuroscience, reproduce, and compare key model findings both within and I suggest that our community needs an online clearinghouse across models. This is the true power of model development for model code and computational models of cognition. To – determining which model “wins” across a variety of options accomplish this, it is recommended that we adopt the practice for explaining a given phenomenon. There has been rapid of the ModelDB4 architecture, in which the computational development towards this goal in systems neuroscience neuroscience community has shared models of basic neural (Almog & Korngreen, 2016), and so it is now needed in processes for the past 20 years. Specifically, this cognitive neuroscience. If further researchers continue to clearinghouse need not be where the code itself is stored. push their models to version control repositories like GitHub Rather, it may serve a purpose similar to the NITRC5 or https://bitbucket.org/, a future feature of a cognitive model database for neuroimaging; a resource from which clearinghouse could be to “pull” distinct models from computational cognitive models may be identified and different studies and apply them to same dataset, and then reached on their own servers. compare which model explains the data better.

To implement a solution like this, given the challenges to Conclusion effective sharing, will require several criteria to be met. First, the success of any database is dependent on the metadata This is not the first time such a call has been made for greater framework that underlies it. For cognitive science, a number openness and transparency in cognitive modeling, and of metadata frameworks exist that can be used to support any cognitive model databases have been attempted before. What model aggregation efforts. These frameworks, such as the is most needed now is initial support for the project, from Cognitive Atlas6 can be used to categorize and identify leaders in the computational cognitive neuroscience models by the cognitive process that they simulate. Second, community, to ensure that the momentum of open science any database will require community buy-in – a minimum extends to cognitive models and enables future researchers to number of users will be necessary to commit to the support conduct the most groundbreaking research possible. of this database, by volunteering their code. Third, a clearinghouse will need to be sustained. Thankfully, given References the low memory requirements of such a database, the general cost will be needed up front to initiate the effort, followed by lower expected costs to maintain. Yet, with increased feature Addyman, C., & French, R. M. (2012). Computational implementation there will be an increasing need for modeling in cognitive science: A manifesto for continued, dedicated support. change. Topics in cognitive science, 4(3), 332-341.

Almog, M., & Korngreen, A. (2016). Is realistic What is at stake? neuronal modeling realistic?. Journal of Neurophysiology, 116(5), 2180-2209. Is a clearinghouse for cognitive models really necessary? Current trends indicate that future cognitive and neuro- Eglen, S., Marwick, B., Halchenko, Y., Hanke, M., Sufi, scientists are becoming increasingly computationally savvy, S., Gleeson, P., ... & Wachtler, T. (2016). Towards as such skills are necessary to succeed, and community standard practices for sharing computer code and engagement is coalescing around best practices for the programs in neuroscience. Nature Neuroscience, development and sharing of analysis code in these fields 20(6) 770-773. (Eglen, et al. 2017). Yet, this process is likely to be slow, as full pervasiveness of a new generation of researchers will Palminteri, S., Wyart, V., & Koechlin, E. (2017). The take time. The bigger danger is that, as models become more Importance of Falsification in Computational complex and widespread, errors in replication may occur. Cognitive Modeling. Trends in Cognitive Sciences. That is, how can one be certain that their implementation of a given model is the same as another researcher’s? Wiener, M., Sommer, F. T., Ives, Z. G., Poldrack, R. A., & Litt, B. (2016). Enabling an Open Data Ecosystem for the Neurosciences. Neuron, 92(3), 617-621.

4 https://senselab.med.yale.edu/modeldb/ 6 http://www.cognitiveatlas.org/ 5 http://www.nitrc.org/