Bio-Instantiated Recurrent Neural Networks
Total Page:16
File Type:pdf, Size:1020Kb
bioRxiv preprint doi: https://doi.org/10.1101/2021.01.22.427744; this version posted January 23, 2021. The copyright holder for this preprint (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made available under aCC-BY-NC 4.0 International license. Bio-instantiated recurrent neural networks Alexandros Goulas 1 Fabrizio Damicelli 1 Claus C Hilgetag 1 2 Abstract 1. Introduction Recent breakthroughs in artificial neural network research Biological neuronal networks (BNNs) constitute have generated a renewed interest in the intersection of bio- a niche for inspiration and analogy making for logical and artificial neural systems (Richards et al., 2019; researchers that focus on artificial neuronal net- Lillicrap et al., 2020; Saxe et al., 2021). This intersection works (ANNs). Moreover, neuroscientists increas- primarily focuses on the biological plausibility of learning ingly use ANNs as a model for the brain. How- rules (Bartunov et al., 2018; Lillicrap et al., 2020) or the ever, apart from certain similarities and analogies similarity of visual stimuli representations in ANNs and that can be drawn between ANNs and BNNs, such BNNs (Cadieu et al., 2014; Gu¨c¸lu¨ & van Gerven, 2015). networks exhibit marked differences, specifically Explicit comparisons of the network topology of BNNs with respect to their network topology. Here, we and ANNs are less prominent. This gap primarily exists investigate to what extent network topology found due to the fact that, despite certain analogies and inspira- in nature can lead to beneficial aspects in recur- tion derived from neuroscientific findings, groundbreaking rent neural networks (RNNs): i) the prediction advancements in ANNs are driven by engineering goals performance itself, that is, the capacity of the net- and computing power (He et al., 2015; Srivastava et al., work to minimize the desired function at hand in 2015; Hochreiter & Schmidhuber, 1997) and not by direct test data and ii) speed of training, that is, how fast translations of empirical data pertaining to neural network during training the network reaches its optimal topologies found in nature. Recent studies that examine net- performance. To this end, we examine different work topologies of ANNs only focus on generic similarities ways to construct RNNs that instantiate the net- with biological systems. For instance, building feedforward work topology of brains of different species. We networks that do not have their weights optimized with refer to such RNNs as bio-instantiated. We exam- backpropagation, but are instead based on abstract topology ine the bio-instantiated RNNs in the context of models that bear certain similarities with BNNs, such as a key cognitive capacity, that is, working mem- the Watts-Strogatz model (Watts & Strogatz, 1998) or the ory, defined as the ability to track task-relevant Barabasi–Albert´ model (Albert & Barabasi´ , 2002), leads to information as a sequence of events unfolds in competitive performances in image classification tasks (Xie time. We highlight what strategies can be used to et al., 2019). In the context of image recognition, the enrich- construct RNNs with the network topology found ment of convolutional networks with recurrent connections, in nature, without sacrificing prediction capacity mimicking a key feature of BNNs (Markov et al., 2012), and speed of training. Despite that we observe leads to competitive performance in benchmark tasks, even no enhancement of performance when compared outperforming state-of-the-art networks without recurrence to randomly wired RNNs, our approach demon- (Ming Liang & Xiaolin Hu, 2015). In addition, studies strates how empirical neural network data can be focusing on RNNs, specifically, echo state networks, and used for constructing RNNs, thus, facilitating fur- evolutionary algorithms applied to ANNs, demonstrate that ther experimentation with biologically realistic networks with a modular network architecture, reminiscent networks topology. of the modular nature of BNNs, exhibit better memory ca- pacity (Rodriguez et al., 2019) and adaptation to new tasks (Clune et al., 2013). However, such studies have only focused on network topolo- 1Institute of Computational Neuroscience, University Medical Center Hamburg-Eppendorf, Hamburg University, Martinistr 52 gies that bear generic similarities to BNNs and do not instan- 20246 Hamburg Germany 2Health Sciences Department, Boston tiate artificial networks with the actual, empirical network University, Boston, MA 02215, USA. Correspondence to: Alexan- topology that experimental work has unravelled. Hence, it is dros Goulas <[email protected]>. unknown how such empirically-discerned network topology can be incorporated in RNNs, what ramifications this incor- 1 bioRxiv preprint doi: https://doi.org/10.1101/2021.01.22.427744; this version posted January 23, 2021. The copyright holder for this preprint (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made available under aCC-BY-NC 4.0 International license. Bio-instantiated recurrent neural networks poration has and if and to what extent such network topol- comparison of ANNs and BNNs without sacrificing predic- ogy can lead to beneficial aspects, such as faster learning tion capacity and speed of training. A lack of enhancement (fewer training epochs) and better performance (minimiza- of the performance of bio-instantiated RNNs when com- tion of loss functions in test sets). Examining the effects of pared to randomly wired RNNs showcases the need for network topology is important, since the highly structured further modifications, possibly encompassing more biologi- network topology of animal brains is suggested to serve as cally realistic learning algorithms and activation functions. a structural prior that is important for rapid learning (Zador, It is important to highlight the distinction between network 2019). Notably, despite universal cross-species brain net- architecture and network topology, as currently used in this work topology principles, divergence of network topology report. State-of-the-art RNNs, such as LSTMs (Hochreiter is also discernible, for instance, when comparing human & Schmidhuber, 1997), exhibit a non-random engineered and monkey brains (Goulas et al., 2014). Therefore, the architecture (different types of interconnected gates), but effects of the incorporation of brain network topology from a random topology (all-to-all connectivity with randomly different species into ANNs should be examined. Here, we initialized weights). In addition, the classic Elman networks explicitly address this gap. Instead of examining network (Elman, 1990) exhibit a non-random architecture, for in- topologies that are biologically inspired or analogous to stance, the input layer connects to the hidden recurrent layer, BNNs, we build RNNs that are bio-instantiated, that is, they but not directly to the output layer. However, the topology embody the empirically discerned network topology found of the hidden recurrent layer is all-to-all, and, thus, in stark in natural neural systems, that is, monkey and human brains. contrast to the network topology of biological systems, such This is a necessary step to understand the possible links be- as the human and monkey brain network. tween the network topology of natural and artificial neural systems, and, thus, avoid looking at this relation through the lens of abstract network models. We examine the impact of 1.1. Related work bio-instantiated RNNs in a series of working memory tasks, In the realm of neuroscience, experimental work and net- that is, the ability to track relevant information as a sequence work analysis revealed that BNNs, such as the worm neural of events unfolds across time. Working memory is a key network, have a characteristic network topology, that is, neu- cognitive capacity of biological agents, extensively stud- rons in BNNs do not connect in a random or all-to-all fash- ied in cognitive psychology and neuroscience. Moreover, ion, but exhibit preferential connectivity that obeys certain such capacity is also desired in engineering tasks where wiring principles (Varshney et al., 2011; Motta et al., 2019). sequential data are processed. Comprehensive, single neuron connectivity at a large-scale, Our main goal is to investigate different strategies to con- whole-brain level in animals, such as humans, is currently struct bio-instantiated RNNs from empirical data on BNNs experimentally intractable. While such information is still and elucidate the significance of brain network topology of lacking, detailed experimental data reveal how neuronal diverse animals as a potentially advantageous structural populations in different brain regions of human and non- prior that would situate the system at hand in an advan- human animals are wired to each other. These experimental tageous starting point, that is, effecting performance and observations highlight that neuronal populations inhabiting speed of training when the networks is subsequently trained the various regions of animal brains, do not connect to each with a common learning algorithm, that is, backpropagation- other in a random or all-to-all fashion, but exhibit prefer- through-time. Since ANNs are hand-engineered and BNNs ential connectivity and connection weights, thus, forming are a product of evolution, apart from intuitive expecta- a characteristic network