Embodying a Computational Model of Hippocampal Replay for Robotic Reinforcement Learning Matthew T. Whelan Supervisors: Prof. E. Vasilaki and Prof. T. J. Prescott Department of Computer Science The University of Sheffield This dissertation is submitted for the degree of Master of Philosophy August 2020 Abstract Hippocampal reverse replay has been speculated to play an important role in biological rein- forcement learning since its discovery over a decade ago. Whilst a number of computational models have recently emerged in an attempt to understand the dynamics of hippocampal replay, there has been little progress in testing and implementing these models in real-world robotics settings. Presented first in this body of work then is a bio-inspired hippocampal CA3 network model. It runs in real-time to produce reverse replays of recent spatio-temporal sequences, represented as place cell activities, in a robotic spatial navigation task. The model is based on two very recent computational models of hippocampal reverse replay. An analysis of these models show that, in their original forms, they are each insufficient for effective performance when applied to a robot. As such, choosing particular elements from each allows for a computational model that is sufficient for application in a robotic task. Having a model of reverse replay applied successfully in a robot provides the groundwork necessary for testing the ways in which reverse replay contributes to reinforcement learning. The second portion of the work presented here builds on a previous reinforcement learning neural network model of a basic hippocampal-striatal circuit using a three-factor learning rule. By integrating reverse replays into this reinforcement learning model, results show that reverse replay, with its ability to replay the recent trajectory both in the hippocampal circuit and the striatal circuit, can speed up the learning process. In addition, for situations where the original reinforcement learning model performs poorly, such as when its time dynamics do not sufficiently store enough of the robot’s behavioural history for effective learning, the reverse replay model can compensate for this by replaying the recent history. These results are inline with experimental findings showing that disruption of awake hippocampal replay events severely diminishes, but does not entirely eliminate, reinforcement learning. This work provides possible insights into the important role that reverse replays could contribute to mnemonic function, and reinforcement learning in particular; insights that could benefit the robotic, AI, and neuroscience communities. However, there is still much tobe done. How reverse replays are initiated is still an ongoing research problem, for instance. Furthermore, the model presented here generates place cells heuristically, but there are iv Abstract computational models tackling the problem of how hippocampal cells such as place cells, but also grid cells and head direction cells, emerge. This leads to the pertinent question of asking how these models, which make assumptions about their network architectures and dynamics, could integrate with the computational models of hippocampal replay which make their own assumptions on network architectures and dynamics. List of figures 1.1 Example of the Morris Water Maze task . .3 2.1 Comparing the hippocampus with a seahorse . 10 2.2 Line drawing of a horizontal section of the hippocampus . 11 2.3 The hippocampal trisynaptic circuit . 12 2.4 A more complete hippocampal systems diagram . 24 3.1 Plot of dendritic spiking . 26 3.2 Synfire-like chain of cell assemblies for producing hippocampal replays . 27 3.3 Weight magnitudes upon initialisation of the network in Haga and Fukai (2018) 31 3.4 Plots showing the replication of the hippocampal replay model by Haga and Fukai (2018) . 35 3.5 Weight changes in the model by Haga and Fukai (2018) . 36 3.6 Showing the network instability in Haga and Fukai (2018) . 38 3.7 Unbounded weight changes cause the instability in Haga and Fukai (2018) . 38 4.1 Simulated MiRo robot with place cells overlaid onto the environment . 45 4.2 Image of the MiRo robot . 49 4.3 Plots of the rates and intrinsic plasticities for the place cell network during exploration and a replay event . 50 4.4 Time course plots of place cells involved in a replay event . 51 4.5 Replay event with intrinsic plasticity removed . 52 5.1 MiRo’s testing environment with the place cell to action cell network . 59 5.2 Results for the non-replay case in order to test that the derived learning rule performs well . 73 5.3 Comparing the effects of a small eligibility trace time constant with and without reverse replays . 75 5.4 Comparing the eligibility trace during reward retrieval . 76 vi List of figures 5.5 Comparison of performance for when the learning rate is small. 77 5.6 Comparing the best cases with and without reverse replays . 78 5.7 Comparing average performance across the last 10 trials . 79 5.8 Population weight vectors after reward retrieval in the non-replay, replay, and modified learning rule cases . 80 5.9 Performance results for the modified learning rule using a post-synapse only eligibility trace time constant . 81 A.1 Example of a straight trajectory by MiRo . 104 A.2 Plots of place cell rates and intrinsic plasticities for the straight line trajectory 105 A.3 Line plots of the place cell rates for the straight line trajectory . 106 A.4 Full results for τe = 0:04s ........................... 107 A.5 Full results for τe = 0:2s ........................... 108 A.6 Full results for τe = 1s ............................ 108 A.7 Full results for τe = 5s ............................ 109 A.8 Full results for h = 0:001 . 109 List of tables 3.1 Summary of computational hippocampal replay models . 29 4.1 Model parameter values used in the experiments for hippocampal reverse replays. 48 5.1 Model parameter values used in the reinforcement learning experiments. 72 Nomenclature Roman Symbols c1, c2 Constants in the action cell Sigmoidal activation function d Constant determining the place field width Dk Short-term depression for neuron k ei j Synaptic eligibility trace at the synapse connecting place cell j to action cell i Fk Short-term facilitation for neuron k I Current Mtarget Magnitude of the population vector of action cells R Reward value ri Rate for place cell i (model of Haga and Fukai) s j Neuron spikes (= 1 or 0; model of Pang and Fairhall) U Constant in the short-term facilitation dynamics vi Cell voltage (model of Pang and Fairhall) wi j Weight projecting from neuron j onto neuron i Xj Place field x-coordinate centre location for place cell j x j Rate for place cell j xy Constant that determines the Sigmoidal midpoint in the calculation for intrinsic plasticity XMiRo MiRo’s x-coordinate position x Nomenclature yi Rate for action cell i Yj Place field y-coordinate centre location for place cell j YMiRo MiRo’s y-coordinate position Greek Symbols a Constant in place cell linear rectifier b Determines the Sigmoidal slope in the intrinsic plasticity computation e Constant in place cell linear rectifier h Learning rate y j Intrinsic plasticity for place cell j s Standard deviation si Intrinsic plasticity for place cell i (model of Pang and Fairhall) q MiRo’s heading in the environment qi Heading that action cell i codes for τ Time constant Table of contents Abstract iii List of figuresv List of tables vii Nomenclature ix 1 Introduction1 1.1 A Brief Introduction to the Hippocampus . .2 1.1.1 Place Cells . .3 1.1.2 Hippocampal Replay . .4 1.1.3 Computational and Robotic Modelling of Hippocampal Replay . .5 1.1.4 Hippocampal Replay and its Role in Reinforcement Learning . .6 1.2 Motivation for this Thesis . .7 1.3 Structure of the Thesis . .7 2 Review of Hippocampal Formation Neuroanatomy9 2.1 Overview of Hippocampal Formation Neuroanatomy . 11 2.2 Entorhinal Cortex . 13 2.2.1 Cytoarchitecture . 13 2.2.2 Extrahippocampal Connections . 14 2.2.3 Intrahippocampal Connections . 15 2.3 Dentate Gyrus . 15 2.3.1 Cytoarchitecture . 15 2.3.2 Intrahippocampal Connections . 16 2.3.3 Extrahippocampal Connections . 17 2.4 Hippocampus . 18 2.4.1 Cytoarchitecture . 18 xii Table of contents 2.4.2 Intrahippocampal Connections . 19 2.4.3 Extrahippocampal Connections . 20 2.5 Subiculum, Presubiculum and Parasubiculum . 21 2.5.1 Cytoarchitecture . 22 2.5.2 Intrahippocampal Connections . 22 2.5.3 Extrahippocampal Connections . 23 2.6 Hippocampal Neuroanatomy: Summary . 24 3 Computational Models of Hippocampal Replay 25 3.1 Overview of the Models . 25 3.2 A Further Examination of Two Computational Models . 30 3.2.1 Model of Haga and Fukai (2018) . 30 3.2.2 Extracting Intrinsic Plasticity from Pang and Fairhall (2019) . 39 3.3 Hippocampal Replay for Robotic Applications . 40 4 Developing a Model of Hippocampal Replay in the MiRo Robot 43 4.1 Model Details . 44 4.1.1 Network Architecture . 44 4.1.2 Network Dynamics . 44 4.1.3 Model Parameters . 47 4.2 Experimental Setup and Results . 48 4.2.1 MiRo Robot and the Testing Environment . 48 4.2.2 Searching for a Hidden Reward . 49 4.2.3 Removing intrinsic plasticity . 51 4.3 Analysing the Model and Results . 52 5 Employing Reverse Replays in a Robotic Reinforcement Learning Task 55 5.1 Background . 56 5.2 A Hippocampal-Striatal Model . 58 5.2.1 Network Architecture . 58 5.2.2 Hippocampal Place Cells . 58 5.2.3 Striatal Action Cells . 60 5.2.4 Place Cell to Action Cell Synaptic Plasticity . 63 5.2.5 Review of the Implementation . 67 5.2.6 A Post-Synapse Only Eligibility Trace . 70 5.3 Experimental Results . 71 5.3.1 Testing the Learning Rule Without Reverse Replays .