Spike-Based Bayesian-Hebbian Learning in Cortical and Subcortical Microcircuits

PHILIP J. TULLY

Doctoral Thesis Stockholm, Sweden 2017 TRITA-CSC-A-2017:11 ISSN 1653-5723 KTH School of Computer Science and Communication ISRN-KTH/CSC/A-17/11-SE SE-100 44 Stockholm ISBN 978-91-7729-351-4 SWEDEN

Akademisk avhandling som med tillstånd av Kungl Tekniska högskolan framläg- ges till oﬀentlig granskning för avläggande av teknologie doktorsexamen i datalogi tisdagen den 9 maj 2017 klockan 13.00 i F3, Lindstedtsvägen 26, Kungl Tekniska högskolan, Valhallavägen 79, Stockholm.

Tryck: Universitetsservice US AB iii

Abstract

Cortical and subcortical microcircuits are continuously modiﬁed throughout life. Despite ongoing changes these networks stubbornly maintain their functions, which persist although destabilizing synaptic and nonsynaptic mechanisms should osten- sibly propel them towards runaway excitation or quiescence. What dynamical phenomena exist to act together to balance such learning with information processing? What types of activity patterns do they underpin, and how do these patterns relate to our perceptual experiences? What enables learning and memory operations to occur despite such massive and constant neural reorganization?

Progress towards answering many of these questions can be pursued through large- scale neuronal simulations. Inspiring some of the most seminal neuroscience experiments, theoretical models provide insights that demystify experimental measurements and even inform new experiments. In this thesis, a Hebbian learning rule for spiking neurons inspired by statistical inference is introduced. The spike-based version of the Bayesian Conﬁdence Propagation Neural Network (BCPNN) learning rule involves changes in both synaptic strengths and intrinsic neuronal currents. The model is motivated by molecular cascades whose functional outcomes are mapped onto biological mechanisms such as Hebbian and homeostatic plasticity, neuromodulation, and intrinsic excitability. Temporally interacting memory traces enable spike-timing dependence, a stable learning regime that remains competitive, postsynaptic activity regulation, spike-based reinforcement learning and intrinsic graded persistent ﬁring levels.

In terms of large-scale networks, the model can account for a diversity of functions, processes and timescales exhibited by the cerebral cortex and basal ganglia. Imbued with spike-based BCPNN, a modular cortical memory network can learn meta-stable sequential or nonsequential transitions. The trajectory of attractor states traversed by the network is governed by externally presented stimuli and internal network parameters. A spike-based BCPNN cortical model can perform basic perceptual grouping operations, and a spike-based BCPNN basal ganglia model can learn to dis-inhibit reward-seeking actions in a multiple-choice learning task with transiently changing reward schedules. Numerical simulations are carried out on both a Cray XC-30 supercomputer and on SpiNNaker: the digital, neuromorphic architecture designed for simulating large-scale spiking neuronal networks using general-purpose ARM processors. Spiking cortical networks are scaled up to 2.0 × 104 neurons and 5.1 × 107 plastic synapses. At the time of publishing, this represents the largest plastic neural network ever simulated on neuromorphic hardware.

The thesis seeks to demonstrate how multiple interacting plasticity mechanisms can coordinate reinforcement, auto- and hetero-associative learning within large-scale, spiking, plastic neuronal networks. It sheds light upon many cognitive and motor functions, including some high-level phenomena associated with Parkinson’s disease. The thesis will argue that spiking neural networks can represent information in the form of probability distributions, and that this biophysical realization of Bayesian computation can help reconcile disparate experimental observations. iv

Sammanfattning

Cortikala och sub-cortikala nätverk I hjärnan som förändras under hela ens liv. De fungerar trots destabiliserande synaptiska och icke-synaptiska mekanismer som borde driva nätverken till okontrollerad aktivitetsökning eller tystnande existerar. Vilka dynamiska fenomen ﬁnns som möjliggör en balansering av nätverken? Hur representerad fenomenen i form av aktivitetsmönster och hur relaterar de till våra sinnesupplevelser? Var möjliggör att inlärning och minnesoperationer kan fungera trots den neurala omorganisation som hela tiden pågår i hjärnan?

Ett steg mot att besvara dessa frågor kan göras genom storskalig simulering av neurala nätverk. Matematisk modellering har gett betydande bidrag till några av de mest innovativa experiment inom neurovetenskapen. I denna avhandling intro- duceras en Hebbiansk inlärningsregel för neuroner med actionspotentialer framtagen med inspiration från statistisk inferens. Den spikbaserade versionen av Bayesian Conﬁdence Propagation Neural Network (BCPNN) inlärningsregel involverar för- ändringar i både av synapsstyrka samt strömstyrka inne i nervcellerna. Modellen bygger på idén att utfallet av molekylära reaktionen i synapserna och nervcellerna beskrivas genom biologiska mekansimer så som hebbiansk inlärning, homeostatisk plasticitet och nervcellers aktiveringsbarhet. Tillfällig integration mellan minnesre- laterade aktivitet möjliggör tidsberoenden mellan aktionspotentialer och en stabil inlärningsregim som förblir funktionell med avseende på postsynaptisk aktivitets- reglering, actionpotentialsberoendebaserad inlärning samt förändring av nervcellers aktivitetsgrad.

Med avseende på storskaliga nätverk kan modellen förklara funktionell mångfald vil- ket inkluderar processer och tidsskalor som kan återﬁnnas i cerebrala cortex (hjärn- barken) och basala ganglierna. Ett modulärt cortikalt minnesnätverk som använder sig har actionspotentialsbaserad BCPNN-inlärning kan lära sig metastabila sekventiella och icke-sekventiella övergångar. Sekvensen av aktivitetsmönster som nätverket genomgår styrs både av externstimuli och av modellparametrar. En cortical BCPNN modell klarar av att genomföra enkla sinnliga grupperingsoperationer samt en modell av basal ganglierna med BCPNN kan lära sig att selektera handlingar i en ﬂervalsuppgift med belöningsbaserad inlärning som har ett skiftande belöningssche- man. Simuleringarna I denna avhandling gjordes på både en Cray XC-30 superdator samt på SpiNNaker - en digital arkitektur inspirerad av hjärnan byggd på ARM pro- cessorer. Nätverksstorleken har skalats upp till 2.0 × 104 nervceller samt 5.1 × 107 plastiska synapser. I skrivandets stund representerar detta det största nätverk med plastiska synapser som någonsin har simulerats på en SpiNNaker arkitektur.

Denna avhandling ämnar visa hur ﬂera olika plasticitetsmekansimer kan intera- gera och åstadkomma belöningsbaserad, auto- och hetero associativ inlärning och inkluderar ﬂera högnivåfenomen som associerats med Parkinsons sjukdom. Denna avhandling kommer visa hur neurala nätverk med aktionspotentialer kan represen- tera information som sannolikhetsdistributioner och hur en biofysikalisk realisering av Bayesianska beräkningar kan sammanföra spretiga experimentella observationer. v

Acknowledgements

My deepest gratitude is to my first supervisor, Professor Anders Lansner. It has been an amazing experience working with a scholar who is so dedicated to ensur- ing my growth both as a student and a person. Your tireless efforts to meet with me to offer advice have helped me to excel in a comfortable atmosphere that was conducive to the free exchange of ideas. Even outside of my doctoral endeavor, you’ve provided the motivation to help me to achieve recognition both at KTH and beyond. Your contributions to this work and to shaping my success cannot merely be expressed by the words on this page. You’ve been a huge part of my life for the past five years, and most of all, I am thankful for your friendship.

I would like to thank my second supervisor, Dr. Matthias Hennig, for the count- less hours you’ve spent helping me explore new possibilities and academic avenues. Your different perspectives and healthy skepticism and have inspired me to achieve a deep and broad set of knowledge that has not only benefitted my PhD thesis work, but also my scientific approach as a whole. I’ll take these invaluable lessons with me as I go forward in the next steps of my career.

To Professor Örjan Ekeberg, for nudging me to pursue the opportunity to do my PhD abroad in the ﬁrst place, and dragging me away from that small liberal arts school in upstate New York. The world is much bigger and better than I could have ever imagined back then, so thanks for helping break me out of my bubble.

To my two oﬃce roommates - Dr. Pierre Berthet and Dr. Bernhard Kaplan - thank you for all the moments of wit and levity, putting up with my groans and keyboard smashes, and for helping drown the PhD stress away with after-work pub sessions.

I would also like to thank my fellow lab members and collaborators over all the years - Dr. Pawel Herman, Dr. Mikael Lindahl, Dr. Henrik Lindén, Pradeep Krishnamurthy, Dr. Jamie Knight, Prof. Steve Furber, Cristina Meli, Dr. Mar- cus Petersson, Prof. Jeanette Hellgren-Kotaleski, Dr. Mikael Lundqvist, Prof. Erik Fransén, Florian Fiebig, Peter Åhlström, Ekaterina Brocke, Prof. Tony Lin- deberg, David Silverstein, Dr. Omar Arenas, Jovana Belic, Ramón Mayorquin, Jan Pieczkowski, Anu Nair, Miha Pelko, Dr. Paolo Puggioni, Jyotika Bahuguna, Nathalie Dupuy, Renato Duarte, Dr. Yann Sweeney, Dinesh Natesan, Daniel Tr- pevski, Ylva Jansson, Ronald Biro, Harriett Johansson, and many, many others.

I would like to thank my family, to whom this thesis is dedicated. Mom, Dad, and Grace - your support, understanding, patience and love has enabled me to achieve everything I could have ever hoped.

Last but certainly not least, I’d like to thank my devoted ﬁancée, Annie Varnum, for vi the endless love and patience you’ve shown throughout this surreal journey. Despite all the obstacles that in hindsight seem impossibly daunting: the long-distance, the timezone switches, and moving back and forth between several diﬀerent foreign countries - all while navigating a career of your own, and propped up only by our student salaries - I knew I could always rely on you and your support. This was only the beginning of many other adventures we’ll have and challenges we’ll overcome together. I love you. vii

Declaration

Spike-Based Bayesian-Hebbian Learning in Cortical and Subcortical Microcircuits PhD in Computer Science (School of Computer Science and Communication, KTH Royal Institute of Technology) Doctor of Philosophy (School of Informatics, University of Edinburgh) supervisors: Anders Lansner (KTH Royal Institute of Technology) Matthias Hennig (University of Edinburgh)

I declare that this thesis was composed by myself, that the work contained herein is my own except where explicitly stated otherwise in the text, that this work has not been submitted for any other degree or professional qualiﬁcation except as speci- ﬁed, and this work isn’t necessarily intended as a novel contribution since content may also appear within the previously peer reviewed manuscripts attached at the end.

Philip J. Tully, Stockholm, May 9, 2017 Contents

Contents viii

List of Figures x

I Dissertation 1

1 Introduction 3 1.1 On the (f)utility of understanding the brain ...... 3 1.2 Thesis overview ...... 5 1.3 List of publications ...... 6 1.4 Contributions to publications ...... 7

2 Background 9 2.1 Neurons ...... 9 2.1.1 Spike-Frequency Adaptation ...... 10 2.1.2 Intrinsic Plasticity and Excitability ...... 10 2.2 Synapses ...... 10 2.2.1 Long-Term Potentiation and Long-Term Depression . . . . . 11 2.2.2 Hebbian Learning ...... 12 2.2.3 Spike Timing-Dependent Plasticity ...... 13 2.2.4 Short-Term Plasticity ...... 13 2.2.5 Homeostatic Synaptic Plasticity ...... 14 2.2.6 Synaptic Tagging ...... 15 2.2.7 Neuromodulated Plasticity ...... 15 2.2.8 Interacting Synaptic and Nonsynaptic plasticity ...... 15 2.3 Anatomical architecture of the mammalian brain ...... 16 2.3.1 Cerebral cortex ...... 16 2.3.2 Basal ganglia and a Systems View of Cortical and Subcortical Memory ...... 17 2.4 Modeling Learning, Memory and Neural Dynamics ...... 19 2.4.1 Model Neuron and Synapse Types ...... 19

viii 2.4.2 Autoassociative Memories and Attractors in Dynamical Sys- tems ...... 21 2.4.3 Gestalt phenomena and perceptual grouping ...... 21 2.4.4 Resolving the Problems of Abstract Attractor Memory Net- works ...... 22 2.4.5 Biophysical cortical memory networks ...... 23 2.4.6 Temporal sequences ...... 24 2.5 Role of computational neuroscience ...... 25 2.5.1 Top-down versus bottom-up modeling approaches ...... 26 2.5.2 Large-scale modeling with high-performance computing . . . 27 2.5.3 Neuromorphic Hardware ...... 28 2.6 Bayesian brain hypothesis ...... 29 2.6.1 Perceptual inference: evidence from psychophysics ...... 29 2.6.2 Representing probabilities within the neural substrate . . . . 29

3 Materials and Methods 31 3.1 Bayesian Conﬁdence Propagation Neural Network ...... 31 3.2 The spike-based BCPNN learning rule and Cortical microcircuit model 33 3.3 Simulation tools and strategies ...... 38 3.3.1 NEST ...... 38 3.3.2 PyNN ...... 38 3.3.3 SpiNNaker ...... 38

4 Results and Discussion 41 4.1 Paper 1: Emergent dynamics and neurobiological correlates . . . . . 41 4.2 Paper 2: Bayesian-Hebbian learning in an attractor network . . . . . 49 4.3 Paper 3: Scaling up Attractor Learning on Neuromorphic Hardware 61 4.4 Paper 4: Reward-Dependent Bayesian-Hebbian learning in a Bio- physical Basal Ganglia model ...... 62

5 Concluding Remarks and Future Directions 65 5.0.1 Summary ...... 65 5.0.2 Temporal Gestalten and other Perceptual Illusions of Time Acquired from Probabilistic Plasticity ...... 65 5.0.3 Natural Language Processing ...... 67 5.0.4 Basal Ganglia Sequences ...... 68 5.0.5 Outlook ...... 69

Bibliography 71

ix x List of Figures

II Publications 89

List of Figures

1.1 Spatial scales of the nervous system ...... 4 1.2 Temporal scales of the nervous system ...... 5

2.1 AMPA and NMDA receptor kinetics ...... 11 2.2 Visualizing synaptic plasticity window outcomes ...... 14 2.3 Anatomical schematic of a cortical microcircuit ...... 17 2.4 Anatomical schematic of the basal ganglia and its relationship to the cortex ...... 18 2.5 Comparison of the realistic and computational properties of spiking neuron models ...... 20 2.6 Gestalt phenomena illustrated by the Necker cube and Rubin vase . . . 22

3.1 Reconciling neuronal and probabilistic spaces using the spike-based BCPNN architecture for a postsynaptic minicolumn with activity yj ...... 32 3.2 More detailed anatomical schematic of a cortical microcircuit ...... 34 3.3 Schematic ﬂow of BCPNN update equations reformulated as spike-based plasticity ...... 36 3.4 Mapping of a spiking neural network to SpiNNaker ...... 40

4.1 The bias term shows logarithmically increasing ﬁring rate values of the neuron for which it is computed...... 42 4.2 Persistent graded activity of entorhinal cortical neurons due to βj . . . 44 4.3 STDP function curves are shaped by the Z trace time constants . . . . 45 4.4 The BCPNN learning rule exhibits a stable equilibrium weight distribution 46 4.5 Delayed reward learning using E traces ...... 47 4.6 Learning spontaneously wandering attractor states ...... 49 4.7 Applying temporally interleaved stimuli to a spiking network ...... 50 4.8 Evolving dynamical phenomena during stimulus learning ...... 51 4.9 Replaying spontaneously wandering attractors after learning ...... 52 4.10 Applying temporally consecutive stimuli to a spiking network ...... 53 4.11 Replaying sequential attractors after learning ...... 54 4.12 The temporal structure of neural recall dynamics reﬂects the temporal interval used during training ...... 55 List of Figures xi

4.13 Upkeep of stable sequential state switching during recall is permitted by the interplay of excitation and inhibition in a single neuron ...... 56 4.14 Dependence of temporal sequence speed on the duration of conditioning stimuli ...... 58 4.15 Characterizing speed changes based on intrinsic network parameters . . 59 4.16 Cue-triggered overlapping sequence completion and disambiguation . . . 60 4.17 SpiNNaker power usage ...... 61 4.18 Learning in the D1, D2, and RP pathways of a Spike-Based BCPNN Basal Ganglia model ...... 63

5.1 Spike-based BCPNN synaptic weight matrix for a Dr. Suess poem . . . 69

Part I

Dissertation

Chapter 1

Introduction

1.1 On the (f)utility of understanding the brain

Disclaimer: at the conclusion of this thesis, and by some measures also the last five years of my life, the mysteries of the brain remain, unfortunately, largely unsolved. When I first stepped into my supervisor’s office all those years ago, I remember remarking how satisfyingly recursive it would be to study the brain: the more I would learn about it, I reasoned, the easier it would become to learn even more because I’d begin to internalize and therefore better understand the phenomena being carried out between my ears. In fact, exactly the opposite happened. Each newly learned tidbit brought with it more and more questions and left me even more puzzled and fascinated than when I began.

Why bother studying something so complex that there’s so little chance of understanding it all within a single lifetime? Part of it stems from curiosity, and another part the desire to contribute to something bigger and more important than myself. Advances in neuroscience oﬀer the promise of a future encompassing longer, more meaningful lives along with more intelligently planned and productive societies. From an altruistic point of view, there are a slew of diseases associated with the brain including Parkinson’s, Alzheimer’s and epilepsy, which enact high emotional tolls upon patients and their families. Even worse, rising life expectancies resulting from improvements in clinical care do not beneﬁt those who go on to develop more mentally debilitating disorders such as dementia. When it comes to extending human life, quantity has not given rise to quality.

While humanity may one day help cure the brain of its ailments through research, by the same token the brain may one day help cure humanity of its own ailments. From a technological point of view, artiﬁcial intelligence and brain-inspired au- tomation have the potential to lift citizens out of poverty and create smarter, more eﬃcient societies. For example, at the time this thesis is being written, the world

3 4 CHAPTER 1. INTRODUCTION is on the precipice of self-driving cars, which could not only help optimize fuel efficiency but also help drive down rates of accidental injury or death. The algorithms underlying such applications generalize to a wide ranging set of problems and are largely based on brain-inspired computing paradigms that have come to light due to the last 30 years of basic research in the fields of machine learning and artificial intelligence. Rather than making sense of patterns embedded within action potentials like the brain, new technologies have leveraged brain-inspired architectures and computational techniques to make sense of patterns contained within everyday experiences like natural language, images and videos.

And we’ve still only scratched the surface of computerizing our neural circuitry. It’s important to note that the most frequently applied state of the art brain- inspired algorithms, like deep neural networks [1], incorporate only a sliver of the brain’s fundamental structures and functions. The diagram in Figure 1.1 helps put this into perspective.

body x=0 m] whole brain x -2

brain regions -3

microcircuits -5

cells spatial scale [10 -6

synapses -7

chromosomes -8

proteins -9

Figure 1.1: Spatial scales of the nervous system. Neural processes control structures that range in size from nanometers to meters. Diagram inspired by [2].

The orders of magnitude diﬀerences in spatial scale between organization levels is astonishing in its own right, let alone that these dynamical hierarchies operate so 1.2. THESIS OVERVIEW 5 robustly day in and day out. Even more impressive is the notion that elements within each of these organizational levels are themselves constantly changing (Fig- ure 1.2).

development x=7

behavior s]

x 5

learning 3

plasticity 2

metabolism 0

spike scale [10 temporal -3

vesicle release -7

molecular -12 dynamics

Figure 1.2: Temporal scales of the nervous system. Brain processes carry out functions that range in runtime from picoseconds to years. Diagram inspired by [2] as in Figure 1.1.

Many of the structures and functions in Figure 1.1 and Figure 1.2 are conserved across species to varying degrees, and this veritable generalizability makes a com- pelling case for further investigation towards incorporating these evolutionary adaptations into silicon. The overarching objective of this thesis will be to marry the structure (Figure 1.1) and function (Figure 1.2) of speciﬁc brain areas, with the twofold goals of having computer simulations improve our understanding of the brain, and having our understanding of the brain improve computer simulations.

1.2 Thesis overview

To structure the proceeding thesis, I’ll ﬁrst motivate the questions and hypotheses it addresses and describe my contributions to the individual publications included 6 CHAPTER 1. INTRODUCTION at the end. Then, I’ll provide general background material to motivate some of the more advanced topics covered further afar. I’ll begin with an overview of the biological foundations of the cerebral cortex and basal ganglia, followed by modeling goals including how to go about understanding these complex neural structures and how they are interdependent. I’ll cover some of the theoretical ideas these models are based upon, and relate their dynamics to those observed in previous experimental ﬁndings.

From there, I’ll begin to talk more speciﬁcally about the unique contributions put forth by this work. I’ll introduce the theoretical underpinnings all of the results are built upon, before describing the approaches taken in simulating these large- scale models on distributed cluster and neuromorphic machines. I’ll pivot from there into a reﬂection of the neurobiological correlates of the developed theoretical model, starting from the level of individual synapses and neurons, and subsequently working my way into more detailed biological circuits. I’ll conclude with highlighted other studies this work has motivated, and more open-ended ideas that could be pursued using the developed concepts.

1.3 List of publications

With a system as complex and dynamic as the brain, the best one can hope to do is both stand on the shoulders of giants and try to provoke the next generation into coming along and taking the scientiﬁc baton. That said, I’ve been lucky to be able to work with inspirational collaborators who’ve helped me leave behind some important contributions:

Poster Presentation 1: Tully, Philip. J., Henrik Lindén, Matthias H. Hen- nig & Anders Lansner, "Probabilistic computation underlying sequence learning in a spiking attractor memory network." BMC Neuroscience 14(Suppl 1): P236 (2013).

Manuscript 1: Tully, Philip. J., Matthias H. Hennig & Anders Lansner, "Synap- tic and nonsynaptic plasticity approximating probabilistic inference." Frontiers in Synaptic Neuroscience 6.8 (2014).

Manuscript 2: Knight, James C., Philip J. Tully, Bernhard A. Kaplan, An- ders Lansner & Steve Furber, "Large-scale simulations of plastic neural networks on neuromorphic hardware." Frontiers in Neuroanatomy 10.37 (2016).

Manuscript 3: Tully, Philip. J., Henrik Lindén, Matthias H. Hennig & Anders Lansner, "Spike-Based Bayesian-Hebbian Learning of Temporal Sequences." PLoS Computational Biology 12.5 (2016).

Manuscript 4: Berthet, Pierre, Mikael Lindahl, Philip J. Tully, Jeanette Hellgren- 1.4. CONTRIBUTIONS TO PUBLICATIONS 7

Kotaleski & Anders Lansner, "Functional Relevance of Diﬀerent Basal Ganglia Pathways Investigated in a Spiking Model with Reward Dependent Plasticity." Frontiers in Neural Circuits 10.53 (2016).

1.4 Contributions to publications

Poster Presentation 1: Co-researched and co-created poster content, then co- presented poster at conference. Forged initial ideas for basis of study, designed model, performed simulations and analyzed results contained in poster.

Manuscript 1: Designed experiments, performed simulations, analyzed the data and wrote the manuscript.

Manuscript 2: Designed experiments, analyzed the data and helped write and review the manuscript.

Manuscript 3: Conceived and designed the experiments, performed the experiments, contributed reagents/materials/analysis tools and wrote the manuscript.

Manuscript 4: Developed and delivered code for the plasticity model, conceived and designed the experiments and helped write and review the manuscript.

Chapter 2

Background

2.1 Neurons

Before trying to tackle some of the grander challenges laid out in Section 1.2, a brief groundwork will be provided from which many of the concepts introduced later on in this work will stem. At a low level, neurons represent the fundamental functional building block of the brain (but see [3]). Encompassing vastly diﬀerent structural and electrophysiological properties, neurons as a whole are remarkably diverse, and its safe to say that no two neurons are exactly the same. Perhaps this is best illustrated by the artwork of Santiago Ramón y Cajal, whose detailed but immaculately detailed illustrations of neurons continue to help students put the intricacy of neural circuits into perspective to this day [4].

However, there are elements shared in common between many diﬀerent neuron types, including their ability to generate electrical pulses known as action potentials (i.e. spikes) and their ability to communicate across proportionally larger spatial distances using synapses. Mechanically, each spike produced by a presynaptic neuron is sent to other postsynaptic neurons through projections called axons. Axons terminate at the synapse where the output signal is transmitted and received as input by dendrites of the postsynaptic neuron.

These morphological characteristics of each neuron help determine its unique physiological characteristics. On the cell membrane of each neuron, there exist ion channels that open and close, allowing ions to travel into or out of the cell. The opening and closing dynamics of ion channels like sodium (Na+), potassium (K+), calcium (Ca2+), or chloride (Cl-), and in turn the flow of ions across the cell, are determined in part by voltage changes and in part by internal and external neural signaling. These ion channels help maintain a difference in electrical potential between the inside and outside of the neuron, where the typical membrane voltage Vm = -70 mV at rest. When current flows into the neuron, Vm increases and the

9 10 CHAPTER 2. BACKGROUND

cell becomes depolarized, and when current ﬂows out of the neuron, Vm decreases and the cell becomes hyperpolarized.

When a neuron is depolarized to the certain point, or voltage threshold (Vt), the neuron ﬁres an action potential or spike, which lasts around ∆T = 1 ms and leads to a refractory period of tref = 2 ms during which another spike cannot be generated again. Spikes are 100 mV deﬂections that can be propagated to other neurons that receive synaptic connections from the spiking neuron. As ion channels can be modulated, neurons have several well-described dynamical phenomena that make them more amenable to either depolarization or hyperpolarization depending on certain conditions.

2.1.1 Spike-Frequency Adaptation The first of these is spike-frequency adaptation, which means that a neuron’s spiking behavior depends on the state of its recently emitted spikes, or lack thereof. Spike- frequency adaptation reduces the firing frequency of spikes during a constantly applied stimulus. Biophysically, the KCa ion channels and voltage-gated calcium channels activate during multiple consecutive depolarizing events allowing the accumulation of intracellular calcium. When the intracellular calcium concentration is too high, these currents cause an after-hyperpolarizing potential and spiking is temporarily terminated while calcium decays with a timescale of around 150 ms [5]. Thus each subsequently generated spike or burst of spikes is slightly more delayed in time. There are several other known mechanisms that could contribute to spike-frequency adaptation, and each of these has a wide range of time courses associated with their ultimate functional effects [6].

2.1.2 Intrinsic Plasticity and Excitability A second neuron-speciﬁc dynamical phenomenon is long-term potentiation of intrinsic excitability (LTP-IE) [7], which occurs when stimulated neurons have an increased likelihood of generating spikes. Seemingly opposing spike-frequency adaptation and involving separate ion channels, Ca2+-sensitive, activity-dependent K+ currents can decrease the spike threshold by enhancing neurotransmitter release and broadening the spike. This mechanism has been conjectured to serve as a memory substrate via locally stored information in the form of a neuron’s activity history.

2.2 Synapses

Synapses can be either excitatory or inhibitory, depending on the makeup of their neurotransmitters and whether they tend to depolarize or hyperpolarize the postsynaptic neuron. By convention, a neuron is an excitatory neuron if it projects excitatory synapses, and inhibitory if it projects inhibitory synapses. The most 2.2. SYNAPSES 11 abundant excitatory neurotransmitter is glutamate, which activates α-amino-3- hydroxy-5-methyl-4-isoxazolepropionic acid (AMPA) and N -methyl-D-aspartate receptor (NMDA) receptors on the postsynaptic cell. These two receptors have sig- nificant structural differences, but functionally, they are differentiated according to the time course of activation (2.1). The inhibitory neurotransmitter is gamma- Aminobutyric (GABA), which activates GABA receptors on the postsynaptic cell. These receptors have a similar time course as AMPA receptors.

.20

.15 [nS]

.10 syn ij g .05

.0 -100 0 100 200 300 400 500 600 700 800 time [ms] Figure 2.1: AMPA and NMDA receptor kinetics. Model synapses connecting neuron i and neuron j have diﬀerent time courses of activation when postsynaptic syn conductances gij is elicited through AMPA (red) or NMDA (blue) receptors. The inset provides a zoomed view of the area contained within the dotted gray lines, ﬁgure adopted from Figure 2B of [8].

The basic mechanism of synaptic transmission involves a presynaptic spike that depolarizes the synaptic terminal, which leads to an inﬂux of calcium through presynaptic calcium channels and subsequent release of neurotransmitter vesicles into the synaptic cleft. When the neurotransmitter transiently binds to postsynaptic channels, it opens them and allows ionic current to ﬂow across the membrane. Models of synaptic transmission typically assume an instantaneous rise of synaptic conductance due to an arriving spike. But sometimes for simplicity, models assume synapses to be sources of current and not a conductance (see Section 2.4.1).

2.2.1 Long-Term Potentiation and Long-Term Depression Like neurons, synapses can also be modulated as a result of dynamical phenomena. One of the most well-researched mechanisms is long-term potentiation (LTP), which is a long-lasting strengthening of the synapse due to pre- and/or postsynaptic neural activity [9]. Correspondingly, long-term depression (LTD) is a long-lasting 12 CHAPTER 2. BACKGROUND weakening of the synapse. These mechanisms are believed to subserve learning and memory, since they biophysically sculpt and are sculpted by previous experience in the form of neural activity. Both AMPA and NMDA synapses can be potentiated, with delayed potentiation of NMDA currents requiring previous AMPA potentiation [10]. Though many complex molecular chains of events are known to be involved in LTP, it is generally caused by repeated synaptic stimulation, which leads to glutamate binding and opening of NMDA receptor voltage gated channels followed by calcium influx and a corresponding increase of intracellular Ca2+ levels . The change in synaptic efficacy typically lasts on the order of minutes to hours, and structural changes like spine growth or decay can mean that these changes can persist for years afterwards. The timescales are similar for LTD, but this opposing effect is caused by decreases in postsynaptic receptor density and presynaptic neurotransmitter release.

2.2.2 Hebbian Learning Donald Hebb’s conceptual theory of cell assemblies is one of the oldest and most in- ﬂuential ideas in neuroscience to this day [11]. He hypothesized about the formation and storage of memories originating from synapses connecting co-active neurons, phrased elegantly in his own words:

When an axon of cell A is near enough to excite a cell B and repeatedly or persistently takes part in firing it, some growth process or metabolic change takes place in one or both cells such that A’s efficiency, as one of the cells firing B, is increased.

With this simple formulation, Hebb helped set oﬀ an arms race among theoreticians and experimentalists alike, who sought to verify and build upon his elegant ideas. In fact, Hebb’s theories preceded even the earliest experiments discovering LTP and LTD (see Section 2.2.1). According to this theory, memories are learned and stored as increased synaptic strengths, and the set of neurons or cell assemblies that are activated form the basis of an internal mental representation for externally presented stimuli. Hebbian learning theory emphasized the role of cell assemblies as fundamental functional cortical units, as opposed to e.g. individual neurons. In practical terms, a basketball would invoke one cell assembly, while a football would invoke a diﬀerent one. In mathematical terms, a synapse would change its strength as ∆wi = ηxiy (2.1)

This equation states that the change in synaptic weight wi from presynaptic neuron i is equal to a learning rate η multiplied by the input from neuron i, xi, multiplied by y the postsynaptic neural response. Though it is a very simple formulation of 2.2. SYNAPSES 13

Hebbian learning, Equation 2.1 clearly demonstrates an inherent instability present in these type of learning rules in general. In a network with Hebbian synapses, the strongest signal present in that network will be inﬁnitely ampliﬁed by runaway growth of recurrent excitatory synaptic strengths over time, and the remaining signals will eventually be silenced.

2.2.3 Spike Timing-Dependent Plasticity A more nuanced view of Hebbian learning was later developed, when a ground- breaking set of experiments determined that the precise timing of presynaptic and postsynaptic spikes can have different effects on the magnitude and direction of change in synaptic strength [12]. Suggesting a temporal coding scheme on the order of milliseconds, this functional outcome was found after stimulating pre- and postsynaptic neurons repeatedly at slightly different intervals, unveiling that a presynaptic spike repeatedly occurring prior to a postsynaptic spike caused LTP whereas a postsynaptic spike occurring prior to a presynaptic spike caused LTD (Figure 2.2).

The finding of temporal contrast between LTP and LTD has not been without controversy though. Firstly, the shape of LTP and LTD is not ubiquitous, in fact it can have different outcomes when carried out in other animals, brain regions, synapse types or using slightly different experimental protocols [13]. Secondly, the fidelity of STDP has been questioned on the grounds that the artificial postsynaptic spike typically elicited in experiments is not realistic enough to draw conclusions about its importance in vivo [14]. Finally, asymmetrical STDP as a generic biological learning rule suffers from some of the same theoretical pitfalls as LTP in terms of being susceptible to runaway excitation [15].

2.2.4 Short-Term Plasticity Another form of synaptic plasticity that is differentiated from classical LTP and LTD is short-term plasticity. Also highly abundant and activity-dependent, facilitation and depression can modulate of synaptic efficacy on more rapid timescales. Depression is the progressive reduction of postsynaptic responses during repetitive presynaptic activity, whereas facilitation is an increase in postsynaptic response during repetitive presynaptic activity [16]. Depression, an activity dampening mechanism, is caused by depletion of neurotransmitters consumed during a presynaptic spike. Facilitation, an activity quenching mechanism, is caused by an influx of calcium into the axon terminal a presynaptic spike, which increases the release probability of neurotransmitters [17]. The timescales of these changes can vary just like LTP and LTD, they fall in a decidedly more rapid timescale classification of hundreds of milliseconds to seconds. Mechanisms like short-term synaptic facilitation are non-associative (non-Hebbian), and don’t account for the encoding of novel associations observed in working memory tasks. Another fast-expressing form of 14 CHAPTER 2. BACKGROUND

1.0

0.5 j i

w 0.0 ∆

0.5

1.0 150 100 50 0 50 100 150 ∆t

Figure 2.2: Visualizing synaptic plasticity window outcomes. Absolute change in peak synaptic amplitude after 100 pre- and postsynaptic events at 1 Hz, where spike latency > 0 denotes a pre before post pairing. In this diagram, STDP is demonstrated by the solid black line, which notably has a pronounced asymmetry around ∆t = 0. The dotted line illustrates what the left side of the plasticity curve would look like in the case where purely associative LTP was at work. In this case, the change in synaptic strength occurs irrespectively of the timing relationship between pre- and postsynaptic spike.

associative short-term potentiation based on Hebbian learning has been shown to address these shortcomings [18].

2.2.5 Homeostatic Synaptic Plasticity At the other end of the temporal spectrum lies homeostatic synaptic plasticity, which seems to contradict the common notions of LTP and LTD since they act in opposition to these modulatory forces: chronic increases in the global levels of cellular activity weaken synaptic weights, and decreases strengthen them. A particular mechanism of homeostatic plasticity is known as synaptic scaling, and the key differentiator is that LTD and LTP take place over the course of seconds to hours, while synaptic scaling takes place over the course of hours to days [19]. The mechanism also involves glutamate and AMPA receptors. It regulates that quantity of AMPA receptors on the postsynaptic neuron, allowing global negative feedback control over the neurons spike firing rate through its own efferent synapses. Synaptic scaling can help solve the problem of runaway excitation by providing a 2.2. SYNAPSES 15 stabilizing mechanism to normalize the otherwise overwhelming forces of LTP and LTD [15].

2.2.6 Synaptic Tagging Protein synthesis occurs within a neuron whereas LTP is dependent on external input. A long-standing puzzle of memory formation due to LTP involved how synaptic specificity could arise in the absence of elaborate intracellular protein trafficking. Independent of protein synthesis, creation of a decaying ’synaptic tag’ for each pre-post activated synapse for delivery of a specific marker that can be targeted for future plasticity-associated protein trafficking was hypothesized to provide an intermediary step in the transition from short-lasting LTP to long-lasting LTP [20].The time course of activation for this synaptic tag is on the order of hours, and its presence indicates that LTP depends in part upon prior activity of the neuron in addition to coincident pre- and postsynaptic activity (see Subsection 2.2.1).

2.2.7 Neuromodulated Plasticity Rather than focusing on single neurons or synapses, yet another form of plasticity called neuromodulation provides more globally applied change to speciﬁc physical areas wherein lie many neurons and synapses. Typically involving G-protein coupled receptors that act on slower timescales compared to the ion channels involved in LTP and LTD, neurotransmittors act through volume transmission meaning that they cause eﬀects in large and more dispersed brain areas.

2.2.8 Interacting Synaptic and Nonsynaptic plasticity With all these cooks in the kitchen, its no wonder some of the issues outlined in Section 1.1 remain unsolved. Because these moving parts occur all at different timescales, it can be difficult to try to understand the ultimate affect they have in terms of circuit functioning. Indeed, there is some overlap when it comes to the underlying molecular processes that govern these observed dynamics. For example, LTP/LTD and LTP-IE can be co-expressed due to their shared intracellular postsynaptic Ca2+ signaling cascades [21]. Indeed, LTP-IE is thought to share many common induction and expression pathways with LTP/LTD [22], and experimental protocols used to study synaptic plasticity have often been shown to incidentally give rise to LTP-IE [9]. As in LTP/LTD, LTP-IE is rapidly induced and long-lasting [7].

Why so many cooks? An ad-hoc approach might involve simply storing everything, but the nervous system is constrained by capacity and speed. Linear memory growth would lead to much higher post-storage access times. Storing things at diﬀerent but overlapping timescales is useful from a computational point of view in order to avoid the deletion of important information through forgetting [23, 24]. 16 CHAPTER 2. BACKGROUND

The fact too that some of these mechanisms are governed by similar molecular phenomena indicates that these processes have built-in redundancy that renders them robust to fast or unexpected changes in the external environment.

2.3 Anatomical architecture of the mammalian brain

Occupying a volume of 1400 cm3, the human brain consumes about 20 W power while computationally outperforming the racks upon racks of processors comprising modern supercomputers [25]. It is an interconnected web of structures with different and overlapping functional specializations. Once environmental input is received by sensory structures like the retina, epithelium and taste buds, a complex set of events is initiated whereby signals are rapidly communicated through different pathways to areas of the brain responsible for higher level processing. Structures like the hippocampus, which has been associated with the consolidation of long-term memories, the amygdala, which has been associated with emotional motivators like fear and food, and the cerebellum, which has been associated with procedural memory learning of fine motor skills, all partially form the neuroanatomy of memory. In addition to these anatomical structures, and of particular importance to the work presented throughout this thesis, are the cortical and subcortical nuclei.

2.3.1 Cerebral cortex As one of the most recent evolutionary additions and most defining physical land- marks of the mammalian brain, the cerebral cortex is thought to confer advanced computational functions including memory, language, abstraction, planning and perception. Early experiments by Hubel and Wiesel provided evidence of a functional neocortical organization [26]. They discovered that neurons in cat visual cortex were most strongly activated by visual stimuli consisting of bars presented at specific orientations, leading them to postulate that functional columns, or local groups of highly connected and coactive neurons with similar receptive fields [27, 28], were the underlying unit responsible for their observed phenomena. Mini- columns were hierarchically grouped together into larger entities called hypercolumns, which discretely and completely accounted for the set of possible ori- entation values that the stimuli could represent. Stronger anatomical evidence subsequently followed these observations [29].

This anatomical organization forms the basis behind many of the concepts elabo- rated upon throughout the remainder of this work (see Figure 2.3). Biologically, stimulus response specificity to incoming stimuli like bars at different orientations could be mediated by basket cell inhibition between hypercolumns, a generic motif commonly referred to as winner-take-all (WTA) that is thought to be fundamental to cortical network organization [30]. The functional ramifications of this stereotypical organization, while not without controversy, are further explored in future sections. 2.3. ANATOMICAL ARCHITECTURE OF THE MAMMALIAN BRAIN 17

Figure 2.3: Anatomical schematic of a cortical microcircuit. WTA hypercolumns (shaded gray areas) consist of excitatory neurons (red circles) belonging to minicolumns (red ovals) that innervate (red arrows) local populations (blue ovals) of basket cells (blue circles), which provide inhibitory feedback (blue arrows). Semi- circular arrowheads indicate the postsynaptic target. Figure adopted from Figure 1A of [8].

2.3.2 Basal ganglia and a Systems View of Cortical and Subcortical Memory The Basal ganglia is a more ancient brain structure than the cortex, it is ubiquitous and similarly organized among vertebrate species and consists of well-deﬁned substructures believed to have been evolutionarily conserved for over 560 million years [31]. It is hypothesized to be involved in action selection, helping to determine the decision of which of several possible behaviors to execute at any given moment, and to assume the role of a centralized selection device that resolves conﬂicts over access to limited motor and cognitive resources [32].

The Basal ganglia receives connections from different areas of the cortex, amygdala, thalamus, and dopaminergic nuclei, interacting with the cortex and thalamus through several loops that travel through sensorimotor, associative and limbic brain regions (Figure 2.4) [33, 34]. Functionally, the basal ganglia has a dual pathway architecture: two different pools of inhibitory medium spiny neurons (MSN) expressing either dopamine D1 receptors in the direct pathway or D2 receptors in the indirect pathway. It is comprised of many different complex substructures, including the striatum, pallidum, substantia nigra (SNr) and subthalamic nucleus. The striatum consists of MSNs that receive topographical connections from the cortex. Specific cortical cells tend to activate a corresponding subgroup of striatal cells. In Figure 2.4, a cortical state layer provides inputs to the striatum, which in turn represents the set of possible actions that can be taken. For example, the population AD1 corresponds to the population of MSNs in the D1 pathway coding for action A. 18 CHAPTER 2. BACKGROUND

Figure 2.4: Anatomical schematic of the basal ganglia and its relationship to the cortex. Figure adopted from Figure 1 of [35].

The striatum is further subdivided into striosomes and matrisomes. Matrisomes consist of D1 and D2 MSNs that project to a large structure within the pallidum called the internal lateral segment of the globus pallidus (GPi) and the SNr. These projections can either be inhibitory, corresponding to the direct pathway, or excitatory, corresponding to the indirect pathway. The GPi and SNr are the output nuclei of the Basal ganglia, projecting to the thalamus and brainstem motor command centra [36]. The high resting activity of these nuclei keeps the target motor structures under tonic inhibition. An eﬀerence copy informs the striatum about the ultimately selected action based on the activity in GPi/SNr.

This web of interconnected microcircuits provides both functional dis-inhibition in the D1 pathway, and functional dis-inhibition of excitatory neurons in thalamus and brain stem in the D2 pathway [37, 31]. An action is performed upon removal of inhibition from the output nuclei, i.e., when the motor command centra are disin- hibited. The inhibition from the BG output nuclei can be decreased via the direct 2.4. MODELING LEARNING, MEMORY AND NEURAL DYNAMICS 19 pathway, and enhanced via the indirect pathway. The MSNs associated with the direct pathway send connections mainly to SNr and GPi, while those associated with the indirect pathway project to the external part of the globus pallidus (GPe) [37, 31]. GPe in turn provides an additional inhibitory stage before projecting to SNr and GPi either directly or via the glutamatergic STN.

There is an additional dopaminergic reward prediction feedback pathway involving striosomes that project to dopaminergic neurons. The level of dopamine codes modulates plasticity in the system, which occurs at cortico-striatal synapses and at synapses from striosomes targeting dopaminergic neurons. Dopamine is involved in the control of the diﬀerent pathways [38], in the modulation of plasticity and learning [39], and in coding the reward prediction error (RPE) [40, 41, 42, 43].

2.4 Modeling Learning, Memory and Neural Dynamics

Much previous theoretical work has focused on developing a generic set of differential equations that captures the most common dynamical properties exhibited by different types of neurons and synapses. Proposed models tend to account for some degree of the underlying biophysical dynamics (see Section 2.1), but level of detail ranges all the way from multi-compartmental models represented by thousands of coupled differential equations, to single-compartmental point models more applicable towards studying network behavior arising from many interconnected units. As an example, voltage-dependent K+ and Na+ conductances that contribute to action potential generation can be accounted for by models with great level of detail, but not explicitly including these mechanisms greatly lowers the computational cost of simulating an action potential, speeding up the simulation time to the benefit of the modeler.

2.4.1 Model Neuron and Synapse Types One of the most basic types of model neurons is the integrate-and-ﬁre (IAF) neuron [44], which relates the membrane current to the conductance and the membrane potential: dV C m = −g (V − E ) + I (2.2) m dt L m L ext

In Eq 2.2, Cm stands for the membrane capacitance, gL the leak conductance, Iext the level of external input current, and EL the reversal potential. Following a spike, Vm is reset to a value Vr where Vr < Vth. It is called leaky because of the assumption that membrane potential dynamics are modeled as a single passive leakage term. The diﬀerence between Vm and EL is a driving force that determines current when multiplied by gL, and the neuron acts like a simple electrical circuit with a resistor and a capacitor. 20 CHAPTER 2. BACKGROUND

Such leaky IAF neuron models can reasonably approximate the dynamics of some cells and are useful for studying large-scale phenomena. But oftentimes the sim- pliﬁcations it introduces don’t adequately account for the complexities of experimentally observed dynamics. For example, other terms can be incorporated such that EL and gL vary with time. Many alternatives have been proposed to simply account for the realities of biology (Figure 2.5), and one of them is the adaptive exponential IAF model (AdEx) [45] that describes the temporal evolution of Vm as well as an adaptation current Iw as:

dV Vm−VT m ∆ Cm = −gL(Vm − EL) + gL∆T e T − Iw(t) + Iext dt (2.3) dI τ w = −I Iw dt w

AdEx theoretically describes spike emission at the time when Vm diverges towards inﬁnity, but in practice, when Vth is approached, Vm is reset to Vr like in Eq 2.2. Except here, Vt is not a strict threshold per se, but rather the center of an exponential nonlinearity modeling each action potential upswing with spike upstroke slope factor ∆T . Concurrently with the spike-induced voltage reset, an adaptation current Iw increases according to Iw = Iw + b where b is a constant additive current, and Iw then decays with time constant τIw . The incorporation of a dynamic adaptation current term not only enriches the dynamics of individual neurons but also networks comprising them as interconnected elements.

Figure 2.5: Comparison of the realistic and computational properties of spiking neuron models. Adopted from Figure 2 of [46].

Synapse models can incorporate diﬀerent levels of sophistication as well. Current- based models are more simplistic than those described thusfar, as they don’t in- 2.4. MODELING LEARNING, MEMORY AND NEURAL DYNAMICS 21

corporate any dependency on Vm as in Eq 2.2. And in the network context, conductance-based models incorporate other presynaptic sources i that can extend the singular view presented in Eqs 2.2 and 2.3:

X X syn syn Itotj (t) = gij (t)(Vmj −Eij ) = IAMP Aj (t)+INMDAj (t)+IGABAj (t) (2.4) syn i

Eq 2.4 describes the total current for postsynaptic neuron j aggregated from different subtypes of excitatory and inhibitory presynaptic inputs (see Section 2.2).

2.4.2 Autoassociative Memories and Attractors in Dynamical Systems Hebb’s ideas saw a resurgence in popularity in part due to important theoretical work later performed by the physicist John Hopfield, who expanded upon Hebbian learning by applying the collective stochastic dynamics of magnetic spin systems to the dynamics of recurrently connected neural networks [47]. In this way, the persistent firing of cell assemblies in Hopfield networks are akin to attractors in dynamical systems: a set of points in the phase space toward which the system tends to evolve. A trajectory that gets close enough to the attractor values will remain close even if they are perturbed. Attractors in this sense represent memories, several distinct memories can exist within a single network and the network connectivity as a whole represents the attractor landscape.

2.4.3 Gestalt phenomena and perceptual grouping Attractor networks display many of the original functions proposed by Hebb including persistent activity, pattern completion and rivalry and figure-background segmentation. For example, these properties confer the ability to stimulate a subset of cells belonging to an attractor, in other words partially activate the pattern, the remaining cells belonging to that pattern become activated resulting in a successful associative memory retrieval [48]. Such behavior is reminiscent of figure-background segmentation, which refers to the human perceptual tendency to see a figure in front of a background. Likewise, due to lateral inhibition between neurons of a pattern, stimulating a subset of cells belonging to one attractor with more vigor than a subset of cells belonging to a different attractor can lead to the higher stimulated one to win out, similar to the phenomena of perpetual rivalry that underlies how humans resolve ambiguous stimuli. These are holistic descriptions of human perception rooted in Gestalt psychology, which stresses that whole being separate from the sum of its parts is important for how the brain creates meaning from the world (Figure 2.6). 22 CHAPTER 2. BACKGROUND

Figure 2.6: Gestalt phenomena illustrated by the Necker cube (left) and Rubin vase (right). In both examples, like dynamical attractors, only one of the two interpretations of stimuli are perceived at a time. In the Necker cube, the frontward- facing square of the cube is multistable, and in the Rubin vase, its either a vase or two human face proﬁles.

2.4.4 Resolving the Problems of Abstract Attractor Memory Networks But there are several aspects of the classical attractor memory network paradigm described thus far that make it diﬃcult to reconcile with neurobiological reality [49]:

• all-to-all symmetric connectivity

• violating Dale’s Law: each neuron should only release one type of neurotransmitter at all of its synapses [50].

• activity levels of cell assemblies are signiﬁcantly higher than experimentally observed

• a ceiling exists for the number of storable patterns, after which the network suﬀers from catastrophic forgetting where all memories are lost when the system becomes overloaded.

• mechanisms for escaping from attractor states are absent

However, all of these issues can be addressed by taking a deeper look at some of the more recent experimental observations concerning cortical anatomy, and additionally using more realistic spike-based models of constituent neurons and synapses. The above bullet point are addressed below, respectively: 2.4. MODELING LEARNING, MEMORY AND NEURAL DYNAMICS 23

• replace the single pyramidal cell with a local population of interconnected pyramidal cells called minicolumns, sparsifying and making asymmetric connections at the neuron-to-neuron level [51] • polysynaptic subcircuit including long distance inhibitory targeting [48] • modularity and lateral inhibition by basket cells [5] • palimpsest memory [23] • spike-frequency adaptation and synaptic depression Another prominent feature of Hebb’s original hypotheses was the presence of phase sequences and association chains: cell assemblies that would sequentially activate in response to stimuli in close temporal proximity. One of the major contributions of the proceeding work will be to describe how this, like the previously described grievances, can be resolved by delving deeper into the level of neurobiological detail.

2.4.5 Biophysical cortical memory networks Phase sequences too can be accounted for in spiking cortical attractor memory network models. Generation of sequential states as an outcome of temporally asymmetric connections between neurons belonging to diﬀerent attractor states was proposed early on [52, 53]. Sequential attractor activity at any one point in time thus depends on the delegation of neurons belonging to minicolumns active in the previous time step, which is consistent with the idea of combinatorial coding [54, 55]. This redundant neural coding scheme should not necessarily be viewed in terms of anatomical columns, but rather functional columns consisting of sub- groups of neurons with similar receptive ﬁelds that are highly connected [27, 56] and coactive [57, 28].

Such distributed systems generally exhibit improved signal-to-noise, error resilience, generalizability and a structure suitable for Bayesian calculations [58, 59]. Like their uniformly interconnected counterparts, they exhibit high variability in their spike train statistics [60, 61], and moreover, these topologies are advantageous for information processing and stimulus sensitivity due to their capacity to exhibit a rich repertoire of behaviorally relevant activity states [62, 63].

Similar stereotypical architectures have been used as building blocks for uniﬁed mathematical frameworks of the neocortex [64, 65, 66]. When such anatomical organization is absent [67] or undetected, analogous clustering of functionally distinct but anatomically intermixed subnetworks are nonetheless believed to support the sequential propagation of activity [27, 68, 69, 70].

The interplay of excitation and inhibition has posed problems for biologically reconciling coexisting propagation and stabilization in recurrent neuronal networks [71]. 24 CHAPTER 2. BACKGROUND

Activity can be transmitted synchronously without disrupting network function under feed-forward dominated conditions [54], and as demonstrated here, stable propagation can be carried out by recurrent networks with high variability that make use of clustered excitatory connectivity [60, 61]. Another approach includes models of non-normal ampliﬁcation though, in which functionally feed-forward pathways can develop in the absence of mutually exciting neurons with similar activity [72].

2.4.6 Temporal sequences Several physiological studies of temporal perception have provided insights into how the cortex tells time. Neural ensembles can encode perceived elapsed time by discriminating stimulus duration using reference cues in the range of hundreds of milliseconds in intraparietal [73], frontal [74] and supplementary motor areas [75], pointing towards a coordinated involvement of many regions as part of a fundamental cortical timing circuit. But the sequential structure of replay dynamics can reﬂect the temporal interval used during training [76], hinting more broadly at an inherent computation of time passage by evolving patterns of activity.

Aided by recent advancements in multi-unit recording technologies and quantitative methods for extracting information from large neural populations, experimentalists and theoreticians are increasingly able to investigate the extent to which spiking activity is sequentially organized in behaving animals. Temporal sequences are thought to engage circuit-level phenomena since their concatenated durations are longer than individual cellular or synaptic time courses, but the intricacies of this structure-function relationship have yet to be fully explored. Though initially believed to result from specialized devices like an internal clock, the emergence of temporal sequences is alternatively explained by generic and non-dedicated, yet still elusive, neural mechanisms [77, 78].

Furthermore, negative deﬂections in the local ﬁeld potential propagate in a salta- tory manner [79], and large-scale spatiotemporal patterns distributed over wide swaths of cortex are believed to play a role in cross-modal sensory associations [80].

Despite initial skepticism regarding their statistical significance [81], the spontaneous presence of precisely repeating feedforward structures at fine temporal scales have been reported [82, 83]. The concept has since been refined to accommodate hierarchical [84] and polychronous [85, 86] motifs. But such temporal structure can be observed in coarser resolutions, too [87]. The massively recurrent and long-range nature of cortical connectivity, taken together with the emergence of temporal sequences at fine scales and distributed over spatial areas, suggests the presence of generic circuit mechanisms. In our model, we focused on a specific but widespread type of neural dynamics comprising transient neuronal coalitions that experience concomitant shifts in their firing rates and manifest as stereotypical trajectories taken through the state space of neocortical networks [88]. 2.5. ROLE OF COMPUTATIONAL NEUROSCIENCE 25

In prefrontal (parietal) cortex, different sequential activity trajectories are traversed in different trials according to whether rats (mice) turn left or right on their journey through a figure-eight (virtual) T-maze during a working memory task involving odor-place matching (navigation-based decision making) [89, 68]. Other task-critical spatial information can be dynamically encoded during object- construction, like whether the visual absence of a square is located on the left or right side of a reference object [90]. In this case, the left or right side was represented in monkey parietal cortex as brief sequential activations of successive neural groups corresponding to distinct network states.

Even prior to the onset of performed motor programs, successive prefrontal activity state transitions can recall the ordered segment positions of geometrical shapes that monkeys are trained to draw on a screen [91]. In this study, task errors indicated premature representation of the subsequent segment position. In other words, monkeys correctly predicted the segment to draw after next, but were wrong by task definition since they were getting ahead of themselves. Consistent with this notion, firing rate changes in pre-supplementary motor area neurons can hold information about the second-next movement in a series of movements, reflecting a build-up of preparatory activity on the population level [92]. Similar dynamics also arise in premotor cortex when monkeys plan intended arm-movements regardless of whether the motor plan is successfully carried out or canceled [93].

Multiple single-unit data from cingulate cortex of rats performing a decision-making task identified sequential ensemble organization only within successful trials [94]. Similarly acquired data analyzed using Hidden Markov Models revealed sequences of firing rate states in taste specific cells from gustatory cortex [95] and in monkey frontal cortex following visual cue presentation [96, 97]. Overall, the prevalence of attractor-like sequence-based circuit dynamics during sequential and non-sequential behaviors and behavioral outcomes hints that common computational strategies may be at play.

2.5 Role of computational neuroscience

Organisms have evolved unique and sophisticated solutions for coping with threat- ening external environments. Accompanying the advent of modern technology is our burgeoning understanding of these complex biological phenomena, and in turn, our ability to take technological inspiration from them to simplify human engineering problems. After all, what would the state of aircraft wing design be without our current understanding of bird ﬂight physics? Imitating nature is no less important in the age of information technology, in which modern society searches for eﬃcient ways to tackle still unresolved, large-scale problems. 26 CHAPTER 2. BACKGROUND

The brain is the cheapest and fastest learner, predictor, memorizer, and recognizer of patterns. It does so despite physical space constraints (i.e. skull size), offering an immense power and capability that is unmatched by alternative computing architectures. The field of neuroscience has greatly benefitted from recent advances in recording hardware, heralding unprecedented access to neural data at high res- olution and fidelity. As these technological developments march forward together with our collective knowledge regarding the computational structure of information processing in the brain, it becomes crucial to apply these insights to other practical problems facing society. By mimicking the synaptic connectivity and signal propagation characteristics of neurons, artificial neural networks (ANNs) represent an important step in this direction.

2.5.1 Top-down versus bottom-up modeling approaches

Is the whole greater than the sum of its parts? Starting from scratch by simulating the physics and chemistry of what is known regarding the physical brain, a bottom- up modeling approach involves combining the simplest measurable elements in the hopes of capturing function as a result of emergent phenomena resulting from these combinations and their interactions. It aims to answer whether details of the neural anatomy can incrementally reveal more and more information that may not be as obvious when individual elements are considered alone. A big proponent of this approach are the scientists involved in the Blue Brain [98] and Human Brain [99] Projects, which aim to build comprehensive models of a cortical column and the entire human brain, respectively. As it stands today, these projects require inor- dinate amounts of computing power in order to simulate such large systems using very detailed biological elements, and they are often run on the IBM Blue Gene or CRAY supercomputers [98].

On the other end of the spectrum lies the top-down modeling approach, which takes a more wholistic view of brain function by aiming to ﬁll in previously un- known details of the biological substrate by uniting as much known information as possible into theoretical frameworks. This approach often makes assumptions and abstractions regarding speciﬁc details, but can often be useful in terms of guid- ing model development by compensating for empirical data that remains missing, emphasizing certain included elements over others and providing more constraints. From a high level, the approach aims to understand computing and information processing of the brain.

In practice, however, its never so cut and dry. Modelers typically adopt a convenient mixture of both approaches depending on the issue in question. For example, the work in this thesis largely adapts a top-down approach, but at the same time tries to incorporate as many relevant detailed biological processes as possible. 2.5. ROLE OF COMPUTATIONAL NEUROSCIENCE 27

2.5.2 Large-scale modeling with high-performance computing

Another issue to consider when pursuing the appropriate modeling approach is the relative size of the model needed to prove or disprove the hypothesis in question. Typically, modelers will subsample the biological network being studied in order to 1) more easily understand the effects of different parameters being changed in relation to outputs being generated, and 2) make each simulation run faster in order to test more and increasingly diverse sets of parameters per trial. However, in subsampling connection strengths and/or densities must be artificially exaggerated in order to compensate for the small fraction of the real neurons, and therefore presynaptic input sources, being simulated [25, 49]. These hacky assumptions can degrade the quality of the model by distorting network dynamics, leading to unrea- sonable predictions and biologically irreproducible conditions. Many weak signals propagate within the biologically realistic networks, producing a rich repertoire of dynamics. Contrastingly introducing unnaturally few and strong signals can lead to artificial network synchronization, altering the dynamical regime of individual cells and distorting the relationship between individual underlying mechanisms and globally observed network phenomena.

Despite their numerous state variables and intensive computational demands, full networks with a one-to-one correspondence between biological and model neurons should be employed to most naturally and realistically imitate real neural substrates. This is the ideal to strive towards, as large-scale models undoubtedly enable the study of more elaborate network architectures that better match the complexity of the real brain. Studying more realistic models will help demystify the huge amount of experimental data gathered from large-scale recording techniques, and simultaneously help us understand how these measurements relate to biological processes and information-processing functions. It would help reconcile experimental results at the behavioral, psychophysical and psychological levels, and further expand the addressable experimental data by newer models.

But how far along are we on this journey? Computational power continues to grow according to Moore’s law, which asserts the number of transistors per square inch on computer circuits approximately doubles each year. Except these days, the hardware harnesses multi-core architectures to keep up with Moore’s law, meaning that software is increasingly designed in a multi-threaded fashion to exploit this parallelism. Extrapolating from Moore’s law, we can then assume that the supercomputers available about 10 years from now will probably be capable of simulating a full-scale human brain model in real-time. In 2008, supercomputer simulations of extremely large neuronal network models were already being performed, like a 22- million neuron 11-billion synapse mouse cortex model run on an IBM BlueGene/L cluster supercomputer with 8192 parallel processors [25]. More recently, the K computer was used to simulate a 1.86-billion neuron 11.1-trillion synapse generic balanced random network model [100]. But much progress is et to be made in terms 28 CHAPTER 2. BACKGROUND of power consumption and runtime of these simulations, which pale in comparison to the real-time 20 W-powered human brain. Studies such as these and others will continue to help enable a detailed understanding of diseases, perception, cognition and behavior in terms of dynamical processes occurring at the cellular, synaptic and molecular levels.

2.5.3 Neuromorphic Hardware

Still, the simulation of a complete human brain will require a supercomputer at least a thousand times more powerful than the state of the art. Additionally, despite Moore’s Law there are worrying signs that traditional CMOS supercomputer chips are beginning to reach their physical capacities. Even if this is not the case though, it’s clear that supercomputers will not be the ultimate solution for simulating a realistic and full-scale human brain: the underlying architectures are completely different, brittle supercomputers aren’t fault tolerant and the power consumption and physical size of modern supercomputers cannot even begin to compete with the extremely energy efficient neural computing structures of the brain. In addition to being more power efficient than their general purpose architecture counterparts, neuromorphic systems are better suited for embedded controllers of autonomous units like robots.

What’s needed is hardware that’s more brain-like and naturally scalable to massively interconnected neural networks. Indeed, neuromorphic systems are beginning to challenge the status quo in artiﬁcial neural network simulation. By providing hardware level realizations of molecular processes like those reproducing neural- and synaptic dynamics, neuromorphic hardware cut out the von-Nuemann bot- tleneck. Brain-like hardware can be implemented by analog, digital, mixed-mode analog/digital very large scale integration (VLSI) and software systems. The analog nature of transistors provide a relative, graded value representation that contrasts with absolute value of digital signals such that the underlying silicon is much better utilized. A single digital operation can be carried out by thousands of transistors and energy is spent charging up wires rather than the 0-1 gate itself, meaning there is a 1000000x disparity in power eﬃciency between digital computation and single transistor operation [101]. Emulation within hardware, involving measuring the circuit output voltage directly representing the result of an equation, turns out to be much cheaper than simulating the entire sequence of operations required to execute the equation digitally. It can even be sped up to perform operations faster than biological real time, capitalizing on the short physical electronic circuit distances [102]. In hybrid implementations, software can be utilized to perform tasks like digital communication, where spikes are broadcast across longer physical chip distances [102, 103]. Other more conventional implementations leverage general- purpose ARM processors to simulate digital currents, permitting programmatic simulation of a wide variety of neuron and synapse models [104]. 2.6. BAYESIAN BRAIN HYPOTHESIS 29

2.6 Bayesian brain hypothesis

Bayesian inference provides an intuitive framework for how the nervous system could internalize uncertainty about the external environment by optimally combining prior knowledge with information accumulated during exposure to sensory evidence. In an uncertain world, the Bayesian brain would generate internal perceptual representations of the statistics of its surroundings, allowing it to recursively make inferences to guide actions and make decisions. Bayesian approaches are advantageous because they assume that each local computational stage represents all possible parameter values and their associated probabilities, allowing the system to efficiently integrate information over space and time, from different sensory modalities and cues and to propagate uncertain information between stages without committing too early to certain conclusions [105]. It is attractive in its simplicity and power: it is evidence-based just like the scientific method in general.

2.6.1 Perceptual inference: evidence from psychophysics Bayesian inference is a viable neurobiological encoding strategy that reasonably captures numerous psychophysical data, including human motor behavior. In a ﬁnger reaching task, subjects were found to internally represent both the statistical distribution of the task and their sensory uncertainty during sensorimotor learning, combining these probabilistic quantities through a performance-optimizing Bayesian estimation process [106]. In a similar ﬁnger reaching task during which subjects had to perform rapid pointing movements at unseen targets, it was found that they employed near-optimal Bayesian strategies by combining stimulus uncertainty and prior information of the distribution of possible target locations [107].

2.6.2 Representing probabilities within the neural substrate Theoreticians have proposed that the notion of probabilistic computation should not be restricted to interpretations on the systems level, but rather that probabilistic computation, if plausible, should occur on the timescale of a single inter-spike interval (ISI) in order to match the temporal scale of perception and sensorimotor integration [108]. This intuition propelled others to devise biophysical substrates for the neural implementation of Bayes rule, but it remains an open theoretical issue at which level of detail within the neural substrate probabilistic computation should be embedded [105].

From single neurons [108], neural population responses [109], speciﬁcally within the parietal [110] and prefrontal [111] cortices, activation levels in the visual cortical hierarchy [112], LTP [113], and short-term synaptic plasticity [114], realizations of disparate phenomena occurring within the cortical circuitry have been hypothesized to represent viable coding schemes for such Bayesian principles. More complex modeling approaches combine multiple synergistic empirically grounded phenom- 30 CHAPTER 2. BACKGROUND ena for learning and inference. Coupled synaptic plasticity and synaptic scaling [115] along with coupled STDP and homeostatic intrinsic excitability [116] have been proposed in the context of the expectation maximization algorithm, whereas a model with coupled synaptic and intrinsic plasticity has been implemented using Gibbs sampling [117].

But inductive frameworks notoriously tend to impose restrictions about when learning should occur (if at all) and account for a fraction of the diversity in physiological processes whose given anatomical granularity is otherwise arbitrary. The focus of the remaining sections will be on deriving and demonstrating a model based on probabilistic computation that encompasses many of the aforementioned biological dynamical phenomena that can operate on the scale of the brain. Chapter 3

Materials and Methods

3.1 Bayesian Conﬁdence Propagation Neural Network

Consider a paradigm in which learning and recall are probabilistically grounded, associative memory mechanisms. According to BCPNN [118, 119], computational units representing stochastic events have an associated activation state reﬂected by a real value between 0 and 1. This corresponds to the probability of that event, given observed events, which are represented by other active units. In spike-based BCPNN, units are viewed as local populations of 30 spiking neurons [120], i.e., minicolumns, that have similar receptive ﬁelds and are highly connected and coactive [29, 27, 28]. Corresponding levels of activation for these minicolumns are represented by their average spike rate normalized from between 0 and 1 activations to between 0 Hz and 20 Hz.

Starting from Bayes’ rule for relating the conditional probabilities of two random variables, observed ﬁring rates collected from n presynaptic minicolumns x1...n, i.e., the evidence P (x1...n), can better inform the ﬁring probabilities of neurons in the postsynaptic minicolumn yj, i.e., the prior P (yj):

P (x1...n|yj) P (yj|x1...n) = P (yj) (3.1) P (x1...n)

By assuming conditional independence between x1...n, Bayes’ rule can be extended by:

P (x1|yj) P (x2|yj) P (xn|yj) P (yj|x1...n) = P (yj) ... (3.2) P (x1) P (x2) P (xn) The described learning approach from Equation 3.2 is tantamount to a naïve Bayes classiﬁer that attempts to estimate the posterior probability distribution P (yj|x1...n) over a class (e.g., yj = "animal") realized by its observed attributes (e.g., xh = "shape," "color," or "size"). The assumption of independent marginals

31 32 CHAPTER 3. MATERIALS AND METHODS above is insigniﬁcant considering that the denominator of Equation 3.2 is identical for each yj. Thus, relative probabilistic ordering of classes remains intact, and probabilities can be recovered by normalizing P (yj|x1...n) to sum to 1. If we deﬁne each attribute xh as a discrete coded or as an interval coded continuous variable (e.g., xhi = "blue," "yellow," or "pink" for xh = "color"), a modular network topology follows: H nh Y X P (xhi|yj) P (yj|x1...n) = P (yj) πxhi (3.3) P (xhi) h=1 i=1 in which nh minicolumns are distributed into each of H hypercolumns (Figure

3.1A). Here, πxhi represents relative activity or uncertainty of the attribute value xhi, and πxhi = 1 indicates that attribute value xhi was observed with maximal certainty. Equation 3.3 may instead be equivalently expressed as a sum of logarithms by: H nh X X P (xhi|yj) logP (yj|x1...n) = logP (yj) + log πxhi (3.4) P (xhi) h=1 i=1

Figure 3.1: Reconciling neuronal and probabilistic spaces using the spike-based BCPNN architecture for a postsynaptic minicolumn with activity yj. A A cartoon of the derived network incorporates NHC = 5 hypercolumns each containing NMC = 4 minicolumns that laterally inhibit each other (red lines) to perform a WTA operation via local inhibitory interneurons (red circles). The dotted gray area is represented by B in detail. B Weighted input rates x1...N are summed and passed through a transfer function to determine the amount of output activation.

Connections wxi yj can be viewed as synaptic strengths (black lines, semicircles) or inverted directed acyclic graph edges representing the underlying generative model of a naïve Bayes classiﬁer. Adopted from Figure 2 of [121].

Equation 3.4 states that contributions via connections from minicolumns in the same hypercolumn need to be summed before taking the logarithm, then summed again. Such an operation might be performed dendritically. More generally, the sum inside the logarithm can be approximated by one term through the elimination 3.2. THE SPIKE-BASED BCPNN LEARNING RULE AND CORTICAL MICROCIRCUIT MODEL 33 of index h, since there are signiﬁcantly more hypercolumns than incoming synapses per neuron in mammalian neocortical networks. Considering the asymptotically large size and sparse connectivity of these networks, it is statistically unlikely that a speciﬁc hypercolumn would receive more than one incoming connection from any other hypercolumn.

n Ph Each hypercolumn is regarded as having normalized activity πxhi = 1, and i=1 such canonical connectivity schemes along with the WTA operations they imply are prevalent throughout neocortex [30]. Hence in analogy to neural transduction, N P a support value πxi wxiyj can be calculated by iterating over the set of possible i=1 conditioning attribute values N = Hnh for yj with weight wxiyj and bias βj update equations (Figure 3.1B):

βj = logP (yj)

P (xi|yj) P (xi, yj) (3.5) wxiyj = = log P (xi) P (xiP (yj)

Activity statistics are gathered during learning and their relative importance is evaluated and expressed as weights and biases. After Bayesian updating, probabilities are recovered by normalizing P (yj|x1...n) to sum to 1 over each yj by using an exponential transfer function since sj = logP (yj|x1...n):

esj logP (yj|x1...n) = n (3.6) P h si i=1 e The variables w and β are henceforth referred to as models of the incoming synaptic strength and excitability of a neuron. In the case where multiple synaptic boutons from a pre- to postsynaptic target neuron exist, they are represented as a single synapse.

3.2 The spike-based BCPNN learning rule and Cortical microcircuit model

Using the previously described synaptic and neural spike-based BCPNN learning models, this section summarizes a computational model of a cortical microcircuit [8] inspired by previous work [5] that was topographically structured according to the columnar organization of neocortex [26, 29]. The network consisted of 2700 pyramidal cells and 270 basket cells divided into nine hypercolumns (NHC = 9), each consisting of 300 pyramidal cells and 30 basket cells. Pyramidal cells had spike-frequency adaptation according to the AdEx model [45].

The pyramidal cells in each hypercolumn were further divided into ten minicolumns 34 CHAPTER 3. MATERIALS AND METHODS

(NMC = 10), each consisting of 30 pyramidal cells [120]. The pyramidal cells in each minicolumn laterally projected to and received feedback from the 30 local inhibitory basket cells within their own hypercolumn to form competitive WTA subnetworks [30]. These connections were static and consisted of both excitatory AMPA-mediated pyramidal-to-basket and inhibitory GABAergic basket-to- pyramidal connections with connection probabilities set to 0.7. In addition to the static local (i.e., within-hypercolumn) feedback connections, there were plastic BCPNN synapses (see below), which randomly formed AMPA- and NMDA- mediated connections among all pyramidal cells across the network with distant- dependent axonal delays and a connection probability of 0.25 (Figure 3.2A).

Figure 3.2: A A more detailed illustration of Figure 2.3. Here, the WTA hypercolumns (shaded gray areas) consisted of excitatory neurons (red circles) belonging to minicolumns (red ovals) that innervate (red arrows) local populations (blue ovals) of basket cells (blue circles), which provide inhibitory feedback (blue arrows). Non- speciﬁc BCPNN synapses (black arrow) are recurrently connected amongst all excitatory neurons (black dotted box), with learned weights (scale) determined by pre- and postsynaptic spike activity. Semicircular arrowheads indicate the postsynaptic target. B Schematic of a polysynaptic subcircuit that elicited a net excitatory eﬀect on the postsynaptic neuron when the BCPNN weight was positive (+ pathway), while a negative BCPNN weight can be thought of as being converted to inhibition via excitation of a local inhibitory interneuron (- pathway). Adopted from Figure 1 of [8].

The logarithmic transformation from Equation 3.5 was motivated by the Bayesian underpinnings of the learning rule, but this caveat meant that multiplexed both positive and negative signals. Learning of excitatory or inhibitory weights was interpreted in the context of a monosynaptic excitatory connection with conduc- 3.2. THE SPIKE-BASED BCPNN LEARNING RULE AND CORTICAL MICROCIRCUIT MODEL 35 tance set by the positive component of alongside a disynaptic inhibitory connection [122, 123, 124] set by the negative component (Figure 3.2B). In practice, when the sign of the BCPNN weight turned negative, an inhibitory reversal potential was used instead of an excitatory one.

Spike-based BCPNN is based on memory traces implemented as exponentially weighted moving averages (EWMAs) [125] of spikes, which were used to estimate Pi, Pj and Pij as deﬁned above. Temporal smoothing corresponds to integration of neural activity by molecular processes and enables manipulation of these traces; it is a technique commonly implemented in synapse [126] and neuron [127] models. EWMAs can ensure newly presented evidence is prioritized over previously learned patterns because as old memories decay, they are gradually replaced by more recent ones.

The dynamics governing the diﬀerential equations of the learning rule with two input spike trains, Si from presynaptic neuron i and Sj from postsynaptic neuron j, are illustrated in Figure 3.3A. A three-stage EWMA procedure (Figures 3.3B-D) was adopted, the time constants of which were chosen to have a phenomenologi- cal mapping to key plasticity-relevant changes within signal transduction pathways that occur during learning.

The Zi and Zj traces had the fastest dynamics (Figure 3.3B), and were deﬁned as

dZ S τ i = i − Z + zi dt f ∆ i max T (3.7) dZj Sj τzj = − Zj + dt fmax∆T

which ﬁltered pre- and postsynaptic activity with time constants τzi , τzj ≈ 5-100 ms to match rapid Ca2+ inﬂux via NMDA receptors or voltage-gated Ca2+ channels [128, 9]. These events initiate synaptic plasticity and can determine the time scale of the coincidence detection window for LTP induction [129].

We assumed that each neuron could maximally fire at fmax Hz and minimally at Hz, which represented absolute certainty and doubt about the evidential context of the input. Relative uncertainty was represented by firing levels between these bounds. Since every spike event had duration ∆T ms, normalizing each spike by fmax∆T meant that it contributed an appropriate proportion of overall probability in a given unit of time by making the underlying Z trace ≈ 1. This established a linear transformation between probability space ∈ {, 1} and neuronal spike rate ∈ {, fmax}. Placing upper and lower bounds on firing rates was reasonable given physiologically relevant firing rates of cortical pyramidal neurons [82].

The Z traces were passed on to the E or eligibility traces [130], which evolved 36 CHAPTER 3. MATERIALS AND METHODS

Figure 3.3: Schematic flow of BCPNN update equations reformulated as spike- based plasticity. A The Si pre- (A-D, red) and Sj postsynaptic (A-D, blue) neuron spike trains are presented as arbitrary example input patterns. Each subsequent row (B-D) corresponds to a single stage in the EWMA estimate of the terms used in the incremental Bayesian weight update. B Z traces low pass filter input spike trains with τzi = τzj . C E traces compute a low pass filtered representation of the Z traces at time scale τe. Co-activity now enters in a mutual trace (C,D, black). D E traces feed into P traces that have the slowest plasticity and longest memory, which is established by τp. Adopted from Figure 1 of [121]. according to (Figure 3.3C): dE τ i = Z − E e dt i i dE τ j = Z − E (3.8) e dt j j dE τ ij = Z − E e dt ij ij At this stage of the EWMAs, a separate equation was introduced to track coincident activity from the Z traces. Eligibility traces have been used extensively to simulate delayed reward paradigms in previous models [131, 132], and are viewed as a potential neural mechanism underlying reinforcement learning [133]. They enabled simultaneous pre-post spiking to trigger a buildup of activity in the E traces, which could then be eligible for externally driven neuromodulatory intervention. The time 3.2. THE SPIKE-BASED BCPNN LEARNING RULE AND CORTICAL MICROCIRCUIT MODEL 37 constant τe ≈ 100-1000 ms was assumed to represent one of the downstream cellular processes that could interact with increased intracellular Ca2+ concentrations, such as CaMKII activation [134]. Creation of a decaying tag for each pre-post activated synapse for delivery of a specific marker that can be targeted for future plasticity-associated protein trafficking [20] has also been hypothesized to provide an intermediary step in the transition from early to late phase LTP.

E traces were subsequently passed on to the P traces (Figure 3.3D). Gene expression, protein synthesis and protein capture are cellular processes that mediate LTP maintenance and long-term memory formation [135, 20]. They tend to be activated in late phase LTP by elevated levels of Ca2+ dependent protein kinases, akin to activation in the P trace dynamics originating from sustained activation in the E traces: dP τ i = κ(E − P ) p dt i i dP τ j = κ(E − P ) (3.9) p dt j j dP τ ij = κ(E − P ) p dt ij ij Since these processes tend to exhibit highly variable timescales lasting anywhere from several seconds up to potentially days or months [136], we simply imposed

τzi , τzj < τe < τp, but typically used τp ≈ 10 s for the sake of conciseness in simulations. Directly regulating the learning rate, parameter κ ∈ {0, ∞} represented the action of an endogenous neuromodulator, e.g., dopamine [41], that signaled the relevance of recent synaptic events. The P trace is considered a versatile process tied closely to the nature of the task at hand by a globally applied κ [41]. Recently stored correlations were propagated when κ 6= 0 and no weight changes take place when κ = 0. Although we show through simulations how delayed reward could be implemented with E traces, they are not required for inference, and having τe approach 0 would not undermine any of the results presented here.

Probabilities were ultimately fed into the ﬁnal learning rule update equations (Equation 3.10) used to compute βj and wij:

βj = log(Pj)

Pij (3.10) wij = log PiPj

The bias current βj enters the equation describing the neuron model, for example in an AdEx neuron (see Eq 2.3) it would enter according to:

dVm Vm−Vt C = −g (V − E ) + g ∆ e ∆T − I (t) − I (t) + I + I (3.11) m dt L m L L T w tot βj ext

Here, the term Iβj appears as an additive hyperpolarizing current. The weight wij is the synaptic weight once it’s normalized by a constant term wgain to reﬂect the 38 CHAPTER 3. MATERIALS AND METHODS current- or conductance-based nature of its residing synapse.

3.3 Simulation tools and strategies

3.3.1 NEST To confer upon simulations context and predictive power, the conditions and results of electrophysiological experiments can be achieved in silico using computer pro- gram libraries like the Neural Simulation Tool (NEST) [137]. The workflow of these programs is straightforward: after setting parameters for the network like connectivity, cellular and synaptic properties and simulation length, output variables like spikes and/or voltages can be written to files for conducting postanalyses. NEST was chosen for all of the work presented here because it is optimized for large networks of spiking neurons, as it can represent spikes on continuous timescales and execute supralinearly on parallel computers without sacrificing mathematical ac- curacy [138]. NEST has a convenient Python interface (i.e. PyNEST) and many built-in, open-source neuron and synapse models to enable fast testing of hypotheses. However, NEST was extended to include models defined by Equations 3.7-3.11 above for the purposes of this work.

3.3.2 PyNN PyNN was created to be a simulator-independent Python interface that supports NEST among other simulation programs [139]. Because the existence of different simulation programs made it difficult to reproduce the work of others working on creating models within computational neuroscience and in turn simulation results had to be verified across different implementations, PyNN was introduced as a way to abstract away the details of particular software setups in favor of more translatable model development.

3.3.3 SpiNNaker SpiNNaker is a digital neuromorphic architecture designed for the simulation of spiking neural networks using PyNN. Although systems built on SpiNNaker chips are available in sizes ranging from single boards to room-size machines [104]. SpiN- Naker chips are connected to their six immediate neighbors using a chip-level in- terconnection network with a hexagonal mesh topology. They each contain 18 ARM cores connected to each other through a network-on-chip, and connected to an external network through a multicast router. Each ARM core has two small tightly-coupled memories: 32 KiB for instructions and 64 KiB for data. They share 128 MiB of oﬀ-chip SDRAM with the other cores on the same chip.

The SpiNNaker neural simulation kernel maps neural networks in a memory-efficient 3.3. SIMULATION TOOLS AND STRATEGIES 39 manner while still obtaining supra-linear speed-up as the number of processors increases [140, 141]. Each processing core can simulate between 100 and 1000 neurons and their afferent synapses (Figure 3.4). When a spike occurs, the core sends a packet containing a 32 bit ID to the SpiNNaker router. Spike packets are then routed to the cores that are responsible for simulating these synapses. Biological neurons have in the order of 103 - 104 afferent synapses, so updating all of these every time step would be extremely computationally intensive. But taking advan- tage of the low spiking rates of biological networks, spikes can only cause synaptic updates as long as a new state can be calculated from the previous state, the time since the last spike and the time-driven simulation of the postsynaptic neuron. Us- ing this event-driven approach is advantageous because of the exploding number of synapses need to be stored in the off-chip SDRAM, which has insufficient band- width for every synapse’s state to be retrieved every simulation time step [143]. Instead, a core retrieves the row of the connectivity matrix associated with the firing neuron from SDRAM upon receipt of a spike packet. Each of these rows describes the parameters associated with the synapses connecting the firing neuron to those simulated on the core. Once a row is retrieved the weights are inserted into an input ring-buffer, where they remain until any synaptic delay has elapsed and they are applied to the neuronal input current.

In addition to enabling large-scale simulations with static synapses, this event- driven approach can be extended to handle other models of plastic synapses. How- ever, simulating plastic synapses has additional overheads in terms of memory and CPU load. Several different approaches have been previously developed that aim to minimize memory usage [144], reduce CPU load [145] or offload the processing of plastic synapses to dedicated cores [146]. An algorithm for distributed spiking neural network simulation including synaptic plasticity in an event-driven manner was developed using a simplified model of synaptic delay to reduce CPU and memory usage [147]. Results will demonstrate how SpiNNaker and its limited resources can overcome such issues. 40 CHAPTER 3. MATERIALS AND METHODS

Figure 3.4: Mapping of a spiking neural network to SpiNNaker. A network consisting of 12 neurons is distributed between two SpiNNaker cores. Each core is responsible for simulating six neurons (filled circles) and holds a list of afferent synapses (non-filled circles) associated with each neuron in the network. The SpiNNaker router routes spikes from firing neurons (filled circles) to the cores responsible for simulating the neurons to which they make efferent synaptic connections. Adopted from Figure 1 of [142]. Chapter 4

Results and Discussion

4.1 Paper 1: Emergent dynamics and neurobiological correlates

In spike-based BCPNN, output firing rates represent the posterior probability of observing a presented pattern. Although it is calculated by exponentiating the support activity (Equation 3.6), exponential input-output curves are rarely measured in experiments despite the apparent computational benefits of non-linear input transformation at the level of single neurons [148]. To account for these biological constraints, an alternative scenario is considered in which a neuron is stimulated by excitatory Poisson background input such that the mean voltage of its membrane potential is subthreshold (Figure 4.1A) and it fires up to intermediate levels. This background-driven regime enables spike production due to fluctuations in subthreshold membrane voltage, and is thought to approximate in vivo conditions during which cortical neurons are bombarded by ongoing synaptic background activity [149].

Linearly increasing the level of presynaptic drive in the presence of background activity caused an expansive non-linearity in the IAF input-output curve within a physiologically relevant <1 up to 20 Hz range of ﬁring, which has been reported previously for conductance-based IAF neurons [150] and cortical neurons [151]. The time-averaged ﬁring rate was well-approximated by an exponential function (Figure 4.1B). Information deemed relevant in the form of increased activity by a subset of presynaptic sources can cause the postsynaptic neuron to ascend its activation function.

The neural input-output relationship is controlled by the abundance, kinetics, and biochemical properties of ion channels present in the postsynaptic cell membrane. This is represented in spike-based BCPNN by the variable βj, which is a function of the prior probability of postsynaptic activity Pj (Equation 3.10, see Figure

41 42 CHAPTER 4. RESULTS AND DISCUSSION

Figure 4.1: A The bias term shows logarithmically increasing ﬁring rate values of the neuron for which it is computed. B When hyperpolarizing current proportional to βj is applied, neurons that have previously been highly active will be more easily excitable (e.g., yellow curve) compared to neurons that have had little recent history of ﬁring (e.g., blue curve). Error bars depict the standard deviation gathered from 50 repeated experiments. Adopted from Figure 8 of [121].

4.1A), and quantifies a general level of excitability and spiking for the postsynaptic neuron. Because βj → log Pj for minimal and βj → log(1) = 0 for maximal postsynaptic firing rates, βj essentially lowered the probability for neurons that were seldom active previously to be driven passed threshold in the future. With regards to the statistical inference process, this excitability represents an a priori estimate of postsynaptic activation. The intuition is if an event is experienced for the first time, it will still be highly unexpected. To account for these effects neurally, βj was treated as a hyperpolarizing current, Iβj , that was continuously injected into the IAF neuron according to Equation 3.11.

The outcome of this type of dynamic modification is illustrated in Figure 4.1B. The input-output curve shifted depending on βj, and the same synaptic input caused differing output levels. Similarly, LTP-IE provides a priming mechanism that can sensitively tune membrane properties of a neuron in response to altered levels of incoming synaptic input [7]. The A-type K+ channel gates the outward flow of current in an activity-dependent manner prescribed by a logarithmic transformation 2+ of Pj [152, 22, 153]. The decay of a Ca -activated non-specific cationic (CAN) current mediated by activation of transient receptor potential (TRP) channels [154] is another candidate that is thought to play a role in these graded changes [155]. Mirroring the cascading trace levels that collectively compute βj, multiple time scales of TRP current decay rate have been identified including a fast decay of 10 ms [156], a medium decay of 200-300 ms [157] and a slow decay of 2-3 s [158].

Intrinsic excitability has been conjectured to serve as a memory substrate via locally stored information in the form of a neuron’s activity history. Despite the lack 4.1. PAPER 1: EMERGENT DYNAMICS AND NEUROBIOLOGICAL CORRELATES 43 of temporal specificity that exists for synapses, intrinsic effects provide an alternative computational device that is presumably beneficial for learning and memory. We therefore asked how βj could account for functional aspects associated with the modulation of intrinsic excitability.

Speciﬁcally, we sought to model the rapid changes in intrinsic excitability found in slice preparations of layer V neurons from entorhinal cortex in rat [159]. In this study, initially silent neurons were repeatedly depolarized leading to a graded increases in their persistent ﬁring levels. It was also shown that persistent activity states were deactivated by applying hyperpolarizing steps until quiescence. Figure

4.2A summarizes this stimulus protocol, which was applied to an Iβj -modulated IAF neuron in the presence of background excitation. Duration and magnitude of the transient events were parameterized according to Egorov et al. (2002), using depolarizing steps of 0.3 nA for 4 s each and hyperpolarizing steps of 0.4 nA for 6 s each. The resulting activity of the neuron is illustrated by Figure 4.2B. Stable periods of elevated and suppressed ﬁring rates were associated with increases and decreases in Iβj , respectively. To achieve quantitatively similar graded persistent ﬁring levels as was shown in Egorov et al. a τp of 60 s was used, similar to induction time courses observed for LTP-IE in neurons from visual cortex [7] and cerebellar deep nuclear neurons [160]. The sustained levels of activation were noisier than the in vitro preparation of Egorov et al., presumably due to the presence of excitatory synaptic background activity in the model.

Importantly, the increased rate of firing caused by each depolarizing stimulus ap- plication period led to a continuum of levels up to fmax Hz, rather than discretely coded activity levels [155]. The number of levels was arbitrary and depended on both the magnitude and duration of the pulse, displaying peak frequencies (<20 Hz) similar to those that were assumed for fmax. To test this, ten depolarizing 2 s current steps were induced, producing a continuum of levels that was approximately linear with a regression coefficient of 1.33 (Figure 4.2B inset, red dotted line). Discharges were sustained by changes in the Pj trace (Figure 4.2C). Each depolarizing step led to the generation of spikes which transiently increased Pj and made βj less negative. Conversely, each hyperpolarizing step tended to silence output activity, decreasing βj and making it more difficult for the neuron to reach threshold. A bidirectional effect of βj was apparent here, as excitability decreased when the neuron was depotentiated [22].

The spiking setup allowed us to consider more detailed temporal aspects of plasticity beyond simple rate-modulated Poisson processes. First, we investigated how the temporal relationship between pre- and postsynaptic activity inﬂuenced expression of plasticity in our model. To evaluate the STDP properties of spike-based BCPNN, a canonical experimental protocol was simulated [129, 12] by inducing pre- (ti) and postsynaptic (tj) spiking in IAF neurons shortly before or after one another 60 times at 1 Hz frequency without background activity (Figure 4.3A). 44 CHAPTER 4. RESULTS AND DISCUSSION

Figure 4.2: Persistent graded activity of entorhinal cortical neurons due to βj. A Stimulation protocol. Repetitive depolarizing followed by hyperpolarizing current injections (switch occurs at black arrow) of the IAF neuron including βj. B Peris- timulus time histogram (2 s bin width) of the elicited discharge. Red bars indicate the time averaged activity of each 1 min post-stimulus interval. Time averaged activity of 1 min post-stimulus intervals using 0.3 nA depolarizing steps each lasting 2 s (stars, red-dotted line: linear ﬁt). C Underlying Pj trace evolution during the simulation. Adopted from Figure 9 of [121].

The strength of the weight changes were bidirectional and weight-dependent (Fig- ure 4.3B), generally exhibiting LTP for tight values of ∆t = ti − tj and LTD for wider values of ∆t (Figure 4.3C). The shape of the learning window was dependent upon the parameters τzi , τzj , and τp, defining the duration of the different memory traces in the model (see Equation 3.7). Manipulation of the Z trace time constants changed the width of the STDP window, and therefore τzi and τzj effectively regulated sensitivity to spike coincidence. Having τzi 6= τzj generated an asymmetric weight structure that allowed for prioritization of pre-post timing (+∆t) over post- pre timing (-∆t, Figure 4.3D) and vice versa (Figure 4.3E). The LTD area shrank for a constant STDP window width when τp was increased because it induced a longer decay time for the P traces (Figure 4.3F), emphasizing a slowness in learning. Temporally symmetrical Hebbian learning was due to an increase of Pij as a result of the amount of overlap between Pi and Pj. A similar form of LTP based on pre- and postsynaptic spike train overlap has been shown for synapses in slices [161].

Long-term stability can be problematic for correlative learning rules (e.g., Fig- ure 4.3C), since bounded Hebbian synapses destabilize plastic networks by maxi- 4.1. PAPER 1: EMERGENT DYNAMICS AND NEUROBIOLOGICAL CORRELATES 45

Figure 4.3: STDP function curves are shaped by the Z trace time constants. A Schematic representation of the STDP conditioning protocol. Each pre (blue)-post (green) pairing is repeated for each time diﬀerence ∆t = ti −tj illustrated in (C-E). B Weight dependence for positive (∆t = 0 ms, solid line) and negative (∆t = 50 ms, dashed line) spike timings. Compare to Figure 5 of Bi and Poo (1998). C

Relative change in peak synaptic amplitude using τzi = 5 ms, τzj = 5 ms, τe = 100 ms, and τp = 10000 ms. This curve is reproduced in (D-F) using dotted lines as a reference. D The width of the LTP window is determined by the magnitude of the Z trace time constants. When τzj is changed to 2 ms, the coincident learning window shifts right. E Instead when τzi is changed to 2 ms, it shifts left. Note that a decrease in τzi is thus qualitatively consistent with the canonical STDP kernel. F Changing the P trace time constant inﬂuences the amount of LTD. When τp is doubled to 20000 ms, the learned correlations tend to decay at a slower rate. Adopted from Figure 5 of [121].

mally potentiating or depressing synapses. Additional mechanisms such as weight- dependent weight changes [15] or ﬁne tuning of window parameters [162, 163] have been shown to be able to keep weights in check. In contrast, owing to its plasticity dynamics during on-line probability estimation, spike-based BCPNN naturally demonstrated weight dependence (Figure 4.3B) along with a stable unimodal equilibrium weight distribution when exposed to prolonged uncorrelated stimulation. 46 CHAPTER 4. RESULTS AND DISCUSSION

We conducted equilibrium experiments (Figure 4.4) using spike-based BCPNN synapses in which each of their mean stationary weight distributions were shifted upwards by the lowest possible allowed weight. This subtrahend was calculated from 2 2 2 2 Equation 3.10, log( /0.5 ) = log(4 ), or the log minimum Pij = (no co-activity) 2 divided by maximum PiPj = 0.5 (both pre- and post-neurons are active half of the time) trace values. Although this normalization would not occur biologically, it was necessary for displaying true equilibrium weight values because the average weight distribution ≈ 0 after τp ms due to P trace decay, and zero-valued average weights would have mitigated any postsynaptic response in the absence of background input. To demonstrate stability, a postsynaptic neuron is shown steadily ﬁring at an average of 7 Hz when innervated by 1000 presynaptic input neurons each producing 5 Hz Poisson spike trains due to background activity (Figure 4.4A). Given this setup (Figure 4.4B), the evolution of the renormalized synaptic weights during this period settled around 0 (Figure 4.4C).

Figure 4.4: The BCPNN learning rule exhibits a stable equilibrium weight distribution. A Progression of averaged rates of firing (3 s bins) for the presynaptic (blue) and postsynaptic (black) neurons in the network. B Setup involves 1000 Poisson-firing presynaptic neurons that drive one postsynaptic cell. C The BCPNN synaptic strengths recorded every 100 ms (blue, dotted white line is their instantaneous mean) has an initial transient but then remains steady throughout the entire simulation despite deviation amongst individual weights within the equilibrium distribution. D BCPNN weight histogram plotted for the final time epoch is unimodal and approximately normally distributed (blue line, µ0 = 0.0 and σ0 = 0.38). Adopted from Figure 6 of [121].

This behavior can be understood by investigating the P traces. Initially, both Pi and Pj increased as presynaptic input elicited postsynaptic spiking, growing the value of the denominator from Equation 3.10. In the numerator, the mutual trace 4.1. PAPER 1: EMERGENT DYNAMICS AND NEUROBIOLOGICAL CORRELATES 47

Pij built up as well, and there was an eventual convergence in the P traces to p PiPj = Pij after an elapsed time τp. Because both neurons fired together, the learning rule initially enhanced their connection strength, creating an initial transient output rate excursion. But as input persisted such that pre- and postsynaptic neurons continued firing at constant rates, correlations were eventually lost due to P trace decay. Statistically speaking, the signals emitted by the two neurons were indistinguishable over long timescales. The steady state of the weights ended up approximately Gaussian distributed around the quotient log(1) ≈ 0 (Figure 4.4D), independent of the approximate rates for the pre- and postsynaptic neurons. This stability was robust to the choice of time constants, given relatively constant pre- and postsynaptic firing rates.

Figure 4.5: Delayed reward learning using E traces. A A pair of neurons fire randomly and elicit changes in the pre- (red) and postsynaptic (blue) Z traces of a BCPNN synapse connecting them. Sometimes by chance (pre before post*, synchronous+, post before pre#), the neurons fire coincidentally and the degree of overlap of their Z traces (inset, light blue), regardless of their order of firing, is propagated to the mutual eligibility trace Eij. B A reward (pink rectangular function, not to scale) is delivered as external supervision. Resulting E traces are indicated (gray line, τe = 100 ms and black line, τe = 1000 ms). C Behavior of color corresponding Pij traces and weights (inset) depends on whether or not the reward reached the synapses in ample time. Adopted from Figure 3 of [121].

A learning scheme involving delayed rewards is depicted with a pair of connected neurons (Figure 4.5A). In this example, a reward was delivered 1-2 s after coincident activity [164] for 500 ms [165] to reinforce deserving stimuli. If τe was too small or positive reward κ arrived after the E trace had decayed to baseline (Figure 4.5B), no signal was propagated to the P traces. As a result, the corresponding Pij trace and weight remained unchanged. However, if the E trace was suﬃciently large such that there was an overlap with κ, the strength of the synapse grew and associative 48 CHAPTER 4. RESULTS AND DISCUSSION learning transpired (Figure 4.5C). Although only one connection is depicted in this example, κ would be modulated in the same way for all synapses in the network context, typical of dopaminergic neuron release characteristics [164]. 4.2. PAPER 2: BAYESIAN-HEBBIAN LEARNING IN AN ATTRACTOR NETWORK 49 4.2 Paper 2: Bayesian-Hebbian learning in an attractor network

We then proceeded to use the spike-based BCPNN learning rule in a simulated attractor network to generate sequential activations. The network was initially trained using 10 orthogonal component patterns. Each pattern consisted of one active minicolumn per hypercolumn without overlap (Figure 4.6), presented for tstim = 100 ms using an inter-pulse interval (IPI) of 2 seconds (Figure 4.7A) during which κ was clamped to 0 (Figure 4.7B).

Figure 4.6: Learning spontaneously wandering attractor states. Alternate network schematic with hypercolumns (large black circles), along with their member minicolumns (smaller colored circles) and local basket cell populations (centered gray circles). Minicolumns with the same color across diﬀerent hypercolumns belong to the same pattern; these color indicators are used in the following subﬁgures. Adopted from Figure 3 of [8].

Over the course of training, plastic changes transpired for intrinsic currents (Figure 4.8A) and weights in both AMPA (Figure 4.8B) and NMDA (Figure 4.8C) synapses. Neurons sharing patterns developed positive connections among themselves, but those belonging to diﬀerent patterns developed strong negative ones. Minicolumn neurons belonging to the same hypercolumn had similar average connection strength proﬁles, so Figures 4.8B and 4.8C would look the same for a single minicolumn within a hypercolumn as they do here for neurons belonging to a single pattern across all hypercolumns.

During memory recall, the network spontaneously recalled stored temporal sequences in the form of temporally coactive patterns (Figure 4.9A). Sorting the rastergram by pattern revealed clear 500 ms periods of attractor state activity (Figure 4.9B). Network dynamics wandered from attractor to attractor, alternating in a seemingly random fashion, while neurons outside the currently active attractor were silenced due to lateral inhibition (Figure 4.9C). A non-coding ground state spontaneously emerged where individual active states were transiently inter- rupted by lower rate (∼2 Hz) volatile periods in which neurons competed via WTA, thwarting any obvious selection of a winner. Attractor demise was precipitated by a combination of inhibition from the network together with neural and synap- 50 CHAPTER 4. RESULTS AND DISCUSSION

Figure 4.7: Applying temporally interleaved stimuli to a spiking network. A Raster- gram (10x downsampled, henceforth) and superimposed ﬁring rates (10 ms bins averaged per pattern, henceforth) associated with the ﬁrst training epoch. B Pro- gression of the "print-now" κ signal whose brief activations are synchronized with the incoming stimuli from A. Adopted from Figure 3 of [8].

tic fatigue, which consisted of an enhanced spike-triggered adaptation current Iw (Figure 4.9D, see Equation 3.11) and short-term depression that depleted available synaptic resources (Figure 4.9E, [17]) amid active attractor states.

A longer IPI (2 s) relative to the NMDA plasticity window width (150 ms) meant that associations between diﬀerent patterns did not develop. To demonstrate how BCPNN learning could memorize sequences, we altered the training protocol by decreasing IPIs and then reexamined network recall dynamics. We began by reduc- ing the IPI to 0 ms (Figure 4.10A). Salient stimuli were always being encountered by the network in this case, so κ = 1 throughout. This enabled an evolution of AMP A Iβj as in Figure 4.8A and gij trajectories as in Figure 4.8B, except for the AMP A two neighboring patterns whose slightly higher gij arose from short temporal overlaps with τ AMP A and τ AMP A. The otherwise similar learning outcomes can zi zj be appreciated by the fact that neurons were exposed to identical stimuli while κ = 1. Stimulus gaps previously encountered while κ = 0 (Figure 4.7B), which would syn have led to gij and Iβj decay if κ was nonzero, were essentially never learned.

However, the lower IPI did lead to a dispersion of higher average NMDA connection strengths that deviated markedly from Figure 4.8C (Figure 4.10B). Filtering by the BCPNN traces reﬂected relative stimulus presentation times, which created a discrepancy in the amount of overlap in the underlying running averages 4.2. PAPER 2: BAYESIAN-HEBBIAN LEARNING IN AN ATTRACTOR NETWORK 51

Figure 4.8: Evolving dynamical phenomena during stimulus learning. A Develop- ment of Iβj during training and averaged over 50 randomly selected neurons per pattern. The period of time shown in Figure 4.7A-B can be discerned as * for refer- AMP A ence. B Development of average gij during training that project from neurons belonging to the ﬁrst stimulated (i.e. red) pattern, with colors denoting target NMDA postsynaptic neuron pattern. C Same as (B), except showing gij development during training. Adopted from Figure 3 of [8].

of coactivity traces Pij (Equation 3.9) when the IPI was reduced. Reﬂecting the temporal disparity between t versus τ NMDA, the modiﬁed training procedure stim zi allowed the network to learn stronger feedforward associations than recurrent ones and produced an asymmetrical terminal structure (Figure 4.10C). NMDA synapses connecting excitatory cells in subsequently presented patterns formed progressively decreasing forward associations. Taken from the other perspective, trailing inhibition was learned in the form of negative-valued towards antecedent pattern cells.

The resulting weight conﬁguration reorganized the network dynamics, generating reliable sequential transitioning between attractor states (4.11A-C). Sequential state traversal stably and perpetually repeated, though including longer temporal gaps between entire epochs rather than individual patterns could alleviate such 52 CHAPTER 4. RESULTS AND DISCUSSION

Figure 4.9: Replaying spontaneously wandering attractors after learning. A Raster- gram snapshot of excitatory neurons during recall. B Relative average firing rates based on (A) and sorted by attractor membership. Each row represents the firing rate of one attractor. C Average firing rate of the attractors displaying the random progression of the network through state space. Arrows highlight the ground state, which are competitive phases lacking a dominantly active attractor. D Evolution of the adaptation current Iw for a single neuron whose activity builds up and at- tenuates as it enters and exits attractor states. E Evolution of the same neuron’s dynamic AMPA strength due to short-term depression, whose postsynaptic target resides within the same attractor. Adopted from Figure 3 of [8].

cyclic behavior. Given the same training protocol as depicted in 4.11A, switching the learning window widths and generated a mirror image of the proﬁle from 4.11C and led to reverse recall of the stored patterns (4.11D).

In networks trained using longer IPIs, the probability of transitioning from one attractor to another seemed uniform (Figure 4.9A-C). However, attractor transitions became more deterministic as IPIs were decreased to 0 ms (Figure 4.11A-C), thus there were diﬀerential eﬀects of the training interval on neural recall dynamics. 4.2. PAPER 2: BAYESIAN-HEBBIAN LEARNING IN AN ATTRACTOR NETWORK 53

Figure 4.10: Applying temporally consecutive stimuli to a spiking network. A Rastergram and firing rates associated with the first out of 50 epochs of training. AMP A B Average gij during training emanating from neurons belonging to the first stimulated pattern as in Figure 4.8B (indicated with red arrow, colors denote target postsynaptic neuron pattern). Contrast the weight trajectories between patterns NMDA with Figure 4.8C. C Average gij after training that depicts an asymmetrical terminal weight profile. As in (B), the red arrow indicates the presynaptic perspective taken from the first stimulated pattern, which is aligned here at index 4. Adopted from Figure 4 of [8].

Since the difference in recall dynamics between the two setups presented here could NMDA be mostly ascribed to the makeup of gij between neurons from different pat- AMP A terns (e.g. rather than gij or Iβj ), we expressed the learned model networks NMDA at different temporal intervals using their terminal gij profiles (Figure 4.12A).

To characterize the transformation from random to nonrandom temporal sequences 54 CHAPTER 4. RESULTS AND DISCUSSION

Figure 4.11: Replaying sequential attractors after learning. A Rastergram snapshot of excitatory neurons during recall. B Relative average ﬁring rates based on (A) and sorted by attractor membership as in Figure 4.9B. The sequence is chronologically ordered according to the trained patterns from 4.10A. C Average ﬁring rate of attractors displaying the sequential progression of the network through state space. D Resulting recall after training the network by exchanging τ NMDA for τ NMDA zi zj showing the reverse traversal of attractor states from (A-C). Firing rates here and in (B) are coded according to the colorbar from Figure 4.9B. Adopted from Figure 4 of [8].

depending on these resulting weight proﬁles, we measured the distribution of attractor transitions for diﬀerent IPIs using Conditional Response Probability (CRP) curves. Shortening the IPI led to narrower CRP curves that peaked at lag 1 (Figure 4.12B), indicating this manipulation decreased recall heterogeneity and made forward transitions more likely. Networks trained with longer IPIs instead displayed CRP curves that hovered around chance levels. 4.2. PAPER 2: BAYESIAN-HEBBIAN LEARNING IN AN ATTRACTOR NETWORK 55

Figure 4.12: The temporal structure of neural recall dynamics reflects the temporal NMDA interval used during training. A Average gij after training as in Figure 4.10C (reproduced here by the 0 ms line) except now depicting terminal weight profiles for many differently trained networks with IPIs varying between 0 and 2000 ms. B CRP curves calculated for networks with representative IPIs = 0, 500, 1000, 1500 and 2000 ms after 1 minute of recall, with colors corresponding to (A). Increasing IPIs flattened the CRP curve, promoting attractor transition distribution evenness. Error bars reflect standard deviations. Adopted from Figure 5 of [8].

As the network with IPI = 0 ms proceeded through sequential states, individual neurons not actively participating in attractors were mostly quiescent, being suppressed through lateral inhibition. Previously referred to as the background state [62], single cells displayed characteristically hyperpolarized Vm (Figure 4.13A, see Equation 2.4) until recruited into an attractor, where they then received an excitatory push through delayed activation of the NMDA receptor (4.13B, Equation 2.4). Companion neurons then followed, and there was a period of mutual excitation that stabilized attractor activity via the AMPA receptor (4.13C, Equation 2.4). Once reaching fruition, the attractor immediately began to perish due to ongoing cellular adaptation and short-term depression that ensured its neurons would even- 56 CHAPTER 4. RESULTS AND DISCUSSION tually lose their competition against neurons of opposing attractors due to GABA receptor activation (Figure 4.13D, Equation 2.4).

Figure 4.13: Upkeep of stable sequential state switching during recall is permitted by the interplay of excitation and inhibition in a single neuron. A Recorded Vm of a randomly chosen single cell whose period of increased tendency for being suprathreshold coincided with its brief engagement within an attractor. The red dotted line represents the membrane voltage threshold Vt for reference, and the shaded area represents a detected attractor state that is reproduced in B-D. B The same cell’s net NMDA current, which combined positive and negative afferent BCPNN weights. C The same cell’s net AMPA current, which combined positive and negative aﬀerent BCPNN weights. D The same cell’s GABA current originating from local basket cell inhibition. Adopted from Figure 6 of [8].

To analyze which factors played a role in determining the relative lifetimes of sequential attractors, we began by taking the perspective that the speed at which sequences were learned was a product of the speed at which conditioning stimuli 4.2. PAPER 2: BAYESIAN-HEBBIAN LEARNING IN AN ATTRACTOR NETWORK 57 were presented to the network. Given its functional importance for sequence development in this model (see Figure 4.12), the strength of terminal NMDA weights are shown for different stimulus durations (Figure 4.14A). As tstim was decreased, the shape of the terminal NMDA weight distribution flattened out, leading to an increased level of overall NMDA excitation and therefore faster network recall speeds (Figure 4.12B). We also measured the strength of terminal NMDA weights by varying the number of training epochs while tstim = 100 ms. Terminal weights could evolve to a relatively stable distribution of values after only 10 epochs, lowering the possibility that observed speed effects were localized only to the 50 epoch condition.

The speed with which attractors were recalled ranged from 3 to 7 attractors per second due to changes in tstim. But recall speeds were mostly dilated, displaying a compression factor < 1. This meant tstim changes (i.e. trained speed changes) alone did not suﬃce to keep an even pace with the recall speed, let alone account for compression factors as high as those from experimental measurements [12,13,43].

Since experiments demonstrating temporal compression similarly elicited ﬁring activity at longer durations [166, 167, 168], we analyzed the extent to which the network could be maximally compressed and dilated for a pattern duration of tstim = 500 ms (i.e. a trained speed of 2 patterns per second). Compression factors for each calibrated parameter at this trained speed are summarized by Figure 4.15.

To further illustrate its robustness and functional ﬂexibility, the network was trained using sequentially overlapping temporal stimulus patterns. Two separate training protocol were designed that were motivated by experimental paradigms [169, 170] and adhered to generic sequential memory recall tasks [171]. For the following protocol, we reverted to a training conﬁguration involving interposed delays between trials, in other words training blocks with IPI = 0 ms were separated by 2000 ms periods of quiescence. Each protocol consisted of two unique subsequences of the complete sequence pattern alternatingly presented 50 times each.

In the first test, alternating stimulus patterns consisted of two subsequences where the last pattern of the first subsequence matched the first pattern of the second subsequence (Figure 4.16A). The network should be able to bind the transitively associated discontiguous subsequences together and perform cue-triggered recall in its entirety despite the different degrees of temporal proximity between subsequences [170]. After training, the resulting NMDA weight matrix exhibited pronounced asymmetry (Figure 4.16B) as in Figure 4.10C. But nonspecific negative connections developed between all of the non-overlapping patterns such that the network could only rely upon the excitatory connections projecting to and from a single pattern in order to bridge the two subsequences together. Nevertheless, the network could overcome these limitations and successfully recall the entire sequence (Figure 4.16C). 58 CHAPTER 4. RESULTS AND DISCUSSION

Figure 4.14: Dependence of temporal sequence speed on the duration of condition- NMDA ing stimuli. A Average gij after training as in Figure 4.10C (reproduced here by the 100 ms line). B Speeds computed for the networks from (A) after 1 minute of recall. The dotted gray line represents a linear relationship between training and recall speeds (compression factor = 1), tstim used previously can be discerned as * for reference, "attrs" abbreviates attractors and the shaded areas denotes the standard deviation. Adopted from Figure 8 of [8].

In the second test, stimuli consisted of two alternating subsequences whose middle two patterns were shared between both subsequences (Figure 4.16D). The ﬁrst two unique patterns of each subsequence were followed by an intersection where the next two patterns were shared by both subsequences; this was followed by a "fork" that divided the subsequences again into two separate branches comprising their last two unique patterns [169].

The resulting NMDA weight matrix after training showed that the connections projecting from the overlapping patterns towards the patterns belonging to each fork were identical (Figure 4.16E). But the unique subsequence patterns occurring be- 4.2. PAPER 2: BAYESIAN-HEBBIAN LEARNING IN AN ATTRACTOR NETWORK 59

Figure 4.15: Characterizing speed changes based on intrinsic network parameters. Compression factor ranges measured for trained speeds of 2 patterns/second achieved by altering one parameter at a time and keeping all others constant. Hor- izontal bars denote compression factor cutoﬀs that were maximally allowable without violating edit distance tolerance levels, and the gray dotted line indicates a compression factor of 1. A second value does not exist for training speed since it was held at 2 patterns/second. Adopted from Figure 10 of [8]. fore the overlapping patterns selectively developed connectivity preferences towards the last two unique patterns of their associated fork branch, and formed negative projections towards patterns belonging to their opposing fork branch. These preferences, enabled by temporal extent of τ NMDA, allowed the network to successfully zi recall each individual subsequence when presented with its respective cue (Figure 4.16F). 60 CHAPTER 4. RESULTS AND DISCUSSION

Figure 4.16: Cue-triggered overlapping sequence completion and disambiguation. A Schematic of sequential training patterns used to demonstrate completion. Roman numerals label uniquely trained subsequences that were alternatingly presented to the network, and the gray box highlights overlapping subsequence patterns. B NMDA Terminal average gij matrix resulting from (A). White Roman numerals identify regions of the weight matrix corresponding to learned connections between the non-overlapping subsequences of (A), which were reciprocally inhibiting. Black Roman numerals identify the crucial associations used for bridging the two subsequences together. C A cue (red star) presented to the ﬁrst pattern 1 second into recall resonates through the network. D Schematic of the training pattern as in (A) demonstrating the problem of sequence disambiguation. E Terminal average NMDA gij matrix resulting from (D) and white Roman numerals as in (B). Here, black Roman numerals identify the crucial associations for bridging each individual subsequence together despite their shared patterns, which each form indistinguishable average connection strengths towards each branch of the fork as emphasized by the equivalent matrix cells contained within the dotted outlines. F Two separate cues (red and blue stars) presented 8 seconds apart each resonate through their corresponding subnetworks. Adopted from Figure 11 of [8]. 4.3. PAPER 3: SCALING UP ATTRACTOR LEARNING ON NEUROMORPHIC HARDWARE 61 4.3 Paper 3: Scaling up Attractor Learning on Neuromorphic Hardware

Next, we wanted to show that a similar network was generalizable enough to run on other computing architectures, and furthermore that these simulations could be enhanced through the use of neuromorphic hardware. In this bake-oﬀ, NEST simulations on Cray XC-30 compute nodes consisting of two 2.5 GHz Intel Ivy Bridge Xeon processors were pitted against SpiNNaker simulations, but since SpiNNaker runs at a ﬁxed-fraction of real-time, we found that 9 supercomputer nodes were required to match the run-time of the SpiNNaker simulation when NHC = 16.

We also derived approximate power usage statistics for simulations based on the 1 W peak power usage of the SpiNNaker chip and the 30 kW power usage of a Cray XC-30 compute rack [172]. While these ﬁgures ignore the power consumed by the host computer connected to the SpiNNaker system; the power consumed by the "blower" and storage cabinets connected to the Cray XC-30; and assume that all CPUs are running at peak power usage, they show that even in the worst case, SpiNNaker uses 45x less power than the Cray XC-30 and, if the limitations of the current SpiNNaker software are addressed, this can be improved to 200x.

Figure 4.17: Comparison of power usage of modular attractor network simulations running on SpiNNaker with simulations distributed across enough compute nodes of a Cray XC-30 system to match SpiNNaker simulation time. Adopted from Table 3 of [142]. 62 CHAPTER 4. RESULTS AND DISCUSSION

4.4 Paper 4: Reward-Dependent Bayesian-Hebbian learning in a Biophysical Basal Ganglia model

In order to demonstrate the ﬂexibility of the spike-based BCPNN learning rule, we sought to replicate the results of an abstract model of the Basal Ganglia that learned to map states to actions on blocks of 200 trials in a simple reinforcement learning task [173]. This task involved the model described in Figure 2.4 with 3 states and 3 actions. During each trial one action was rewarded while all the others presented a zero-valued reward, meaning each action had one state representing a time period during which a positive reward was given. We found that the spike-based BCPNN imbued model was able to learn a correct reward mapping in simulations consisting of 15 blocks of 40 trials (Figure 4.18A).

The synaptic strengths from the striosomes to the dopaminergic population in the RP pathway learned to correctly predict the reward (Figure 4.18B). Both the RPE and weight volatility decreased over many trials. The slow decay of the D2 weights after the initial surge at the beginning of a block resulted from the small variations around baseline of the dopamine level, which also led to a gradual increase in D1 weights. Learning of the task was considered successful since the average success approaching the maximum value at the end of each block (Figure 4.18C). Thus like its abstract counterpart, the spike-based model could learn to select the correct action for each state.

To explain these dynamics, we need to revisit Figure 2.4 and dissect how different combinations of Go, NoGo and RP pathways perform during reward learning tasks. During trials, activity of the dopaminergic neurons was modulated by input from the striosomes, but the delivery of a reward as predicted by the RP pathway enacted little change in the dopaminergic neurons’ firing rates. This was because inhibition received from the specific striosomal sub-population coding for the relevant state-action pair compensated for the increased excitatory input brought by the reward on the dopaminergic neurons.

New reward mappings at the beginning of trials resulted from increased suppres- sion from the then incorrect action coding D2 MSNs, together with a subsequent decrease of promotion by the D1 population coding for that action. Decreased promotion occurred when another action had been associated with the same state. When an action-coding population learned higher D2 synaptic strengths, this led to lower D2 weights associated with other action-coding populations. A diﬀerence between the expected reward from the previous block and the new negative outcome in the current block led to higher RPEs at the beginning of new blocks, which subsequently enabled more rapid and volatile changes in D2 weights compared with D1 weights. Thus, negative RPEs essentially allowed this learning system to stop associating subsequent selection of the same action [35]. 4.4. PAPER 4: REWARD-DEPENDENT BAYESIAN-HEBBIAN LEARNING IN A BIOPHYSICAL BASAL GANGLIA MODEL 63

Figure 4.18: Learning in the D1, D2, and RP pathways of a Spike-Based BCPNN Basal Ganglia model. A The three colored lines represent the average weight of the three action coding populations in D1 and D2 from state 1, where vertical dashed gray lines denote the start of a new block. B The color coded average weights from the nine state-action pairing striosomal sub-populations to the dopaminergic population. C The moving average success ratio of the model over the simulation. Adopted from Figure 3 of [35].

Lastly, the degeneration of dopaminergic neurons in SNc as observed in Parkin- son’s disease was simulated by silencing fractions of dopaminergic neurons, which prevented them from impacting network dynamics after 8 blocks. Decreasing the number of dopaminergic neurons during learning to 16%, 33% and 66% of their original amount and deteriorated task performance was observed during each condition, indicating that restoring plasticity within cortico-striatal connections towards 64 CHAPTER 4. RESULTS AND DISCUSSION

D1 MSNs in Parkinsonian patients could help increase learning ability and action selection performance. Chapter 5

Concluding Remarks and Future Directions

5.0.1 Summary Cortical and subcortical neuronal networks stubbornly maintain their functioning despite ongoing plastic changes that give rise to a host of dynamical phenomena. Through large-scale neural simulations on supercomputing clusters and neuromorphic hardware, this work has demonstrated how an empirically-grounded probabilistic learning rule can help account for a diversity of biological processes with ranging physical and time scales. Spike-based BCPNN has helped drive hypothesis- driven and experimentally testable predictions about the mechanisms underlying cerebral cortex and basal ganglia functioning.

More generally, it demonstrated how multiple interacting plasticity mechanisms can coordinate reinforcement, auto- and hetero-associative learning within large- scale, spiking, plastic neuronal networks. Spiking neural networks are capable of representing information in the form of probability distributions, and this biophysical realization of Bayesian computation can help reconcile disparate experimental observations. The work is a rung in the ladder of computational neuroscience that provides value from both the theoretical and experimental perspectives. We review below a few potential avenues of future investigation that can be carried out beyond the results of this work.

5.0.2 Temporal Gestalten and other Perceptual Illusions of Time Acquired from Probabilistic Plasticity Illusions of time can help reveal how the brain organizes and interprets behavioral experiences. It was initially mentioned that discovering sequential neuronal activity within neocortical microcircuits could be expected given that planned and performed actions could be boiled down to step-wise processes. In support of this

65 66 CHAPTER 5. CONCLUDING REMARKS AND FUTURE DIRECTIONS idea, humans are thought to execute motor programs and perceive their experiences as delimited subsequences in order to cope with a continuously arriving stream of complex sensory information. For example, during handwriting one must recite ordered combinations of ﬁnger movements, which themselves are controlled by serial sequences of muscle group activations [174]. Such internal representations can be clustered according to temporal community structure [175] and parsed into sequences of instances of exploration and stabilization in decision-making [176].

In visual cortex, well-aligned receptive fields that mutually excite one another and increase each other’s saliency are called local association fields [177]. It is through perceptual grouping of collinear line segment orientations at the level of receptive fields (i.e. minicolumns) that observers can integrate more complex information like smooth contours at the level of local association fields (i.e. attractors). According to the incremental grouping model [178], such expanded input representations are mediated by long-range recurrent horizontal connectivity, similar to the role played by spike-based BCPNN synapses in the model presented here. Adhering to these models, the anisotropic nature of tangential intracortical connections is thought to underlie Gestalt principles of perceptual organization [179] such as similarity, or the tendency to group items that resemble each other together [180].

In the parlance of attractor memory models, pattern completion can produce the essential dynamical properties required by such grouping operations. Pattern completion can be demonstrated by stimulating of a subset of minicolumns belonging to a stored pattern, which is adequate to then activate the remaining minicolumns belonging to that pattern [5]. The model presented in this thesis could be used to explore how the connectivity of such a network could be learned with the interesting corollary that Gestalt criteria for perceptual grouping are not innate but rather the result of previously experienced stimuli [181]. In support of this idea, the model can perform a hallucinatory form of pattern completion in which the presented cue not only ignites a single stored pattern but leads to the spontaneous recall of the recently experienced temporal sequence in its entirety [168].

While usually illustrated by and applied to spatial stimuli, Gestalt phenomena and their neural origins have received surprisingly little attention in the temporal domain. The model here has been shown to simulate a temporal analog of figure- ground segmentation, or the tendency to differentiate items based on their relationship with other surrounding items, if one stimulates the network with isochronous stimuli. Akin to spatial distance in a static image presented within the visual field, introducing latency into training would allow arriving events to stand out and create the impression that these interruptions should be differentially perceived against the surrounding silence. This effect is more easily invoked in the auditory context, where the incoming stimuli are sounds rather than pictures [182].

On the cognitive level, several illusory phenomena exist in which timing-independent 67 stimulus properties can create a distorted judgment of recalled event duration, including relative magnitude [183], visibility [184], emotional content [185], predictability [186], and granted level of attention [187]. While such high level psychophysical eﬀects undoubtedly involve mechanisms beyond what can be accounted for by this simple cortical microcircuit, if they result from the dynamic patterns produced by networks of neurons, then model simulations can help make predica- tions regarding the neural factors that may play a role in forming these perceptual experiences.

In motivating the ability of the network to learn various pattern constellations, other temporal Gestalten could also be constructed. Future work could focus on implementing the Gestalt grouping laws of proximity and connectedness. Proxim- ity, or the tendency to perceive stimulus elements placed closely together as one group, might consist of the following stimulus conﬁguration: ’AB–CD’. One would then expect the network to recall items that are temporally close to one another. For example, if ’A’ was presented to the network as a cue during recall, ’B’ should closely follow, and likewise for ’D’ when ’C’ is presented. Temporal relationships between delay periods would be enhanced and therefore grouped together in the Gestalt context. Connectedness, or the tendency to group objects that share common elements together rather than as unrelated individual items, could be construed as ’AB–BC’. The network should be able to bind the transitively associated discontiguous subsequences together and perform cue-triggered recall in its entirety when presented with ’A’ during recall.

Continuity, or the tendency to group aligned objects together, could be explored in the temporal context by presenting the network with two diﬀerent patterns e.g. ABC and DBE. Since the middle element B is present in both patterns, there is ambiguity in how the network should perform sequential pattern completion after replaying B when presented with an A or D cue. Sequential pattern rivalry could also be simulated by initiating cue-triggered recall, but presenting another cue to the network during pattern completion. We would expect the sequence trajectory to be overwritten depending on if the novel stimulus can overcome the inclination to complete the sequence.

5.0.3 Natural Language Processing Word vectors are distributed word representations that have successfully been employed in several diﬃcult natural language processing (NLP) tasks. They are the new state-of-the-art in machine learning for extracting meaningful semantic information from text, the quintessential example being that they can help solve the following problem: king - man + woman = queen 68 CHAPTER 5. CONCLUDING REMARKS AND FUTURE DIRECTIONS without any a priori notion of gender or royal hierarchy. But how is it able to complete this task? Word vectors are typically trained several times through huge corpora, for example all of Wikipedia, using methods optimized for speed and storage such as continuous bag-of-words (CBOW) and Skip-gram [188].

The goal of such advanced methods involve building up an accurate and concise summary of word co-occurrences of the trained corpus, the idea being that words that occur often enough together in these huge swaths of data can already take you a long way in determining underlying semantic meaning. Given the proven power of word vectors in a variety of NLP tasks, a hypothesis to explore is that the brain might be capable of performing something vaguely similar to computing word vectors. One could develop a spiking neural network that took words as inputs and learned co-occurrences of these words using spike-based BCPNN [121]. An example below are words taken from the famous One fish, two fish, red fish, blue fish children’s poem written by Dr. Suess [189]. This corpus has lots of repeated words, which leads to richer synaptic representation of co-occurrences. This prototype was simulated used NEST [190] to encode each unique word as a 30-neuron population, providing 100 ms of stimulation to each population in order to make it spike at 20 Hz when the word was encountered in the corpus. Going word-by-word, all neurons in the network were simulated in this way. Using periods (’.’) as sentence delimiters, we provided temporal gaps in which no stimulation was present to avoid associating words between sentences. Over the course of training, and due to a long Hebbian learning window, the network was able to learn the full co-occurrence matrix of words in the form of synaptic strengths 5.1. Since this way of storing word co-occurrences does not scale so well because the entire NxN matrix scales as a square, it remains future work to encode the input using multiple synapses in order to increase the dimensionality such that output neurons are synthesized to respond to a particular spatio-temporal pattern of input spikes [191].

5.0.4 Basal Ganglia Sequences

Our approach could be extended in the context of cortico-basal ganglia circuits [173], where population activity patterns encode time in agreement with reinforcement learning theory [192], and where switching between behavioral sequences might be a beneﬁcial strategy for successful task completion [193]. Spike-based reinforcement learning [132] could be incorporated into BCPNN learning as an added level of EWMAs with an intermediate time course between the Z and P traces [121], representing downstream cellular processes that could interact with increased intracellular Ca2+ concentrations [194]. This eligibility trace would represent interactions with neuromodulatory systems, wherein the basal ganglia are a key player, and with which the emergence of sequential activity has been linked [83, 195]. 69

Figure 5.1: Spike-based BCPNN synaptic weight matrix for a Dr. Suess poem. Learning the full co-occurrence matrix of words in the form of synaptic strengths.

5.0.5 Outlook This work generalizes across spatial and temporal scales in neuroscience, and is applicable to a variety of biological and practical applications. As technology continues to improve and computational and experimental techniques allow for ﬁner grained study of dynamical phenomena, its fundamental assumptions will be tested and called into question. Whatever those ﬁnal set of answers may be though, the insights brought to light in this work represent an an important step towards uniting the disparate worlds of experimental measurement, mathematical theory and high level brain function.

Bibliography

[1] Yann LeCun, Yoshua Bengio, and Geoﬀrey Hinton. Deep learning. Nature, 521(7553):436–444, 2015. Cited on page 4.

[2] Richard Frackowiak and Henry Markram. The future of human cerebral cartography: a novel approach. Phil. Trans. R. Soc. B, 370(1668):20140171, 2015. Cited on pages 4 and 5.

[3] Nicola J Allen and Ben A Barres. Neuroscience: glia - more than just brain glue. Nature, 457(7230):675–677, 2009. Cited on page 9.

[4] Santiago Ramón y Cajal. Comparative study of the sensory areas of the human cortex. Worcester, 1899. Cited on page 9.

[5] Mikael Lundqvist, Martin Rehn, Mikael Djurfeldt, and Anders Lansner. At- tractor dynamics in a modular network model of neocortex. Network: Com- putation in Neural Systems, 17(3):253–276, 2006. Cited on pages 10, 23, 33, and 66.

[6] Jan Benda and Andreas VM Herz. A universal model for spike-frequency adaptation. Neural computation, 15(11):2523–2564, 2003. Cited on page 10.

[7] Robert H Cudmore and Gina G Turrigiano. Long-term potentiation of intrinsic excitability in lv visual cortical neurons. Journal of neurophysiology, 92(1):341–348, 2004. Cited on pages 10, 15, 42, and 43.

[8] Philip J Tully, Henrik Lindén, Matthias H Hennig, and Anders Lansner. Spike-based bayesian-hebbian learning of temporal sequences. PLoS Comput Biol, 12(5):e1004954, 2016. Cited on pages 11, 17, 33, 34, 49, 50, 51, 52, 53, 54, 55, 56, 58, 59, and 60.

[9] Tim VP Bliss, Graham L Collingridge, et al. A synaptic model of memory: long-term potentiation in the hippocampus. Nature, 361(6407):31–39, 1993. Cited on pages 11, 15, and 35.

[10] Alanna J Watt, Per Jesper Sjöström, Michael Häusser, Sacha B Nelson, and Gina G Turrigiano. A proportional but slower nmda potentiation follows

71 72 BIBLIOGRAPHY

ampa potentiation in ltp. Nature neuroscience, 7(5):518–524, 2004. Cited on page 12. [11] Donald Olding Hebb. The organisation of behaviour: a neuropsychological theory. Wiley, 1952. Cited on page 12. [12] Guo-qiang Bi and Mu-ming Poo. Synaptic modiﬁcations in cultured hippocampal neurons: dependence on spike timing, synaptic strength, and postsynaptic cell type. The Journal of neuroscience, 18(24):10464–10472, 1998. Cited on pages 13 and 43. [13] Larry F Abbott and Sacha B Nelson. Synaptic plasticity: taming the beast. Nature neuroscience, 3:1178–1183, 2000. Cited on page 13. [14] John Lisman and Nelson Spruston. Questions about stdp as a general model of synaptic plasticity. Spike-timing dependent plasticity, 26:53, 2010. Cited on page 13. [15] Mark CW Van Rossum, Guo Qiang Bi, and Gina G Turrigiano. Stable hebbian learning from spike timing-dependent plasticity. The Journal of Neuro- science, 20(23):8812–8821, 2000. Cited on pages 13, 15, and 45. [16] Matthias H Hennig. Theoretical models of synaptic short term plasticity. Neural Information Processing with Dynamical Synapses, page 5, 2015. Cited on page 13. [17] Misha V Tsodyks and Henry Markram. The neural code between neocortical pyramidal neurons depends on neurotransmitter release probability. Proceed- ings of the National Academy of Sciences, 94(2):719–723, 1997. Cited on pages 13 and 50. [18] Florian Fiebig and Anders Lansner. A spiking working memory model based on hebbian short-term potentiation. Journal of Neuroscience, 37(1):83–96, 2017. Cited on page 14. [19] Gina G Turrigiano and Sacha B Nelson. Homeostatic plasticity in the developing nervous system. Nature Reviews Neuroscience, 5(2):97–107, 2004. Cited on page 14. [20] Uwe Frey, Richard GM Morris, et al. Synaptic tagging and long-term potentiation. Nature, 385(6616):533–536, 1997. Cited on pages 15 and 37. [21] Wei Zhang and David J Linden. The other side of the engram: experience- driven changes in neuronal intrinsic excitability. Nature Reviews Neuro- science, 4(11):885–900, 2003. Cited on page 15. [22] Gaël Daoudal and Dominique Debanne. Long-term plasticity of intrinsic excitability: learning rules and mechanisms. Learning & Memory, 10(6):456– 465, 2003. Cited on pages 15, 42, and 43. BIBLIOGRAPHY 73

[23] Anders Sandberg, Anders Lansner, Karl Magnus Petersson, and Ekeberg. A bayesian attractor network with incremental learning. Network: Computation in neural systems, 13(2):179–194, 2002. Cited on pages 15 and 23.

[24] Christian Tetzlaﬀ, Christoph Kolodziejski, Irene Markelic, and Florentin Wörgötter. Time scales of memory, learning, and plasticity. Biological cyber- netics, 106(11-12):715–726, 2012. Cited on page 15.

[25] Mikael Djurfeldt, Mikael Lundqvist, Christopher Johansson, Martin Rehn, Örjan Ekeberg, and Anders Lansner. Brain-scale simulation of the neocortex on the ibm blue gene/l supercomputer. IBM Journal of Research and Development, 52(1-2):31–41, 2008. Cited on pages 16 and 27.

[26] David H Hubel and Torsten N Wiesel. Receptive ﬁelds and functional architecture of monkey striate cortex. The Journal of physiology, 195(1):215–243, 1968. Cited on pages 16 and 33.

[27] Yumiko Yoshimura, Jami LM Dantzker, and Edward M Callaway. Excitatory cortical neurons form ﬁne-scale functional networks. Nature, 433(7028):868– 873, 2005. Cited on pages 16, 23, and 31.

[28] Brice Bathellier, Lyubov Ushakova, and Simon Rumpel. Discrete neocortical dynamics predict behavioral categorization of sounds. Neuron, 76(2):435–449, 2012. Cited on pages 16, 23, and 31.

[29] Vernon B Mountcastle. The columnar organization of the neocortex. Brain, 120(4):701–722, 1997. Cited on pages 16, 31, and 33.

[30] Rodney J Douglas and Kevan AC Martin. Neuronal circuits of the neocortex. Annu. Rev. Neurosci., 27:419–451, 2004. Cited on pages 16, 33, and 34.

[31] André Parent and Lili-Naz Hazrati. Functional anatomy of the basal ganglia. i. the cortico-basal ganglia-thalamo-cortical loop. Brain Research Reviews, 20(1):91–127, 1995. Cited on pages 17, 18, and 19.

[32] Ann M Graybiel. Building action repertoires: memory and learning functions of the basal ganglia. Current opinion in neurobiology, 5(6):733–741, 1995. Cited on page 17.

[33] Garrett E Alexander, Mahlon R DeLong, and Peter L Strick. Parallel organization of functionally segregated circuits linking basal ganglia and cortex. Annual review of neuroscience, 9(1):357–381, 1986. Cited on page 17.

[34] André Parent. Extrinsic connections of the basal ganglia. Trends in neurosciences, 13(7):254–258, 1990. Cited on page 17. 74 BIBLIOGRAPHY

[35] Pierre Berthet, Mikael Lindahl, Philip Tully, Jeanette Hällgren Kotaleski, and Anders Lansner. Functional relevance of different basal ganglia pathways investigated in a spiking 1 model with reward dependent plasticity. Frontiers in neural circuits, 10:53, 2016. Cited on pages 18, 62, and 63. [36] John E Hoover and Peter L Strick. Multiple output channels in the basal ganglia. SCIENCE-NEW YORK THEN WASHINGTON-, 259:819–819, 1993. Cited on page 18. [37] Charles R Gerfen, Thomas M Engber, Lawrence C Mahan, Zvi Susel, Thomas N Chase, Frederick J Monsma Jr, and David R Sibley. D1 and d2 dopamine receptor-regulated gene expression of striatonigral and striatopall- idal neurons. Science, 250(4986):1429–1432, 1990. Cited on pages 18 and 19. [38] D James Surmeier, Jun Ding, Michelle Day, Zhongfeng Wang, and Weixing Shen. D1 and d2 dopamine-receptor modulation of striatal glutamatergic signaling in striatal medium spiny neurons. Trends in neurosciences, 30(5):228– 235, 2007. Cited on page 19. [39] John NJ Reynolds and Jeffery R Wickens. Dopamine-dependent plasticity of corticostriatal synapses. Neural Networks, 15(4):507–521, 2002. Cited on page 19. [40] P Read Montague, Peter Dayan, and Terrence J Sejnowski. A framework for mesencephalic dopamine systems based on predictive hebbian learning. The Journal of neuroscience, 16(5):1936–1947, 1996. Cited on page 19. [41] Wolfram Schultz, Peter Dayan, and P Read Montague. A neural substrate of prediction and reward. Science, 275(5306):1593–1599, 1997. Cited on pages 19 and 37. [42] Wolfram Schultz and Anthony Dickinson. Neuronal coding of prediction errors. Annual review of neuroscience, 23(1):473–500, 2000. Cited on page 19. [43] Nathaniel D Daw and Kenji Doya. The computational neurobiology of learning and reward. Current opinion in neurobiology, 16(2):199–204, 2006. Cited on page 19. [44] Louis Lapicque. Recherches quantitatives sur l’excitation électrique des nerfs traitée comme une polarisation. J. Physiol. Pathol. Gen, 9(1):620–635, 1907. Cited on page 19. [45] Romain Brette and Wulfram Gerstner. Adaptive exponential integrate-and- fire model as an effective description of neuronal activity. Journal of neurophysiology, 94(5):3637–3642, 2005. Cited on pages 20 and 33. [46] Eugene M Izhikevich. Which model to use for cortical spiking neurons? IEEE transactions on neural networks, 15(5):1063–1070, 2004. Cited on page 20. BIBLIOGRAPHY 75

[47] John J Hopﬁeld. Neural networks and physical systems with emergent collective computational abilities. Proceedings of the national academy of sciences, 79(8):2554–2558, 1982. Cited on page 21. [48] A Lansner and E Fransén. Modelling hebbian cell assemblies comprised of cortical neurons. Network: Computation in Neural Systems, 3(2):105–119, 1992. Cited on pages 21 and 23. [49] Anders Lansner. Associative memory models: from the cell-assembly theory to biophysically detailed cortex simulations. Trends in neurosciences, 32(3):178–186, 2009. Cited on pages 22 and 27.

[50] JC Eccles. The physiology of synapses. Berlin, Gottingen and Heidelberg: Springer-Verlag, 1964. Cited on page 22. [51] Erik Fransén and Anders Lansner. A model of cortical associative memory based on a horizontal network of connected columns. Network: Computation in Neural Systems, 9(2):235–264, 1998. Cited on page 23.

[52] Haim Sompolinsky and I Kanter. Temporal association in asymmetric neural networks. Physical review letters, 57(22):2861, 1986. Cited on page 23. [53] David Kleinfeld. Sequential state generation by model neural networks. Pro- ceedings of the National Academy of Sciences, 83(24):9469–9473, 1986. Cited on page 23.

[54] Markus Diesmann, Marc-Oliver Gewaltig, and Ad Aertsen. Stable propagation of synchronous spiking in cortical neural networks. Nature, 402(6761):529–533, 1999. Cited on pages 23 and 24. [55] Daniel S Reich, Ferenc Mechler, and Jonathan D Victor. Independent and redundant information in nearby cortical neurons. Science, 294(5551):2566– 2568, 2001. Cited on page 23. [56] Björn M Kampa, Johannes J Letzkus, and Greg J Stuart. Cortical feedforward networks for binding diﬀerent streams of sensory information. Nature neuroscience, 9(12):1472–1473, 2006. Cited on page 23.

[57] Rosa Cossart, Dmitriy Aronov, and Rafael Yuste. Attractor dynamics of network up states in the neocortex. Nature, 423(6937):283–288, 2003. Cited on page 23. [58] James L McClelland, David E Rumelhart, and Geoﬀrey E Hinton. The appeal of parallel distributed processing. MIT Press, Cambridge MA, pages 3–44, 1986. Cited on page 23. [59] Horace Barlow. Redundancy reduction revisited. Network: computation in neural systems, 12(3):241–253, 2001. Cited on page 23. 76 BIBLIOGRAPHY

[60] Mikael Lundqvist, Pawel Herman, and Anders Lansner. Theta and gamma power increases and alpha/beta power decreases with memory load in an attractor network model. Journal of cognitive neuroscience, 23(10):3008– 3020, 2011. Cited on pages 23 and 24. [61] Ashok Litwin-Kumar and Brent Doiron. Slow dynamics and high variability in balanced cortical networks with clustered connections. Nature neuroscience, 15(11):1498–1505, 2012. Cited on pages 23 and 24. [62] Mikael Lundqvist, Albert Compte, and Anders Lansner. Bistable, irregular firing and population oscillations in a modular attractor memory network. PLoS Comput Biol, 6(6):e1000803, 2010. Cited on pages 23 and 55. [63] Sheng-Jun Wang, Claus Hilgetag, and Changsong Zhou. Sustained activity in hierarchical modular neural networks: self-organized criticality and oscillations. Frontiers in computational neuroscience, 5:30, 2011. Cited on page 23. [64] Christopher Johansson and Anders Lansner. Towards cortex sized artificial neural systems. Neural Networks, 20(1):48–61, 2007. Cited on page 23. [65] Dileep George and Jeff Hawkins. Towards a mathematical theory of cortical micro-circuits. PLoS Comput Biol, 5(10):e1000532, 2009. Cited on page 23. [66] Andre M Bastos, W Martin Usrey, Rick A Adams, George R Mangun, Pas- cal Fries, and Karl J Friston. Canonical microcircuits for predictive coding. Neuron, 76(4):695–711, 2012. Cited on page 23. [67] Ye Li, Hui Lu, Pei-lin Cheng, Shaoyu Ge, Huatai Xu, Song-Hai Shi, and Yang Dan. Clonally related visual cortical neurons show similar stimulus feature selectivity. Nature, 486(7401):118–121, 2012. Cited on page 23. [68] Christopher D Harvey, Philip Coen, and David W Tank. Choice-specific sequences in parietal cortex during a virtual-navigation decision task. Nature, 484(7392):62–68, 2012. Cited on pages 23 and 25. [69] Alexander J Sadovsky and Jason N MacLean. Mouse visual neocortex supports multiple stereotyped patterns of microcircuit activity. The Journal of Neuroscience, 34(23):7769–7777, 2014. Cited on page 23. [70] Luis Carrillo-Reid, Fatuel Tecuapetla, Dagoberto Tapia, Arturo Hernández- Cruz, Elvira Galarraga, René Drucker-Colin, and José Bargas. Encod- ing network states by striatal cell assemblies. Journal of neurophysiology, 99(3):1435–1450, 2008. Cited on page 23. [71] Arvind Kumar, Stefan Rotter, and Ad Aertsen. Conditions for propagat- ing synchronous spiking and asynchronous firing rates in a cortical network model. The Journal of neuroscience, 28(20):5268–5280, 2008. Cited on page 23. BIBLIOGRAPHY 77

[72] Mark S Goldman. Memory without feedback in a neural network. Neuron, 61(4):621–634, 2009. Cited on page 24.

[73] Matthew I Leon and Michael N Shadlen. Representation of time by neurons in the posterior parietal cortex of the macaque. Neuron, 38(2):317–327, 2003. Cited on page 24.

[74] Aldo Genovesio, Satoshi Tsujimoto, and Steven P Wise. Feature-and order- based timing representations in the frontal cortex. Neuron, 63(2):254–266, 2009. Cited on page 24.

[75] Akihisa Mita, Hajime Mushiake, Keisetsu Shima, Yoshiya Matsuzaka, and Jun Tanji. Interval time coding by neurons in the presupplementary and supplementary motor areas. Nature neuroscience, 12(4):502–507, 2009. Cited on page 24.

[76] Hope A Johnson, Anubhuthi Goel, and Dean V Buonomano. Neural dynamics of in vitro cortical networks reﬂects experienced temporal patterns. Nature neuroscience, 13(8):917–919, 2010. Cited on page 24.

[77] Michael D Mauk and Dean V Buonomano. The neural basis of temporal processing. Annu. Rev. Neurosci., 27:307–340, 2004. Cited on page 24.

[78] Richard B Ivry and John E Schlerf. Dedicated and intrinsic models of time perception. Trends in cognitive sciences, 12(7):273–280, 2008. Cited on page 24.

[79] Tara C Thiagarajan, Mikhail A Lebedev, Miguel A Nicolelis, and Dietmar Plenz. Coherence potentials: loss-less, all-or-none network events in the cortex. PLoS Biol, 8(1):e1000278, 2010. Cited on page 24.

[80] Friedemann Pulvermüller and Yury Shtyrov. Spatiotemporal signatures of large-scale synﬁre chains for speech processing as revealed by meg. Cerebral cortex, 19(1):79–88, 2009. Cited on page 24.

[81] MW Oram, MC Wiener, R Lestienne, and BJ Richmond. Stochastic nature of precisely timed spike patterns in visual system neuronal responses. Journal of neurophysiology, 81(6):3021–3033, 1999. Cited on page 24.

[82] Moshe Abeles. Corticonics: Neural circuits of the cerebral cortex. Cambridge University Press, 1991. Cited on pages 24 and 35.

[83] Yuji Ikegaya, Gloster Aaron, Rosa Cossart, Dmitriy Aronov, Ilan Lampl, David Ferster, and Rafael Yuste. Synﬁre chains and cortical songs: temporal modules of cortical activity. Science, 304(5670):559–564, 2004. Cited on pages 24 and 68. 78 BIBLIOGRAPHY

[84] Sven Schrader, Markus Diesmann, and Abigail Morrison. A compositionality machine realized by a hierarchic architecture of synﬁre chains. Frontiers in computational neuroscience, 4:154, 2011. Cited on page 24. [85] Eugene M Izhikevich. Polychronization: computation with spikes. Neural computation, 18(2):245–282, 2006. Cited on page 24. [86] John D Rolston, Daniel A Wagenaar, and Steve M Potter. Precisely timed spatiotemporal patterns of neural activity in dissociated cortical cultures. Neuroscience, 148(1):294–303, 2007. Cited on page 24. [87] Claude ZH Wang, Joshua A Herbst, Georg B Keller, and Richard HR Hahn- loser. Rapid interhemispheric switching during vocal production in a songbird. PLoS Biol, 6(10):e250, 2008. Cited on page 24. [88] Daniel G Almeida-Filho, Vitor Lopes-dos Santos, Nivaldo AP Vasconcelos, José GV Miranda, Adriano BL Tort, and Sidarta Ribeiro. An investigation of hebbian phase sequences as assembly graphs. Frontiers in neural circuits, 8, 2014. Cited on page 24. [89] Shigeyoshi Fujisawa, Asohan Amarasingham, Matthew T Harrison, and György Buzsáki. Behavior-dependent short-term assembly dynamics in the medial prefrontal cortex. Nature neuroscience, 11(7):823–833, 2008. Cited on page 25. [90] David A Crowe, Bruno B Averbeck, and Matthew V Chafee. Rapid sequences of population activity patterns dynamically encode task-critical spatial information in parietal cortex. The Journal of Neuroscience, 30(35):11640–11653, 2010. Cited on page 25. [91] Bruno B Averbeck, Matthew V Chafee, David A Crowe, and Apostolos P Georgopoulos. Parallel processing of serial movements in prefrontal cortex. Proceedings of the National Academy of Sciences, 99(20):13172–13177, 2002. Cited on page 25. [92] Toshi Nakajima, Ryosuke Hosaka, Hajime Mushiake, and Jun Tanji. Covert representation of second-next movement in the pre-supplementary motor area of monkeys. Journal of neurophysiology, 101(4):1883–1889, 2009. Cited on page 25. [93] Maurizio Mattia, Pierpaolo Pani, Giovanni Mirabella, Stefania Costa, Paolo Del Giudice, and Stefano Ferraina. Heterogeneous attractor cell assemblies for motor planning in premotor cortex. The Journal of Neuroscience, 33(27):11155–11168, 2013. Cited on page 25. [94] Christopher C Lapish, Daniel Durstewitz, L Judson Chandler, and Jeremy K Seamans. Successful choice behavior is associated with distinct and coher- ent network states in anterior cingulate cortex. Proceedings of the National Academy of Sciences, 105(33):11963–11968, 2008. Cited on page 25. BIBLIOGRAPHY 79

[95] Lauren M Jones, Alfredo Fontanini, Brian F Sadacca, Paul Miller, and Don- ald B Katz. Natural stimuli evoke dynamic sequences of states in sensory cortical ensembles. Proceedings of the National Academy of Sciences, 104(47):18772–18777, 2007. Cited on page 25.

[96] Moshe Abeles, Hagai Bergman, Itay Gat, Isaac Meilijson, Eyal Seide- mann, Naftali Tishby, and Eilon Vaadia. Cortical activity ﬂips among quasi-stationary states. Proceedings of the National Academy of Sciences, 92(19):8616–8620, 1995. Cited on page 25.

[97] Eyal Seidemann, Isaac Meilijson, Moshe Abeles, Hagai Bergman, and Eilon Vaadia. Simultaneously recorded single units in the frontal cortex go through sequences of discrete and stable states in monkeys performing a delayed local- ization task. Journal of Neuroscience, 16(2):752–768, 1996. Cited on page 25.

[98] Henry Markram. The blue brain project. Nature Reviews Neuroscience, 7(2):153–160, 2006. Cited on page 26.

[99] Henry Markram. The human brain project. Scientiﬁc American, 306(6):50– 55, 2012. Cited on page 26.

[100] Susanne Kunkel, Maximilian Schmidt, Jochen M Eppler, Hans E Plesser, Gen Masumoto, Jun Igarashi, Shin Ishii, Tomoki Fukai, Abigail Morrison, Markus Diesmann, et al. Spiking network simulation code for petascale computers. Frontiers in neuroinformatics, 8:78, 2014. Cited on page 27.

[101] Carver Mead. Neuromorphic electronic systems. Proceedings of the IEEE, 78(10):1629–1636, 1990. Cited on page 28.

[102] Johannes Schemmel, Daniel Briiderle, Andreas Griibl, Matthias Hock, Karl- heinz Meier, and Sebastian Millner. A wafer-scale neuromorphic hardware system for large-scale neural modeling. In Proceedings of 2010 IEEE Inter- national Symposium on Circuits and Systems, pages 1947–1950. IEEE, 2010. Cited on page 28.

[103] Giacomo Indiveri, Bernabé Linares-Barranco, Tara Julia Hamilton, André Van Schaik, Ralph Etienne-Cummings, Tobi Delbruck, Shih-Chii Liu, Piotr Dudek, Philipp Häﬂiger, Sylvie Renaud, et al. Neuromorphic silicon neuron circuits. Frontiers in neuroscience, 5:73, 2011. Cited on page 28.

[104] Steve B Furber, Francesco Galluppi, Steve Temple, and Luis A Plana. The spinnaker project. Proceedings of the IEEE, 102(5):652–665, 2014. Cited on pages 28 and 38.

[105] David C Knill and Alexandre Pouget. The bayesian brain: the role of uncertainty in neural coding and computation. TRENDS in Neurosciences, 27(12):712–719, 2004. Cited on page 29. 80 BIBLIOGRAPHY

[106] Konrad P Körding and Daniel M Wolpert. Bayesian integration in sensorimotor learning. Nature, 427(6971):244–247, 2004. Cited on page 29. [107] Hadley Tassinari, Todd E Hudson, and Michael S Landy. Combining priors and noisy visual cues in a rapid pointing task. The Journal of neuroscience, 26(40):10154–10163, 2006. Cited on page 29. [108] Sophie Deneve. Bayesian spiking neurons i: inference. Neural computation, 20(1):91–117, 2008. Cited on page 29. [109] Wei Ji Ma, Jeffrey M Beck, Peter E Latham, and Alexandre Pouget. Bayesian inference with probabilistic population codes. Nature neuroscience, 9(11):1432–1438, 2006. Cited on page 29. [110] Tianming Yang and Michael N Shadlen. Probabilistic reasoning by neurons. Nature, 447(7148):1075–1080, 2007. Cited on page 29. [111] Mathieu d’Acremont, Eleonora Fornari, and Peter Bossaerts. Activity in infe- rior parietal and medial prefrontal cortex signals the accumulation of evidence in a probability learning task. PLoS Comput Biol, 9(1):e1002895, 2013. Cited on page 29. [112] Pietro Berkes, Gergő Orbán, Máté Lengyel, and József Fiser. Spontaneous cortical activity reveals hallmarks of an optimal internal model of the environment. Science, 331(6013):83–87, 2011. Cited on page 29. [113] Alireza Soltani and Xiao-Jing Wang. Synaptic computation underlying probabilistic inference. Nature neuroscience, 13(1):112–119, 2010. Cited on page 29. [114] Jean-Pascal Pfister, Peter Dayan, and Máté Lengyel. Synapses with short- term plasticity are optimal estimators of presynaptic membrane potentials. Nature neuroscience, 13(10):1271–1275, 2010. Cited on page 29. [115] Christian Keck, Cristina Savin, and Jörg Lücke. Feedforward inhibition and synaptic scaling–two sides of the same coin? PLoS Comput Biol, 8(3):e1002432, 2012. Cited on page 30. [116] Bernhard Nessler, Michael Pfeiffer, Lars Buesing, and Wolfgang Maass. Bayesian computation emerges in generic cortical microcircuits through spike- timing-dependent plasticity. PLoS Comput Biol, 9(4):e1003037, 2013. Cited on page 30. [117] Cristina Savin, Peter Dayan, and Máté Lengyel. Optimal recall from bounded metaplastic synapses: predicting functional adaptations in hippocampal area ca3. PLoS Comput Biol, 10(2):e1003489, 2014. Cited on page 30. [118] Anders Lansner and Örjan Ekeberg. A one-layer feedback artificial neural network with a bayesian learning rule. International journal of neural systems, 1(01):77–87, 1989. Cited on page 31. BIBLIOGRAPHY 81

[119] Anders Lansner and Anders Holst. A higher order bayesian neural network with spiking units. International Journal of Neural Systems, 7(02):115–128, 1996. Cited on page 31.

[120] Alan Peters and Engin Yilmaz. Neuronal organization in area 17 of cat visual cortex. Cerebral Cortex, 3(1):49–68, 1993. Cited on pages 31 and 34.

[121] Philip J Tully, Matthias H Hennig, and Anders Lansner. Synaptic and nonsynaptic plasticity approximating probabilistic inference. Frontiers in synaptic neuroscience, 6:8, 2014. Cited on pages 32, 36, 42, 44, 45, 46, 47, and 68.

[122] Thomas R Tucker and Lawrence C Katz. Recruitment of local inhibitory networks by horizontal connections in layer 2/3 of ferret visual cortex. Journal of Neurophysiology, 89(1):501–512, 2003. Cited on page 35.

[123] Ming Ren, Yumiko Yoshimura, Naoki Takada, Shoko Horibe, and Yukio Ko- matsu. Specialized inhibitory synaptic actions between nearby neocortical pyramidal neurons. Science, 316(5825):758–761, 2007. Cited on page 35.

[124] Gilad Silberberg and Henry Markram. Disynaptic inhibition between neocortical pyramidal cells mediated by martinotti cells. Neuron, 53(5):735–746, 2007. Cited on page 35.

[125] SW Roberts. Control chart tests based on geometric moving averages. Tech- nometrics, 42(1):97–101, 1959. Cited on page 35.

[126] Richard Kempter, Wulfram Gerstner, and J Leo Van Hemmen. Hebbian learning and spiking neurons. Physical Review E, 59(4):4498, 1999. Cited on page 35.

[127] Wulfram Gerstner. Time structure of the activity in neural network models. Physical review E, 51(1):738, 1995. Cited on page 35.

[128] John Lisman. A mechanism for the hebb and the anti-hebb processes underlying learning and memory. Proceedings of the National Academy of Sciences, 86(23):9574–9578, 1989. Cited on page 35.

[129] Henry Markram, Joachim Lübke, Michael Frotscher, and Bert Sakmann. Reg- ulation of synaptic eﬃcacy by coincidence of postsynaptic aps and epsps. Science, 275(5297):213–215, 1997. Cited on pages 35 and 43.

[130] A Harry Klopf. Brain function and adaptive systems: a heterostatic theory. Technical report, DTIC Document, 1972. Cited on page 35.

[131] Răzvan V Florian. Reinforcement learning through modulation of spike- timing-dependent synaptic plasticity. Neural Computation, 19(6):1468–1502, 2007. Cited on page 36. 82 BIBLIOGRAPHY

[132] Eugene M Izhikevich. Solving the distal reward problem through linkage of stdp and dopamine signaling. Cerebral cortex, 17(10):2443–2452, 2007. Cited on pages 36 and 68. [133] Verena Pawlak, Jeﬀery R Wickens, Alfredo Kirkwood, and Jason ND Kerr. Timing is not everything: Neuromodulation opens the stdp gate. Frontiers in Synaptic Neuroscience, 2, 2010. Cited on page 36. [134] K Fukunaga, Luc Stoppini, E Miyamoto, and Dominique Muller. Long-term potentiation is associated with an increased activity of ca2+/calmodulin- dependent protein kinase ii. Journal of Biological Chemistry, 268(11):7863– 7867, 1993. Cited on page 37. [135] Peter V Nguyen, Ted Abel, and Eric R Kandel. Requirement of a critical period of transcription for induction of a late phase of ltp. Science, 265(5175):1104–1107, 1994. Cited on page 37. [136] Wickliﬀe C Abraham. How long will long-term potentiation last? Philo- sophical Transactions of the Royal Society of London B: Biological Sciences, 358(1432):735–744, 2003. Cited on page 37. [137] Marc-Oliver Gewaltig and Markus Diesmann. Nest (neural simulation tool). Scholarpedia, 2(4):1430, 2007. Cited on page 38. [138] Abigail Morrison, Sirko Straube, Hans Ekkehard Plesser, and Markus Dies- mann. Exact subthreshold integration with continuous spike times in discrete- time neural network simulations. Neural computation, 19(1):47–79, 2007. Cited on page 38. [139] Andrew Davison, Daniel Brüderle, Jens Kremkow, Eilif Muller, Dejan Pecevski, Laurent Perrinet, and Pierre Yger. Pynn: a common interface for neuronal network simulators. Frontiers in Neuroinformatics, 2009. Cited on page 38. [140] Abigail Morrison, Carsten Mehring, Theo Geisel, AD Aertsen, and Markus Diesmann. Advancing the boundaries of high-connectivity network simulation with distributed computing. Neural computation, 17(8):1776–1801, 2005. Cited on page 39. [141] Susanne Kunkel, Tobias C Potjans, Jochen Martin Eppler, Hans Ekke- hard Ekkehard Plesser, Abigail Morrison, and Markus Diesmann. Meeting the memory challenges of brain-scale network simulation. Frontiers in neuroinformatics, 5:35, 2012. Cited on page 39. [142] James C Knight, Philip J Tully, Bernhard A Kaplan, Anders Lansner, and Steve B Furber. Large-scale simulations of plastic neural networks on neuromorphic hardware. Frontiers in neuroanatomy, 10, 2016. Cited on pages 40 and 61. BIBLIOGRAPHY 83

[143] Steve B Furber, David R Lester, Luis A Plana, Jim D Garside, Eustace Painkras, Steve Temple, and Andrew D Brown. Overview of the spinnaker system architecture. IEEE Transactions on Computers, 62(12):2454–2467, 2013. Cited on page 39.

[144] Xin Jin, Mikel Lujan, Luis A Plana, Sergio Davies, Steve Temple, and Steve Furber. Modeling spiking neural networks on spinnaker. Computing in Sci- ence & Engineering, 12(5):91–97, 2010. Cited on page 39.

[145] Peter U Diehl and Matthew Cook. Eﬃcient implementation of stdp rules on spinnaker neuromorphic hardware. In 2014 International Joint Conference on Neural Networks (IJCNN), pages 4288–4295. IEEE, 2014. Cited on page 39.

[146] Francesco Galluppi, Xavier Lagorce, Evangelos Stromatias, Michael Pfeiﬀer, Luis A Plana, Steve B Furber, and Ryad B Benosman. A framework for plasticity implementation on the spinnaker neural architecture. Frontiers in neuroscience, 8:429, 2015. Cited on page 39.

[147] Abigail Morrison, Ad Aertsen, and Markus Diesmann. Spike-timing- dependent plasticity in balanced random networks. Neural computation, 19(6):1437–1467, 2007. Cited on page 39.

[148] Christof Koch. Biophysics of computation: information processing in single neurons. Oxford university press, 2004. Cited on page 41.

[149] Alain Destexhe, Michael Rudolph, J-M Fellous, and Terrence J Sejnowski. Fluctuating synaptic conductances recreate in vivo-like activity in neocortical neurons. Neuroscience, 107(1):13–24, 2001. Cited on page 41.

[150] Nicolas Fourcaud-Trocmé, David Hansel, Carl Van Vreeswijk, and Nicolas Brunel. How spike generation mechanisms determine the neuronal response to ﬂuctuating inputs. The Journal of neuroscience, 23(37):11628–11640, 2003. Cited on page 41.

[151] Alexander Rauch, Giancarlo La Camera, Hans-Rudolf Lüscher, Walter Senn, and Stefano Fusi. Neocortical pyramidal cells respond as integrate-and- ﬁre neurons to in vivo–like input currents. Journal of neurophysiology, 90(3):1598–1612, 2003. Cited on page 41.

[152] Dax A Hoﬀman, Jeﬀrey C Magee, Costa M Colbert, and Daniel Johnston. K channel regulation of signal propagation in dendrites of hippocampal pyramidal neurons. Nature, 390:199, 1997. Cited on page 42.

[153] Sung-Cherl Jung and Dax A Hoﬀman. Biphasic somatic a-type k+ channel downregulation mediates intrinsic plasticity in hippocampal ca1 pyramidal neurons. PLoS One, 4(8):e6549, 2009. Cited on page 42. 84 BIBLIOGRAPHY

[154] Marcus E Petersson, Motoharu Yoshida, and Erik A Fransén. Low-frequency summation of synaptically activated transient receptor potential channel- mediated depolarizations. European Journal of Neuroscience, 34(4):578–593, 2011. Cited on page 42. [155] Erik Fransén, Babak Tahvildari, Alexei V Egorov, Michael E Hasselmo, and Angel A Alonso. Mechanism of graded persistent cellular activity of entorhinal cortex layer v neurons. Neuron, 49(5):735–746, 2006. Cited on pages 42 and 43. [156] ESL Faber, P Sedlak, Maria Vidovic, and Pankaj Sah. Synaptic activation of transient receptor potential channels by metabotropic glutamate receptors in the lateral amygdala. Neuroscience, 137(3):781–794, 2006. Cited on page 42. [157] Claire Wyart, Simona Cocco, Laurent Bourdieu, Jean-Francois Léger, Cather- ine Herr, and Didier Chatenay. Dynamics of excitatory synaptic components in sustained firing at low rates. Journal of neurophysiology, 93(6):3370–3380, 2005. Cited on page 42. [158] Kyriaki Sidiropoulou, Fang-Min Lu, Melissa A Fowler, Rui Xiao, Christopher Phillips, Emin D Ozkan, Michael X Zhu, Francis J White, and Donald C Cooper. Dopamine modulates an mglur5-mediated depolarization underlying prefrontal persistent activity. Nature neuroscience, 12(2):190–199, 2009. Cited on page 42. [159] Alexei V Egorov, Bassam N Hamam, Erik Fransén, Michael E Hasselmo, and Angel A Alonso. Graded persistent activity in entorhinal cortex neurons. Nature, 420(6912):173–178, 2002. Cited on page 43. [160] Carlos D Aizenman and David J Linden. Rapid, synaptically driven increases in the intrinsic excitability of cerebellar deep nuclear neurons. Nature neuroscience, 3(2):109–111, 2000. Cited on page 43. [161] Katsunori Kobayashi and Mu-ming Poo. Spike train timing-dependent associative modification of hippocampal ca3 recurrent synapses by mossy fibers. Neuron, 41(3):445–454, 2004. Cited on page 44. [162] Richard Kempter, Wulfram Gerstner, and J Leo Van Hemmen. Intrinsic stabilization of output rates by spike-based hebbian learning. Neural Computation, 13(12):2709–2741, 2001. Cited on page 45. [163] Baktash Babadi and Larry F Abbott. Intrinsic stability of temporally shifted spike-timing dependent plasticity. PLoS Comput Biol, 6(11):e1000961, 2010. Cited on page 45. [164] Pascale Waelti, Anthony Dickinson, and Wolfram Schultz. Dopamine responses comply with basic assumptions of formal learning theory. Nature, 412(6842):43–48, 2001. Cited on pages 47 and 48. BIBLIOGRAPHY 85

[165] François Gonon. Prolonged and extrasynaptic excitatory action of dopamine mediated by d1 receptors in the rat striatum in vivo. The Journal of neuroscience, 17(15):5972–5978, 1997. Cited on page 47.

[166] David R Euston, Masami Tatsuno, and Bruce L McNaughton. Fast-forward playback of recent memory sequences in prefrontal cortex during sleep. science, 318(5853):1147–1150, 2007. Cited on page 57.

[167] Daoyun Ji and Matthew A Wilson. Coordinated memory replay in the visual cortex and hippocampus during sleep. Nature neuroscience, 10(1):100–107, 2007. Cited on page 57.

[168] Shengjin Xu, Wanchen Jiang, Mu-ming Poo, and Yang Dan. Activity recall in a visual cortical ensemble. Nature neuroscience, 15(3):449–455, 2012. Cited on pages 57 and 66.

[169] Paul A Lipton, John A White, and Howard Eichenbaum. Disambiguation of overlapping experiences by neurons in the medial entorhinal cortex. The Journal of neuroscience, 27(21):5787–5795, 2007. Cited on pages 57 and 58.

[170] JB Hales and JB Brewer. Activity in the hippocampus and neocortical working memory regions predicts successful associative memory for temporally discontiguous events. Neuropsychologia, 48(11):3351–3359, 2010. Cited on page 57.

[171] William B Lew. A sequence predicting ca3 is a ﬂexible associator that learns and uses context to solve hippocampal-like tasks. Hippocampus, 6:579–590, 1996. Cited on page 57.

[172] Greg Pautsch, Duncan Roweth, and Scott Schroeder. The cray xc supercomputer series: Energy-eﬃcient computing. Technical Report, 2013. Cited on page 61.

[173] Pierre Berthet, Jeanette Hellgren Kotaleski, and Anders Lansner. Action selection performance of a reconﬁgurable basal ganglia inspired model with hebbian–bayesian go-nogo connectivity. Frontiers in behavioral neuroscience, 6:65, 2012. Cited on pages 62 and 68.

[174] Paolo Viviani and Carlo Terzuolo. Trajectory determines movement dynamics. Neuroscience, 7(2):431–437, 1982. Cited on page 66.

[175] Anna C Schapiro, Timothy T Rogers, Natalia I Cordova, Nicholas B Turk- Browne, and Matthew M Botvinick. Neural representations of events arise from temporal community structure. Nature neuroscience, 16(4):486–492, 2013. Cited on page 66. 86 BIBLIOGRAPHY

[176] Martín Graziano, Pablo Polosecki, Diego Edgar Shalom, and Mariano Sig- man. Parsing a perceptual decision into a sequence of moments of thought. Frontiers in integrative neuroscience, 5:45, 2011. Cited on page 66.

[177] David J Field, Anthony Hayes, and Robert F Hess. Contour integration by the human visual system: evidence for a local association ﬁeld. Vision research, 33(2):173–193, 1993. Cited on page 66.

[178] Pieter R Roelfsema, Victor AF Lamme, and Henk Spekreijse. The implementation of visual routines. Vision research, 40(10):1385–1411, 2000. Cited on page 66.

[179] Max Wertheimer. Untersuchungen zur lehre von der gestalt. ii. Psychological Research, 4(1):301–350, 1923. Cited on page 66.

[180] Kerstin E Schmidt, Rainer Goebel, Siegrid Löwel, and Wolf Singer. The perceptual grouping criterion of colinearity is reﬂected by anisotropies of connections in the primary visual cortex. European Journal of Neuroscience, 9(5):1083–1089, 1997. Cited on page 66.

[181] Carl B Zuckerman and Irvin Rock. A reappraisal of the roles of past experience and innate organizing processes in visual perception. Psychological Bulletin, 54(4):269, 1957. Cited on page 66.

[182] Albert S Bregman. Auditory scene analysis: The perceptual organization of sound. MIT press, 1994. Cited on page 66.

[183] Bin Xuan, Daren Zhang, Sheng He, and Xiangchuan Chen. Larger stimuli are judged to last longer. Journal of vision, 7(10):2–2, 2007. Cited on page 67.

[184] Masahiko Terao, Junji Watanabe, Akihiro Yagi, and Shin’ya Nishida. Reduc- tion of stimulus visibility compresses apparent time intervals. Nature neuroscience, 11(5):541–542, 2008. Cited on page 67.

[185] Sylvie Droit-Volet and Warren H Meck. How emotions colour our perception of time. Trends in cognitive sciences, 11(12):504–513, 2007. Cited on page 67.

[186] Vani Pariyadath and David Eagleman. The eﬀect of predictability on subjec- tive duration. PloS one, 2(11):e1264, 2007. Cited on page 67.

[187] Jennifer T Coull, Franck Vidal, Bruno Nazarian, and Francoise Macar. Func- tional anatomy of the attentional modulation of time estimation. Science, 303(5663):1506–1508, 2004. Cited on page 67.

[188] Tomas Mikolov, Kai Chen, Greg Corrado, and Jeﬀrey Dean. Eﬃcient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781, 2013. Cited on page 68. BIBLIOGRAPHY 87

[189] Seuss. One fish, two fish, red fish, blue fish. HarperCollins UK, 2003. Cited on page 68. [190] Marc-Oliver Gewaltig and Markus Diesmann. Nest (neural simulation tool). Scholarpedia, 2(4):1430, 2007. Cited on page 68. [191] Jonathan Tapson, Gregory Cohen, Saeed Afshar, K Stiefel, Yossi Buskila, Runchun Wang, Tara J Hamilton, and André van Schaik. Synthesis of neural networks for spatio-temporal spike pattern recognition and processing. arXiv preprint arXiv:1304.7118, 2013. Cited on page 68. [192] Dezhe Z Jin, Naotaka Fujii, and Ann M Graybiel. Neural representation of time in cortico-basal ganglia circuits. Proceedings of the National Academy of Sciences, 106(45):19156–19161, 2009. Cited on page 68. [193] Adam Ponzi and Jeff Wickens. Sequentially switching cell assemblies in random inhibitory networks of spiking neurons in the striatum. The Journal of Neuroscience, 30(17):5894–5911, 2010. Cited on page 68. [194] Sho Yagishita, Akiko Hayashi-Takagi, Graham CR Ellis-Davies, Hidetoshi Urakubo, Shin Ishii, and Haruo Kasai. A critical time window for dopamine actions on the structural plasticity of dendritic spines. Science, 345(6204):1616–1620, 2014. Cited on page 68. [195] Jeffrey P Gavornik and Mark F Bear. Learned spatiotemporal sequence recognition and prediction in primary visual cortex. Nature neuroscience, 17(5):732, 2014. Cited on page 68.

Part II

Publications