Computational Dissection of the Basal Ganglia Functions and Dynamics in Health and Disease

Computational Dissection of the Basal Ganglia functions and dynamics in health and disease

MIKAEL LINDAHL Doctoral Thesis in Computational Biology KTH Royal Institute of Technology

Akademisk avhandling som med tillstånd av Kungliga Tekniska högskolan framlägges till offentlig granskning av teknologie doktorsexamen onsdagen den 4e Maj 2016 kl 13.00 i F3, Lindstedtsvägen 26, Kungliga Tekniska högskolan, Stockholm

ISSN-1653-5723

TRITA-CSC-A-2016:07

ISRN-KTH/CSC/A—16/7-SE

ISBN 978-91-7595-925-2

Abstract

The basal ganglia (BG), a group of nuclei in the forebrain of all vertebrates, are important for behavioral selection. BG receive contextual input from most cortical areas as well as from parts of the thalamus, and provide output to brain systems that are involved in the generation of behavior, i.e. the thalamus and the brain stem. Many neurological disorders such as Parkinson’s disease and Huntington’s disease, and several neuropsychiatric disorders, are related to BG. Studying BG enhances the understanding as to how behaviors are learned and modified. These insights can be used to improve treatments for several BG disorders, and to develop brain- inspired algorithms for solving special information-processing tasks. In this thesis modeling and simulations have been used to investigate function and dynamics of BG. In the first project a model was developed to explore a new hypothesis about how conflicts between competing actions are resolved in BG. It was proposed that a subsystem named the arbitration system, composed of the subthalamic nucleus (STN), pedunculopontine nucleus (PPN), the brain stem, central medial nucleus of thalamus (CM), globus pallidus interna (GPi) and globus pallidus externa (GPe), resolve basic conflicts between alternative motor programs. On top of the arbitration system there is a second subsystem named the extension systems, which involves the direct and indirect pathway of the striatum. This system can modify the output of the arbitration system to bias action selection towards outcomes dependent on contextual information. In the second project a model framework was developed in two steps, with the aim to gain a deeper understanding of how synapse dynamics, connectivity and neural excitability in the BG relate to function and dynamics in health and disease. First a spiking model of STN, GPe and substantia nigra pars reticulata (SNr), with emulated inputs from striatal medium spiny neurons (MSNs) and the cortex, was built and used to study how synaptic short-term plasticity affected action selection signaling in the direct-, hyperdirect- and indirect pathways. It was found that the functional consequences of facilitatory synapses onto SNr neurons are substantial, and only a few presynaptic MSNs can suppress postsynaptic SNr neurons. The model also predicted that STN signaling in SNr is mainly effective in a transient manner. The model was later extended with a striatal network, containing MSNs and fast spiking interneurons (FSNs), and modified to represent GPe with two types of neurons: type I, which projects downstream in BG, and type A, which have a back-projection to striatum. Furthermore, dopamine depletion dependent modification of connectivity and neuron excitability were added to the model. Using this extended BG model, it was found that FSNs and GPe type A neurons controlled excitability of striatal neurons during low cortical drive, whereas MSN collaterals have a greater impact at higher cortical drive. The indirect pathway increased the dynamical range over which two possible action commands were competing, while removing intrastriatal inhibition deteriorated action selection capabilities. Dopamine-depletion induced effects on spike synchronization and oscillations in the BG were also investigated here. For the final project, an abstract spiking BG model which included a hypothesized control of the reward signaling dopamine system was developed. This model incorporated dopamine- dependent synaptic plasticity, and used a plasticity rule based on probabilistic inference called Bayesian Confidence Propagation Neural Network (BCPNN). In this paradigm synaptic connections were controlled by gathering statistics about neural input and output activity. Synaptic weights were inferred using Bayes’ rule to estimate the confidence of future observations from the input. The model exhibits successful performance, measured as a moving average of correct selected actions, in a multiple-choice learning task with a changeable reward schedule. Furthermore, the model predicts a decreased performance upon dopamine lesioning, and suggests that removing the indirect pathway may disrupt learning in profound ways.

Svensk sammanfattning

Basala ganglierna (BG) är en grupp nervcellskärnor i storhjärnan hos ryggradsdjur som är viktiga för beteendeselektion. BG får information från de flesta delarna av hjärnbarken samt från talamus, och genererar i sin tur en utsignal till delar av talamus och hjärnstam som är involverade i initiering av beteenden. Många neurologiska sjukdomar, såsom Parkinsons sjukdom, Huntingtons sjukdom och flera neuropsykiatriska tillstånd, är relaterade till BG. Genom att studera BG kan man uppnå en djupare förståelse för hur beteenden lärs in och modifieras. Dessa insikter kan användas till att både förbättra behandlingar av BG-relaterade sjukdomar samt för att utveckla hjärninspirerade beräkningsalgoritmer. I denna avhandling har modeller och simuleringar använts för att undersöka olika aspekter av BG. I det första delprojektet byggdes en modell för att testa en ny hypotes som handlar om hur selektion görs i BG då man har konkurrerande beteenden att välja mellan. Ett delsystem, ”arbitration system”, som består av subthalamiska kärnan (STN), pedunculopontina kärnan (PPN), hjärnstammen, centrala mediala kärnan i talamus (CM), globus pallidus interna (GPi) and globus pallidus externa (GPe), kan göra enkla val mellan olika motorprogram. Ovanpå detta ”arbitration system”, finns ytterligare ett system som kallas ”extension system”, som involverar den direkta och indirekta vägen från striatum. Detta system kan modifiera resultatet från ”arbitration system” utifrån kontextuell information från hjärnbarken. I det andra delprojektet utvecklades en modell i två steg med syftet att skapa en djupare förståelse för hur synapsdynamik, nervcellskopplingar och neural retbarhet i BG relaterar till funktion och dynamik. Först utvecklades en spikande delmodell som bestod av STN, GPe och substantia nigra pars reticulata (SNr), som i sin tur aktiverades med emulerat input från striatala medium spiny neurons (MSNs) och från hjärnbarken. Modellen användes för att studera hur beteendeselektion påverkas av synaptisk korttidsplasticitet i den s.k. direkta, indirekta eller hyperdirekta vägen genom BG. Det visade sig att de funktionella konsekvenserna av faciliterande synapser i SNr var stor, och endast ett fåtal presynaptiska MSN behövs för att hämma postsynaptiska SNr neuroner. Modellen förutspådde också att STNs signalering i SNr endast är effektiv för ett transient input. Modellen utvidgades sedan med ett striatalt nätverk, innehållande MSN och ’fast spiking interneurons’ (FSN), och GPe var nu uppdelad i två typer av neuroner; typ I, som signalerar nedströms i BG och typ A som kopplar tillbaka till striatum. Vidare introducerades dopaminberoende synapskopplingar och neuroner i modellen. Med den utvidgade BG-modellen såg vi att FSN och GPe typ A neuroner kontrollerade aktiviteten hos MSN vid svagt input från hjärnbarken, medan MSN-kollateralerna hade större inflytande vid starkt input från hjärnbarken. Den indirekta vägen genom BG var viktig för att utvidga intervallet mellan den lägsta och högsta spikfrekvens i hjärnbarkens neuroner som striatala celler kunde läsa av, medan detta intervall minskade när inhibitionen i striatum försvagades eller togs bort. Effekten av att ta bort dopamin i systemet undersöktes också med avseende på spiksynkronisering och nätverksoscillationer. I det sista delprojektet byggdes en abstrakt, spikande modell av BG som användes för att testa en hypotes om hur dopamin kan användas för inlärning av nya beteenden. Denna modell inkorporerade dopaminberoende synapsplasticitet och använde sig av en plasticitetsregel som bygger på probabilistisk inferens kallad Bayesian Confidence Propagation Neural Network (BCPNN). I detta paradigm kontrolleras styrkan av de synaptiska kopplingarna av statistik från pre- och postsynaptisk aktivitet. Vikter tas fram genom att använda Bayes regel för att ta estimera kommande observationer av pre- och postsynaptisk aktiviteter. Modellen kan framgångsrikt lära sig en flervalsuppgift med föränderligt belöningsschema. Vidare visar modellen hur prestandan går ner vid lesion av dopamin neuronerna eller då man tar bort den s.k. indirekta vägen genom BG.

Acknowledgement My PhD were supported by EU SELECT AND ACT project, Vetenskapsrådet and SerC. Stockholm brain institute have granted me generous mobility support for attending several summer and winder schools as well as conferences. I want to thank my supervisor professor Jeanette Hellgren Kotaleski for here support, encouragement and inspiration and my co-supervisor professor Örjan Ekeberg for insightful discussions. Both have made my PhD journey very pleasant. I am grateful to have work with professor Anders Lansner who I have enjoyed many interesting discussions with during lunch break. I am glad to have met Arvind Kumar and thrived from our talks about scientific and general stuff. Pawel, who I shared the master thesis room with for 6 months in the beginning and Marcus have meant a lot to me as friends and as supporting colleagues. I am also very happy to have met Stina and Martin through SBI. My fellow CB friends, who not yet mentioned, Phil, Iman, Pierre, Bernhard, Pradeep, Simon, Mikael, Peter, Anu, Jan, Ekatarina, Dinesh, Jovana, Christina, Daniel, Erik, Tony, Ramon, Ylva, Florian, Olivia and Alex. I cherish all the moments we have shared. Thank you Erik Aurell for your policy as a department head to give new PhD students a free course in improvisational theater. That lead me into a fantastic new world where I have met a lot of great friends and fellow improvisers, who I have shared a many unforgettable moments with on and off stage. I further want to acknowledge all my current colleagues and friends at Greencargo and Loredge. Thank you for making my present so enjoyable. Also, thank you Jonatan for letting me take time of and finish my PhD. Forever grateful. Thank you Magnus and Kristina for helping me with the defense festivities, you both have a special place in my heart. My friends outside work have meant the world to me and you all have helped me through my PhD with laughter and inspiration. Throughout my PhD, my family have been there for me in good and bad times. I especially want to thank my father for his inspiring energy, my mother for her emotional support and my sister for being such a funny, talented and caring person. I love you all. Finally, I thank you Sanne for your love, for your enthralling humor and for all the beautiful moments we have shared. I am yours with all my heart.

Contents 1 Introduction 7 1.1 Why study basal ganglia? ...... 7 1.2 What is a model? ...... 7 1.3 Why use computational models? ...... 7 1.4 Scope of the thesis ...... 7 1.4.1 Project I (paper 1) ...... 7 1.4.2 Project II (paper 2 and 3) ...... 8 1.4.3 Project III (paper 4) ...... 8 1.5 List of papers included in this thesis ...... 8 1.6 Contribution to papers ...... 9 2 Biological background 10 2.1 The basal ganglia ...... 10 2.1.1 Input ...... 10 2.1.2 Striatum ...... 10 2.1.3 Subthalamic nucleus ...... 11 2.1.4 Globus pallidus ...... 11 2.1.5 Substantia nigra ...... 11 2.1.6 Pedunculopontine nucleus ...... 11 2.2 Synaptic dynamics ...... 11 2.2.1 Short-term plasticity ...... 11 2.2.2 Long-term plasticity ...... 12 2.2.3 Synaptic pruning ...... 12 2.2.4 Synaptogenesis ...... 12 2.3 Dopamine...... 12 2.4 Action selection ...... 12 2.5 The direct, indirect and hyperdirect pathways ...... 13 2.6 Reinforcement learning ...... 13 3 Models and methods 14 3.1 Computational neuroscience ...... 14 3.2 Model components ...... 14 3.2.1 Hodgkin-Huxley model ...... 14 3.2.2 Integrate and fire model ...... 15 3.2.3 Adaptive exponential integrate and fire model ...... 15 3.2.4 Izhikevich simple model ...... 15 3.2.5 Conductance based synapses ...... 16 3.2.6 Tsodyks model ...... 16 3.2.7 Bayesian Confidence Propagation Neural Network ...... 17 3.3 Parameter estimation ...... 18 3.3.1 Short-term plasticity ...... 18 3.3.2 GPe neuron ...... 19 3.3.3 SNr neuron ...... 20

3.3.4 STN neuron ...... 21 3.4 Computational models of the basal ganglia ...... 21 3.4.1 Houk and Beiser 1998 ...... 21 3.4.2 Bar-Gad, Havazelet-Heimer, Goldberg, Ruppin and Bergman 2000 ...... 22 3.4.3 Terman, Rubin, Yew and Wilson 2002 ...... 22 3.4.4 Frank 2006 ...... 22 3.4.5 Lo and Wang 2006 ...... 22 3.4.6 Humphries, Stewart, Gurney 2006 ...... 22 3.4.7 Humphries, Wood and Gurney 2009 ...... 22 3.4.8 Ponzi and Wickens 2010 ...... 22 3.4.9 Steward, Bekolay and Eliasmith 2012 ...... 23 3.4.10 Nevado-Holgado, Mallet and Magill 2014 ...... 23 3.4.11 Bahuguna, Aertsen and Kumar 2015 ...... 23 4 Results and Discussion 24 4.1 The arbitration and extension system hypothesis of basal ganglia ...... 24 4.1.1 The arbitration system can select the most salient response ...... 24 4.1.2 The extension system can store conjunction and disjunction patterns ...... 26 4.1.3 The extension system can store negation ...... 26 4.2 Lessons learned through detailed spiking models of the basal ganglia ...... 26 4.2.1 Filtering effect of synaptic short-term facilitation in MSN-SNr synapses ...... 26 4.2.2 Filtering effect of depressive STN-SNr synapses in the hyperdirect pathway...... 27 4.2.3 The control of MSN excitability through MSN collaterals, FSN and GPe ...... 28 4.2.4 Dopamine dependent changes enhancing oscillations and synchrony ...... 29 4.2.5 Does basal ganglia support action selection? ...... 31 4.2.6 Synaptic and neural perturbations improving dynamics and function ...... 32 4.3 Learning action selection in basal ganglia ...... 33 4.3.1 Action selection can be learned through Bayesian inference in basal ganglia ..... 33 4.3.2 Synaptic homeostasis with Bayesian inference ...... 37 4.3.3 Relative importance in learning for MSN D1 and MSN D2 pathways ...... 37 4.3.4 Consequences of dopamine lesioning on learning ...... 37 4.3.5 RPE pathway affect learning rate ...... 38 5 Conclusions and further investigations 39 6 References 41

Introduction

1.1 Why study basal ganglia? The basal ganglia (BG) are present in all vertebrates ranging from primates, with complex behavior repertoires, down to the primitive lamprey thought to reflect the vertebrate system in early ancestors. The conservative evolution of the vertebrate BG implicates that they serve a function important to all vertebrates. BG are believed to be involved in important cognitive functions such as reward-based learning, exploratory behavior, goal-oriented behavior, motor initiation, working memory and action selection. BG receive input from a large portion of the cortex and thalamus, and project to motor centers in the thalamus and brainstem effectively influencing behavior. In addition, BG can control cortical processing, e.g. short term memory or motor preparation, through loops back to prefrontal cortical and supplementary motor areas via thalamus. Indeed BG are heavily involved in motor, associative and limbic information (Mink, 1996) and play a major role in some disorders such as Parkinson’s disease (PD), Huntington’s disease (HD), Schizophrenia, Dystonia and Hemiballismus. Thus, by studying BG a better understating of how behaviors are stored, utilized, formed and modified in all vertebrates can be gained, potentially resulting in the discovery of novel treatments for severe debilitating neurological diseases

1.2 What is a model? Modelling is a crucial tool in many scientific disciplines including neuroscience. As the complexity of a system grows, there is a need to integrate data and knowledge, and also to make simplifications and build example systems from which a deeper understanding of aspects of the complex system under study can be obtained. A model representation can have vast variations, some consisting of a single equation, some requiring page after page of equations and computer code, and some with only a descriptive presentation of a hypothetical system. Models are important tools to test hypotheses or theories. Thus, a model can have many different shapes but with a common goal; to reduce or make sense of the complexity of the system studied and help us develop hypotheses and theories that deepen our knowledge of how the system works.

1.3 Why use computational models? Computational models are used within the computational neuroscience field to help uncover principles behind the development, organization, information-processing and functional capacity of a neural system, on multiple levels of biological description and abstraction. A computational model can be used in order to get a better comprehension of brain mechanism on an explanatory level, developping information-processing algorithms where traditional approaches have not produced satisfactory results or to generate ideas of new treatments for neurological disorders.

1.4 Scope of the thesis This thesis is based on three modeling projects (four papers). In the first project, a model with focus on functionality was used to evaluate a new hypothesis about the organization of BG. The second project, and the main body of the work, focused on broadening our understanding of how different components of the BG network, such as synaptic dynamics, connectivity patterns, neuron types and dopamine levels, affect function and network dynamics. In the third project, a spiking network model with a novel biological plasticity rule was used to study how connectivity patterns in BG can be learned.

1.4.1 Project I (paper 1) In this project we took a fresh and brave look at the most recent findings on BG organization and came up with a novel hypothesis on how different subsystems in BG come together to produce action selection. The two-pathway model, with the direct- and indirect pathways, have dominated BG research over the last 20 years. Although this model captures many important aspects of BG information processing, there is room for improvement since many aspects of BG organization have not yet been incorporated into a functional framework. Here, an idea was developed that the arbitration system, composed of subthalamic nucleus (STN), globus pallidus interna (GPi), globus pallidus externa (GPe), pendunculupontine nucleus (PPN) and central

7 medial (CM) nucleus of the thalamus, constitutes a winner-take-all network in which the strongest input from the brainstem or thalamus is selected in the output nuclei, here GPi, and that the extension system is made up of the direct and indirect pathway from striatum which in turn can utilized cortical contextual information to modify the output from the arbitration system.

1.4.2 Project II (paper 2 and 3) The ambition here was to build a detailed spiking model that could push the boundary for our understanding of how different parts of BG physiology contribute to dynamics and function in healthy and pathological brain states. It was an ambitious goal and the project had to be divided into two subprojects. The first part of the project (paper 2) focused on developing a model of GPe), STN and substantia nigra pars reticulata (SNr). Great effort was made to make sure the model complied with latest BG findings. In the second part (paper 3), the model was extended with the striatum, GPe type A and I neurons, and also dopamine modulation effects were included. In the first part of the project, action selection in BG was explored with a focus on the role of dynamic synapses originating in striatum GPe and STN. Transient strengthening or weakening of synapses through short-term plasticity on a time scale of hundreds of milliseconds can significantly affect functional signaling between neural groups in the brain. This short-term plasticity is a common feature in several parts of BG, but still, prior to this project, models had yet not incorporated it. Her, we could show how SNr cells become particularly responsive to a burst in striatal cells due to facilitation in striato-nigral connections and depression in pallido- nigral connections. Furthermore, the standard model suggests that the indirect pathway controls SNr activity with striatal cells disinhibiting the STN via GPe. Our model rather indicates that SNr activity is controlled more directly by the GPe projections to SNr. This is partly because GPe cells strongly inhibits SNr cells, but also due to depressing STN to SNr synapses. In the second part of the project we broadened the scope to investigate not only action selection but also oscillatory dynamics in both healthy and pathological brain states. The extended model predicts that local inhibition in striatum and the indirect pathway are important for BG to function properly over a larger range of cortical drive. The changes of AMPA efficacy in corticostriatal synapses, the reduction of connectivity between MSNs and the decreased excitability of GPe neurons were found to be the main contributors to the facilitation of oscillations seen in PD. Finally, we found multiple changes of synaptic efficacy and neural excitability which could restore action selection ability and at the same time reduce the oscillations. Identification of such targets could potentially generate new ideas for treatment of Parkinson’s disease.

1.4.3 Project III (paper 4) In this project we started to explore how action selection can be learned in BG. The brain enables animals to behaviourally adapt in order to survive in a complex and dynamic environment, but how reward-oriented behaviours are achieved by its underlying neural circuitry is an open computational question. We addressed this concern by developing a spiking model of BG utilizing a novel probabilistic synaptic model that learns to dis-inhibit the action leading to a reward despite ongoing changes in the reward schedule. The paper demonstrates how a learning rule based on probabilistic inference in a spiking neural network model can be used for long-term synaptic plasticity by the successful performance of the model in a multiple- choice learning task with a changeable reward schedule. Furthermore, it shows how the severity of dopamine depletion correlates with decrease in model performance and that removing the indirect pathway is far more debilitating than removing the direct pathway, although removing the direct pathway still results in a significant performance drop.

1.5 List of papers included in this thesis Paper 1: Kamali Sarvestani I, Lindahl M, Hellgren-Kotaleski J, Ekeberg Ö (2011) The arbitration-extension hypothesis: a hierarchical interpretation of the functional organization of the Basal Ganglia. Front Syst Neurosci 5:13. Paper 2: Lindahl M, Kamali Sarvestani I, Ekeberg O, Kotaleski JH (2013) Signal enhancement in the output stage of the basal ganglia by synaptic short-term plasticity in the direct, indirect, and hyperdirect pathways. Front Comput Neurosci 7:76.

Paper 3: Lindahl M, Kotaleski JH. Untangling basal ganglia network dynamics and function – role of dopamine depletion and inhibition investigated in a spiking network model (Manuscript in preparation) Paper 4: Berthet P, Lindahl M, Tully P, Hellgren-Kotaleski J, Lansner A. Functional Relevance of Different Basal Ganglia Pathways Investigated in a Spiking Model with Reward Dependent Plasticity. (In revision Frontiers in neural circuits)

1.6 Contribution to papers Paper 1: Conception and design of research, interpreted results of simulations, drafted manuscript, edited and revised manuscript, approved final version of manuscript. Paper 2 and 3: Conception and design of research, performed simulations, analyzed data, interpreted results of simulations, prepared figures, drafted manuscript, edited and revised manuscript, approved final version of manuscript.

Paper 4: Conception and design of research, implemented the BCPNN model that works with dopamine volume transmission in NEST, interpreted results of simulations, drafted manuscript, edited and revised manuscript, approved final version of manuscript.

2 Biological background

2.1 The basal ganglia BG consist of several nuclei in the brain of vertebrates and are situated in the forebrain. Nuclei included are striatum consisting of the caudate, putamen and nucleus accumbens, subthalamic nucleus (STN), globus pallidus pars externa (GPe), globus pallidus pars interna (GPi), substantia nigra pars reticulata (SNr) and substantia nigra pars compacta (SNc), but also other nuclei such as the pedunculupontine nucleus (PPN) have been suggested to be part of BG. Below follows a more detailed description of each BG nucleus and the connectivity (also see Figure 1).

Figure 1 | Schematic pictures of BG nuclei location and connectivity. (A) Illustration of BG location in the brain, (B) Illustration of BG connectivity

2.1.1 Input The striatum receives excitatory input form the cortex, thalamus and amygdala but also inhibitory input from GPe. The excitatory input is topographically organized, but any given region of striatum receives overlapping input from multiple, and often related cortical areas. The input is thought to provide BG with motor planning information to execute its role in motor control (Reiner, 2010). Cortical input to STN is more restricted and mainly originates from motor areas in cortex such as the primary and supplementary motor areas (M1 and SMA) as well as frontal eye field and supplementary frontal eye field (FEF and SFEF) (Parent and Hazrati 1995). A major thalamic input to BG is the central medial nucleus (CM) and the parafascicular nucleus (PF) of the thalamus (Smith et al., 2010) that projects to both striatum and STN. The CM and PF nuclei get inputs from nuclei liable for preliminary transformation of sensory cues into motor commands. In addition to the motor command inputs from CM and PF, shared with the STN, striatum also receives incoming connections from the sensory and the associative thalamus with visual, auditory, and somatosensory association information.

2.1.2 Striatum The striatum is the main input stage of BG and comprises the caudate nucleus, putamen and nucleus accumbens. The striatum contains one principal projection neuron, the mediumspiny neuron (MSN) (Bishop et al., 1982) which makes up around 95% of the neural population (Kemp and Powell, 1971). The remaining striatal neurons are interneurons (Bishop et al., 1982) that preferably project inside striatum. Despite being relatively few they constitute a variety of morphological and neurochemically defined types. Included in these are the large aspiny neurons with acetylcholine as a synaptic transmitter and the fast spiking neurons (FSN) with GABA as a synaptic transmitter (Kawaguchi, 1993).

2.1.2.1 The direct and indirect pathways The striatal projection neurons, MSNs, are currently seen as originating from two separate but intermingled neuronal populations. The MSNs with dopamine receptor D1 targets GPi or SNr whereas the MSNs with dopamine receptor D2 projects to the GPe. These projections have been referred to as the direct and indirect pathways (Wichmann and DeLong, 1996). The direct pathway is thought to arise from GABAergic neurons that co-express substance P and/or dynorphin and the indirect pathway is believed to originate from GABAergic striatal neurons that co-express encephalin. 2.1.2.2 Matrix and striosomes/patch The striatum can also be divided along another axis: the striosomes (called patch in rat) and the surrounding extrastriosomal matrix, which are known to have different chemical markers and also have partly different connections (Gerfen et al., 1987; Graybiel, 1990; Gerfen, 1992). The function of this compartmentalization is still poorly known, but experiments in rats and monkeys have suggested differential relations of these two compartments with the SN. The matrix is believed to get inputs from sensory and motor cortical areas and to project to the SNr, whereas striosomes are thought to receive inputs from limbic cortical areas and to project to SNc and SNr (Gerfen, 1985; Gerfen et al., 1987; Lévesque and Parent, 2005; Fujiyama et al., 2015).

2.1.3 Subthalamic nucleus The STN is the primary excitatory input nucleus of BG, and it consists of medium sized glutamatergic projection neurons (Kita and Kitai, 1987). The STN receives input from cortex and thalamus and projects to GPe, GPi, SNr and PPN.

2.1.4 Globus pallidus The globus pallidus (GP) contains the medium sized GABAergic projections neurons and can be subdivided into two parts GPe and GPi. GPe sits in the center of BG like a spider in a web, whereas GPi is part of BG output (Smith et al., 1998). GPe receives excitatory input from STN, and thalamus and inhibitory input from striatum, and sends inhibitory projections to striatum, STN, neighboring GPe and SNr (Smith et al., 1998; Bolam et al., 2000). The internal segment receives excitatory input from STN and inhibitory input from GPe (Smith et al., 1998) and sends inhibitory projects to PPN, CM, some motor nuclei of ventral thalamus and several brain stem nuclei.

2.1.5 Substantia nigra The substantia nigra (SN) consists of two main cell types, SNc that use dopamine, and the SNr that use GABA as neurotransmitter (Nakanishi et al., 1997). The SN receives a majority of GABAergic input, 70%, mainly from striatum, GP and collaterals (Rinvik and Grofová, 1970). The SNc projects sends mainly dopaminergic projections to the striatum in BG, but also to other partslike GP SN and STN (Tepper, 2010). The SNr similar to GPi also projects to PPN, CM, some motor nuclei of ventral thalamus and several brain stem nuclei.

2.1.6 Pedunculopontine nucleus PPN, has traditionally not been thought of as part of BG, although it has been suggested (Mena- Segovia et al., 2004). PPN receives glutamatergic projections from STN, GABAergic input from GPi and SNr and sends mixed cholinergic/glutamatergic projections to STN (Bevan and Bolam, 1995; Sadikot and Rymar, 2009). PPN is included in BG model in the first project where a hypothesis about BG organizations in action selection is developed.

2.2 Synaptic dynamics

2.2.1 Short-term plasticity Short-term plasticity refers to facilitation or depression of synaptic efficacy from pre-synaptic spiking lasting on the timescale of hundreds to thousands of milliseconds. Synapses showing short-term plasticity are common throughout the whole brain (Creager et al., 1980; McCormick, 1989; Douglas and Martin, 1991; Nathan and Lambert, 1991; Tsodyks and Markram, 1997; Varela et al., 1997; Dittman et al., 2000; Hanson and Jaeger, 2002; Sims et al., 2008; Connelly et al., 2010). Such synapses are critical components contributing to the computational function of neural networks (Abbott and Regehr, 2004). These synapses can dynamically enhance either

11 low frequency input or high frequency input (Fortune and Rose, 2001; Abbott and Regehr, 2004) or detect a transient increase or decrease of presynaptic input (Abbott et al., 1997; Lisman, 1997; Markram et al., 1998; Puccini et al., 2007). Short-term plasticity is also prominent in BG (Hanson and Jaeger, 2002; Sims et al., 2008; Connelly et al., 2010; Gittis et al., 2010; Planert et al., 2010).

2.2.2 Long-term plasticity Long-term plasticity is viewed as the corresponding neural mechanism for experience based alteration of neural circuits. Synapses can increase in strength through long-term potentiation (LTP) or decrease in strength through long-term depression (LTD). LTP and LTD are believed to be the most important processes that enable learning and memory in the nervous system (Cooke and Bliss, 2006). Spike time dependent plasticity is a type of long-term plasticity where modulation of synaptic efficacy occurs on the basis of precise pre- and postsynaptic activation times and as other LTP or LTD forms lasts from hours up to years. With respect to BG such plasticity is prominent in the synapses between cortex and medium spiny neurons in the striatum but not understood in great detail (Reynolds and Wickens, 2000, 2002; Surmeier et al., 2007; Berretta et al., 2008; Pawlak and Kerr, 2008; Shen et al., 2008; Fino and Venance, 2011; Paille et al., 2013).

2.2.3 Synaptic pruning Synaptic pruning is a phenomenon in the mammalian brain where synapses are removed during the period between from early childhood until the onset of puberty. It is a process that is considered to reflect learning and is affected by the environment (Craik and Bialystok, 2006).

2.2.4 Synaptogenesis Synaptogenesis is the growth of synaptic connections between neurons in the nervous system. It is an ongoing process during the whole lifespan of a person, although particularly active during initial brain development (Huttenlocher and Dabholkar, 1997). It is a process that is especially important during the critical period since some connections formed during the critical period cannot be form later in life (Comery et al., 1997). Both formation of new synapses and removal of existing ones are seen throughout life, and together with LTP and LTD these processes are likely all important for learning, memory and/or adaptation in the brain.

2.3 Dopamine Dopamine has been shown to be important both as a reward signal (Schultz, 1997) and as modulator of neural excitability and synaptic efficacy (Cepeda et al., 1993; Shen and Johnson, 2000; Bracci et al., 2002; Hernández-Echeagaray et al., 2004; Hernández et al., 2006; Baufreton and Bevan, 2008; Taverna et al., 2008; Zhou et al., 2009; Humphries et al., 2009a; Chan et al., 2011; Chuhma et al., 2011; Gittis et al., 2011; Miguelez et al., 2012). Many neurological disorders stem from a disruption in the dopaminergic system. It projects heavily to striatum but also to other parts of BG. In PDthe dopaminergic neurons die which leads to a deficiency of dopamine in BG. This has major consequences for the spike dynamics in BG with increases in oscillations and spike synchronization.

2.4 Action selection BG have for a long time been suggested to be involved in action selection (DeLong, 1990; Mink, 1996; Redgrave et al., 1999). The cortex/thalamus-BG-brain stem pathway is, besides the more direct pathway from cortex to the brain stem, in a perfect position to modify motor program execution in the brain stem based on contextual input from the cortex and thalamus. Recently Kravitz et al. (2010) showed that activation of medium spiny neurons with dopamine receptor D1 promotes actions and activation of medium spiny neurons while dopamine receptor D2 inhibits actions. This strongly supports the hypothesis that BG are involved in action selection.

2.5 The direct, indirect and hyperdirect pathways In the BG system, three major pathways have been identified: the direct, the indirect and the hyperdirect pathways, respectively which converge on GPi and SNr (Nambu, 2008). The direct pathway is made up by the MSNs with dopamine receptor D1 projecting directly onto GPi and SNr. The indirect pathway consists of the MSNs with dopamine D2 receptor projecting to GPe and then via STN or directly influence GPi and SNr. The hyperdirect pathway is considered a fast pathway where STN excites GPi and SNr. The direct pathway seems to be important for promoting actions, whereas the indirect pathway seems to inhibit actions, and, finally, the hyperdirect pathway works as a transient stop signal giving striatum and GP enough time to resolve a high level of conflict in action selection (Frank, 2006; Kravitz et al., 2010).

2.6 Reinforcement learning In engineering literature three main types of learning paradigms are commonly referred to: supervised learning, reinforcement learning and unsupervised learning. In supervised learning a model is built to predict an output given an input. This is done by supplying an algorithm with example input data where the output is known. In unsupervised learning we try to find a hidden structure in unlabeled data. Thus, there is no a priori knowledge of what output to expect. In reinforcement learning an input is given, in response to which, during training, a reinforcement signal is generated to indicate whether a good decision or a bad decision was made by the model. During a training period parameters in the model are adjusted such that the model in the end ‘knows’ exactly what the correct choices given any input are. The striatum receives massive inputs from dopamine neurons. The dopamine signal has been shown by Schultz et al. (1997) to behave like a reinforcement signal that can shift from predicting a primary reward to a conditioning stimuli. The behavior of the dopamine neurons resemble the reward prediction error in the temporal difference learning algorithm used in reinforcement learning. In fact, it seems as if BG could be the biological substrate in the brain for reinforcement learning, while cortex rather seems to be suited for unsupervised learning and cerebellum seems to be involved in supervised learning (Doya, 1999).

3 Models and methods

3.1 Computational neuroscience Computational neuroscience is one of the approaches to advance progress in the neuroscience field, in addition to disciplines such as psychology, physiology, and medicine. The field of computational neuroscience emerged from the realization that interdisciplinary studies are necessary in order to deepen our understanding of how the nervous system works. The nervous system has multiple levels of spatial organization ranging from molecules to the nervous system as a whole. Neurons specialize in signal processing mainly through electrical impulses called action potentials. Single neuron models can be utilized to study how ion channels and biochemical signaling affect and modify the dynamics of neurons. Insights from such studies may in turn be used to search for putative drug compounds able to restore normal function of malfunctioning neurons. Basic computational neural network models are used to study the information processing of multiple connected neurons and their signaling through electrical impulses. Such models have the potential to enhance the understanding of general computational principles of networks with different organizations, such as random-, feed forward-, competitive and point attractor networks. System level network models are used to study specific parts of the brain where the computational model can be developed based on the known or hypothesized anatomical organization of the region. System level models are beneficial to address questions about what computation a specific part of the brain can perform, and also can help us to identify important mechanisms that are responsible for network dynamics seen in neurological diseases as well as in the healthy system. In addition, there are models that have abstracted away the neuron as a computational unit, and instead use an equation to represent a population of neurons or describe the computation of a particular area of the brain as one function. Such models are generally called top down models since they are theory driven, as compared to bottom up models that are driven more by biological constrains. In reality most models have borrowed features from both the top-down and bottom-up approaches. Together, all these models are important for mapping out the bigger picture of how the nervous system is functionally and structurally organized. Top down models can be used to precede modeling studies with more detailed biologically constrained models that thus are used to verify or reject hypotheses generated by more abstract models.

3.2 Model components In this thesis, systems level network models are used, where the neuron is a basic computational unit connected through chemical synapses. Examples of neuron- and synaptic models suitable for such an abstraction level are given below.

3.2.1 Hodgkin-Huxley model The Hodgkin-Huxley model came about through pioneering work by Hodgkin and Huxley (1952) working on the axon of the giant squid. It was determined that the axon had three major + 4 currents: voltage-gated persistent 퐾 current (퐼퐾 = 푔퐾푛 (푉 − 퐸푘)) with four activation gates (the 4 + 3 term 푛 in the equation below), voltage gated transient 푁푎 current (퐼푁푎 = 푔푁푎푚 ℎ(푉 − 퐸푁푎)) with three activation gates and one inactivation gate (the term 푚3ℎ in the equation below) and a − Ohmic leak current (퐼퐿 = 푔퐿(푉 − 퐸퐿)) carried mostly by 퐶푙 ions. The equations Hodgkin-Huxley come up with to describe the voltage dynamics on the giant squid is given by the equation below where the 푉 is the voltage, 푛 is the activation variable for 퐾+, 푚 is the activation variable for 푁푎+ and ℎ the inactivation variable for 푁푎+.

푑푉 퐶 = 퐼 − 푔 푛4(푉 − 퐸 ) − 푔 푚3ℎ(푉 − 퐸 ) − 푔 (푉 − 퐸 ) 푑푡 퐾 푘 푁푎 푁푎 퐿 퐿

푑푛 = 훼 (푉)(1 − 푛) − 훽 (푉)푛 푑푡 푛 푛 푑푚 = 훼 (푉)(1 − 푚) − 훽 (푉)푚 푑푡 푚 푚 푑ℎ = 훼 (푉)(1 − ℎ) − 훽 (푉)ℎ (1) 푑푡 ℎ ℎ

Here 훼푖 and 훽푖 are voltage dependent (but not time) rate constants for the 𝑖-th ion channel, 퐼 is a current source and 푔퐾, 푔푁푎 and 푔퐿 are respectively the conductance for ion channels K, Na and leakage. Most of the currents activated in the neuron can be represented in the manner above, and if the 3-D morphology of the neuron is taken into account and one wants to represent active membrane properties throughout the dendrites, this model approach can result in thousands of equations for one neuron. A general challenge accompanying this approach is to find reasonable parameters for all the ionic currents modelled.

3.2.2 Integrate and fire model The leaky integrate and fire model (Tuckwell, 1988), represents a neuron as an unit having a leakage current and several voltage-gated currents that are inactivated at rest. Subthreshold behavior is given by the linear differential equation where 푉 is the membrane voltage: 푑푉 퐶 = −푔 (푉 − 퐸 ) + 퐼 푑푡 퐿 퐿

푓 𝑖푓 푉 > 푡 푡ℎ푒푛 푉 = 푉푟 (2)

Here 퐶 is the capacitance, 푔퐿 is the leak conductance, 퐸퐿 is the resting potential and 퐼 is a 푓 current source. When the membrane potential 푉 reaches 푡 it is reset to 푉푟. Often one wants to represent that neurons show spike adaptation or behave in characteristic ways when they start to spike, etc. Below some of these ways are summarized.

3.2.3 Adaptive exponential integrate and fire model The adaptive exponential integrate and fire model, also called AdEx, was introduced by Brette and Gerstner (2005) and is a hybrid spiking neural model with two variables which uses an exponential function to generate the upstroke of the action potential. It is a hybrid spiking model since it combines a smooth spike generation with sharp spike reset. Unlike Hodgkin- Huxley conductance based models, a hybrid model has only a few parameters derived from bifurcation theory. These models however lack the same degree of connection to physiology as parameters used in Hodgkin-Huxley formalism. The following equations control the dynamics of the exponential integrate and fire model with adaptation, where 푉 is the membrane potential and 푤 is the contribution of the neurons slow currents:

푑푉 푉 − 푉푇 퐶 = −푔퐿(푉 − 퐸퐿) + 푔퐿훥푇 푒푥푝 ( ) − 푤 + 퐼 푑푡 훥푇 푑푡 휏 = 푎(푉 − 퐸 ) − 푤 푤 푑푡 퐿

푓 𝑖푓 푉 > 푡 푡ℎ푒푛 푉 = 푉푟 푎푛푑 푤 = 푤 + 푏 (3)

Here 퐶 is the capacitance, 푔퐿 is the leak conductance, 퐸퐿 and 푉푇 are the resting and threshold potentials, Δ푇 is the slope factor, 퐼 is a current source, 휏푤 is the recovery current time constant and 푎 is the voltage dependence of the recovery current above. When the membrane potential 푉 푓 reaches 푡 it is reset to 푉푟 and then the recovery current 푤 is increased with 푏.

3.2.4 Izhikevich simple model The Izhikevich simple model (Izhikevich, 2003) is a two variable hybrid model which uses a quadratic function for generating the upstroke of an action potential. It is similar to the AdEx model in that is can capture the dynamics of neurons with few parameters.

The following equations control the dynamics of the quadratic integrate and fire model with adaptation, where 푉 is the membrane potential and 푢 is the contribution of the neuron’s slow currents:

푑푉 퐶 = 푘(푉 − 푣 )(푉 − 푣 ) − 푢 + 퐼 푑푡 푟 푡ℎ

푢̇ = 푎(푏(푉 − 푣푟) − 푢)

(4) 𝑖푓 푉 > 푣푝푒푎푘 푡ℎ푒푛 푣 = 푐 푎푛푑 푢 = 푢 + 푑

Here 퐶 is the capacitance, 푣푟 and 푣푡 are the resting and threshold potentials, 퐼 is a current source, 푎 is the recovery current time constant, 푏 is the voltage dependence of the recovery current and 푘 is a parameter determining the steady-state current voltage (I-V) relation. When the membrane potential 푉 reaches 푣푝푒푎푘 it is reset to 푐 and then the recovery current 푢 is updated with 푑.

3.2.5 Conductance based synapses A simple but realistic way to model the synapse dynamics is to use a conductance based synapse model. The model is given by the equation below where 푔 is the conductance.

푔 (5) 푔̇ = + 푔표 ∗ 훿(푡 − 푡푠푝푖푘푒) 휏푔푎푏푎

When a pre-synaptic spike arrives, the conductance 푔 is updated with 푔0 and then, in between the spikes, the conductance decays towards zero with time constant 휏푔푎푏푎. The postsynaptic current is given by 퐼 = 푔 ∗ (퐸푟푒푣 − 푉).

3.2.6 Tsodyks model Synaptic short-term plasticity can be captured with the Tsodyks-Markram model (Tsodyks et al., 1998). with the common FD formalism (Equations 6 and 7) (Abbott et al., 1997; Dittman et al., 2000; Abbott and Regehr, 2004; Puccini et al., 2007). The FD formalism dictates that the synaptic weight is changed by the product of facilitating (F) and depressing (D) components. This characterization display quantitatively good similarity to experimentally measured synapse dynamics (Tsodyks and Markram, 1997; Markram et al., 1998; Planert et al., 2010; Klaus et al., 2011). The model formalism assumes a fixed pool of synaptic assets in active (푦), inactive (푧) and recovered (푥) states. At rest 푦 and 푧 are 0 and 푥 is 1. Depression arise because some of the assets remain for a while in the inactive state before going into the recovered state with a rate set by the recovery time constant 휏푟푒푐. The facilitation is modeled by 푢 which is a variable that is step- wise increased at each spike with the product of the utilization factor 푈 and 1 − 푢 (U is between 0 and 1) and decays exponentially towards 0 with time constant 휏푓푎푐 in between spikes (Equation 6). The assets in the active state 푦 are increased with the product of the variables 푥 and 푢 (enabling depression and facilitation respectively) and are then rapidly inactivated by decaying towards zero with time constant 휏푠푦푛 (Equation 7). The postsynaptic conductance is proportional to the fraction of assets in the active state and is determined by 푔 = 푔0 ∗ 푦 with the post-synaptic current 퐼푠푦푛 = 푔 ∗ (퐸푟푒푣 − 푉).

푑푢 푢 (6) = − + 푈 ∗ (1 − 푢) ∗ 훿(푡 − 푡푠푝푖푘푒) 푑푡 휏푓푎푐

푑푥 푧 = − 푢 ∗ 푥 ∗ 훿(푡 − 푡푠푝푖푘푒) 푑푡 휏푟푒푐 푑푦 푦 = − + 푢 ∗ 푥 ∗ 훿(푡 − 푡푠푝푖푘푒) 푑푡 휏푠푦푛 푑푧 푦 푧 = − (7) 푑푡 휏푠푦푛 휏푟푒푐

3.2.7 Bayesian Confidence Propagation Neural Network The Bayesian Confidence Propagation Neural Network (BCPNN) is a learning framework that describes the relative activation levels of connected units together with their level of co- activation to probabilities that can be couple with Bayes rule (Lansner and Ekeberg, 1989; Lansner and Holst, 1996). The assumptions made in the derivation of this rule are functionally consistent with canonical microcircuits of the cerebral cortex (Johansson and Lansner, 2007) and BG (Berthet et al., 2012). The framework can be approximated by spiking neurons and their inter-synaptic connections (Tully et al., 2014), in which Hebbian learning arise both at the synaptic level in the form of long-term potentiation and at the neural level in the form of long- term potentiation of intrinsic excitability. The default, abstract representation of this network can learn randomly changing attractor states (Sandberg et al., 2002). But the spiking implementation flexibly allows for a vast set of dynamics within recurrent networks of integrate-and-fire neurons. Since probabilities can be approximated by three sets of cascading memory traces at separate time scales (Figure 2), different operating regimes can be reached by introducing relative differences between the time constants of these memory traces. For example, asymmetric fast-acting synaptic traces can allow the network to produce sequences of attractor states, and eligibility traces can enable the network to perform spike-based delayed reward learning. Altogether, the framework allows a principled methodology for modeling, and thus understanding, the interaction of different plasticity mechanisms in large-scale networks of spiking neurons

Figure 2 | Trace-based synaptic learning. (A) A pair of neurons fire randomly and elicit changes in the pre- (red) and postsynaptic (blue) primary Z traces of a BCPNN synapse connecting them. Sometimes by chance (pre before post*, synchronous+, post before pre#), the neurons fire coincidentally and the degree of overlap of their Z traces (inset, light blue), is propagated to the mutual eligibility trace Eij. (B) A reward (pink rectangle) is delivered as external supervision. resulting in E trace modification (gray line, τe = 100 ms and black line, τe= 1000 ms). (C) Behavior of color corresponding Pij traces and weights (inset) depends on whether or not the reward arrived in time.

The pre- Si and postsynaptic Sj spike trains are defined by summed Dirac delta pulses with 푖 푗 respective spike times 푡푠푝 and 푡푠푝:

푖 푗 (8) 푆푖(푡) = ∑ 훿(푡 − 푡푠푝) 푆푗(푡) = ∑ 훿(푡 − 푡푠푝) 푠푝 푠푝

Traces with the fastest dynamics, Zi and Zj, are exponentially smoothed spike trains:

푑푍푖 푆푖 푑푍푗 푆푗 (9) 휏푍 = − 푍푖 + 휀 휏푍 = − 푍푗 + 휀 푖 푑푡 푓max ∆푡 푗 푑푡 푓max ∆푡

17 which lowpass filters pre- and postsynaptic activity with time constants 휏푍푖 and 휏푍푗 , like what would be expected from rapid Ca2+ influx via NMDA channels or voltage-gated Ca2+ channels. It is assumed that each neuron could fire maximally at 푓푚푎푥 Hz and minimally at 휀푓푚푎푥 Hz, which serve as absolute certainty and doubt concerning the suggestive context of the input. In that range, firing levels correspond to the approximated probability. Each spike event had a duration of ∆푡 ms. These Z traces are then passed on to the E eligibility traces:

푑퐸푖 푑퐸푗 푑퐸푖푗 휏 = 푍 − 퐸 휏 = 푍 − 퐸 휏 = 푍 푍 − 퐸 (10) 푒 푑푡 푖 푖 푒 푑푡 푗 푗 푒 푑푡 푖 푗 푖푗 where, in order to track the coincident activity from the Z traces, a separate equation is introduced. τe is the time constant for these traces which are assumed to represent intracellular Ca2+-dependent processes (Fukunaga et al., 1993). The E traces are then used in the calculation of the P traces, whose longer time courses could correspond to processes like gene expression or protein synthesis. These values correspond to the final probability estimates based on smoothed activity levels:

푑푃 푑푃 푑푃 휏 푖 = 휅(퐸 − 푃 ) 휏 푗 = 휅(퐸 − 푃 ) 휏 푖푗 = 휅(퐸 − 푃 ) (11) 푝 푑푡 푖 푖 푝 푑푡 푗 푗 푒 푑푡 푖푗 푖푗

where κ is the reward prediction error (RPE) value and τp the time constant of these P traces.

3.3 Parameter estimation Many parameters for the neuron and synaptic models were retrieved from already published studies in this work, however in some cases where no previous model existed we estimated parameters ourselves. The models were previously derived in Lindahl et al. (2013), here follows a summary of what was presented in the article plus additional information about the method behind the estimation of model parameters. To estimate the parameters to the AdEx model, different approaches were used depending on the type of data on neuron dynamics that could be obtained. It is not a trivial task to estimate the relatively few parameters in adaptive exponential integrate and fire model. Brette and Gerstner (2005) describe how the parameters for a pyramidal cell can be estimated from specific type of recordings. Similar techniques were used when applicable. Otherwise the parameters were hand tuned such that whether the behavior of the neuron was in accordance with recordings of current-voltage curves, current-frequency curves and miscellaneous stimulation protocols. The following equations were used to approximate EL, VT and ΔT for GPe, SNr and STN neuron model.

푑푉 푉 − 푉푇 퐶 ∗ = −푔퐿(푉 − 퐸퐿) + 푔퐿훥푇 푒푥푝 ( ) − 푤 + 퐼 = 0 푑푡 훥푇 푑푡 휏 = 푎(푉 − 퐸 ) − 푤 = 0 푤 푑푡 퐿

⇒

푑푉 푉 − 푉푇 = −푔퐿(푉 − 퐸퐿) + 푔퐿훥푇 푒푥푝 ( ) − 푎(푉 − 퐸퐿) (12) 푑푡 훥푇 푑2푉 푉 − 푉 푇 2 = −푔퐿 + 푔퐿 푒푥푝 ( ) − 푎 (13) 푑푡 훥푇

3.3.1 Short-term plasticity The parameters for the Tsodyks synapse model were estimated in Matlab using a least square method minimizing the squared error between experimental and model current pair pulse data.

To find the solution the fminsearch method in Matlab which implements the Nelder-Mead Simplex method (Lagarias et al., 1998) was employed. For facilitating and depressing synapses in SNr a data set collected from Connelly et al. (2010) was used. These data show the relative synaptic current increase/decrease over 10 successive spikes at 10, 50, and 100 Hz as well as the relative size of a recovery post synaptic current after 5 pulses at 100 Hz and measured after 60, 160, 560, 3000, and 9000 ms. For the facilitating MSN synapse in GPe a dataset from Sims et al. (2008) with the relative synaptic current increase over 10 successive spikes at 20 and 50 Hz was used. Figure 3A shows an example of the fit of the depressing GPe to SNr synapse, with 푈 = 0.191, 푡푟푒푐 = 0.196 ms and 푡푓푎푐 = 0 ms and Figure 3B show an example of the fit of facilitating MSN to SNr synapse where 푈 = 0.02, 푡푟푒푐 = 551 ms and 푡푓푎푐 = 621 ms.

3.3.2 GPe neuron The hyperpolarization triggered spike of the GPe neuron was estimated with a subthreshold adaptation at 2.5 nS and a time constant 휏푤 at 20 ms. Steady-state current voltage relation was captured with 푔퐿 a 1.0 nS and a capacitance 퐶 at 40 pF. Note that with these parameters a GPe neuron exhibits subthreshold oscillations close to rheobase current (the minimal current necessary to elicit a spike) (Touboul and Brette, 2008). To mimic the frequency acceleration and the spike frequency adaptation of the, summed recovery current contribution, 푏, at spike reset was set to 70 pA. With the spike voltage reset, 푉푟, at −60 mV and spike cut off, 푉푝푒푎푘, at 15 mV the after hyperpolarization and spike amplitude were in accordance with literature. 퐸퐿, 푉푇 and Δ푇 were estimated on membrane potential of spike initiation, threshold membrane potential and acceleration of membrane potential at spike generation. Experiments indicate that the GPe neuron goes from silent to spiking at around −53 mV and have a spike threshold at −43 mV, defined as when the acceleration of the membrane potential reaches 50% of its max, estimated to 1270 mV/ms2. Both the spike initiation and threshold membrane potential were given in Bugaysen et al. (2010), whereas the acceleration had to be estimated, This was done by fitting a sigmoid (Figure 3A) to the up stroke of an action potential (Bugaysen et al. 2010). With equation 12 and 13 the following equation system was set up and then solved for 퐸퐿, 푉푇 and Δ푇. 푑푉 (−53) = 0 푑푡 푑2푉 (−53) = 0 푑푡2 푑2푉 (−43) = 1270 (14) 푑푡2

In Figure 3A we show the acceleration nullcline of the solution 퐸퐿 = −55.1 (V), 푉푇 = −54.7 (V) and Δ푇 = 1.7 (ms).

Figure 3 | Parameter estimation of synapse and neuron models. (A) Blue, green and red dots in left panel show relative synaptic current decrease over 10 successive spikes at 10, 50, and 100 Hz for GPe to SNr. The blue, green and red solid lines in left panel show the model fit. The black dots in the middle panel shows the relative size of a recovery spike after 5 pulses at 100 Hz and measured after 60, 160, 560, 3000, and 9000. The solid black line in the middle plot shows the model fit. In the right panel the steady state amplitude for the synapse model over a range of post-synaptic spike frequencies are shown. (B) Same as figure (A) but for facilitating synapses between MSN and SNr. (C) In the first panel the diamond marker shows the data point V=-43, d2V/dt=1270 and dot marker shows the data point V=-43, d2V/dt=0 for the GPe neuron. The solid line shows the fit of the adaptive integrate and fire model to these data points. The middle diamond shows the fit of a sigmoidal curve to the upstroke of an action potential of the GPe neuron, where the dots show experimental data and the solid line the model fit. In the right panel we show the data point where the speed of the upstroke are 50 percent of the max speed of the sigmodal fit of the action potential. (D) The diamond shows the point V=-52, dV/dt=10.2 and the dot shows the point V=-54, dv/dt=0 for the SNr neuron, and the solid line is the fit of the first voltage derivative of the adaptive integrate and fire model to this data. (E) The diamond shows the point V=-52, d2V/dt=0 and the dot shows the point V=-35, d2v/dt=50 for the STN neuron, and the solid line is the fit of the second voltage derivative of the adaptive integrate and fire model to this data.

3.3.3 SNr neuron

The subthreshold adaptation 푎, time constant 푡푤, capacitance 퐶, summed current contribution 푏 at spike reset of the SNr neuron was set to 80 pF and 200 pA such that spike frequency acceleration and spike frequency adaptation were in accordance with experiments. Spike voltage reset 푉푟 and spike cut-off 푉푝푒푎푘 were set to capture the after-hyperpolarization and spike amplitude of recorded neurons. Then to estimate resting and threshold potentials and slope

20 factor, 퐸퐿, 푉푇 and Δ푇 we used experimental data on SNr neuron that goes from silent to spiking at approximately −54 mV and have spike threshold at −52 mV, defined as when the rate of rise is 10.2 mV/ms (Richards et al., 1997; Atherton and Bevan, 2005; Chuhma et al., 2011). The following equation system set up and then solve for EL, VT and ΔT.

푑푉 (−52) = 10.2 푑푡 푑푉 (−54) = 0 푑푡 푑2푉 (−54) = 0 (15) 푑푡2

In Figure 3A the nullcline of the solution 퐸퐿 = −55.8 (V), 푉푇 = −55.2 (V) and Δ푇 = 1.8 (ms) is shown.

3.3.4 STN neuron

The subthreshold adaptation of the STN model was set at 0.3 nS below −70 mV with 휏푤 at 333 ms, such that 333푤̇ = 0.3(푉 + 70) − 푤 to account for the hyperpolarization activated inward current that is behind the rebound bursts and was set to 0 nS above −70 mV, such that 333푤̇ = −푤, to get minimal spike frequency adaptation. The STN neuron's steady-state current-voltage relation was estimated by setting 푔퐿 to 10.0 nS in accordance with experiments. To capture delayed afterhypolarization caused by increased current injection as well as the spike frequency acceleration the capacitance, 퐶, the summed recovery current contribution, 푏, at spike reset and the spike voltage reset, 푉푟, was respectively set to 60 pF, 0.05 pA, and −70 mV. The hyperpolarization-induced bursts were estimated by resetting V following a spike to 푉푟 + max (푤 × −10,10) if 푤 < 0 and else to 푉푟. A similar modification to the spike reset point has been done by Izhikevich (2003). With the spike cut off, 푉푝푒푎푘, at 15 mV we got a spike amplitude in accordance with literature. To estimate 퐸퐿, 푉푇 and Δ푇 we again used equation 12 and 13. Experiments show that STN are silent at a membrane potential at −64 mV and a have a spike threshold at −35 mV, within the acceleration of membrane potential at 50 mV/ms2 (Kass and Mintz, 2006; Farries et al., 2010). Thus we got the following equation system and solve for 퐸퐿, 푉푇 and Δ푇. 푑푉 (−64) = 0 푑푡 푑2푉 (−64) = 0 푑푡2 푑2푉 (−35) = 50 (16) 푑푡2

In Figure 3A the nullcline of the solution 퐸퐿 = −80.2 (V), 푉푇 = −64.0 (V) and Δ푇 = 16.2 (ms) is shown.

3.4 Computational models of the basal ganglia Below a brief summary of selected BG related models are done. The model presented have been chosen two show the broad spectrum of questions that system levels models of BG have addressed and intended to put the models presented in this thesis in relation to previous work.

3.4.1 Houk and Beiser 1998 A model of a macroscopic module of prefrontal cortex-BG-thalamus-prefrontal cortex and several similar microscopic modules within the macroscopic module was built (Houk and Beiser, 1998). A single membrane-bound compartment with passive leakage was use to model neurons. A sigmoidal activation function was used to convert membrane potential into normalized spike count between 0-1. The model tested the hypothesis that BG are important for serial order behavior in prefrontal cortex. It was shown that the activity sequences could be represented in bi-stable cortical-thalamic loops which could be activated by MSNs. It was shown

21 that the model, when instantiated with randomly distributed corticostriatal weights, could produce different patterns of prefrontal activity in response to different target sequences. Such patterns were a clear spatially distributed encoding of the sequence in prefrontal cortex.

3.4.2 Bar-Gad, Havazelet-Heimer, Goldberg, Ruppin and Bergman 2000 A three-layer feedforward network with linear activation functions and lateral connectivity of BG were built to test if BG could perform reinforcement driven dimensionality reduction (Bar- Gad et al., 2000). The model had a cortical input layer, a striatal intermediate layer and a pallidal output layer. The model shows how cortical salient information can be extracted by means of dimensionality reduction, that used by prefrontal cortex through thalamus.

3.4.3 Terman, Rubin, Yew and Wilson 2002 A spiking neural network model of STN and GPe were used to investigate how the connectivity strength between STN and GPe and within GPe effected activity patterns in STN and GPe (Terman et al., 2002). It is shown how the network can sustain three types of firing, clustering, propagated waves and regular firing dependent on the connectivity strength between STN and GPe and within GPe. The model predicts that the STN-GPe network is capable of both correlated rhythmic activity and irregular self-sustained firing that can block the rhythmic activity. It is suggested that altered connectivity in STN-GPe network through dopamine depletions could be responsible for the oscillations seen in PD.

3.4.4 Frank 2006 A rate model of BG is presented that includes all major BG nuclei, including STN. The model is used to show that STN can prevent premature action selection when modulating the time of response execution. It is also shows how oscillations emerge when dopamine is removed and how lesioning STN have a beneficial effect by decreasing oscillations.

3.4.5 Lo and Wang 2006 A spiking model of cortex, superior colliculus and BG was built and used to se if the neural substrate of the threshold passing seen in monkey experiments reaction time task could be found (Lo and Wang, 2006). Here BG are represented by caudate nucleus and SNr. The model shows that cortico-striatal synapses as opposed to cortico-colicular synapses can be used to optimally tune the threshold passing of cortical neurons in a time reaction task.

3.4.6 Humphries, Stewart, Gurney 2006 A detailed spiking model of BG is built including all major features of BG known at that time and verified by experimental data on nuclei firing rates and oscillation dynamics in dopamine depleted state (Humphries et al., 2006). It is shown how BG can perform action selection and action switching based on input salience. The model further predicts that action selection deteriorates during low and high dopamine. The model also confirms results that were previously obtained in a rate model of BG (Gurney et al., 2001).

3.4.7 Humphries, Wood and Gurney 2009 A large (~80000 neurons) network model of striatum including dopamine modulated MSNs and FSNs are used to study striatal computations (Humphries et al., 2009b). The model predicts that groups of synchronized cell assemblies on multiple time scale spontaneously can form. The number of cell assemblies are shown to depend on the dopamine concentration. Counter intuitively, the model also predicted that FSN neurons connected with both chemical synapses and electrical gap junctions increase the firing rate of MSNs.

3.4.8 Ponzi and Wickens 2010 A spiking network model with sparse asymmetric randomly connected MSN neurons was built by Ponzi and Wickens (2010). It was shown how episodic sequential firing of cell assemblies emerged from the model. Members of the same assembly showed correlation on relevant timescales of hundreds of milliseconds whereas members of different assemblies showed strong negative correlation. It is speculated that such cell assemblies could be important for information processing in the striatum.

3.4.9 Steward, Bekolay and Eliasmith 2012 A spiking model of BG, with a spike based learning rule and dopamine as a reward signal is built and tested against experimental data (Stewart et al., 2012). A good match in terms of behavioral learning and spike patterns in ventral striatum is obtained. The model is built on the model of Gurney et al.( 2001). The model successfully learns the utilities of multiple actions, and how to choose actions in different states. For dopamine signal a local signal for each action is used instead of a global signal, which is not the current consensus of how dopamine is considered to be released in striatum, i.e. on a global level. But as they argue, it might be that since experiments have shown a variation of dopamine levels at different sites in striatum this could reflect a localization of dopamine release.

3.4.10 Nevado-Holgado, Mallet and Magill 2014 Several candidate rate models of GPe type I, GPe type A and STN were matched against experimental data (Nevado-Holgado et al., 2014). The best candidate model predicted that the input and output to GPe type I differed significantly from GPe type A, counting striatal, STN and thalamic connections. It was also predicted that the two types of GPe neurons where to have different physiological properties reflected in their respective separate firing rates.

3.4.11 Bahuguna, Aertsen and Kumar 2015 Two striatal models, one rate based and one with spiking units, including MSNs and FSNs accounting, were built (Bahuguna et al., 2015). The models accounted for recent experimental finding showing that the recurrent as well as feedforward connectivity markedly differs. The rate model predicts that the asymmetric connectivity of the two types of MSNs turn striatum into a threshold device, which is later confirmed in the spiking model. MSN D2 neurons make stronger inhibition on MSN D1 than for the opposite relation. It is show that MSN D1 neurons firing rate for low cortical input surpass MSN D2 in firing rate, driven by the same cortical drive, but at higher rates MSN D2 firing is higher than MSN D1. It is described how the decision threshold crossing can be shifted by FSNs, dopamine level and pallidal back projections from GPe type A neurons.

4 Results and Discussion My different research projects concern action selection and network dynamics in BG. In section 4.1 the hypothesis on action selection created in project I is presented, and how it was tested with a model. In section 4.2 selected results from the detailed model developed in project II is presented. Finally, in section 4.3 results from the spiking neuron model with reinforcing learning of project III is shown.

4.1 The arbitration and extension system hypothesis of basal ganglia In the first project we developed a hypothesis of the functional organization of BG. It was hypothesized that BG could be divided into two different subsystem, the arbitration system and the extension system. The arbitration system was postulated to account for solving competition between different action based on salience in a winner-take-all manner, whereas the extension system extends the arbitration system and adds the possibility to learn logical expressions such as conjunction, disjunction and negation. Below are selected results presented from project I, for more details se paper 1.

Figure 4 | BG projections and connections to other CNS regions (excitatory and inhibitory projections are shown by arrows and stars respectively). Decisions are made by several mechanisms organized hierarchically. (Modified from Kamali Sarvestani et al. (2011))

4.1.1 The arbitration system can select the most salient response The winner take all network in the arbitration system (Kamali Sarvestani et al., 2011) consisting of PPN, brain stem, CM, GPe and STN (Figure 4) can successfully be used to escape an aversive stimuli and to target the strongest stimuli. In a simulation with an artificial point-like animal without BG each of the stimuli triggered a motor response, thus the model ends up activating multiple direction neurons at once (Figure 5A). Instead in Figure 5B the winner-take-all property of the arbitration system is demonstrated, the animal successfully suppresses all but one of the responses at a time, resulting in an effective escape followed by a precise targeting.

The activities in all nuclei of the arbitration system show domination of a single action at any given time and a proper soft switch between actions when the relative strength of the second response takes over.

Figure 5 | Activity of the BG nuclei during decision making and corresponding decisions. The subplots to the left display the activity of the nuclei indexed. The horizontal axes represent time (in seconds) whereas the vertical axes show 128 different neurons each corresponding to a certain directions of movement, i.e., the competing actions. The neuronal activity is color coded in brightness of each neuron in a given point in time. (A) An animal without the BG will average the mixed responses it receives. Such an animal is deprived of effective escape and precise targeting behaviors. (B) An animal with arbitration system is capable of selecting one action and suppress all of its competitors hence enhancing the escape and targeting behaviors. (C) An animal with MSN D1 connections to the GPe and the GPi can enforce learned responses and suppress the otherwise strongest action in certain states. (D) An animal with MSN D2 collaterals can suppress the learned responses in certain states. (Modified from Kamali Sarvestani et al. (2011))

4.1.2 The extension system can store conjunction and disjunction patterns The capability of the extension system involving striatum direct and indirect pathways (Figure 4) of storing conjunction and disjunction patterns in an action unit is shown in Figure 5C. The animal is assumed to have learned that either the combination of landmarks a and b or the combination of landmarks e and f (interchangeable with landmark a and b in Figure 5C) will transform the red stimulus (originally aversive) into an appetitive one. Lack of a proper combination of landmark stimuli (a and b together or e and f together) fails to push the membrane potential of the MSNs to the vicinity of threshold. However, a proper combination of landmarks available activates either of the MSN D1 in the action unit responsible for approach behavior. Activation of MSN D1 inhibits the GPe neurons representing the escape response hence suppressing the innate tendency of the animal to escape from the aversive stimulus. The same striatal neurons also inhibit GPi neurons representing approach response thus lifting inhibition from corresponding PPN. The PPN neurons fire by the virtue of their intrinsic spontaneous activity, enforcing the learned approach response. The same GPi neurons disinhibit the CM neurons which in turn activates corresponding STN.

4.1.3 The extension system can store negation The capability of MSN D1-MSN D2 inhibitory collaterals in negating a certain situation (Boolean NOT) is shown in (Figure 5C). Landmarks c and d the combination of which is assumed to restore the original nature of the red stimulus (as an aversive stimulus) are added in this demonstration (still keeping landmark a and b). Since the learned action caused by stimulation of the MSN D1 in previous demonstration is to be suppressed here, the MSN D2 stimulated by components c and d of the state sends input not only to GPe but also to the MSN D1 hence inhibiting them and removing the influence of the extension system on the GPi. It is worth noting that although there is enough contextual support to activate both MSNs, since MSN D2 is more excitable than the MSN D1, it fires more easily and wins the mutual competition.

4.2 Lessons learned through detailed spiking models of the basal ganglia In the above systems level model both simplified neurons and synapses have been used. In the project relating to paper 2 and 3 we study BG dynamics and function using more detailed model which were built in two steps. In the first step STN, GPe and SNr were included. It was studied how signaling through the direct-, indirect- and hyperdirect pathways affect the output stage of the BG and how this is controlled by synapses exhibiting short-term dynamics. Then in the second step, the model is extended with striatum, including MSN and FSN as well as dividing GPe into the two distinct population GPe type I and GPe type A. Dopamine modulation is also included. With the second version of the model a more comprehensive coverage of BG was accomplished. It enabled us to study how connectivity, neuron excitability and dopamine affect function and dynamics in normal and dopamine depleted state. Below are selected results presented from project II, for more details se paper 2 and 3.

4.2.1 Filtering effect of synaptic short-term facilitation in MSN-SNr synapses Input arriving at high frequency rates is strengthen by facilitating synapses whereas low frequency input is filter out. An experiment was set up where a group of emulated MSN D1 were stimulated for five hundred milliseconds and then the firing rate in SNr where measured over time (Figure 6A-C). This were done for three models, one with facilitating synapses and two with static synapses (the minimum or maximum conductance of the facilitating synapse) as references. Several simulation experiments were run where the frequency of the bursts and the size of the bursting MSN population were varied. It was seen that stimulation of only a few percent of pre-synaptic direct pathway MSNs results in powerful inhibition of SNr during

26 steady-state (Figure 6D) and that action signaling becomes more resource demanding for lower MSN D1 spike frequencies requiring stimulation of significantly more pre-synaptic MSNs (Figure 6E). Thus facilitating MSN D1 synapses in SNr indeed empower high frequency input and filter out low frequency input.

Figure 6 | Short-term facilitation in MSN-SNr synapses have a profound filtering effect at different biologically realistic spike frequencies. (A) Raster plot of the emulated activity of 15,000 pre-synaptic MSNs with 4% of the neurons bursting (red) at 20 Hz for 500 ms and the rest of the population (blue) firing at 0.1Hz. (B) Firing frequency of pre-synaptic MSNs shown in (A) averaged over the whole population (blue), and over the bursting inputs (red) (triangular kernel window 100 ms used). (C) The resulting inhibitory response in SNr over time. The 푓푎푐푀푆푁퐷1 synapses (magenta) need time to be 푀푆푁퐷1 푀푆푁퐷1 fully activated, delaying the threshold crossing for 200 ms here. With the 푟푒푓푖푛푖푡 (blue) and 푟푒푓푚푎푥 (green) synapses the inhibitory effect appears immediately (triangular kernel window 100 ms). The standard deviation of population activity between simulations is shown as shaded areas around the mean (solid or dotted lines). (D) The number of MSN D1 bursting with a certain frequency (7–48Hz) which are needed for action selection, defined as decreasing SNr firing under a certain threshold. If facilitated synapses are used (magenta), only a few MSNs are needed when bursting in the 푀푆푁퐷1 푀푆푁퐷1 interval 17–48Hz, and with performance closer to 푟푒푓푚푎푥 (green) synapses than to 푟푒푓푖푛푖푡 (blue) synapses during the last 100 ms of the 500 ms burst. (E) Steady-state firing rate in post-synaptic SNr cells when all pre-synaptic MSN D1 successively increase their firing. Facilitating synapses (magenta) allow background activity to increase up to 1.2Hz before suppressing SNr to action signal threshold. (Modified from Lindahl et al. (2013))

4.2.2 Filtering effect of depressive STN-SNr synapses in the hyperdirect pathway Postulate that both the GPe and STN synaptic weights in SNr were fixed, then one would predict that they counteracted each other, e.g., they might even cancel each other out where increased stimulation of STN only leads to very small rate change in SNr (Figure 7A, blue dotted line). But since GPe synapses in SNr are depressing (Connelly et al., 2010), the synaptic input from STN would come to control the response in SNr where increased firing of STN cells causes increased firing of SNr cells (Figure 7A, blue solid) because the inhibitory signal through the depressing GPe-SNr synapses would saturate (Tsodyks and Markram, 1996) while the excitatory input from

STN would continue to increase with frequency. However, data from experiments in rat and monkey contradict such results, and instead implicate that increased firing rate in STN do not lead to increased firing rate in SNr/GPi (Maurice et al., 2003; Kita et al., 2005; Moran et al., 2011). These results are well explained by published (Moran et al., 2011) and unpublished observations (Rosenbaum et al., 2012) and by model simulations (Figure 7A, solid green) where STN is hypothesized to connect to SNr with depressing synapses. MSN D2 neurons have a significant effect on SNr activity with and without depressing STN to SNr synapses (Figure 7B) whereas stimulation of STN for five hundred milliseconds shows minor effect on SNr for the first hundred milliseconds, two-three hundred milliseconds and for the last hundred milliseconds Figure 7C. Only for a very short synchronous activation of STN (Figure 7D) a significant increase of SNr rate is seen. The time course of responses in SNr due to such stimulation are supported by experiments in rat and monkey where a triphasic response is triggered by a brief pulse directly in STN or in cortex (Maurice et al., 2003; Kita et al., 2005; Jaeger and Kita, 2011). The inhibitory response in SNr following the short STN stimulation can be quenched by removing STN to GPe connections (Figure 7E) which is in accordance with experiments showing how application of Gabazine in GPi (homolog to SNr) in monkeys quenches the inhibitory and late excitatory response in GPi succeeding cortical activation in vivo (Tachibana et al., 2008). As foreseen, STN will indirectly inhibit SNr via GPe for a longer duration when the connections between STN and SNr instead are removed (Figure 7F).

Figure 7 | Steady-state and temporal effects succeeding activation of the indirect and hyperdirect pathway. (A) Effects 퐺푃푒 퐺푃푒 on SNr frequency when increasing the total STN population activity for 푑푒푝 (solid) and 푟푒푓30 퐻푧 (dotted) GPe to SNr synapses, and with static (blue) and depressing (green) STN-SNr synapses. (B) Effects on SNr frequency when 퐺푃푒 퐺푃푒 increasing MSN D2 population activity for 푑푒푝 (solid) and 푟푒푓30 퐻푧 (dotted) synapses. (C) SNr activity in response to a 500ms burst in STN during the first 100 ms (blue), between 250 and 350 ms (green) and during the last 100 ms (red) using depressing (solid) and static (dotted) STN synapses in SNr. (D) Rate in SNr (blue), GPe (green) and STN (red) after a brief (3 ms) high frequency excitatory pulse into STN. (E) Same as (D) but with STN to GPe lesioned. (F) Same as (D) but with STN to SNr lesioned. (Modified from Lindahl et al. (2013))

4.2.3 The control of MSN excitability through MSN collaterals, FSN and GPe In version two of the model, the GPe type A neurons were included. Thus the three main contributors of GABAerigic inhibition in striatum, MSN collaterals, FSN and GPe, could now be accounted for. Given the model, a simple question was asked; how are the excitability of striatal MSNs controlled by the three sources of inhibition. It was found that inhibition from FSNs and GPe were more important in controlling MSNs firing rate at low cortical drive, whereas collaterals from neighboring MSNs were more important at higher cortical drive (Figure 8A).

Activated MSNs that are not connected by collaterals have significantly higher firing rate than connected MSNs. In the arbitration and extension hypothesis the collaterals in MSN are assumed to be strong enough to have an effect on surrounding MSNs firing rate, but physiological evidence suggest that the connections are weak (Tunstall et al., 2002; Taverna et al., 2004; Tepper et al., 2004). To test if weak connections between MSNs (between 0.15-0.45 nS) still can be important for action selection we simulated two scenarios. We planted in a striatal network with 3000 neurons a number of non-connected groups of MSN neurons (Figure 8B-C). In the first scenario we randomly activated neurons from the pool of MSNs neurons, and in the second scenario we specifically activated only non-connected MSNs. It is clear that non- connected MSNs spikes higher than if connected (Figure 8D first panel) and that this holds over a range of different percentage of activated MSNs (Figure 8D second panel). Weak collateral MSN connectivity may thus be able to restrict the firing rate of neighboring MSNs and work as a mechanism in increasing the contrast in action selection.

Figure 8 | Control of MSN neurons firing rate by striatal inhibition: (A) How firing of MSN D1 and MSN D2 neurons are differentially influenced by inhibition form FSN, GPe and MSN collaterals. Left top panel shows the firing rate of MSN D1 neurons where cortical input is increased with a factor 1 to 1.5, when keeping all inhibition (dark blue), only using MSN- MSN inhibition (light blue), only using FSN-MSN inhibition (cyan), only using FSN-MSN static synapse inhibition (yellow), only GPe TA-MSN (orange) and without any inhibition (dark red). Right top panel shows the relative addition to the inhibition with increased cortical input for MSN-MSN inhibition (light blue), FSN-MSN inhibition (cyan), FSN-MSN static synapse inhibition (yellow) and GPe TA-MSN (orange). Left and right bottom panels shows the same as left and right top panel respectively except it is for MSN D2. (B) Illustration of the role of lateral inhibition when a subpopulation, 10%, of MSNs are activated. Left top panel shows the result when randomly selected MSN D1 neurons are activated and right top panel shows the result when only non-connected MSN D1 neurons are activated. Left and right bottom panels show the same as left and right top panels for MSN D2. (C) Connection diagram for MSNs where a blue dot indicated connection between MSNs. It is illustrating how groups of MSN are not connected. (D) Left top panel shows the firing rate of the 10 percent of MSN D1 neurons activated with a 500 ms long input burst for the case when randomly connected MSNs (blue) and specifically non-connected MSNs (red) are targeted. Right top panel shows the mean difference between the firing rate of the two differently selected populations of MSN D1 neurons when varying the activated MSN population size between 2-20 percent. Left and right bottom panels shows the same thing as left and right top panels but for MSN D2.

4.2.4 Dopamine dependent changes enhancing oscillations and synchrony Dopamine depletion leads to multiple changes in synaptic strength and neural excitability (Figure 9A), all potentially contributing to the emergence of synchrony and oscillations in

29 dopamine depleted rats. We found that dopamine induced changes to neural excitability and synaptic coupling have a diverse effect on synchrony and oscillations, with some nuclei showing increase and others a decrease in synchrony and oscillations for a single dopamine parameter. For each of the perturbations induced by dopamine depletion we run simulations where the control state parameter value was restored one at a time, and then compared the simulation result with the full lesioned simulation (Figure 9B-C).

 Restoring the dopamine-depletion induced increase in coupling within GPe leads to a decrease in synchrony for GPe TA neurons and to a smaller degree for MSN D2, but there is also an increase in GPe TI. The effect on oscillations is predicted to be negligible.  There is a decrease in synchrony and oscillations in STN when restoring the increased synaptic strength in the STN to GPe synapse, however there is an increase in synchrony in MSN D2 and TA, and to a lesser extent in MSN D1 and FSN.  Restoring the dopamine-depletion induced strenghtening of MSN D2 to GPe synapses results in a decrease of synchronization in GPe TI and SNr neurons and a increase is seen in MSN D2, and GPe TA, and to a lesser extent in MSN D1 and FSN. The effect on oscillations seems to be negligible in MSN D2, GPe TA, GPe TI, SNr and STN and resulting in a small increase in oscillations in MSN D1 and FSN.  Restoring the dopamine-depletion induced increase in the coupling between FSN and MSN D2 leads to a decease in synchrony in MSN D2, FSN, GPe TA and GPe TI. The effect on oscillations for FSN-MSN D2 synaptic dopamine parameter is predicted to be negligible.  It can be seen that increased synaptic coupling between CTX and MSN D2 strongly affect syncrony and/or oscillations in BG with a decrease seen in MSN D2, GPe TA, GPe TI and SNr, and an increase seen in MSN D1, FSN and STN.  Restoring the dopamine induced decrease in the coupling between MSNs decrease synchrony in MSN D1, MSN D2, GPe TA, GPe TI and SNr whereas it increase synchrony in FSN and STN. The affect on oscilations is neglectible except for MSN D1 and to a small degree for FSN.  Restoring the dopamine depletion induced decrease in excitability in GPe neurons have almost exclusively a dampening effect on synchrony (MSN D1, MSN D2, FSN, GPe TA, GPe TI and SNr) and oscillations (MSN D1 and FSN) and only minor increase in synchrony of STN.  Restoring the dopamine depletion induced increase in CTX to STN synapse has a dampening effect on synchrony (GPe TA, GPe TI, SNr, STN) and oscillations (STN).

Figure 9 | Local network dynamics and the role of dopamine dependent perturbations: (A) Illustration of dopamine depletion effect on neurons and connections in the network model. (B) Synchrony and oscillations in control and lesioned network components. Upper panel shows amount of synchrony in MSN D1, MSN D2, FSN, GPe TA, GPe TI, SNr and STN during control (black bars) and following lesion (white bars). Lower panel shows the same but for oscillations instead of synchrony. (C) The upper panel shows the relative change in synchrony in MSN D1, MSN D2, FSN, GPe TA, GPe TI, SNr and STN (y axis) compared to the lesioned network when restoring the parameter on the x

30 axis to the value it had in the control network (no dopamine depletion). Lower panels show the same but for oscillations instead of synchrony.

4.2.5 Does basal ganglia support action selection? BG are generally thought to be important in action selection. Kravitz et al. (2010) showed that D1 receptor activation promotes actions and D2 receptor activation inhibits actions. To test action selection with the model two competing actions (action 1 and action 2) were inserted into the network, and then activated over a range of cortical input and with the output rate in SNr measured. Selection of action 1 is defined as when the firing rate in SNr for action 2 is below 50 percent normal activity whereas action 1 is above. The opposite holds for selection of action 2. Dual selection occurs when the SNr firing rate for both actions are below 50 percent of SNr base rate whereas no selections occurs if SNr firing rate is above base rate for both actions. It was found that the indirect pathway is important for improving the dynamical range of the input- output, given that cortical input should result in action selection (compare Figure 10A top panels). Action selection was also robust when varying the action pools between 15-100 percent (Figure 10B).

Figure 10 | Action selection performance of BG network: (A) Top left panel shows the action selection with only the direct pathway activated with respectively cortical input from 1 x base level to 3x base level for action 1 and 2. Top right panel shows the action selection when both direct and indirect pathways are activated (similar action inputs as in top left panel). Middle left panel shows the action selection when both direct pathways are activated and MSN collaterals have been removed (similar inputs as in top left panel). Middle right panel shows the result when both direct and indirect pathways are activated and FSN-MSN inhibition has been removed (similar action inputs as in top left panel). Bottom left panel shows the outcome when both direct and indirect pathways are activated and GPe TA-MSN inhibition has been removed (action similar inputs as in top left panel). Bottom right panel shows the action selection when both direct and indirect pathways are activated and in addition the hyperdirect pathway is co-activated through a burst in STN (similar action inputs as in top left panel). For all six plots 20 percent of the MSN action pool was activated (one MSN action pool equaled half of the simulated MSNs) (B) Action selection performance when scaling the size of the activated MSN pool. First row shows the action selection with only direct pathway activated, same as top left figure in A, but varying the relative size of the MSN pool between 10-100 percent. Second row shows the same as the first row, but for the scenario when both direct and indirect pathways are activated. Third to fifth rows show the result for the scenario when both direct and indirect pathways are activated (as in row 2) but when MSN collaterals are removed, FSN-MSN inhibition is removed or GPe TA-MSN inhibition is removed. Sixth row show scaling effect when the hyperdirect pathway is activated during action selection.

Action selection deteriorates when MSN collaterals and/or feed-forward striatal inhibition are removed, whereas GPe inhibition to striatum has little effect. It was found that when removing the inhibition on MSNs from either collaterals or the FSNs, the action selection performance deteriorated with dual selection for high cortical input combinations (Figure 10A middle panels). The model thus predicts that both MSN collaterals and FSNs are important for action selection in BG. STN, through the cortical hyperdirect pathway, is proposed to act as a stop signal (Frank, 2006), giving striatum and GPe enough time to resolve high conflict in action selection. Experiments suggest that STN is involved in the cancellation of already initiated motor responses (Eagle and Robbins, 2003; Eagle et al., 2008). Input to STN as a strong hundered milisecond pulse was injected in parallel with activation of the striatal action pools (Figure 10A bottom right panel). The model predicted that action selection at low cortical input could be stopped in line with the claim that STN is responsible for stop signaling in BG.

4.2.6 Synaptic and neural perturbations improving dynamics and function Decreasing or increasing the synaptic efficacy in around half of the modeled connections can lead to a decrease in synchrony and/or oscillations, and in 6 of them also improve action selection ability (Figure 11). Thus all connections that have an effect on reducing synchrony and/or oscillations could serve as potential treatment methods for PD. Especially interesting are the six manipulations that also improves action selection (see Figure 11C) By silencing and/or increasing the activity of specific BG nuclei, an improvement in action selection is achieved (Figure 11B-C). Lesioning GPe and STN both decrease the synchrony and oscillations in several nuclei in the network, especially lesioning STN proves more effective. Both predictions are in accordance with experiments. Lesion therapies targeting GPe, GPi and STN have successfully been used to elevate PDsymptoms (Okun and Vitek, 2004). The model predicts that synchrony and oscillations in GPe and SNr are decreased when lesioning GPe, and additionally also in STN when lesioning STN (Figure 8B). Thus it seems STN lesions are more effective than GPe lesions. Interestingly STN have been the more common target in lesion treatments of PD(Okun and Vitek, 2004). It can also be seen that increases in the activity in GPe type I and STN leads to decrease in synchrony and oscillations, which is in line with that deep brain stimulation has proven to be an effective method for relieving symptoms in Parkinson’s disease. Both decreasing and increasing MSN D2 activity counteracts synchrony and oscillations in GPe and SNr. Loss of dopamine leads to increased activity in MSN D2 neurons. Decreasing MSN D2 activity (Figure 11A) proves to be an effective measure to annihilate synchrony and oscillations in GPe and SNr. Interestingly, increasing their activity also is effective (Figure 11A). When increasing MSN D2 activity, GPe becomes suppressed thus resulting in the same output as a GPe lesion.

Figure 11 | Restoring network dynamics and function after dopamine lesion: ((A) Change in synchrony and oscillation in MSN D1, MSN D2, FSN, GPe TA, GPe TI, SNr and STN relative to the lesioned network model when silencing/increasing each connection. (B) Change in synchrony and oscillation in MSN D1, MSN D2, FSN, GPe TA, GPe TI, SNr and STN relative to lesioned network model when inhibiting/exciting each nucleus separately. A current were injected over a range of values from negative to positive in each nucleus as indicated on the x axes. Each nucleus was silent at the most hyperpolarizing current thus mimicking a lesion of the nucleus. (C) Action selection performance for a number of manipulations either to neuron excitability or to synaptic efficacy.

4.3 Learning action selection in basal ganglia Finally, in project III a very interesting but difficult topic in BG were studied, that is how actions are learned. Below are selected results presented, for more details se paper 4.

4.3.1 Action selection can be learned through Bayesian inference in basal ganglia It was found that a spiking model (Figure 12) with a learning rule utilizing Bayesian inference could learn the correct reward mapping in simulations consisting of 15 blocks of 40 trials (Figure 13) The average success was well above chance level (1/3) and approached the maximum value 1 at the end of each block Figure 14C. Thus the model had successful performance in a multiple-choice learning task with a transiently changing reward schedule. It can be seen in the Figure 14A upper panel that the D1 weight of the previously rewarded action stay high for a couple of trial after a reward mapping shift followed by a rapid decline when D2 started to suppress it and pre and post correlation are lost. It can be seen in Figure 14A lower panel how the weights in the D2 population that can suppress the previous rewarded action, grow very rapidly in the beginning of the new block. Thus the model predicted that the synaptic weight change comes first in the D2 when learning a new reward mapping, results that are in line with (Groman et al., 2011).

Figure 12 | Schematic representation of the model with relevant biological substrate. In striatum, two actions A and B are here used as an example. The population “AD1” thus corresponds to the population of MSNs in the D1 pathway coding for action A. Striosomes and matrisomes are segregated for visual guidance, but they were intermingled in the model, with striosomes often referred to as “islands” in the matrix. Thalamus and brainstem were not explicitly implemented in the model but are shown above for completeness.

Figure 13 | Raster plot of the model consisting of 3 states and 3 actions, illustrated over 30 seconds of activity during a change in the reward mapping. In (A), a single trial is detailed. In (A) and (B) the states (blue), D1 (green), D2 (red) and GPi/SNr (purple) populations, grouped by representation coding, are shown. The indices of the states and actions begin from the top. For example, neurons with an ID between 160-220 represent the D1 and D2 populations coding for action 1. In (C) both the raster plot and a histogram of the dopaminergic neurons spiking activity are displayed. The width of the bins is 10 ms. A trial lasts for 1500 ms and starts with the onset of a new state. The simultaneous higher phasic firing rates in the D1 and D2 populations correspond to the population coding for the selected action receiving inputs form the efference copy. The vertical orange dashed line signals a change in the reward mapping.

Figure 14 | Evolution of weights in the D1, D2 and RP pathways, as well as the performance over a simulation of 15 blocks with 40 trials each. In (A), the three colored lines represent the average weight of the three action coding populations in D1 and D2 from state 1. (B) represents the color coded average weights from the nine state-action pairing striosomal sub-population to the dopaminergic population. (C) displays the moving average success ratio of the model over the simulation. Vertical dashed grey lines denote the start of a new block. In (A) and (B), the black line is the total average of the plotted weights. Color-coded shaded areas represent standard deviations.

4.3.2 Synaptic homeostasis with Bayesian inference During phasic dopamine changes, synaptic modification occurred not only between the active pre- and postsynaptic populations, but also at synapses where either only the pre- or postsynaptic population were active. The relative changes in magnitude taking place between inactive units were very small. Furthermore, changes in the weights were of opposite signs for the connections between co-active neurons and connections where only one was active. These features led to some degree of homeostasis of the average weight (Figure 14B).

4.3.3 Relative importance in learning for MSN D1 and MSN D2 pathways Lesioning the MSN D2 have a more profound negative effect on success rate than lesioning MSN D1. Two simulations were run either with MSN D2 weights set to zeros or MSN D1 weights. In Figure 15 it is shown that the success ratio at the end of a learning block when lesioning MSN D1 is significantly higher than when lesioning MSN D2. The selection pattern revealed that the system with MSN D2 lesioned got stuck constantly selecting the same action: the one that was initially associated with the reward before lesioning the MSN D2 connections. Thus in one block the performance was good, around 70 percent, whereas in the two other blocks the system got stuck always choosing the wrong actions with performance close to 0 percent. MSN D2 is needed to suppress the previously selected action such that pre and post correlations are lost. It is only D2 that can learn from negative reward thus there is no change in D1 weights for two out of three blocks. Without the D2 pathway the RPE could not become positive for the remaining actions in subsequent block. It has been reported that Parkinson’s patients exhibit better learning from negative than from positive outcomes. The model predict that the MSN D2 pathway have a more beneficial effect on selection because it can preserve plasticity despite low dopamine levels. This is in line with other authors who argues that D2 pathway is more valuable since it is impacted by negative RPE (Frank et al., 2004; Cox et al., 2015). Thus our model supports observations indicating that dopaminergic medication in mild Parkinson’s patients impaired reversal learning when reversals were signaled by unexpected punishment (Swainson et al., 2000),

4.3.4 Consequences of dopamine lesioning on learning Deletion of dopamine neurons had a more severe effect on model performance, with degrading performance depending on the lesion size. Following deletion, performance immediately deteriorated for all conditions with 70 percent success rate at 16 percent lesioned, 50 percent success rate at 33 percent lesioned and below 40 percent success rate at 66 percent lesions Figure 15A-B. Thus the model predicts that dopamine depletion severely deteriorates action selection learning.

Figure 15 | Box plot of the mean success ratio and standard deviation of the examined conditions. In (A), the first 20 trials of each block were used whereas in (B), the analysis was carried out on the last 20 trials. Data from the last 7 blocks of the PD conditions were used. All differences are significant (p<.0001) unless stated otherwise (ns = non- significant). The horizontal dotted line represents chance level. For all conditions except PD33, PD66, noD2 and no Efference, differences within conditions between the first and last 20 trials are significantPD16, PD33 and PD66 display the results of the 7 blocks following the deletion of respectively 16%, 33% and 66% of the dopaminergic neurons.

4.3.5 RPE pathway affect learning rate Recall how D2 weights grow very rapidly for the previously rewarded actions right after change of reward schedule in intact model (Figure 14). We wondered if the RPE pathway somehow could be responsible for the growth speed. A simulation without the RPE pathways, now with a model that only depended on primary reward, were run. It can be seen in Figure 15A, showing the performance after 20 trials (out of 40) how the performance is well below intact model, while during the following 20 trial the performance improve and become as good as the intact model. Thus, the model predict that the RPE pathway is important for learning rate. Both the RPE and the primary reward inhibits dopamine neurons for the previously rewarded action right after reward schedule shift (since a reward is expected but a punishment is received) producing a stronger dopamine dip than with only primary reward, thus affecting D2 plasticity change more rapidly. Faster growth of D2 weight leads to faster suppression of D1 neurons of previous chosen action so that the weights of the correct D1 population can grow.

5 Conclusions and further investigations In this thesis I have explored BG function and dynamics, with different types of computational spiking neural network system level models. Three different projects were carried out and here are the main lessons learned from them:

 BG can be divided into an arbitration system, acting as a conflict resolver in a winner- take-all network, and an extension system which can modify the selection of the arbitration system by logical operations: conjunction, disjunction and negations.  Multiple synapses in BG exhibit short-term plasticity, which significantly affects signaling in the direct-, indirect- and hyper-direct pathways. Short-term plastcity enables both the direct and indirect pathway to become sensitive to bursting presynaptic MSN neurons  MSNs in striatum are controlled by three main sources of inhibition; neighboring MSNs, FSNs and GPe type A. Simulations predict that FSN and GPe type A are relatively more important at low cortical input drive whereas MSN collaterals are more important at higher cortical drive.  Simulations predict that a) the dopamine-depletion induced increases in connectivity between cortex and MSN D2; b) the decrease in connectivity in MSN-MSN; and c) the decrease in excitability in GPe neurons, are the main contributors to the enhancement of BG oscillations and synchrony.  The direct pathway enhances the dynamical range over which BG can perform selection between cortical commands, and removing either MSN collaterals or FSN to MSN inhibition reduce the dynamical range.  Simulations show that Bayesian inference together with synaptic plasticity in D1- and D2-MSNs can successfully lead to learning to select the correct actions. Here corticostriatal synapses onto D1 MSNs are only updated following increased dopamine, and synapses onto D2 MSNs are only updated for decreased dopamine signals.  The indirect pathway seems to be more important than the direct pathway, since action selection performance without the indirect pathway is worse than without the direct pathway, something which is also partly supported by literature. It is important to further understand how changes to dopamine levels affect the dynamics in BG. In the present thesis, I have only investigated how dopamine depletion affect the dynamics in BG but it would be equally important to study what happens following dopamine elevation and compare the effect to repeated injections of the dopamine stimulant amphetamine (Wang and Rebec, 1993; Waszczak et al., 2001; Gulley et al., 2004) or injection into striatum of nonselective dopamine agonists (Waszczak et al., 2002). The system level model presented in project II can be further extended to include, CM, PPN and brainstem to further challenge our hypothesis about the arbitration and extension system in a quantitatively more detailed model. It could also be used to study how the dynamics in BG during dopamine depletion are affected by a loop through CM back to striatum. This pathway has yet to be included in a detailed spiking model of BG. It would be interesting to study the dynamics of a detailed model where cortical and thalamic inputs are separated. Would thalamic input increase or decrease correlations in BG? Another line of inquiry would be to connect the more detailed BG model in project II with dopamine dependent synaptic plasticity as in project III. Thus, to include more nuclei and pathways and test weather learning of action selection can be done in a more detailed model. There is also a need to further develop the model of the reward prediction system when more experimental data becomes available. The hypothesis that BG function as an action selection device has been strengthened by the work in this thesis. It has expanded the concepts of how BG organization enables action selection and has contributed to a deeper understanding of how BG physiology and structure relate to action selection capabilities and influence network dynamics, and has furthermore proposed hypotheses of how learning of new actions in BG is realized. Still, much remain to be done, the knowledge of basal ganglia physiology and organization is constantly pushed by new experiments and future models to test the action selection hypothesis need to take into account new data. Much focus in BG research has been on models which have no learning, thus there is a

39 need to build additional models that test how actions can be learned in a biologically realistic manner.

6 References

Abbott LF, Regehr WG (2004) Synaptic computation. Nature 431:796–803. Abbott LF, Varela J a, Sen K, Nelson SB (1997) Synaptic depression and cortical gain control. Science (80- ) 275:220–224. Atherton JF, Bevan MD (2005) Ionic mechanisms underlying autonomous action potential generation in the somata and dendrites of GABAergic substantia nigra pars reticulata neurons in vitro. J Neurosci 25:8272–8281. Bahuguna J, Aertsen A, Kumar A (2015) Existence and control of Go/No-Go decision transition threshold in the striatum. PLoS Comput Biol 11:e1004233. Bar-Gad I, Havazelet-Heimer G, Goldberg JA, Ruppin E, Bergman H (2000) Reinforcement- driven dimensionality reduction--a model for information processing in the basal ganglia. J Basic Clin Physiol Pharmacol 11:305–320. Baufreton J, Bevan MD (2008) D2-like dopamine receptor-mediated modulation of activity- dependent plasticity at GABAergic synapses in the subthalamic nucleus. J Physiol 586:2121–2142. Berretta N, Nisticò R, Bernardi G, Mercuri NB (2008) Synaptic plasticity in the basal ganglia: a similar code for physiological and pathological conditions. Prog Neurobiol 84:343–362. Berthet P, Hellgren-Kotaleski J, Lansner A (2012) Action selection performance of a reconfigurable basal ganglia inspired model with Hebbian-Bayesian Go-NoGo connectivity. Front Behav Neurosci 6:65. Bevan MD, Bolam JP (1995) Cholinergic, GABAergic, and glutamate-enriched inputs from the mesopontine tegmentum to the subthalamic nucleus in the rat. J Neurosci 15:7105–7120. Bishop GA, Chan CS, Kitai ST (1982) Morphological and physiological properties of neostriatal neurons: an intracellular horseradish peroxidase study in the rat. Neuroscience 7:179–191. Bolam JP, Hanley JJ, Booth P a, Bevan MD (2000) Synaptic organisation of the basal ganglia. J Anat 196:527–542. Bracci E, Centonze D, Bernardi G, Calabresi P, Bracci E, Centonze D, Centonze D, Bernardi G, Bernardi G, Calabresi P, Calabresi P (2002) Dopamine excites fast-spiking interneurons in the striatum. J Neurophysiol 87:2190–2194. Brette R, Gerstner W (2005) Adaptive exponential integrate-and-fire model as an effective description of neuronal activity. J Neurophysiol 94:3637–3642. Bugaysen J, Bronfeld M, Tischler H, Bar-Gad I, Korngreen A (2010) Electrophysiological characteristics of globus pallidus neurons. PLoS One 5. Cepeda C, Buchwald NA, Levine MS (1993) Neuromodulatory actions of dopamine in the neostriatum are dependent upon the excitatory amino acid receptor subtypes activated. Proc Natl Acad Sci U S A 90:9576–9580. Chan CS, Glajch KE, Gertler TS, Guzman JN, Mercer JN, Lewis AS, Goldberg AB, Tkatch T, Shigemoto R, Fleming SM, Chetkovich DM, Osten P, Kita H, Surmeier DJ (2011) HCN channelopathy in external globus pallidus neurons in models of Parkinson’s disease. Nat Neurosci 14:85–92. Chuhma N, Tanaka KF, Hen R, Rayport S (2011) Functional Connectome of the Striatal Medium Spiny Neuron. J Neurosci 31:1183–1192. Comery TA, Harris JB, Willems PJ, Oostra BA, Irwin SA, Weiler IJ, Greenough WT (1997) Abnormal dendritic spines in fragile X knockout mice: maturation and pruning deficits. Proc Natl Acad Sci U S A 94:5401–5404. Connelly W, Schulz J, Lees G (2010) Differential short-term plasticity at convergent inhibitory

synapses to the substantia nigra pars reticulata. J Neurosci 30:14854–14861. Cooke S, Bliss (2006) No Title. Brain 129:1659–1673. Cox SML, Frank MJ, Larcher K, Fellows LK, Clark CA, Leyton M, Dagher A (2015) Striatal D1 and D2 signaling differentially predict learning from positive and negative outcomes. Neuroimage 109:95–101. Craik FIM, Bialystok E (2006) Cognition through the lifespan: mechanisms of change. Trends Cogn Sci 10:131–138. Creager R, Dunwiddie T, Lynch G (1980) Paired-pulse and frequency facilitation in the CA1 region of the in vitro rat hippocampus. J Physiol 299:409–424. DeLong MR (1990) Primate models of movement disorders of basal ganglia origin. Trends Neurosci 13:281–285. Dittman JS, Kreitzer AC, Regehr WG (2000) Interplay between facilitation, depression, and residual calcium at three presynaptic terminals. J Neurosci 20:1374–1385. Douglas RJ, Martin K a (1991) A functional microcircuit for cat visual cortex. J Physiol 440:735– 769. Doya K (1999) What are the computations of the cerebellum, the basal ganglia and the cerebral cortex? Neural networks 12:961–974. Eagle DM, Baunez C, Hutcheson DM, Lehmann O, Shah AP, Robbins TW (2008) Stop-signal reaction-time task performance: role of prefrontal cortex and subthalamic nucleus. Cereb Cortex 18:178–188. Eagle DM, Robbins TW (2003) Inhibitory control in rats performing a stop-signal reaction-time task: effects of lesions of the medial striatum and d-amphetamine. Behav Neurosci 117:1302–1317. Farries MA, Kita H, Wilson CJ (2010) Dynamic Spike Threshold and Zero Membrane Slope Conductance Shape the Response of Subthalamic Neurons to Cortical Input. J Neurosci 30:13180–13191. Fino E, Venance L (2011) Spike-timing dependent plasticity in striatal interneurons. Neuropharmacology 60:780–788. Fortune ES, Rose GJ (2001) Short-term synaptic plasticity as a temporal filter. Trends Neurosci 24:381–385. Frank MJ (2006) Hold your horses: a dynamic computational role for the subthalamic nucleus in decision making. Neural Netw 19:1120–1136. Frank MJ, Seeberger LC, O’reilly RC (2004) By carrot or by stick: cognitive reinforcement learning in parkinsonism. Science (80- ) 306:1940–1943. Fujiyama F, Takahashi S, Karube F (2015) Morphological elucidation of basal ganglia circuits contributing reward prediction. Front Neurosci 9:6. Fukunaga K, Stoppini L, Miyamoto E, Muller D (1993) Long-term potentiation is associated with an increased activity of Ca2+/calmodulin-dependent protein kinase II. J Biol Chem 268:7863–7867. Gerfen CR The neostriatal mosaic: compartmentalization of corticostriatal input and striatonigral output systems. Nature 311:461–464. Gerfen CR (1985) The neostriatal mosaic. I. Compartmental organization of projections from the striatum to the substantia nigra in the rat. J Comp Neurol 236:454–476. Gerfen CR (1992) The neostriatal mosaic: multiple levels of compartmental organization. Trends Neurosci 15:133–139. Gerfen CR, Herkenham M, Thibault J (1987) The neostriatal mosaic: II. Patch- and matrix-

directed mesostriatal dopaminergic and non-dopaminergic systems. J Neurosci 7:3915– 3934. Gittis AH, Hang GB, LaDow ES, Shoenfeld LR, Atallah B V., Finkbeiner S, Kreitzer AC (2011) Rapid target-specific remodeling of fast-spiking inhibitory circuits after loss of dopamine. Neuron 71:858–868. Gittis AH, Nelson AB, Thwin MT, Palop JJ, Kreitzer AC (2010) Distinct roles of GABAergic interneurons in the regulation of striatal output pathways. J Neurosci 30:2223–2234. Graybiel AM (1990) Neurotransmitters and neuromodulators in the basal ganglia. Trends Neurosci 13:244–254. Groman SM, Lee B, London ED, Mandelkern M a, James AS, Feiler K, Rivera R, Dahlbom M, Sossi V, Vandervoort E, Jentsch JD (2011) Dorsal striatal D2-like receptor availability covaries with sensitivity to positive reinforcement during discrimination learning. J Neurosci 31:7291–7299. Gurney KN, Prescott TJ, Redgrave P (2001) A computational model of action selection in the basal ganglia. I. A new functional anatomy. Biol Cybern 84:401–410. Hanson JE, Jaeger D (2002) Short-term plasticity shapes the response to simulated normal and parkinsonian input patterns in the globus pallidus. J Neurosci 22:5164–5172. Hernández A, Ibáñez-Sandoval O, Sierra A, Valdiosera R, Tapia D, Anaya V, Galarraga E, Bargas J, Aceves J (2006) Control of the subthalamic innervation of the rat globus pallidus by D2/3 and D4 dopamine receptors. J Neurophysiol 96:2877–2888. Hernández-Echeagaray E, Starling AJ, Cepeda C, Levine MS (2004) Modulation of AMPA currents by D2 dopamine receptors in striatal medium-sized spiny neurons: are dendrites necessary? Eur J Neurosci 19:2455–2463. Hodgkin a. L, Huxley a. F (1952) A quantitative description of membrane current and its application to conduction and excitation in nerve. J Physiol 117:500–544. Houk JC, Beiser D (1998) Model of cortical-basal ganglionic processing: encoding the serial order of sensory events. J Neurophysiol 79:3168–3188. Humphries MD, Lepora N, Wood R, Gurney KN (2009a) Capturing dopaminergic modulation and bimodal membrane behaviour of striatal medium spiny neurons in accurate, reduced models. Front Comput Neurosci 3:26. Humphries MD, Stewart RD, Gurney KN (2006) A physiologically plausible model of action selection and oscillatory activity in the basal ganglia. J Neurosci 26:12921–12942. Humphries MD, Wood R, Gurney KN (2009b) Dopamine-modulated dynamic cell assemblies generated by the GABAergic striatal microcircuit. Neural Netw 22:1174–1188. Huttenlocher PR, Dabholkar AS (1997) Regional differences in synaptogenesis in human cerebral cortex. J Comp Neurol 387:167–178. Izhikevich EM (2003) Simple model of spiking neurons. IEEE Trans Neural Netw 14:1569– 1572. Jaeger D, Kita H (2011) Functional connectivity and integrative properties of globus pallidus neurons. Neuroscience 198:44–53. Johansson C, Lansner A (2007) Towards cortex sized artificial neural systems. Neural Netw 20:48–61. Kamali Sarvestani I, Lindahl M, Hellgren-Kotaleski J, Ekeberg Ö, Ekeberg O (2011) The arbitration-extension hypothesis: a hierarchical interpretation of the functional organization of the Basal Ganglia. Front Syst Neurosci 5:13. Kass JI, Mintz IM (2006) Silent plateau potentials, rhythmic bursts, and pacemaker firing: three patterns of activity that coexist in quadristable subthalamic neurons. Proc Natl Acad Sci U S A 103:183–188.

Kawaguchi Y (1993) Physiological, morphological, and histochemical characterization of three classes of interneurons in rat neostriatum. J Neurosci 13:4908. Kemp JM, Powell TPS (1971) The Structure of the Caudate Nucleus of the Cat: Light and Electron Microscopy. Philos Trans R Soc B Biol Sci 262:383–401. Kita H, Kitai ST (1987) Efferent projections of the subthalamic nucleus in the rat: light and electron microscopic analysis with the PHA-L method. J Comp Neurol 260:435–452. Kita H, Tachibana Y, Nambu A (2005) Balance of monosynaptic excitatory and disynaptic inhibitory responses of the globus pallidus induced after stimulation of the subthalamic nucleus in the monkey. J Neurosci 25:8611–8619. Klaus A, Planert H, Hjorth JJJ, Berke JD, Silberberg G, Kotaleski JH (2011) Striatal fast-spiking interneurons: from firing patterns to postsynaptic impact. Front Syst Neurosci 5:57. Kravitz AA V., Freeze BBS, Parker PRLP, Kay K, Thwin MT, Deisseroth K, Kreitzer AC (2010) Regulation of parkinsonian motor behaviours by optogenetic control of basal ganglia circuitry. Nature 466:622–626. Lagarias JC, Reeds J a., Wright MH, Wright PE (1998) Convergence Properties of the Nelder-- Mead Simplex Method in Low Dimensions. SIAM J Optim 9:112–147. Lansner A, Ekeberg Ö (1989) A ONE-LAYER FEEDBACK ARTIFICIAL NEURAL NETWORK WITH A BAYESIAN LEARNING RULE. Int J Neural Syst 01:77–87. Lansner A, Holst A (1996) A higher order Bayesian neural network with spiking units. Int J Neural Syst 7:115–128. Lévesque M, Parent A (2005) The striatofugal fiber system in primates: a reevaluation of its organization based on single-axon tracing studies. Proc Natl Acad Sci U S A 102:11888– 11893. Lindahl M, Kamali Sarvestani I, Ekeberg O, Kotaleski JH (2013) Signal enhancement in the output stage of the basal ganglia by synaptic short-term plasticity in the direct, indirect, and hyperdirect pathways. Front Comput Neurosci 7:76. Lisman JE (1997) Bursts as a unit of neural information: making unreliable synapses reliable. Trends Neurosci 20:38–43. Lo C-C, Wang X-J (2006) Cortico-basal ganglia circuit mechanism for a decision threshold in reaction time tasks. Nat Neurosci 9:956–963. Markram H, Gupta A, Uziel A, Wang Y, Tsodyks M (1998) Information processing with frequency-dependent synaptic connections. Neurobiol Learn Mem 70:101–112. Maurice N, Thierry A-M, Glowinski J, Deniau J-M (2003) Spontaneous and evoked activity of substantia nigra pars reticulata neurons during high-frequency stimulation of the subthalamic nucleus. J Neurosci 23:9929–9936. McCormick D a (1989) GABA as an inhibitory neurotransmitter in human cerebral cortex. J Neurophysiol 62:1018–1027. Mena-Segovia J, Bolam JP, Magill PJ (2004) Pedunculopontine nucleus and basal ganglia: distant relatives or part of the same family? Trends Neurosci 27:585–588. Miguelez C, Morin S, Martinez A, Goillandeau M, Bezard E, Bioulac B, Baufreton J (2012) Altered pallido-pallidal synaptic transmission leads to aberrant firing of globus pallidus neurons in a rat model of Parkinson’s disease. J Physiol 590:5861–5875. Mink JW (1996) The basal ganglia: focused selection and inhibition of competing motor programs. Prog Neurobiol 50:381–425. Moran A, Stein E, Tischler H, Belelovsky K, Bar-Gad I (2011) Dynamic stereotypic responses of Basal Ganglia neurons to subthalamic nucleus high-frequency stimulation in the parkinsonian primate. Front Syst Neurosci 5:21.

Nakanishi H, Tamura A, Kawai K, Yamamoto K (1997) Electrophysiological studies of rat substantia nigra neurons in an in vitro slice preparation after middle cerebral artery occlusion. Neuroscience 77:1021–1028. Nathan T, Lambert JD (1991) Depression of the fast IPSP underlies paired-pulse facilitation in area CA1 of the rat hippocampus. J Neurophysiol 66:1704–1715. Nevado-Holgado AJ, Mallet N, Magill PJ, Bogacz R (2014) Effective connectivity of the subthalamic nucleus-globus pallidus network during Parkinsonian oscillations. J Physiol 592:1429–1455. Okun MS, Vitek JL (2004) Lesion therapy for Parkinson’s disease and other movement disorders: update and controversies. Mov Disord 19:375–389. Paille V, Fino E, Du K, Morera-Herreras T, Perez S, Kotaleski JH, Venance L (2013) GABAergic circuits control spike-timing-dependent plasticity. J Neurosci 33:9353–9363. Parent A, Hazrati LN (1995) Functional anatomy of the basal ganglia. II. The place of subthalamic nucleus and external pallidum in basal ganglia circuitry. Brain Res Brain Res Rev 20:128–154. Pawlak V, Kerr JND (2008) Dopamine receptor activation is required for corticostriatal spike- timing-dependent plasticity. J Neurosci 28:2435–2446. Planert H, Szydlowski SN, Hjorth J, Grillner S, Silberberg G (2010) Dynamics of synaptic transmission between fast-spiking interneurons and striatal projection neurons of the direct and indirect pathways. J Neurosci 30:3499–3507. Ponzi A, Wickens JR (2010) Sequentially Switching Cell Assemblies in Random Inhibitory Networks of Spiking Neurons in the Striatum. J Neurosci 30:5894–5911. Puccini GD, Sanchez-Vives M V, Compte A (2007) Integrated mechanisms of anticipation and rate-of-change computations in cortical circuits. PLoS Comput Biol 3:e82. Redgrave P, Prescott TJ, Gurney KN (1999) The basal ganglia: a vertebrate solution to the selection problem? Neuroscience 89:1009–1023. Reiner A (2010) Organization of corticostriatal projection neuron types. In: Handbook of basal ganglia structure and function (Steiner H, Tseng KY, eds), pp 323–339. London: Elsevier. Reynolds J, Wickens J (2000) Substantia nigra dopamine regulates synaptic plasticity and membrane potential fluctuations in the rat neostriatum, in vivo. Neuroscience 99:199– 203. Reynolds JNJJNJ, Wickens JR (2002) Dopamine-dependent plasticity of corticostriatal synapses. Neural Netw 15:507–521. Richards CD, Shiroyama T, Kitai ST (1997) Electrophysiological and immunocytochemical characterization of GABA and dopamine neurons in the substantia nigra of the rat. Neuroscience 80:545–557. Rinvik E, Grofová I (1970) Observations on the fine structure of the substantia nigra in the cat. Exp Brain Res 11:229–248. Rosenbaum R, Rubin J, Doiron B (2012) Short term synaptic depression imposes a frequency dependent filter on synaptic information transfer. PLoS Comput Biol 8:e1002557. Sadikot AF, Rymar V V (2009) The primate centromedian-parafascicular complex: anatomical organization with a note on neuromodulation. Brain Res Bull 78:122–130. Sandberg A, Lansner A, Petersson KM, Ekeberg Ö (2002) A Bayesian attractor network with incremental learning. Network 13:179–194. Schultz W (1997) Dopamine neurons and their role in reward mechanisms. Curr Opin Neurobiol 7:191–197. Schultz W, Dayan P, Montague PR (1997) A Neural Substrate of Prediction and Reward. Science

(80- ) 275:1593–1599. Shen KZ, Johnson SW (2000) Presynaptic dopamine D2 and muscarine M3 receptors inhibit excitatory and inhibitory transmission to rat subthalamic neurones in vitro. J Physiol 525 Pt 2:331–341. Shen W, Flajolet M, Greengard P, Surmeier DJ (2008) Dichotomous dopaminergic control of striatal synaptic plasticity. Science (80- ) 321:848–851. Sims RE, Woodhall GL, Wilson CL, Stanford IM (2008) Functional characterization of GABAergic pallidopallidal and striatopallidal synapses in the rat globus pallidus in vitro. Eur J Neurosci 28:2401–2408. Smith Y, Bevan MD, Shink E, Bolam JP (1998) Microcircuitry of the direct and indirect pathways of the basal ganglia. Neuroscience 86:353–387. Smith Y, Galvan A, Raju D, Wichmann T (2010) Anatomical andFunctional Organization if Thalamostriatal System. In: Handbook of Basal Ganglia Structure and Function (Steiner H, Tseng K, eds), pp 381–396. London: Elsevier. Stewart TC, Bekolay T, Eliasmith C (2012) Learning to select actions with spiking neurons in the Basal Ganglia. Front Neurosci 6:2. Surmeier DJ, Ding J, Day M, Wang Z, Shen W (2007) D1 and D2 dopamine-receptor modulation of striatal glutamatergic signaling in striatal medium spiny neurons. Trends Neurosci 30:228–235. Swainson R, Rogers RD, Sahakian BJ, Summers BA, Polkey CE, Robbins TW (2000) Probabilistic learning and reversal deficits in patients with Parkinson’s disease or frontal or temporal lobe lesions: possible adverse effects of dopaminergic medication. Neuropsychologia 38:596–612. Tachibana Y, Kita H, Chiken S, Takada M, Nambu A (2008) Motor cortical control of internal pallidal activity through glutamatergic and GABAergic inputs in awake monkeys. Eur J Neurosci 27:238–253. Taverna S, Ilijic E, Surmeier DJ (2008) Recurrent collateral connections of striatal medium spiny neurons are disrupted in models of Parkinson’s disease. J Neurosci 28:5504–5512. Taverna S, van Dongen YC, Groenewegen HJ, Pennartz CMA (2004) Direct physiological evidence for synaptic connectivity between medium-sized spiny neurons in rat nucleus accumbens in situ. J Neurophysiol 91:1111–1121. Tepper JM (2010) Neurophysiology of Substantia Nigra Dopamine Neurons: Modulation by GABA. In: Handbook of Basal Ganglia Structure and Function (Steiner H, Tseng KY, eds), pp 275–296. London: Elsevier. Tepper JM, Koós T, Wilson CJ (2004) GABAergic microcircuits in the neostriatum. Trends Neurosci 27:662–669. Terman D, Rubin JE, Yew AC, Wilson CJ (2002) Activity Patterns in a Model for the Subthalamopallidal Network of the Basal Ganglia. J Neurosci 22:2963–2976. Touboul J, Brette R (2008) Dynamics and bifurcations of the adaptive exponential integrate- and-fire model. Biol Cybern 99:319–334. Tsodyks M V, Markram H (1997) The neural code between neocortical pyramidal neurons depends on neurotransmitter release probability. Proc Natl Acad Sci U S A 94:719–723. Tsodyks M, Pawelzik K, Markram H (1998) Neural networks with dynamic synapses. Neural Comput 10:821–835. Tuckwell H (1988) Introduction to Theoretical Neurobiology. Cambridge: Cambridge Univ Press. Tully PJ, Hennig MH, Lansner A (2014) Synaptic and nonsynaptic plasticity approximating probabilistic inference. Front Synaptic Neurosci 6:8.

Tunstall MJ, Oorschot DE, Kean A, Wickens JR (2002) Inhibitory interactions between spiny projection neurons in the rat striatum. J Neurophysiol 88:1263–1269. Varela JA, Sen K, Gibson J, Fost J, Abbott LF, Nelson SB (1997) A quantitative description of short-term plasticity at excitatory synapses in layer 2/3 of rat primary visual cortex. J Neurosci 17:7926–7940. Wichmann T, DeLong MR (1996) Functional and pathophysiological models of the basal ganglia. Curr Opin Neurobiol 6:751–758. Zhou F-W, Jin Y, Matta SG, Xu M, Zhou F-M (2009) An ultra-short dopamine pathway regulates basal ganglia output. J Neurosci 29:10424–10435.