Effects of Multilayer Network Interactions on Neural Network

Eﬀects of multilayer network interactions on neural network dynamics

Y. S. Virkar

B.E., University of Mumbai, 2010

M.S., University of Colorado, Boulder, 2012

A thesis submitted to the

Faculty of the Graduate School of the

University of Colorado in partial fulﬁllment

of the requirements for the degree of

Doctor of Philosophy

Department of Computer Science

2017 This thesis entitled: Eﬀects of multilayer network interactions on neural network dynamics written by Y. S. Virkar has been approved for the Department of Computer Science

Prof. Juan Restrepo

Prof. Elizabeth Bradley

Prof. Aaron Clauset

Prof. Rafael Frongillo

Date

The ﬁnal copy of this thesis has been examined by the signatories, and we ﬁnd that both the content and the form meet acceptable presentation standards of scholarly work in the above mentioned discipline. iii

Virkar, Y. S. (Ph.D., Computer Science)

Eﬀects of multilayer network interactions on neural network dynamics

Thesis directed by Prof. Juan Restrepo

Networks of excitable units are found in varied disciplines such as social science, neuroscience, genetics, epidemiology, etc. Previous studies have shown that some aspects of network function can be optimized when the network operates in the ‘critical regime’, i.e., at the boundary between order and disorder where the statistics of node excitations correspond to those of a classical branching process. In this thesis, we introduce and study a mathematical model of a neural network with the goal of understanding the long-standing problem of determining the mechanisms by which a neural network regulates its activity so as to operate in the critical regime. In particular, we study the dynamics of a two-layered network model consisting of an excitable node network and a complementary network that supplies resources required for node ﬁring. More speciﬁcally, we study the dynamics of an excitable neural network consisting of neurons (nodes) connected via synapses

(edges). Synaptic strengths are mediated by resources supplied by the complementary glial cell network. Resources from the bloodstream are supplied to the glial network at some fixed rate, resources transport diffusively within the glial cell network and ultimately to the synapses, and each time a presynaptic neuron fires the resources for all outgoing synapses get consumed at some

fixed rate. We show that this natural and very compelling mechanism for feedback control can stabilize the critical state. Additionally, the neural network can learn, remember and recover the critical state after learning. The critical state is characterized by power-law distributed avalanche sizes that are robust to changes in the supply, consumption and diffusion rates. Finally, we show that our findings are fairly robust to heterogeneity in model parameters or network structure. iv

Acknowledgements

It is my immense pleasure to acknowledge and thank the many people who made this thesis possible.

First and foremost, I would like to thank my advisor, Prof. Juan Restrepo. His guidance, support and advice helped me throughout my Ph.D. degree program. I thank him for being patient with me and for making mathematics lucid and intuitive.

I would also like to thank my thesis committee members, Prof. Elizabeth Bradley, Prof.

Aaron Clauset, Prof. Rafael Frongillo and Prof. Edward Ott for taking the time from their busy schedules and being a part of my thesis committee. Their comments and suggestions helped improve this work. I would also like to thank Prof. Edward Ott and Prof. Woodrow Shew with whom Prof.

Juan Restrepo and I collaborated with for this work. It has been a pleasure working with them and I feel like I learned a lot from our weekly meetings.

I am grateful to my friends and peers for their invaluable comments on my defense presentation.

Finally, I would like to thank my grandmother, Lata Virkar and my parents, Shrikant Virkar and Manda Virkar, for raising me and for being a constant source of love and support and also my brother, Siddhesh Virkar for giving me invaluable advice throughout my life. To them, I dedicate this thesis. v

Contents

Chapter

1 Introduction 1

2 Literature review 7

2.1 Networks ...... 7

2.1.1 Network representation and terminology ...... 9

2.1.2 Multilayer networks ...... 10

2.1.3 Dynamics on multilayer networks ...... 13

2.2 Neuron anatomy ...... 16

2.3 Biological neuron models ...... 19

2.4 Simpliﬁed neuron models ...... 20

2.5 Spike-timing-dependent plasticity (STDP) ...... 30

2.5.1 Synaptic strength limitation (SSL) ...... 30

2.5.2 Synaptic scaling (SS) ...... 32

2.6 Self-organized quasi-criticality ...... 32

3 Two-layered network model 37

3.1 Background ...... 38

3.1.1 Usefulness of computational models ...... 38

3.1.2 Model 0: A computational model of learning neural networks ...... 39

3.1.3 Problems with incorporating the STDP learning rule ...... 40 vi

3.1.4 Instability of critical state with STDP learning ...... 42

3.1.5 Synaptic strength limitation (SSL) ...... 48

3.1.6 Synaptic scaling (SS) ...... 48

3.2 Model 1: Resource-transport dynamics with learning and inhibition ...... 50

3.3 Model 2: Resource-transport dynamics without learning or inhibition ...... 53

3.3.1 Simpliﬁcation of model 2 to a 3-dimensional map ...... 55

3.3.2 Analysis of 3-D map without noise ...... 57

4 Numerical Experiments 62

4.1 Model 1: Results on model with learning and inhibition ...... 62

4.2 Model 2: Results on model without learning or inhibition ...... 72

5 Conclusions & future work 84

5.1 A more realistic view of resource-transport dynamics ...... 86

5.2 Eﬀect of refractory states and time delays ...... 86

5.3 Extensions of the learning experiment...... 88

5.4 Eﬀects of complex network topologies...... 88

5.5 Quantifying critical dynamics in neural networks...... 88

5.6 Applications to machine learning ...... 89

Bibliography 91 vii

Tables

Table

2.1 Literature classiﬁcation with example references. The literature is divided into 4

topics as shown above: Balanced/Critical dynamics, STDP learning rule, Glia and

Multilayer networks. Each topic is further subdivided into multiple subtopics: Is it a

review? Does it include experimental work? Does it include mathematical modeling?

Some example references are provided for each subtopic. The goal is to make the

relevant literature easily accessible and to ﬁgure out the gaps in the literature space,

i.e., to ﬁgure out the areas that are yet to be explored: for instance, the literature

cited under the mathematical modeling subtopic for glia, reveals that there have

been no mathematical models connecting glia to the area of critical dynamics. Our

present work ﬁlls that gap...... 8

4.1 The speciﬁcations and relevant terminology for the two-layered multilayer network

consisting of neural and glial network layers. The neural layer is directed and

weighted while the glial layer is undirected and unweighted. Unless speciﬁed other-

wise, in all the experiments, we use the above speciﬁed parameter values...... 63 viii

Figures

Figure

1.1 a) Cartoon based on existing experiments, illustrating how glia distribute metabolic

resources from the bloodstream to neural synapses. Red arrows indicate paths of

metabolite transport. b) A simpliﬁed directed graph representation of our two-layer

network model. Black arrows indicate neural synaptic interactions. Red arrows ter-

minating on black arrows represent the resource supply to the corresponding synapse.

Arrow thickness indicates synaptic strength which evolves according to STDP. (Fig-

ure reproduced from [20].) ...... 3

2.1 An example network G = (V,E) is shown in panel a) has N = 6 vertices given by

the set V = {1, 2, 3, 4, 5, 6}. The edges E are represented using an adjacency matrix

that is shown in panel b). This matrix is populated using the rule given by Eq. (2.1).

For example the edges from node 1 to nodes 2, 3, 4 are represented by the entries of

1 in the ﬁrst row, i.e., the entries w21 = w31 = w41 = 1. The edges E may also be

described using a sparser representation such as the adjacency list shown in panel

c). While adjacency matrices are typically implemented in computer software using

2-dimensional arrays, adjacency lists are implemented as linked lists such that we

have a list for each vertex and this list consists of vertex ids that have edges from

the given vertex. For example, in the adjacency list for vertex 3, we put vertex ids

1, 2, 5 since we have edges to these vertices from vertex 3...... 11 ix

2.2 An example multilayer network with two layers, i.e., l = 2. The multilayer network

can be described in its entirety by G = {G, I}, where G = {G(1),G(2)} is the set

of graphs and I = {I(12),I(21)} is the set of interactions. Each graph consists of

a set of nodes and edges (as described in Sec. 2.1.1). We model the interactions

very generally as the edges from any node or edge from one layer to any node or

edge of other layers. In most studies, however, the interactions between layers are

modeled only amongst the nodes [45, 71]. Finally, we depict the interactions using

matrices I(12),I(21) shown in the bottom two panels. Note that if these matrices are

sparse (i.e., they have few non-zero entries), we could instead use an adjacency list

representation...... 12

2.3 The two-layered multiplex network of [76]. Panel (a) shows a multiplex representa-

tion where one of the layers models neural network synchronization dynamics and

the other layer models resource transport dynamics. In particular, synchroniza-

tion amongst model neurons (nodes) is modeled using the Kuramoto model (see

Eq. (2.2).) The global order parameter that quantiﬁes the amount of synchrony

amongst the neurons is depicted in panel (b). The resource transport is modeled

as a biased random walk and the fraction of random walkers at a node deﬁnes the

amount of resource at that node. Inspired from biological ﬁndings, the bias of the

random walkers is modeled such that high frequency neurons receive a higher allo-

cation of walkers and hence resource. An example random walk is depicted in panel

(c). Reprinted ﬁgure with permission from [V. Nicosia, P. S. Skardal, A. Arenas, V.

Latora APS, Physical Review Letters, 118, 138302, 2017] © (2017) by the American

Physical Society...... 14 x

2.4 A cartoon sketch showing the anatomy of a neuron. A neuron consists of three

main parts– the soma, dendrites and axons. The main cell body is called the soma.

Dendrites collect information from neighboring neuronal cells and axons transmit

information to other neurons. A synapse is a small gap or (a cleft) between the axon

terminal of one neuron and the dendrite of some other neuron...... 17

2.5 States of the biological neuron model of Hodgkin-Huxley (HH) in panel (a) and the

equivalent states in the Greenberg-Hastings (GH) model in panel (b). Kinouchi-

Copelli (KC) model [11] is one of the examples of a computational model based

on the GH model. In the KC model the transition from ready to excited state is

stochastic and depends on the amount of stimulus from neighboring neuronal cells.

Transitions from excited to refractory state 1 and so on back to the ready state are

deterministic. (Figure courtesy of Daniel Larremore [47].) ...... 18

2.6 Typical stimulus-response curves from simulations on the Kinouchi-Copelli (KC)

model (The plot shows ﬁgure 2 of [11]). Each data point (circle) is obtained from

simulating the KC model on a network with N = 105 neurons, average degree hdi =

10, number of refractory states n = 5 and time window of averaging T = 103 ms. The

solid lines corresponds to a mean-ﬁeld approximation of the model [11]. The inset

shows the response F0 for small r values. Reprinted by permission from Macmillan

Publishers Ltd: Nature Physics 2: 348-351, ©(2006)...... 22

2.7 Results of the Kinouchi-Copelli model reproduced from Figure 2 of their paper [11].

Panel (a) shows a stimulus-response curve for σ = 1.2. Panel (b) shows the dynamic

range, ∆ as a function of the local branching ratio, σ. Optimal signal processing

happens when ∆ is maximized at the critical value of σ = 1. Reprinted by permission

from Macmillan Publishers Ltd: Nature Physics 2: 348-351, ©(2006)...... 24 xi

2.8 Figure reproduced from [12]. Panel (a) shows the low-stimulus response F0 as a

function of varying mean degree, hdi. For degree-degree correlations the value of

hdi = 1 does not correctly predict the phase transition. Panel (b) shows the dynamic

range ∆ as a function of the principal eigenvalue of the network adjacency matrix,

λ. When λ = 1 the dynamic range is maximized. Reprinted ﬁgure with permission

from [D. B. Larremore, W. L. Shew, J. G. Restrepo, APS, Physical Review Letters,

106, 058101, 2011] © (2011) by the American Physical Society...... 25

2.9 (a) Stimulus response curve obtained experimentally using electric shocks as external

stimulus and response measured using multielectrode arrays [9]. (b) The dynamic

range as a function of the parameter κ (see text for deﬁnition). When the activity

is balanced between hypoexcitable state (induced by drug AP 5 and shown in blue)

and hyperexcitable state (induced by drug PTX and shown in red), the dynamic

range is maximized. Here, the dark black line shows the binned average. (Figure

adapted with permission from [4, 9])...... 28

2.10 Shows the amount of long-term potentiation (LTP) or long-term depression (LTD)

induced as a function of the diﬀerence of the timing between presynaptic (tpre) and

postsynaptic (tpost) action potentials. The qualitative and quantitative behavior

varies for diﬀerent preparations and diﬀerent species. Reprinted by permission from

Macmillan Publishers Ltd: Nature Neuroscience 3: 1178-1183, ©(2000)...... 31 xii

2.11 Self-organized criticality in a model of dynamical synapses put forth by Levina et al.

[43] (see Eqs. (2.9), (2.10)). For certain parameter regimes, the model self-organizes

to a critical state characterized by power-law distributed avalanche size distributions.

The avalanche size L is plotted on the x-axis and the distribution P (L, α) is plotted

on the z-axis. The parameter α controls the synapse strengths. In particular, varying

the coupling strength α (on y-axis) we get three diﬀerent regimes of operation: (1)

Subcritical for α < 1.3 (2) Critical for α ≈ 1.4 and (3) Supercritical for α > 1.6.

Reprinted by permission from Macmillan Publishers Ltd: Nature Physics 3: 857-860,

©(2007)...... 33

3.1 Results for the computational model in Sec. 3.1.2 using STDP learning rule. Panel

(a) shows that over time the largest eigenvalue of W , i.e., λ, goes above 1 and hence

in the supercritical state. The inset shows a magniﬁcation of the ﬁrst 1500 time

steps. Panel (b) shows that the network activity, i.e., the fraction of neurons in the

‘active’ state, S, approaches 1 corresponding to a supercritical state...... 41

3.2 Actual eigenvalue λt (solid lines) and the estimated eigenvalue λt (circles) for (k, α)

equal to (100, 0.1) (blue), (20, 0.30) (black), and (50, 0.20) (red) as a function of time

(in units of 10 time steps, for a total of 100 time steps)...... 43

3.3 Results for the computational model in Sec. 3.1.2 using STDP learning rule along

with synaptic strength limitation (SSL). Maximum synaptic strength is capped at

w¯. Panel (a) shows that over time the largest eigenvalue of W (see Sec. 2.5), i.e.,

λ, goes above its desired value of 1. Panel (b) shows that the network activity, i.e.,

the fraction of neurons in the ‘active’ state, S, goes close to 1 corresponding to a

supercritical state...... 47 xiii

3.4 Results for the computational model in Sec. 3.1.2 using STDP learning rule along

with synaptic scaling rule. Panel (a) shows that over time the largest eigenvalue of

W (see Sec. 2.5), i.e., λ goes above its desired value of 1. The inset shows a blow

up of the ﬁrst 1500 time steps. Panel (b) shows that the spread of network activity,

i.e., the fraction of neurons in the ‘active’ state, S, decreases as λ goes above 1. This

corresponds to a network state in which some neurons are always active and some

neurons are always resting...... 49

3.5 Two-layered multilayer network consisting of neural and glial network layers. The

neural layer (bottom), represented by adjacency matrix W , is directed and weighted.

The glial layer (top), represented by adjacency matrix U, is undirected and un-

weighted. The interaction between glial cells and synapses (dotted lines) is modeled

using an adjacency matrix G that maps each glial cell to some subset of all synapses. 51

−5 3.6 Assuming the parameter choicesw ¯ = 0.06, k = q = 50 and C2 = 10 , the unshaded

or white region corresponds to all inequalities being satisﬁed. The parameter choices

within this region should get us a stable critical state, or more concretely, λ = 1 over

time. The regions shaded purple and orange correspond to two of the inequalities not

being satisﬁed, namely, Eqs. (3.63) and (3.64). The inequalities given by Eqs. (3.60)-

(3.62) do not tell us anything new...... 60

3.7 We plot the largest eigenvalue of the neural adjacency matrix, λ, as a function of

time t. Panels (a), (b), (c) and (d) show the results for the four settings shown in

Fig. 3.6. Panel (a) corresponds to all inequalities being satisﬁed and hence λ → 1

over time. Panels (b), (c) violate inequality given by Eq. (3.63). Panel (d) violates

inequality given by Eq. (3.64). Hence, in these three cases, we ﬁnd that the critical

state λ = 1 is not stable...... 61 xiv

4.1 Results of Experiment 1 : (a) Time series of λt (largest eignenvalue of W t) reveal rapid

convergence to stable network dynamics (λ ≈ 1), independent of initial conditions.

Three diﬀerent initial conditions were tested: hyperexcitable (blue, λ0 = 1.5), stable

(black, λ0 = 1), and hypoexcitable (red, λ0 = 0.5). The inset is an expanded view

of the ﬁrst 5000 time steps. (b) After a longer transient the total resource R also

stabilizes to a steady value. (c) Similarly, in all three cases, the average activity S

reaches a statistical steady state with large ﬂuctuations...... 65

4.2 Results for Experiment 2 : (a) the maximum eigenvalue λ versus t, and (b) the total

resource R versus t. The data plotted in black are ‘baseline’ results obtained using

model 1 as described in Sec. 3.2. For the data plotted in red (labelled ‘instability’),

the initial evolution is the same as for the baseline data up until t = 100000 (marked

in the figure by a vertical arrow). At this point, turning off the diffusion of resources

amongst the glia results in runaway growth of λ and R...... 67

4.3 Results for Experiment 3 : We divide the neurons into two equally sized groups, G1

and G2, consisting of 500 neurons each. This results in four groups of synapses:

synapses within the ﬁrst group (within G1), synapses that convey signals from neu-

rons in G1 to neurons in G2 (G1 to G2), synapses that convey signals from neurons

in G2 to neurons in G1 (G2 to G1) and synapses within the second group (within

G2). Panel (a) depicts the learning protocol (see text). Panel (b) and (c) show λ and

R versus t. The learning regime spans t = [80000, 100000] (delimited by the vertical

arrows). Panel (b) shows that λ becomes subcritical during learning [109]. Panels

(d) shows the mean synaptic strength for the four groups of synapses for excitatory

synapses during learning. In accord with the STDP learning rule, the mean synap-

tic strength increases for G1 to G2 synapses. In the post-learning regime, spanning

t = [80000, 160000], panels (d) shows that the model remembers what it learned.

Panel (b) shows that after learning, λ quickly evolves back to the critical state λ =∼ 1. 69 xv

4.4 Time average of the largest eigenvalue, hλit, as a function of C1/Cb1, C2/Cb2 and D/Db

where Cb1, Cb2 and Db are the parameter values used for Fig.4.1-4.3. All three curves

show that this model is fairly robust to parameter changes, e.g., a 25% change in C1

or C2 yields a change in hλit of about 0.3%...... 71

4.5 Results of numerical simulations considering diﬀerent initial states of the model such

as critical (black circles), subcritical (red triangles) or supercritical (blue squares)

states. Panel (a) shows the evolution of the largest eigenvalue, λ, of the neural

network adjacency matrix. In all three cases, λ ≈ 1 after a transient period. Panel

(b) shows that the average glial resource, hRii, settles to a stable value over time. In

both panels, the dotted (subcritical case) and the dashed (supercritical case) lines

show that the predictions from the 3-D map with noise [Eqs.(3.36), (3.37), (3.39)]

qualitatively agree with the simulations from the full model for both λ and hRii... 73

−8 4.6 The critical state is robust to changes in model parameters. For a ﬁxed C2 = 10 ,

panel (a) shows the avalanche size distributions, P (L) for diﬀerent parameter settings

shown in panel (b). As shown, the exponent for the power-law ﬁt (γ) is near the

characteristic value of −3/2 for ﬁts spanning 3 (red) or 4 (blue) decades. In all

these cases, we get a plausible p-value for the power-law ﬁt found by using maximum

likelihood methods [73]...... 74

4.7 Avalanches are deﬁned as the excursions of activity S above a threshold S∗ [13]. We t X2 calculate an avalanche size using the sum St, i.e., the area under the S curve

t=t1 over the duration d = t2 − t1 of the excursion. For each avalanche we use a diﬀerent

color to depict the area under the S curve. Finally, in order to count the number

of excitations (or spikes) in an avalanche, we scale this sum by N to get the actual

avalanche size L given by Eq. (4.6)...... 76 xvi

4.8 Using heterogeneous source rates, the critical state is stable only in the presence

of resource-transport amongst the glial cells. Panel (a) shows that when there is

diﬀusion amongst glia, i.e., DG = D (blue circles), the largest eigenvalue of the

neural network adjacency matrix, λ ≈ 1. However, if resource transport amongst

the glial cells is absent, i.e. DG = 0, the neural network becomes supercritical with

λ > 1 (red triangles). The dashed line shows λ = 1 for reference. Panel (b) shows

that the total resource R reaches a steady state when diﬀusion is turned on (blue

circles) and keeps increasing over time if diﬀusion is turned oﬀ (red triangles). This

further highlights the importance of resource-transport dynamics on the stability of

the critical state...... 78

4.9 In a heterogeneous interaction network, G, the critical state is stable only if resources

are allowed to transport amongst the glial cells. Panel (a) shows that when there

is diﬀusion amongst glia (blue circles), the largest eigenvalue of the neural network

adjacency matrix, λ ≈ 1. However, if resource transport amongst the glial cells is

absent, the neural network becomes supercritical with λ > 1 (red triangles). The

dashed line shows λ = 1 for reference. Panel (b) shows that the total resource R

reaches a steady state when diﬀusion is turned on (blue circles) and keeps increas-

ing over time if diﬀusion is turned oﬀ (red triangles). This further highlights the

importance of resource-transport dynamics on the stability of the critical state. . . . 80 xvii

4.10 3-Dimensional map reproduces behavior of the full model: We show that the behavior

of the full model can be qualitatively described using the 3-dimensional map. Panels

(a) and (b) show the plots for the model simulations and the map with noise respec-

tively. The 3-D map reproduces qualitatively the plots for activity, S, the largest

eigenvalue of the neural adjacency matrix, λ, and the average glial resource, hRii,

for a particular setting of C1,C2. For this particular setting, we see oscillations for λ

and hRii with peaks for S that correspond to the rising edges of λ and hRii. Panel

for C1,C2 shown in the table in panel (d). For any given setting, since λ ﬂuctuates

about its mean value near 1, lower values of σ(λ) indicate a tighter control near criti-

cality. As seen, increasing C1 or C2 results in a higher σ(λ) and thus lesser control of

the critical state. Furthermore, there is excellent agreement between values of σ(λ)

from the 3-D map without noise (dashed black line), map with noise (blue circles)

and those obtained from model simulations (red triangles). Additionally, the grey

area highlights the region of parameter space for C1 or C2 where the critical state is

predicted to be unstable by the 3-D map without noise (see text for explanation). . . 81 Chapter 1

Introduction

A well-established hypothesis in neuroscience is that the brain operates at a critical point, poised between a phase where neuronal activity dies out and a phase where the activity saturates

[1]. One of the premises for this hypothesis is that a critical brain is most flexible and adaptable to an ever-changing external environment [2, 3], such as the environment we live in. Additionally, experiments and models have suggested that there are many functional benefits of criticality [4] such as optimized information capacity, information transfer and dynamic range that represents the range of inputs that can be encoded with high fidelity. There has been evidence for criticality in both in vivo [5, 6, 7] and in vitro preparations [8, 9, 10] as evidenced by a power-law distribution for size and duration of intermittent spontaneous bursts of neuronal activity called neuronal avalanches.

Simple computational models that treat each neuron as a cellular automaton have also shown such signatures of criticality [8, 11, 12, 13].

Previous computational models [12, 13] consider a network of neurons (nodes) connected via synapses (edges). Results using numerical simulations show the presence of critical dynamics: when the largest eigenvalue, λ, of the neural network adjacency matrix is equal to 1, neuronal avalanche sizes follow a power-law distribution with the characteristic exponent value of −3/2. In these models λ = 1 is a necessary condition for critical dynamics and this result is robust to changes in the network structure such as heterogeneity, assortativity and degree-degree correlations. However, these models lack learning. Neuronal networks have the ability to evolve: learning mechanisms allow certain pathways in the neuronal circuits to become stronger via synaptic modiﬁcation [14]. 2

In the context of the model, varying synaptic strengths over time would eﬀectively change the entries of the neural network adjacency matrix, subsequently changing λ at each time step. Thus, the following question is of signiﬁcance to neurobiologists: How does the brain maintain critical dynamics (or λ ≈ 1) in the face of learning? One of the goals of this thesis is to propose a novel network-based computational model that accomplishes this. Other goals include studying robustness of model parameters to the critical state using simulations and theory, quantifying the critical state using typical measures of criticality and studying the impact of heterogeneity in model parameters and network structure on critical neural network dynamics.

In 1949, Donald Hebb [15] put forth his postulate for a potential learning mechanism that was succinctly summarized by the American neurobiologist Carla Shatz as, “Cells that ﬁre together, wire together.” Thus a synapse between cell A and cell B is reinforced if A takes part in ﬁring

B frequently. In 1983, Levy & Steward [16] found out that the precise timing of the ﬁring is more important than the ﬁring rates. In the subsequent years, consensus developed amongst neurobiologists that ultimately led to the formation of a new learning rule called spike-timing- dependent plasticity or STDP [17]. For a historical review of STDP, refer to Markram, Gerstner

& Sj¨ostr¨om[18]. While there has been experimental evidence for STDP, incorporating it in a computational model can lead to unstable network dynamics. In chapter 3 sec. 3.1.2, we incorporate

STDP in the model put forth by Larremore et al. [13] and show that stabilizing λ near its desired value of 1 and, hence, keeping the neural network activity in the critical regime can be diﬃcult.

Subsequent sections in chapter 3 show that existing regulatory mechanisms such as limiting synaptic strengths or synaptic scaling [19] do not provide a satisfactory solution to stabilize λ ≈ 1. In this thesis, we propose and explore a novel regulatory mechanism based on how metabolic resources are distributed to the neurons by the glial cells. In what follows, we ﬁrst introduce and give a broad overview of two variants of our model termed as model 1 and model 2 and highlight our key

ﬁndings. The ﬁrst variant, i.e., model 1 has been introduced and studied in [20].

Next, we describe the motivation and the idea behind resource-transport dynamics common to both model variants. Glial brain cells play important and diverse roles regulating the dynamics 3

a blood stream b

glia

neurons

Figure 1.1: a) Cartoon based on existing experiments, illustrating how glia distribute metabolic resources from the bloodstream to neural synapses. Red arrows indicate paths of metabolite transport. b) A simpliﬁed directed graph representation of our two-layer network model. Black arrows indicate neural synaptic interactions. Red arrows terminating on black arrows represent the resource supply to the corresponding synapse. Arrow thickness indicates synaptic strength which evolves according to STDP. (Figure reproduced from [20].) 4 and structure of neural networks [21, 22], including learning-related changes in synapses [23, 24].

In this thesis we focus on one of the most important functions thought to be served by the glial network – the transport and distribution of metabolic resources among the neural synapses [25].

This hypothesis originated from early anatomical studies which showed that the glia form a bridge between the neural synapses and the brain vasculature [26] (Fig 1.1a). More recently, experiments have directly demonstrated that glia, astrocytes more specifically, deliver metabolic resources to synapses depending on how active the synapses are [27]. En route to the synapses, these resources diffuse through an extensive network of astrocytes [25]. The biophysical properties of such diffusive transport of resources may have a fundamental influence on the dynamics of the activity of the neural network that consumes the resources [28, 29, 30, 31]. For example, a highly active synapse may consume all of its local resources, thus forcing it to become less active until more resources arrive. It may also drain resources away from less active synapses, thus shaping functional differ- ences among synapses. Here, in order to study these possibilities, we introduce a computational model incorporating both a neural network and a glial network.

For model 1, the neurons interact via synapses whose eﬃcacy evolves according to activity- dependent learning rules, namely spike timing dependent plasticity (STDP) [19, 32]. Under many circumstances, modeling of STDP can result in unstable growth of synaptic eﬃcacy; additional types of learning rules must typically be added to the model to prevent such runaway growth (see

Refs. [19, 33]). The main finding of model 1 is that diffusive transport of resources via the glial network can serve to prevent runaway synaptic growth due to STDP, thereby maintaining stable neural network dynamics. We show that this phenomenon requires resource transport among the glia; locally confined production and consumption of resources result in unstable neural network dynamics. The known roles played by the glia in synaptic plasticity are diverse and numerous [34], but, to our knowledge, this work is the first to show that metabolic resource distribution can play such a stabilizing role.

More broadly, there are many examples of dynamical processes on networks in which the macroscopic network dynamics undergoes a phase transition as the strength of interactions between 5 the network nodes is increased, including synchronization [35, 36], Boolean models of gene regulation networks [37, 38], and functional brain networks [39, 40]. In some important cases, it has been argued that it is desirable for the system to operate at the onset of the phase transition. Few examples are as follows: for Boolean gene regulatory networks, it has been proposed that operating at the “edge of chaos” provides the network with enough flexibility to have a number of different, useful attractors, but without being too sensitive to perturbations [37]; for neuronal networks, it has been hypothesized that operating at a critical point where the strengths of inhibitory and excitatory synapses are balanced provides various benefits for information processing and storage, both in neuronal network models [4] and in coarser models based on synchronization of neuronal rhythms

[41]; for wireless networks, it has been suggested that operating just past a phase transition in connectivity can minimize costs while achieving operational requirements [42]. A natural question is how these networks can robustly maintain operation at the critical point without centralized control, while at the same time experiencing functional state changes, as well as changes of inputs and external environment. This long-standing question has been the subject of much interest, and various mechanisms designed so that the system’s parameters self-tune to operate at the critical point have been proposed [43, 44]. In some cases, however, there might be constraints in the dynamics of the network that either result in a net drift of the system away from the critical point, or prevent fluctuations due to noise or finite size effects from being controlled. In this thesis, we introduce a mechanism based on the transport of a resource through a secondary network which results in the stabilization of the primary network’s dynamics at the critical point. In the broad context of network science, this mechanism illustrates one benefit of the dynamical interaction between different networks [45], namely providing a novel avenue for organized criticality.

Model 2 is a simpliﬁed version of model 1 such that we do not have learning and inhibition.

Motivation for studying such a model is twofold: (1) some brain areas are involved in sensing input rather than learning and require to be in the critical state so that the dynamic range of inputs can be maximized and (2) each node in this simpliﬁed model may represent a collection of neurons, and links between nodes represent aggregate output of the collection. We can represent 6 such aggregate output using only positive synaptic weights and this results in networks that are purely excitatory. In this simpliﬁed model we show that the neuronal avalanche sizes follow a power-law distribution with λ ≈ 1 as the network self-organizes to the critical state. We examine the importance of resource-transport amongst the glial cells by studying the behavior of the model with heterogeneity in source rates or in network structure. For homogeneous networks, we derive a 3-dimensional map that qualitatively reproduces the behavior of the full system.

The rest of this thesis is organized as follows. In Sec. 2, we review the vast literature spanning the areas of critical dynamics, STDP and glia. We introduce and describe the two variants of our model in Sec. 3 and present results of numerical simulations on both variants, using Erdös-Renyi random network structure for both neural and glial networks, in Sec. 4. Our main finding is that the diffusion of resources amongst the glial cells is required to control the largest eigenvalue λ so that it remains close to 1. We show that this result is fairly robust to changes in model parameters.

We also demonstrate using model 1 that the neural network can learn, memorize what it learned and after learning recover the critical state with λ ≈ 1. Finally, we present our conclusions and discuss open problems in Sec. 5. Chapter 2

Literature review

In this section, we review some of the vast literature that spans diverse topics such as critical dynamics in experimental and computational neuroscience, learning in a neural network, the glial network, multilayer networks, etc. For a quick review of the relevant areas, we classify the literature space and provide example references in Table 2.1. We begin with a brief introduction to networks focusing on some of the recent advancements in the area of dynamics on multilayer networks. Next, we give a brief introduction to the anatomy of a neuron and how a neuron fires. This is followed by a brief history of biological neuron models– and subsequently by an introduction to simplified mathematical models of the neuron. We argue that this simplification is useful and can shed light on the collective dynamics of neural networks. In these mathematical models, we show that Hebbian

(after Donald O. Hebb [15]) learning rules can cause the network dynamics to become unstable.

We propose a novel mechanism based on transport of metabolic resources via the glial network that stabilizes the network dynamics as the network learns.

2.1 Networks

Networks or graphs occur in many disciplines of science such as physics, neuroscience, biology, chemistry, etc. A network can represent a set of arbitrary interactions amongst a set of entities.

For many years, researchers from varied disciplines have studied networks both theoretically and empirically. Networks have thus become invaluable to the understanding of complex systems. In this section, we start by describing how networks are represented in practice and some of the basic 8

Topic Subtopics Example references Balanced cortical dynamics/ ◦ Review [1, 2, 3, 4, 46, 47, 48] Critical Dynamics ◦ Experiments: in vivo or in vitro [8, 9, 49, 50, 51, 52, 53, 54] ◦ Mathematical modeling [11, 12, 13, 43, 47, 55] STDP learning rule ◦ Review [17, 18, 19, 56] ◦ Experiments: in vivo or in vitro [16, 57, 58, 59] ◦ Mathematical modeling [19, 60] Glia ◦ Review [21, 61, 62] ◦ Experiments: in vivo or in vitro [22, 25, 63, 64, 65, 66] ◦ Mathematical modeling [31, 67, 68, 69, 70] Multilayer networks ◦ Review [45, 71]

Table 2.1: Literature classification with example references. The literature is divided into 4 topics as shown above: Balanced/Critical dynamics, STDP learning rule, Glia and Multilayer networks. Each topic is further subdivided into multiple subtopics: Is it a review? Does it include experimental work? Does it include mathematical modeling? Some example references are provided for each subtopic. The goal is to make the relevant literature easily accessible and to figure out the gaps in the literature space, i.e., to figure out the areas that are yet to be explored: for instance, the literature cited under the mathematical modeling subtopic for glia, reveals that there have been no mathematical models connecting glia to the area of critical dynamics. Our present work fills that gap. 9 network terminology that we use in this thesis. Second, we review some of the literature spanning the area of dynamics on multilayer networks.

2.1.1 Network representation and terminology

A network or a graph G = (V,E) consists of set of vertices V and a set of edges E. Typi- cally, the vertices denote the entities and the edges denote the interactions between these entities.

Assuming N vertices and M edges, we can represent the network using a N × N adjacency matrix

W with entries   1 if there is an edge from vertex m to vertex n , wnm = (2.1)  0 otherwise .

This representation is typically used when the network is dense, i.e., it consists of a high number of edges (note that the maximum number of edges a network with N vertices may have is N 2).

Considering computer software implementation for networks, note that if the network is sparse storing N 2 entries for the adjacency matrix could unnecessarily take up a lot of storage space.

Hence, in practice, we use a sparser representation such as the adjacency list that consists of a N lists (one for each vertex) such that the list for vertex n stores all the vertices that have edges from node n.

An example network along with its adjacency matrix and adjacency list representations is shown in the Fig. 2.1. Since the example shown here is an undirected network, the adjacency matrix, W , is symmetric with wnm = wmn for all (m, n). Also, this network is unweighted, i.e., the edges have a uniform strength of 1 unit. In practice, this is seldom the case as interactions often have both directionality and varying strengths. In such cases the matrix W may not be symmetric and may have real-valued weights with either a positive or a negative sign. In computers, adjacency matrices may be implemented with Boolean arrays (unweighted networks) or they may be implemented with arrays that allow for signed floating point values (weighted networks). In contrast, adjacency lists are typically implemented with linked-list like data structures such that each node in the list consists of two attributes: vertex id and address of the next node in the list. In 10 the case of weighted networks, each linked-list node may be given an additional attribute that stores the weight. While the example network is a simple network, i.e., one without self-loops or multiple edges between a given pair of nodes, such a simplified representation has been quite successful to model empirical networks with skewed degree distributions [72, 73], small-world property [74] or community structure [75]. However, many real-world networks have multiple levels of interaction between the same set of entities. In what follows, we describe a different view of networks called the multilayer network that allows arbitrary levels of representation thus making networks even more applicable in empirical contexts.

2.1.2 Multilayer networks

Consider the example of social interactions amongst a set of people. The individuals in such a network may communicate via online social platforms such as Facebook as well as via offline social gatherings. The level or strength of interactions is often quite different in these two cases. Offline interactions may indicate stronger bonds as opposed to online interactions amongst the same set of individuals. Thus, depending on the final goal, one may choose to model online and offline interactions differently. This gives rise to multiple layers of interactions between the same set of entities. Such networks are called as multiplex networks. Generally, however, the entities may vary in each layer. An individual may be connected to many more individuals online than offline and some individuals may not be available on online platforms. This gives rise to a more generalized network view that is referred to as a multilayer network.

Formally, a multilayer network can be deﬁned as the pair G = (G, I) where G represents the set of graphs G = {G(1),G(2),...,G(l)} for each of the l layers in the multilayer network. Each graph can be represented as a set of vertices and edges in the usual way as: G(i) = V (i),E(i). Let

N (i) and M (i) denote the number of nodes and the number of edges respectively for layer i. Next, we deﬁne a set I that represents the set of interactions between components (nodes or edges) of a graph from one layer G(i) and the components (nodes or edges) of a graph from any other layer

G(j). Thus I = {I(12),I(13),...,I(1l),I(21),I(23),...,I(2l),...,...,I(l(l−1))}, where each entry I(ij) 11

Figure 2.1: An example network G = (V,E) is shown in panel a) has N = 6 vertices given by the set V = {1, 2, 3, 4, 5, 6}. The edges E are represented using an adjacency matrix that is shown in panel b). This matrix is populated using the rule given by Eq. (2.1). For example the edges from node 1 to nodes 2, 3, 4 are represented by the entries of 1 in the ﬁrst row, i.e., the entries w21 = w31 = w41 = 1. The edges E may also be described using a sparser representation such as the adjacency list shown in panel c). While adjacency matrices are typically implemented in computer software using 2-dimensional arrays, adjacency lists are implemented as linked lists such that we have a list for each vertex and this list consists of vertex ids that have edges from the given vertex. For example, in the adjacency list for vertex 3, we put vertex ids 1, 2, 5 since we have edges to these vertices from vertex 3. 12

Figure 2.2: An example multilayer network with two layers, i.e., l = 2. The multilayer network can be described in its entirety by G = {G, I}, where G = {G(1),G(2)} is the set of graphs and I = {I(12),I(21)} is the set of interactions. Each graph consists of a set of nodes and edges (as described in Sec. 2.1.1). We model the interactions very generally as the edges from any node or edge from one layer to any node or edge of other layers. In most studies, however, the interactions between layers are modeled only amongst the nodes [45, 71]. Finally, we depict the interactions using matrices I(12),I(21) shown in the bottom two panels. Note that if these matrices are sparse (i.e., they have few non-zero entries), we could instead use an adjacency list representation. 13 deﬁnes a matrix with dimensionality N (j) + M (j) × N (i) + M (i) such that each non-zero entry in I(ij) deﬁnes an edge (or interaction) from a component in graph j to a component in graph i.

Although in most studies (e.g., [76]) the interactions across the diﬀerent layers are modeled amongst the nodes, in general they could be modeled between nodes/edges from one layer to nodes/edges from another layer (e.g., [20]). Fig. 2.2 shows an example of a multilayer network with l = 2 such that G = {G(1),G(2)} and I = {I(12),I(21)}. Though this example shows the matrices in set I as unweighted, they may be modeled as weighted and also using a diﬀerent representation such as an adjacency list representation. In this thesis we model the interactions such that nodes in one layer connect to the edges in the other layer. For implementation we use an adjacency-list like representation to store these sparse matrices.

2.1.3 Dynamics on multilayer networks

There have been studies on dynamics on multilayer networks including spreading processes

[77], synchronization [71], evolutionary dynamics [78], etc. More relevant to our present study,

Nicosia et al. [76] study synchronization dynamics of a neural network that is bidirectionally coupled to a transport network that provides nutrient support. In particular they explore a model of a multiplex network where synchronization dynamics in one layer aﬀects and is aﬀected by the transport dynamics in the other layer. Such a multiplex network is shown in panel (a) of Fig. 2.3

(figure reproduced from [76]). In this two-layer multiplex network model of N nodes, each layer can have different topologies that govern the two types of interactions. The state of a node n in the synchronization layer is defined as xn and it denotes the phase for node n. Thus, we have a fully synchronized global state if all xn are equal. The firing rate for a node n (and hence its phase xn) depends on the amount of resource at that node. Resources are modeled using biased random walk processes across the transport layer. The state of a node n in the transport layer is defined as yn and it denotes the fraction of random walkers at that node. This fraction of random walkers in turn defines the amount of resource at that node. The movement of walkers across the transport layer defines the flow of resources. Thus modeling the state variables xn and yn allows us to study 14

Figure 2.3: The two-layered multiplex network of [76]. Panel (a) shows a multiplex representation where one of the layers models neural network synchronization dynamics and the other layer models resource transport dynamics. In particular, synchronization amongst model neurons (nodes) is modeled using the Kuramoto model (see Eq. (2.2).) The global order parameter that quantifies the amount of synchrony amongst the neurons is depicted in panel (b). The resource transport is modeled as a biased random walk and the fraction of random walkers at a node defines the amount of resource at that node. Inspired from biological findings, the bias of the random walkers is modeled such that high frequency neurons receive a higher allocation of walkers and hence resource. An example random walk is depicted in panel (c). Reprinted figure with permission from [V. Nicosia, P. S. Skardal, A. Arenas, V. Latora APS, Physical Review Letters, 118, 138302, 2017] © (2017) by the American Physical Society. 15 the impact of resource-transport on synchronization dynamics. The states xn ∈ [0, 2π) evolve as per the Kuramoto model [79]

N X [1] x˙ n = wn + K anm sin (xm − xn) , (2.2) m=1

[1] where K denotes the coupling strength, wn denotes the natural frequency and anm is an entry in the adjacency matrix A[1] for layer one. In this case, we can deﬁne the amount of global synchronization

iψ 1 PN ixn using the global order parameter re = N n=1 e depicted in panel (b) of Fig. 2.3. The ﬂow of nutrients or energy is modeled as a biased random walk process on the transport layer using the states yn that deﬁne the fraction of random walkers at node n and evolve as follows:

N [2] α ! 1 X amnχ y˙ = n − δ y . (2.3) n [2] nm m τy P α m=1 l amlχl

[2] [2] Here τy denotes the time scale, anm is an entry in the adjacency matrix A for layer two, and the [2] α amnχn variables χn deﬁne node properties that are used to bias the random walk. The term P [2] α l amlχl denotes the probability that a walker transitions from node m to node n. Additionally, the bias is controlled using a parameter α such that α > 0 (or α < 0) results in walkers moving preferentially towards nodes with higher χn (or lower χn) values as shown in panel (c) of Fig. 2.3. Finally, we can recover an unbiased random walk using α = 0.

These two separate dynamics are intertwined such that the ﬁring rate of node n depends on the availability of resource at node n (in the transport layer) and the amount of resource at node n in turn depends on the amount of local synchronization for node n (in the synchronization layer).

In particular the evolution of natural frequencies wn depends on the amount of available random walkers at node n:

1 w˙ n = [Nyn(t) − wn] . (2.4) τw

The motivation for this comes from the fact that higher oscillation frequency usually corresponds to a higher energy requirement and hence an increased blood ﬂow [80]. Thus, in this case, the frequencies wn relax to the number of random walkers with time scale τw and hence the amount of 16 resource at the respective nodes. Finally, the node parameters χn evolve

1 h dyn i χ˙ n = sn − χn (2.5) τχ

dyn dyn such that χn relax to sn with time scale τχ. The term sn measures the amount of local synchronization for node n in the synchronization layer. Since parameters χn are used to bias the random walk, coupling to the amount of local synchronization results in the probability of transition for random walkers to be biased towards strongly-synchronized nodes for α > 0 and vice-versa for

α < 0. This correctly models the notion that areas of the brain with higher electrical activity receive a higher blood inflow [81, 82] or resource. Using such coupled dynamics of resource transport and synchronization, Nicosia et al. [76] obtain interesting collective phenomena such as first-order (or explosive) synchronization and heterogeneous allocations of random walkers (resources) across the transport network. Our present work shares some of the ideas of modeling a coupled two-layer neural network where one layer provides resources required by the other layer. However our work differs from [76] in that we are interested in modeling the seemingly opposing goals of stability and learning, and critical dynamics in neural networks.

2.2 Neuron anatomy

We now turn our attention to biological neural networks and start by describing the anatomy of a neuron. A neuron is one of the fundamental working units of the brain. It is an excitable cell that is capable of transmitting information to other nerve cells. Figure 2.4 shows a cartoon of the anatomy of a neuron. As shown, each neuron has a soma (the cell body that contains the nucleus) and projections called dendrites and axons. Two neurons may be connected via a synaptic cleft

(or a synapse) that allows for signals to pass from the axon of a presynaptic (source) neuron to the dendrite of a postsynaptic (target) neuron. Thus dendrites bring information from neighboring presynaptic neurons and axons transmit information to the postsynaptic neurons. A synapse is thus a junction between the axon of a presynaptic neuron and the dendrite of a postsynaptic neuron. 17

Figure 2.4: A cartoon sketch showing the anatomy of a neuron. A neuron consists of three main parts– the soma, dendrites and axons. The main cell body is called the soma. Dendrites collect information from neighboring neuronal cells and axons transmit information to other neurons. A synapse is a small gap or (a cleft) between the axon terminal of one neuron and the dendrite of some other neuron. 18

Figure 2.5: States of the biological neuron model of Hodgkin-Huxley (HH) in panel (a) and the equivalent states in the Greenberg-Hastings (GH) model in panel (b). Kinouchi-Copelli (KC) model [11] is one of the examples of a computational model based on the GH model. In the KC model the transition from ready to excited state is stochastic and depends on the amount of stimulus from neighboring neuronal cells. Transitions from excited to refractory state 1 and so on back to the ready state are deterministic. (Figure courtesy of Daniel Larremore [47].) 19

2.3 Biological neuron models

A neuron is said to be in the resting/ready state when it is ready to accept inputs via its dendrites from neighboring neuronal cells. In this state, the cell membrane potential is roughly

−70mV; this means that the electrical potential diﬀerence between the neuron and its surrounding environment is 70mV. As the neighboring neurons ﬁre, they transmit electrical signals and the neuron accepts these inputs via the dendrites. This causes chemical changes both within the neuron and its surrounding environment and the neuron becomes more and more depolarized– causing the cell membrane potential to increase. If a neuron is depolarized above a certain threshold then there is a burst of electrical activity and the neuron transmits a spike (or an action potential) down its axon. This is called the excited state. Excitation causes subsequent chemical changes that cause the neuron to start polarizing. After some time has passed, the neuron becomes hyperpolarized as the membrane potential drops below resting state potential to roughly −82mV. In this state, the neuron is unable to accept inputs from neighboring neurons and is hence said to be in the refractory state. As time passes further chemical changes help the neuron to regain its ready state and the cell membrane potential goes back near −70mV. This cycle is depicted in panel (a) of Fig. 2.5. For a detailed review on the biology of a neuronal cell, refer to Levitan and Kaczmarek [14].

The study of biological neuron models has a long history. One of the most pioneering works was by Hodgkin and Huxley in 1952 [83, 84, 85, 86, 87]. Using both experiments and mathematical model, they studied the basis of generation and propagation of the action potential (or a spike) in the giant axon of the squid Loligo– for which they received the Nobel Prize in 1963. Their mathematical model is referred to as the Hodgkin-Huxley (HH) model and consists of a set of ordinary differential equations that describe the chemical changes in a neuronal cell that gives the typical plot for cell membrane potential depicted in Fig. 2.5(a). In 1961, Fitzhugh [88] proposed a simple two-dimensional system characterizing the spike propagation in the HH model. The following year Nagumo et al. [89] described the equivalent electrical circuit capturing the simplification of HH proposed by Fitzhugh. This simplified two-dimensional model is hence referred to as the Fitzhugh- 20

Nagumo model. Since then there have been many variants and derivatives of these models. For in-depth reviews refer to Gerstner and Kistler [90] and Izhikevich [60].

2.4 Simpliﬁed neuron models

Biological models mentioned thus far describe neuronal ﬁring in terms of changes in cell membrane potential in continuous time. We can abstract out more details by using a discrete state model such that each neuron is either in a “ready”, “excited” or “refractory” state. For the HH model these would respectively correspond to the resting state membrane potential (in which the neuron is “ready” to transmit a spike), the action potential (in which the neuron ﬁres or is “excited”) and the hyperpolarized state (in which the neuron is recovering and hence is in the “refractory” state). Such an abstraction is provided by the Greenberg-Hastings (GH) cellular automaton [91] that uses discrete states and discrete time [see panel (b) of Fig. 2.5].

While models based on the GH cellular automaton are biologically less plausible, they can still be quite useful to explain certain biological phenomena. For instance, it is known that an isolated neuron can only encode signals that span a single order of magnitude [92, 93]. However, our senses can typically process a wide range of inputs, spanning several orders of magnitude. One of the earliest works on relating the stimulus intensity (I) and sensation intensity (S) was done by

Weber, Fencher and Stevens [94]. Their law shows a nonlinear power relation between S and I as

S ∝ Iα, where the exponent α determines the growth of S with respect to I and is found to be diﬀerent for diﬀerent types of stimuli [94]. Of particular interest to neurobiologists is the question–

What neuronal mechanisms can give rise to such psychophysical laws?

Kinouchi and Copelli [11] showed that the key to answering this question lies in emergent complex dynamics owing to the interactions between coupled excitable neurons. Their model is based on the cyclic GH cellular automaton. Each excitable cell (or neuron) i = 1 ...N has n states.

t t We let si denote the state of neuron i at a discrete time step t; si = 0 denotes the ready or resting

t t state, si = 1 denotes the excited state and the remaining si = 2, . . . n − 1 denote the refractory

t states. This is depicted in panel (b) of Fig 2.5. For each neuron the transition from ready (si = 0) 21

t+1 to excited state (si = 1) is stochastic and is governed by two things– (1) an external stimulus modeled by a Poisson process with rate r that implies a transition probability 1 − e−r∆t and (2) a

t−1 probability pij due to neighbor j being in the excited state in the previous time step (sj = 1). This latter point refers to the stimulus received from the neighboring neurons. In their original work, neurons are connected in an undirected Erd¨os-Renyi random network. Once in the excited state, a neuron deterministically goes through the n refractory states and back to the ready state.

Transitions happen in discrete time steps (with the assumption that ∆t = 1 ms). A local branching ratio σj refers to the expected number of excitations caused by neuron j in the next time step

N X σj = pij . (2.6) i=1

The average of σj over the network nodes is called the branching ratio and is denoted by σ. When

σ < 1 it is expected that the network activity dies out. Similarly, when σ > 1 the network activity saturates. At σ = 1 one neuron, on average, excites one other neuron and hence we can think of this as a balanced cortical state. The network activity at time t, St, is deﬁned as the fraction of

t −1 P t neurons in the excited state, i.e., S = N i si. The time averaged activity over a large time window T is deﬁned as:

T X F = T −1 St . (2.7) t=1

In this model, one is interested in the network response F to a stimulus represented by r, the rate of external stimulation to each node. Typical stimulus-response curves for different values of σ are depicted in figure 2.6. In general, the network response F (on the y-axis) increases with stimulus r (on the x-axis). If we let F0 denote the response in the limit r → 0, the inset shows that for very low σ, F0 ≈ 0. This corresponds to the black lines of the figure. For high σ values, F0 is non-zero and the blue curves show that the network activity is ceaseless and in general high for all r values. There is a phase transition for F0 at a critical value of σ beyond which the network activity becomes self-sustained even in the absence of external stimulus. This corresponds to the red curve with σ = 1. Fig. 2.6 also shows a mean-field approximation of the model (solid lines) that 22

Figure 2.6: Typical stimulus-response curves from simulations on the Kinouchi-Copelli (KC) model (The plot shows figure 2 of [11]). Each data point (circle) is obtained from simulating the KC model on a network with N = 105 neurons, average degree hdi = 10, number of refractory states n = 5 and time window of averaging T = 103 ms. The solid lines corresponds to a mean-field approximation of the model [11]. The inset shows the response F0 for small r values. Reprinted by permission from Macmillan Publishers Ltd: Nature Physics 2: 348-351, ©(2006). 23 provides a good fit to the results from simulations (circles). A notable aspect of this result is that for low stimulus values, the exponents m obtained for the theoretical solid lines are correctly fitted for the Weber-Fechner-Stevens law (F ∝ rm); with 1 ≤ m < 1/2 representing the subcritical state, m = 1/2 representing the critical state and m < 1/2 representing the supercritical state. Thus independent of the sophistication of the individual neuron model, the overall KC model provides a result for their collective dynamics that is of significant importance to neurobiologists.

The value of σ = 1 is special for another reason: it corresponds to optimal signal processing at a phase transition (although, as discussed below, this applies only to homogeneous networks).

Consider a typical response curve for high σ value, σ = 1.2, shown in figure 2.7(a). For low values of r, F0 remains low but does not drop to 0 owing to σ > 1. As r is increased systematically the response F increases and finally saturates at Fmax. The plot shows a range of values [F0.1,F0.9] where F is very sensitive to changes in r. Note that below F0.1 and above F0.9 the network response does not vary much and hence the neural network cannot encode r efficiently in those regimes. The range [F0.1,F0.9] is calculated as 10% − 90% of the full range [F0,Fmax] [11]. The corresponding

−1 range [r0.1, r0.9] is calculated by using the inverse function F and represents the range of stimuli that the neural network can encode with high ﬁdelity. This is called the dynamic range and is calculated in decibels (dB) using the following formula:

∆ = 10 log10(r0.9/r0.1) . (2.8)

Fig 2.7(b) shows the plot for dynamic range as a function of the mean branching ratio, σ, and shows why σ = 1 is a special value. The dynamic range is maximum when the neural network is in a balanced cortical state (σ = 1) as one neuron on average excites one other neuron. This is also referred to as the critical state. The dynamic range drops for values of σ < 1 where activity is low

(sub-critical state) and σ > 1 (super-critical state). This result is robust to changes in parameters such as the average degree hdi, the number of refractory states n, etc.

Larremore, Shew and Restrepo [12] showed that this result, however, is not robust in networks with a heterogeneous degree distribution, degree-assortativity or degree-degree correlations. For 24

Figure 2.7: Results of the Kinouchi-Copelli model reproduced from Figure 2 of their paper [11]. Panel (a) shows a stimulus-response curve for σ = 1.2. Panel (b) shows the dynamic range, ∆ as a function of the local branching ratio, σ. Optimal signal processing happens when ∆ is maximized at the critical value of σ = 1. Reprinted by permission from Macmillan Publishers Ltd: Nature Physics 2: 348-351, ©(2006). 25

Figure 2.8: Figure reproduced from [12]. Panel (a) shows the low-stimulus response F0 as a function of varying mean degree, hdi. For degree-degree correlations the value of hdi = 1 does not correctly predict the phase transition. Panel (b) shows the dynamic range ∆ as a function of the principal eigenvalue of the network adjacency matrix, λ. When λ = 1 the dynamic range is maximized. Reprinted ﬁgure with permission from [D. B. Larremore, W. L. Shew, J. G. Restrepo, APS, Physical Review Letters, 106, 058101, 2011] © (2011) by the American Physical Society. 26 instance, Fig. 2.8(a) shows the results for the KC model (circles): the response in the r → 0 limit,

F0, is shown on the y-axis as the mean degree, hdi, is varied on the x-axis. Thus each circle denotes a diﬀerent network structure. The model and approximations presented in [12] are depicted by solid and dashed lines respectively. In the presence of degree-degree correlations, it can be seen that the value of hdi = 1 does not correctly predict the phase transition of F0. In fact it can be shown that this phase transition occurs when the largest eigenvalue of the network adjacency matrix,

λ, is exactly one. Network structures having this property are marked by the red arrows. Thus

λ = 1 correctly predicts the phase transition. Finally, the dynamic range of the network response is maximized when the adjacency matrix has λ = 1. Fig. 2.8(b) (reproduced from [12]) shows that the curve for ∆(λ) is qualitatively similar to the one in the KC model (see Fig. 2.7(b)) except that the x-axis denotes the principal eigenvalue of the adjacency matrix as opposed to the branching ratio. This result is very robust and holds for networks with homogeneous and heterogenous degree distributions, varying the number of refractory states, presence of delays [95] and the inclusion of inhibitory nodes [13, 96].

The critical value of the branching ratio σ = 1 (or, more generally, λ = 1) has other important implications for the collective dynamics of a neural network. More speciﬁcally it has been found both experimentally and using simulations [8, 9], that when σ = 1, the distribution of avalanche sizes shows no characteristic scale: it follows a power-law distribution. A power law distribution has the mathematical form– P (x) ∝ x−γ, where x is the event (in this case avalanche size) and γ is the exponent. An avalanche is deﬁned as the cascade of activity that follows an initial stimulus.

t The avalanche size can be deﬁned as the sum of the neural activity S between t = [T1,T2] such that T1 indicates the start time of the network activity owing to the input stimulus and T2 indicates the time at which network activity dies out.

Using multielectrode arrays, Beggs and Plenz [8] recorded neuronal activity in cultured and acute (short-living) rat slices (see Fig. 1A of [8]). They deﬁned the branching ratio as the average number of electrodes that are active in the current time interval, given that one electrode was active in the previous time interval. To take into account the refractoriness at individual electrodes, they 27 approximate the branching ratio using only the ﬁrst and the second time interval. When the branching ratio was held near its critical value, they observed neuronal avalanches with avalanche sizes following a power-law distribution with the characteristic exponent value of γ = −3/2. This power-law distribution has been later supported by statistical analyses as shown in [97]. This is depicted by the dashed line in Fig. 4F of [8]. For varying the total number of electrodes n = {15, 30, 60}, they get power-law distributions for avalanche sizes each following γ ≈ −3/2.

This signature of criticality has also been found in other experimental works both in vivo [5, 6, 7] and in vitro [9, 10].

Shew et al. [9] showed experimentally that network activity has to be balanced delicately between the two extremes of subcritical and supercritical states in order for the neuronal circuit to exhibit maximum dynamic range. This was the first experimental evidence of this potential information handling benefit of criticality. To test the deviations from criticality, they defined a

1 Pm T E measure κ = 1 + m k=1 F (Sk) − F (Sk) where F measures the fraction of avalanche sizes, s < S, i.e. the cumulative density function (CDF). The parameter κ measures the diﬀerence between empirical distribution F E and the theoretical reference CDF given by F T in such a way that a perfect power law with the desired −3/2 exponent for the empirical avalanche size distribution gives κ = 1. An empirical distribution characteristic of the subcritical state (e.g. exponential) gives

κ < 1, and one characteristic of the supercritical state gives κ > 1. Fig. 2.9 shows the results. Panel a) shows an example of a typical stimulus-response curve. The external stimulus was provided by electrical shocks and the response was measured using microelectrode arrays. Panel b) shows the dynamic range (∆) as a function of κ. The presence of a drug such as AP 5 (blue) makes network activity subcritical as it suppresses a glutamate receptor (responsible for excitations), while the presence of a drug such as PTX (red) results in inhibition of GABA receptor (responsible for inhibition), thus making the network activity supercritical. As seen from panel b) the dynamic range is maximized in the absence of drugs (black symbols) when the activity is well balanced between hypoexcitable and hyperexcitable states.

Finally, a general criterion for such signatures of criticality has been obtained for compu- 28

Figure 2.9: (a) Stimulus response curve obtained experimentally using electric shocks as external stimulus and response measured using multielectrode arrays [9]. (b) The dynamic range as a function of the parameter κ (see text for deﬁnition). When the activity is balanced between hypoexcitable state (induced by drug AP 5 and shown in blue) and hyperexcitable state (induced by drug PTX and shown in red), the dynamic range is maximized. Here, the dark black line shows the binned average. (Figure adapted with permission from [4, 9]). 29 tational models using excitatory [98] and inhibitory [13] nodes: when the largest eigenvalue of the adjacency matrix is one, i.e., λ = 1, we see avalanche sizes being power-law distributed with

γ = −3/2. This result has been shown to be robust to homogenous and heterogenous degree distributions, degree assortativity and degree-degree correlations [12]. Thus, transition from low to high activity, a power-law distribution of avalanche sizes, and a maximized dynamic range coexist both in experiments and in simpliﬁed mathematical models.

Computational models considered thus far explain some of the neurobiological phenomena such as optimized dynamic range, statistics of neuronal avalanches, etc. However, they do not include learning: the ability of neural circuits to adapt to a constantly changing external environment.

Synaptic modification is the basis for learning and memory. From the perspective of computational models, synaptic modification could result in a constantly varying largest eigenvalue λ of the neural network adjacency matrix. Since λ = 1 (critical state) corresponds to a balanced cortical state and provides important functional benefits [4] such as optimal dynamic range, information transfer and information capacity, one question of interest is: How does the brain maintain the desired balanced cortical state in the face of constant synaptic modifications? In this thesis, we introduce and study a novel mathematical model of a neural network to explore a possible solution to this long-standing question. Before presenting our approach, we review some of the pertinent learning mechanisms.

In 1949, D. O. Hebb [15] put forth his postulate for learning or synaptic modiﬁcation: “When an axon of a cell A is near enough to excite a cell B and repeatedly and persistently takes part in

firing it, some growth or metabolic change takes place in one or both cells such that A’s efficiency, as one of the cells firing B, is increased”. In other words, if A takes part in firing B frequently, the strength of the synapse from cell A to cell B is increased. If we consider a network of neurons and synapses, the Hebbian learning rule can be applied independently at any synapse thus making it quite powerful. The downside, however, is that synaptic modification across the neural network becomes difficult to manage. Without regulatory mechanisms, Hebbian learning can cause synaptic strengths to grow or shrink uncontrollably. Abbott and Nelson [19] review some of these regulatory mechanisms that could potentially be used to control runaway synaptic strengths. We review these 30 regulatory mechanisms and put them in the context of computational models such as the models of Kinouchi & Copelli [11] and Larremore et al. [12, 13].

2.5 Spike-timing-dependent plasticity (STDP)

In 1983, Levy and Steward [16] studied the neuronal cells in the EC (Entorhinal Cortex) hippocampus system, an area of the brain that is primarily responsible for memories. They varied the relative timing of inputs from one part of the EC to another part (dentate gyrus) and studied its effects on synaptic modification. They found that synapses were reinforced (potentiated) if a weak input preceded a strong input and the synapses weakened (depressed) if the ordering of the inputs were reversed. This was the first evidence of timing dependence on Hebbian learning. The long-term potentiation (LTP) and depression (LTD) thus depend on temporal order of firing and this result varies across preparations and species as shown in Fig. 2.10, reproduced from [19]. Panel

(a) shows that for layer-5 neocortex preparation if the tpre, i.e., timing of the presynaptic action potential (spike) is less than tpost, i.e., timing of the postsynaptic action potential (spike), then LTP is induced. The amount induced is more for a smaller diﬀerence tpre −tpost. This behavior of STDP is qualitatively diﬀerent for other cases; for instance, panel (c) shows that the LTD as opposed to

LTP is induced in the ELL (electrosensory lateral line) of electric ﬁsh if tpre < tpost. Further studies by Magee & Johnston [58], Markram et al. [57] and Bi & Poo [59] were in agreement with these results. For a comprehensive review refer to Caporale & Dan [17] and Markram et al. [18].

2.5.1 Synaptic strength limitation (SSL)

In most modeling studies, synaptic strengths are capped oﬀ at a maximum value [99]. From a biological perspective, having an upper bound for synapse strength makes sense as one can imagine a limit on the amount of neurotransmitter or physical dimensions of a neuron. In chapter 3, we demonstrate how this rule is inadequate to control runaway synaptic growth in computational models considered in this thesis. 31

Figure 2.10: Shows the amount of long-term potentiation (LTP) or long-term depression (LTD) induced as a function of the difference of the timing between presynaptic (tpre) and postsynaptic (tpost) action potentials. The qualitative and quantitative behavior varies for different preparations and different species. Reprinted by permission from Macmillan Publishers Ltd: Nature Neuro- science 3: 1178-1183, ©(2000). 32

2.5.2 Synaptic scaling (SS)

Another mechanism to control runaway excitability is to globally adjust all synapses incident on a given postsynaptic neuron according to its firing rate. This mechanism is called synaptic scaling. It was first observed in cultured neocortical neurons by Turrigiano et al. [100]. They studied a cultured neural network generating robust spontaneous activity, which consisted of both excitatory and inhibitory neurons. Blocking a fraction of inhibitory neurons, they observed the network firing rates over a time scale of many hours. They found that although the firing rates initially increase (owing to the blocking of inhibition), after many hours the firing rates return back to the control values.

There are in general two ways of scaling input synaptic strengths [99]– (i) Additive: In this case, synaptic strengths are changed by the same amount. (ii) Multiplicative: Changes are proportional to the synaptic strengths. In the long run, additive adjustments result in a maximum value for synapse strength; this well modeled by the SSL rule described in the previous section. The synaptic scaling rule described above, is an example of a multiplicative adjustment. In chapter 3, we discuss the issues in incorporating this control mechanism in the computational models considered here.

2.6 Self-organized quasi-criticality

Levina, Hermann and Geisel [43] introduced a model that allows for a regulatory feedback control of network dynamics resulting in self-organization to criticality. In particular, they deﬁne a model of N integrate-and-ﬁre neurons that are characterized by membrane potential hi, where i ∈ {1,...,N}. At any given time t, if the membrane potential exceeds a threshold θ, i.e., hi(t) > θ, then neuron i transmits a spike and its membrane potential, hi, is reset by subtracting the amount

θ. At time t, a random external stimulus process ξτ (t) ∈ {1,...,N} selects neuron i, ξτ (t) = i

ext with rate τ. If i is selected, its membrane potential hi increases by an amount I . Additionally, there is some delay in synaptic transmission that is denoted by τd. The dynamics of the membrane 33

Figure 2.11: Self-organized criticality in a model of dynamical synapses put forth by Levina et al. [43] (see Eqs. (2.9), (2.10)). For certain parameter regimes, the model self-organizes to a critical state characterized by power-law distributed avalanche size distributions. The avalanche size L is plotted on the x-axis and the distribution P (L, α) is plotted on the z-axis. The parameter α controls the synapse strengths. In particular, varying the coupling strength α (on y-axis) we get three diﬀerent regimes of operation: (1) Subcritical for α < 1.3 (2) Critical for α ≈ 1.4 and (3) Supercritical for α > 1.6. Reprinted by permission from Macmillan Publishers Ltd: Nature Physics 3: 857-860, ©(2007). 34 potential can be thus described as follows:

N 1 X h˙ = δ Iext + uJ δ(t − tj − τ ) (2.9) i i,ξτ (t) N ij sp d j=1

Here, the first term denotes the external stimulus while the second term defines the intrinsic network dynamics. In particular, Jij defines the strength of the synapse from neuron j to neuron i in terms of some resource such as the amount of neurotransmitter. The dynamics of Jij is as follows. If the presynaptic neuron j spikes, Jij decreases by a fraction u; this is intended to model consumption of neurotransmitter that is required for spiking. In the absence of spiking, Jij relaxes to a value

α/u with a slow time scale, τJ = τνN, where 1 < ν N. This results in the following update equation for Jij:

˙ 1 α j Jij = − Jij − uJijδ(t − tsp) (2.10) τJ u

Note that the parameter α controls the synaptic strength. Varying this single parameter allows us to explore diﬀerent regimes of operation. Fig. 2.11 shows the avalanche size distributions P (L, α) as a function of parameter α. As shown, for a suﬃciently wide range of α (red) the model self- organizes to a critical state characterized by P (L, α ∈ (1.3, 1.6)) ∝ L−3/2. For values of α < 1.3, the model operates in the subcritical regime and for values of α > 1.6, the model operates in the supercritical regime. In this thesis, we show that the models we propose also self-organize to the critical state. For the model described above, the critical state is stable for a narrow range of

α ∈ (1.3, 1.6). In contrast, we show in chapter 4 that for the models we introduce in this thesis in chapter 3, the critical state is stable for a large range of parameter settings.

Bonachela and Mu˜noz[101] showed that the scale-invariance found in slowly-driven systems without a conservation law (such as the one discussed above) is not true scale-invariance. Instead, in such cases, the system merely oscillates about the critical state with amplitudes based on the rates of loading and dissipating mechanisms, i.e., in this case, the rate of resource replenishment in the absence of spiking and the rate of resource consumption in the presence of presynaptic spike.

They call this quasi-criticality, since the system oscillates about the critical point with some am- 35 plitude that does not vanish in the large N limit.

In this chapter, we have reviewed the pertinent literature focussing on diverse topics such as critical dynamics, spike-timing-dependent plasticity and the mechanisms used to control runaway excitation. We now discuss in brief the motivation behind the approach we use in this thesis.

The brain makes up only 2% of human body weight, but is responsible for over 20% of energy consumption [102, 103]. Most of this energy is consumed at the synapses both during on- going neural network dynamics and during learning. This energy or resource consumption impacts learning processes such as the STDP. Delattre et al. [33] studied the STDP learning rule on acute slices of juvenile rat somatosensory cortex. Their study revealed that the neural network activity ﬂips STDP-induced LTP to LTD owing to the depletion of metabolic resources. They model a resource-dependent STDP learning rule that demonstrates a self-organized critical state for neuronal activity. However, their mathematical model assumes a global resource that varies according to the dynamics of STDP learning. This model, while useful, is somewhat unrealistic.

In this thesis, we consider a more general case, in which the resources transport via secondary glial network with a particular network structure.

It is well-known that glial brain cells play important and diverse roles regulating the dynamics and structure of neural networks [21, 22], including the transport and distribution of metabolic resources among the neural synapses [25]. More recently, experiments have directly demonstrated that glia astrocytes, in particular, deliver metabolic resources to synapses depending on how active the synapses are [27]. En route to the synapses, these resources diﬀuse through an extensive network of astrocytes [25]. It is thus natural to incorporate the glia network to model a resource-dependent

STDP learning rule.

In what follows, we describe two variants of a multilayered neural network model whose

ﬁrst network layer is formed by neurons (nodes) connected via synapses (edges) and the second network layer is formed by glial cells (nodes) connected via gap junctions (edges). The glial cells provide resources to the synapses in the ﬁrst layer, enabling them to transmit neuronal excitations. 36

Resources are transmitted diﬀusively between the glial cells and between glial cells and synapses.

For model 1, the ﬁrst variant, the neural network learns via STDP. Similar to Delattre et al. [33], our results have shown that our model can control STDP learning dynamics. In addition, we also have found that our model can learn, store a learned pattern and recover the desired λ ≈ 1 state after learning. The second variant, i.e., model 2 does not include STDP learning and inhibition.

This simpliﬁcation allows us to derive a 3-dimensional map that can reproduce the behavior of the full system and correctly predict the good and bad parameter regimes. In this way, we gain some insight into stability of critical state that is characterized by power-law distributed neuronal avalanches. Finally, we show that the critical state is stable to heterogeneity in model parameter and network structure so long as resources are allowed to diﬀuse amongst the glial cells. Chapter 3

Two-layered network model

In this section, we ﬁrst motivate the need for a regulatory mechanism that can mediate synapse strengths in the presence of learning rules such as the spike-timing-dependent plasticity rule

(STDP). In particular, we include STDP in the existing cellular automata model by Larremore et al.

[13] in Sec. 3.1.2 and show using model simulations (see Sec. 3.1.3) and theory (see Sec. 3.1.4) that the neural network can become supercritical. In the subsequent sections (Sec. 3.1.5 and Sec. 3.1.6), we show that implementing some of the existing regulatory mechanisms introduced in the previous chapter such as synaptic strength limitation (SSL) and synaptic scaling (SS) does not help stabilize the critical state. We call these variants of the adapting the model from [13] collectively as model

In Sec. 3.2 and Sec. 3.3, we introduce two models based on the idea of resource transport via a complementary network that we outlined in previous chapter. These two models operate at different levels of abstraction. In the first model (called as model 1 ), we consider the effects of learning and inhibitory neurons and show how resource-transport dynamics can achieve the competing goals of stability and learning. In this case, we can think of each node in our network as an individual neuron. This work, the topic of Sec. 3.2, was published in Ref. [20]. An alternative view is to think of each node as a functional group of neurons. In this case, a signal transmitted from a node can be viewed as the result of aggregate activity of a collection of neurons and hence can be modeled as a positive number leading to purely excitatory coupling. In the second model (called as model

2 ), the topic of Sec. 3.3, we show that the critical dynamics of such excitable networks can be 38 stabilized using resource transport.

3.1 Background

3.1.1 Usefulness of computational models

Computational models are not only useful to gain insights but also to predict something about the natural phenomena they are intended to describe. Additionally, with modern computers, we can quickly simulate the model dynamics and predict model behavior to compare with actual experiments or nature.

As an example, we reviewed models describing diﬀerent aspects of a neural network and operating at diﬀerent scales in chapter 2. For instance, the pioneering work of Hodgkin-Huxley

[83, 84, 85, 86, 87] describes how a spike propagates by modeling the chemical changes at individual neurons. In contrast, the Kinouchi-Copelli model [11], based on simple cellular automata models, describes the collective dynamics of a neural network. Hence, in this model, the details of how individual neurons ﬁre are abstracted out making the model very simple. Despite this simplicity, the model correctly predicts that at criticality the neural network can maximize the dynamic range.

These model predictions qualitatively agree with experimental ﬁndings using rat cortical slices [9].

In this section, we describe the two main variants of our model based on such simple cellular automata models. While our models are simple, they may be useful to gain intuitions on future experimental work. Our main point here is to show that the transport of resources amongst glia may be an important feature of the brain that allows it to stabilize the desired critical state. More recently, there have been experiments that highlight the importance of resource-transport amongst glia [21]. In particular, these experiments use genetically modiﬁed rats devoid of gap-junctions, i.e., the connections between the glial cells. In some pathological cases, signs of epilepsy [104] and hypoglycemia [25] have been found. In the next chapter, numerically simulating model 1 and model

2 described in Sec. 3.2 and Sec. 3.3 respectively, we obtain similar qualitative results.

Finally, we note that such simple mathematical models of neural networks also form the basis 39 for artiﬁcial neural networks used in diﬀerent areas of machine learning such as deep learning [105].

While the learning algorithms for biological and artiﬁcial neural networks are rarely same, the idea of activations and activity propagation uses similar simple mathematical simplicity.

3.1.2 Model 0: A computational model of learning neural networks

In this section, we describe an adaption of one of the computational models, namely, the model of Larremore et al., [13] to the case of learning. This model considers neuronal dynamics on a sparse network of N nodes that represent the neurons, m = {1,...,N}. The parameter α denotes the fraction of inhibitory neurons. Activation of inhibitory neurons (with m = −1) causes neighboring neurons to ﬁre with a lesser probability and activation of excitatory neurons (with

m = 1) causes neighboring neurons to ﬁre with a greater probability. In this model, unlike the

Kinouchi-Copelli model [11], there are no refractory states: at each discrete time step t any node

t t m can only be in one of the two states: sm = 0 (resting state) or sm = 1 (active state). We now modify the existing model [13] by incorporating learning. We do this by deﬁning a time-varying

t N × N network adjacency matrix, W , whose entry Wnm gives the strength of a synapse from node

t m to node n at time step t. If we let wbnm represent the absolute strength of the synapse from t t m → n, then Wnm = mwbnm. Thus the presynaptic neuron m is either excitatory or inhibitory t t t resulting in Wnm ≥ 0 or Wnm ≤ 0 respectively; Wnm = 0 indicates absence of a synapse from node

t m to n. At each time step t, neuron n sums its inputs Wnm from active presynaptic neurons, i.e.,

t neurons m for which sm = 1. If the sum is positive, neuron n ﬁres in the next time step with some probability. Thus to get the state in the next time step we use the following rule:

 N !  X t t  1 with probability σ Wnmsm , t+1  sn = m=1 (3.1)   0 otherwise .

The model transfer function, σ, is piecewise linear; σ(x) = 0 for x ≤ 0, σ(x) = x for 0 < x < 1 and

σ(x) = 1 for x ≥ 1. We thus have a stochastic model- in presence of net excitatory input the sum is greater than 0 and neuron n ﬁres with some probability while in the presence of net inhibitory 40

t input the sum is less than 0 and neuron n does not ﬁre. The synaptic strengths Wnm undergo spike-timing-dependent plasticity (STDP), a learning rule that strengthens the synapse from cell

A to cell B depending on the their temporal order of ﬁring.

In what follows, we review the STDP learning rule along with a few control mechanisms such as synaptic strength limitation (SSL) and synaptic scaling. We present results on simulating the model described above and explain why the existing control mechanisms are not a satisfactory solution to control the neural network dynamics for the considered simpliﬁed mathematical model.

3.1.3 Problems with incorporating the STDP learning rule

To incorporate STDP in model 0 described in the previous section, we simplify STDP learning dynamics by considering discrete time: the absolute strength of synapse from neuron m to n, i.e.,

t−1 wbnm, increases at time t + 1, if neuron m is active (sm = 1) at time step t − 1 and neuron n is t active (sn = 1) at time step t. The converse is true if the temporal order of ﬁring is reversed. That

t t−1 is, if we have sm = 1 and sn = 1, then wbnm decreases at time t + 1. The STDP learning rule can be modeled as follows:

m t−1 t t t−1 t+1 t (sm sn−smsn ) wbnm = wbnme τ , (3.2) where the constant τ represents the learning timescale.

We now present the results of this model on an Erd¨os-Renyi random graph with the number of neurons N = 1000. We build the N × N directed weighted ER neural network adjacency matrix

W by creating a link from node m to node n (i.e., setting Wnm 6= 0) with probability p = 0.05 and setting Wnm = 0 otherwise. This gives the mean degree of a neuron, d = (N − 1)p ≈ Np = 50 and the expected number of edges M ≈ Nd = 50000. To specify the initial state of each synapse, at

0 t = 0 we set the initial value of each wbnm to be an independent draw from a uniform distribution over [0, 1]. We then rescale all entries in W by a constant to obtain a desired largest eigenvalue of W , λ. We set the fraction of inhibitory nodes α = 0.2, the learning timescale τ = 50, and the initial state of the network at criticality, i.e., λ0 = 1. Finally we recall that the average activity, 41

1 (b) 0.8

0.6 S 0.4

0.2

0 500 1000 1500 2000 t

Figure 3.1: Results for the computational model in Sec. 3.1.2 using STDP learning rule. Panel (a) shows that over time the largest eigenvalue of W , i.e., λ, goes above 1 and hence in the supercritical state. The inset shows a magniﬁcation of the ﬁrst 1500 time steps. Panel (b) shows that the network activity, i.e., the fraction of neurons in the ‘active’ state, S, approaches 1 corresponding to a supercritical state. 42

St, is given by the fraction of neurons that are in active state at time step t,

N 1 X St = st . (3.3) N n n=1 Using the initial conditions described above, we simulate the model for a total of 3000 time steps. Fig. 3.1(a) shows that λ becomes greater than 1. The network activity S saturates as shown in Fig. 3.1(b). Thus, without additional regulatory mechanisms, STDP results in the undesirable supercritical regime (see review articles [19, 106]).

3.1.4 Instability of critical state with STDP learning

In this section, we explore how spike-timing dependent plasticity (STDP) leads to instability of the critical state in model 0. To do this, we will construct a rough estimate of how the eigenvalue

λ increases with time due to STDP. Assuming a large sparse homogeneous network, we will use the approximation that λ is approximately given by the mean degree or, in our case, by

1 X λt ≈ ε wt . (3.4) N m nm m,n

Now consider how the eigenvalue changes in one time step. Since the nodes do not change their

t t t−1 t t t−1 inhibitory or excitatory nature, we have, denoting ∆λ = λ − λ and ∆wnm = wnm − wnm ,

1 X ∆λt ≈ ε ∆wt . (3.5) N m nm m,n

Taking an average we get

1 X E[∆λt] ≈ E[ε ∆wt ] . (3.6) N m nm m,n

Conditioning on whether a node is excitatory or inhibitory, we can rewrite this as

1 X E[∆λt] ≈ E[∆wt |ε = 1]P (ε = 1) − E[∆wt |ε = −1]P (ε = −1) . (3.7) N nm m m nm m m m,n

Since we have a fraction of α inhibitory nodes

1 X E[∆λt] ≈= (1 − α)E[∆wt |ε = 1] − αE[∆wt |ε = −1] . (3.8) N nm m nm m m,n 43

1.003

1.0025

1.002

1.0015

1.001

1.0005

1 0 2 4 6 8 10

Figure 3.2: Actual eigenvalue λt (solid lines) and the estimated eigenvalue λt (circles) for (k, α) equal to (100, 0.1) (blue), (20, 0.30) (black), and (50, 0.20) (red) as a function of time (in units of 10 time steps, for a total of 100 time steps). 44

To calculate E[∆wnm], we assume that the learning time τ is large so that we have

t+1 t εm (st−1st −st st−1) t εmwnm t−1 t t t−1 w = w e τ m n m n ≈ w + (s s − s s ) , (3.9) nm nm nm τ m n m n from which we can derive

ε w ∆wt ≈ m nm (st−1st − st st−1) . (3.10) nm τ m n m n

The possible values for ∆wnm are wnm/τ, −wnm/τ, and 0. By calculating the probability of each, we can calculate the expected values in Eq. (3.8). We neglect the probability to have simultaneous causal and anticausal excitations to get

t t−1 t t t−1 E[∆wnm|εm = 1] = (wnm/τ)P (sm sn = 1|εm = 1) − (wnm/τ)P (smsn = 1|εm = 1) ,

t t t−1 t−1 t E[∆wnm|εm = −1] = (wnm/τ)P (smsn = 1|εm = −1) − (wnm/τ)P (sm sn = 1|εm = −1) .

(3.11)

Now we will estimate each one of these probabilities as

t−1 t t t−1 t−1 P (sm sn = 1|εm = 1) = P (sn = 1|sm = 1, εm = 1)P (sm ) . (3.12)

t−1 t Since we assume a large sparse homogeneous network we can assume that for all m, P (sm ) = S , the average network activity for the assumed network. Using this and Eq. (3.1) from Sec. 3.1.2, we get   t−1 t X t−1 t−1 t−1 P (sm sn = 1|εm = 1) = σ  εjwnj sj  S , (3.13) j

t−1 th where in the sum we condition on sm = 1, εm = 1. We separate the m term in the sum to get   t−1 t X t−1 t−1 t−1 P (sm sn = 1|εm = 1) = σ wnm + εjwnj sj  S , (3.14) j6=m

Now we make the following two approximations. The ﬁrst is to assume that we are in the linear

t−1 part of σ(·), and the second is to replace all sj ’s with j 6= m by their expected value, which we take to be St−1. This is a nontrivial assumption because the right hand side is nonlinear even after 45

t−1 t we remove σ (because we already replaced P (sm ) = S ). This gives us   t−1 t t−1 X t−1 t−1 P (sm sn ≈ 1|εm = 1) = wnm + S εjwnj  S , (3.15) j6=m which we can rewrite as   t−1 t t−1 t−1 X t−1 t−1 P (sm sn ≈ 1|εm = 1) = wnm(1 − S ) + S εjwnj  S . (3.16) j For simplicity of notation, in the following equations we remove the t−1 superscripts and put them

P in back at the end. So we have, deﬁning the in-degree j εjwnj ≡ kn

t−1 t in P (sm sn ≈ 1|εm = 1) = wnm(1 − S) + Skn S. (3.17)

Similarly, for the other cases we get

t−1 t in P (sm sn ≈ 1|εm = −1) = −wnm(1 − S) + Skn S, (3.18)

t t−1 in P (smsn ≈ 1|εm = 1) = εnwmn(1 − S) + Skm S, (3.19)

t t−1 in P (smsn ≈ 1|εm = −1) = εnwmn(1 − S) + Skm S. (3.20)

Substituting these expressions in Eqs. (3.11), inserting the results in Eq. (3.8), and simplifying, we get

1 X E[∆λt] ≈ (1 − S)(1 − 2α)w2 − (1 − S(1 − 2α))w w ε + w S(kin − kin) (3.21) Nτ nm nm mn n nm n m m,n (1 − S)(1 − 2α) X [1 − S(1 − 2α)] X S X = w2 − w w ε + w (kin − kin) . Nτ nm Nτ nm mn n Nτ nm n m m,n m,n m,n (3.22)

For an undirected, random sparse network, the number of bidirectional links should be much smaller than the number of links, so we neglect the second term compared with the ﬁrst term, and get

1 X S X E[∆λt] ≈ (1 − S)(1 − 2α) w2 + w (kin − kin) . (3.23) Nτ nm Nτ nm n m m,n m,n Similarly, the in-degree at the ends of a link should be uncorrelated; besides, because the network

in in was assumed homogeneous we have kn ≈ km , and so we get the main contribution from

1 X E[∆λt] ≈ (1 − S)(1 − 2α) w2 . (3.24) Nτ nm m,n 46

To get a useful estimate, we assume that all the nodes have the same weight, wnm = w. Then, we estimate w by requiring that the row sum of the matrix be λ = 1. Assuming there are k nonzero entries per row, a fraction α of which are inhibitory, we get wk(1 − α) − wkα = 1, so

1 w = . (3.25) k(1 − 2α)

In the sum over n, m there will be Nk nonzero terms, each of magnitude w2, which leads to our main result:

St−1(1 − St−1) E[∆λt] ≈ . (3.26) kτ(1 − 2α)

To test this, we construct Erd¨os-Renyi networks with diﬀerent values for degree k and fraction of inhibitory nodes α. Initially the weights are chosen so that λ = 1. We then simulate model 0 with STDP learning. Given the time series of activity {St}, the equation above can be integrated to give an estimate λˆt of the eigenvalue λ at time t

t 1 X λˆt ≈ 1 + Sj−1(1 − Sj−1) . (3.27) kτ(1 − 2α) j=1

Figure 3.2 shows the actual eigenvalue that come from model 0 simulations λt (solid lines) and the estimated eigenvalue λt (circles) for (k, α) equal to (100, 0.1) (blue), (20, 0.30) (black), and

(50, 0.20) (red). While the agreement is very rough, the estimate captures the trend; in particular, the growth rate of the eigenvalue is positive, so that the critical state is unstable.

Finally, note that despite the simplifying assumptions we make in deriving Eq. 3.10, it provides the following key insight. If on average, we have more causal excitations than anticausal excitations, the synapse strengths will keep increasing resulting in a supercritical state as shown in the above numerical experiment. Since we assume large and sparse network structure, given a link from node m to node n, a link from node n to node m is less likely. As a result, on average, causal excitations are more likely as opposed anticausal excitations and this results in the supercritical state. 47

1.2 1 (a) (b) 0.8 1.15 0.6 λ 1.1 S 0.4 1.05 0.2

1 0 1000 2000 3000 4000 5000 1000 2000 3000 4000 5000 t t

Figure 3.3: Results for the computational model in Sec. 3.1.2 using STDP learning rule along with synaptic strength limitation (SSL). Maximum synaptic strength is capped atw ¯. Panel (a) shows that over time the largest eigenvalue of W (see Sec. 2.5), i.e., λ, goes above its desired value of 1. Panel (b) shows that the network activity, i.e., the fraction of neurons in the ‘active’ state, S, goes close to 1 corresponding to a supercritical state. 48

3.1.5 Synaptic strength limitation (SSL)

In this section, we try to control the runaway excitation by including the SSL rule introduced in Sec. 2.5.1. To capture this eﬀect in model 0, we complement the STDP learning rule as given by

Eq. (3.2) by imposing a maximum synapse strengthw ¯:

t+1 t+1 wnm =wφ ¯ wbnm /w¯ . (3.28)

The transfer function φ is piecewise linear; φ(x) = 0 for x ≤ 0, φ(x) = x for 0 < x < 1 and φ(x) = 1 for x ≥ 1.

Using the same initial conditions and network structure as mentioned in the previous section, we run the dynamics described by Eqs. (3.1), (3.2), (3.28) for a total of 5000 time steps. The simulation results for λ and S are shown in Fig. 3.3. Panel (a) shows that imposing a strength limitation does not maintain λ ≈ 1. In fact we get a similar result as λ > 1 and, even though λ is now bounded, S still saturates.

3.1.6 Synaptic scaling (SS)

In this section, we consider the multiplicative adjustment rule to control runaway excitations.

In particular, we incorporate the synaptic scaling rule reviewed in Sec. 2.5.2 as follows:

t t+1 t+1 t t+1 t+1 t m/τ[smsn −sm sn] −βsn [m+1] wnm = wnme e . (3.29)

The parameter β deﬁnes the strength of the multiplicative adjustment. Since experimental evidence for this has been mainly found for excitatory synapses [106], we introduce the factor m + 1. It evaluates to 0 for inhibitory synapses and 2 (i.e., a non-zero value) for excitatory synapses.

Using model simulations, we found that the resulting dynamics for a given β is sensitive to the particular network realization. That is, prescribing a value for β given the network parameters

(N, p) and τ is quite hard; making this mechanism infeasible to implement. As an example, we show the result for one such value with β = 0.0001. Fig. 3.4 shows the results for λ and S as a 49

1 (b) 0.8

0.6 S 0.4

0.2

0 500 1000 1500 2000 2500 3000 t

Figure 3.4: Results for the computational model in Sec. 3.1.2 using STDP learning rule along with synaptic scaling rule. Panel (a) shows that over time the largest eigenvalue of W (see Sec. 2.5), i.e., λ goes above its desired value of 1. The inset shows a blow up of the ﬁrst 1500 time steps. Panel (b) shows that the spread of network activity, i.e., the fraction of neurons in the ‘active’ state, S, decreases as λ goes above 1. This corresponds to a network state in which some neurons are always active and some neurons are always resting. 50 function of time. The synaptic scaling mechanism is unable to stabilize λ and hence S reaches a low saturation state in which some neurons are always active and some neurons are always resting.

This results in S reaching a saturation value that is not close to 1 (see Fig. 3.4(b)). The inset in

Fig. 3.4(a) shows that the synaptic scaling stabilizes λ near its desired value of 1 for time steps

[1, 1500]. However, running the simulation further reveals that the network is supercritical with

λ > 1.

3.2 Model 1: Resource-transport dynamics with learning and inhibition

As shown in Fig 3.5, model 1 consists of a two-layered network whose ﬁrst layer is a weighted and directed neural network and whose second layer is an unweighted undirected glial network. The neural network is composed of N excitable nodes that represent neurons, labeled n = 1, 2,...,N,

t and M directed edges, labeled η = 1, 2,...,M on which synapses are located. The state sn of

t t neuron n at a discrete time step t is represented either as sn = 0 (quiescent) or sn = 1 (active). We

t t deﬁne W as the N × N adjacency matrix whose entry Wnm denotes the weight of the synapse on the edge from neuron m to neuron n at time t. If there exists a synapse from neuron m to neuron n, then neuron m is called as the presynaptic neuron and the neuron n is called the postsynaptic neuron. Any presynaptic neuron m can be either excitatory (m = 1) or inhibitory (m = −1).

t t t t That is, if we let wnm = |Wnm| denote the absolute value of synapse strength, then Wnm = mwnm.

At each time step t (where t = 0, 1, 2,... ), the state of neuron n is updated probabilistically based on the sum of its synaptic input from active presynaptic neurons at time t − 1,

 N !  X t t  1 with probability σ Wnmsm , t+1  sn = m=1 (3.30)   0 otherwise .

As in Ref. [13], the model transfer function probability σ is piecewise linear; σ(x) = 0 for x ≤ 0,

σ(x) = x for 0 < x < 1, and σ(x) = 1 for x ≥ 1.

The second layer of this model, the unweighted and undirected glial network, consists of T

t glial cells labeled i = 1, 2,...,T . Each glial cell i holds an amount of resource Ri at time step 51

Figure 3.5: Two-layered multilayer network consisting of neural and glial network layers. The neural layer (bottom), represented by adjacency matrix W , is directed and weighted. The glial layer (top), represented by adjacency matrix U, is undirected and unweighted. The interaction between glial cells and synapses (dotted lines) is modeled using an adjacency matrix G that maps each glial cell to some subset of all synapses. 52 t. Resources diffuse between the glial cells that are connected to each other. While we do not focus on a particular resource, we note that various metabolites such as glucose and lactate [25] are transported diffusively among the glial cells. We define a T × T symmetric glial adjacency matrix

U such that entry Uij = 1 if glial cell j is connected to glial cell i and Uij = 0 otherwise. Each glial cell serves a set of synapses by supplying resource to them. Hence we deﬁne a T × M matrix G with entries Giη = 1 if glial cell i serves synapse η and Giη = 0 otherwise. Consistent with recent experimental studies [64], we assume that each synapse is served by a unique glial cell and that all incoming synapses of one neuron (i.e., its dendrites) are served by a single glial cell. So, given a synapse η, there is a unique glial cell i(η) such that Gi(η)η = 1. In chapter 4 we show that our results are robust to relaxing this assumption.

Learning: Let η denote the synapse that connects presynaptic neuron m to postsynaptic neuron n, i.e., the synapse η that corresponds to the neural network edge m → n. We assume that the absolute strength of synapse η, i.e., wnm, depends on its past learning history as determined

t t from the STDP learning rule, via an auxiliary variable, wbnm, and on the amount of resource, Rη, at synapse η,

t t t wnm = f Rη, wbnm , (3.31)

t where ∂f(x, y)/∂x ≥ 0, ∂f(x, y)/∂y ≥ 0, and wbnm evolves according to the STDP learning rule:

h i wt+1 = wt exp m st−1st − st st−1 . (3.32) bnm bnm τ m n m n

Moreover, we implement synaptic strength limitation, by requiring f not to exceed a maximum valuew ¯, f ≤ w¯. For excitatory synapses (m = +1), causal ﬁring corresponds to ﬁring of the

t−1 presynaptic neuron m at the previous time step t − 1 (i.e., sm = 1), followed by the ﬁring of

t the postsynaptic neuron n at the current time step t (i.e., sn = 1). Thus for causal excitations

t+1 t wbnm > wbnm and the excitatory synapse is reinforced. Similarly, for anticausal excitations excitatory t+1 t synapses are weakened, wbnm < wbnm. The corresponding analogous conditions hold for inhibitory neurons (m = −1). The constant τ sets the learning timescale. 53

Resource-transport dynamics: Resources diﬀuse amongst glia through their connection network (characterized by the adjacency matrix U) and between glia and the synapses they serve (via the glial-neural connection network characterized by the adjacency matrix G). Our model for the

t t evolution of the amount of resource Ri at glial cell i and the amount of resource Rη at synapse η is

T M t+1 t X t t X t t Ri = Ri + C1 + DG Uij Rj − Ri + DS Giη Rη − Ri , (3.33) j=1 η=1

t+1 t t t t Rη = Rη + DS Ri(η) − Rη − C2sm(η) , (3.34)

where DG is the rate of diﬀusion between glial cells, and DS is the rate of diﬀusion between glia

t+1 and synapses. Moreover, we enforce Rη ≥ 0, i.e., if Eq. (3.34) yields Rη < 0, then we replace it

t by 0. The ﬁrst term on the right hand side of Eq. (3.33), Ri, is the amount of resource in glial cell i at time t. The parameter C1 denotes the amount of resource added to each glial cell at each time step (e.g., supplied by capillary blood vessels). For simplicity, we assume each glial cell has the same C1, although we later discuss the eﬀect of heterogeneous source rates. The last two terms are the amount of resource transported to (or from) glial cell i, respectively, from its neighboring glial cells and from the synapses that it serves.

In (3.34), the ﬁrst term denotes the amount of resource at synapse η at time t. The term

t t t t proportional to DS denotes the amount of resource gained (if Ri(η) > Rη) or lost (if Ri(η) < Rη) from glial cell i(η) that serves synapse η. If the presynaptic neuron m(η) ﬁres at time step t

t (sm(η) = 1), then all outgoing synapses for neuron m(η), including η, consume some resource, thus decreasing the resource at each synapse by an amount C2 (where C2 is a model parameter).

3.3 Model 2: Resource-transport dynamics without learning or inhibition

In this section, we present the second version of our model, i.e., model 2. In this version, we use Eq. (3.30) to describe the evolution of state sn of the neuron n. Reﬂecting increasing synaptic strengths with increase in resource, we model synapse strengths by taking f in Eq. (3.31) 54 as f(x, y) = xy, i.e.,

t t Wnm = wnmRη (3.35) where wnm > 0 denotes the initial strength of that synapse chosen from a uniform distribution.

Since resource Rη is always positive, this eﬀectively amounts to removing inhibitory links from this model. Also, unlike the model in the previous section, we remove STDP learning rules applied to the synaptic weights and thus consider the no learning case. Finally, to model resource-transport dynamics, we use Eq. (3.33)-(3.34).

One of the advantages of using the above mentioned simpler model is that by assuming homogeneous network structure for both the neural and the glial networks, we can derive a three dimensional map that reproduces the behavior of the full model and thus we can gain insights on the eﬀect of changing various parameters. We approximate the largest eigenvalue, λt, of the neural

t 1 P t 1 P t network adjacency matrix as the mean degree [107], λ ≈ N n,m Wnm = N n,m wnmRη(n,m) and t 1 P t deﬁne the average resource in glial cells as R = T i Ri. Summing Eq. (3.33) over i and Eq. (3.34) over n and m, we obtain

Dλt Rt+1 = Rt + C + − qDRt , (3.36) 1 w¯ t+1 t t t t λ = λ + qDwR¯ − Dλ − C2wkS¯ . (3.37)

t 1 P t The above equations are coupled to the average activity, S = N n sn, which is stochastic. We model St considering the following two options:

(1) A deterministic approximation near the critical point is to assume that

St+1 = λtSt . (3.38)

This neglects the nonlinear eﬀects that keep St bounded between 0 and 1. We refer to this

option along with Eqs. (3.47), (3.48) as the 3-D map without noise. While this approxima-

tion neglects the eﬀects of noise and is valid only close to the ﬁxed point, it is useful since

the stability properties of the ﬁxed point λ = 1 underlie the robustness of the critical state

to changes in model parameters. 55

(2) Another approximation is to assume

St+1 = max 0, min 1, λtSt + rt + µt , (3.39)

where rt is a noise term which has zero mean and standard deviation given by pSt(1 − St)/N

as estimated in [13] while µt represents the external stimulus,   1/N with probability ζ , µt = (3.40)  0 otherwise .

In this case, one neuron ﬁres every time step with probability ζ. Coupled with eqs. (3.47),

(3.48), this is referred to as the 3-D map with noise. Since in the full model, S is stochastic,

this variant of the map is useful to make comparisons and to predict the behavior of the

full model. This is shown in chapter 4.

We now present the detailed analysis and derivation for the two variants of the 3-dimensional map.

3.3.1 Simpliﬁcation of model 2 to a 3-dimensional map

In the absence of learning, we can assume without loss of generality that the weights in the coupling network are completely determined by the associated resource, Wnm = Rη. Let

t w¯ = hwnmiM , i.e., the average synaptic strength at time step, t = 0. The synaptic strength Wnm

t t can be expressed as Wnm =wR ¯ η. Further, assuming that the network is homogeneous, the largest eigenvalue λt can be approximated well by the mean degree, or

N 1 X w¯ X λt ≈ W t = Rt , (3.41) N nm N η n,m η where the sum over the entries of W is expressed as a sum over synapses. Summing Eq. (3.34) over

η, we get

Dw¯ X C2w¯ X λt+1 = λt + Rt − Dλt − st . (3.42) N i(η) N m(η) η η

t 1 P t Next, we deﬁne R = N i Ri to be the average amount of resource per glial cell at time t. If the number of synapses served by glial cell i is qi, and the qi have a narrow distribution with qi ≈ q 56

P t P t P t then η Ri(η) = i qiRi ≈ q i Ri, and so

C2w¯ X λt+1 = λt + qDwR¯ t − Dλt − st . (3.43) N m(η) η

1 P t t In the term N η sm(η) we have to sum sm for each synapse starting from neuron m. If the out- degree distribution of the neuronal network is narrow with mean k, this can be approximated as

1 P t 1 PN t k N m sm = kS(t), where S(t) = N n=1 sn:

t+1 t t t t λ = λ + qDwR¯ − Dλ − C2wkS¯ . (3.44)

To get the evolution of Rt, we sum Eq. (3.33) over i and divide by N. The third term vanishes by symmetry, yielding

M D X X Rt+1 = Rt + C + G (Rt − Rt) . (3.45) 1 N iη η i η=1 i=1

D PM P t D P t Since there is a unique glial cell serving synapse η, we have N η=1 i=1 GiηRη = N η=1 Rη = t PM P t P t PM P t Dλ . Furthermore, η=1 i=1 GiηRi = i=1 Ri η=1 Giη = i=1 Riqi, where again qi is the number of synapses served by glial cell i. Assuming the distribution of qi’s is narrow and qi ≈ q, then this is approximately qNRt:

Dλt Rt+1 = Rt + C + − qDRt . (3.46) 1 w¯

Therefore, for homogeneous networks we obtain the coupled equations

Dλt Rt+1 = Rt + C + − qDRt , (3.47) 1 w¯ t+1 t t t t λ = λ + qDwR¯ − Dλ − C2wkS¯ . (3.48)

These equations were obtained under the approximation that both the neural and glial networks are homogeneous. Although they look simple, the complication is that they are coupled to St, which is a stochastic variable whose evolution depends on λt. To model S we use one of the two options given by Eqs. (3.38), (3.39) mentioned before. In this way, we have two variants of the

3-dimensional map: (i) without noise (Eqs. (3.36), (3.37), (3.38)) and (ii) with noise (Eqs. (3.36),

(3.37), (3.39)). 57

3.3.2 Analysis of 3-D map without noise

As described in the previous section if we use the map without noise, we get the following closed system for three variables:

Dλt Rt+1 = Rt + C + − qDRt , 1 w¯ t+1 t t t t λ = λ + qDwR¯ − Dλ − C2wkS¯ ,

St+1 = λtSt, which has the ﬁxed point

C C 1 λ¯ = 1, S¯ = 1 , R¯ = 1 + . (3.49) kC2 qD qw¯

The stability of the ﬁxed point is determined by whether the eigenvalues of the Jacobian,

 D  1 − qD 0  w¯      J =  qDw¯ 1 − D −kC2w¯  (3.50)    C  0 1 1 kC2 are inside the complex unit circle. We obtain the following characteristic polynomial for J:

p(t) = t3 + [D (q + 1) − 3] t2

+ [−2D (q + 1) + C1w¯ + 3] t

+ [qDC1w¯ + D (q + 1) − C1w¯ − 1] (3.51)

3 z+1 0 We can use the following transformation– p(t) → (z − 1) p( z−1 ) = p (z) – to map the inside of the unit circle into the left half open complex plane and apply the Routh-Hurwitz criterion to determine the stability of the ﬁxed point. Doing this, we obtain

0 3 2 p (z) = (C1Dqw¯) z + (2C1w¯ − 3C1Dqw¯) z

1 + (4D + 4Dq − 4C1w¯ + 3C1Dqw¯) z

0 + (8 − 4D − 4Dq + 2C1w¯ − C1Dqw¯) z . (3.52) 58

0 1 2 3 This equation is of the form a0z + a1z + a2z + a3z . We thus have,

a0 = 8 − 4D − 4Dq + 2C1w¯ − C1Dqw¯ , (3.53)

a1 = 4D + 4Dq − 4C1w¯ + 3C1Dqw¯ , (3.54)

a2 = 2C1w¯ − 3C1Dqw¯ , (3.55)

a3 = C1Dqw¯ , (3.56) with the following conditions for the stability of the ﬁxed point:

ai > 0 ∀i , (3.57)

a1a2 > a3a0 . (3.58)

Since the activity S ∈ [0, 1], we have the following additional constraint:

C 1 < 1 . (3.59) kC2

This results in the following set of inequalities for the stable critical state:

2 Dq − < 0 , (3.60) 3 1 1 + q 3 − − < 0 , (3.61) qD C1qw¯ 4 C qDw¯ C w¯ Dq D 1 − 1 + + − 1 < 0 , (3.62) 8 4 2 2 2 2 2 2 2 2 2 2 C1 D q w¯ − 2C1 Dqw¯ + C1 w¯ + C1D q + C1D q − C1D < 0 , (3.63) C 1 < 1 . (3.64) kC2

Since we have numerous parameters, an exhaustive validation of the inequalities using numerical simulations is infeasible. As an illustrative example, we choosew ¯ = 0.06, k = q = 50 and

−5 C2 = 10 and plot the inequalities in Fig. 3.6 by varying parameters D and C1 on x-axis and y-axis respectively. The white region corresponds to all the inequalities being satisﬁed. As an example, we consider the four settings for the model parameters– C1 = {0.00006, 0.00006, 0.00006, 0.0006} and D = {0.0002, 0.0005, 0.0000005, 0.0002}. This is shown in Fig. 3.6 with the points marked as 59 a, b, c, d. The largest eigenvalue of the neural network adjacency matrix, λ, is shown in Fig. 3.7 with panels (a)-(d) corresponding to the respective settings. As expected, since only setting (a) satisﬁes all inequalities (since a is in the white region), in this case λ converges to 1 and hence the critical state is stable. In all other cases, the critical state is unstable as seen from panels (c)-(d) in

Fig. 3.7. Although, here we show the results for a few settings, in Sec. 4.2 we show the results for parameter search along the C1 axis and show how the 3-dimensional map can be used to predict model behavior. 60

−5 Figure 3.6: Assuming the parameter choicesw ¯ = 0.06, k = q = 50 and C2 = 10 , the unshaded or white region corresponds to all inequalities being satisﬁed. The parameter choices within this region should get us a stable critical state, or more concretely, λ = 1 over time. The regions shaded purple and orange correspond to two of the inequalities not being satisﬁed, namely, Eqs. (3.63) and (3.64). The inequalities given by Eqs. (3.60)-(3.62) do not tell us anything new. 61

Figure 3.7: We plot the largest eigenvalue of the neural adjacency matrix, λ, as a function of time t. Panels (a), (b), (c) and (d) show the results for the four settings shown in Fig. 3.6. Panel (a) corresponds to all inequalities being satisﬁed and hence λ → 1 over time. Panels (b), (c) violate inequality given by Eq. (3.63). Panel (d) violates inequality given by Eq. (3.64). Hence, in these three cases, we ﬁnd that the critical state λ = 1 is not stable. Chapter 4

Numerical Experiments

4.1 Model 1: Results on model with learning and inhibition

In this section, we present results of numerical experiments on model 1 that includes learning and inhibition. For simplicity, we assume that both the neural network and the glial network have an Erd¨os-Renyi (ER) network structure. Following the terminology described in Chapter. 3 the experimental setup is as follows. We build the N × N directed weighted ER neural network adjacency matrix W by creating a link from node m to node n (i.e., setting Wnm 6= 0) with probability p and setting Wnm = 0 otherwise. This gives the mean number of incoming and outgoing synapses per neuron, kN = (N − 1)p, and the expected total number of synapses M = NkN . To

0 specify the initial state of each synapse, at t = 0 we set the initial resource at synapse η Rη = 1

0 and take the initial value of each wbnm to be an independent draw from a uniform distribution over [0, 1]. We then rescale all entries in W by a constant to obtain a desired largest eigenvalue of W , as discussed below.

The glial network, represented by the matrix U having T nodes that represent glial cells, is taken to be an undirected and unweighted ER network. If the glial cell j is connected to the glial cell i, then Uij = Uji = 1; and Uij = 0 otherwise. If the probability of forming a link is q, then the mean degree of a glial cell is kG = (T − 1)q. Recent evidence suggests that the number of glial cells is roughly equal to the number of neurons [63], and hence in our experiments we set T = N.

0 The initial resource for each glial cell is taken to be Ri = 1. Based on this setup we now present various numerical experiments that illustrate our results. 63

Layer Nodes Adjacency Prob. of Mean Mean no. matrix an edge degree of edges

Neural N W p kN = Np M = NkN (weighted, directed) 1000 0.05 50 50000 Glial T U q kG = T q E = T kG/2 (unweighted, undirected) 1000 0.05 50 25000

Table 4.1: The specifications and relevant terminology for the two-layered multilayer network consisting of neural and glial network layers. The neural layer is directed and weighted while the glial layer is undirected and unweighted. Unless specified otherwise, in all the experiments, we use the above specified parameter values. 64

In all our experiments, for simplicity, we take the function f in Eq. (3.31) to be f(x, y) =

t t t xy for xy < w¯ and f(x, y) =w ¯ for xy ≥ w¯, i.e., we take wnm = Rη(n,m)wbnm. We make two random draws of an ER random graph: ﬁrst for the directed neural network and second for the undirected glial network. We use the parameter settings described in Table 4.1. For all our numerical experiments we take DG and DS to be the same, DG = DS = D; we also set the fraction of inhibitory nodes to be 0.2 [108] and use the following additional parameter choices

C1 = 0.0188,C2 = 0.001,

D = 0.005 , w¯ = 0.14.

We chose these parameter values somewhat arbitrarily but, as shown later, our results are fairly robust to the choice of these values.

In the following, we report the three main ﬁndings from model 1. First, we show that network dynamics are stable (i.e., that the network robustly self-tunes to operate at the critical point λ ≈ 1), avoiding saturation or extinction of neural activity. Second, we show that resource transport among the glia is essential to maintain this stability. Third, we verify that the neural network can learn, i.e., external input can result in long-lasting synaptic changes.

Experiment 1 : To quantitatively assess the stability of the network dynamics we study λ, the largest eigenvalue of the matrix W . Previous studies on purely excitatory networks [12] and networks having inhibitory nodes [13, 96] show that λ dictates the nature of the network’s dynamics:

λ < 1 corresponds to a hypoexcitable, or subcritical, state where activity dies out; λ = 1 corresponds to the stable, critical state where activity is balanced, neither growing nor decaying on average; and

λ > 1 corresponds to a hyperexcitable, supercritical state where the activity grows until nearly all neurons are firing at every time step. In this first experiment we choose different initial conditions for λ (obtained by rescaling the initial W ), i.e., at t = 0 we start in the critical, subcritical and supercritical states respectively, λ0 = {1, 0.5, 1.5}. Figure 4.1(a) shows a plot of λ as a function of time, t. In all three cases we find that after a brief transient, the network dynamics become stable, i.e., λ settles near 1 after sufficient time has passed. Fig. 4.1(b) shows the total resource R held in 65

× 4 10 10 (b) 8

6 R 4

0 1 2 3 4 5 6 t ×104

1 0.9 (c) 0.8 0.7 0.6

S 0.5 0.4 0.3 0.2 0.1 0 2000 4000 6000 8000 10000 t

Figure 4.1: Results of Experiment 1 : (a) Time series of λt (largest eignenvalue of W t) reveal rapid convergence to stable network dynamics (λ ≈ 1), independent of initial conditions. Three different initial conditions were tested: hyperexcitable (blue, λ0 = 1.5), stable (black, λ0 = 1), and hypoexcitable (red, λ0 = 0.5). The inset is an expanded view of the first 5000 time steps. (b) After a longer transient the total resource R also stabilizes to a steady value. (c) Similarly, in all three cases, the average activity S reaches a statistical steady state with large fluctuations. 66 all glia and synapses as a function of time t, where R is given by

T M t X t X t R = Ri + Rη . (4.1) i=1 η=1

In all three cases R reaches a steady state value. Fig. 4.1(c) shows that the average activity,

N 1 X S = st , (4.2) N n n=1 is initially below the activity for the critical case for λ0 = 0.5 and above the activity for the critical case for λ0 = 1.5, indicative of the subcritical and supercritical regimes. Starting in these regimes, over time, the dynamics of S becomes statistically similar to the dynamics of S for the critical initial state of λ0 = 1. Thus this model naturally operates in the stable regime. This can be understood on the basis that high activity rapidly consumes resources at the synapses, thus reducing their weights, and leading to decrease in λ; while, with low activity, synapses consume at a low rate, allowing buildup of resource with time and consequent increase of synaptic weights. As an example, starting in the subcritical state (red triangles) results in an initial build up of resource owing to low activity. However, as the activity increases, the resource gets consumed; this feedback control results in a steady-state for R as shown in Fig. 4.1(b).

Experiment 2 : The previous experiment demonstrated that STDP and resource distribution dynamics are both active during the stabilization. Next, we pose the question: Is the diffusion of resources via the glial network important for stable cortical dynamics? Or can we still get stability if we switch off transport among the glia (i.e., set DG = 0)? To do this experiment, for t = 1, 2,...,T1 = 80000, we let the system reach a steady state with the glial network operative as in Fig. 4.1, using Eq. (3.33), and define an equivalent time averaged resource supply rate Ci for each glial cell i,

* T + X t t Ci = DG Uij Rj − Ri + C1 . (4.3) j=1 T1,T2

In the above equation h.iT1,T2 represents the time average over the interval t = (T1,T2). We switch 67

× 4 2 4 10 baseline (a) (b) 1.8 instability 3.5

1.6 3 λ 1.4 R 2.5

1.2 2

1 1.5

0.8 1 5 10 15 5 10 15 t 4 t ×104 ×10

Figure 4.2: Results for Experiment 2 : (a) the maximum eigenvalue λ versus t, and (b) the total resource R versus t. The data plotted in black are ‘baseline’ results obtained using model 1 as described in Sec. 3.2. For the data plotted in red (labelled ‘instability’), the initial evolution is the same as for the baseline data up until t = 100000 (marked in the figure by a vertical arrow). At this point, turning off the diffusion of resources amongst the glia results in runaway growth of λ and R. 68 off transport among the glia at t = T2 by setting DG = 0, and replace (3.33) by

M t+1 t X t t Ri = Ri + DS Giη Rη − Ri + Ci . (4.4) η=1

In this way, the average rate of total nonsynapse resource supply to each glial cell is preserved after the glial diﬀusion is turned oﬀ. Replacing Eq. (3.33) by Eq. (4.4) we run the dynamics for a total of 160000 time steps. Since the total resource R stabilizes by time step 80000 and we sever the links amongst glia at time step 100000, choosing T1 = 80000 and T2 = 100000 results in a good approximation of the Ci term in Eq. (4.4).

For initial condition λ0 = 1, Fig. 4.2 shows the results for two runs– one in which we use the dynamics described by Eq. (4) (baseline) and the other in which we switch oﬀ the glial network and run the dynamics as described in the previous paragraph (instability). Fig. 4.2(b) shows that after the glial network is switched oﬀ, Rt increases as resource starts to accumulate at some synapses and get used up at others. Such increases and decreases in Rη change the weights of the matrix

W , resulting in an increase in λ as shown in Fig. 4.2(a). This suggests that the dynamical nature of the diﬀusion plays an important role in stabilizing the neural network learning dynamics.

Experiment 3 : In the next experiment we demonstrate that the neural network can learn and memorize while maintaining λ close to the stable value of 1. To do this we divide the neurons into two equally sized groups, G1 and G2, consisting of 500 neurons each. This results in four groups: synapses that connect neurons within G1, synapses from G1 to G2, synapses from G2 to G1 and synapses that connect neurons within G2.

The aim of this experiment is to diﬀerentiate these groups of synapses in terms of their mean synaptic strengths. In particular, we employ a learning protocol that results in a higher mean synaptic strength for the group of synapses from G1 to G2 and a lower mean synaptic strength for the group of synapses from G2 to G1. Per the STDP learning rule described in chapter 2, we would require causal excitations for the group of synapses from G1 to G2 and anticausal excitations for the other. Additionally, noting that there is an inherent delay in STDP learning of two time steps, we use the learning protocol depicted in Fig. 4.3(a). The entire experimental setup is described 69

1.5 (a) (b)

G λ 1 1

stimulus 0 ζ

G 2

stimulus 0 0.5 5 10 15 5 10 15 ×104 t ×104

× 4 4 10 0.04 (c) (d) 3.5

3 0.035

R 2.5 0.03 Within G 2 1 G to G 1 2 G to G 1.5 2 1 mean synaptic strength Within G 0.025 2 1 5 10 15 5 10 15 t ×104 t ×104

Figure 4.3: Results for Experiment 3 : We divide the neurons into two equally sized groups, G1 and G2, consisting of 500 neurons each. This results in four groups of synapses: synapses within the ﬁrst group (within G1), synapses that convey signals from neurons in G1 to neurons in G2 (G1 to G2), synapses that convey signals from neurons in G2 to neurons in G1 (G2 to G1) and synapses within the second group (within G2). Panel (a) depicts the learning protocol (see text). Panel (b) and (c) show λ and R versus t. The learning regime spans t = [80000, 100000] (delimited by the vertical arrows). Panel (b) shows that λ becomes subcritical during learning [109]. Panels (d) shows the mean synaptic strength for the four groups of synapses for excitatory synapses during learning. In accord with the STDP learning rule, the mean synaptic strength increases for G1 to G2 synapses. In the post-learning regime, spanning t = [80000, 160000], panels (d) shows that the model remembers what it learned. Panel (b) shows that after learning, λ quickly evolves back to the critical state λ =∼ 1. 70 below.

We run the dynamics for a total of 160000 time steps, observing three distinct phases: pre- learning (1 ≤ t ≤ 80000), learning (80000 < t ≤ 100000) and post-learning (t > 100000). In the pre-learning phase, the dynamics are as described in the previous section. The total resource R reaches a steady-state value and the eigenvalue λ ﬂuctuates near 1 (viz., Figs. 4.3(b), 4.3(c)). In the learning phase, for neurons in group Gν (ν = 1 or 2) we modify Eq. (3.30) by introducing a

(ν) time-dependent external stimulus, ζt ,

 N !  X t t (ν)  1 with prob. σ Wnmsm + ζt , t+1  sn = m=1 (4.5)   0 otherwise . where ν is the group to which neuron n belongs, and, letting ζ = 0.15, the learning protocol

(ν) deﬁning ζt is as shown in Fig. 4.3(a). That is, starting at the beginning of the learning phase

(t = T1 = 80000), we stimulate neurons only in G1; then, in the next time step, we stimulate neurons only in G2; then, in the next two time steps, no stimulus is applied to either group. This four step sequence is successively repeated until the end of the learning phase (t = T2 = 100000), past which no stimuli are applied. We plot the mean synaptic strength for the four groups of synapses in Fig. 4.3(d). As expected, sequential ﬁring of G1 neurons followed by G2 neurons results in strengthening of excitatory synapses from G1 to G2 and weakening of excitatory synapses from

G2 to G1. Importantly, these learning-related changes in strengths of the four groups of synapses are preserved in the post-learning phase (after time step 100000), thus conﬁrming that the neural network model remembered what it learned.

Finally, Fig. 4.3(c) shows that during the learning phase there is a corresponding decrease in total resource R. The increased resource consumption and the consequent decrease in R can be attributed to the increase in neuronal ﬁring rates owing to the external stimulus. As the stimulus is removed in the post-learning phase, the plots in Figs. 4.3(b), 4.3(c) show that the resource R is replenished and λ resets to 1 with ﬂuctuations. Hence, in the post-learning phase we have a balanced cortical state, and the neural network remembers what it learned. Thus, although the 71

1.5 D 1.4 Db 1.3 C2 Cb 1.2 2 1.1 t i 1 λ h 0.9 0.8 C1 Cb 0.7 1 0.6 0.5 0 0.5 1 1.5 2

Figure 4.4: Time average of the largest eigenvalue, hλit, as a function of C1/Cb1, C2/Cb2 and D/Db where Cb1, Cb2 and Db are the parameter values used for Fig.4.1-4.3. All three curves show that this model is fairly robust to parameter changes, e.g., a 25% change in C1 or C2 yields a change in hλit of about 0.3%. 72 glial transport stabilizes a unique attracting macrostate with λ =∼ 1, it, nevertheless, still potentially allows for distinct microstates representing stored information.

The qualitative results reported here are robust to parameter variations over a 25% range in

C1 and C2 and an even larger range for D. One indication of this is shown in Fig. 4.4 where we plot the time averaged largest eigenvalue hλit of W versus the parameters C1, C2 and D normalized to their values used in Figs. 4.1-4.3. We note that hλit changes by roughly 0.3% when C1 or C2 changes by 25%. This approach used to quantify robustness is somewhat arbitrary. However, for the simpliﬁed model without learning and inhibition we derive a 3-dimensional map (see Sec. 3.3.2) that allows to more robustly predict the good and bad parameter regimes.

4.2 Model 2: Results on model without learning or inhibition

In this section, we present the results of numerical experiments on the model with only excitatory nodes and no learning. The goal here is twofold: (i) to quantify the critical state using a typical signature of criticality [4, 12], power-law distributed avalanche size distributions with the characteristic exponent of −3/2 for the power law and (ii) to gain insights on the good and bad parameter regimes for the stable critical state. Additionally, we quantify robustness using both simulations on the full model as well as using the 3-dimensional map. We show that the

3-dimensional map with noise (introduced in Sec. 3.3) can robustly reproduce the behavior of the full model. Finally, in all the experiments, we use the parameter values for the two layers as shown in Table 4.1.

In the ﬁrst experiment, we show that using suitable parameter choices the resource-transport dynamics causes the system to self-organize to the critical state corresponding to λt = 1 after a transient period (see panel (c) of Fig. 4.5). We consider three diﬀerent initial conditions λ0 =

{1, 0.95, 1.05} (obtained from rescaling the initial W ). Panel (b) of Fig. 4.5 shows that the average glial resource, R, reaches a steady state in all three cases. Additionally, we run the 3-D map with noise (see Eqs. (3.36), (3.37), (3.39) and the accompanying explanation) for the same parameter settings and qualitatively recover the results shown as the dotted (subcritical case) and dashed 73

1.02 (a)

1.01

0 = 1 0.99 0 = 0.98 0 = 1.02 0.98 2 4 6 8 10 106

0.31 (b)

0.305

0.3

0.295

0.29 2 4 6 8 10 106

Figure 4.5: Results of numerical simulations considering diﬀerent initial states of the model such as critical (black circles), subcritical (red triangles) or supercritical (blue squares) states. Panel (a) shows the evolution of the largest eigenvalue, λ, of the neural network adjacency matrix. In all three cases, λ ≈ 1 after a transient period. Panel (b) shows that the average glial resource, hRii, settles to a stable value over time. In both panels, the dotted (subcritical case) and the dashed (supercritical case) lines show that the predictions from the 3-D map with noise [Eqs.(3.36), (3.37), (3.39)] qualitatively agree with the simulations from the full model for both λ and hRii. 74

(a) 100 10-2 -4 ) 10 L ( 10-6 P 10-8 10-10 0 10 2 10 9 104 7 8 6 5 6 10 8 3 4 10 10 1 2 L 10 setting

setting C1 D decades γ

1 3 × 10−8 2.5 × 10−5 4 −1.48 2 3 × 10−8 5 × 10−5 4 −1.5 3 3 × 10−8 10 × 10−5 4 −1.46 4 6 × 10−8 2.5 × 10−5 3 −1.45 5 6 × 10−8 5 × 10−5 4 −1.44 6 6 × 10−8 10 × 10−5 3 −1.46 7 12 × 10−8 2.5 × 10−5 3 −1.46 8 12 × 10−8 5 × 10−5 3 −1.47 9 12 × 10−8 10 × 10−5 3 −1.44

(b)

−8 Figure 4.6: The critical state is robust to changes in model parameters. For a fixed C2 = 10 , panel (a) shows the avalanche size distributions, P (L) for different parameter settings shown in panel (b). As shown, the exponent for the power-law fit (γ) is near the characteristic value of −3/2 for fits spanning 3 (red) or 4 (blue) decades. In all these cases, we get a plausible p-value for the power-law fit found by using maximum likelihood methods [73]. 75

(supercritical state) lines in Fig. 4.5. Thus, we show that starting with diﬀerent initial states, the full model can self-organize to the stable critical state. Also, owing to the simpliﬁcation of model

2, we could predict this result using the 3-dimensional map.

Although in the previous experiment, we quantified the critical state regime by λ ≈ 1, experimentally (e.g. [4, 8]) it is often quantified by avalanches whose sizes follow a power-law distribution with the characteristic exponent of −3/2. We define neuronal avalanches as the excursions of activity S above a certain threshold S∗ [13]. In particular, we calculate an avalanche size L as follows:

t X2 L = N St . (4.6)

t=t1

∗ Here we deﬁne d = t2 − t1 as the duration of the excursion of S > S . Thus, the sum is an approximation of the area under S over the duration d. Finally, we scale the sum by N so as to count the number of spikes (excitations) in an avalanche. Fig. 4.7 shows the average neural network activity, S, obtained by simulating the model. In this example plot we use a threshold S∗ = 0.15.

Each avalanche is shown in a diﬀerent color.

In the next experiment, we simulate the model for a total of 107 time steps and calculate avalanche sizes using the time series for S. Panel (a) of Fig. 4.6 shows that for S∗ = 0.15, the critical state regime as characterized by avalanche size distributions P (L) ∝ Lγ is robustly observable over a wide range of parameter settings. Panel (b) of Fig. 4.6 shows the parameter settings used and the corresponding power-law fit exponent, γ ≈ −3/2. We use standard fitting techniques based on maximum-likelihood estimation and the Kolmogorov-Smirnov statistic [73] with one key difference– we allow for an upper cutoff for the fitting procedure, similar to some of the previous studies quantifying neuronal avalanche size distributions [52, 97, 110]. That is, we fit a power law with three parameters {γ, Lmin,Lmax} where Lmin and Lmax are lower and upper cutoffs respectively.

Given {Lmin,Lmax} we can ﬁnd the best γ using a grid search over the typical range for γ [−1.1, −4]

[73]. The best fit is found by minimizing the KS-statistic. Additionally, we discard a power-law p fit if the KS ≥ 1/ Nsamp where Nsamp is the sample size in the fitted range. This empirical rule 76

0.8

0.6 S 0.4 S∗ 0.2

0 1000 2000 3000 4000 5000 6000 7000 8000 9000 10000 t

Figure 4.7: Avalanches are deﬁned as the excursions of activity S above a threshold S∗ [13]. We t X2 calculate an avalanche size using the sum St, i.e., the area under the S curve over the duration

t=t1 d = t2 − t1 of the excursion. For each avalanche we use a different color to depict the area under the S curve. Finally, in order to count the number of excitations (or spikes) in an avalanche, we scale this sum by N to get the actual avalanche size L given by Eq. (4.6). 77 comes from [52] (see supplementary information therein), which generalizes the existing tabulated values approach used in [111]. When introducing an upper cutoff, it is important to make sure that we fit a large enough range, since we can always make Lmax and Lmin close enough to get a “good” Lmax (or plausible) fit. To address this, we select {Lmin,Lmax} that maximizes log . Lmin

Given a choice of C2, we ﬁnd that choosing a C1 such that C1 < C2kI , where kI denotes the average number of synapses served by a glial cell, results in stable critical state. If C1 does not satisfy this inequality, the system receives more resource than the synapses can consume, resulting in a build-up of resource and a supercritical state. In this model, we ﬁnd that, for the tested

−8 −8 range C1 = [3 × 10 , 12 × 10 ], choosing smaller values of C1 results in power-law ﬁts over Lmax more decades, i.e., with larger log . For example, as shown in panel (b), using smaller C1 Lmin resulted in power-law distributions spanning 4 decades as opposed to 3. This numerical ﬁnding will be validated with the results obtained from the 3-D map in Sec. 3.3.

For homogeneous undirected glial networks, if we use the same source rate C1 for all glial cells the diﬀusion of resource amongst the glial cells can be shown to be unnecessary. In particular summing Eq. (3.33) over i and dividing by T gives us an equation for average glial resource Rt and the term for diﬀusion amongst the glial cells vanishes by symmetry. Thus setting DG = 0 does not result in any instability.

However, if we consider the likely realistic situation of heterogeneous source rates, the neural network dynamics can become supercritical if DG = 0. We note that the sources of heterogeneity are diverse such as heterogeneous source, consumption or diﬀusion rates and heterogeneous network structures. Here, as an illustrative example, we consider heterogeneous source rates by choosing a

Gaussian distribution for C1i, i.e., the source rate for glial cell i. We draw C1i ∼ N (µ, σ) with µ and σ such that approximately 5% of the C1i > C2kI , i.e., approximately 5% of glial cells violate inequality Eq. (3.64) shown in Sec. 3.3.2. The other 95% source rates do not violate this inequality.

In this way, some glial cells receive more resource than what can be consumed by the synapses that the glial cells serve. Fig. 4.8 shows the results. Panel (a) shows that in the absence of resource- transport amongst the glial cells, i.e., setting DG = 0, the network converges to a supercritical state 78

1.5 (a)

D = D = D G S D = 0, D = D G S 0.5 1 2 3 4 5 6 106

104 2.2 (b) 2

1.8

1.6

1.4 1 2 3 4 5 6 106

Figure 4.8: Using heterogeneous source rates, the critical state is stable only in the presence of resource-transport amongst the glial cells. Panel (a) shows that when there is diffusion amongst glia, i.e., DG = D (blue circles), the largest eigenvalue of the neural network adjacency matrix, λ ≈ 1. However, if resource transport amongst the glial cells is absent, i.e. DG = 0, the neural network becomes supercritical with λ > 1 (red triangles). The dashed line shows λ = 1 for reference. Panel (b) shows that the total resource R reaches a steady state when diffusion is turned on (blue circles) and keeps increasing over time if diffusion is turned off (red triangles). This further highlights the importance of resource-transport dynamics on the stability of the critical state. 79 such that λ > 1. However, in the presence of resource transport amongst the glial cells (DG = D),

λ = 1 is stable. Panel (b) of Fig. 4.8 shows that for DG = 0 case, the total resource R keeps increasing while for DG = D case, R reaches a steady-state. We conjecture that this model can protect the system against other sources of heterogeneity such as heterogeneous consumption and diﬀusion rates so long as the inequalities are satisﬁed and we leave the required analysis for future.

In the next experiment, we study the impact of network heterogeneity on the critical state.

More speciﬁcally, we make the interaction network, G, heterogeneous such that some glial cells serve a much larger number of synapses while others serve few. To do this, we use the following arbitrary rule: every glial cell i = x serves one incoming synapse of neuron n = x while glial cell i = x + 1 serves all the incoming synapses of neuron n = x + 1 and the remaining synapses of neuron n = x.

We repeat this process for half of the glial cell network, i.e., from glial cell i = {1, 2,...,T/2}.

Each of the remaining glial cells serve all of the incoming synapses for only one neuron. While this might seem arbitrary, the aim of this experiment is not to test the stability of the critical state by exhaustively varying the amount of heterogeneity in the interaction network. Instead, we want to provide one example of a heterogeneous interaction network structure that shows that the presence of resource transport amongst glia is an important feature of the model that is crucial for both stability and criticality.

Fig. 4.9 shows that the critical state is stable only if resources are allowed to diﬀuse via the glial network (blue circles). Panel (a) of Fig. 4.9 shows that λ is near its critical value and panel (b) shows that R reaches a steady state. If diﬀusion amongst the glial cells is absent (red triangles), panel (a) shows that λ goes supercritical. Also, R keeps increasing over time.

In the ﬁnal experiment, we show that the 3-dimensional map can qualitatively reproduce the behavior of the full model. In this case we use two variants of the 3-D map: with (Eqs. (3.36),

(3.37), (3.39)) and without (Eqs. (3.36), (3.37), (3.38)) noise. In the ﬁrst part of this experiment, we show that the map with noise can be used to predict full model behavior for some arbitrary parameter setting. As an illustrative example, we ﬁx arbitrary values for source and consumption

−4 −4 −5 rates C1 = 48 × 10 ,C2 = 8 × 10 and diﬀusion rate, D = 5 × 10 and simulate the model and 80

1.02 (a) 1.01

D = D = D 0.99 G S D = 0, D = D G S 0.98 0.5 1 1.5 2 2.5 3 106

104 1.545 (b) 1.54

1.535

1.53

1.525 0.5 1 1.5 2 2.5 3 106

Figure 4.9: In a heterogeneous interaction network, G, the critical state is stable only if resources are allowed to transport amongst the glial cells. Panel (a) shows that when there is diffusion amongst glia (blue circles), the largest eigenvalue of the neural network adjacency matrix, λ ≈ 1. However, if resource transport amongst the glial cells is absent, the neural network becomes supercritical with λ > 1 (red triangles). The dashed line shows λ = 1 for reference. Panel (b) shows that the total resource R reaches a steady state when diffusion is turned on (blue circles) and keeps increasing over time if diffusion is turned off (red triangles). This further highlights the importance of resource-transport dynamics on the stability of the critical state. 81

Simulations (a) Map with noise (b) 1 1 ) ) t t

( 0.5 ( 0.5 S S 0 0 5000 6000 7000 8000 9000 10000 1.5 1.6 1.7 1.8 1.9 2 t t ×104 1.1 1.1 λ 1 λ 1

0.9 0.9 5000 6000 7000 8000 9000 10000 1.5 1.6 1.7 1.8 1.9 2 t t ×104 2.27 2.225 i i i i 2.26 2.22 R R h h 2.25 2.215 5000 6000 7000 8000 9000 10000 1.5 1.6 1.7 1.8 1.9 2 t t ×104

setting C1 C2

1 6 × 10−8 10−8 2 6 × 10−7 10−7 3 6 × 10−6 10−6 4 6 × 10−5 10−5 5 6 × 10−4 10−4 6 12 × 10−4 2 × 10−4 7 24 × 10−4 4 × 10−4 8 48 × 10−4 8 × 10−4 9 96 × 10−4 16 × 10−4 10 192 × 10−4 32 × 10−4

(d)

Figure 4.10: 3-Dimensional map reproduces behavior of the full model: We show that the behavior of the full model can be qualitatively described using the 3-dimensional map. Panels (a) and (b) show the plots for the model simulations and the map with noise respectively. The 3-D map reproduces qualitatively the plots for activity, S, the largest eigenvalue of the neural adjacency matrix, λ, and the average glial resource, hRii, for a particular setting of C1,C2. For this particular setting, we see oscillations for λ and hRii with peaks for S that correspond to the rising edges of λ and hRii. Panel (c) plots the standard deviation σ(λ) as a function of C1 for 10 diﬀerent settings for C1,C2 shown in the table in panel (d). For any given setting, since λ ﬂuctuates about its mean value near 1, lower values of σ(λ) indicate a tighter control near criticality. As seen, increasing C1 or C2 results in a higher σ(λ) and thus lesser control of the critical state. Furthermore, there is excellent agreement between values of σ(λ) from the 3-D map without noise (dashed black line), map with noise (blue circles) and those obtained from model simulations (red triangles). Additionally, the grey area highlights the region of parameter space for C1 or C2 where the critical state is predicted to be unstable by the 3-D map without noise (see text for explanation). 82 the 3-D map with noise for a total of 20000 time steps and plot the average activity, S, largest eigenvalue of the neural adjacency matrix, λ and the average glial resource, hRii. Panels (a) and

(b) of Fig. 4.10 show the results of the model simulations and the 3-D map with noise respectively.

These images demonstrate that the 3-D map qualitatively reproduces the model simulation. Though not shown here, note that such an agreement holds for a wide range of parameter settings.

In the second part of this experiment, we are interested in quantifying the deviation of the system from criticality as we vary model parameters. To quantify this, we use the standard deviation in the time series of λ that we denote as σ(λ). Since λ ﬂuctuates about 1 for any selected parameter setting, σ(λ) captures the deviation of the system from criticality. Since an exhaustive search of parameter space is infeasible, we select parameter settings along the C1 axis of Fig. 3.6 in

−5 Sec. 3.3.2. That is, ﬁxing D = 5 × 10 , we plot σ(λ) as a function of parameter C1 in panel (c) of

Fig. 4.10. The parameter values for the 10 settings are shown in panel (d) of Fig. 4.10. As shown, we increase C2 proportional to C1. Panel (c) shows that increasing C1 or C2 results in reduced control of λ = 1 as σ(λ) increases.

Furthermore, panel (c) shows that both 3-D map variants quantitatively recover the curve for

σ(λ) as the agreement with model simulations is quite excellent. Finally, the grey area highlights the region of parameter space for C1 or C2 such that one of the inequalities that governs the stability of λ = 1 ﬁxed point, i.e., Eq. (3.63), is violated (see Sec. 3.3.2). This inequality serves as one of the conditions, derived from the map without noise, that must be satisﬁed for a stable critical state.

The map without noise thus predicts roughly the onset of instability, with results from the 3-D map with noise and model simulations agreeing quite well.

Finally, we calculated the avalanche size distributions using the time series for S for all 10 setting from the model simulations. For the ﬁrst four settings (see panel (d) of Fig. 4.10) where

σ(λ) is small, we obtain plausible power-law ﬁts with the exponent near its characteristic value of

−3/2. For the remaining 6 settings where σ(λ) starts to increase, we do not get plausible power- law fits and the fitted region is confined to a very small range of avalanche sizes. In particular, the number of samples in the fitted region denoted by Nsamp is so small that the value of the 83

Kolmogorov-Smirnov (KS) statistic, KS > √ 1 . Thus, the 3-D map can be quite helpful in Nsamp predicting good and bad parameter regimes for the stability of the critical state. Chapter 5

Conclusions & future work

The study of neural network learning mechanisms has a long history– from Aristotle’s early observation that repeated causal activation sequences are necessary to link mental representations

[112] to the most current view on the subject such as the STDP learning rule. While there is ample experimental evidence for STDP [18], incorporating it in a computational model is nontrivial. In a recurrent network of neurons, STDP, or indeed any other Hebbian plasticity mechanism, results in synapses becoming either too strong or too weak resulting in saturation or dying out of the neural network activity respectively. As pointed out in [19, 113], some homeostatic plasticity mechanisms are needed to dynamically adjust synapse strengths in the correct direction so as to promote network stability.

In some computational models of neural networks, the dynamics can be well-characterized by the largest eigenvalue, λ, of the neural network adjacency matrix [12, 13]. Incorporating STDP in such existing computational models [13], we observe that, over time, λ becomes greater than 1.

This results in saturation of the network activity (i.e., attainment of a supercritical state). Thus it is vital to answer the question– How can the neural network maintain λ ≈ 1 in the face of learning?

In this thesis, we described a novel answer to this fundamental question; that the transport of resources in the secondary glial cell network can play a crucial role in stabilizing the dynamics of the primary neural network. In particular, we studied the following two variants of this idea:

(1) Model 1 operates at the scale of individual neurons such that we have both excitatory

and inhibitory synapses and learning via spike-timing-dependent plasticity. In this case, 85

we were interested in providing a plausible hypothesis to the long-standing question– How

does the human brain achieve the opposing goals of stability and learning? Using numerical

simulations of this model, we collected promising results: (i) regardless of the initial state,

our model can eﬀectively stabilize the dynamics of learning, (ii) diﬀusion of metabolic

resources amongst the glial cells is vital to maintain stable overall dynamics, (iii) our

model can learn and remember. During learning, the neural network becomes subcritical

and during rest, it is restored back to a balanced cortical state recovering λ ≈ 1. Although

the model does not maintain the critical state during learning we are encouraged that a

similar observation has also been made for in vivo experiments on human beings [109].

(2) Model 2 operates at a mesoscopic scale such that a node may be considered as a group of

neurons and the edges can be modeled with only positive weights, making the model purely

excitatory. This can be useful in practical contexts where recordings are obtained from an

aggregate response of a set of neurons that are modeled as a single node. We used this model

to provide a plausible hypothesis for how critical dynamics can be stable. In a simpliﬁed

version of our model, without learning and inhibition, we have shown that the resource-

transport dynamics complements the neuronal dynamics such that the system is poised at

the desired critical state. For this version of the model, our main results are as follows:

(i) Under suitable model parameters, the simpliﬁed model can self-organize to a critical

state that is characterized by power-law distributed avalanche sizes with an exponent value

near the characteristic −3/2 exponent found in various experimental studies, (ii) this state

is robust to a wide range of model parameters, (iii) using heterogeneous source rates or a

heterogeneous interaction network, the critical state is stable only if the diﬀusion of resource

between the glial cells is turned on that further emphasizes the need for resource transport,

(iv) theory correctly predicts the evolution of λt and Rt. Finally, we show that (v) the

3-dimensional map with or without the noise term can correctly reproduce the behavior of

the full model. 86

Adding more realistic features to any variant of our model such as refractory states for neurons, time delays for synaptic transmission, recycling of resources via the glial network, etc. may provide potential avenues for future studies. While in our present work we study the dynamics for simple homogeneous networks, the study of network dynamics with more complicated network topologies such as networks with communities may be closer to biologically observed network structure. To conclude, we outline some possible avenues to further the ideas presented in this thesis and discuss how some of the ideas used here may be applicable outside the ﬁeld of dynamics on complex networks.

5.1 A more realistic view of resource-transport dynamics

Glia perform several regulatory roles that enable neurotransmission [114]. Astrocytes, a particular type of glial cells, surround pre and postsynaptic terminals. During neurotransmission, astrocytes convert glutamate into glutamine and release it to the surrounding environment. Neurons consume glutamine to produce glutamate and GABA, the excitatory and inhibitory neurotrans- mitters respectively. Introducing this recycling mechanism of resource in our resource-transport dynamics would help add more realistic transport dynamics to our model.

5.2 Eﬀect of refractory states and time delays

The idea of resource production and consumption may indirectly model both refractoriness and time delays. For instance, assume that a certain neuron ﬁres consistently for some number of consecutive time steps. Since the synaptic resource gets consumed at a certain rate, such persistent

ﬁring can result in a state where all the outgoing synapses of such a neuron run out of resource.

Thus, in the subsequent time steps the neuron cannot excite any of its neighbors until at least one of its outgoing synapse receives some resource from the glial network. Thus, local resource consumption may force the neuron to be in a refractory period. Additionally, since there is an inherent delay in moving resources via the glial networks, some synapses might have to wait before they are able to transmit a signal from the presynaptic neuron to the postsynaptic neuron. Indirectly, then, 87 there may also be some delay in synaptic transmission as well.

Modeling these effects explicitly would allow us to have a finer control on refractoriness or transmission delay. As shown in panel (a) of Fig. 2.5, after a node is active, it becomes hyperpolarized and is unable to fire immediately. To model this we can introduce rn − 1 refractory

t t states so that the rn + 1 possible states for node n are: sn = 0 (resting), sn = 1 (active) and

t sn = {2, 3, . . . , rn} (refractory).

Introducing time delays and refractoriness in their model, Larremore et al. [95] found that the critical state is preserved with λ = 1 maximizing the dynamic range for homogeneous distributions of the number of refractory states and the amount of time delay. Also, considering synchronization dynamics, time delays can play an important role when modeling neural network dynamics; they can enhance synchrony in certain neuronal models such as in a network of Hindmarsh-Rose neurons

[115], while they may have more subtle eﬀects in other neuronal models such as the Rulkov map

[116].

To match biology, we could also introduce axonal conduction delay, i.e., the time required for an action potential to travel from the neuronal cell body to its axon terminals [14]. This addition to the model could be used to study both critical and synchronization dynamics. We can model this by modifying Eq. (3.30) as follows:

 N !  X t t−τnm  1 with probability σ Wnmsm , t+1  sn = m=1 (5.1)   0 otherwise .

t That is, if the postsynaptic neuron n is resting at time step t, i.e. sn = 0, then a presynaptic

t+1 neuron m can excite n in the next time step, i.e. sn = 1, with probability that now depends on the time delay for the action potential to travel from neuron m to neuron n. This is introduced by τnm. Finally, to study both refractoriness and time delays we can initially make a homogeneous assumption where rn = r or τnm = τ and latter study the likely realistic scenario of heterogeneous distributions for the number of refractory states and the amount of time delay. One of the questions of interest here could be– does the resource-transport dynamics stabilize the critical state on 88 including these realistic modeling choices?

5.3 Extensions of the learning experiment.

To illustrate our point in Experiment 3 of Sec. 4.1, we stimulate half the neural network in one time step. However, a more realistic experiment would be to divide the neural network into smaller communities of neurons, each representing some brain regions. Some questions of interest here are: Can memory be better retained in smaller communities? Does competing for resources impact learning simultaneously in two diﬀerent regions of the brain?

5.4 Eﬀects of complex network topologies.

The experiments in this thesis use the Erdös-Renyi network structure for both the neural and the glial networks. However, there is some evidence that the brain network structure is motivated by a tradeoff between minimizing wiring costs for synapses and maximizing navigability of information [117]. The Nash equilibria network structures could result in realizations of small-world network or other random networks. In any case, it is important to test our model using different network structures. How robust are the resource-transport dynamics of Sec. 3 to changes in network structure? Is the result, λ ≈ 1, also robust to such changes? These questions are left for future work.

5.5 Quantifying critical dynamics in neural networks.

Criticality in brain network dynamics is an often debated topic [46]. The classical signature of criticality, a power-law distribution for neuronal avalanche sizes, is often criticized as power laws can potentially emerge from noise (which can be exacerbated by learning) and hence they might not be representative of optimal brain function [118]. Also, we note that as shown in [101], quasi- critical behavior obtained for models like the ones described here can result in oscillations around the stable critical state. In this thesis, we showed that in general choosing smaller values of source rate (loading rate) and consumption rate (dissipating rate) reduces the amplitude of the oscillation 89 around the stable critical state resulting in power-law distributed avalanche size distributions for a wide range of parameters. Thus, to distinguish between quasi-critical and critical behavior, quantifying the critical state using avalanche size distributions may not be enough. Due to these reasons, we must look for diﬀerent quantitative ways to establish critical dynamics.

One of the potential ways to do this is to consider entropy measures such as the Shannon entropy measure for the neural network activity, S. For supercritical or subcritical regime, entroopy would be, in general, lower than that for the critical regime. We could do statistical tests to assess the diﬀerence between the entropy measures for the λ ≈ 1 state (the learning case, using our model) and for the λ = 1 state (the no-learning case using the existing mathematical model of [13]).

5.6 Applications to machine learning

Biological neural networks inspire many of the modeling philosophies in artificial neural networks. In the recent past, the idea of deep learning [105] (inspired in part from the multilayer structure of the human brain) has helped advance the field of artificial intelligence in diverse areas such as natural language processing, computer vision, etc. Machine learning algorithms frequently use the idea of regularization to prevent overfitting of model parameters to training data. Regularization techniques such as dropout [119] are also inspired by biology, namely, the idea that synaptic transmission fails at a certain rate. In the particular case of neural network learning algorithms, the idea of regularization is to penalize large values for model parameters, i.e., large values of synaptic weights. This ensures that the neural networks generalize well as they do not overfit the training data. Indeed, one of the potential ideas for regularization would be to constrain synaptic weights using resource-transport dynamics. Presently, the best learning algorithms for computer vision or natural language processing perform worse than or match humans. While the idea of resource-transport seems to be computationally costly, the following could be a very interesting theoretical question of interest to both neuroscience and machine learning communities- does the regulation of resource via the glial network makes the human brain generalize well? If this is indeed the case, it may be very useful to mimic this behavior in neural network learning 90 algorithms. Bibliography

[1] D. Plenz, E. Niebur, and H. G. Schuster. Criticality in Neural Systems. Wiley, 2014.

[2] D. Chialvo. Are our senses critical? Nature Physics, 2:301–302, 2006.

[3] D. Chialvo. Emergent complex neural dynamics. Nature Physics, 6:744–750, 2010.

[4] W. L. Shew and D. Plenz. The functional beneﬁts of criticality in the cortex. The Neuroscientist, 19:88–100, 2013.

[5] E. D. Gireesh and D. Plenz. Neuronal avalances organize as nested theta- and beta/gamma oscillations during development of cortical layer 2/3. Proceedings of the National Academy of Sciences, USA, 105:7576–7581, 2008.

[6] T. Petermann, T. C. Thiagarajan, M. A. Lebedev, M. A. L. Nicolelis, D. R. Chialvo, and D. Plenz. Spontaneous cortical activity in awake monkeys composed of neuronal avalanches. Proceedings of the National Academy of Sciences, USA, 106:15921–15926, 2009.

[7] T. L. Ribeiro, M. Copelli, F. Caixeta, H. Belchior, and et al. Spike avalanches exhibit universal dynamics across the sleep-wake cycle. PLoS ONE, 5:e14129, 2010.

[8] J. M. Beggs and D. Plenz. Neuronal avalanches in neocortical circuits. Journal of Neuroscience, 23:11167–11177, 2003.

[9] W. L. Shew, H. Yang, T. Petermann, R. Roy, and D. Plenz. Neuronal avalanches imply maximum dynamic range in cortical networks at criticality. The Journal of Neuroscience, 29:15595–15600, 2009.

[10] V. Pasquale, P. Massobrio, L. L. Bologna, M. Chiappalone, and S. Martinoia. Self- organization and neuronal avalanches in networks of dissociated cortical neurons. Neuroscience, 153:1354–1369, 2008.

[11] O. Kinouchi and M. Copelli. Optimal dynamical range of excitable networks at criticality. Nature Physics, 2:348–351, 2006.

[12] D. B. Larremore, W. L. Shew, and J. G. Restrepo. Predicting criticality and dynamic range in complex networks: eﬀects of topology. Physical Review Letters, 106:058101, 2011.

[13] D. B. Larremore, W. L. Shew, E. Ott, F. Sorrentino, and J. G. Restrepo. Inhibition causes ceaseless dynamics in networks of excitable nodes. Physical review letters, 112:138103, 2014. 92

[14] I. B. Levitan and L. K. Kaczmarek. The Neuron: Cell and Molecular Biology. Oxford Press, 2002.

[15] D. O. Hebb. The Organization of Behavior, A Neuropsychological Theory. New York, Wiley, 1949.

[16] W. B. Levy and D. Steward. Temporal contiguity requirements for long-term associative potentiation/depression in the hippocampus. Neuroscience, 8:791–797, 1983.

[17] N. Caporale and Y. Dan. Spike timing-dependent plasticity: A Hebbian learning rule. The Annual Review of Nueroscience, 31:25–46, 2008.

[18] H. Markram, W. Gerstner, and P. Jesper Sj¨ostr¨om. A history of spike-timing-dependent plasticity. Frontiers in Synaptic Neuroscience, 3(4):1–24, 2011.

[19] L. F. Abbott and S. B. Nelson. Synaptic plasticity: Taming the beast. Nature Neuroscience, 3:1178–1183, 2000.

[20] Y. S. Virkar, W. L. Shew, J. G. Restrepo, and E. Ott. Feedback control stabilization of critical dynamics via resource transport on multilayer networks: How glia enable learning dynamics in the brain. Physical Review E, 94:042310, 2016.

[21] C. Giaume, A. Koulakoﬀ, L. Roux, D. Holcman, and N. Rouach. Astroglial networks: a step further in neuroglial and gliovascular interactions. Nature Reviews Neuroscience, 11:87–99, 2010.

[22] R. D. Fields, D. H. Woo, and P J. Basser. Glial regulation of the neuronal connectome through local and long-distant communication. Neuron, 86:374–386, 2015.

[23] A. Zuzuki et al. Astrocyte-neuron lactate transport is required for long-term memory formation. Cell, 144:810–823, 2011.

[24] L. A. Newman, D. L. Korol, and P. E. Gold. Lactate produced by glycogenolysis in astrocytes regulates memory processing. PLoS ONE, 6:e28427, 2011.

[25] N. Rouach, A. Koulakoﬀ, V. Abudara, K. Willecke, and C. Giaume. Astroglial metabolic networks sustain hippocampal synaptic transmission. Science, 322:1551–1555, 2008.

[26] M. Tsacopoulos and P. J. Magistretti. Metabolic coupling between glia and neurons. Journal of Neuroscience, 16:877–885, 1996.

[27] L. Pellerin and P. J. Magistretti. Glutamate uptake into astrocytes stimulates aerobic gly- colysis: a mechanism coupling neuronal activity to glucose utilization. Proceedings of the National Academy of Sciences, USA, 91:10625–10629, 1994.

[28] J. Wade et al. Biophysically based computational models of astrocyte neuron coupling and their functional signiﬁcance. Frontier E-books, 2014.

[29] S. Y. Gordleeva et al. Bi-directional astrocytic regulation of neuronal activity within a network. Frontiers in Computational Neuroscience, 6:92, 2012. 93

[30] R. Jolivent, J. S. Coggan, I. Allaman, and P. J. Magistretti. Multi-timescale modeling of activity-dependent metabolic coupling in the neuron-glia vasculature ensemble. PLoS Computational Biology, 11:e1004036, 2015.

[31] I. A. Simpson, A. Carruthers, and S. J. Vannucci. Supply and demand in cerebral energy metabolism: the role of nutrient transporters. Journal of Cerebral Blood Flow and Metabolism, 27:1766–1791, 2007.

[32] D. D. Feldman. The spike-timing dependence of plasticity. Neuron, 75:556–571, 2012.

[33] V. Delattre, D. Keller, M. Perich, H. Markram, and E. B. Muller. Network-timing-dependent plasticity. Frontiers in Cellular Neuroscience, 9:220, 2015.

[34] R. Min, M. Santello, and T. Nevian. The computational power of astrocyte mediated synaptic plasticity. Frontiers in Computational Neuroscience, 6:1–15, 2012.

[35] J. Acebr´on,L. Bonilla, C. Vicente, F. Ritort, and R. Spigler. The Kuramoto model: a simple paradigm for synchronization phenomena. Reviews of Modern Physics, 77:137, 2005.

[36] J. G. Restrepo, E. Ott, and B. Hunt. Onset of synchronization in large networks of coupled oscillators. Physical Review E, 71:036151, 2005.

[37] S. Kaufmann. Metabolic stability and epigenesis in randomly constructed genetic nets. Journal of Theoretical Biology, 22:437, 1969.

[38] A. Pomerance, E. Ott, M. Girvan, and W. Losert. The eﬀect of topology on the stability of discrete state models of genetic control. Proceedings of the National Academy of Sciences, USA, 106:8209, 2009.

[39] K. D. Harris, J. Csicsvari, H. Hirase, G. Dragoi, and G. Buzs´aki.Organization of cell assem- blies in the hippocampus. Nature, 424:552, 2003.

[40] T. Sasaki, N. Matsuki, and Y. Ikegaya. Metastability of active ca3 networks. Journal of Neuroscience, 27:517, 2007.

[41] M. G. Kitzbichler et al. Broadband criticality of human brain network. PLoS Computational Biology, 5:e1000314, 2009.

[42] B. Krishnamachari, S. Wicker, and R. Bejar. Phase transition phenomena in wireless ad-hoc networks. in GLOBECOM, san Antonio, TX, 2001.

[43] A. Levina, J. M. Herrmann, and T. Geisel. Dynamical synapses causing self-organized criticality in neural networks. Nature Physics, 3:857–860, 2007.

[44] S. Bornholdt and T. Rohlf. Topological evolution of dynamical networks: Global criticality from local dynamics. Physical Review Letters, 84:6114, 2000.

[45] M. Kivel¨a,A. Arenas, M. Barthelemy, J. P. Gleeson, Y. Moreno, and M. A. Porter. Multilayer networks. Journal of Complex networks, 2:203–271, 2014.

[46] P. Massobrio, L. de Arcangelis, V. Pasquale, H. J. Jensen, and D. Plenz. Criticality as a signature of healthy neural systems. Frontiers in systems neuroscience, 9:22, 2015. 94

[47] D. B. Larremore. Critical dynamics in complex excitable networks. Thesis: University of Colorado, Boulder, 2012.

[48] J. M. Beggs and N. Timme. Being critical of criticality in the brain. Frontiers in physiology, 3:163, 2012.

[49] J. M. Beggs and D. Plenz. Neuronal avalanches are diverse and precise activity patterns that are stable for many hours in cortical slice cultures. Journal of Neuroscience, 24:5216–5229, 2004.

[50] C. Meisel, E. Olbrich, O. Shriki, and P. Achermann. Fading signatures of critical brain dynamics during sustained wakefulness in humans. Journal of Neuroscience, 33:17363–17372, 2013.

[51] W. L. Shew, H. Yang, S. Yu, R. Roy, and D. Plenz. Information capacity and transmission are maximized in balanced cortical networks with neuronal avalanches. The Journal of Neuroscience, 31:55–63, 2011.

[52] W. L. Shew, W. P. Clawson, J. Probst, Y. Karimipanah, N. C. Wright, and R. Wessel. Adaptation to sensory input tunes visual cortex to criticality. Nature Physics, 11:659–663, 2015.

[53] C. Haldeman and J. M. Beggs. Critical branching captures activity in living neural networks and maximizes the number of metastable states. Physical Review Letters, 94:058101, 2005.

[54] S. S. Poil, R. Hardstone, H. D. Mansvelder, and K. Linkenkaer-Hansen. Critical-state dynamics of avalanches and oscillations jointly from balanced excitation/inhibition in neuronal networks. Journal of Neuroscience, 32:9817–9823, 2012.

[55] L. de Arcangelis, C. Perrone-Capano, and H. J. Herrmann. Self-organized criticality model for brain plasticity. Physical review letters, 96:028107, 2006.

[56] Y. Dan and M. Poo. Spike timing-dependent plasticity: From synapse to perception. Physiological Reviews, 86:1033–1048, 2006.

[57] H. Markram, J. Lubke, M. Frotscher, and B. Sakmann. Regulation of synaptic eﬃcacy by coincidence of postsynaptic aps and epsps. Science, 275:213–215, 1997.

[58] J. C. Magee and D. Johnston. A synaptically controlled, associative signal for hebbian plasticity in hippocampal neurons. Science, 275:209–213, 1997.

[59] G. Q. Bi and M. M. Poo. Synaptic modiﬁcations in cultured hippocampal neurons: dependence on spike timing, synaptic strength and postsynaptic cell type. Journal of Neuroscience, 18:10464–10472, 1998.

[60] E. M. Izhikevich. Dynamical systems in neuroscience: the geometry of excitability and bursting. The MIT Press, 2007.

[61] N. J. Allen and B. A. Barres. Glia– more than just brain glue. Nature, 457:675–677, 2009.

[62] C. Eroglu, B. A. Barres, and B. Stevens. Glia as active participants in the development and function of synapses. In: Structural and Functional Organization of the Synapse. Springer, 2008. 95

[63] F. A. Azevedo, L. R. Carvalho, L. T. Grinberg, J. M. Farfel, R. E. Ferretti, R. E. Leite, W. Jacob Filho, R. Lent, and S. Herculano-Houzel. Equal numbers of neuronal and non- neuronal cells make the human brain an isometrically scaled-up primate brain. Journal of Comparative Neurology, 512(532–541), 2009.

[64] M. M. Halassa, T. Fellin, H. Takano, J. H. Dong, and P. G. Haydon. Synaptic islands deﬁned by the territory of a single astrocyte. Journal of Neuroscience, 27:6473–6477, 2007.

[65] U. Pannasch, L. Vargov´a,J. Reingruber, P. Ezan, D. Holcman, C. Giaume, E. Sykov´a,and N. Rouach. Astroglial networks scale synaptic activity and plasticity. Proceedings of the National Academy of Sciences, USA, 108:8467–8472, 2010.

[66] D. Stellwagen and R. C. Malenka. Synaptic scaling mediated by glial tnf-alpha. Nature, 440:1054–1059, 2006.

[67] S. Mangia, I. A. Simpson, S. J. Vannucci, and A. Carruthers. The in vivo neuron-to-astrocyte lactate shuttle in human brain: evidence from modeling of measured lactate levels during visual stimulation. Journal of Neurochemistry, 109:55–62, 2009.

[68] A. Aubert, L. Pellerin, P. J. Magistretti, and R. Costalat. A coherent neurobiological frame- work for functional neuroimaging provided by a model integrating compartmentalized energy metabolism. Proceedings of the National Academy of Sciences, USA, 104:4188–4193, 2007.

[69] A. Neves, R. Costalat, and L. Pellerin. Determinants of brain cell metabolic phenotypes and energy substrate utilization unraveled with a modeling approach. PLoS Computational Biology, 8:e1002686, 2012.

[70] R. Jolivent, J. S. Coggan, I. Allaman, and P. J. Magistretti. Multi-timescale modeling of activity-dependent metabolic coupling in the neuron-glia-vasculature ensemble. PLoS Computational Biology, 11:e1004036, 2015.

[71] S. Boccaletti, G. Bianconi, R. Criado, C. I. del Genio, J. Gómez-Gardeñes,M. Romance, I. Sendiña-Nadal,Z. Wang, and M. Zanin. The structure and dynamics of multilayer networks. Physics Reports, 544:1–122, 2014.

[72] A.-L. Barab´asiand R. Albert. Emergence of scaling in random networks. Science, 286:509– 512, 1999.

[73] A. Clauset, C. R. Shalizi, and M. E. J. Newmann. Power-law distributions in empirical data. SIAM Review, 51:661–703, 2009.

[74] D. J. Watts and S. H. Strogatz. Collective dynamics of ‘small-world’ networks. Nature, 393:440–442, 1998.

[75] M. A. Porter, J. Onnela, and P. J. Mucha. Communities in networks. Notices of the AMS, 56:1082–1097, 2009.

[76] V. Nicosia, P. S. Skardal, A. Arenas, and V. Latora. Collective phenomena emerging from the interactions between dynamical processes in multiplex networks. Physical Review Letters, 118:138302, 2017. 96

[77] M. De Domenico, C. Granell, M. A. Porter, and A. Arenas. The physics of spreading processes in multilayer networks. Nature Physics, 12:901–906, 2016.

[78] J. Gómez-Gardeñes,C. Gracia-Lázaro,L. M. Flor´ıa,and Y. Moreno. Evolutionary dynamics on interdependent populations. Physical Review E, 86:056113, 2012.

[79] Y. Kuramoto. Chemical oscillations, waves and turbulence. Springer, 1984.

[80] L. Edvinsson and D. N. Krause. Cerebral Blood Flow and Metabolism. Williams & Wilkins, Philadelphia, 2002.

[81] D. Malonek et al. Vascular imprints of neuronal activity: Relationships between the dynamics of cortical blood ﬂow, oxygenation, and volume changes following sensory stimulation. Proceedings of the National Academy of Sciences, USA, 94:14826, 1997.

[82] E. A. Allen, B. N. Pasley, T. Doung, and R. D. Freeman. Transcranial magnetic stimulation elicits coupled neural and hemodynamic consequences. Science, 317:1918, 2007.

[83] A. Hodgkin and A. Huxley. Currents carried by sodium and potassium ions through the membrane of the giant axon of loligo. The Journal of Physiology, 116:449–472, 1952.

[84] A. Hodgkin and A. Huxley. The components of membrane conductance in the giant axon of loligo. The Journal of Physiology, 116:473–496, 1952.

[85] A. Hodgkin and A. Huxley. The dual eﬀect of membrane potential on sodium conductance in the giant axon of loligo. The Journal of Physiology, 116:497–506, 1952.

[86] A. Hodgkin and A. Huxley. A quantitative description of membrane current and its ap- plication to conduction and excitation in nerve. The Journal of Physiology, 117:500–544, 1952.

[87] A. Hodgkin, A. Huxley, and B. Katz. Measurement of current-voltage relations in the membrane of the giant axon of loligo. The Journal of Physiology, 116:424–448, 1952.

[88] R. Fitzhugh. Impulses and physiological states in theoretical models of nerve membrane. Biophysical Journal, 1:445–466, 1961.

[89] J. Nagumo, S. Arimoto, and S. Yoshizawa. An active pulse transmission line simulating nerve axon. Proceedings of the IRE, 50:2061–2070, 1962.

[90] W. Gerstner and W. Kistler. Spiking Neuron Models: Single neurons, populations, plasticity. Cambridge University Press, 2002.

[91] J. M. Greenberg and S. P. Hastings. Spatial patterns for discrete models of diﬀusion in excitable media. SIAM Journal on Applied Mathematics, 34:515–523, 1978.

[92] J. Reiser and H. Matthews. Response properties of isolated mouse olfactory receptor cells. Journal of Physiology, 530:113–122, 2001.

[93] A. Tomaru and T. Kurahashi. Mechanisms determining the dynamic range of the bullfrog olfactory receptor cell. Journal of Neurophysiology, 93:1880–1888, 2005. 97

[94] S. S. Stevens. Psychophysics: Introduction to its Perceptual, Neural and Social Prospects. Wiley, New York, 1975.

[95] D. B. Larremore, W. L. Shew, E. Ott, and J. G. Restrepo. Eﬀects of network topolgy, transmission delays, and refractoriness on the response of coupled excitable systems to a stochastic stimulus. Chaos, 21:025117, 2011.

[96] S. Pei, S. Tang, S. Yan, S. Jiang, X. Zhang, and Z. Zheng. How to enhance the dynamic range of excitatory-inhibitory excitable networks. Physical Review E, 86:021909, 2012.

[97] A. Klaus, S. Yu, and D. Plenz. Statistical analyses support power law distributions found in neuronal avalanches. PLoS ONE, 6:e19779, 2011.

[98] D. B. Larremore, M. Y. Carpenter, E. Ott, and J. G. Restrepo. Statistical properties of avalanches in networks. Physical Review E, 85:066131, 2012.

[99] K. D. Miller and D. J. C. MacKay. The role of constraints in hebbian learning. Neural Computation, 6:100–126, 1994.

[100] G. G. Turrigiano, K. R. Leslie, N. S. Desai, L. C. Rutherford, and S. B. Nelson. Activity- dependent scaling of quantal amplitude in neocortical neurons. Nature, 391:892–896, 1998.

[101] J. A. Bonachela and M. A. Mu noz. Self-organization without conservation: True or just apparent scale-invariance? Journal of Statistical Mechanics Theory and Experiment, 2009:P09009, 2009.

[102] S. S. Kety. The general metabolism of the brain in vivo. In: Metabolism of the nervous system, London: Pergamon, pages 221–237, 1957.

[103] L. Sokoloﬀ. The metabolism of the central nervous system in vivo. In: Handbook of Physiology, Sec. I, Neurophysiology, APS, 3:1843–1864, 1960.

[104] A. Wallraﬀ and et al. The impact of astrocytic gap junctional coupling on potassium buﬀering in the hippocampus. Journal of Neuroscience, 26:5438–5447, 2006.

[105] Y. LeCun, Y. Bengio, and G. Hinton. Deep learning. Nature, 521:436–444, 2015.

[106] G. G. Turrigiano. The self-tuning neuron: Synaptic scaling of excitatory synapses. Cell, 135:422–435, 2008.

[107] J. G. Restrepo, E. Ott, and B. Hunt. Approximating the largest eigenvalue of network adjacency matrices. Physical Review E, 76:056119, 2007.

[108] D. L. Meinecke and A. Peters. Gaba immunoreactive neurons in rat visual cortex. Journal of Computational Neurology, 261:388, 1987.

[109] E. D. Fagerholm, R. Lorenz, G. Scott, M. Dinov, P. J. Hellyer, N. Mirzaei, C. Leeson, D. W. Carmichael, D. J. Sharp, W. L. Shew, and R. Leech. Cascades and cognitive state: Focused attention incurs subcritical dynamics. The Journal of Neuroscience, 35:4626–4634, 2015.

[110] D. Langlois, D. Cousineau, and J. P. Thivierge. Maximum likelihood estimators for truncated and censored power-law distributions show how neuronal avalanches may be misevaluated. Physical Review E, 89:012709, 2014. 98

[111] M. L. Goldstein, S. A. Morris, and G. G. Yen. Problems with ﬁtting to the power-law distribution. European Physics Journal B, 41:255–258, 2004.

[112] Y. Fr´egnac. “Hebbian synaptic plasticity”, The Handbook of Brain Theory and Neural Networks. The MIT Press, 2002.

[113] G. G. Turrigiano. Homeostatic plasticity in neuronal networks: the more things change, the more they stay the same. Trends in Neurosciences, 22:221–227, 1999.

[114] C. Eroglu and B. A. Barres. Regulation of synaptic connectivity by glia. Nature, 468:223–231, 2010.

[115] M. Dhamala, V. K. Jirsa, and M. Ding. Enhancement of neural synchrony by time delay. Physical Review Letters, 92:074104, 2004.

[116] Q. Wang, M. Perc, Z. Duan, and G. Chen. Synchronization transitions on scale-free neuronal networks due to ﬁnite information transmission delays. Physical Review E, 80:026206, 2009.

[117] A. Gulyás,J. J. B´ıró,A. K orösi,G. Rétvári,and D. Krioukov. Navigable networks as Nash equilibria of navigation games. Nature Communications, 6:7651, 2015.

[118] J. Touboul and A. Destexhe. Can power-law scaling and neuronal avalances arise from stochastic dynamics? PLoS ONE, 5:e8982, 2010.

[119] N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov. Dropout: A simple way to prevent neural networks from overﬁtting. Journal of Machine Learning Research, 15:1929–1958, 2014.