Recurrent Network Models of Sequence Generation and Memory

Article

Highlights Authors d Sequences emerge in random networks by modifying a small Kanaka Rajan, Christopher D. Harvey, fraction of their connections David W. Tank d Analysis reveals new circuit mechanism for input-dependent Correspondence sequence propagation [email protected] (K.R.), [email protected] (C.D.H.), d Sequential activation may provide a dynamic mechanism for [email protected] (D.W.T.) short-term memory In Brief Rajan et al. show that neural sequences similar to those observed during memory- based decision-making tasks can be generated by minimally structured networks. Sequences may effectively mediate the short-term memory engaged in these tasks.

Rajan et al., 2016, Neuron 90, 128–142 April 6, 2016 ª2016 Elsevier Inc. http://dx.doi.org/10.1016/j.neuron.2016.02.009 Neuron Article

Recurrent Network Models of Sequence Generation and Memory

Kanaka Rajan,1,* Christopher D. Harvey,2,* and David W. Tank3,* 1Joseph Henry Laboratories of Physics and Lewis–Sigler Institute for Integrative Genomics, Princeton University, Princeton, NJ 08544, USA 2Department of Neurobiology, Harvard Medical School, Boston, MA 02115, USA 3Department of Molecular Biology and Princeton Neuroscience Institute, Princeton University, Princeton, NJ 08544, USA *Correspondence: [email protected] (K.R.), [email protected] (C.D.H.), [email protected] (D.W.T.) http://dx.doi.org/10.1016/j.neuron.2016.02.009

SUMMARY sequence. The ubiquity of neural sequences suggests that they are of widespread functional use and thus may be produced Sequential activation of neurons is a common feature by general circuit-level mechanisms. of network activity during a variety of behaviors, Sequences can be produced by highly structured neural cir- including working memory and decision making. cuits or by more generic circuits adapted through the learning Previous network models for sequences and mem- of a specific task. Highly structured circuits of this type have a ory emphasized specialized architectures in which long history (Kleinfeld and Sompolinsky, 1989; Goldman, a principled mechanism is pre-wired into their 2009), e.g., as synfire chain models (Hertz and Pru¨ gel-Bennett, 1996; Levy et al., 2001; Hermann et al., 1995; Fiete et al., connectivity. Here we demonstrate that, starting 2010), in which excitation flows unidirectionally from one active from random connectivity and modifying a small neuron to the next along a chain of connected neurons, or as fraction of connections, a largely disordered recurring attractor models (Ben-Yishai et al., 1995; Zhang, 1996), in rent network can produce sequences and implement which increased (central) excitation between nearby neurons working memory efficiently. We use this process, surrounded by long-range inhibition and asymmetric connectiv- called Partial In-Network Training (PINning), to model ity are responsible for time-ordered neural activity. These and match cellular resolution imaging data from the models typically require imposing a task-specific mechanism posterior parietal cortex during a virtual memory- (e.g., for sequences) into their connectivity, producing special- guided two-alternative forced-choice task. Analysis ized networks. Neural circuits are highly adaptive and involved of the connectivity reveals that sequences propagate in a wide variety of tasks, and sequential activity often emerges by the cooperation between recurrent synaptic inter- through learning of a task and retains significant variability. It is therefore unlikely for highly structured approaches to produce actions and external inputs, rather than through models with flexible circuitry or to generate dynamics with the feedforward or asymmetric connections. Together temporal complexity needed to recapitulate experimental data. our results suggest that neural sequences may In contrast, random networks interconnected with excitatory emerge through learning from largely unstructured and inhibitory connections in a balanced state (Sompolinsky network architectures. et al., 1988), rather than being specifically designed for one single task, have been modified by training to perform a variety of tasks (Buonomano and Merzenich, 1995; Buonomano, 2005; Williams INTRODUCTION and Zipser, 1989; Pearlmutter, 1989; Jaeger and Haas, 2004; Maass et al., 2002, 2007; Sussillo and Abbott, 2009; Jaeger, Sequential firing has emerged as a prominent motif of population 2003). Here we built on these lines of research and asked whether activity in temporally structured behaviors, such as short-term a general implementation using relatively unstructured random memory and decision making. Neural sequences have been networks could create sequential dynamics resembling experi- observed in many brain regions including the cortex (Luczak mental data. We used data from sequences observed in the et al., 2007; Schwartz and Moran, 1999; Andersen et al., 2004; posterior parietal cortex (PPC) of mice trained to perform a two- Pulvermu¨ ller and Shtyrov, 2009; Buonomano, 2003; Ikegaya alternative forced-choice (2AFC) task in a virtual reality environ- et al., 2004; Tang et al., 2008; Seidemann et al., 1996; Fujisawa ment (Harvey et al., 2012), and we also constructed models et al., 2008; Crowe et al., 2010; Harvey et al., 2012), hippocam- that extrapolated beyond these experimental data. pus (Na´ dasdy et al., 1999; Louie and Wilson, 2001; Pastalkova To address how much network structure is required for se- et al., 2008; Davidson et al., 2009), basal ganglia (Barnes et al., quences like those observed in the recordings, we introduced 2005; Jin et al., 2009), cerebellum (Mauk and Buonomano, a new modeling framework called Partial In-Network Training 2004), and area HVC of the songbird (Hahnloser et al., 2002; or PINning. In this scheme, any desired fraction of the initially Kozhevnikov and Fee, 2007). In all these instances, the observed random connections within the network can be modified by a sequences span a wide range of time durations, but individual synaptic change algorithm, enabling us to explore the full range neurons fire transiently only during a small portion of the full of network architectures between completely random and fully

128 Neuron 90, 128–142, April 6, 2016 ª2016 Elsevier Inc. structured. Using networks constructed by PINning, we ﬁrst of a balanced set of excitatory and inhibitory weights drawn inde- demonstrated that sequences resembling the PPC data were pendently from a random distribution (Experimental Procedures). most consistent with minimally structured circuitry, with small In random network models, when excitation and inhibition are amounts of structured connectivity to support sequential activity balanced on average, the ongoing dynamics have been shown patterns embedded in a much larger fraction of unstructured to be chaotic (Sompolinsky et al., 1988); however, the presence connections. Next we investigated the circuit mechanism of of external stimuli can channel the ongoing dynamics in these sequence generation in such largely random networks contain- networks by suppressing their chaos (Molgedey et al., 1992; Bert- ing some learned structure. Finally, we determined the role se- schinger and Natschla¨ ger, 2004; Rajan et al., 2010, 2011). In ex- quences play in short-term memory, e.g., by storing information periments, strong inputs have also been shown to reduce the during the delay period about whether a left or right turn was indi- Fano factor and trial-to-trial variability associated with sponta- cated early in the 2AFC task (Harvey et al., 2012). Going beyond neous activity (Churchland et al., 2010; White et al., 2012). Thus, models meant to reproduce the experimental data, we analyzed we asked whether PPC-like sequences could be constructed multiple sequences initiated by different sensory cues and either from the spontaneous activity or the input-driven dynamics computed the capacity of this form of short-term memory. of random networks. We simulated a random network of rate- based neurons operating in a chaotic regime (Figure 1A; Experi- RESULTS mental Procedures; N = 437, network size chosen to match the dataset under consideration). The individual ﬁring rates were Sequences from Highly Structured or Random Networks normalized by the maximum over the trial duration (10.5 s, Fig- Do Not Match PPC Data ure 1C) and sorted in ascending order of their times of center-of-

Our study is based on networks of rate-based model neurons, in mass (tCOM), matching the procedures applied to the experimental which the outputs of individual neurons are firing rates and the data (Harvey et al., 2012). Although the resulting ordered sponta- units are interconnected through excitatory and inhibitory synap- neous activity was sequential (not shown, but similar to Figure 1B), ses of various strengths (Experimental Procedures). To interpret the level of extra-sequential background (bVar = 5% ± 2% and the outputs of the rate networks in terms of experimental data, pVar = 0.15% ± 0.1%) was higher than data. Sparsifying this we extracted firing rates from the calcium fluorescence signals background activity by increasing the threshold of the sigmoidal recorded in the PPC using two complementary deconvolution activation function increased bVar to a maximum of 22%, still methods (Figures 1C and 1D; see also Figure S1 and Experi- considerably smaller than the data value of 40% (Figure S4). mental Procedures). We defined two measures to compare the Next we added time-varying inputs to the random network to rates from the model to the rates extracted from data. The first, represent the visual stimuli in the virtual environment (right panel bVar, measures the stereotypy of the data or the network output of Figure 1A; Experimental Procedures). The sequence obtained by quantifying the variance explained by the translation of an ac- by normalizing and sorting the rates from the input-driven random tivity profile with an invariant shape (Figure 1G; Experimental network did not match the PPC data (bVar = 10% ± 2% and Procedures). The second metric, pVar, measures the percentage pVar = 0.2% ± 0.1%; Figures 1B, left panel of 1H, and S4). variance of the PPC data (Harvey et al., 2012) captured by the External inputs and disordered connectivity were insufficient different network outputs, and it is useful for tracking network to evoke sequences resembling the data (Figure 1B), which are performance across different parameters (Figure 1H; Experi- more structured and temporally constrained (Figures 1C, 1D, mental Procedures). bVar and pVar are used throughout this pa- and 2A compared to Figure 1B). Therefore, sequences like those per (Figures 1, 2, 5, and 6). observed during timing and memory experiments (Harvey et al., Many models have suggested that highly structured synaptic 2012) are unlikely to be an inherent property of completely connectivity, i.e., containing ring-like or chain-like interactions, random networks. Furthermore, since neural sequences arise is responsible for neural sequences (Ben-Yishai et al., 1995; during the learning of various experimental tasks, we asked Zhang, 1996; Hertz and Pru¨ gel-Bennett, 1996; Levy et al., 2001; whether initially disordered networks could also be modified by Hermann et al., 1995; Fiete et al., 2010). We therefore first asked training to produce realistic sequences. how much of the variance in long-duration neural sequences observed experimentally, e.g., in the PPC (Harvey et al., 2012), Temporally Constrained Neural Sequences Emerge with measured as bVar, was consistent with a bump of activity moving Synaptic Modification across the network, as expected from highly structured connec- To construct networks that match the activity observed in the tivity. The stereotypy of PPC sequences, which were averaged PPC, we developed a training scheme, Partial In-Network over hundreds of trials and pooled across different animals, Training or PINning, in which a user-defined fraction of synap- was found to be quite small (bVar = 40% for Figures 1D and ses were modified (Experimental Procedures). In this synaptic 2A). bVar was lower in both single-trial data (10%–15% for the modification scheme, the inputs to individual model neurons data in Figure S13 of Harvey et al., 2012) and trial-averaged were compared directly with target functions or templates data from a single mouse (15% for the data in Figure 2c of Harvey derived from the experimental data (Fisher et al., 2013), on et al., 2012). The small fraction of the variance explained by a both left and right correct-choice outcome trials from 437 moving bump (low bVar), combined with a weak relationship be- trial-averaged neurons, pooled across six mice, during a tween the activity of a neuron and anatomical location in the PPC 2AFC task (Figures 2A and 5D). During training, the internal syn- (Figure 5d in Harvey et al., 2012), motivated us to consider aptic weights in the connectivity matrix of the recurrent network network architectures with disordered connectivity composed were modified using a variant of the recursive least-squares

Neuron 90, 128–142, April 6, 2016 ª2016 Elsevier Inc. 129 Output from input-driven random network Figure 1. Stereotypy of PPC Sequences and Random Network Output A Random network + external inputs B Driven random network output (A) Schematic of a randomly connected network of rate neurons (blue) oper- 1 1 ating in a spontaneously active regime and a few example irregular inputs h1 (orange traces) are shown here. Random synapses are depicted in gray. Note that our networks were at least as big as the size of the data under consideration h2 (Harvey et al., 2012) and were typically all-to-all connected, but only a fraction of these neurons and their interconnections are depicted in these schematics. h3 (B) When the individual ﬁring rates of the random (untrained) network driven by external inputs in (A) were normalized by the maximum per neuron and

h4 Neurons, sorted sorted by their times of center-of-mass (tCOM), the activity appeared ordered...... However, this sequence contained large amounts of background activity hN Random 437 0 relative to the PPC data (bVar = 12%, pVar = 0.2%). synapses 2s 0 10.5 (C) Calcium fluorescence (i.e., normalized DF=F) data were collected from 437 Time, s trial-averaged PPC neurons during a 10.5 s-long 2AFC experiment from both Stereotypy of PPC sequence left and right correct-choice outcomes, pooled from six mice. Vertical gray C Calcium fluorescence from PPC D Firing rates extracted from data 1 1 1 1 lines indicate the time points corresponding to the different epochs of the task: the cue period ending at 4 s, the delay period ending at 7 s, and the turn period concluding at the end of the trial, at 10.5 s. (D) Normalized firing rates extracted from (C) using deconvolution methods are shown here (see also Figure S1). (E) The firing rates from the 437 neurons shown in (D) were realigned by their

tCOM and plotted here. Neurons, sorted Neurons, sorted (F) (Inset) A typical waveform (Rave in red), obtained by averaging the realigned rates shown in (E) over neurons, and a Gaussian curve with mean = 0 and variance = 0.3 (f(t), green) that best fit the neuron-averaged waveform are 437 0 437 0 0 10.5 0 10.5 plotted here. (Main panel) Residual activity not explained by translations of the Time, s Time, s best fit to Rave, f(t), is shown here. t E Firing rates realigned by COM F Average waveform and residual (G) Variance in the population activity explained by translations of the best fit to 1 1 1 0.7 1 R , f(t), is a measure of the stereotypy (bVar). Left panel shows the normalized R ave ave f(t) firing rates from four example PPC neurons (red) from (D) and the curves f(t) for each neuron (green); bVar = 40% for these data. Right panel shows the 0 normalized firing rates of four model neurons (red) from a network generating 1s an idealized sequence (see also Figure S2A) and the corresponding curves f(t) Neurons

Neurons for each (green); bVar = 100% here. (H) Variance of the PPC data explained by the outputs of different PINned networks is given by pVar and illustrated with four example neurons here. The left panel shows the normalized firing rates of four example PPC neurons (red) picked 437 0 437 –0.5 from (D) and four model neuron outputs from a random network driven by time- –5.2 0 5.2 –5.2 0 5.2 t t varying external inputs (gray, network schematized in A) with no training (p =0). Time relative to COM, s Time relative to COM, s For this example,pVar = 0.2%. The right panel shows the same PPC neurons asin G Variance explained by moving bump dynamics bVar the left panel in red, along with four model neuron outputs from a sparsely PINned bVar = 40% bVar = 100% 1 1 network with p = 12% plastic synapses. For this example, pVar =85%. PPC Idealized f(t) neuron sequence target 0 0 (RLS) or first-order reduced and controlled error (FORCE) learning rule (Haykin, 2002; Sussillo and Abbott, 2009) until the network rates matched the target functions (Experimental Procedures). Crucially, the learning rule was applied only to all the synapses of a randomly selected and often small fraction Example neuron firing rates Example neuron firing rates of neurons, such that only a fraction p (pN2 << N2) of the total 0 10.5 0 10.5 number of synapses was modified (plastic synapses, depicted Time, s Time, s in orange in Figures 2B, 5A, and 6A). While every neuron had H Variance of PPC data explained by network outputs pVar pVar = 0.2% pVar = 85% a target function, only the outgoing synapses from a subset 1 1 of neurons (i.e., from pN chosen neurons) were subject to Random network PINned network neuron, p = 0 neuron, p = 12% the learning rule (Figure 2B; Experimental Procedures). The 0 0 remaining elements of the synaptic matrix remained unmodified and in their random state (random synapses, depicted in gray in Figures 1A, 2B, 5A, and 6A). The PINning method therefore provided a way to span the entire spectrum of possible neural architectures, from disordered networks with random connec-

Example neuron firing rates Example neuron firing rates tions (p = 0 for Figures 1A and 1B), through networks with partially structured connectivity (p < 25% for Figures 2, 5, and 0 10.5 0 10.5 Time, s Time, s 6), to networks containing entirely trained connections (p = 100%, Figures 3D–3F).

130 Neuron 90, 128–142, April 6, 2016 ª2016 Elsevier Inc. A Rates from correct trials of 2AFC task B PINning all outgoing synapses from a few network neurons C Output of PINned network (p = 12%) 1 1 1 1 Input to network Firing rate of neuron i i z (t) pVar = 85% Plastic neuron , i synapses i 1s

AFTER BEFORE PINning 1s Target derived from Ca2+ data, fi(t) Neurons, sorted Neurons, sorted Error, zi(t) – fi(t)

Synaptic update, ΔJij(t) 437 0 437 0 0 10.5 j = 1 ... pN 0 10.5 Time, s Time, s D Variance of data explained increases with p E Magnitude of synaptic change required F Dimensionality of sequential activity

100 10 100 Q eff = 14 95 85 Q = 38 80 8 80 eff

pVar, % pVar, 40 eff Q 60 6 60 30 20

40 4 40 10

Dimensionality 0 05 12 20 25 20 2 20 p, % PINned network, p = 12% Normalized mean synaptic change Variance of data explained Variance Random network

0 0 Cumulative variance explained by PC, % 0 0 12 20 40 60 80 100 0 20 40 60 80 100 0 10 20 30 40 50 Fraction of plastic synapses p, % Fraction of plastic synapses p, % Principal component

Figure 2. Partial In-Network Training Matches PPC-like Sequences (A) This is identical to Figure 1C. (B) Schematic of the activity-based modification scheme we call Partial In-Network Training or PINning is shown here. Only the synaptic weights carrying inputs from a small and randomly selected subset, controlled by the fraction p of the N = 437 rate neurons (blue), were modified (plastic synapses, depicted in orange) at every time step by an amount proportional to the difference between the input to the respective neuron at that time step (zi(t), plotted in gray) and a target waveform (fi(t), plotted in green), the presynaptic firing rate rj, and the inverse cross-correlation matrix of the firing rates of PINned neurons (denoted by the matrix P; see Experimental Procedures). Here the target functions for PINning were extracted from the rates shown in (A). (C) Normalized activity from the network with p = 12% plastic synapses (red circle in D). This activity was temporally constrained, had a relatively small amount of extra-sequential activity (bVar = 40% for C and A), and showed a good match with data (pVar = 85%). (D) Effect of increasing the PINning fraction, p, in a network producing a single PPC-like sequence as in (A), is shown here. pVar increased from 0 for random networks with no plasticity (i.e., p =0)topVar = 50% for p = 8% (not shown) and asymptoted at pVar = 85% as p R 12% (highlighted by red circle, outputs in C).

(E) The total magnitude of synaptic change to the connectivity as a function of p is shown here for the matrix, JPINned, 12%. The normalized mean synaptic change grew from a factor of 7 for sparsely PINned networks (p = 12%) to 9 for fully PINned networks (p = 100%), producing the PPC-like sequence. This means that individual synapses changed more in small-p networks, but the total change across the synaptic matrix was smaller. (F) Dimensionality of the sequence is computed by plotting the cumulative variance explained by the different principal components (PCs) of the 437-neuron PINned network generating the PPC-like sequence (orange circles) with p = 12% and pVar = 85%, relative to a random network (p = 0, gray circles). When p = 12%, Qeff = 14, and smaller than the 38 for the random untrained network. Inset shows Qeff (depicted in red circles) of the manifold of the overall network activity as p increased.

We first applied PINning to a network with templates obtained simulate the effect of unobserved but active neurons present in from a single PPC-like sequence (Figures 2A–2C), using a range the data (Figure S3). The dependence of pVar on p is shown in of values for the fraction of plastic synapses, p. Modification of Figure 2D. only a small percentage (p = 12%) of the connections in a disor- Figure 2D quantifies the amount of structure needed dered network was sufficient for its sequential outputs to for generating sequences in terms of the relative fraction of become temporally constrained (bVar = 40%) and to match the synapses modified from their initially random values, but PPC data with high fidelity (pVar = 85%, Figure 2C; see Experi- what is the overall magnitude of synaptic change required? mental Procedures for cross-validation analysis). Although As shown in Figure 2E, we found that, although the individual our networks were typically as large as the size of the experi- synapses changed more in sparsely PINned (small p) net- mental dataset (N = 437 for Figures 1 and 2 and N = 569 for Fig- works, the total amount of change across the synaptic con- ure 5), our results are consistent both for larger N networks and nectivity matrix was smaller (other implications are described for networks in which non-targeted neurons are included to subsequently).

Neuron 90, 128–142, April 6, 2016 ª2016 Elsevier Inc. 131 A Sparsely PINned matrix B Distance-dependent synaptic interactions D 100% PINned matrix E Distance-dependent synaptic interactions J J PINned, 8% PINned, 100% 1 15 2 J 1 5 2 J p = 8% mean PINned, 8% +/– std. dev. p = 100% mean PINned, 100% +/– std. dev. J J mean Rand +/– std. dev. mean Rand +/– std. dev.

0 0 1 1 J J Postsynaptic neuron Postsynaptic neuron 500 –15 500 –5 1 500 0 1 500 0 J Presynaptic neuron Presynaptic neuron Rand F Distribution of weights Elements of 1 0.25 J Elements of p = 0 0 PINned, 100% J PINned, 8% J –1 Rand –1 –2 0

–4 –2 –2 Postsynaptic neuron 500 –0.25 –6 1 500 –200 0 200 10^ Log probability density, –15 –5 0 5 15 –200 0 200 Presynaptic neuron i – j Synaptic strength i – j

J J C Dynamics from the mean of PINned, 8% G Dynamics from the mean of PINned, 100% Mean connectivity Rates from the mean Mean connectivity Rates from the mean 1 1 1 1 1 0.5 1 1

0 0 Neurons, sorted Neurons, sorted Postsynaptic neuron Postsynaptic neuron 500 –1 500 0 500 –2 500 0 1 500 0 10.5 1 500 0 10.5 Presynaptic neuron Time, s Presynaptic neuron Time, s

Figure 3. Properties of PINned Connectivity Matrices (A) Synaptic connectivity matrix of a 500-neuron network with p = 8% producing an idealized sequence with pVar = 92% (highlighted by a red circle in Figure S2B), denoted by JPINned, 8%, and that of the randomly initialized network with p = 0, denoted by JRand, are shown here. Color bars indicate in pseudocolor the magnitudes of synaptic strengths after PINning. (B) Influence of neurons away from the sequentially active neurons was estimated by computing the mean (circles) and the standard deviation (lines) of the elements of JRand (in blue) and JPINned, 8% (means in orange, standard deviations in the red lines) in successive off-diagonal stripes away from the principal diagonal. These quantities are plotted as a function of the inter-neuronal distance, i j. In units of i – j, 0 corresponds to the principal diagonal or self-interactions, and the positive and the negative terms are the successive interaction magnitudes a distance i – j away from the primary sequential neurons. (C) Dynamics from the band averages of JPINned, 8% are shown here. The left panel is a synthetic matrix generated by replacing the elements of JPINned, 8% by their means (orange circles in B). The normalized activity from a network with this synthetic connectivity is shown on the right. Although there was a localized bump around i – j = 0 and long-range inhibition, these features led to fixed-point activity (right panel). (D) Same as (A), except the connectivity matrix from a fully PINned (p = 100%) network, denoted by JPINned, 100%, is shown. (E) Same as (B), except comparing JPINned, 100% and JRand. Band averages (orange circles) were bigger and more asymmetric compared to those for JPINned, 8%. Notably, these band averages were also negative for i – j = 0 and in the neighborhood of 0. (F) Logarithm of the probability density of the elements of JRand (gray squares), JPINned, 8% (yellow squares), and, for comparison, JPINned, 100% (red squares) is shown here. (G) Same as (C), except showing the firing rates from the mean JPINned, 100%. In contrast with JPINned, 8%, the band averages of JPINned, 100% could be sufficient to evoke a Gaussian bump qualitatively similar to the moving bump produced by a ring attractor model, since its movement is driven by the asymmetry in the mean connectivity.

To uncover how the sequential dynamics are distributed p values (inset of Figure 2F). The circuit dynamics are therefore across the population of active neurons in PINned networks, higher dimensional for the output of the PINned network than we used principal component analysis (PCA) (e.g., Rajan et al., for a sinusoidal bump attractor, but lower than the data. 2011; Sussillo, 2014; and references therein). For an untrained random network (here with N = 437 and p = 0) operating in a Circuit Mechanism for Sequential Activation through spontaneously active regime (Experimental Procedures), the PINning top 38 principal components accounted for 95% of the total vari- To develop a simpliﬁed prototype for further investigation of the ance (therefore, the effective dimensionality, Qeff = 38; gray cir- mechanisms of sequential activation, we created a synthetic cles in Figure 2F). In comparison, Qeff of the data in Figure 1D sequence idealized from data, as long as the duration of the is 24. In the network with p = 12% with outputs matching the PPC sequence (10.5 s, Figure 2A). We ﬁrst generated a Gaussian

PPC data, the dimensionality was lower (Qeff = 14; orange circles curve, f(t) (green curve in inset of Figure 1F and left panel of Fig- in Figure 2F). Qeff asymptoted around 12 dimensions for higher ure S2A) that best ﬁt the average extracted from the tCOM-aligned

132 Neuron 90, 128–142, April 6, 2016 ª2016 Elsevier Inc. PPC data (Rave, red curve in inset of Figure 1F and left panel of the band averages of JPINned, 8% (shown in Figure 3C) nor the fluc- Figure S2A). The curve, f(t), was translated uniformly over a tuations (not shown) were, by themselves, sufficient to produce 10.5-s period to derive a set of target functions (N = 500, right moving sequences similar to the full matrix JPINned, 8%. Instead, panel of Figure S2A, bVar = 100%). Increasing fractions, p,of the outputs of the synthetic networks built from the components the initially random connectivity matrix were trained by PINning of the band-averaged JPINned, 8% were stationary bumps (right with the target functions of the idealized sequence. As before, panel of Figure 3C). In this case, what causes the bump to move? pVar increased with p, reached pVar = 92% at p = 8% (red circle We considered the fluctuations around the band averages of in Figure S2B), asymptoting at p 10%. The plasticity required the sparsely PINned connectivity matrix, JPINned, 8%.Asexpected for producing the idealized sequence was therefore smaller than (see color bars in Figure 3A and lines in Figures 3B and 3E), the the p = 12% required for the PPC-like sequence (Figure 2D), due fluctuations around the band averages of JPINned, 8% (red lines in to the lack of the idiosyncrasies present in the data, e.g., irregu- Figure 3B) were much larger and more structured than those of larities in the temporal ordering of individual transients and back- JRand (small blue lines in Figure 3B). To uncover the mechanistic ground activity off the sequence. role of these fluctuations, we examined the input to each neuron Two features are critical for the production of sequential activ- produced by the sum of the fluctuations of JPINned, 8% and the ity. The first is the formation of a subpopulation of active neurons external input (Figure 4). We realigned the sums of the fluctuations (bump), maintained by excitation between co-active neurons and the external inputs for all the neurons in the network by their and restricted by inhibition. The second is an asymmetry in the tCOM and then averaged over neurons (e.g., see Experimental Pro- synaptic inputs from neurons ahead in the sequence and those cedures). This yielded an aligned population average (bottom right behind, needed to make the bump move. We therefore looked panel in Figure 4) that clearly revealed the asymmetry responsible for these two features in PINned networks by examining both for the movement of the bump across the network. Therefore, in the structure of their connectivity matrices and the synaptic cur- the presence of external inputs that are constantly changing in rents into individual neurons. time, the mean synaptic interactions do not have to be asym- In a classic moving bump/ring attractor network, the synaptic metric, as observed for JPINned, 8% (Figure 3B). Instead, the varia- connectivity is only a function of the distance between pairs of tions in the fluctuations of JPINned, 8% (i.e., after the mean has been network neurons in the sequence, i – j, with distance correspond- subtracted) and the external inputs create the asymmetry that ing to how close neurons appear in the ordered sequence (for our moves the bump along. It is difficult to visualize this asymmetry analysis of the connectivity in PINned networks, we also ordered at an individual neuron level because of fluctuation, necessitating the neurons in a similar manner). Furthermore, the structure that this type of population-level measure. While the mean synaptic in- sustains, constrains, and moves the bump is all contained in the teractions in sparsely PINned networks cause the formation of the connectivity matrix, localized in ji – jj, and asymmetric. We localized bump of excitation, it is the non-trivial interaction of the looked for similar structure in the trained networks. fluctuations in these synaptic interactions with the external inputs We considered three synaptic matrices interconnecting a pop- that causes the bump to move across the network. Therefore, this ulation of model neurons, N = 500, before PINning (randomly is a novel circuit mechanism for non-autonomous sequence prop- initialized matrix with p = 0, denoted by JRand), after sparse agation in a network. PINning (JPINned, 8%), and after full PINning (JPINned, 100%, built Additionally, we looked at other PINning-induced trends in the as a useful comparison). To analyze how the synaptic strengths elements of JPINned, 8% more directly. For all the three connectiv- in these three matrices varied with i – j, we first computed the ity matrices of sequential networks constructed by PINning, means and the standard deviations of the diagonals and the there was a substantial increase in the magnitudes spanned by means and the standard deviations of successive off-diagonal the synaptic weights as p decreased (compare the color bars bands moving away from the diagonals, i.e., i – j = constant in Figures 3A and 3D). These magnitude increases were mani- (Experimental Procedures). These band averages and the fluctu- fested in PINned networks with different p values in different ations were then plotted as a function of the interneuronal dis- ways (Figure 3F). The initial weight matrix JRand had 0 mean tance, i – j (Figures 3B and 3E). Next we generated ‘‘synthetic’’ (by design), 0.005 variance (of order 1/N, Experimental Proce- interaction matrices, in which all the elements along each diago- dures), 0 skewness, and 0 kurtosis. The partially structured ma- nal were replaced by their band averages and fluctuations, trix JPINned, 8% had a mean of –0.1, variance of 2.2, skewness at respectively. Finally, these synthetic matrices were used in net- –2, and kurtosis of 30, all of which were indicative of a probability works of rate-based neurons and driven by the same inputs as distribution that was asymmetric about 0 with heavy tails from a the original PINned networks (Experimental Procedures). small number of strong weights. This corresponds to a network In the p = 100% model, the band averages of JPINned, 100% in which the large sequence-facilitating synaptic changes formed a localized and asymmetric profile (orange circles in Fig- come from a small fraction of the weights, as suggested exper- ure 3E) that is qualitatively consistent with moving bump/ring imentally (Song et al., 2005). In JPINned, 8%, the ratio of the size of attractor dynamics (Ben-Yishai et al., 1995; Zhang, 1996). In the largest synaptic weight to the size of the typical is 20. If we contrast, the band averages for JPINned, 8% (orange circles in Fig- assume the typical synapse corresponded to a postsynaptic po- ure 3B) exhibited a localized zone of excitation for small values of tential (PSP) of 0.05 mV, then the large synapses had a 1 mV i – j that was symmetric, a significant inhibitory self-interactive PSP, which is within the range of experimental data (Song feature at i – j = 0, and diffuse flanking inhibition for larger values et al., 2005). For comparison purposes, the connectivity for a of i – j. This is reminiscent of the features expected in the station- fully structured network, JPINned, 100%, had a mean of 0, variance ary bump models (Ben-Yishai et al., 1995). Furthermore, neither of 0.7, skewness of –0.02, and kurtosis of 0.2, corresponding to a

Neuron 90, 128–142, April 6, 2016 ª2016 Elsevier Inc. 133 Figure 4. Mechanisms for Formation and Propagation of Sequence The currents from the matrix JPINned, 8% (red trace in the top left panel) and from its components are examined here. Band averages of JPINned, 8% caused the bump to form, as shown in the plot of the current from mean JPINned, 8% to one neuron in the network (red trace in the top panel on the right; see also Figure 3C). The cooperation of the ﬂuc- tuations (whose currents are plotted in red in the middle panel on the left) with the external inputs (in yellow in the bottom left) caused the bump to move. We demonstrated this by considering the currents from the ﬂuctuations around mean JPINned, 8% for all the neurons in the network combined with the currents from the external inputs. The summed

currents were realigned to the tCOM of the bump and then averaged over neurons. The resulting curve, an aligned population average of the sum of the ﬂuctuations and external inputs to the network, is plotted in orange in the bottom panel on the right and revealed the asymmetry responsible for the movement of the bump across the network.

we modeled the cue period and the turn period of the maze by two different inputs (blue and red traces in Figure 5C). In the middle third, the delay period, when the left- or right-specific visual cues were matrix in which the synaptic changes responsible for sequences off, the mouse ran through a section of the maze that was visually were numerous and distributed throughout the network. identical for both types of trials. In our simulation of this task, the Finally, we determined that synaptic connectivity matrices inputs to individual network neurons coalesced into the same obtained by PINning were fairly sensitive to small amounts of time-varying waveform during the delay periods of both the left structural noise, i.e., perturbations in the matrix JPINned, 8%. and the right trials (purple traces in Figure 5C). The correct However, when stochastic noise (Experimental Procedures) execution of the task therefore depended on the network gener- was used during training, slightly more robust networks were ob- ating more than one sequence to maintain the memory of the tained (Figure S6E). cues separately, even when the sensory inputs were identical. A network with only p = 16% plastic synapses generated out- Delayed Paired Association and Working Memory Can puts that were consistent with the data (Figures 5D and 5E, Be Implemented through Sequences in PINned pVar = 85%, bVar = 40%, compare with Figure 2c in Harvey Networks et al., 2012; here N = 569, with 211 network neurons selected Delayed paired association (DPA) tasks, such as the 2AFC task at random to activate in the left trial condition, depicted in blue from Harvey et al. (2012), engage working memory because in Figure 5A; 226 to fire in the right sequence, red in Figure 5A; the mouse must remember the identity of the cue, or cue-asso- and the remaining 132 to fire in the same order in both left and ciated motor response, as it runs through the T-maze. Therefore, right sequences, non-choice-specific neurons, depicted in green in addition to being behavioral paradigms that exhibit sequential in Figure 5A). This network retained the memory of cue identity activity (Harvey et al., 2012), DPA tasks are useful for exploring by silencing the left-preferring neurons during the delay period the different neural correlates of short-term memory (Gold and of a right trial and the right-preferring neurons during a left Shadlen, 2007; Brunton et al., 2013; Hanks et al., 2015; Amit, trial, and generating sequences with the active neurons. 1995; Amit and Brunel, 1995; Hansel and Mato, 2001; Hopfield Non-choice-specific neurons were sequentially active in the and Tank, 1985; Shadlen and Newsome, 2001; Harvey et al., same order in both trials, like real no-preference PPC neurons 2012). By showing that the partially structured networks we con- observed experimentally (Figures 5D and 5E; also Figure 7b in structed by PINning could accomplish a 2AFC task, here we Harvey et al., 2012). argued that sequences could mediate an alternative form of short-term memory. Comparison with Fixed-Point Memory Networks During the first third of the 2AFC experiment (Harvey et al., Are sequences a comparable alternative to fixed-point models 2012), the mouse received either a left or a right visual cue, for storing memories? We compared two types of sequential and during the last third, it had two different experiences de- memory networks with a fixed-point memory network (Figure 5F). pending on whether it made a left turn or a right turn. Therefore, As before, different fractions of plastic synapses, controlled by p,

134 Neuron 90, 128–142, April 6, 2016 ª2016 Elsevier Inc. A Schematic of PINned network of delayed paired association BExample firing rates CTask-relevant inputs Left Trial Right Trial Left/Right Common Left/Right inputs Plastic Left sequential inputs “inputs” synapses neuron h 1

h 2 Right sequential neuron

h 3 Non-choice-specific neuron

i ...... h N

Random BEFORE AFTER PINning synapses ΔJ (t) Synaptic update, ij 04710.5 j = 1 ... pN Time, s D Normalized firing rates extracted from PPC data E Normalized firing rates from PINned network (p = 16%) Left trial, Right preferring Right trial, Right preferring Left trial, Right preferring Right trial, Right preferring 1 1 1 1 Neurons, sorted Neurons, sorted

226 226 Left trial, Left preferring Right trial, Left preferring Left trial, Left preferring Right trial, Left preferring 1 1 Neurons, sorted Neurons, sorted

211 211

Left trial, Non-choice-specific Right trial, Non-choice-specific Left trial, Non-choice-specific Right trial, Non-choice-specific 1 1

132 0 132 0 0 4 7 10.5 0 4 7 10.5 04710.504710.5 Time, s Time, s Time, s Time, s F Selectivity of PINned networks of working memory G Magnitude of synaptic change required 1 10

0.91 0.85 0.81 0.8 8

0.6 6

0.4 4 Selectivity index

0.2 Normalized mean synaptic change 2 Idealized sequential memory network PPC-match delayed paired association network Idealized sequential memory network Fixed point memory network Fixed point memory network 0 0 040608010020 040608010020 10 16 23 Fraction of plastic synapses p, % Fraction of plastic synapses p, %

(legend on next page) Neuron 90, 128–142, April 6, 2016 ª2016 Elsevier Inc. 135 were embedded by PINning against different target functions ing to chance performance when the animal made a turn. Given a (Experimental Procedures). These targets represented the 50% probability of turning in the correct direction by chance, this values of a variable being stored and were chosen for the dynam- implies that the cue identity was forgotten on 30%–40% of trials. ical mechanism by which memory is implemented: idealized for Adding noise to our model, we could reproduce this level of delay h = : : the sequential memory network (orange in Figure 5F, based on period forgetting at a noise amplitude of c 0 4--0 5(Fig- Figure S2A), firing rates extracted from PPC data (Harvey ure S6B). It should be noted that this value is in a region where et al., 2012) for the PPC-like DPA network (green in Figure 5F, the model shows a fairly abrupt decrease in performance. outputs from the PPC-like DPA network with p = 16% plasticity are shown in Figure 5E), and constant valued targets for the Capacity of Sequential Memory Networks fixed-point-based memory network (blue in Figure 5F). We have discussed one specific instantiation of DPA through a To compare the task performance of these three types of sequence-based memory mechanism, the 2AFC task. We next memory networks, we computed a selectivity index (Experi- extended the same basic PINning framework to ask whether mental Procedures, similar to Figure 4 in Harvey et al., 2012). PINned networks could accomplish memory-based tasks that We found that the network exhibiting long-duration population require the activation of more than two non-interfering se- dynamics and memory activity through idealized sequences (or- quences. We also computed the capacity of such multi-sequen- ange triangles in Figure 5F) had a selectivity of 0.91, when only tial networks (denoted by Ns) as a function of the network 10% of its synapses were modified by PINning (p = 10%). In size (N), PINning fraction (p), fraction of non-choice-specific comparison, the PPC-like DPA network (Figure 5E) needed p = neurons (NNon-choice-specific/N), and temporal sparseness (fraction 16% of its synapses to be structured to match data (Harvey of neurons active at any instant, NActive/N, Experimental et al., 2012) and to achieve a selectivity of 0.85 (green triangles Procedures). in Figure 5F). Both sequential memory networks performed We adjusted the PINning parameters for a type of memory comparably with the fixed-point network in terms of their selec- task that is mediated by multiple non-interfering sequences (Fig- tivity-p relationship. The fixed-point memory network achieved ure 6). Different fractions of synaptic weights in an initially an asymptotic selectivity of 0.81 for p = 23%. The magnitude random network were trained by PINning to match different tar- of synaptic change required (Experimental Procedures) was gets (Figure 6A), non-overlapping sets of Gaussian curves also comparable between sequential and fixed-point memory (similar to Figure S2A), each evenly spaced and collectively networks, suggesting that sequences may be a viable alternative spanning 8 s. The width of these waveforms, NActive/N, was var- to fixed points as a mechanism for storing memories. ied as a parameter that controlled the sparseness of the During experiments, the fraction of trials on which the mouse sequence (Experimental Procedures). Because the turn period made a mistake was about 15%–20% (accuracy of the perfor- was omitted here for clarity, the modeled multi-sequential tasks mance of mice was found to be 83% ± 9% correct, Harvey were 8 s long. Similar to DPA (Figure 5), each neuron received a et al., 2012). We interpret errors as arising from trials in which different filtered white noise input for each cue during the cue delay period activity failed to retain the identity of the cue, lead- period (0–4 s); but, during the delay period (4–8 s), they

Figure 5. DPA in PINned Networks of Working Memory (A) Schematic of a sparsely PINned network implementing DPA through a 2AFC task (Harvey et al., 2012). The targets for PINning were firing rates extracted from Ca2+ imaging data from left-preferring PPC cells (schematized in blue), right-preferring cells (schematized in red), and cells with no choice preference (in green, called non-choice-specific neurons). As in Figure 2B, the learning rule was applied only to p% of the synapses. (B) Example single-neuron firing rates, normalized to the maximum, before (gray) and after PINning (blue trace for left-preferring, red trace for right-preferring, and green trace for non-choice-specific neurons) are shown here.

(C) A few example task-specific inputs (hi for i=1, 2, 3, ., N) are shown here. Each neuron got a different irregular, spatially delocalized, filtered white noise input, but received the same one on every trial. Left trial inputs are in blue and the right trial ones, in red. (D) Normalized firing rates extracted from trial-averaged Ca2+ imaging data collected in the PPC during a 2AFC task are shown here. Spike trains were extracted by deconvolution (see Experimental Procedures) from mean calcium fluorescence traces for the 437 choice-specific and 132 non-choice-specific, task- modulated cells (one cell per row) imaged on preferred and opposite trials (Harvey et al. 2012). These firing rates were used to extract the target functions for

PINning (schematized in A). Traces were normalized to the peak of the mean ﬁring rate of each neuron on preferred trials and sorted by the tCOM. Vertical gray lines indicate the time points corresponding to the different epochs of the task: the cue period ending at 4 s, the delay period ending at 7 s, and the turn period concluding at the end of the trial, at 10.5 s.

(E) The outputs of the 569-neuron network with p = 16% plastic synapses, sorted by tCOM and normalized by the peak of the output of each neuron, showed a match with the data (D). For this network, pVar = 85%. (F) Selectivity index, shown here, was computed as the ratio of the difference and the sum of the mean activities of preferred neurons at the 10-s time point during preferred trials and the mean activities of preferred neurons during opposite trials (see Experimental Procedures). Task performance of three different PINned networks of working memory as a function of the fraction of plastic synapses, p—an idealized sequential memory model (orange triangles, N = 500), a model that exhibits PPC-like dynamics (green triangles, N = 569, using rates from D), and a fixed-point memory network (blue triangles, N = 500)—are shown here. The network that exhibited long-duration population dynamics and memory activity through idealized sequences (orange triangles) had selectivity = 0.91, when p = 10% of its synapses were plastic; the PPC-like network needed p = 16% plastic synapses for selectivity = 0.85, and the fixed-point network needed p = 23% of its synapses to be plastic for selectivity = 0.81. (G) Magnitude of synaptic change (computed as for Figure 2E) for the three networks shown in (F) is plotted here. The idealized sequential (orange squares) and the fixed-point memory network (blue squares) required comparable amounts of mean synaptic change to execute the DPA task, growing from 3to9 for p = 10%–100%. Across the matrix, the total amount of synaptic change required was smaller, even though individual synapses changed more in the sparsely PINned case. The PPC-like memory network is omitted here for clarity.

136 Neuron 90, 128–142, April 6, 2016 ª2016 Elsevier Inc. A Schematic for multi-sequential memory networks B Network output, 5 choice-specific sequences Plastic N N /N p synapses Cue 1 = 500, Active = 3%, = 25%, 1 1

Cue 2 100

200 Cue 3

... 300 N Cue s Neurons, sorted 400

Random 500 0 synapses 048048048048048 Non-choice-specific neuron Time, s C Temporally sparse sequences rescued by inclusion of non-choice-specific neurons N N /N , p N /N p N /N = = 500, Active = 1.6% = 25%, Active = 1.6%, = 25% + Non-choice-specific 4% 1 1 1 1 20

100 116

200 212

300 308 Neurons, sorted Neurons, sorted 400 404

500 0 500 0 048048048048048 048048048048048 Time, s Time, s D Capacity of multi-sequential memory networks E Resilience to stochastic noise 25 0.5 p N /N = p = 100% + N /N = 4% = 100% + Non-choice-specific 4% Non-choice-specific p p = 100% = 100% p N /N = p N /N = = 25% + Non-choice-specific 4% = 25% + Non-choice-specific 4% 20 0.4 p = 25% s p = 25% N c η 15 0.3

10 0.2 Noise tolerance 5 0.1 Number of sequences

0 0 100300 500 700 900 1100 0 0.02 0.04 0.06 0.08 0.1 N /N Network size N Capacity : Network size s

Figure 6. Capacity of Multi-sequential Memory Networks

(A) Schematic of a multi-sequential memory network (N = 500) is shown here. N/Ns neurons were assigned to each sequence that we wanted the network to simultaneously produce. Here Ns is the capacity. Only p% of the synapses were plastic. The target functions were identical to Figure S2A, however, their widths, denoted by NActive/N, were varied as a task parameter controlling how many network neurons were active at any instant.

(B) The normalized firing rates of the 500-neuron network with p = 25% and NActive/N=3% are shown here. The memory task (the cue periods and the delay periods only) was correctly executed through five choice-specific sequences; during the delay period (4–8 s), neurons fired in a sequence only on trials of the same type as their cue preference.

(C) The left panel shows the same network as (B) failing to perform the task correctly when the widths of the targets were halved (NActive/N=1.6%), sparsifying the sequences in time. Memory of the cue identity was not maintained during the delay period. The right panel shows the result of including a small number of

(legend continued on next page)

Neuron 90, 128–142, April 6, 2016 ª2016 Elsevier Inc. 137 coalesced to a common cue-invariant waveform, different for rons were more stable against stochastic perturbations (Fig- each neuron. In the most general case, we assigned N/Ns neu- ure 6E). Once the amplitude of added stochastic noise exceeded h rons to each sequence. Here Ns was the number of memories the maximum tolerance (denoted by c, Experimental Proce- (capacity), which was also the number of trial types. Once again, dures), however, non-choice-specific neurons were not effective only p% of the synapses in the network were plastic. at repairing the memory capacity (not shown). Non-choice-spe- A correctly executed multi-sequential task is one in which, dur- cific neurons could not rescue inadequately PINned schemes ing the delay period, network neurons fire in a sequence only on either (small p, not shown). trials of the same type as their cue preference and are silent during other trials. A network of 500 neurons with p = 25% plastic DISCUSSION synapses performed such a task easily, generating Ns = 5 sequences with delay period memory of the appropriate cue iden- In this paper, we used and extended earlier work based on liquid- tity with temporal sparseness of NActive/N=3% (Figure 6B). state (Maass et al., 2002) and echo-state machines (Jaeger, The temporal sparseness was found to be a crucial factor 2003), which has shown that a basic balanced-state network that determined how well a network performed a multi-se- with random recurrent connections can act as a general purpose quential memory task (Experimental Procedures). When se- dynamical reservoir, the modes of which can be harnessed for quences were forced to be sparser than a minimum (e.g., by performing different tasks by means of feedback loops. These using narrower target waveforms, for this 8 s-long task, when models typically compute the output of a network through

NActive/N<1.6%), the network failed. Although there was weighted readout vectors (Buonomano and Merzenich, 1995; sequential activation, the memory of the five separately memo- Maass et al., 2002; Jaeger, 2003; Jaeger and Haas, 2004) rable cues was not maintained across the delay period. Surpris- and feed the output back into the network as an additional ingly, this failure could be rescued by adding a small number of current, leaving the synaptic weights within the dynamics- non-choice-specific neurons (NNon-choice-specific/N=4%) that producing network unchanged in their randomly initialized fired in the same order in all five trial types (Figure 6C). In other configuration. The result was that networks generating chaotic words, the N – NNon-choice-specific network neurons (

N, and inversely with sparseness, NActive/N (Figure 6D). The fied in a task-specific manner and by a biologically reasonable slope of this capacity-to-network size relationship increased amount, leaving the majority of the network heterogeneously when non-choice-specific neurons were included, because wired. Furthermore, the fraction of plastic connections is a they enabled the network to carry sparser sequences. We also tunable parameter that controls the contribution of the structured found that networks with a bigger fraction of plastic synaptic portion of the network relative to the random part, allowing these connections and networks containing non-choice-specific neu- models to interpolate between highly structured and completely

non-choice-specific neurons (NNon-choice-specific/N=4%) that fired in the same order in trials of all five types. Including non-choice-specific neurons restored the memory of cue identity during the delay period, without increasing p.

(D) Capacity of multi-sequential memory networks, Ns, as a function of network size, N, for different values of p is shown here. Mean values are indicated by orange circles for p = 25% PINned networks, red circles for p = 25% + NNon-choice-specific/N=4%, light blue squares for p = 100%, and dark blue squares for p = 100% + NNon-choice-specific/N=4% neurons. PINned networks containing additional non-choice-specific neurons had temporally sparse target functions with

NActive/N=1.6%, the rest had NActive/N=3%. Error bars were calculated over five different random instantiations each and decreased under the following conditions: as network size was increased, as fraction of plastic synapses p was increased, and moderately, with the inclusion of non-choice-specific neurons. The green square highlights the 500-neuron network whose normalized outputs are plotted in (B). h (E) Resilience of different multi-sequential networks, computed as the critical amount of stochastic noise (denoted here by c) tolerated before the memory task fails, is shown here. Tolerance, plotted here as a function of the ratio of multi-sequential capacity and network size, Ns/N, decreased as p was lowered and as capacity increased, although it fell slower with the inclusion of non-choice-specific neurons. Only mean values are shown here for clarity.

138 Neuron 90, 128–142, April 6, 2016 ª2016 Elsevier Inc. random architectures until we find the point that best matches the in Figure S5). In particular, the model in Fiete et al. (2010) gener- relevant experimental data (a similar point is made in Barak et al., ates highly stereotyped sequences similar to area HVC by initial- 2013). PINning is not the most general way to restrict learning to a izing as a recurrently connected network and subsequently using limited number of synapses, but it allows us to do so without spike timing-dependent plasticity, heterosynaptic competition, losing the efficiency of the learning algorithm. and correlated external inputs to learn sequential activity. A crit- To illustrate the applicability of such partially structured net- ical difference between the model in Fiete et al. (2010) and our works, we used data recorded from the PPC of mice performing approach lies in the circuitry underlying sequences: their learning a working memory-based decision-making task in a virtual reality rule results in synaptic chain-like connectivity. environment (Harvey et al., 2012). It might be unlikely that an as- In this paper, we suggest an alternative hypothesis that sociation cortex such as the PPC evolved specialized circuitry sequences might be a more general and effective dynamical solely for sequence generation, because the PPC mediates other form of working memory (Results), making the prediction that complex temporal tasks, e.g., from working memory and deci- sequences may be increasingly observed in many working sion making to evidence accumulation and navigation (Shadlen memory-based tasks (also suggested in Harvey et al., 2012). and Newsome, 2001; Gold and Shadlen, 2007; Freedman and This contrasts with previous models that relied on fixed-point at- Assad, 2011; Snyder et al., 1997; Andersen and Cui, 2009; Bisley tractors to retain information and exhibited sustained activity and Goldberg, 2003; McNaughton et al., 1994; Nitz, 2006; Whit- (Amit, 1992, 1995; Amit and Brunel, 1995; Hansel and Mato, lock et al., 2008; Calton and Taube, 2009; Hanks et al., 2015). We 2001; Hopfield and Tank, 1985). We computed the capacity also computed a measure of stereotypy from PPC data (bVar, of sequential memory networks for storing more than two Figures 1C–1G) and found it to be much lower than if the PPC memories (Results) by extending the sparse PINning approach were to generate sequences based on highly specialized chain- developed above, and we interpreted this capacity as the like or ring-like connections or if it read out sequential activity ‘‘computational bandwidth’’ of a general purpose circuit to from a structured upstream region. We therefore started with perform different timing-based computations. random recurrent networks, imposed a small amount of struc- The term pre-wired is used in this paper to mean a scheme in ture in their connectivity through PINning, and duplicated many which a principled mechanism for executing a certain task is first features of the experimental data (Harvey et al., 2012), e.g., assumed and then incorporated into the network circuitry, e.g., a choice-specific neural sequences and delay period memory of moving bump architecture (Ben-Yishai et al., 1995; Zhang, 1996) cue identity (Results). or a synfire chain (Hertz and Pru¨ gel-Bennett, 1996; Levy et al., We analyzed the structural features in the synaptic connectiv- 2001; Hermann et al., 1995; Fiete et al., 2010). In contrast, the ity matrices of sparsely PINned networks, concluding that the models built and described in this paper are constructed without probability distribution of the synaptic strengths is heavy tailed bias or assumptions. If a moving bump architecture (Ben-Yishai due to the presence of a small percentage of strong interaction et al., 1995; Zhang, 1996) had been assumed at the beginning terms (Figure 3F). There is experimental evidence that there and the network pre-wired accordingly, we would of course might be a small fraction of very strong synapses embedded have uncovered it through the analysis of the synaptic connectiv- within a milieu of relatively weak synapses in the cortex (Song ity matrix (similar to Results; Figures 3, 4, and S5; see also Exper- et al., 2005), most recently from the primary visual cortex (Cossell imental Procedures). However, by starting with an initially et al., 2015). Synaptic distributions measured in slices have been random configuration and learning a small amount of structure, shown to have long tails (Song et al., 2005), and the experimental we found an alternative mechanism for input-dependent result in Cossell et al. (2015) has demonstrated that rather sequence propagation (Figure 4). We would not have encoun- than occurring at random, these strong synapses significantly tered this mechanism for the non-autonomous movement of contribute to network tuning by preferentially interconnecting the bump by pre-wiring a different mechanism into the connec- neurons with similar orientation preferences. These strong syn- tivity of the model network. While the models constructed here apses may be the plastic synapses that are induced by PINning are indeed trained to perform the task, the fact that they are un- in our scheme. The model also exhibited some structural noise biased means that the opportunity was present to uncover sensitivity (Figure S6E), and this was not completely removed mechanisms that were not thought of a priori. by training in the presence of noise. It is possible that dynamic mechanisms of ongoing plasticity in neural circuits could EXPERIMENTAL PROCEDURES enhance stability to structural fluctuations. Network Elements In this paper, the circuit mechanisms underlying both the We considered a network of N fully interconnected neurons described by a formation of a localized bump of excitation in the connectivity standard firing-rate model, where N = 437 for the PPC-like sequence in Figures and the manner in which the bump of excitation propagates 1D and 2A, N = 500 for the single idealized sequence in Figure S2 and the multi- across the network were elucidated separately (Figures 3 and sequential memory task in Figure 6, and N = 569 for the 2AFC task in Figure 5 4). The circuit mechanism for the propagation of the sequence (we generally built networks of the same size as the experimental dataset we was found to be non-autonomous, relying on a complex interplay were trying to model, however, the results obtained remain applicable to larger between the recurrent connections and the external inputs. networks, Figure S3). For mathematical details of the network, see the Supple- mental Experimental Procedures. The mechanism for propagation was therefore distinct from the standard moving bump/ring attractor model (Ben-Yishai et al., Design of External Inputs 1995; Zhang, 1996), but it had similarities to the models devel- To represent sensory (visual and proprioceptive) stimuli innervating the oped in Fiete et al. (2010) and Hopfield (2015) (explored further PPC, the external inputs to the neurons in the network were made from

Neuron 90, 128–142, April 6, 2016 ª2016 Elsevier Inc. 139 filtered and spatially delocalized white noise that was frozen (repeated from Analyzing the Structure of PINned Synaptic Connectivity Matrices trial to trial, Figure 1A). For motivation of this choice and for mathematical This is pertinent to the Results (Figures 3 and 4), in which we quantified how the details of the frozen noise input, see the Supplemental Experimental Proce- synaptic strength varies with distance between pairs of network neurons in dures. In addition, we also tested the resilience of the memory networks we connectivity space, i–j. These were computed as described in the Supple- built (Figure 6E) to injected stochastic noise (see the Supplemental Exper- mental Experimental Procedures. See Figure 3B for the sparsely PINned imental Procedures). We defined resilience or noise tolerance as the critical matrix, JPINned, 8%, relative to the randomly initialized matrix, JRand, and Fig- h J amplitude of this stochastic noise, denoted by c, at which the delay period ure 3E for the fully PINned matrix, PINned, 100%. The same analysis also was memory failed and the selectivity dropped to 0 (Figure 6E; see also used for Figure S5. Figure S6). Cross-Validation Analysis Extracting Target Functions from Calcium Imaging Data This pertains to quantifying how well PINning-based models did and is To derive the target functions for our activity-dependent synaptic modification described in the Supplemental Experimental Procedures. scheme termed PINning, we converted the calcium fluorescence traces from the PPC data (Harvey et al., 2012) into firing rates using two complementary SUPPLEMENTAL INFORMATION methods. See the Supplemental Experimental Procedures for details of both deconvolution methods (see also Figure S1). Supplemental Information includes Supplemental Experimental Procedures and six figures and can be found with this article online at http://dx.doi.org/ Synaptic Modification Rule for PINning 10.1016/j.neuron.2016.02.009. During PINning, the inputs of individual network neurons were compared directly with the target functions extracted from data to compute a set of error AUTHOR CONTRIBUTIONS functions. During learning, the subset of plastic internal weights in the connectivity matrix of the random network, denoted by p, underwent modification at a Conceptualization, K.R., C.D.H., and D.W.T.; Methodology, K.R.; Formal Anal- rate proportional to the error, the presynaptic firing rate of each neuron, and a ysis, K.R.; Investigation, K.R.; Data Curation, C.D.H.; Writing – Original Draft, pN 3 pN matrix, P, that tracked the rate fluctuations across the network. For K.R.; Writing – Review and Editing, K.R., C.D.H., and D.W.T.; Resources, further details of the learning rule, see the Supplemental Experimental D.W.T.; Supervision, D.W.T. Procedures. ACKNOWLEDGMENTS Dimensionality of Network Activity, Qeff

We defined the effective dimensionality of the activity, Qeff, as the number of The authors thank Larry Abbott for providing guidance and critiques throughout principal components that captured 95% of the variance in the activity (Fig- this project; Eftychios Pnevmatikakis and Liam Paninski for their deconvolution ure 2F), computed as in the Supplemental Experimental Procedures. algorithm (Pnevmatikakis et al., 2016; Vogelstein et al., 2010); and Dmitriy Stereotypy of Sequence, bVar Aronov, Bill Bialek, Selmaan Chettih, Cristina Domnisoru, Tim Hanks, and Matthias Minderer for comments. This work was supported by the NIH bVar quantified the variance of the data or the network output explained by the (D.W.T., R01-MH083686, RC1-NS068148, and 1U01NS090541-01; C.D.H., translation of an activity profile with an invariant shape. See the Supplemental R01-MH107620 and R01-NS089521), a grant from the Simons Collaboration Experimental Procedures for details. on the Global Brain (D.W.T.), a National Alliance for Research on Schizophrenia Percentage Variance of Data Explained by Model, pVar and Depression (NARSAD) Young Investigator Award from the Brain & Behavior Research Foundation (K.R.), a fellowship from the Helen Hay Whitney Founda- We quantified the match between the experimental data or the set of target tion (C.D.H.), the New York Stem Cell Foundation (C.D.H.), and a Burroughs Well- functions and the outputs of the model by pVar, the amount of variance in come Fund Career Award at the Scientific Interface (C.D.H.). C.D.H. is a New the data captured by the model. Details of how this was computed are in the York Stem Cell Foundation-Robertson Investigator and a Searle Scholar. Supplemental Experimental Procedures.

Magnitude of Synaptic Change Received: August 22, 2015 Revised: December 3, 2015 In Figures 2E and 5G, we computed the magnitude of the synaptic change Accepted: February 2, 2016 required to implement a single PPC-like sequence, an idealized sequence, Published: March 10, 2016 and three types of memory tasks. In combination with the fraction of plastic synapses in the PINned network, p, this metric characterizes the amount REFERENCES of structure imposed in an initially random network to produce the desired temporally structured dynamics. See also the Supplemental Experimental Procedures. Amit, D.J. (1992). Modeling Brain Function: The World of Attractor Neural Networks (Cambridge University Press). Selectivity Index for Memory Task Amit, D.J. (1995). The Hebbian paradigm reintegrated: local reverberations as In Figure 5F, a selectivity index was computed (similar to Figure 4 in internal representations. Behav. Brain Sci. 18,617. Harvey et al., 2012) to assess the performance of different PINned net- Amit, D.J., and Brunel, N. (1995). Learning internal representations in an attrac- works at maintaining memories during the delay period of DPA tasks. tor neural network with analogue neurons. Network 6, 359–388. This metric is based on the ratio of the difference and the sum of the Andersen, R.A., and Cui, H. (2009). Intention, action planning, and decision mean activities of preferred neurons at the end of the delay period ( 10 s making in parietal-frontal circuits. Neuron 63, 568–583. for Figure 5,afterHarvey et al., 2012 and 7sforFigure 6) during preferred Andersen, R.A., Burdick, J.W., Musallam, S., Pesaran, B., and Cham, J.G. trials and the mean activities of preferred neurons during opposite (2004). Cognitive neural prosthetics. Trends Cogn. Sci. 8, 486–493. trials. This was computed as described in the Supplemental Experimental Procedures. Barak, O., Sussillo, D., Romo, R., Tsodyks, M., and Abbott, L.F. (2013). From ﬁxed points to chaos: three models of delayed discrimination. Prog. Neurobiol.

Temporal Sparseness of Sequences, NActive/N 103, 214–222. The temporal sparseness of a sequence is deﬁned as the fraction of neurons Barnes, T.D., Kubota, Y., Hu, D., Jin, D.Z., and Graybiel, A.M. (2005). Activity of active at any instant during the sequence, denoted here by NActive/N. This striatal neurons reﬂects dynamic encoding and recoding of procedural mem- was computed as in the Supplemental Experimental Procedures. ories. Nature 437, 1158–1161.

140 Neuron 90, 128–142, April 6, 2016 ª2016 Elsevier Inc. Ben-Yishai, R., Bar-Or, R.L., and Sompolinsky, H. (1995). Theory of orientation Hopfield, J.J. (2015). Understanding emergent dynamics: Using a collective tuning in visual cortex. Proc. Natl. Acad. Sci. USA 92, 3844–3848. activity coordinate of a neural network to recognize time-varying patterns. Bertschinger, N., and Natschla¨ ger, T. (2004). Real-time computation at the Neural Comput. 27, 2011–2038. edge of chaos in recurrent neural networks. Neural Comput. 16, 1413–1436. Hopfield, J.J., and Tank, D.W. (1985). ‘‘Neural’’ computation of decisions in optimization problems. Biol. Cybern. 52, 141–152. Bisley, J.W., and Goldberg, M.E. (2003). Neuronal activity in the lateral intra- parietal area and spatial attention. Science 299, 81–86. Ikegaya, Y., Aaron, G., Cossart, R., Aronov, D., Lampl, I., Ferster, D., and Yuste, R. (2004). Synfire chains and cortical songs: temporal modules of Brunton, B.W., Botvinick, M.M., and Brody, C.D. (2013). Rats and humans can cortical activity. Science 304, 559–564. optimally accumulate evidence for decision-making. Science 340, 95–98. Jaeger, H. (2003). Adaptive nonlinear system identification with echo state net- Buonomano, D.V. (2003). Timing of neural responses in cortical organotypic works. In Advances in Neural Information Processing Systems 15 (NIPS 2002), slices. Proc. Natl. Acad. Sci. USA 100, 4897–4902. S. Becker, S. Thrun, and K. Obermayer, eds. (MIT Press), pp. 593–600. Buonomano, D.V. (2005). A learning rule for the emergence of stable dynamics Jaeger, H., and Haas, H. (2004). Harnessing nonlinearity: predicting chaotic and timing in recurrent networks. J. Neurophysiol. 94, 2275–2283. systems and saving energy in wireless communication. Science 304, 78–80. Buonomano, D.V., and Merzenich, M.M. (1995). Temporal information trans- Jin, D.Z., Fujii, N., and Graybiel, A.M. (2009). Neural representation of time in formed into a spatial code by a neural network with realistic properties. cortico-basal ganglia circuits. Proc. Natl. Acad. Sci. USA 106, 19156–19161. Science 267, 1028–1030. Kleinfeld, D., and Sompolinsky, H. (1989). Associative network models for cen- Calton, J.L., and Taube, J.S. (2009). Where am I and how will I get there from tral pattern generators. In Methods in Neuronal Modeling, C. Koch and I. here? A role for posterior parietal cortex in the integration of spatial information Segev, eds. (MIT Press), pp. 195–246. and route planning. Neurobiol. Learn. Mem. 91, 186–196. Kozhevnikov, A.A., and Fee, M.S. (2007). Singing-related activity of identified Churchland, M.M., Yu, B.M., Cunningham, J.P., Sugrue, L.P., Cohen, M.R., HVC neurons in the zebra finch. J. Neurophysiol. 97, 4271–4283. Corrado, G.S., Newsome, W.T., Clark, A.M., Hosseini, P., Scott, B.B., et al. Laje, R., and Buonomano, D.V. (2013). Robust timing and motor patterns by (2010). Stimulus onset quenches neural variability: a widespread cortical phe- taming chaos in recurrent neural networks. Nat. Neurosci. 16, 925–933. nomenon. Nat. Neurosci. 13, 369–378. Levy, N., Horn, D., Meilijson, I., and Ruppin, E. (2001). Distributed synchrony in Cossell, L., Iacaruso, M.F., Muir, D.R., Houlton, R., Sader, E.N., Ko, H., Hofer, a cell assembly of spiking neurons. Neural Netw. 14, 815–824. S.B., and Mrsic-Flogel, T.D. (2015). Functional organization of excitatory syn- Louie, K., and Wilson, M.A. (2001). Temporally structured replay of awake hip- aptic strength in primary visual cortex. Nature 518, 399–403. pocampal ensemble activity during rapid eye movement sleep. Neuron 29, Crowe, D.A., Averbeck, B.B., and Chafee, M.V. (2010). Rapid sequences of 145–156. population activity patterns dynamically encode task-critical spatial informa- Luczak, A., Bartho´ , P., Marguet, S.L., Buzsa´ ki, G., and Harris, K.D. (2007). tion in parietal cortex. J. Neurosci. 30, 11640–11653. Sequential structure of neocortical spontaneous activity in vivo. Proc. Natl. Davidson, T.J., Kloosterman, F., and Wilson, M.A. (2009). Hippocampal replay Acad. Sci. USA 104, 347–352. of extended experience. Neuron 63, 497–507. Maass, W., Natschla¨ ger, T., and Markram, H. (2002). Real-time computing Fiete, I.R., Senn, W., Wang, C.Z., and Hahnloser, R.H.R. (2010). Spike-time- without stable states: a new framework for neural computation based on per- dependent plasticity and heterosynaptic competition organize networks to turbations. Neural Comput. 14, 2531–2560. produce long scale-free sequences of neural activity. Neuron 65, 563–576. Maass, W., Joshi, P., and Sontag, E.D. (2007). Computational aspects of feed- Fisher, D., Olasagasti, I., Tank, D.W., Aksay, E.R., and Goldman, M.S. (2013). back in neural circuits. PLoS Comput. Biol. 3, e165. A modeling framework for deriving the structural and functional architecture of Mauk, M.D., and Buonomano, D.V. (2004). The neural basis of temporal pro- a short-term memory microcircuit. Neuron 79, 987–1000. cessing. Annu. Rev. Neurosci. 27, 307–340. Freedman, D.J., and Assad, J.A. (2011). A proposed common neural mecha- McNaughton, B.L., Mizumori, S.J., Barnes, C.A., Leonard, B.J., Marquis, M., nism for categorization and perceptual decisions. Nat. Neurosci. 14, 143–146. and Green, E.J. (1994). Cortical representation of motion during unrestrained Fujisawa, S., Amarasingham, A., Harrison, M.T., and Buzsa´ ki, G. (2008). spatial navigation in the rat. Cereb. Cortex 4, 27–39. Behavior-dependent short-term assembly dynamics in the medial prefrontal Molgedey, L., Schuchhardt, J., and Schuster, H.G. (1992). Suppressing chaos cortex. Nat. Neurosci. 11, 823–833. in neural networks by noise. Phys. Rev. Lett. 69, 3717–3719. Gold, J.I., and Shadlen, M.N. (2007). The neural basis of decision making. Na´ dasdy, Z., Hirase, H., Czurko´ , A., Csicsvari, J., and Buzsa´ ki, G. (1999). Annu. Rev. Neurosci. 30, 535–574. Replay and time compression of recurring spike sequences in the hippocam- Goldman, M.S. (2009). Memory without feedback in a neural network. Neuron pus. J. Neurosci. 19, 9497–9507. 61, 621–634. Nitz, D.A. (2006). Tracking route progression in the posterior parietal cortex. Hahnloser, R.H.R., Kozhevnikov, A.A., and Fee, M.S. (2002). An ultra-sparse Neuron 49, 747–756. code underlies the generation of neural sequences in a songbird. Nature Pastalkova, E., Itskov, V., Amarasingham, A., and Buzsa´ ki, G. (2008). Internally 419, 65–70. generated cell assembly sequences in the rat hippocampus. Science 321, 1322–1327. Hanks, T.D., Kopec, C.D., Brunton, B.W., Duan, C.A., Erlich, J.C., and Brody, C.D. (2015). Distinct relationships of parietal and prefrontal cortices to evi- Pearlmutter, B.A. (1989). Learning state space trajectories in recurrent neural dence accumulation. Nature 520, 220–223. networks. Neural Comput. 1, 263–269. Hansel, D., and Mato, G. (2001). Existence and stability of persistent states in Pnevmatikakis, E.A., Soudry, D., Gao, Y., Machado, T.A., Merel, J., Pfau, D., large neuronal networks. Phys. Rev. Lett. 86, 4175–4178. Reardon, T., Mu, Y., Lacefield, C., Yang, W., et al. (2016). Simultaneous denoising, deconvolution, and demixing of calcium imaging data. Neuron Harvey, C.D., Coen, P., and Tank, D.W. (2012). Choice-specific sequences in 89, 285–299. parietal cortex during a virtual-navigation decision task. Nature 484, 62–68. Pulvermu¨ ller, F., and Shtyrov, Y. (2009). Spatiotemporal signatures of large- Haykin, S. (2002). Adaptive Filter Theory (Prentice Hall). scale synfire chains for speech processing as revealed by MEG. Cereb. Hermann, M., Hertz, J.A., and Pru¨ gel-Bennett, A. (1995). Analysis of synfire Cortex 19, 79–88. chains. Network 6, 403–414. Rajan, K., Abbott, L.F., and Sompolinsky, H. (2010). Stimulus-dependent sup- Hertz, J., and Pru¨ gel-Bennett, A. (1996). Learning short synfire chains by self- pression of chaos in recurrent neural networks. Phys. Rev. E Stat. Nonlin. Soft organization. Network 7, 357–363. Matter Phys. 82, 011903.

Neuron 90, 128–142, April 6, 2016 ª2016 Elsevier Inc. 141 Rajan, K., Abbott, L.F., and Sompolinsky, H. (2011). Inferring stimulus selec- Sussillo, D. (2014). Neural circuits as computational dynamical systems. Curr. tivity from the spatial structure of neural network dynamics. In Advances in Opin. Neurobiol. 25, 156–163. Neural Information Processing Systems 23 (NIPS 2010), J.D. Lafferty, C.K.I. Sussillo, D., and Abbott, L.F. (2009). Generating coherent patterns of activity Williams, J. Shawe-Taylor, R.S. Zemel, and A. Culotta, eds. (Neural from chaotic neural networks. Neuron 63, 544–557. Information Processing Systems Foundation). Tang, A., Jackson, D., Hobbs, J., Chen, W., Smith, J.L., Patel, H., Prieto, A., Schwartz, A.B., and Moran, D.W. (1999). Motor cortical activity during drawing Petrusca, D., Grivich, M.I., Sher, A., et al. (2008). A maximum entropy model movements: population representation during lemniscate tracing. applied to spatial and temporal correlations from cortical networks in vitro. J. Neurophysiol. 82, 2705–2718. J. Neurosci. 28, 505–518. Seidemann, E., Meilijson, I., Abeles, M., Bergman, H., and Vaadia, E. (1996). Vogelstein, J.T., Packer, A.M., Machado, T.A., Sippy, T., Babadi, B., Yuste, R., Simultaneously recorded single units in the frontal cortex go through and Paninski, L. (2010). Fast nonnegative deconvolution for spike train infer- sequences of discrete and stable states in monkeys performing a delayed ence from population calcium imaging. J. Neurophysiol. 104, 3691–3704. localization task. J. Neurosci. 16, 752–768. Shadlen, M.N., and Newsome, W.T. (2001). Neural basis of a perceptual deci- White, B., Abbott, L.F., and Fiser, J. (2012). Suppression of cortical neural vari- sion in the parietal cortex (area LIP) of the rhesus monkey. J. Neurophysiol. 86, ability is stimulus- and state-dependent. J. Neurophysiol. 108, 2383–2392. 1916–1936. Whitlock, J.R., Sutherland, R.J., Witter, M.P., Moser, M.B., and Moser, E.I. Snyder, L.H., Batista, A.P., and Andersen, R.A. (1997). Coding of intention in (2008). Navigating from hippocampus to parietal cortex. Proc. Natl. Acad. the posterior parietal cortex. Nature 386, 167–170. Sci. USA 105, 14755–14762. Sompolinsky, H., Crisanti, A., and Sommers, H.J. (1988). Chaos in random Williams, R.J., and Zipser, D. (1989). A learning algorithm for continuously neural networks. Phys. Rev. Lett. 61, 259–262. running fully recurrent neural networks. Neural Comput. 1, 270–280. Song, S., Sjo¨ stro¨ m, P.J., Reigl, M., Nelson, S., and Chklovskii, D.B. (2005). Zhang, K. (1996). Representation of spatial orientation by the intrinsic dy- Highly nonrandom features of synaptic connectivity in local cortical circuits. namics of the head-direction cell ensemble: a theory. J. Neurosci. 16, 2112– PLoS Biol. 3, e68. 2126.

142 Neuron 90, 128–142, April 6, 2016 ª2016 Elsevier Inc. Neuron, Volume 90

Supplemental Information

Recurrent Network Models of Sequence Generation and Memory

Kanaka Rajan, Christopher D. Harvey, and David W. Tank SUPPLEMENTAL INFORMATION

A Deconvolving Ca2+ into rates C Extracting firing rates from data Ca2+ data from 437 PPC neurons Figure S1 (related to Figure 1): Top Panel, Extracting Target Data Ca2+ 1 1 1 Functions from Ca2+ Fluorescence Data by Deconvolution Rate = P[spike] and Bottom Panel (D), Convergence of the PINning

0 Algorithm 1 A.The use of deconvolution algorithms for the extraction of firing rates is illustrated here. A few example PPC neurons 2+ 0 showing the Ca fluorescence signals in red (replotted from Neurons, sorted 1 [Harvey, Coen & Tank, 2012]) and their extracted firing rates (normalized to a maximum of 1) in green. These rates were 437 0 obtained using both methods described in Experimental 0 Firing rates extracted from data Procedures (see also Supplemental Experimental Procedures 1 1 1 S7.3). The implementation of the Bayesian algorithm (method #2 from Supplemental Experimental Procedures 0 0 10.5 S7.3) yields spike times along with the statistical confidence Time, s of a spike arriving at that particular time point (P[spike]). B Inferred Ca2+ from rates However, since the frame rate of the imaging experiments in 2+ 1 Data Ca [Harvey, Coen & Tank, 2012] was relatively slow (64ms per 2+ Neurons, sorted Inferred Ca frame), we used these “probabilistic” spike trains as a

0 437 0 normalized firing rate estimate. To verify, we first 2+ 1 Inferred Ca2+ from rates deconvolved single trial Ca data, and then performed an 1 1 average over all the single trial P[spike] estimates to obtain the smooth firing rates for each of the 437 sequential neurons 0 that we show in the middle panel of (C). 1 B.To see how accurate the firing rate estimates extracted from the data were, we re-convolved the extracted firing rates 0 1 of the example units in (A) through a difference of Neurons, sorted exponentials with a rise time of 52ms and decay time of 384ms (see Supplemental Experimental Procedures S7.3). 0 0 437 2+ 2+ 0 10.5 0 4 7 10.5 The PPC Ca data are plotted in red and the inferred Ca are Time, s Time, s plotted in blue here. Convergence of algorithm for multiple networks C.Top panel shows the original Ca2+ fluorescence signals 0.06 from the PPC (similar to Figure 2A of main text, adapted 2 Convergence criteria for one p = 8% network χ 0.45 from Figure 2c in [Harvey, Coen & Tank, 2012]). Middle panel shows the firing rates (from 0 to a maximum of 6Hz,

0.05 2 normalized by the peak to 1) extracted from these data by χ 0.35 using deconvolution methods. Bottom panel shows the Run 5 inferred Ca2+ signals obtained by convolving these extracted Run 4 0.04 0.25 firing rates by a Ca2+ impulse response function as explained Run 3 for (B) (see also Experimental Procedures 3 and Run 2 0.15 Supplemental Experimental Procedures S7.3). 0.03 Run 1

Error during learning 0.05

0.02 0 0 100 200 300 400 500 Learning steps per run

0.01 p = 8% p = 100%

Error across multiple networks after learning 0 0 500 1000 2000 3000 4000 5000 Maximum number of learning steps D. Bottom Panel shows the convergence of the PINning algorithm. The convergence of our PINning algorithm (Experimental Procedures 4, see also Supplemental Experimental Procedures S4) is shown here for multiple sequential networks generating sequential outputs (each, similar to Supplemental Figure S2A). We plot the ! χ 2 error between the target functions and the outputs of several PINned networks as a function of the number of learning steps for two values of p – p = 8% plastic synapses (mean values in the red circles, means computed over 5 instantiations each) and p = 100% plastic synapses (mean values in the yellow circles, means computed over 5 instantiations each) for comparison purposes. When the ! χ 2 error drops below 0.02, which for both sparsely and fully PINned networks occurs before the 500th learning step, we terminate the learning and simulate the network with the PINned connectivity matrix (denoted as JPINned, p% in general) for an additional 50 steps before the program graphs the network outputs (firing rates, inferred calcium to compare with data, statistics of JPINned, p%, etc.). The point highlighted in the gray square corresponds to a PINned network with p = 8% that ran for 500 learning steps, at the end of which, the ! χ 2 error was 0.018. Additionally, this network had a pVar (Experimental Procedures 7, see also Supplemental Experimental Procedures S7.7) of 92%. Inset shows the speed of convergence for runs of different lengths for the p = 8% PINned network. Run #5 corresponds to the example network highlighted in the gray square. A Deriving target functions for an idealized sequential PINned network Gaussian fit to averaged waveform Sequential target functions for PINning 1 1 1 Rave Gaussian fit, f(t) Normalized firing rate Target functions, sorted Target

0 500 0 –5 0 5 0 10.5 Time, s Time, s

B PINning an idealized sequence C Dimensionality of sequential activity

100 100 95 92 Qeff = 10 Magnitude of synaptic change 80 80 10

pVar, % pVar, 8 60 60 6

40 4 40

2 20 20 0 Variance explained Variance

Normalized mean synaptic change 0 20 40 60 80 100 Fraction of plastic synapses p, % PINned network, p = 8% 0 0

0 8 20 40 60 80 100 Cumulative variance explained by PC, % 0 10 20 30 40 50 Fraction of plastic synapses p, % Principal component

Figure S2 (related to Figures 3 and 6): Idealized Sequence Generating PINned Network A. Left panel shows a Gaussian with mean = 0 and variance = 0.3, denoted by f(t) (green trace), that best fits the neuron- averaged waveform (red trace, Rave, identical to the inset of Figure 1F). This waveform, f(t) is used to generate the target functions (right panel) for a network of 500 rate-based model neurons PINned to produce an idealized sequence of population activity (bVar = 100%). B. Effect of increasing the fraction of plastic synapses, p, in a network constructed by PINning to produce the single idealized sequence is shown here. pVar (Experimental Procedures 7, see also Supplemental Experimental Procedures S7.7) plotted as a function of p, increases from 0 for a random unmodified network (p = 0) network and plateaus at pVar = ~92% for and above p = 8%. The sequence-facilitating properties of the connectivity matrix in the p = 8% network highlighted in red are analyzed in Figures 3 and 4 of the main text. Inset shows the magnitude of synaptic change required to generate an idealized sequence as a function of p, computed as in Figure 2E (Experimental Procedures 8, see also Supplemental Experimental Procedures S7.8). The overall magnitude grows from a factor of ~3 for sparsely PINned networks (p = 8%) to between 8 and 9 for fully PINned networks (p = 100%) producing a single idealized sequence that match the targets in (A). As explained in the main text, although the individual synapses change more in sparsely PINned (small p) networks, the total amount of change across the synaptic connectivity matrix is smaller. C. Dimensionality of sequential activity is computed (as in Figure 2F, Experimental Procedures 5, see also Supplemental Experimental Procedures S7.5) for the 500-neuron PINned network generating an idealized sequence (orange circles) with p = 8% and pVar = 92%. Of the 500 possible PCs that can capture the total variability in the activity of this 500-neuron network, 10 PCs account for over 95% of the variance when p = 8%, i.e., Qeff = 10. A Effect of introducing non-targeted neurons 100 p = 10% Var, % Var,

p Idealized sequence 80 PPC-like sequence

p = 15%

Variance captured by targeted neurons Variance 0 0 10 20 30 40 50 60 70 80 90

Non-targeted : targeted neurons Nnt /N, % B Idealized sequential network outputs, N = 1000

Nnt /N = 5% Nnt /N = 50% Nnt /N = 75% 1 1 pVar = 90% pVar = 95% pVar = 98% Neurons, sorted

1000 0 0 Time, s 10.5 0 Time, s 10.5 0 Time, s 10.5 C PPC-like sequential network, N = 437

Nnt /N = 25% Nnt /N = 50% Nnt /N = 75% 1 1 pVar = 80% pVar = 85% pVar = 88% sorted Neurons,

437 0 0 Time, s 10.5 0 Time, s 10.5 0 Time, s 10.5

Figure S3 (related to Figures 1, 2 and 5): Simulating Unobserved Neurons by Including Non-targeted Neurons in PINned Networks A. Variance of the target functions captured by the outputs of the network neurons that have been PINned, pVar (evaluated as described in Experimental Procedures 7, see also Supplemental Experimental Procedures S7.7) is plotted here as a function of the ratio of non-targeted to targeted neurons in the network, denoted by Nnt/N. Overall, pVar does not decrease appreciably when the relative number of untrained neurons introduced into the PINned network is increased. Interestingly however, pVar improves slightly for sparsely PINned (p = 10% networks in the red circles for an idealized sequence- generating network and p = 15% networks in green for a PPC-like sequence) when untrained neurons are introduced. This improvement gets smaller as p increases (not shown). The inclusion of non-targeted neurons in the networks constructed by PINning simulates the effect of unobserved but active neurons that may exist in the experimental data and might influence neural activity. It should be noted, however, that these additional neurons, however, might add irregularity to the sorted outputs of the full network (including both targeted and non-targeted neurons) and reduce the stereotypy of the overall outputs (indicated by a decrease in the bVar computed over the full network, not shown). This effect is independent of p.

B. Example network outputs are shown here for 3 values of Nnt/N. for a network generating an idealized sequence similar to Supplemental Figure 4. pVar = 90% for Nnt/N = 5%, 95% for Nnt/N = 50% and 98% for Nnt/N = 75%. The sequences become noisier overall as more randomly fluctuating untrained neurons are introduced, but the percent variance of the targets captured by the PINned neurons remains largely unaffected, even showing a slight improvement. C. Same as panel (B), except for a 437-neuron network constructed by PINning to generate a PPC-like sequence similar to Figure 2C with different fractions of neurons left untrained. pVar = 80% for Nnt/N = 25%, 85% for Nnt/N = 50% and 88% for Nnt/N = 75%, confirming the same general trend as above. A Effect of sparsifying random network output 25

20 bVar, %

5 1 Response function r = Stereotypy of random network [1 + e – (x – θ) ] 0 –10 –5 0 5 10 15 20 Threshold of response function θ B Example outputs with low & high thresholds θ = 0, bVar = 12% θ = 5, bVar = 22% 1 1 Neurons, sorted

500 0 0 10.5 0 10.5 Time, s Time, s

Figure S4 (related to Figure 1): Changing the Threshold of the Response Function and Sparsifying the Random Network Output A. Effect of sparsifying the outputs of random networks of model neurons constructed with different thresholds, denoted by !θ , is shown here. Stereotypy of the sequences made by sorting the outputs of these random networks, bVar, increases from 0 to 12% for !θ = 0 networks used as initial configurations throughout the paper, and saturates at 22% for networks with threshold values of !θ > 2. B. Example outputs from two random networks, one with !θ = 0 (left, with bVar = 12%, this is identical to Figure 1B) and one with θ = 5 (right, with bVar = 22%) are shown here. Firing rates are normalized by the tCOM and sorted to yield the sequences shown here. A Sparsely PINned matrix B Distance-dependent synaptic interactions C Hybird matrix, JHybrid D Distance-dependent synaptic interactions

JPINned, 12% JMoving bump + JRand 1 15 2 mean J +/– std. dev. 1 2 2 mean J +/– std. dev. p = 12% PINned, 12% Hybrid

0 0 1 1 Postsynaptic neuron Postsynaptic neuron

437 –25 500 –2.5 Hybrid J 1 Presynaptic neuron 500 PINned, 12% 0 1 Presynaptic neuron 500 0 J Rates from Rates from JPINned, 12% JHybrid 1 1 1 1

–1 Elements of –1 Elements of

–2 –2

437 0 500 0 1 10.5 –437 0 437 1 10.5 –500 0 500 Time, s i – j Time, s i – j

Figure S5 (related to Figure 3): Comparison of the Sparsely PINned Matrix, JPINned, 12% and the Additive Hybrid Connectivity Matrix of the Form, JHybrid = JMoving bump + JRand A. Synaptic connectivity matrix of a 437-neuron network with p = 12% that produces a PPC-like sequence with pVar = 85%, denoted by JPINned, 12%, and the firing rates obtained (identical to Figure 2C, also highlighted by the red circle in Figure 2D) are shown here. B. Influence of neurons away from the sequentially active neurons is estimated by computing the mean (circles) and the standard deviation (lines) of the elements of JRand (in blue) and JPINned, 8% (in orange) in successive off-diagonal “stripes” away from the principal diagonal (as described in Experimental Procedures 11 and analogous to Figure 3 in the main text). These quantities are plotted as a function of the “inter-neuron distance”, i – j. In units of i – j, 0 corresponds to the principal diagonal or self-interactions, and the positive and the negative terms are the successive interaction magnitudes of neurons a distance i – j away from the primary sequential neurons.

C. Same as panel (A), except for the synaptic connectivity matrix from an additive hybrid of the form, JHybrid = JMoving bump + JRand is shown here, where, JRand. is a random matrix similar to the one shown in the lower panel of Figure 3A and JMoving bump is the connectivity for a moving bump model [Yishai, Bar-Or & Sompolinsky, 1995]. The hybrid matrix contains a structured and a random part, and is constructed by the addition of a moving bump connectivity matrix [Yishai, Bar-Or &

Sompolinsky, 1995],! JMoving bump = –J0 + J2[cos(φi –φ j )]+ 0.06 ×[sin(φi –φ j )] , !φ = 0,...,π ) and a random matrix,

JRand, similar to the one used to initialize PINning (lower panel of Figure 3A). Mathematically, the hybrid is of the form, 1 1 J = [A J ]+ [A J ] , where AB is the relative amplitude of the structured or moving bump Hybrid N B Moving bump N R Rand

part, scaled by network size, N, and AR is the relative amplitude of the random part of the N-neuron hybrid network, scaled by ! N . The lower panel shows the firing rates from the additive hybrid network whose connectivity is given by JHybrid. pVar for the output of this hybrid network is only 1%, however, its stereotypy, bVar = 92%.

D. Same as (B), except for JHybrid. Band-averages (orange circles) are bigger and more asymmetric compared to those for JPINned, 12%. Notably, these band-averages are positive for i – j = 0 and in the neighborhood of 0. Fluctuations around the band-averages (red lines) for JHybrid are less structured than those for JPINned, 12% and result from JRand. A Sensitivity of single sequences to stochastic noise B Resilience of sequential memory tasks to noise 1 Idealized sequence network 1 Sequential memory network PPC-match delayed PPC-like sequential network Tolerance paired association network 0.5 0.8 0.8 0.4 , %

0.6 0.6 pVar Sensitivity to structural noise Tolerance = 1 1 0.4 0.4

Selectivity index 0.8 0.2 0.2 Percent variance , %

0.6

0 0 pVar

0 0.5 1 1.5 2 0 0.5 1 1.5 Noise amplitude Noise amplitude 0.4

C Correctly performed task in sequential PINned network D “Error” caused by stochastic noise in inputs (N = 500, p = 9%, Selectivity = 0.91) (N = 500, p = 9%, = 0.6, Selectivity = 0) ηc 0.2 Left trial, Right preferring Right trial, Right preferring Left trial, Right preferring Right trial, Right preferring 1 1 1 1 1 1

Percent variance 0 Trained without noise Trained with stochastic noise –0.2 Neurons, sorted Neurons, sorted Neurons, sorted Neurons, sorted 0 0.2 0.4 0.6 0.8 1 200 200 200 200 Structural noise level Left trial, Left preferring Right trial, Left preferring Left trial, Left preferring Right trial, Left preferring 1 1 1 1 Neurons, sorted Neurons, sorted Neurons, sorted Neurons, sorted

200 200 200 200 0 8 0 8 0 0 8 0 8 0 Time, s Time, s Time, s Time, s

Figure S6 (related to Figures 3 and 6): Left Panel, Robustness of PINned Networks to Stochastic Noise in Inputs and Right Panel, Sensitivity to Structural Noise A. Stochastic noise, !η is injected as an additional current to test whether, and how much, the neural sequences learned through PINning are stable against perturbations (as described in Experimental Procedures 2, see also Supplemental Experimental Procedures S7.2), and the results are shown here. Percent variance of the target functions explained by the network outputs, pVar, drops as amplitude of the injected noise is increased. The maximum tolerance is indicated by the gray line and for single sequences, we define it as the amplitude of noise at which pVar drops below 50% and denote it by

!ηc . For the idealized sequence and for the PPC-like sequence, !ηc = 1. B. Noise tolerance of memory networks that implement working memory through an idealized sequence (orange triangles) and through PPC-like sequences (green triangles, identical to the network shown in Figure 5) is plotted here. Selectivity of both networks drops at different values of ! η , indicating the maximum resilience or noise-tolerance of each, denoted by

!ηc – the sequential memory network has a maximum tolerance of ! ηc = 0.5 while the PPC-match memory network has

a maximum noise tolerance of ! ηc = 0.4 . C. Correctly performed delayed paired association task with delay period memory of the cue identity implemented through two idealized sequences (each similar to Supplemental Figure S2A) in a network of 500 neurons with p = 9% plastic synapses. For the outputs shown here, the turn period is omitted for clarity, and the delay period starts at the 5s time-point in the 10.5s-long task. The performance of this network, quantified by the selectivity index (Experimental Procedures 9) is 0.91. D. The same network from panel (D) fails to perform the task (selectivity = 0) because the high levels of stochastic noise present in the inputs (! η = 0.6 , here) quenches delay period memory. E. Right Panel shows sensitivity of PINned networks to structural noise in the synaptic connectivity matrix. Structural noise N (0,1) (described by ! level × , where N = 500 here) is added to the connections in the sparsely PINned connectivity N

matrix, JPINned, 8% to test whether, and by how much, the synaptic connections are finely tuned. pVar, computed as in Experimental Procedures 9, is plotted as a function of structural noise amplitude, for sequential networks obtained by PINning in the presence of stochastic noise in the inputs (red squares) and for networks trained without any noise (blue squares). Gray line is at pVar = 50%, the noise-free-PINned network has a tolerance (noise amplitude at which pVar drops below 50%) of 0.6, and the network trained with noise, 0.8. Training in the presence of stochastic noise therefore leads to slightly more robust networks, although the drop in pVar with the addition of structural noise does not fully recover with training noise. SUPPLEMENTAL EXPERIMENTAL PROCEDURES S7: Detailed Experimental Procedures 7.1. Network elements (see also Experimental Procedures 1 in main text) We consider a network of N fully interconnected neurons described by a standard firing rate model. Each model neuron is characterized by an activation variable, ! xi for i = 1, 2, ... N, where, N = 437 for the PPC-like sequence in Figures 1D and 2A, N = 500 for the single idealized sequence in Supplemental Figure S2 and the multi-sequential memory task in Figure 6, and N = 569 for the 2AFC task in Figure 5 (we generally build networks of the same size as the experimental dataset we are trying to model, however the results obtained remain applicable to larger networks, see for example, Supplemental Figure 1 S3), and a nonlinear response function, !φ(x) = . This function ensures that the firing rates, ! r = φ(x ) , go 1+ e[−(x−θ )] i i from a minimum of 0 to a maximum at 1. Adjusting !θ allows us to set the firing rate at rest, x = 0 , to some convenient and biologically realistic background firing rate, while retaining a maximum gradient at ! x = θ . We use !θ = 0 (but see also Supplemental Figure S4).

We introduce a recurrent synaptic weight matrix J with element ! Jij representing the strength of the connection from presynaptic neuron j to postsynaptic neuron i (schematic in Figure 1A) The individual synaptic weights are initially chosen independently and randomly from a Gaussian distribution with mean and variance given by ! 〈 Jij 〉J = 0 and 2 2 ! 〈Jij 〉J = g / N , and are either held fixed or modifiable, depending on the fraction of plastic synapses p that can change by applying a learning algorithm (Experimental Procedures 4).

The activation variable for each network neuron ! xi is determined by, dx N ! i . τ = –xi + ∑ Jijφ(x j ) + hi dt j In the above equation, !τ = 10ms is the time constant of each unit in the network and the control parameter g determines whether (g > 1) or not (g < 1) the network produces spontaneous activity with non-trivial dynamics [Sompolinsky, Crisanti & Sommers, 1988; Rajan, Abbott & Sompolinsky, 2010; Rajan, Abbott & Sompolinsky, 2011]. We use g values between 1.2 and 1.5 for the networks in this paper, so that the randomly initialized network generates chaotic spontaneous activity prior to activity-dependent modification (gray trace in the schematic in Figure 2B), but produces a variety of regular non-chaotic outputs that match the imposed target functions afterward (red trace in the schematic in Figure 2B, see also results in Figures 2, 5 and 6, see also Experimental Procedures 4 later). The network equations are integrated using Euler method with an integration time step, dt = 1ms. ! hi is the external input to the unit i.

7.2. Design of External Inputs (see also Experimental Procedures 2 in main text) During the course of the real two-alternative forced-choice (2AFC) experiment [Harvey, Coen & Tank, 2012], as the mouse runs through the virtual environment, the different patterns projected onto the walls of the maze (colored dots, stripes, pillars, hatches, etc.) translate into time-dependent visual inputs arriving at the PPC. Therefore, to represent sensory (visual and proprioceptive) stimuli innervating the PPC neurons, the external inputs to the neurons in the network, denoted by ! h(t) , are made from filtered and spatially delocalized white noise that is frozen (repeated from trial to trial), using the equation, dh !τ = −h(t) + h η(t), where !η is a random variable drawn from a Gaussian distribution with 0 mean and unit WN dt 0 variance, and the parameters ! h0 and !τ WN control the scale of these inputs and their correlation time, respectively. We use

! h0 = 1and !τ WN = 1s. There are as many different inputs as there are model neurons in the network, with individual model neurons receiving the same input on every simulated trial. A few example inputs are shown in the right panel of Figure 1A. In addition to the frozen noise h, which acts as external inputs to these networks, described above, we also test the resilience of the memory networks we built (Figure 6E) to injected stochastic noise. This stochastic injected noise varies randomly (i.e., is a Gaussian random variable between 0 and 1, drawn from a zero mean and unit variance distribution) and 2 2 independently at every time step. The diffusion constant of the white noise is given by ! Aη / 2τ , where the amplitude is ! Aη and !τ is the time constant of the network units (we use 10ms, as detailed in Experimental Procedures 1). We define

“Resilience” or “Noise Tolerance” as the critical amplitude of this stochastic noise, denoted by !ηc , at which the delay period memory fails and the Selectivity Index of the memory network drops to 0 (Figure 6E, see also Supplemental Figure S6).

7.3. Extracting Target Functions From Calcium Imaging Data (see also Experimental Procedures 3 in main text) To derive the target functions for our activity-dependent synaptic modification scheme termed Partial In-Network Training or PINning, we convert the calcium fluorescence traces from PPC recordings [Harvey, Coen & Tank, 2012] into firing rates using two complementary methods. We find that for this dataset, the firing rates estimated by the two methods agree quite well (Supplemental Figure S1). The first method is based on the assumption that the calcium impulse response function, which is a difference of exponentials (! K ∝ e−t/384 − e−t/52 , with a rise time of 52ms and a decay time of 384ms [Tian et al, 2009; Harvey, Coen & Tank, 2012]), is approximated by an alpha function of the form ! K ∝ te−t/τ Ca , where there is only a single (approximate) time constant for the filter, !τ Ca = 200ms. According to this assumption, the scaled firing rate s and calcium concentration, [Ca2+] are related by, dCa dx !τ i = −Ca (t) + x (t) and !τ i = −x (t) + s (t), Ca dt i i dt i i where, x(t) is an auxiliary variable.The inverse of the above model is obtained by taking a derivative of the calcium data, writing, dCa dx ! x (t) = Ca (t) +τ i and ! s (t) = x (t) +τ i . i i Ca dt i i dt

Once we have s(t), we rectify it and choose a smoothing time constant !τ R for the firing rate we need to compute. Finally, dR integrating the equation, !τ i = −R (t) + s (t) , and normalizing by the maximum gives us an estimate for the firing R dt i i rates extracted from the calcium data, denoted by R (Supplemental Figure S1). The second method is a fast Bayesian deconvolution algorithm [Pnevmatikakis et al, 2016, available online at https:// github.com/epnev/continuous_time_ca_sampler] that infers spike trains from calcium fluorescence data. The inputs to this algorithm are the rise time (52ms) and the decay time (384ms) of the calcium impulse response function [Tian et al, 2009; Harvey, Coen & Tank, 2012] and a noise parameter [Pnevmatikakis et al, 2016]. Typically, if the frame rate for acquiring the calcium images is low enough (the data in [Harvey, Coen & Tank, 2012] are imaged at 64ms per frame), the outputs from this algorithm can be interpreted as a normalized firing rate. To verify the accuracy of the firing rate outputs obtained from trial- averaged calcium data (for example, from Figure 2c in [Harvey, Coen & Tank, 2012]), we smoothed the spike trains we got

−(t−ti ) 2 from the above method for each trial separately through a Gaussian of the form ! 2τ R , normalized by! , and ∑ e 2π ×τ R i then averaged over single trials to get trial-averaged firing rates (this smoothing and renormalization procedure has also been recommended for faster imaging times [Pnevmatikakis et al, 2016]).

Once the values of !τ R and ! τ Ca are determined that make the results obtained by both deconvolution methods consistent

(we used !τ R = 100ms and !τ Ca = 384ms), we used the firing rates extracted as target functions for PINning through the ⎡ R (t) ⎤ transform,! i . The above expression is obtained by solving the activation function relating input current fi (t) = ln ⎢ ⎥ ⎣1− Ri (t)⎦ 1 to firing rate of model neurons, ! Ri (t) = , since the goal of PINning is to match the input to neuron i, say, denoted 1+ e− fi (t ) by ! zi (t) to its target function, denoted by! fi (t). Finally, to verify our estimates, we re-convolved (Supplemental Figure S1) the output firing rates from the network neurons with a difference of exponentials using a rise time of 52ms and a decay time of 384ms.

7.4. Synaptic Modification Rule For PINning (see also Experimental Procedures 4 in main text) During PINning, the inputs of individual network neurons are compared directly with the target functions to compute a set of error functions, i.e., ! ei (t) = zi (t) − fi (t), for i = 1, 2, ... N. Individual neuron inputs are expressed as , where is the firing rate of the jth or the presynaptic neuron. zi (t) = ∑ Jijrj (t) rj (t) j During learning, the subset of plastic internal weights in the connectivity matrix J of the random recurrent network, denoted by the fraction p, undergo modification at a rate proportional to the error term, the presynaptic firing rate of each neuron, ! rj and a pN x pN matrix, P (with elements ! Pij ) that keeps track of the rate fluctuations across the network at every time step. Here, p is the fraction of neurons whose outgoing synaptic weights are plastic; since this is a fully connected −1 network, this is also the fraction of plastic synapses in the network. Mathematically, ! Pij = < rirj > , the inverse cross- correlation matrix of the firing rates of the network neurons (Pij is computed for all i but is restricted to j = 1, 2, 3,…,pN). The basic algorithm is schematized in Figure 2B. At time t, for i = 1, 2, ... N neurons, the learning rule is simply that the elements of the matrix J are moved from their values at a time step ! Δt earlier through ! Jij (t) = Jij (t −1) + ΔJij (t). Here, the synaptic update term, according to the RLS/FORCE procedure [Haykins, 2002; Sussillo & Abbott, 2009] (since other methods for training recurrent networks, such as backpropagation would be too laborious for our purposes) follows, ! where the above update term is restricted to the p% of plastic synapses in the ΔJij (t) = c[zi (t) − fi (t)]∑ Pjk (t)rk (t), k network, which are indexed by j and k in the above expression. While c can be thought of as an effective learning rate, it is 1 given by the formula, ! c = . The only free parameter in the learning rule is P(0) (but the value to which it 1+ r′(t)P(t)r(t) is set is not critical [Sussillo & Abbott, 2009]). When there are multiple sequences (such as in Figures 5 and 6), we choose pN synapses that are plastic and we use those same synapses for all the sequences. The matrix P is generally not explicitly calculated but rather updated according to the rule, P(t −1)r(t)r′(t)P(t −1) ! P(t) = P(t −1) − in matrix notation, which includes a regularizer [Haykins, 2002]. In our 1+ r′(t)P(t −1)r(t) scheme, all indices in the above expression are restricted to the neurons with plastic synapses in the network. The algorithm requires the matrix P to be initialized to the identity matrix times a factor that controls the overall learning rate, i.e., P(0) = α × I , and in practice, values from 1 to 10 times the overall amplitude of the external inputs (denoted by h0 in Experimental Procedures 2) driving the network are effective (other values are explored in [Sussillo & Abbott, 2009]). For numerically simulating the PINned networks whose sequential outputs are shown in Figures 2C, 5 and 6, the integration time step used is dt = 1ms (as described in Experimental Procedures 1 and 2, we use Euler method for integration). The learning occurs at every time step for the p% of pre-synaptic neurons with plastic outgoing synaptic weights. Starting from a random initial state (Experimental Procedures 1), we first run the program for 500 learning steps, which include both the network dynamics and the PINning algorithm, and then an additional 50 steps with only the network dynamics after the learning has been terminated (convergence metrics below, see also Supplemental Figure S1D). A “step” is defined as one run of the program for the duration of the relevant trial, denoted by T. Each step is equivalent to T = 10500 time points (10.5s) in Figures 2C and 5E, and 8000 time points (8s) in Figure 6B. On a standard laptop computer, the first 500 such steps for a 500-unit rate-based network producing a 10s-long sequence (such as in Supplemental Figure S2A) take approximately 8 minutes to complete in realtime and the following 50, about 30 seconds in realtime (about 1s/step for the 10s-long single sequence example, scaling linearly with network size N, total duration or length of the trial T, and number of sequences produced.) The convergence of the PINning algorithm was assayed as follows: (a) By directly comparing the outputs with the data (as in the case of Figures 2A–C and 5D–E) or the set of target functions used (Supplemental Figure S2A) and (b) By calculating and following the ! χ 2 -squared error between the network rates and the targets, both during PINning (inset of Supplemental Figure S1D) and at the end of the simulation (Supplemental Figure S1D). The performance of the PINning algorithm was assayed by computing the percent variance of the data or the targets captured by the sequential outputs of different networks (pVar, see Experimental Procedures 7, see also Figure 1H) or by computing the Selectivity Index of the memory network (Selectivity, see Experimental Procedures 9, see also Figure 5F).

7.5. Dimensionality Of Network Activity (Qeff) (see also Experimental Procedures 5 in main text) We use state space analysis based on PCA (see for example, [Rajan, Abbott & Sompolinsky, 2011; Sussillo, 2014] and references therein) to describe the instantaneous network state by diagonalizing the equal-time cross-correlation matrix of network firing rates given by, !Qij = 〈(ri (t)− < ri >)(rj (t)− < rj >)〉 , where <> denotes a time average. The eigenvalues of this matrix expressed as a fraction of their sum indicate the distribution of variances across different orthogonal directions in the activity trajectory. We define the effective dimensionality of the activity, Qeff, as the number of principal components that capture 95% of the variance in the dynamics (Figure 2F).

7.6. Stereotypy Of Sequence (bVar), % (see also Experimental Procedures 6 in main text) bVar quantifies the variance of the data or the network output that is explained by the translation along the sequence of an activity profile with an invariant shape. For example, for Figure 1C–G, we extracted an aggregate waveform, denoted by

Rave (red trace in Figures 1D and the left panel of Supplemental Figure S2A), by averaging the tCOM-realigned firing rates extracted from trial-averaged PPC data collected during a 2AFC task [Harvey, Coen & Tank, 2012]. Undoing the tCOM shift, we can write this function for the aggregate or “typical” bump-like waveform as ! Rave (t − ti ). The amount of variability in the data that is explained by the moving bump, ! Rave (t − ti ) is given by a measure we call bVar. ⎡ 〈R (t) − R (t − t )〉2 ⎤ N T ! bVar 1 i ave i , where ! . In the above expressions, R’s denote the firing rates extracted = ⎢ − 2 ⎥ <> = ∑∑ ⎣ 〈Ri (t) − R(t)〉 ⎦ i t

th from calcium data (Experimental Procedures 3); Ri (t) is the firing rate of the i PPC neuron at time t and R(t) is the average over neurons. The total duration,T is 10.5s in Figures 1, 2 and 5, and 8s in Figure 6. In some ways, bVar is similar to the ridge-to-background ratio computed during the analysis of experimental data for measuring the level of background activity (see for example, Supplemental Figure 14 in [Harvey, Coen & Tank, 2012]); however, bVar additionally quantifies the stereotypy of the shape of the transient produced by individual neurons.

7.7. Percent Variance Of Data Explained By Model (pVar), % (see also Experimental Procedures 7 in main text) We quantify the match between the experimental data or the set of target functions, and the outputs of the model by the ⎡ 〈D (t) − r (t)〉2 ⎤ amount of variance of the data that is captured by the model,! i i which is one minus the ratio pVar = ⎢1− 2 ⎥, ⎣ 〈Di (t) − D(t)〉 ⎦ of the Frobenius norm of the difference between the data and the outputs of the network, and the variance of the data. The data referred to here, denoted by D, is trial-averaged data, such as from Figures 1D, 2A and 5D.

7.8. Magnitude Of Synaptic Change (see also Experimental Procedures 8 in main text) In Figure 2E and in Figure 5G, we compute the magnitude of the synaptic change required to implement a single PPC- like sequence, an idealized sequence and three memory tasks, respectively. In combination with the fraction of plastic synapses in the PINned network, p, this metric characterizes the amount of structure that needs to be imposed in an initially random network to produce the desired temporally structured dynamics. This is calculated as, normalized mean synaptic PINned, p% Rand ∑|Jij − Jij | ij change = Rand , where, following the same general notation as in the main text, JPINned, p% denotes the ∑|Jij | ij connectivity matrix of the PINned network constructed with p% plastic synapses and JRand denotes the initial random connectivity matrix (p = 0).

7.9. Selectivity Index For Memory Task (see also Experimental Procedures 9 in main text) In Figures 5F, a Selectivity Index is computed (similar to Figure 4 in [Harvey, Coen & Tank, 2012]) to assess the performance of different PINned networks at maintaining cue-specific memories during the delay period of delayed paired association tasks. This metric is based on the ratio of the difference and the sum of the mean activities of preferred neurons at the end of the delay period during preferred trials, and the mean activities of preferred neurons during opposite trials. We 1 ⎡< r >r.n − < r >r.n < r >l.n − < r >l.n ⎤ compute Selectivity Index as, ! r.t l.t l.t r.t where the notation is as follows: ⎢ r.n r.n + l.n l.n ⎥, 2 ⎣< r >r.t + < r >l.t < r >l.t + < r >l.t ⎦ 1 right pref neurons 1 left pref neurons ! r.n and ! l.n are the average firing rates of right- < r >r.t = ∑ ri < r >r.t = ∑ ri Nright pref neurons i Nleft pref neurons i preferring and left-preferring model neurons on right trials; 1 right pref neurons 1 left pref neurons ! r.n and ! l.n are the average firing rates of right- < r >l.t = ∑ ri < r >l.t = ∑ ri Nright pref neurons i Nleft pref neurons i preferring and left-preferring model neurons on left trials of the simulated task. The end of the delay period is at approximately 10s for the network in Figure 5 (after [Harvey, Coen & Tank, 2012]) and at ~7s time point for the network in Figure 6.

7.10. Temporal Sparseness Of Sequences (NActive/N) (see also Experimental Procedures 10 in main text) The temporal sparseness of a sequence is defined as the fraction of neurons active at any instant during the sequence, the fraction, NActive/N. To compute this, first, the normalized firing rate from each model neuron in the network or from data [Harvey, Coen & Tank, 2012], denoted by Ri(t), is realigned by the center-of-mass, iCOM(t), given by,

! iCOM (t) = ∑i i * R(i,t) / ∑i R(i,t), where i is the neuron number and t is time. The realigned rates are then averaged over time to obtain time, i.e., 〈R(i)〉time = 〈R(i–iCOM (t), t)〉time after using a circular shift rule to undo the iCOM-shift. The standard deviation of the best Gaussian that fits this curve time is the number, NActive, and the ratio of NActive to the network size, N, yields the temporal sparseness of the sequence, NActive/N. For the data in Figure 2A, for example, NActive/N = 3% (i.e., 16 neurons out of a total of 437). Practically speaking, decreasing this fraction makes the sequence narrower and at a critical value of sparseness (NActive/N = 1.6% in Figure 6), there is not enough current in the network to propagate the sequence. However, up to a point, making the sequences sparser increases the capacity of a network for carrying multiple non-interfering sequences (i.e., sequences with delay period memory of cue identity), without demanding an increase in either the fraction of plastic synapses, p, or in the overall magnitude of synaptic change.

7.11. Analyzing the Structure of PINned Synaptic Connectivity Matrices (see also Experimental Procedures 11 in main text) This is pertinent to Section 3 (Figures 3 and 4), in which we quantify how the synaptic strength varies with the “distance” between pairs of network neurons in connectivity space, i – j, in PINned sequential networks. We first compute the N J N J means and the standard deviations of the principal diagonals, i.e., ! ∑ ij → ∑ ii , in the 3 connectivity matrices under i= j N i=1 N consideration here – JPINned, 8%, JRand, and just for comparison purposes, JPINned, 100%. Then, we compute the means and the standard deviations of successive off-diagonal “stripes” moving away from the principal diagonal, i.e., N Jij Ji(i−a) ∑ → ∑ , for the same three matrices. These are plotted in Figures 3B for the sparsely PINned matrix, JPINned, i− j=a N i=a+1 N

8%, relative to the randomly initialized matrix, JRand, and in Figure 3E, for the fully PINned matrix constructed for comparison purposes, JPINned, 100%. The same analysis is also used to compare different partially structured matrices in Supplemental Figure S5.

7.12: Cross-Validation Analysis (see also Experimental Procedures 12 in main text) We first divided the data from 436 neurons in Figure 1D into two separate “synthetic” sequences by assigning the even- numbered cells to one (let’s call this Sequence A, containing 218 neurons) and the odd-numbered cells to another sequence (say, Sequence B, also with 218 neurons). Then we constructed a PINning-based network with 218 model neurons, exactly like we described in the main text, using data from Sequence A as target functions for PINning. Next, we computed the percent variance of the data in Sequence A and the data in Sequence B, denoted by pVarTest A, Train A and pVarTest B, Train A, respectively, that are captured by the outputs of the PINned network (Experimental Procedures 7). Following a similar procedure, we also computed pVarTest A, Seq B and pVarTest B, Seq B, after PINning a second network against target functions derived from Sequence B.

We obtained the following estimates for all fractions of plastic synapses, p :

pVarTest A, Train A = 91 + 2% pVarTest B, Train A = 45 + 2%

pVarTest A, Train B = 46 + 2% pVarTest B, Train B = 90 + 2%

For comparison purposes, a random network such as the one in Figure 1B only captures a tiny amount of the variability of data (in this notation, pVarTest Data, Random Network = 0.2%, see also, right panel of Figure 1H). Additionally, the data from one set only accounts for 49% of the variance of the data of the other set, i.e., pVar = 49%. Thus the model does almost as well as it possibly could.

REFERENCE

Tian L, Hires SA, Mao T, Huber D, Chiappe ME, Chalasani SH, Petreanu L, Akerboom J, McKinney SA, Schreiter ER, Bargmann CI, Jayaraman V, Svoboda K, Looger LL (2009) Imaging neural activity in worms, flies and mice with improved GCaMP calcium indicators. Nat Methods. 6(12): 875-881.