Extending the Neural Engineering Framework for Nonideal Silicon Synapses

Extending the Neural Engineering Framework for Nonideal Silicon Synapses Aaron R. Voelker∗, Ben V. Benjaminy, Terrence C. Stewart∗, Kwabena Boaheny and Chris Eliasmith∗ farvoelke, tcstewar, [email protected] fbenvb, [email protected] ∗Centre for Theoretical Neuroscience, University of Waterloo, Waterloo, ON, Canada. yBioengineering and Electrical Engineering, Stanford University, Stanford, CA, U.S.A. Abstract—The Neural Engineering Framework (NEF) is a for parasitic capacitances. We also account for the variability theory for mapping computations onto biologically plausible (i.e., heterogeneity) introduced by transistor mismatch in the networks of spiking neurons. This theory has been applied to a extended pulse’s width and height, and in the LPF’s two time- number of neuromorphic chips. However, within both silicon and constants. In II, we demonstrate how to extend the NEF to real biological systems, synapses exhibit higher-order dynamics directly harnessx these features for system-level computation. and heterogeneity. To date, the NEF has not explicitly addressed This extension is tested by software simulation in IV using how to account for either feature. Here, we analytically extend the circuit models described in III. x the NEF to directly harness the dynamics provided by heteroge- x neous mixed-analog-digital synapses. This theory is successfully validated by simulating two fundamental dynamical systems in II. EXTENDING THE NEURAL ENGINEERING Nengo using circuit models validated in SPICE. Thus, our work FRAMEWORK reveals the potential to engineer robust neuromorphic systems with well-defined high-level behaviour that harness the low- The NEF consists of three principles for describing neural level heterogeneous properties of their physical primitives with computation: representation, transformation, and dynamics [2]. millisecond resolution. This framework enables the mapping of dynamical systems onto recurrently connected networks of spiking neurons. We begin by providing a self-contained overview of these three I. THE NEURAL ENGINEERING FRAMEWORK principles using an ideal first-order LPF. We then extend these The field of neuromorphic engineering is concerned with principles to the heterogeneous pulse-extended second-order building specialized hardware to emulate the functioning of LPF, and show how this maps onto a target neuromorphic the nervous system [1]. The Neural Engineering Frame- architecture. work (NEF; [2]) compliments this goal with a theory for “compiling” dynamical systems onto spiking neural networks, A. Principle 1 – Representation and has been used to develop the largest functioning model The first NEF principle states that a vector k may of the human brain, capable of performing various perceptual, x(t) R be encoded into the spike-trains of neurons with2 rates: cognitive, and motor tasks [3]. This theory allows one to map δi n an algorithm, expressed in software [4], onto some neural ri(x) = Gi [αiei x(t) + βi] , i = 1 : : : n, (1) substrate realized in silicon [5]. The NEF has been applied to · neuromorphic chips including Neurogrid [5], [6] and a VLSI where Gi is a neuron model whose input current to the soma prototype from ETH Zurich [7]. is the linear encoding αiei x(t) + βi with gain αi > 0, unit- · n k length encoding vector ei (row-vectors of E R × ), and 2 However, the NEF assumes that the postsynaptic current bias current βi. The state x(t) is typically decoded from spike- (PSC) induced by a presynaptic spike is modelled by a trains (see Principle 2) by convolving them with a first-order first-order lowpass filter (LPF). That is, by convolving an LPF that models the PSC triggered by spikes arriving at the impulse representing the incoming spike with an exponentially synaptic cleft. We denote this filter as h(t) in the time-domain decaying impulse-response. Furthermore, the exponential time- and as H(s) in the Laplace domain: constant is assumed to be the same for all synapses within 1 t 1 the same population. In silicon, synapses are neither first- h(t) = e− τ H(s) = . (2) order nor homogeneous and spikes are not represented by τ () τs + 1 impulses.1 Synapse circuits have parasitic elements that re- Traditionally the same time-constant is used for all synapses sult in higher-order dynamics, transistor mismatch introduces projecting to a given population.2 variability from circuit to circuit, and spikes are represented by pulses with finite width and height. Previously, these features B. Principle 2 – Transformation restricted the overall accuracy of the NEF within neuromorphic hardware (e.g., in [5], [6], [7]). The second principle is concerned with decoding some desired vector function f : S Rk of the represented The silicon synapses that we study here are mixed-analog- vector. Here, S is the domain of! the vector x(t) represented digital designs that implement a pulse-extender [9] and a first- via Principle 1—typically the unit k-cube or the unit k-ball. th order LPF [10], modelled as a second-order LPF to account Let ri(x) denote the expected firing-rate of the i neuron in 1These statements also hold for real biological systems [8]. 2See [11] for a recent exception. w x δ y 1 g(x) τs+1 G [ ] D w1 x · Γ1 H1 u τ + . w τf(x)+x Φ j x δ y D Γj Hj G [ ] Dg(x) · . Fig. 1. Standard Principle 3 (see (6)) mapped onto an ideal architecture to wm u Φ x implement a general nonlinear dynamical system (see (5)). The state-vector D Γm Hm x is encoded in a population of neurons via Principle 1. The required signal w is approximated by τu plus the recurrent decoders for τf(x) + x applied to δ, such that the first-order LPF correctly outputs x. The output vector y is g(x) approximated using the decoders D . Fig. 2. Using extended Principle 3 (see (11)) to implement a general nonlinear dynamical system (see (5)) on a neuromorphic architecture. The th matrix representation Φ is linearly transformed by Γj to drive the j synapse. response to a constant input x encoded via (1). To account Dashed lines surround the silicon-neuron array that filters and encodes x into for noise from spiking and extrinsic sources of variability, spike-trains (see Fig. 3). we introduce the noise term η (0; σ2). Then the matrix f(x) n k ∼ N D R × that optimally decodes f(x) from the spike- j s 1 2 III, is an extended pulse γj (1 e− ) s− convolved with trains δ encoding x is obtained by solving the following x − 1 problem (via regularized least-squares): a second-order LPF ((τj;1s + 1) (τj;2s + 1))− . These higher- order effects result in incorrect dynamics when the NEF is Z n 2 applied using the standard Principle 3 (e.g., in [5], [6], [7]), f(x) X k D = arg min f(x) (ri(x) + η) di d x (3) as shown in IV. n×k − D R S i=1 x 2 From (7), observe that we must drive the jth synapse with n some signal wj(t) that satisfies the following (W(s) denotes X f(x) = (δi h)(t)d (f(x) h)(t). (4) the Laplace transform of w(t) and we omit j for clarity): ) ∗ i ≈ ∗ i=1 X(s) = H(s) The quantity in (4) may then be encoded via Principle 1 to W(s) complete the connection between two populations of neurons.3 s 1 W(s) 1 e− s− () 1 − 2 C. Principle 3 – Dynamics = γ− (1 + (τ1 + τ2)s + τ1τ2s )X(s). (9) The third principle addresses the problem of implementing s the following nonlinear dynamical system: To solve for w in practice, we first substitute 1 e− = 2 2 3 3 − k s ( s )=2 + O( s ), and then convert back to the time- x_ = f(x) + u; u(t) R − 2 domain to obtain the following approximation: y = g(x). (5) 1 w = (γ)− (x + (τ1 + τ2)x_ + τ1τ2x¨) + w_ . (10) Since we take the synapse (2) to be the dominant source 2 of dynamics for the represented vector [2, p. 327], we must Next, we differentiate both sides of (10): essentially “convert” (5) into an equivalent system where the 1 ... integrator is replaced by a first-order LPF. This transformation w_ = (γ)− (x_ + (τ1 + τ2)x¨ + τ1τ2 x) + w¨ , is accomplished by driving the filter h(t) with: 2 and substitute this w_ back into (10) to obtain: w := τx_ + x = (τf(x) + x) + (τu) (6) 1 = (w h)(t) = x(t), (7) w = (γ)− (x + (τ1 + τ2 + /2)x_ + ) ∗ (τ1τ2 + (/2)(τ1 + τ2))x¨) + so that convolution with h(t) achieves the desired integration. 1 ... 2 Therefore, the problem reduces to representing x(t) in a (γ)− (/2)τ1τ2 x + ( =4)w¨ . population of neurons using Principle 1, while recurrently 1 ... Finally, we make the approximation (γ)− (/2)τ1τ2 x + decoding w(t) using the methods of Principle 2 (Fig. 1). (2=4)w¨ w, which yields the following solution to (9): D. Extensions to Silicon Synapses wj = ΦΓj; (11) Consider an array of m heterogeneous pulse-extended Φ := [x x_ x¨] ; second-order LPFs (in the Laplace domain): " 1 # 1 Γj := (jγj)− τj;1 + τj;2 + j=2 , j s 1 γj (1 e− ) s− τ τ + ( =2)(τ + τ ) Hj(s) = − , j = 1 : : : m, (8) j;1 j;2 j j;1 j;2 (τ s + 1) (τ s + 1) j;1 j;2 4 where jγj is the area of the extended pulse. We compute where j is the width of the extended pulse, γj is the the time-varying matrix representation Φ in the recurrent con- height of the extended pulse, and τj;1, τj;2 are the two time- nection (plus an input transformation) via Principle 2 (similar constants of the LPF.

Load more