Memristor-Based Circuit Design for Multilayer Neural Networks Yang Zhang, Xiaoping Wang, Member, IEEE, and Eby G

Home , Computer vision, Gradient descent, Layer (deep learning)

IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS–I: REGULAR PAPERS, VOL. 65, NO. 2, FEBRUARY 2018 677 Memristor-Based Circuit Design for Multilayer Neural Networks Yang Zhang, Xiaoping Wang, Member, IEEE, and Eby G. Friedman, Fellow, IEEE

Abstract— Memristors are promising components for applica- realizing synaptic weights in artificial neural networks [7], [8] tions in nonvolatile memory, logic circuits, and neuromorphic as the innate property of a reconfigurable resistance with computing. In this paper, a novel circuit for memristor-based memory makes memristor highly suitable for synapse weight multilayer neural networks is presented, which can use a single memristor array to realize both the plus and minus weight refinement. of the neural synapses. In addition, memristor-based switches Neuron circuits were originally developed in are utilized during the learning process to update the weight CMOS [9], [10]. Later, hybrid CMOS-memristor synaptic of the memristor-based synapses. Moreover, an adaptive back circuits were developed [11]–[13]. The area and power propagation algorithm suitable for the proposed memristor-based consumption of transistors are however much greater than multilayer neural network is applied to train the neural networks and perform the XOR function and character recognition. memristors. A memristor bridge synapse-based neural network Another highlight of this paper is that the robustness of the and learning are proposed in [14]–[16], which implement proposed memristor-based multilayer neural network exhibits multilayer neural networks (MNN) trained by a back higher recognition rates and fewer cycles as compared with other propagation (BP) algorithm, and the synapse weight updates multilayer neural networks. are performed by a host computer. The major computational Index Terms— Memristor, synaptic weight, crossbar array, bottleneck is however the learning process itself which could multilayer neural networks, XOR function, character recognition. not be completely implemented in hardware with massive memristor-based crossbar arrays. I. INTRODUCTION Many previous memristor-based learning rules have focused RTIFICIAL neural networks have been exploited to on Spike-Timing Dependent Plasticity (STDP) [17]. For exam- Asolve many problems in the area of pattern recognition, ple, neuromorphic character recognition system with two exhibiting the potential to provide high speed computation. PCMO memristors (2M) as a synapse was presented in [18], One possible device to achieve high speed computation is and a learning rule proposed in [19] for visual pattern recog- memristors, the discovery of which greatly broadened the area nition with a CMOS neuron. The filamentary switching binary of hybrid CMOS architectures to nonconventional logic [1] 2M synapse was used for speech recognition [20]. The con- such as threshold logic [2] and neuromorphic computing [3]. vergence of STDP-based learning is however not guaranteed Memristors were theoretically postulated by Chua in 1971 [4] for general inputs [21]. and later Williams’s team presented a resistance variable New methods have since been proposed for memristor- device as a memristor at HP Labs in 2008 [5]. As a novel based neuromorphic architectures. For example, brain-state- nanoscale device, memristors provide several advantageous in-a-box (BSB) neural networks are presented in [22], which features such as non-volatility, high density, low power, and also use 2M crossbar arrays to represent, respectively, plus- good scalability [6]. Memristors are particularly appealing for polarity and minus-polarity connection matrices. Memristor- Manuscript received February 5, 2017; revised April 25, 2017 and based multilayer neural networks with online gradient descent June 28, 2017; accepted July 15, 2017. Date of publication August 11, 2017; training are proposed in [23] and [24], which use a single date of current version January 25, 2018. This work was supported in part memristor and two CMOS transistors (2T1M) per synapse. by the National Natural Science Foundation of China under Grant 61374150, Grant 61272050, and Grant 61672357, in part by the Science Foundation A training method of a 2M hardware neuromorphic network is of Guangdong Province under Grant 2014A030313556, and in part by the proposed in [25]. To reduce the circuit size, fewer memristors China Scholarship Council. This paper was recommended by Associate Editor and transistors are desired. A memristor-based crossbar array T. Serrano-Gotarredona. (Corresponding author: Xiaoping Wang.) Y. Zhang is with the School of Automation, Huazhong University of Science architecture is therefore presented in [26], where both plus- and Technology, Wuhan 430074, China, with the Shenzhen Key Laboratory polarity and minus-polarity connection matrices are realized of Spatial Smart Sensing and Services, School of Computer Science and Soft- by a single crossbar array and a simple constant-term circuit, ware Engineering, Computer Vision Institute, Shenzhen University, Shenzhen 518060, China, and also with the Department of Electrical and Computer thereby reducing the physical size and power dissipation. The Engineering, University of Rochester, Rochester, NY 14627 USA. memristor-based neural network in [26] is however limited to a X. Wang is with the School of Automation, Huazhong University of single layer neural network (SNN). On-chip learning methods Science and Technology, Wuhan 430074, China (e-mail: wangxiaoping@ mail.hust.edu.cn). remain a challenge in most memristor-based neural networks. E. G. Friedman is with the Department of Electrical and Computer Engi- Neuromorphic processors with memristor-based synapses neering, University of Rochester, Rochester, NY 14627 USA. are investigated in [27]–[29] to achieve the digital pattern Color versions of one or more of the figures in this paper are available online at http://ieeexplore.ieee.org. recognition. Training algorithms of 2M crossbar neuromorphic Digital Object Identifier 10.1109/TCSI.2017.2729787 processors is proposed in [30] and [31], which could be 1549-8328 © 2017 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information. 678 IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS–I: REGULAR PAPERS, VOL. 65, NO. 2, FEBRUARY 2018 used in MNN; however, two memristors per synapse are required. An on-chip supervised learning rule for an ultra-high density neural crossbar using a memristor for the synapse and neuron is described in [32] to perform XOR and AND logic operations. Realizing the BP algorithm on a 1M crossbar array remains an issue. The primary contributions of this paper are: 1) A memristor-based AND (MRL) gate [33] is utilized as a memristor-based switch (MS) [2] in updating the synaptic crossbar circuits. A memristive model for synaptic circuits based on experimental data is utilized in the simulations. Formulae for determining the relevant Fig. 1. Memristor-based logic switch. (a) A memristor-based logic switch, time for the weight updating process are also available. and (b) truth table for AND operation. Moreover, an amplifier is added to generate the errors, creating an opportunity for updating the synaptic weights w( ) on-chip. where t is the width of the doped region (initial width w( ) = w ∈ ( ) ( ) = ( ) 2) The memristor-based SNN in [26] is expanded to MNN 0 0 R), R t is the memristance (R t M q ), and provides enhanced robustness despite the memris- D is the thickness of the TiO2, RON denotes the internal tance variations. The proposed memristor-based synaptic low memristance when the memristor is completely doped w( ) = crossbar circuit uses fewer memristors and no transistors ( t D), ROFF denotes the internal high memristance w( ) = as compared with the synaptic circuits discribed in [11], when the memristor is completely undoped ( t 0), and ( ) v( ) [12], [14]-[16], [18]-[20], [22], [23], [25], [30], and [31]. i t and t are, respectively, the current and voltage of the 3) An adaptive BP algorithm suitable for the proposed memristor [5]. memristor-based MNN is developed to train neural net- To consider the characteristics of the memristors, differ- works and perform the XOR function and character ent memristor models are compared in [35]–[37]. Moreover, recognition. Moreover, the weight adjustment process and different memristor-based SPICE models and circuits are the proposed MNNs exhibit higher recognition rates and presented in [38]–[40]. A new model is therefore proposed require fewer cycles. which matches the synaptic behavior of recent memristive The remainder of this paper is organized as follows. The devices [41]. The derivative of the state variable in the proposed memristive model is memristor and the MS are discussed in section II. The ⎧ proposed memristor-based SNN and the expanded MNN are ⎪ RON ioff ⎪μv f (w(t)), v(t)>V + > 0 presented in section III. The operation of the MNN based on ⎨⎪ ( ) − T dw(t) D i t i0 the proposed adaptive BP algorithm is described in Section IV. = 0, V − v(t) V + dt ⎪ T T The robustness of the crossbar architecture is discussed ⎪ RON i(t) ⎩μv f (w(t)), v(t)

VR is the output. To ensure the correctness of the AND logic operation, RR RP, RQ, as described in [2]. The truth table for the memristor-based AND operation is shown in Fig. 1(b). To simplify the calculation, the window function is ignored. The nonlinear ion drift phenomenon is considered in the simulation with a window function. The time required for a memristor to change from RON to ROFF or from ROFF to RON is assumed to be the same. The initial weight of P and Q memristors is arbitrary and the switching time T1 for MS is [2] 2 2ionRROFF 2β1ion D T1 ≈ = , (5) kVH μv VH 2 where k = μv RRON/D , R is the difference between ROFF and RON (R = ROFF − RON), and β1 = ROFF/RON. ROFF and RON are, respectively, the high and low memristances of memristors P and Q, as shown in Fig. 1(a). No threshold in the MS is assumed. The output error Ve is [2] RQ RON Fig. 2. Proposed memristor-based SNN. Ve ≈ VH = VH. (6) RP + RQ ROFF + RON

To change the memristance of MR from an arbitrary initial synaptic weights on chip. The requirement of T1 to be smaller memristance RRi to a final memristance RR f , the relationship than T2 can be achieved by using different memristors in the between R and time T is [2] MS and synapses. The memristance of the synaptic memristor ⎧ R V ( R − R ) − i (R − R ) is much higher than in the MS. ⎪ R ln Ri ln R f 0 Ri R f , > ⎨ VR 0 Assume a learning system that operates on K discrete = k ioff = , ,..., T ⎪ R2 − R2 (7) iterations of inputs, indexed by k 1 2 K. During each ⎩⎪ Ri R f , < , ion VR 0 iteration k, the system receives a pair of two vectors of size 2k VR (k) M (k) N M and N: inputs VI ∈ R and outputs VO ∈ R . = μ / 2 × where k v RRON D ,andRON is the low memristance For example, assume W is an adjustable N M matrix, and of MR. consider the estimator [23], When the memristance of the memristor MR changes from (k) (k) (k) VO = W VI , (9) RON to ROFF, from (7), the time T2 is or R2 − R2 (β + ) 2 = OFF ON = 2 1 D , M T2 ion ion (8) (k) (k) (k) 2k V 2μv V = , R R VO j W ji VIi (10) i=1 where β2 = R /R ,andR is the high memristance OFF ON OFF = , ,..., = , ,..., of MR. T1 is desired to be as small as possible with respect where i 1 2 M and j 1 2 N. to T ,soβ <β and μv >μv . A new synaptic array composed of a single crossbar array 2 1 2 0 − of M (G ji) and the constant-term circuit of Gs is shown = / III. MEMRISTIVE NEURAL NETWORK CIRCUITS in Fig. 2. Here, Gs (Gs 1 Rs) is the conductance of Rs , and G (G = 1/R ) is the memristor conductance at the In section III.A, a memristor-based single layer neural ji ji ji crossing point between the i th row and j th column. V is the network (SNN) is expanded into a multilayer neural net- Ii input voltage applied to the i th row. According to Kirchhoff’s work (MNN). Moreover, the modified back propagation (BP) law, V is [26] learning method is proposed for on-chip MNN in section III.B. F M R f VF =− VIi . (11) Rs A. Memristor-Based SNN i=1 In this section, the MS is utilized in the memristor-based th The output voltage of the j column VO j is synaptic crossbar circuit in [26], where both plus-polarity M and minus-polarity connection matrices are realized by a R0 VO j =− (R0 × G ji × VIi ) + VF . (12) single crossbar array and a simple constant-term circuit [26]. R f i=1 The second amplifier A2 is changed (as shown in Fig. 2) in the output part [26] to correctly operate the neural network Combining (12) with (11), VO j is (the connection of the positive and negative terminals of M the amplifier are switched). Moreover, amplifier A3 is added VO j = R0 × (Gs − G ji) × VIi . (13) (as shown in Fig. 2) to generate the errors, updating the i=1 680 IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS–I: REGULAR PAPERS, VOL. 65, NO. 2, FEBRUARY 2018

The synaptic weight is

W ji = R0 × (Gs − G ji). (14) The comparator enables VO j as > = ( ) = VH if VO j 0 VO j f VO j (15) VL if VO j ≤ 0, where VH and VL (VL = 0) are, respectively, the high and low voltages of the comparator. The output of the estimator VO = WVI predicts the target output VT for new unseen inputs VI. To solve this problem, the synapse weights W are updated to minimize the error between the outputs and target outputs over a K0 long subset of the training set (k = 1, 2,...,K0). The error vector is (k) = (k) − (k). V VT VO (16) A common measure is the mean square error (MSE) [23] which is K0 ( ) MSE = V k 2. (17) k=1 The performance of the outputs is tested over another subset, Fig. 3. Proposed memristor-based MNN. called the test set (k = K0 + 1, K0 + 2,...,K). A feasible iterative algorithm for minimizing the objective (17) is ∈[− , ] ∈[ , ] Assuming W ji 1 1 while R ji RON ROFF , R0, Rs , (k) (k+1) (k) 1 (k) 2 R f are determined from (14). From (7), the time T in each W = W − η∇ (k) V , (18) W step is 2 ⎧ η ( ) ( + ) ( ) ( + ) where is the learning rate. Using the chain rule (9) and (16), ⎪ ( k − k 1 )− ( k − k 1 ) ( ) (k) (k) (k) ⎪VW ln R ji ln R ji i0 R ji R ji ∇ ( ) k 2 =−( − )( ) ⎪ , > W k V 2 VT VO VI . Therefore, ⎨⎪ VW 0 (k) ≡ (k+1) − (k) k ioff defining W W W ,theouter product is T (k) = ( ) ( ) ( ) ⎪ (k) (k+ ) W k = ηV k (V k ) , (19) ⎪(R )2 − (R 1 )2 I ⎩⎪ ji ji , < . ion VW 0 or 2k VW ( ) ( + ) ( ) ( ) ( ) W k = W k 1 − W k = ηV k (V k ). (20) ji ji ji j Ii (24) Specifically, the MNNs are commonly trained using the BP algorithm, which is an efficient form of online gradient B. Memristor-Based MNN descent [43]. Importantly, note that the update rule in (20) (k) A simple two-layer neural network can be expanded to a is local, i.e., the change in the synaptic weight W ji depends (k) (k) multilayer NN, as shown in Fig. 3. A two-layer crossbar is only on the input V i and error V j . I considered. In layer 1, an N×M matrix W(1) (W ji) corresponds To implement BP training for the neural network, the to N neurons and M inputs. In layer 2, a P × N matrix W(2) relevant time is determined for each step. From (14), (Wpj) corresponds to P neurons and N inputs. (k) =− (k). An adaptive BP algorithm is [30] W ji R0 G ji (21) 1) Initialize the memristors with a high memristance R . From (20), OFF 2) Randomly apply the weight updating voltage VW to all ( ) ( ) ( ) ( ) W k = η(V k − V k )V k of the memristors to record all of the random values of ji ⎧ T j O j Ii (1) (1) (k) (k) (k) the initial weights (W ji and Wpj ) and memristances ⎪0ifV = V or V = 0 ( ) ( ) ⎨ T j O j Ii (R 1 and R 1 ). ( ) ( ) ( ) ji pj = ηV 2 if V k = V , V k = V , V k = 0 (22) 3) Apply the input patterns V to the crossbar circuit and ⎪ H Ii H T j H O j I ⎩ 2 (k) (k) (k) evaluate any hidden and output neuron values. −ηV if V = VH, V = 0, V = VH. H Ii T j O j 4) For output layer neurons, determine the error V between (k) = the neuron output V and the target output V . If W ji 0, O T 2 P ( ) 1 1 ηV (k) (k) (k) | G k |=| − |= H . (23) V = V W . (25) ji (k+1) (k) 1 j 2p pj R0 R ji R ji p=1 ZHANG et al.: MEMRISTOR-BASED CIRCUIT DESIGN FOR MULTILAYER NEURAL NETWORKS 681

5) Back propagate the error. ( ) ( ) ( ) ( ) W k = η(V k − V k )V k pj ⎧ T2p O2p I2 j ⎪ (k) = (k) (k) = ⎨⎪0ifVT2p VO2p or VI2 j 0 ( ) ( ) ( ) = ηV 2 if V k = V , V k = V , V k = 0 ⎪ H I2 j H T2p H O2p ⎩−η 2 (k) = , (k) = , (k) = , VH if VI2 j VH VT2p 0 VO2p VH (26) where neuron p is a connected with the previous layer neuron j. Fig. 4. Memristor-based synaptic weight adjustment method. (a) M22 is 6) Apply write pulses to the crossbar with the pulse adjusted from 1 to −1, and (b) M22 is adjusted from −1to1. width proportional to Wpj to update the memristor ( ) conductance. If W k = 0, pj 3 Update layer 2 synaptic weights based on layer 2 2 ( ) 1 1 ηV errors. | G k |=| − |= H . (27) pj (k+1) (k) 4 Update layer 1 synaptic weights based on layer 1 errors. R R R0 pj pj The four-step operation of the proposed BP training is (k) 7) Determine W ji to ensure that each memristor conduc- described as follows: tance is changed. Step 1: The input signals, Ctrl1 and Ctrl2, are set ( ) ( ) ( ) to VH, turning on the MS. A set of input voltages is W k = ηV k V k ji ⎧ 1 j I1i applied to the layer 1 neurons, and the layer 2 neu- ( ) ⎨ k = ron outputs VO21, VO22,...,VO2p,...,VO2P are compared 0ifVI1i 0 = P V , V ,...,V ,...,V ⎩ (k) (k) (k) (28) with the expected outputs T21 T22 T2p T2P. ηVH V W if V = 0. p=1 2p pj I1i This process is shown in Fig. 3. The error terms V ,V ,...,V ,...,V are based on the differ- 8) Apply write pulses to the crossbar with the pulse 21 22 2p 2P (k) ence between the observed outputs and the expected outputs. width proportional to W ji to update the memristor ( ) These values are generated using comparators that provide a conductance. If W k = 0, ji discretized error value of VH, −VH, or 0. These errors are − ( ) 1 1 amplified (to 2 V, 2 V, or 0) and applied to the synaptic array | G k |=| − | ji (k+1) (k) to change the value of the memristances, as shown in Fig. 4. R R | | ji ji The value of W ji can be obtained in testing process and η P (k) (k) the corresponding |G | (or |R |) can be calculated with VH p=1 V2p Wpj ji ji = . (29) FPGA or LUT (lookup table). The initial states of memristors R 0 are recorded and the calculation of adjustment time can be 9) If the error does not converge to a sufficiently small value, processed in FPGA. Each synaptic adjustment is controlled return to step 2). with peripheral circuits. Remark 1: Using a similar method, (24)-(29) can be written Step 2: To back propagate the errors of layer 2, as a general expression describing N layers. Ctrl1 and Ctrl2 are set to VH. The layer 2 errors (V21,V22,...,V2p,...,V2P) are applied to IV. SYSTEM OPERATION the layer 2 weights to generate the layer 1 errors (V11,V12,...,V1 j ,...,V1N). The methods of operating the proposed BP training and Step 3: Ctrl1 and Ctrl2 are set to 0 (MS is turned off) synaptic weight adjustment process on-chip are introduced to isolate the second layer crossbar from the first layer. in this section, which exhibit the advantages of no sneak A training unit amplifies the layer 2 errors along with the paths. In section IV.A, a four step BP on-chip training layer 2 intermediate outputs to generate a set of training pulses. method is presented. Sneak path methods that change the These pulses are applied to the layer 2 memristor crossbar to synaptic weight in the neural network array are described in update the layer 2 synaptic weights. section IV.B. Step 4: To isolate the first layer crossbar from the input voltages, Ctrl1 and Ctrl2 are set to 0. A process similar A. BP Training Circuit to step 3 is applied to update the synaptic weights in the The circuit implementation of BP training for the neural memristor crossbar array of layer 1. network circuits is shown in Fig. 3. The training is composed of four steps [30]: B. Synaptic Weight Updating 1 Apply input voltages to layer 1 and record layer 2 neuron In the testing process, the applied voltage is the input output errors. voltage 0 or VH. The final output voltages are error signal 2 Back propagate layer 2 errors through the second layer voltages 0, VH or −VH. The applied weight update voltages weights and record layer 1 errors. are amplified to 0, 2 V or −2 V, which are controlled with 682 IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS–I: REGULAR PAPERS, VOL. 65, NO. 2, FEBRUARY 2018

TABLE I TABLE II WEIGHT UPDATE DIRECTION PARAMETERS OF THE MEMRISTOR-BASED MNN

peripheral circuits. The memristor weight updating process utilized by the training unit is presented in this section. All the weights within a crossbar can be updated using the following method. Different grid sizes of the crossbar arrays have been evaluated, and a 3×3 crossbar array is considered as an example. When a voltage greater than the memristor threshold is applied across a memristor, the memristance increases or decreases depending upon the terminal of the device. The two possible update situations of V j are listed in Table I along with the direction that the weight is changed, or V j = 0and the weight remains unchanged. 1) Decreasing Synaptic Weight: If V j < 0, VW > 0. B. Physical Challenges Assume only memristor M22 is changed and the other memris- There are four major physical constraints: 1) The memristors remain unchanged. One possible solution is to apply dif- tors in the MS and synaptic array are different; 2) The bound- = ferent voltages on different rows and columns. For VW 2V ary voltage VH is smaller than the VT of the memristors; 3) The on row 2 and the other rows are connected to GND, column 2 sum of VH and the maximum amplitude of the noise sources is . is connected to GND and 0 9 V is applied to the other columns, smaller than VT. 4) The precision of the synaptic adjustment as shown in Fig. 4(a). The threshold voltage (VT)ofthe process depends upon the precision of the conductance of the synapse memristors is 1.5V,andR f is much smaller than memristors and is adjusted by applying a different number of the memristance of M ji. Only the voltage of memristor M22 voltage pulses. is 2 V, which is larger than VT. The memristance of M22 therefore decreases. The voltage of memristors M11, M13, C. Sneak Path Currents − . M31,andM33 is 0 9 V, while the voltage of memristors The training process is not affected by sneak path . M12 and M32 is 1 1 V and the voltage of memristors M21 currents [47]. Only one write line is raised to reach the weight M V and 23 is 0 V. The absolute values are all smaller than T, changing voltage VW. Other lines are protected to ensure the memristances therefore remain the same. the absolute value of the voltages remain smaller than VT, 2) Increasing Synaptic Weight: V > V < If j 0, W 0. as shown in Fig. 4. VW is set higher than VT since only a small A similar method is utilized, as shown in Fig. 4(b). change in the memristance is needed during each training step. To make changes in the memristor conductances, a volt- Hence, the voltage drop on the other memristors is smaller age of appropriate magnitude and polarity for a suitable than VT, and does not result in unexpected changes in the duration across the memristor is applied [44]–[46]. In each memristance. However, in larger arrays, after a significant (1), (2),..., (k),..., (K) × 6 step, the required durations T2 T2 T2 T2 number of pulses (such as a 16 kb array after 5 10 pulses), (1), (2),..., the write operation is disturbed and exhibits 164 false can be determined from (27) and (24), and T1 T1 (k),..., (K) bits [48]. T1 T1 can be determined from (29) and (24). VI. SIMULATION RESULTS V. ROBUSTNESS ANALYSIS PSPICE is used to evaluate the proposed memristor-based Process variations and noise can significantly affect circuit MNN circuits. The circuits and learning process are also performance. In this section, these noise sources, the impact evaluated in Matlab. The basic memristor model [38] and the of physical design challenges, and sneak path currents are synaptic model with threshold voltages [41] are both used in discussed. the simulations. The parameters of the memristor-based MNN are listed in Table II. The drive voltages of amplifiers A0 and A. Sources of Noise A1 are ±5V,0.9Vand0VforA2,and±0.9 V for A3. Electrical noise from the power supplies and neighboring Many different voltage levels are required, ±5 V voltages is wires can significantly degrade the quality of the analog sig- utilized to produce ±0.9 V and ±2 V voltages through step nals. Different from process variations that remain unchanged down circuits. The range of weights supported by a memristor- once the circuit is fabricated, signal fluctuations vary dur- based synapse is [−1, +1]. ing circuit operation. When character recognition is operated on-chip, the input voltages can be affected by random noise. A. Synaptic Weight Updating The inputs are changed from 0 to VH to evaluate the robustness Different grid sizes of crossbars have been considered. of the proposed MS and synaptic circuit. A3× 3 crossbar array, for example, considers that memristor ZHANG et al.: MEMRISTOR-BASED CIRCUIT DESIGN FOR MULTILAYER NEURAL NETWORKS 683

Fig. 6. Changes in input and output voltages during each iteration of the XOR operation. (a) Input voltage V , (b) input voltage V , and (c) output I11 I12 voltage VO21. Fig. 5. Simulation results of memristor-based synaptic weight adjustment process. (a) Input voltage VW for weight adjustment operation, (b) memristance of memristor M11 (M13, M31, M33 are the same), (c) memristance of memristor M12 (M32 is the same), (d) memristance of memristor M21 (M23 is the same), and (e) memristance of memristor M22. X represents time and Y represents voltage.

M22 in row two, column two is accessed. Using the synaptic weight updating approach, the simulation results demonstrate that only M22 can be updated from R to R or from R OFF ON ON Fig. 7. XOR operation, (a) MNN, and (b) truth table. to ROFF (shown in Fig. 5) while the other memristors remain the same during the updating process, verifying that the sneak path currents have been eliminated during the weight updating process. The switching time (gate delay) is approximately equal to 0.02 ns from (5), and the error is approximately equal to 0.01 V from (6). A pulse of amplitude 2 V and width 0.61 ns from (8) change the memristance from ROFF to RON (the weight of the synapse from 1 to −1). Similarly, a pulse of amplitude −2Vandwidth0.61 ns change the memristance − from RON to ROFF (the weight of the synapse from 1to1), as shown in Fig. 5.

B. XOR Function Fig. 8. Training error in each cycle for the XOR operation in the MNN. A simple two layer neural network is simulated to verify the correctness of the learning method. The XOR function is learned on a network of 2 inputs × 3 hiddens × 1 output, as shown in Fig. 7. In layer 1, a 3 × 2 memristor matrix corresponds to three neurons and two inputs. In layer 2, a 1×3 memristor matrix corresponds to three inputs and one output. One cycle contains four iterations for four different input patterns and the learning rate η = 0.1. The output voltages during each iteration are shown in Figs. 6. The error of the hardware training process for each neuron during each cycle is shown in Fig. 8 (1,000 events are tested to produce different samples). The error Ee is Fig. 9. Ten digit images used as an input to the network for character recognition. = 1 . Ee MSE (30) layer composed of 30 inputs × 10 outputs. The network is K0 trained on black and white images of size 5×6 pixels, as shown in Fig. 9. Consider, for example, image 5. The input and output C. Character Recognition voltages are shown in Table III (“1” represents VH). One cycle A more complex function is simulated on two different contains ten iterations for ten different input patterns and the networks for character recognition. One network is a single learning rate η = 0.1. The error (K0 = 10) of hardware 684 IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS–I: REGULAR PAPERS, VOL. 65, NO. 2, FEBRUARY 2018

Fig. 12. Training error in each cycle for character recognition in the MNN.

TABLE IV Fig. 10. Input image 5 in the left column is shown. Images with 20% COMPARISON OF RECOGNITION RATE FOR DIFFERENT NOISE LEVELS random pixels as white noise are shown in the middle column to produce a noisy image mask. The resulting noisy images used for testing are shown in the right column.

(e.g. number 5) to generate noisy images, as shown in Fig. 10. The original images for number 5 are shown in the left column, and uniform random noise mask images are shown in the middle column. Noisy images are acquired by ﬂipping all of the pixels in the character images wherever there is a white pixel in the noise mask image (assuming 20% noise). The average number of cycles for the correct recognition is shown in Fig. 11(b). The recognition rate of the network is 99.0% Fig. 11. Training error in each cycle for character recognition in the SNN. when 20% noise is added to the images, as listed in Table IV. (a) Input images are without noise, and (b) input images are with 20% noise. Another example network is two layers of 30 inputs × 6 hiddens × 4 outputs. Consider again, for example, image 5. TABLE III The input and output voltages are listed in Table III. One cycle INPUT AND OUTPUT VOLTAGE ARRAYS OF IMAGE 5 contains ten iterations for ten different input patterns and the learning rate η = 0.04. The error (K0 = 10) of hardware training for each neuron in each cycle is shown in Fig. 12 (1,000 events are tested to produce different samples). From simulation, the recognition rate of the network is 95.4% when 20% noise is added to the images. The recognition rate of the proposed memristor-based neural networks when noise is added to the images is compared with [18] in Table IV. The proposed memristor-based BP algorithm is compared with the winner-take-all algorithm [18] in Table V. Note that the proposed system exhibits a higher recognition rate and requires fewer cycles.

D. Effects of Memristance Variations In a practical fabricated memristor array, the the updating process is achieved by applying different numbers of positive or negative voltage pulses. In each step, the memristance or conductance of memristors are not adjusted precisely, training for each neuron in each cycle is shown in Fig. 11(a) the effects of memristance variations are therefore discussed (1,000 events are tested to produce different samples). in this section. The robustness of the memristive neural network circuits is To obtain a desired change in the memristance, a voltage tested using following method. Noise is added to input images of appropriate magnitude and polarity over a suitable duration ZHANG et al.: MEMRISTOR-BASED CIRCUIT DESIGN FOR MULTILAYER NEURAL NETWORKS 685

TABLE V [3] G. Indiveri, B. Linares-Barranco, R. Legenstein, G. Deligeorgis, and COMPARISON OF THE PROPOSED MEMRISTOR-BASED BP T. Prodromakis, “Integration of nanoscale memristor synapses in neu- ALGORITHM WITH THE WINNER-TAKE-ALL romorphic computing architectures,” Nanotechnol., vol. 24, no. 38, ALGORITHM IN [18] p. 384010, Sep. 2013. [4] L. Chua, “Memristor-The missing circuit element,” IEEE Trans. Circuit Theory, vol. 18, no. 5, pp. 507–519, Sep. 1971. [5] D. B. Strukov, G. S. Snider, D. R. Stewart, and R. S. Williams, “The missing memristor found,” Nature, vol. 453, pp. 80–83, May 2008. [6] P. Junsangsri and F. Lombardi, “Design of a hybrid memory cell using memristance and ambipolarity,” IEEE Trans. Nanotechnol., vol. 12, no. 1, pp. 71–80, Jan. 2013. [7] Y. Zhang, Y. Li, X. Wang, and E. G. Friedman, “Synaptic characteristics of Ag/AgInSbTe/Ta-based memristor for pattern recognition applications,” IEEE Trans. Electron Devices, vol. 64, no. 4, pp. 1806–1811, Apr. 2017. [8] L. Wang, Y. Shen, Q. Yin, and G. Zhang, “Adaptive synchronization of memristor-based neural networks with time-varying delays,” IEEE Trans. Neural Netw. Learn. Syst., vol. 26, no. 9, pp. 2033–2042, Sep. 2015. [9] M. Walker, P. Hasler, and L. A. Akers, “A CMOS neural network for pattern association,” IEEE Micro, vol. 9, no. 5, pp. 68–74, Oct. 1989. [10] B. V. Benjamin et al., “Neurogrid: A mixed-analog-digital multichip system for large-scale neural simulations,” Proc. IEEE, vol. 102, no. 5, pp. 699–716, May 2014. [11] K. D. Cantley, A. Subramaniam, H. J. Stiegler, R. A. Chapman, and E. M. Vogel, “Neural learning circuits utilizing nano-crystalline silicon transistors and memristors,” IEEE Trans. Neural Netw. Learn. Syst., vol. 23, no. 4, pp. 565–573, Jun. 2012. [12] H. Manem, J. Rajendran, and G. S. Rose, “Stochastic gradient descent inspired training technique for a CMOS/nano memristive trainable threshold gate array,” IEEE Trans. Circuits Syst. I, Reg. Papers, vol. 59, no. 5, pp. 1051–1060, May 2012. Fig. 13. Training error during each cycle with different levels of memristance [13] Z. Wang, W. Zhao, W. Kang, Y. Zhang, J.-O. Klein, and C. Chappert, variations for character recognition in the SNN. “Ferroelectric tunnel memristor-based neuromorphic network with 1T1R crossbar architecture,” in Proc. Int. Joint Conf. Neural Netw. (IJCNN), Jul. 2014, pp. 29–34. (pulse number) is applied across the memristor. However, due [14] S. P. Adhikari, H. Kim, R. K. Budhathoki, C. Yang, and L. O. Chua, to cycle-to-cycle variations, the change during each step cannot “A circuit-based learning architecture for multilayer neural networks be accurately controlled. Different random noise ranging from with memristor bridge synapses,” IEEE Trans. Circuits Syst. I, Reg. Papers, vol. 62, no. 1, pp. 215–223, Jan. 2015. 5% to 15% is therefore added to the memristance during each [15] S. P. Adhikari, C. Yang, H. Kim, and L. O. Chua, “Memristor bridge step, as shown in Fig. 13. The corresponding cycle numbers synapse-based neural network and its learning,” IEEE Trans. Neural are, respectively, 14, 30, and 48. The proposed memristor- Netw.Learn.Syst., vol. 23, no. 9, pp. 1426–1435, Sep. 2012. [16] H. Kim, M. P. Sah, C. Yang, T. Roska, and L. O. Chua, “Neural synaptic based MNN is shown to be inherently tolerant to memristance weighting with a pulse-based memristor circuit,” IEEE Trans. Circuits variations. Syst. I, Reg. Papers, vol. 59, no. 1, pp. 148–158, Jan. 2012. [17] C. Zamarreño-Ramos, L. A. Camuñas-Mesa, J. A. Pérez-Carrasco, T. Masquelier, T. Serrano-Gotarredona, and B. Linares-Barranco, VII. CONCLUSIONS “On spike-timing-dependent-plasticity, memristive devices, and build- A single memristor-based synaptic architecture for mul- ing a self-learning visual cortex,” Frontiers Neurosci., vol. 5, p. 26, Mar. 2011. tilayer neural networks with on-chip learning is proposed. [18] A. M. Sheri, H. Hwang, M. Jeon, and B.-G. Lee, “Neuromorphic Moreover, an adaptive BP algorithm suitable for the pro- character recognition system with two PCMO memristors as a synapse,” posed memristor-based multilayer neural network is applied IEEE Trans. Ind. Electron., vol. 61, no. 6, pp. 2933–2941, Jun. 2014. [19] M. Chu et al., “Neuromorphic hardware system for visual pattern to train neural networks and perform the XOR function and recognition with memristor array and CMOS neuron,” IEEE Trans. Ind. character recognition. A simple, compact, and reliable neural Electron., vol. 62, no. 4, pp. 2410–2419, Apr. 2015. network can be used for applications in pattern recognition [20] S. N. Truong, S.-J. Ham, and K.-S. Min, “Neuromorphic crossbar by combining the advantages of the memristor-based synaptic circuit with nanoscale ﬁlamentary-switching binary memristors for speech recognition,” Nanoscale Res. Lett., vol. 9, no. 1, pp. 1–9, architecture with the proposed BP weight change algorithm. Nov. 2014. The advantages of the proposed architecture are veriﬁed [21] R. Legenstein, C. Naeger, and W. Maass, “What can a neuron learn with through simulations, demonstrating that the proposed adaptive spike-timing-dependent plasticity?” Neural Comput., vol. 17, no. 11, pp. 2337–2382, 2005. BP algorithm exhibits higher recognition rates and requires [22] M. Hu, H. Li, Y. Chen, Q. Wu, G. S. Rose, and R. W. Linderman, “Mem- fewer cycles. ristor crossbar-based neuromorphic computing system: A case study,” IEEE Trans. Neural Netw. Learn. Syst., vol. 25, no. 10, pp. 1864–1878, Oct. 2014. REFERENCES [23] D. Soudry, D. Di Castro, A. Gal, A. Kolodny, and S. Kvatinsky, “Memristor-based multilayer neural networks with online gradient [1] Y. Zhang, Y. Shen, X. Wang, and Y. Guo, “A novel design for memristor- descent training,” IEEE Trans. Neural Netw. Learn. Syst., vol. 26, no. 10, based OR gate,” IEEE Trans. Circuits Syst. II, Exp. Briefs, vol. 62, no. 8, pp. 2408–2421, Oct. 2015. pp. 781–785, Aug. 2015. [24] E. Rosenthal, S. Greshnikov, D. Soudry, and S. Kvatinsky, “A [2] Y. Zhang, Y. Shen, X. Wang, and L. Cao, “A novel design for memristor- fully analog memristor-based neural network with online gradient based logic switch and crossbar circuits,” IEEE Trans. Circuits Syst. I, training,” in Proc. IEEE Int. Symp. Circuits Syst. (ISCAS), 2016, Reg. Papers, vol. 62, no. 5, pp. 1402–1411, May 2015. pp. 1394–1397. 686 IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS–I: REGULAR PAPERS, VOL. 65, NO. 2, FEBRUARY 2018

[25] M. Prezioso, F. Merrikh-Bayat, B. D. Hoskins, G. C. Adam, [48] H. Li et al., “Write disturb analyses on half-selected cells of cross-point K. K. Likharev, and D. B. Strukov, “Training and operation of an RRAM arrays,” in Proc. IEEE Int. Rel. Phys. Symp. (IRPS), Jun. 2014, integrated neuromorphic network based on metal-oxide memristors,” pp. MY.3.1–MY.3.4. Nature, vol. 521, pp. 61–64, May 2015. [26] S. N. Truong and K.-S. Min, “New memristor-based crossbar array architecture with 50-% area reduction and 48-% power saving for matrix- vector multiplication of analog neuromorphic computing,” J. Semicond. Yang Zhang received the B.S and Ph.D. degrees Tech. Sci., vol. 14, no. 3, pp. 356–363, Jun. 2014. in control science and engineering from the School of Automation, Huazhong University of Science and [27] Q. Wang, Y. Kim, and P. Li, “Architectural design exploration for Technology, in 2013 and 2017, respectively. Proc. 14th Int. neuromorphic processors with memristive synapses,” in From 2015 to 2016, he was a Visiting Ph.D. Conf. Nanotechnol. (IEEE-NANO) , Aug. 2014, pp. 962–966. Student with the University of Rochester. He is [28] D. Querlioz, O. Bichler, A. F. Vincent, and C. Gamrat, “Bioinspired currently a Researcher with the Shenzhen Key Labo- programming of memory devices for implementing an inference engine,” ratory of Spatial Smart Sensing and Services, School Proc. IEEE, vol. 103, no. 8, pp. 1398–1416, Aug. 2015. of Computer Science and Software Engineering, [29] D. Zhang et al., “All spin artificial neural networks based on compound Shenzhen University, Shenzhen, China. His current spintronic synapse and neuron,” IEEE Trans. Biomed. Circuits Syst., research interests include memristor-based memory, vol. 10, no. 4, pp. 828–836, Aug. 2016. deep neural networks, and image recognition. [30] R. Hasan and T. M. Taha, “Enabling back propagation training of memristor crossbar neuromorphic processors,” in Proc. Int. Joint Conf. Neural Netw. (IJCNN), Beijing, China, Jul. 2014, pp. 21–28. [31] I. Kataeva, F. Merrikh-Bayat, E. Zamanidoost, and D. Strukov, “Efficient training algorithms for neural networks based on memristive crossbar Xiaoping Wang (M’14) was born in Hubei, China, circuits,” in Proc. Int. Joint Conf. Neural Netw. (IJCNN), 2015, pp. 1–8. in 1974. She received the B.S. degree and the M.S. [32] D. Chabi, Z. Wang, W. Zhao, and J.-O. Klein, “On-chip super- degree in automation from Chongqing University, vised learning rule for ultra high density neural crossbar using Chongqing, China, in 1997 and 2000, respectively, memristor for synapse and neuron,” in Proc. Int. Symp. Nanoscale and the Ph.D. degree in systems engineering from Archit. (NANOARCH), 2014, pp. 7–12. the Huazhong University of Science and Technology, [33] S. Kvatinsky, N. Wald, G. Satat, A. Kolodny, U. C. Weiser, and Wuhan, China, in 2003. Since 2011, she has been a E. G. Friedman, “MRL–memristor ratioed logic,” in Proc. 13th Int. Professor with the School of Automation, Huazhong Workshop Cellular Nanoscale Netw. Appl., Aug. 2012, pp. 1–6. University of Science and Technology. Her current [34] S. P. Adhikari, M. P. Sah, H. Kim, and L. O. Chua, “Three fingerprints research interests include memristors and its applica- of memristor,” IEEETrans.CircuitsSyst.I,Reg.Papers, vol. 60, no. 11, tion to memory storage, modeling, simulation, and pp. 3008–3021, Nov. 2013. optimization. [35] S. Kvatinsky, E. G. Friedman, A. Kolodny, and U. C. Weiser, “TEAM: Threshold adaptive memristor model,” IEEE Trans. Circuits Syst. I, Reg. Papers, vol. 60, no. 1, pp. 211–221, Jan. 2013. [36] S. Kvatinsky, M. Ramadan, E. G. Friedman, and A. Kolodny, “VTEAM: Eby G. Friedman (S’80–M’81–SM’90–F’00) rece- A general model for voltage-controlled memristors,” IEEE Trans. Cir- ived the B.S. degree from Lafayette College in 1979, cuits Syst. II, Express Briefs, vol. 62, no. 8, pp. 786–790, Aug. 2015. and the M.S. and Ph.D. degrees from the University [37] A. Ascoli, F. Corinto, V. Senger, and R. Tetzlaff, “Memristor model of California at Irvine, Irvine, CA, USA, in 1981 comparison,” IEEE Circuits Syst. Mag., vol. 13, no. 2, pp. 89–105, and 1989, respectively, all in electrical engineering. 2nd Quart., 2013. From 1979 to 1991, he was with Hughes Air- [38] Z. Biolek, D. Biolek, and V. Biolkova, “SPICE model of memristor with craft Company, promoted to Manager of the Signal nonlinear dopant drift,” Radioengineering, vol. 18, no. 2, pp. 210–214, Processing Design and Test Department, responsible Jun. 2009. for the design and test of high performance digital [39] D. Batas and H. Fiedler, “A memristor SPICE implementation and a and analog ICs. He has been with the Department of new approach for magnetic flux-controlled memristor modeling,” IEEE Electrical and Computer Engineering, University of Trans. Nanotechnol., vol. 10, no. 2, pp. 250–255, Mar. 2011. Rochester, since 1991, where he is a Distinguished Professor and the Director [40] C. Yakopcic, T. M. Taha, G. Subramanyam, and R. E. Pino, “Gener- of the High Performance VLSI/IC Design and Analysis Laboratory. He is also alized memristive device SPICE model and its application in circuit a Visiting Professor with the Technion âŁ” Israel Institute of Technology. He design,” IEEE Trans. Comput.-Aided Design Integr., vol. 32, no. 8, has authored over 500 papers and book chapters, 13 patents, and the author pp. 1201–1214, Aug. 2013. or editor of 18 books in the fields of high speed and low power CMOS design [41] Y. Zhang, X. Wang, Y. Li, and E. G. Friedman, “Memristive model for techniques, 3-D design methodologies, high speed interconnect, and the theory synaptic circuits,” IEEE Trans. Circuits Syst. II, Exp. Briefs, vol. 64, and application of synchronous clock and power distribution networks. His no. 7, pp. 767–771, Jul. 2017. current research and teaching interests are in high performance synchronous [42] Y. Li et al., “Activity-dependent synaptic plasticity of a chalcogenide digital and mixed-signal microelectronic design and analysis with application electronic synapse for neuromorphic systems,” Sci. Rep., vol. 4, p. 4906, to high speed portable processors and low power wireless communications. May 2014. Dr. Friedman is a Senior Fulbright Fellow. He was a recipient of the IEEE Circuits and Systems Charles A. Desoer Technical Achievement Award, the [43] Y. A. LeCun, L. Bottou, G. B. Orr, and K.-R. Müller, “Efficient University of Rochester Graduate Teaching Award, the College of Engineering backprop,” in Neural Networks: Tricks of the Trade, 2nd ed. Berlin, Teaching Excellence Award, and a member of the University of California Germany: Springer, 2012, pp. 9–48. at Irvine Engineering Hall of Fame. He is an Editor-in-Chief of the Micro- [44] L. Gao, F. Alibart, and D. B. Strukov, “Programmable CMOS/memristor electronics Journal, a member of the editorial boards of the Journal of Low threshold logic,” IEEE Trans. Nanotechnol., vol. 12, no. 2, pp. 115–119, Power Electronics and the Journal of Low Power Electronics and Applications, Mar. 2013. and a member of the technical program committee of numerous conferences. [45] G. Papandroulidakis, I. Vourkas, N. Vasileiadis, and G. C. Sirakoulis, He was an Editor-in-Chief and the Chair of the Steering Committee of the “Boolean logic operations and computing circuits based on memris- IEEE TRANSACTIONS ON VERY LARGE SCALE INTEGRATION SYSTEMS, tors,” IEEE Trans. Circuits Syst. II, Express Briefs, vol. 61, no. 12, the Regional Editor of the Journal of Circuits, Systems and Computers, pp. 972–976, Dec. 2014. a member of the Editorial Board of the PROCEEDINGS OF THE IEEE, the IEEE [46] I. Vourkas and G. C. Sirakoulis, “On the generalization of composite TRANSACTIONS ON CIRCUITS AND SYSTEMS II: ANALOG AND DIGITAL memristive network structures for computational analog/digital circuits SIGNAL PROCESSING, the IEEE JOURNAL ON EMERGING AND SELECTED and systems,” Microelectr. J., vol. 45, no. 11, pp. 1380–1391, Nov. 2014. TOPICS IN CIRCUITS AND SYSTEMS, Analog Integrated Circuits and Signal [47] M. A. Zidan, H. A. H. Fahmy, M. M. Hussain, and K. N. Salama, Processing,andtheJournal of Signal Processing Systems, a member of the “Memristor-based memory: The sneak paths problem and solutions,” Circuits and Systems Society, Board of Governors, and Program and Technical Microelectron. J., vol. 44, no. 2, pp. 176–183, 2013. chair of several IEEE conferences.