SIMULATING LARGE SCALE MEMRISTOR BASED CROSSBAR FOR NEUROMORPHIC APPLICATIONS Thesis Submitted to The School of Engineering of the UNIVERSITY OF DAYTON In Partial Fulfillment of the Requirements for The Degree of Master of Science in Electrical Engineering By Roshni Uppala UNIVERSITY OF DAYTON Dayton, Ohio May, 2015 SIMULATING LARGE SCALE MEMRISTOR BASED CROSSBAR FOR NEUROMORPHIC APPLICATIONS Name: Uppala, Roshni APPROVED BY: Tarek M. Taha, Ph.D. Vijayan K. Asari, Ph.D. Advisor Committee Chairman Committee Member Associate Professor, Department of Professor, Department of Electrical and Electrical and Computer Engineering Computer Engineering Guru Subramanyam, Ph.D. Committee Member Chair, Professor, Department of Electrical and Computer Engineering Engineering John G. Weber, Ph.D. Eddy M. Rojas, Ph.D., M.A., P.E., Associate Dean Dean, School of Engineering School of Engineering ii c Copyright by Roshni Uppala All rights reserved 2015 ABSTRACT SIMULATING LARGE SCALE MEMRISTOR BASED CROSSBAR FOR NEUROMORPHIC APPLICATIONS Name: Uppala, Roshni University of Dayton Advisor: Dr. Tarek M. Taha The memristor is a novel nano-scale device discovered in 2008. Memristors are basically non- volatile variable resistors. Various breakthroughs of memristive devices have shown the potential of memristive crossbar designs for their ultra-high density and low-power memory. Initial studies have shown that memristor based neuromorphic processors could potentially consume 300,000 times less power than a traditional Intel Xeon Processor for neural network applications. These neuromorphic processors require large memristor arrays for evaluating neural networks. A key problem in design- ing these processors is simulating the large arrays of thousands of memristors as these simulations require extremely long times (sometimes weeks to months). Additionally, most existing simulation tools are unable to handle the large number of computations seen in these simulations. The simu- lations of these large arrays has not been examined in literature. One of the key objectives of this work is to evaluate large scale memristor crossbars that allow high density layout of synapses and thus enable building of highly capable neuromorphic systems. In order to achieve this, this study utilized a newly released parallel SPICE simulator, Xyce, developed by Sandia National Labs. Al- though it is 4 times faster than the existing SPICE simulators, Xyce is still too slow for evaluating iii the capabilities of large crossbar arrays. Therefore, this thesis also examines the development of an equivalent mathematical representation of the circuit level memristor based neuromorphic circuits for faster computation of the crossbars without the use of SPICE. This system is then used as an offline training approach to approximate the SPICE circuit. These trainings require thousands of iterations of the crossbar array. This thesis examined the training of memristor-based crossbars for neural network based and pattern recognition applications such as Image classification at the circuit level. Finally the analysis presented in this work will be crucial in understanding the future of the memristor-based crossbars in developing highly reliable and extremely low power processors and neuromorphic systems. iv Dedicated to my mother, sisters and all the women. “Life is not easy for any of us. But what of that? We must have perseverance and above all confidence in ourselves. We must believe that we are gifted for something and that this thing must be attained.” - Marie Skodowska-Curie “The woman who follows the crowd will usually go no further than the crowd. The woman who walks alone is likely to find herself in places no one has ever been before.” - Albert Einstein v ACKNOWLEDGMENTS I express my sincere gratitude to my advisor Dr. Tarek M. Taha for his time and consideration throughout my thesis. I would also like to thank my committee members Dr. Vijayan K. Asari and Dr. Guru Subramanyam for their support and encouragement. I would also like to thank Christopher Yakopcic for helping me throughout my thesis and letting me build my work over his. I would also like to extend my thanks to Maureen Schlangen and all the University of Dayton Roesch Library staff for their endless support, strength and for funding me throughout my study here at UD in the form of student employment and graduate assistantship. Lastly, I would like to thank Binu Nair for his trust and belief in me and for being a pillar of strength all through this. I finally thank my family in providing me the right kind of assistance. vi TABLE OF CONTENTS ABSTRACT . iii DEDICATION . .v ACKNOWLEDGMENTS . vi LIST OF FIGURES . ix LIST OF TABLES . xii I. INTRODUCTION . .1 1.1 Evolution of Memristors - Mathematical Model . .1 1.2 Re-discovery of the Memristor in its Physical Form . .2 1.3 Memristors in Neuromorphic Computing . .3 1.4 Contributions . .4 1.5 Overview of the Thesis . .4 II. RELATED WORK . .5 2.1 Insights in to Memristors . .5 2.2 Neuromorphic Computing . .7 2.2.1 Early Architectures . .8 2.2.2 Modern Architectures . 10 2.2.3 Memristor based Architectures . 13 III. MEMRISTOR MODEL . 16 3.1 University of Dayton (UD) Memristor Model . 17 3.2 Memristor as Neurons . 19 3.3 Memristor based Neuromorphic Crossbar . 22 3.3.1 Write Operation . 22 3.3.2 Read Operation . 24 vii IV. LARGE SCALE CIRCUIT SIMULATIONS . 25 4.1 Need for Parallel Circuit Simulation . 26 4.2 Xyce . 27 4.2.1 Parallel Simulation using Xyce . 28 4.2.2 Installation of Xyce [1] . 30 4.3 Memristor based Neuromorphic Circuit for SPICE Simulation . 38 4.3.1 Finite drivers . 40 4.3.2 Training of SPICE Circuit . 43 4.4 Results . 44 4.4.1 System Setup . 45 4.4.2 Analysis . 46 V. ALTERNATIVE APPROACH IN MODELING MEMRISTOR BASED CROSSBAR FOR NEUROMORPHIC APPLICATIONS . 51 5.1 Traditional Single Layer Perceptron . 52 5.2 Single Layer Perceptron with Memristor based Crossbars . 56 5.2.1 Approximate Solution to a Circuit Level Memristive based Crossbar. 56 5.2.2 Training Single Layer MATLAB based Memristor Crossbar. 60 5.3 Testing of Memristor based Crossbar as a Single Layer Perceptron in Xyce . 63 5.4 Results . 65 5.4.1 MNIST Digit Database . 66 5.4.2 MIT-CBCL Face Database . 71 VI. CONCLUSION . 77 BIBLIOGRAPHY . 79 viii LIST OF FIGURES 2.1 Insights into memristor. .6 2.2 A multi-chip add-in board with Ni1000 Recognition Accelerators where the CPU manages the flow of data through these accelerators [2]. .9 2.3 SYNAPSE-1 system architecture [3]. 10 2.4 Benchmarks for mapping kernels to neural networks [4]. 11 2.5 IBM Neurosynaptic Core [5, 6]. 12 2.6 DARPA SyNAPSE chipboard and its TrueNorth core [7, 8]. 13 2.7 The mrFPGA architecture [9]. (a) Overview of mrFPGA, and (b) Detailed design of the connection blocks and switch blocks. 14 2.8 SRAM array replaced by a memristor crossbar [10]. 15 2.9 Efficiency of memristors over other high performance architectures [10]. 15 3.1 Input voltage and current simulation waveforms of model in [11] based on device in [12]. The memristor mode parameters mentioned is as follows: Vp = 4V , Vn = 4V ,Ap = 816000,An = 816000,xp = 0:985,xn = 0:985,αp = 0:1,alphan = −4 −4 0:1,a1 = 1:6 × 10 ,a2 = 1:6 × 10 , b = 0:05, x0 = 0:01. This simulation result has been taken from [13]. 17 3.2 Illustration of Memristor as Synapse . 20 3.3 Weight Distribution [14]. 21 3.4 Memristor as a synapse and its neuro-morphic crossbar representation. 23 ix 3.5 Writing and reading into memristors in a crossbar. 24 4.1 Memristor . 39 4.2 Complimentary CMOS based finite driver circuit . 40 4.3 Memristor based neuromorphic crossbar . 41 4.4 Training memristor based crossbar in Xyce. 43 4.5 Effect of total training time in Xyce with change in number of cores to train a linearly separable 2-input logic function using level-3 Xyce MOSFET in the driver circuit. 46 4.6 Effect of total training time in Xyce with change in number of cores to train a lin- early separable 2-input logic function using Mosis-180nm MOSFETs in the driver circuit. 47 4.7 Convergence error curve from training 2-bit input linearly separable logic functions with 10,096 memristors. 48 4.8 Effect of total training time in Xyce with change in number of cores to train a linearly separable 3-bit input logic function using level-3 Xyce MOSFET and Mosis 180nm MOSFET in the driver circuit. 50 5.1 Single layer perceptron . 53 5.2 Matlab Memristive system . 56 5.3 Effective resistances for computation of voltage V1.................. 57 5.4 Effective resistances for computation of voltage V2.................. 58 5.5 Memristor based neuromorphic crossbar for testing offline trained weights for out- put voltage. 65 5.6 MNIST digit dataset. 67 5.7 Error curves obtained with the traditional perceptron with two different learning rates. 69 5.8 Error curves obtained with Matlab memristor crossbar during training with two dif- ferent learning rates. 69 x 5.9 Comparisons of accuracies with traditional, matlab memristor crossbar and Xyce implementation. 70 5.10 MIT-CBCL Face dataset. 73 5.11 Error curves obtained with traditional perceptron during training with two different learning rates for CBCL-MIT Face dataset. 74 5.12 Error curves obtained with Matlab memristor crossbar during training with two dif- ferent learning rates for CBCL-MIT Face dataset. 75 5.13 Comparisons of accuracies with traditional, MATLAB memristor crossbar and Xyce implementation for CBCL-MIT Face dataset. 76 xi LIST OF TABLES 4.1 2-input linearly separable logic functions. 45 4.2 Linearly separable 3-bit input logic functions using two different MOSFET in the driver circuit using 2 processor cores.