Analog Spiking Neuromorphic Circuits and Systems for Brain- and Nanotechnology-Inspired Cognitive Computing
Total Page:16
File Type:pdf, Size:1020Kb
ANALOG SPIKING NEUROMORPHIC CIRCUITS AND SYSTEMS FOR BRAIN- AND NANOTECHNOLOGY-INSPIRED COGNITIVE COMPUTING by Xinyu Wu A dissertation submitted in partial fulfillment of the requirements for the degree of Doctor of Philosophy in Electrical and Computer Engineering Boise State University December 2016 © 2016 Xinyu Wu ALL RIGHTS RESERVED BOISE STATE UNIVERSITY GRADUATE COLLEGE DEFENSE COMMITTEE AND FINAL READING APPROVALS of the dissertation submitted by Xinyu Wu Dissertation Title: Analog Spiking Neuromorphic Circuits and Systems for Brain- and Nanotechnology-Inspired Cognitive Computing Date of Oral Examination: 2 November 2016 The following individuals read and discussed the dissertation submitted by student Xinyu Wu, and they evaluated his presentation and response to questions during the final oral examination. They found that the student passed the oral examination. John Chiasson, Ph.D. Co-Chair, Supervisory Committee Vishal Saxena, Ph.D. Co-Chair, Supervisory Committee Hao Chen, Ph.D. Member, Supervisory Committee Hai Li, Ph.D. External Examiner The final reading approval of the dissertation was granted by Vishal Saxena, Ph.D., Co- Chair of the Supervisory Committee. The dissertation was approved by the Graduate College. ABSTRACT Human society is now facing grand challenges to satisfy the growing demand for computing power, at the same time, sustain energy consumption. By the end of CMOS technology scaling, innovations are required to tackle the challenges in a radically different way. Inspired by the emerging understanding of the computing occurring in a brain and nanotechnology-enabled biological plausible synaptic plasticity, neuromorphic computing architectures are being investigated. Such a neuromorphic chip that combines CMOS analog spiking neurons and nanoscale resistive random-access memory (RRAM) using as electronics synapses can provide massive neural network parallelism, high density and online learning capability, and hence, paves the path towards a promising solution to future energy-efficient real-time computing systems. However, existing silicon neuron approaches are designed to faithfully reproduce biological neuron dynamics, and hence they are incompatible with the RRAM synapses, or require extensive peripheral circuitry to modulate a synapse, and are thus deficient in learning capability. As a result, they eliminate most of the density advantages gained by the adoption of nanoscale devices, and fail to realize a functional computing system. This dissertation describes novel hardware architectures and neuron circuit designs that synergistically assemble the fundamental and significant elements for brain-inspired computing. Versatile CMOS spiking neurons that combine integrate-and-fire, passive iv dense RRAM synapses drive capability, dynamic biasing for adaptive power consumption, in situ spike-timing dependent plasticity (STDP) and competitive learning in compact integrated circuit modules are presented. Real-world pattern learning and recognition tasks using the proposed architecture were demonstrated with circuit-level simulations. A test chip was implemented and fabricated to verify the proposed CMOS neuron and hardware architecture, and the subsequent chip measurement results successfully proved the idea. The work described in this dissertation realizes a key building block for large-scale integration of spiking neural network hardware, and then, serves as a step-stone for the building of next-generation energy-efficient brain-inspired cognitive computing systems. v TABLE OF CONTENTS Abstract .............................................................................................................................. iv List of Tables ..................................................................................................................... xi List of Figures ................................................................................................................... xii List of Abbreviations ....................................................................................................... xxi List of Publication .......................................................................................................... xxiii Chapter 1 Introduction .........................................................................................................1 Grand Challenges and Rebooting Computing .........................................................3 Human Society Desires a Continued Growing Computing Capability .......3 New Ways Are Required to Tackle Unstructured Big Data ........................4 Unsustainable Energy for Sustainable Computing Capability Growth .......5 The End of Semiconductor Transistor Scaling ............................................6 Von Neumann Bottleneck ............................................................................8 Brain and Nanotechnology-Inspired Neuromorphic Computing ...........................10 This Dissertation ....................................................................................................13 Chapter 2 Brain Inspiration for Computing .......................................................................15 A Big Picture of Neuron Properties .......................................................................15 Neuron Morphology...................................................................................15 Neuron Electrical Properties ......................................................................17 vi Synapse ......................................................................................................19 Neuron Models.......................................................................................................22 McCulloch-Pitts Model ..............................................................................22 Leaky Integrate-and-Fire Model ................................................................23 Hodgkin and Huxley Model.......................................................................25 Izhikevich Model .......................................................................................26 Brain-Inspired Learning .........................................................................................27 Hebbian Learning.......................................................................................27 Spike Timing Dependent Plasticity ...........................................................29 Associative Learning .................................................................................31 Competitive Learning ................................................................................32 Brain-Inspired Architectures ..................................................................................35 Perceptron ..................................................................................................35 Multi-Level Perceptron ..............................................................................35 Recurrent Networks ...................................................................................37 Hierarchical Models and Deep Neural Networks ......................................38 Summary ................................................................................................................42 Chapter 3 Nanotechnology for Neuromorphic Computing ...............................................44 Overview of Emerging Memory Technologies .....................................................45 Phase Change Memory (PCM) ..............................................................................47 Spin-Transfer-Torque Random-Access-Memory (STT-RAM) .............................50 Resistive Random-Access-Memory (RRAM) .......................................................51 Resistance Switching Modes .....................................................................52 vii Switching Mechanisms and Operation ......................................................54 STDP in Bipolar Switching RRAMs .........................................................58 Device Characteristics for Neuromorphic Computing ..........................................64 Energy Efficiency ......................................................................................64 Device Dimensions ....................................................................................65 Resolution ..................................................................................................65 Retention and Endurance ...........................................................................66 Crossbar and 3D Integration ..................................................................................67 Summary ................................................................................................................70 Chapter 4 Introduction to Building Blocks of Analog Spiking Neurons ...........................71 Spatio-Temporal Integration ..................................................................................71 Passive Integrators .....................................................................................73 Opamp Integrators .....................................................................................76 Threshold and Firing Functionality .......................................................................80 Spike Shaping ........................................................................................................82 Spike-Frequency Adaptation and Adaptive Thresholds ........................................84