Neuromorphic chips for the Artificial Brain Jaeseung Jeong, Ph.D Program of Brain and Cognitive Engineering, KAIST Silicon-based artificial intelligence is not efficient Prediction, expectation, and error Artificial Information processor vs. Biological information processor Core 2 Duo Brain • 65 watts • 10 watts • 291 million transistors • 100 billion neurons • >200nW/transistor • ~100pW/neuron Comparison of scales Molecules Channels Synapses Neurons CNS 0.1nm 10nm 1mm 0.1mm 1cm 1m Silicon Transistors Logic Multipliers PIII Parallel Gates Processors Motivation and Objective of neuromorphic engineering Problem • As compared to biological systems, today’s intelligent machines are less efficient by a factor of a million to a billion in complex environments. • For intelligent machines to be useful, they must compete with biological systems. Human Cortex Computer Simulation for Cerebral Cortex 20 Watts 1010 Watts 10 Objective I.4 Liter 4x 10 Liters • Develop electronic, neuromorphic machine technology that scales to biological level for efficient artificial intelligence. 9 Why Neuromorphic Engineering? Interest in exploring Interest in building neuroscience neurally inspired systems Key Advantages • The system is dynamic: adaptation • What if our primitive gates were a neuron computation? a synapse computation? a piece of dendritic cable? • Efficient implementations compute in their memory elements – more efficient than directly reading all the coefficients. Biology and Silicon Devices Similar physics of biological channels and p-n junctions • Drift and Diffusion equations form a built-in Barrier (Nernst equation for cable theory) • Exponential distribution of particles (Ions in biology and electrons/holes in silicon) Both biological channels and transistors have a gating mechanism that modulates a channel. Inspiration Biological-Scale Neuromorphic Electronic Devices Human NeoCortex Neuromorphic Electronics 1010 intersection/cm2 in crossbar ~1010 synapses/cm2 arrays w/ 100 nm pitch ~5x108 transistors/cm2 in state of the ~106 Neurons/cm2 art CMOS ~5 x 108 long range axons ~30 Gbit/sec multiplexed digital @ ~1 Hz addressing Conclusion: Gross statistics of biological neural systems might be realized in modern electronics. Approved for Public Release, Distribution 14 Unlimited Why Analog? • Much lower power than digital • Can perform many computations faster and more efficiently than digital in a more complicated way. • Follows the same physical laws as biological systems Analog: Power Savings Gene's Law 100W DSP Power CADSP Power 1W Gene’s Law • Power consumption of integrated circuits 10mW decreases exponentially over time Programmable • Follows Moore’s Law Analog Power 0.1mW Savings • Analog computation yields tremendous power savings equal to a >20 year leap in >20 Year Leap 1mW in Technology technology Power Dissipated / MMAC 10nW 1980 1990 2000 2010 2020 2030 Year e.g., Analog Cochlear Model • 32 subbands at 44.1kHz • Analog consumes <5μW • cf. Digital CM consumes ~5mW (audio-streamlined DSP) • Analog power savings of >1000 times Bio-Inspired Systems using analog - digital Smart Embedded Low-Power Analog Sensors •Consumer Electronics •Implantable Devices •Hearing Aids •Subthreshold Design •Cochlear Implants Analog Programmability Powerful Mixed-Signal Systems Provides digital features to the analog domain •Programmability •Accuracy •Reconfigurability •“Silicon Simulation” Analog alleviates the burden of the digital Synapse: The memristor (memory resistor) • The memristor (memory resistor) is a missing non-linear passive two- terminal electrical component relating electric charge and magnetic flux linkage (Circuit theorist Leon Chua, 1971). • The memristor would hypothetically operate in the following way: The memristor's electrical resistance is not constant but depends on the history of current that had previously flowed through the device, i.e., its present resistance depends on how much electric charge has flowed in what direction through it in the past. • The device remembers its history—the so-called non-volatility property. When the electric power supply is turned off, the memristor remembers its most recent resistance until it is turned on again. This is a basic element for neuromorphic chips. Neuromorphic engineering • A key aspect of neuromorphic engineering is understanding how the morphology of individual neurons, circuits, applications, and overall architectures creates desirable computations, affects how information is represented, influences robustness to damage, incorporates learning and development, adapts to local change (plasticity), and facilitates evolutionary change. • In recent times the term neuromorphic has been used to describe analog, digital, and mixed-mode analog/digital VLSI and software systems that implement models of neural systems (for perception, motor control, or multisensory integration). • The implementation of neuromorphic computing on the hardware level can be realized by oxide-based memristors, threshold switches and transistors. Neuromorphic/Bio-Mimetic Engineering Neuromorphic/Bio-Mimetic Engineering – Using biology to inspire better engineering • High-quality processing • Low power consumption Sensorimotor Systems •Intelligent robotics •Intelligent controls •Locomotive systems Neurons Silicon Retina •Systems that learn •CMOS imagers •Systems that adapt •Intelligent imagers •Neural networks •Retinal implants •Understanding biology Electronic Nose Audio Systems •“Sniff out” odors •Audio front ends •Chemical sensors •Signal processing systems •Drug traffic control •Hearing aids •Bio terror detection •Cochlear implants Analog VLSI • Carver Mead and his students pioneered the development aVLSI technology for use in neural circuits • They developed a silicon retina which electronically emulated the first 3 layers of the retina Very-large-scale integration (VLSI) • Very-large-scale integration (VLSI) is the process of creating an integrated circuit (IC) by combining thousands of transistors into a single chip. VLSI began in the 1970s when complex semiconductor and communication technologies were being developed. • The microprocessor is a VLSI device. Before the introduction of VLSI technology, most ICs had a limited set of functions they could perform. • An electronic circuit might consist of a CPU, ROM, RAM and other logic gates. VLSI lets IC designers add all of these into one chip. Why VLSI? • Cheaper (and easier to mass produce) • Smaller • Reduces power • Keeps everything contained • Reduces noise • Reduces coupling from the environment • Need a large number of transistors to perform real-world computations/tasks • Allows a high density or circuit elements (therefore, VLSI reduces costs) Artificial Neural Network Chips • Early neuromorphic architectures were artificial neural network chips • Examples: • ETANN : (1989) Entirely analog chip that was designed for feed forward artificial neural network operation. • Ni1000 : (1996) Significantly more powerful than ETANN, however has narrower functionality SYNAPSE-1 System Architecture SYNAPSE-1 is a modular system arranged as a 2D array of MA16s, weight memories, data units, and a control unit • As early as 2006, researchers at Georgia Tech published a field programmable neural array. This chip was the first in a line of many increasingly complex arrays of floating gate transistors that allows programmability of charge on the gates of MOSFETs to model the channel-ion characteristics of neurons in the brain and was one of the first cases of a silicon programmable array of neurons. • In November 2011, a group of MIT researchers created a computer chip that mimics the analog, ion-based communication in a synapse between two neurons using 400 transistors and standard CMOS manufacturing techniques. • In June 2012, Spintronic Researchers at Purdue presented a paper on design for a neuromorphic chip using lateral spin valves and memristors. • They argue that the architecture they have designed works in a similar way to neurons and can therefore be used to test various ways of reproducing the brain’s processing ability. In addition, they are significantly more energy efficient than conventional chips. Neurogrid • (2005) Neurogrid is a multi-chip system developed by Kwabena Boahen and his group at Stanford University • Objective is to emulate neurons • Composed of a 4x4 array of Neurocores • Each Neurocore contains a 256x256 array of neuron circuits with up to 6,000 synapse connections • Mead succeeded in mimicking ion-flow across a neuron's membrane with electron-flow through a transistor's channel—this should not have come as a surprise—the same physical forces are at work in both cases! • Stanford researchers made these chips programmable by coming up with a versatile ion-channel analog and reconfigurable synaptic connections. This tiny chip—packaged in black plastic and mounted on a printed circuit board—models 1,024 excitatory pyramidal cells and 256 inhibitory basket cells. Their cellular properties and synaptic organization are downloaded to the chip over a USB link, which also allows their activity to be visualized in real-time. The FACETS Project • (2005) Fast Analog Computing with Emergent Transient States (FACETS) • A project designed by an international collective of scientists and engineers funded by the European Union • Recently developed a chip containing 200,000 neuron circuits connected by 50 million synapses. Large Scale Simulations IBM: Blue Brain Project: IBM & EPFL (Switzerland) IBM Almaden Research Center Los Alamos National