An Artificial Brain Mechanism to Develop a Paradigm for Robots Manish Kumar1, Rashmi Jha2 1Associate Professor, Department of Mechanical Engineering, University of Cincinnati, Cincinnati, Ohio [email protected], 513-556-5311 2Associate Professor, Dept. of Electrical Engineering and Computing Systems, University of Cincinnati, Cincinnati, Ohio [email protected], 513-556-1361 Need for Biological Brain Inspired Ultra-Low Energy Computing vs. Robot for Space Exploration

Humans hold a number of advantages over robots. They can make quick decisions in response to changing conditions or new discoveries, rather than waiting for time- delayed instructions from Earth.

http://www.wired.com/2012/04/space-humans-vs-robots/ Limitations of Currently Available Machine Learning: Deep Neural Network (DNN)

DNN on conventional computing architecture are compute intensive, power hungry, need a large set of training data , and are trained to solve just some specific sets of problems. Limitations of Traditional CMOS Transistor Scaling and Computing

Cost of CMOS transistor is rising at 20 nm node and beyond for the first time in history.

http://epc-co.com/epc/EventsandNews/FastJustGotFasterBlog/Issue11.aspx http://www.extremetech.com/computing/116561-the-death-of-cpu-scaling-from- one-core-to-many-and-why-were-still-stuck Motivation for Artificial Brain

“Challenge is to create an exascale Brain computing system Scalable by 2018 that consumes only 20 megawatts (MW) of Ultra Low- power.”- DOE grand Energy challenge.

“Taking advantage of the almost 83,000 “1014 , 1015 processors of one of the world's most Synapses, 1013 to 1016 powerful supercomputers, the team was able to Instructions per sec, 10 W mimic just one percent of one second's worth of Power (e.g. retinal of activity—and even that took 40 operation).” minutes.” – Gizmodo, 2013 Motivation for Artificial Brain

“Challenge is to create an exascale Supercomputers Brain computing system Scalable by 2018 that consumes only 20 megawatts (MW)• How of does a biological “Brain”Ultra Low work- ?? power.”- DOE grand Energy challenge. • How can we make an artificial brain on chip???

“Taking advantage of the almost 83,000 “1014 Neurons, 1015 processors of one of the world's most Synapses, 1013 to 1016 powerful supercomputers, the team was able to Instructions per sec, 10 W mimic just one percent of one second's worth of Power (e.g. retinal of human brain activity—and even that took 40 operation).” minutes.” – Gizmodo, 2013 How Does a Biological Brain Process Information?

Today’s Computing Brain-Inspired Paradigm of Computing

STDP Von-Neumann Architecture Components: Neurons, https://computing.llnl.gov/tutorials/parallel_comp/ Reconfigurable Synapses,

Interconnects Synaptic Efficacy Synaptic

Markram et. al., Front. Synp. NeurosSc. 2011 Operation and Action Potential Firing

Synapse Sensory Signal Processing

Temperature, odor etc.

(Effector Cells)

Central Nervous System What does it mean from neuro- inspired device perspective?

Neuron

Synapse • High fan-out spiking device • Reconfigurable • Ultra-low energy consumption ~10 fJ/spike • Ultra Low-power • Scalable • Scalable • High reliability and endurance • High endurance and reliability • Minimal Variability Sensory Information Coding Spike Coding of Odor

Mainland et. al., Trends in August 2014, Vol. 37, No. 8 Sensory Neurons in Silicon

Axon-Hillock Circuit, proposed by Prof. Carver Mead, 1980’s Indiveri et. al., Frontiers in , 2011 Learning • Supervised Learning – Feed-Forward – Back-Propagation – Gradient-Descent • Unsupervised Learning for Spiking Neural Network – Hebbian Learning (Spike Timing Dependent Plasticity) • Neurons that fire together, wire together • Basis of Associative Memory Spike Timing Dependent Plasticity

Which synapses are strengthened? Which ones are depressed? Bi et. al. J. of NeuroSci, 1998 Doped Oxide Dynamics for Synaptic Memory Positive Hysteresis 1.2 10-5 Cycle1 Cycle2 1 10-5 Cycle3 Cycle4 Cycle5 -6 Cycle6 8 10 Cycle7 Cycle8 Cycle9 -6 Cycle10 6 10 Cycle11 Cycle12 -6 Cycle13

Current Current (A) 4 10 Cycle14 Cycle15 Cycle16 2 10-6 Initial IV -9 0 10 0 0.5 1 1.5 2 2.5 3 Voltage (V) Negative Hysteresis 10-10 0 -2 10-8

-4 10-8 -11 Device 1 Cycle1 10 Cycle2 Device 2 -8 Device 3 -6 10 Cycle3 Cycle4 Device 4 -8 10-8 Cycle5 Current Current (A) Device 5

-12 Current (A) 10 Device 6 -7 Device 7 -1 10 Device 8 -7 Device 9 -1.2 10 Device 10 -13 100µm x 100 µm -1.4 10-7 10 -1.6 -1.4 -1.2 -1 -0.8 -0.6 -0.4 -0.2 0 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 Voltage (V) Voltage (V) 17 Mandal/Jha et. al., Nature Sci. Rep, 2014 Potentiation and Depression with Pulses

Conductance vs Pulse 1ms Conductance vs Pulse 10ms 4.5 6 Excitation (5Hz) Excitation (5Hz) 4 Depression (5Hz) Depression (5Hz) Excitation (15Hz) 5 Excitation (15Hz) Depression (15Hz) Depression (15Hz) 3.5 4 3

2.5 3

2 2 1.5 Current read (nA) @0.5V

Current @0.5V readCurrent (nA) @0.5V 1 1 0 100 200 300 400 500 600 700 0 100 200 300 400 500 600 700 Pulse # Pulse # 10 Potentiating Pulses 2.5V/1ms applied at given frequency and Current measured at 0.5V read after excitation. Potentiation was repeated 30 times (total 300 pulses) with conductance measuring intervals of 10 pulses. After potentiation for 300 pulses, 300 depression pulses at -1.5V/1ms were applied at the given frequency and measurement of current at 0.5V read was done in intervals of 10 pulses.

18 Endurance with Pulses of Different Pulse-Widths 40 1 ms Pot. 1 ms Dep. 35 10ms Pot. 10 ms Dep. 20 ms Pot. 20 ms Dep. 30 50 ms Pot. 50 ms Dep. 100 ms Pot. 25 100 ms Dep. 200 ms Pot. 200 ms Dep. 20

15

Current @ 0.5V read (nA) read 0.5V @ Current 10 0 200 400 600 800 1000 1200 Cycle #

19 Distribution over Cycle to Cycle 200 250  : 16.116 200 dep. Pot. (10 ms) Pot.1ms Dep. (10 ms) Pot. (20ms)  : 0.586 Dep. (20ms) Dep. 1ms dep.  : 16.394 200 150  : 20.3828 dep.  : 21.478  : 12.936 pot. 150 pot. dep.  : 1.55  : 0.8272 dep.  : 2.876  : 0.2416 pot. pot. 150 dep.  : 14.766 pot. 100 100  : 0.57 100 pot.

Count

Count

Count 50 50 50

0 0 0 10.8 12.8 14.8 16.8 18.8 20.8 12 14 17 19 22 24 12 14 16 18 20 22 24 Range (nA) Range (nA) Range (nA) 300 200 200 Pot. (50ms) Pot. (100 ms) Dep. (50ms) Dep. (100 ms) Pot. (200ms) Dep. (200 ms) 250  : 18.53  : 18.7035 dep. dep.  : 27.9565 150 pot. 150  : 0.356  : 25.303  : 0.982  : 19.7548 dep. pot. dep.  : 0.9163 dep. 200 pot.  : 0.896  : 1.493 pot. dep.  : 33.6524 150 100 100 pot.  : 1.3408 pot.

Count

Count 100 Count 50 50 50 0 0 0 14 16.8 19.6 22.4 25.2 28 12 16 20 24 28 32 16 20 25 30 35 40 Range (nA) Range (nA) Range (nA)

20 Spike Timing Dependent Plasticity

80 =1.3713 60

40 Δt (+/-) Feedback (2.5V/-1.5V) 20 (t)=83.03*exp(-t/41.3) 0 (t)=-47.03*exp(t/59.03) 10 ms 200 ms -20 %age %age change 20 ms 100 ms -40 =0.9934 30 ms 50 ms -60 -60 -40 -20 0 20 40 60 t (ms) 40 ms 20 ms 50 ms 10 ms

21 Synaptic Memory Device Model TiN/W TiN/ TiN/ W - Mn:HfO2 - W + Mn:HfO2 + + Mn:HfO2 Ru Ru - Ru Initial State Potentiation (LTP) Depression (LTD)

푞퐸 푞퐸 휙 − 휙 − 퐵 휋휀 푞퐸 퐵 휋휀 퐼 = 푞휇퐸퐴푛 exp − 휙 − 퐼 = 푞휇퐸퐴푛 exp − 0 푘푇 퐵 휋휀 2 퐼 = 푞휇퐸퐴푛 exp − 푘푇 1 푘푇

Mandal/Jha et. al., Nature Sci. Rep, 2014 Sarim/Kumar/Jha et. al., NAECON, 2016 Neuromorphic Platform

• A neuromorphic platform is configured as an array of several Synaptic Memory devices arranged as shown. • The arrays are connected to proximity sensors that send in the information about the vicinity of the robot. The motor neuron circuits move the robot wheels. Simulation Framework • Two different models, viz., mathematical device model and experimentally derived device model, for synaptic memory devices with the neuromorphic platform were implemented to demonstrate unsupervised learning in a robot. • This approach was validated by simulating the robot to navigate in an unknown environment with randomly placed obstacles. • The commercially available Khepera III robot [4] is modeled with a two-wheeled differential drive robot kinematics. The robot consists of five ultrasonic sensors that give the information about the vicinity of the robot.

[4] http://www.k-team.com/mobile-robotics-products/old-products/khepera-iii Robot Kinematics 푟 휈 = 휔 + 휔 2 푅 퐿 푟 휔 = 휔 − 휔 푏 푅 퐿

푥 = 휈 cos 휃 푦 = 휈 sin 휃 휃 = 휔

,Gianluca et. al., IEEE Transactions on 21.5 (2005): 994-1004. Learning Scheme

target Mathematical Model

Device Model:

change in structural parameter of the device 푤 = 푓 휈푀푅

휈푀푅 /휈0 휈푡ℎ/휈0 where 푓 휈푀푅 = 퐼0푠푖푔푛 휈푀푅 푒 − 푒

memristor voltage 휈푀푅 푡, Δ푇 = 훼푝표푠푠푝푘 푡 − 훼푝푟푒푠푝푘 푡 + Δ푇

푠푝푘 푡 is the spike shape

Δ푇 is the difference in the spike times of pre- and post-synaptic neurons Δ푤 Δ푇 = 푓 휈푀푅 푡, Δ푇 푑푡 = 휉 Δ푇

STDP learning function:

퐼 , 푣 are device parameters, 푣 is the 0 0 푡ℎ + threshold voltage of the device above which it 푎+푒−Δ푇 휏 푖푓 Δ푇 > 0 휉 Δ푇 = spikes, 훼 are attenuation parameters in pre- and − −Δ푇 휏− post-synaptic neurons. 휉 is the change in the −푎 푒 푖푓 Δ푇 < 0 synaptic weight that is used to implement STDP. Robot Navigation Results

1 2 3

4 5 6

o : start | × : target Navigation Results with Synaptic Memory Device Model TiN/W TiN/ TiN/ W - Mn:HfO2 - W + Mn:HfO2 + + Mn:HfO2 Ru Ru - Ru Initial State Potentiation (LTP) Depression (LTD)

푞퐸 푞퐸 휙 − 휙 − 퐵 휋휀 푞퐸 퐵 휋휀 퐼 = 푞휇퐸퐴푛 exp − 휙 − 퐼 = 푞휇퐸퐴푛 exp − 0 푘푇 퐵 휋휀 2 퐼 = 푞휇퐸퐴푛 exp − 푘푇 1 푘푇

Mandal/Jha et. al., Nature Sci. Rep, 2014 Sarim/Kumar/Jha et. al., NAECON, 2016 Robot Navigation Results

1 2 3

4 5 6

o : start | × : target Conclusions • We demonstrated the potential for having an onboard “artificial brain” for Robots based on emerging neuromorphic devices. • Using artificial brain architecture, a successful Robotic navigation was demonstrated using unsupervised learning scheme to guide the robot in complex environments using the local knowledge of obstacles only. – Our approach overcomes the issue of local minima which is a challenge for other navigation algorithms. • Our approach is projected to be highly energy-efficient and scalable for implementation on any robot. • Future work is targeted towards the actual implementation of these neuromorphic devices based artificial brain on Robots and field verification of the navigation. Student Contributors

1. Mohammad Sarim Robotics Lab, Department of Mechanical and Materials Engineering, University of Cincinnati [email protected]

2. Thomas Schultz EDACS Lab, Department of Electrical Engineering and Computing Systems , University of Cincinnati

3. Saptarshi Mandal (Now at Arizona State University) Acknowledgement • This project is currently supported by National Science Foundation under CAREER (Award # 1556294), and SaTC (Award # 1556301). • We would like to thank Dr. Mark Ritter and his group at IBM TJ Watson Research Center. • We would like to thank our collaborators Dr. Gennadi Bersuker (Sematech), Dr. David Gilmer (Sematech), Dr. Prashant Majhi (Intel), Dr. Kevin Leedy (AFRL), Dr. Marc Cahay (U. of Cincinnati), Dr. Ali Minai ( U. of Cincinnati), Dr. Swaroop Ghosh (USF), Dr. Scott Molitor (U.Toledo). Thank You! Questions and Suggestions?