1 Neuromorphic Processing & Sensing: Evolutionary Progression of AI to Spiking Philippe Reiter, Geet Rose Jose, Spyridon Bizmpikis, Ionela-Ancuța Cîrjilă machine learning and deep learning (MLDL) developments. Abstract— The increasing rise in machine learning and deep This accelerated pace of innovation was spurred on by the learning applications is requiring ever more computational seminal works of LeCun, Hinton and others in the 1990s on resources to successfully meet the growing demands of an always- convolutional neural networks, or CNNs. Since then, numerous, connected, automated world. Neuromorphic technologies based on more advanced learning models have been developed, and Spiking Neural Network algorithms hold the promise to implement advanced artificial intelligence using a fraction of the neural networks have become integral to industry, medicine, computations and power requirements by modeling the academia and commercial devices. From fully autonomous functioning, and spiking, of the human brain. With the vehicles to rapid facial recognition entering popular culture to proliferation of tools and platforms aiding data scientists and enumerable innovations across almost all domains, it is not an machine learning engineers to develop the latest innovations in exaggeration to claim that CNNs and their progeny have artificial and deep neural networks, a transition to a new paradigm will require building from the current well-established changed both the technological and physical worlds. foundations. This paper explains the theoretical workings of CNNs can be exceedingly computationally intensive, neuromorphic technologies based on spikes, and overviews the however, as will be explained, often limiting their use to high- state-of-art in hardware processors, software platforms and performance computers and large data centres. As MLDL neuromorphic sensing devices. A progression path is paved for applications span a vast range of domains, they will be current machine learning specialists to update their skillset, as well increasingly required in lower-performing mobile, edge and as classification or predictive models from the current generation of deep neural networks to SNNs. This can be achieved by Internet-of-Things (IoT) devices. Spiking neural networks leveraging existing, specialised hardware in the form of present a biologically-plausible evolution to CNNs, and their SpiNNaker and the Nengo migration toolkit. First-hand, reduced-power and lower-latency characteristics enable their experimental results of converting a VGG-16 neural network to an deployment into even minimally-performing computing SNN are shared. A forward gaze into industrial, medical and systems. The necessary knowledge foundation, and a facilitated commercial applications that can readily benefit from SNNs wraps migration path for MLDL practitioners to transfer their up this investigation into the neuromorphic computing future. expertise to spiking models are presented in this paper. Index Terms—Artificial Neural Networks, Deep Learning, Machine Learning, Neuromorphic, Spiking Neural Networks II. EVOLUTION TO SPIKING NEURAL NETWORKS The rate of inventions and innovations is only accelerating. As computational capabilities and MLDL discoveries increase, I. INTRODUCTION so does the ability for neuroscience to better understand the S early as the middle of the last century, researchers have brain. A positive feedback loop is formed with ground-breaking A theorized computational models for mimicking the neuroscience research accelerating MLDL ever-forward, and operations of the human brain. These were the beginnings of vice-versa. neuromorphic or brain-like computing. A. Inspired by the Brain The pivotal turning point occurred in 1958, when, working out of the Cornell Aeronautical Laboratory, Frank Rosenblatt Complex deep neural networks (DNNs), which include invented what is considered one of the first-known learning CNNs, are increasingly effective at the exponential number of machines, the Perceptron. Born of this early binary tasks to which they are being applied. This has the side effect classification neural network (NN), advances in the field of of also requiring a similar staggering increase in heavily machine learning have ebbed and flowed through the years. energy- and space-consuming computing systems to power The last couple of decades have seen a more rapid rise in these DNN algorithms. As convolutional networks were inspired by the The article was submitted to the University of Strathclyde on Friday, April G. R. Jose is pursuing an MSc in Electronic and Electrical Engineering from 24, 2020. the Electronic and Electrical Engineering Department at the University of S. Bizmpikis is undertaking an MSc in Electronic and Electrical Engineering Strathclyde, Glasgow, UK, G1 1XQ (linkedin.com/in/geetrosejose). from the Department of Electronic and Electrical Engineering at the University P. Reiter is completing an MSc in Machine Learning and Deep Learning of Strathclyde, Glasgow, UK, G1 1XQ (linkedin.com/in/spyridon-bismpikis). with the Electronic and Electrical Engineering Department at the University of I. A. Cîrjilă is a MSc student with the Electronic and Electrical Engineering Strathclyde, Glasgow, UK, G1 1XQ (linkedin.com/in/philippereiter). Department at the University of Strathclyde, Glasgow, UK, G1 1XQ (linkedin.com/in/cirjilaanca). 2 understanding of the brain’s processing of information through autonomous systems, robotics and drones, vision and a number of layers, each extracting increasingly finer details, recognition sensors, and edge devices. With artificial the future of MLDL will again rest on leading-edge intelligence (AI) and MLDL becoming an intrinsic component neuroscience discoveries. of, not only the computing field, but the very fabric of the Comparing modern computers to the human brain, the latter modern world’s intelligence, security and social infrastructure, has evolved to be an exceedingly efficient, self-adapting and the shortlist of future applications presented comprises only a low-power sensing and learning system. Processing in the brain sampling of the advancements to be realised in the coming is highly distributed, parallel and sparse, with learning years. occurring at each synapse and with each neuron being To establish the groundwork for the rest of this paper’s connected to tens-of-thousands of others. Contrast this to the neuromorphic exploration, the following section overviews the ten-to-one or less fanout in a central processing unit (CPU) [1], neuroscientific and mathematical underpinnings that led to the which is often referred to the “brain” of a computer and which development of SNNs. centrally controls learning. Moreover, the brain captures information in an event-based manner, with the processing and III. FUNDAMENTALS OF SNNS storage of information being time-encoded. This temporal input As mentioned, the lightning advances in CNN research and and computing stimuli are known as “spikes,” and they are machine learning applications have come at the expense of fundamental to Spiking Neural Networks (SNN) and crucial for massive, expensive and power-hungry cloud server operations. developments in neuromorphic sensing and processing CNN algorithms require the highest in computational technologies. performance to quickly execute. This kind of performance is In addition to new algorithms, inventive new hardware not available on the devices that most leverage the benefits of architectures are required to both capture real-world physical MLDL networks – distributed mobile and edge smart devices. events as spikes, as well as drive these spikes through Revisiting the biological functioning of the brain, it was innovative architectures that excel at SNN computations. From determined that neurons quantise impulses as either “ON” or analog to mixed-signal and purely digital implementations, the “OFF” – not as the smooth, highly precise, fractional neuromorphic processors powering the SNN revolution are quantisation found in CNN weights. described in the State of Neuromorphic Hardware section, The modeling of these ON/OFF biological neural spikes has below. A special focus is given to the SpiNNaker led to numerous research institutions and large technology implementation, as it directly arises from using an ingenious companies around the world investing in Spiking Neural repurposing of existing technology. This theme of using Network (SNN) algorithms and the burgeoning field of current, widely-available technology as a springboard to future neuromorphic computing – that is, "brain-like" computing. neuromorphic developments will resonate throughout this The spiking properties of neuromorphic software and paper. hardware promise significant reductions in power consumption B. Path to SNN Adoption and processing delays, compared to current CNNs and DNNs. The human brain consumes on average 20W, and it is expected With thousands upon thousands of data scientists and that neuromorphic developments mirroring the brain's machine learning engineers proficient with today’s MLDL tools operation can achieve comparable levels of low-power and DNN algorithms, a path has to be paved to transition this performance. Similar to the brain, SNNs are massively parallel, expertise to also leverage low-power, low-latency and highly which further benefits rapid computations, especially that of parallel SNN models, where appropriate. A novel suite of new NN calculations. software is becoming available and is being developed to Methods