DOCSLIB.ORG

Spiking Deep Neural Networks: Engineered and Biological Approaches to Object Recognition

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Spiking Deep Neural Networks: Engineered and Biological Approaches to Object Recognition Spiking Deep Neural Networks: Engineered and Biological Approaches to Object Recognition by Eric Hunsberger A thesis presented to the University of Waterloo in fulfillment of the thesis requirement for the degree of Doctor of Philosophy in Systems Design Engineering Waterloo, Ontario, Canada, 2017 c Eric Hunsberger 2017 Examining Committee Membership The following served on the Examining Committee for this thesis. The decision of the Examining Committee is by majority vote. External Examiner Bruno Olshausen Professor, Helen Wills Neuroscience Institute and School of Optometry, University of California, Berkeley Supervisor Chris Eliasmith Professor, Department of Philosophy and Department of Systems Design Engineering, University of Waterloo Supervisor Jeff Orchard Associate Professor, Cheriton School of Computer Science, University of Waterloo Internal Member Bryan Tripp Associate Professor, Department of Systems Design Engineering, University of Waterloo Internal Member Alexander Wong Associate Professor, Department of Systems Design Engineering, University of Waterloo Internal-External Member Steven Waslander Associate Professor, Department of Mechanical and Mechatronics Engineering, University of Waterloo ii I hereby declare that I am the sole author of this thesis. This is a true copy of the thesis, including any required final revisions, as accepted by my examiners. I understand that my thesis will be made electronically available to the public. iii Abstract Modern machine learning models are beginning to rival human performance on some re- alistic object recognition tasks, but we still lack a full understanding of how the human brain solves this same problem. This thesis combines knowledge from machine learning and computational neuroscience to create models of human object recognition that are increasingly realistic both in their treatment of low-level neural mechanisms and in their reproduction of high-level human behaviour. First, I present extensions to the Neural Engineering Framework to make its preferred type of model|the “fixed-encoding" network|more accurate for object recognition tasks. These extensions include better distributions|such as Gabor filters|for the encoding weights, and better loss functions|namely weighted squared loss, softmax loss, and hinge loss|to solve for decoding weights. Second, I introduce increased biological realism into deep convolutional neural networks trained with backpropagation, by training them to run using spiking leaky integrate-and- fire (LIF) neurons. Convolutional neural networks have been successful in machine learn- ing, and I am able to convert them to spiking networks while retaining similar levels of performance. I present a novel method to smooth the LIF rate response function in order to avoid the common problems associated with differentiating spiking neurons in general and LIF neurons in particular. I also derive a number of novel characterizations of spiking variability, and use these to train spiking networks to be more robust to this variability. Finally, to address the problems with implementing backpropagation in a biological system, I train spiking deep neural networks using the more biological Feedback Alignment algorithm. I examine this algorithm in depth, including many variations on the core algorithm, methods to train using non-differentiable spiking neurons, and some of the limitations of the algorithm. Using these findings, I construct a spiking model that learns online in a biologically realistic manner. The models developed in this thesis help to explain both how spiking neurons in the brain work together to allow us to recognize complex objects, and how the brain may learn this behaviour. Their spiking nature allows them to be implemented on highly efficient neuromorphic hardware, opening the door to object recognition on energy-limited devices such as cell phones and mobile robots. iv Acknowledgements First and foremost, I would like to thank my supervisors, Chris Eliasmith and Jeff Orchard. Their guidance, ideas, and keen questions have pushed me to look more deeply at some of the more pressing and interesting problems in the field. Chris has gone beyond his duties by being not only a great mentor, but also a friend. He has filled the CNRG lab at UW with amazing people, and I would like to thank everyone for her or his unique contribution to making the lab both a fertile ground for new and exciting ideas, and a fun and awesome place to be. I owe a special gratitude to those who first welcomed me to the lab, and who greatly influenced my early growth as a researcher and software developer: Trevor Bekolay, Terry Stewart, Dan Rasmussen, Travis DeWolf, Xuan Choo, and James Bergstra. More recently, thanks to Peter Duggins, Mariah Martin Shein, and Peter Blouw for the numerous musical interludes that helped me survive the last months of writing. Many of these past and present CNRG members also form the core development team for Nengo. Without this outstanding piece of software, I could not have done all the things I did in this thesis, so a special thanks to the whole Nengo team. I was also fortunate to have such a great committee to help me through the whole process. In particular, I would like to acknowledge Bryan Tripp, who has for years provided insightful comments, feedback, and ideas across many areas of my research. Finally, I could not have come this far without the continued support of my family and friends. Thank you for being genuinely interested in what I do, even when I am sure it sounds esoteric or downright dull. A special thanks to Sean Anderson, Matt McGill, Matt Wilson, and Nick Fischer, for over 75 combined years of being great friends to me. A very special thanks my family: Brian, Jocelyn, Alex, and Marg. And the specialest thanks to Ruth, who has not borne the burden with me as long as some of the others, but who has helped to carry so much more of it. v Dedication Half-way through this thesis project|which one may know is often the point when moti- vation begins to wane|I received the following fortune inside a cookie: If the brain were so simple we could understand it, we would be so simple we couldn't. Since then, I have kept this fortune on my desk to motivate me. I dedicate this thesis to all of us who work to prove this fortune cookie wrong. vi Table of Contents Examining Committee Membership ii Author's Declaration iii Abstract iv Acknowledgementsv Dedication vi Table of Contents vii List of Tables xiii List of Figures xiv 1 Introduction1 1.1 Outline......................................3 2 Computational Neuroscience5 2.1 Biology......................................6 2.1.1 Neuron physiology...........................7 2.1.2 Visual system anatomy......................... 10 vii 2.2 Neuron models................................. 12 2.2.1 The integrate-and-fire neuron model................. 14 2.2.2 The leaky integrate-and-fire (LIF) model............... 16 2.3 Synapse models................................. 18 2.4 Rate codes, population codes, and timing codes............... 19 2.5 The Neural Engineering Framework (NEF).................. 21 2.6 Neuromorphic hardware............................ 23 3 Machine Learning 24 3.1 Machine learning basics............................ 25 3.1.1 Goal................................... 26 3.1.2 Learning paradigms........................... 26 3.1.3 Basic components............................ 27 3.1.4 Overfitting and underfitting...................... 28 3.1.5 Dataset usage.............................. 30 3.1.6 Objective functions........................... 31 3.1.7 Datasets................................. 32 3.2 Backpropagation................................ 36 3.3 Stochastic gradient descent (SGD)....................... 39 3.3.1 Extensions................................ 40 3.3.2 Gradients and initialization...................... 42 3.4 Convolutional neural networks (CNNs).................... 43 3.4.1 Convolution operation......................... 43 3.4.2 Convolutional layer........................... 45 3.4.3 Nonlinearities.............................. 46 3.4.4 Pooling layer.............................. 48 3.4.5 Local response normalization layer.................. 50 3.4.6 Dropout layer.............................. 51 viii 4 Fixed-Encoding Networks 52 4.1 Background................................... 53 4.1.1 The Neural Engineering Framework.................. 53 4.1.2 Extreme Learning Machines...................... 54 4.2 Encoding methods............................... 56 4.2.1 Independent-element random encoders................ 57 4.2.2 Receptive fields............................. 57 4.2.3 Gabor filters............................... 57 4.2.4 Computed Input Weights (CIW)................... 58 4.2.5 Constrained Difference weights (CD)................. 60 4.3 Decoding methods............................... 61 4.3.1 Squared loss............................... 62 4.3.2 Weighted squared loss......................... 64 4.3.3 Softmax loss............................... 65 4.3.4 Hinge loss................................ 66 4.3.5 Weight norm regularization...................... 68 4.4 Spiking methods................................ 70 4.5 Results...................................... 72 4.5.1 Encoding................................ 72 4.5.2 Decoding................................ 75 4.5.3 Spiking................................. 77 4.5.4 Computation time........................... 82 4.6 Discussion.................................... 85 5 Spiking Deep Networks 89 5.1 Background..................................
Load more

Recommended publications
  • Spiking Neural Networks: Asurvey Oscar T
    Oscar T. Skaug Spiking Neural Networks: A Survey Bachelor’s project in Computer engineering Supervisor: Ole Christian Eidheim June 2020 Bachelor’s project NTNU Engineering Faculty of Information Technology and Electrical Faculty of Information Technology Norwegian University of Science and Technology Abstract The purpose of the survey is to support teaching and research in the department of Computer Science at the university, through fundamental research in recent advances in the machine learning field. The survey assumes the reader have some basic knowledge in machine learning, however, no prior knowledge on the topic of spiking neural networks is assumed. The aim of the survey is to introduce the reader to artificial spiking neural networks by going through the following: what constitutes and distinguish an artificial spiking neural network from other artificial neural networks; how to interpret the output signals of spiking neural networks; How supervised and unsupervised learning can be achieved in spiking neural networks; Some benefits of spiking neural networks compared to other artificial neural networks, followed by challenges specifically associated with spiking neural networks. By doing so the survey attempts to answer the question of why spiking neural networks have not been widely implemented, despite being both more computationally powerful and biologically plausible than other artificial neural networks currently available. Norwegian University of Science and Technology Trondheim, June 2020 Oscar T. Skaug The bachelor thesis The title of the bachelor thesis is “Study of recent advances in artificial and biological neuron models” with the purpose of “supporting teaching and research in the department of Computer Science at the university through fundamental research in recent advances in the machine learning field”.
    [Show full text]
  • Neural Network Transformation and Co-Design Under Neuromorphic Hardware Constraints
    NEUTRAMS: Neural Network Transformation and Co-design under Neuromorphic Hardware Constraints Yu Ji ∗, YouHui Zhang∗‡, ShuangChen Li†,PingChi†, CiHang Jiang∗,PengQu∗,YuanXie†, WenGuang Chen∗‡ Email: [email protected], [email protected] ∗Department of Computer Science and Technology, Tsinghua University, PR.China † Department of Electrical and Computer Engineering, University of California at Santa Barbara, USA ‡ Center for Brain-Inspired Computing Research, Tsinghua University, PR.China Abstract—With the recent reincarnations of neuromorphic (CMP)1 with a Network-on-Chip (NoC) has emerged as a computing comes the promise of a new computing paradigm, promising platform for neural network simulation, motivating with a focus on the design and fabrication of neuromorphic chips. many recent studies of neuromorphic chips [4]–[10]. The A key challenge in design, however, is that programming such chips is difficult. This paper proposes a systematic methodology design of neuromorphic chips is promising to enable a new with a set of tools to address this challenge. The proposed computing paradigm. toolset is called NEUTRAMS (Neural network Transformation, One major issue is that neuromorphic hardware usually Mapping and Simulation), and includes three key components: places constraints on NN models (such as the maximum fan- a neural network (NN) transformation algorithm, a configurable in/fan-out of a single neuron, the limited range of synaptic clock-driven simulator of neuromorphic chips and an optimized runtime tool that maps NNs onto the target hardware for weights) that it can support, and the hardware types of neurons better resource utilization. To address the challenges of hardware or activation functions are usually simpler than the software constraints on implementing NN models (such as the maximum counterparts.
    [Show full text]
  • Spiking Neural Networks: Neuron Models, Plasticity, and Graph Applications
    Virginia Commonwealth University VCU Scholars Compass Theses and Dissertations Graduate School 2015 Spiking Neural Networks: Neuron Models, Plasticity, and Graph Applications Shaun Donachy Virginia Commonwealth University Follow this and additional works at: https://scholarscompass.vcu.edu/etd Part of the Theory and Algorithms Commons © The Author Downloaded from https://scholarscompass.vcu.edu/etd/3984 This Thesis is brought to you for free and open access by the Graduate School at VCU Scholars Compass. It has been accepted for inclusion in Theses and Dissertations by an authorized administrator of VCU Scholars Compass. For more information, please contact [email protected]. c Shaun Donachy, July 2015 All Rights Reserved. SPIKING NEURAL NETWORKS: NEURON MODELS, PLASTICITY, AND GRAPH APPLICATIONS A Thesis submitted in partial fulfillment of the requirements for the degree of Master of Science at Virginia Commonwealth University. by SHAUN DONACHY B.S. Computer Science, Virginia Commonwealth University, May 2013 Director: Krzysztof J. Cios, Professor and Chair, Department of Computer Science Virginia Commonwealth University Richmond, Virginia July, 2015 TABLE OF CONTENTS Chapter Page Table of Contents :::::::::::::::::::::::::::::::: i List of Figures :::::::::::::::::::::::::::::::::: ii Abstract ::::::::::::::::::::::::::::::::::::: v 1 Introduction ::::::::::::::::::::::::::::::::: 1 2 Models of a Single Neuron ::::::::::::::::::::::::: 3 2.1 McCulloch-Pitts Model . 3 2.2 Hodgkin-Huxley Model . 4 2.3 Integrate and Fire Model . 6 2.4 Izhikevich Model . 8 3 Neural Coding Techniques :::::::::::::::::::::::::: 14 3.1 Input Encoding . 14 3.1.1 Grandmother Cell and Distributed Representations . 14 3.1.2 Rate Coding . 15 3.1.3 Sine Wave Encoding . 15 3.1.4 Spike Density Encoding . 15 3.1.5 Temporal Encoding . 16 3.1.6 Synaptic Propagation Delay Encoding .
    [Show full text]
  • Deep Learning with Tensorflow 2 and Keras Second Edition
    Deep Learning with TensorFlow 2 and Keras Second Edition Regression, ConvNets, GANs, RNNs, NLP, and more with TensorFlow 2 and the Keras API Antonio Gulli Amita Kapoor Sujit Pal BIRMINGHAM - MUMBAI Deep Learning with TensorFlow 2 and Keras Second Edition Copyright © 2019 Packt Publishing All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews. Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the authors, nor Packt Publishing or its dealers and distributors, will be held liable for any damages caused or alleged to have been caused directly or indirectly by this book. Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information. Commissioning Editor: Amey Varangaonkar Acquisition Editors: Yogesh Deokar, Ben Renow-Clarke Acquisition Editor – Peer Reviews: Suresh Jain Content Development Editor: Ian Hough Technical Editor: Gaurav Gavas Project Editor: Janice Gonsalves Proofreader: Safs Editing Indexer: Rekha Nair Presentation Designer: Sandip Tadge First published: April 2017 Second edition: December 2019 Production reference: 2130320 Published by Packt Publishing Ltd. Livery Place 35 Livery Street Birmingham B3 2PB, UK. ISBN 978-1-83882-341-2 www.packt.com packt.com Subscribe to our online digital library for full access to over 7,000 books and videos, as well as industry leading tools to help you plan your personal development and advance your career.
    [Show full text]
  • A Sparsity-Driven Backpropagation-Less Learning Framework Using Populations of Spiking Growth Transform Neurons
    ORIGINAL RESEARCH published: 28 July 2021 doi: 10.3389/fnins.2021.715451 A Sparsity-Driven Backpropagation-Less Learning Framework Using Populations of Spiking Growth Transform Neurons Ahana Gangopadhyay and Shantanu Chakrabartty* Department of Electrical and Systems Engineering, Washington University in St. Louis, St. Louis, MO, United States Growth-transform (GT) neurons and their population models allow for independent control over the spiking statistics and the transient population dynamics while optimizing a physically plausible distributed energy functional involving continuous-valued neural variables. In this paper we describe a backpropagation-less learning approach to train a Edited by: network of spiking GT neurons by enforcing sparsity constraints on the overall network Siddharth Joshi, spiking activity. The key features of the model and the proposed learning framework University of Notre Dame, United States are: (a) spike responses are generated as a result of constraint violation and hence Reviewed by: can be viewed as Lagrangian parameters; (b) the optimal parameters for a given task Wenrui Zhang, can be learned using neurally relevant local learning rules and in an online manner; University of California, Santa Barbara, United States (c) the network optimizes itself to encode the solution with as few spikes as possible Youhui Zhang, (sparsity); (d) the network optimizes itself to operate at a solution with the maximum Tsinghua University, China dynamic range and away from saturation; and (e) the framework is flexible enough to *Correspondence: incorporate additional structural and connectivity constraints on the network. As a result, Shantanu Chakrabartty [email protected] the proposed formulation is attractive for designing neuromorphic tinyML systems that are constrained in energy, resources, and network structure.
    [Show full text]
  • Simplified Spiking Neural Network Architecture and STDP Learning
    Iakymchuk et al. EURASIP Journal on Image and Video Processing (2015) 2015:4 DOI 10.1186/s13640-015-0059-4 RESEARCH Open Access Simplified spiking neural network architecture and STDP learning algorithm applied to image classification Taras Iakymchuk*, Alfredo Rosado-Muñoz, Juan F Guerrero-Martínez, Manuel Bataller-Mompeán and Jose V Francés-Víllora Abstract Spiking neural networks (SNN) have gained popularity in embedded applications such as robotics and computer vision. The main advantages of SNN are the temporal plasticity, ease of use in neural interface circuits and reduced computation complexity. SNN have been successfully used for image classification. They provide a model for the mammalian visual cortex, image segmentation and pattern recognition. Different spiking neuron mathematical models exist, but their computational complexity makes them ill-suited for hardware implementation. In this paper, a novel, simplified and computationally efficient model of spike response model (SRM) neuron with spike-time dependent plasticity (STDP) learning is presented. Frequency spike coding based on receptive fields is used for data representation; images are encoded by the network and processed in a similar manner as the primary layers in visual cortex. The network output can be used as a primary feature extractor for further refined recognition or as a simple object classifier. Results show that the model can successfully learn and classify black and white images with added noise or partially obscured samples with up to ×20 computing speed-up at an equivalent classification ratio when compared to classic SRM neuron membrane models. The proposed solution combines spike encoding, network topology, neuron membrane model and STDP learning.
    [Show full text]
  • Multilayer Spiking Neural Network for Audio Samples Classification Using
    Multilayer Spiking Neural Network for Audio Samples Classification Using SpiNNaker Juan Pedro Dominguez-Morales, Angel Jimenez-Fernandez, Antonio Rios-Navarro, Elena Cerezuela-Escudero, Daniel Gutierrez-Galan, Manuel J. Dominguez-Morales, and Gabriel Jimenez-Moreno Robotic and Technology of Computers Lab, Department of Architecture and Technology of Computers, University of Seville, Seville, Spain {jpdominguez,ajimenez,arios,ecerezuela, dgutierrez,mdominguez,gaji}@atc.us.es http://www.atc.us.es Abstract. Audio classification has always been an interesting subject of research inside the neuromorphic engineering field. Tools like Nengo or Brian, and hard‐ ware platforms like the SpiNNaker board are rapidly increasing in popularity in the neuromorphic community due to the ease of modelling spiking neural networks with them. In this manuscript a multilayer spiking neural network for audio samples classification using SpiNNaker is presented. The network consists of different leaky integrate-and-fire neuron layers. The connections between them are trained using novel firing rate based algorithms and tested using sets of pure tones with frequencies that range from 130.813 to 1396.91 Hz. The hit rate percentage values are obtained after adding a random noise signal to the original pure tone signal. The results show very good classification results (above 85 % hit rate) for each class when the Signal-to-noise ratio is above 3 decibels, vali‐ dating the robustness of the network configuration and the training step. Keywords: SpiNNaker · Spiking neural network · Audio samples classification · Spikes · Neuromorphic auditory sensor · Address-Event Representation 1 Introduction Neuromorphic engineering is a discipline that studies, designs and implements hardware and software with the aim of mimicking the way in which nervous systems work, focusing its main inspiration on how the brain solves complex problems easily.
    [Show full text]
  • Spiking Neural Network Vs Multilayer Perceptron: Who Is the Winner in the Racing Car Computer Game
    Soft Comput (2015) 19:3465–3478 DOI 10.1007/s00500-014-1515-2 FOCUS Spiking neural network vs multilayer perceptron: who is the winner in the racing car computer game Urszula Markowska-Kaczmar · Mateusz Koldowski Published online: 3 December 2014 © The Author(s) 2014. This article is published with open access at Springerlink.com Abstract The paper presents two neural based controllers controller on-line. The RBF network is used to establish a for the computer car racing game. The controllers represent nonlinear prediction model. In the recent paper (Zheng et al. two generations of neural networks—a multilayer perceptron 2013) the impact of heavy vehicles on lane-changing deci- and a spiking neural network. They are trained by an evolu- sions of following car drivers has been investigated, and the tionary algorithm. Efficiency of both approaches is experi- lane-changing model based on two neural network models is mentally tested and statistically analyzed. proposed. Computer games can be perceived as a virtual representa- Keywords Computer game controller · Spiking neural tion of real world problems, and they provide rich test beds network · Multilayer perceptron · Evolutionary algorithm for many artificial intelligence methods. An example which can be mentioned here is successful application of multi- layer perceptron in racing car computer games, described for 1 Introduction instance in Ebner and Tiede (2009) and Togelius and Lucas (2005). Inspiration for our research were the results shown in Neural networks (NN) are widely applied in many areas Yee and Teo (2013) presenting spiking neural network based because of their ability to learn. controller for racing car in computer game.
    [Show full text]
  • Neuromorphic Computing Using Emerging Synaptic Devices: a Retrospective Summary and an Outlook
    electronics Review Neuromorphic Computing Using Emerging Synaptic Devices: A Retrospective Summary and an Outlook Jaeyoung Park 1,2 1 School of Computer Science and Electronic Engineering, Handong Global University, Pohang 37554, Korea; [email protected]; Tel.: +82-54-260-1394 2 School of Electronic Engineering, Soongsil University, Seoul 06978, Korea Received: 29 June 2020; Accepted: 5 August 2020; Published: 1 September 2020 Abstract: In this paper, emerging memory devices are investigated for a promising synaptic device of neuromorphic computing. Because the neuromorphic computing hardware requires high memory density, fast speed, and low power as well as a unique characteristic that simulates the function of learning by imitating the process of the human brain, memristor devices are considered as a promising candidate because of their desirable characteristic. Among them, Phase-change RAM (PRAM) Resistive RAM (ReRAM), Magnetic RAM (MRAM), and Atomic Switch Network (ASN) are selected to review. Even if the memristor devices show such characteristics, the inherent error by their physical properties needs to be resolved. This paper suggests adopting an approximate computing approach to deal with the error without degrading the advantages of emerging memory devices. Keywords: neuromorphic computing; memristor; PRAM; ReRAM; MRAM; ASN; synaptic device 1. Introduction Artificial Intelligence (AI), called machine intelligence, has been researched over the decades, and the AI now becomes an essential part of the industry [1–3]. Artificial intelligence means machines that imitate the cognitive functions of humans such as learning and problem solving [1,3]. Similarly, neuromorphic computing has been researched that imitates the process of the human brain compute and store [4–6].
    [Show full text]
  • Neuromorphic Processing and Sensing
    1 Neuromorphic Processing & Sensing: Evolutionary Progression of AI to Spiking Philippe Reiter, Geet Rose Jose, Spyridon Bizmpikis, Ionela-Ancuța Cîrjilă machine learning and deep learning (MLDL) developments. Abstract— The increasing rise in machine learning and deep This accelerated pace of innovation was spurred on by the learning applications is requiring ever more computational seminal works of LeCun, Hinton and others in the 1990s on resources to successfully meet the growing demands of an always- convolutional neural networks, or CNNs. Since then, numerous, connected, automated world. Neuromorphic technologies based on more advanced learning models have been developed, and Spiking Neural Network algorithms hold the promise to implement advanced artificial intelligence using a fraction of the neural networks have become integral to industry, medicine, computations and power requirements by modeling the academia and commercial devices. From fully autonomous functioning, and spiking, of the human brain. With the vehicles to rapid facial recognition entering popular culture to proliferation of tools and platforms aiding data scientists and enumerable innovations across almost all domains, it is not an machine learning engineers to develop the latest innovations in exaggeration to claim that CNNs and their progeny have artificial and deep neural networks, a transition to a new paradigm will require building from the current well-established changed both the technological and physical worlds. foundations. This paper explains the theoretical workings of CNNs can be exceedingly computationally intensive, neuromorphic technologies based on spikes, and overviews the however, as will be explained, often limiting their use to high- state-of-art in hardware processors, software platforms and performance computers and large data centres.
    [Show full text]
  • Neuromorphic Computing in Ginzburg-Landau Lattice Systems
    Neuromorphic computing in Ginzburg-Landau polariton lattice systems Andrzej Opala and Michał Matuszewski Institute of Physics, Polish Academy of Sciences, Warsaw Sanjib Ghosh and Timothy C. H. Liew Division of Physics and Applied Physics, Nanyang Technological University, Singapore < Andrzej Opala and Michał Matuszewski Institute of Physics, Polish Academy of Sciences, Warsaw Sanjib Ghosh and Timothy C. H. Liew Division of Physics and Applied Physics, Nanyang Technological University, Singapore < Machine learning with neural networks What is machine learning? Modern computers are extremely efficient in solving many problems, but some tasks are very hard to implement. ● It is difficult to write an algorithm that would recognize an object from different viewpoints or in di erent sceneries, lighting conditions etc. ● It is difficult to detect a fradulent credit card transaction. !e need to combine a large number of weak rules with complex dependencies rather than follow simple and reliable rules. "hese tasks are often relatively easy for humans. Machine learning with neural networks Machine learning algorithm is not written to solve a speci#c task, but to learn how to classify / detect / predict on its own. ● A large number of examples is collected and used as an input for the algorithm during the teaching phase. ● "he algorithm produces a program &eg. a neural network with #ne tuned weights), which is able to do the job not only on the provided samples, but also on data that it has never seen before. It usually contains a great number of parameters. Machine learning with neural networks "hree main types of machine learning: 1.Supervised learning – direct feedback, input data is provided with clearly de#ned labels.
    [Show full text]
  • Spinnaker Project
    10/21/19 The SpiNNaker project Steve Furber ICL Professor of Computer Engineering The University of Manchester 1 200 years ago… • Ada Lovelace, b. 10 Dec. 1815 "I have my hopes, and very distinct ones too, of one day getting cerebral phenomena such that I can put them into mathematical equations--in short, a law or laws for the mutual actions of the molecules of brain. …. I hope to bequeath to the generations a calculus of the nervous system.” 2 1 10/21/19 70 years ago… 3 Bio-inspiration • Can massively-parallel computing resources accelerate our understanding of brain function? • Can our growing understanding of brain function point the way to more efficient parallel, fault-tolerant computation? 4 2 10/21/19 ConvNets - structure • Dense convolution kernels • Abstract neurons • Only feed-forward connections • Trained through backpropagation 5 The cortex - structure Feedback input Feed-forward output Feed-forward input Feedback output • Spiking neurons • Two-dimensional structure • Sparse connectivity 6 3 10/21/19 ConvNets - GPUs • Dense matrix multiplications • 3.2kW • Low precision 7 Cortical models - Supercomputers • Sparse matrix operations • Efficient communication of spikes • 2.3MW 8 4 10/21/19 Cortical models - Neuromorphic hardware • Memory local to computation • Low-power • Real time • 62mW 9 Start-ups and industry interest 5 10/21/19 SpiNNaker project • A million mobile phone processors in one computer • Able to model about 1% of the human brain… • …or 10 mice! 11 SpiNNaker system 12 6 10/21/19 SpiNNaker chip Multi-chip packaging
    [Show full text]