MANAGING A REAL-TIME MASSIVELY-PARALLEL NEURAL ARCHITECTURE A THESIS SUBMITTED TO THE UNIVERSITY OF MANCHESTER FOR THE DEGREE OF DOCTOR OF PHILOSOPHY IN THE FACULTY OF ENGINEERING AND PHYSICAL SCIENCES 2012 By James Cameron Patterson School of Computer Science Contents Abstract 11 Declaration 12 Copyright 13 Acknowledgements 14 The Author 15 1 Introduction 16 1.1 Research Motivation .......................... 17 1.1.1 Not Chips,but Cores with Everything ............. 17 1.1.2 Large System Management ................... 20 1.1.3 Neural Networks on Parallel Computers ............ 20 1.1.4 Monitoring the SpiNNaker Neural System ........... 21 1.2 Contributions .............................. 23 1.3 Publications ............................... 24 1.4 Thesis Overview ............................ 26 2 Neural Computing 29 2.1 Biology ................................. 30 2.2 Neural Network Modelling ....................... 33 2.2.1 First Generation Artificial Neural Networks (ANNs) ..... 34 2.2.2 Second Generation ANNs ................... 34 2.2.3 Third Generation ANNs .................... 36 2.3 Systems for Neural Network Modelling ................ 39 2.3.1 General Purpose Hardware ................... 40 2.3.2 Specialised Hardware Platforms ................ 45 2 2.3.3 The Centre Ground ....................... 48 3 The SpiNNaker Neural Architecture 50 3.1 Architectural Requirements ....................... 51 3.2 System Architecture ........................... 51 3.3 SpiNNaker Systems ........................... 53 3.4 SpiNNaker Communications ...................... 55 3.4.1 Operation ............................ 55 3.4.2 SpiNNaker NoCs and the Router ................ 56 3.4.3 Packet Formats ......................... 57 3.4.4 Layered Networking ...................... 60 3.4.5 Ex-System, Ethernet ...................... 61 3.4.6 Internet Encapsulation ..................... 62 3.4.7 SpiNNaker Datagram Protocol (SDP) ............. 62 3.5 Software on SpiNNaker ......................... 63 3.6 Summary ................................ 66 4 Bootstrapping SpiNNaker 67 4.1 Node-Boot ................................ 68 4.1.1 First Steps ........................... 71 4.1.2 Monitor Processor Arbitration ................. 72 4.1.3 Chip-Level POST and Initialisation .............. 73 4.1.4 Processor-Level Initialisation .................. 76 4.1.5 TheMainLoop ......................... 77 4.2 Loading the System-Boot Image .................... 77 4.2.1 Ethernet Flood-Fill ....................... 78 4.2.2 Inter-Chip Flood-Fill ...................... 79 4.3 IVB – the ITCM Validation Block ................... 82 4.4 DHCP Node-Boot Image ........................ 83 4.5 Results from Node-Boot Operation ................... 87 4.5.1 Fault Detection and Isolation .................. 87 4.5.2 Response Time and Data Rate ................. 87 4.5.3 Time taken to Flood-Fill a System-Boot Image ........ 91 4.6 Application Loading .......................... 93 4.7 Summary and Contributions ...................... 93 3 5 Imaging Neural Networks 96 5.1 ‘Wetware’ Monitoring ......................... 96 5.1.1 In-Vivo Structural Imaging ................... 98 5.1.2 In-Vivo Functional Activity Imaging .............. 100 5.1.3 Single-Unit Monitoring ..................... 104 5.1.4 Biological Imaging Software .................. 105 5.2 Artificial Neural Network Monitoring ................. 106 5.2.1 Non-Spiking Neural Networks ................. 107 5.2.2 Spiking Neural Networks .................... 110 5.3 Summary ................................ 118 6 Visualising Neural Networks on SpiNNaker 119 6.1 Principles of Visualisation ....................... 120 6.2 It’s All Just Data ............................ 121 6.3 Visualisation Targets .......................... 121 6.4 Implementing SpiNNaker’s Visualiser ................. 124 6.4.1 Execution Environment ..................... 124 6.4.2 Packet Decode ......................... 124 6.4.3 Visualisation Interface ..................... 127 6.4.4 Mapping ............................ 129 6.4.5 SpiNNaker Neural Software and Accessing its Data ...... 130 6.5 Results .................................. 131 6.5.1 Neural Dynamics ........................ 131 6.5.2 Spike Activity ......................... 133 6.5.3 Aggregated Information .................... 134 6.5.4 Interaction ........................... 135 6.5.5 V1 Modelling, Multi-View Simulation ............. 135 6.5.6 Manipulating Visualiser Data ................. 137 6.5.7 Capacity of the Visualiser ................... 137 6.6 Non-Neural Plotting ........................... 138 6.7 Summary and Contributions ...................... 141 7 Managing Large Network Attached Systems 143 7.1 Principles of System Management ................... 143 7.1.1 Fault Management ....................... 144 7.1.2 Configuration Management .................. 144 4 7.1.3 Accounting Management .................... 145 7.1.4 Performance Management ................... 145 7.1.5 Security Management ..................... 146 7.2 Managing Large Systems ........................ 146 7.2.1 Remote Monitoring ....................... 146 7.2.2 Hardware Management ..................... 147 7.2.3 Systems Management Software ................ 150 7.2.4 Protocol and Schema Standards ................ 150 7.2.5 SNMP: A Walk Through .................... 153 7.2.6 The Management Information Base (MIB) .......... 155 7.2.7 Previous Large System Research with SNMP ......... 157 7.3 Summary ................................ 158 8 SpiNNaker Management Framework 160 8.1 SpiNNaker – a Memory-Mapped Architecture ............. 160 8.2 A Protocol Translator – SpiNNmate .................. 161 8.2.1 Primitive Operations on SpiNNaker .............. 163 8.2.2 SpiNNmate Structure ...................... 163 8.2.3 Protocol Modules ........................ 166 8.3 Memory and Communications ..................... 169 8.3.1 Memory Operations – When to use DMA ........... 169 8.3.2 Memory Performance Optimisation .............. 171 8.3.3 Communications Bandwidth .................. 175 8.4 Management Framework Results .................... 176 8.4.1 Memory Operations – the Performance of SpiNNmate .... 176 8.4.2 A Real-Time SNMP Temperature Plotter ........... 178 8.4.3 Processor Utilisation Monitor ................. 179 8.4.4 A Network Packet Counter ................... 180 8.4.5 Long Term Monitoring ..................... 180 8.4.6 Alerting using Nagios ..................... 182 8.5 Monitoring a SpiNNaker Machine in Production ............ 183 8.6 Summary and Contributions ...................... 184 9 Discussion and Conclusions 186 9.1 Bootstrapping SpiNNaker ........................ 187 9.1.1 Primary Targets ......................... 187 5 9.1.2 Time Taken to Boot ....................... 188 9.1.3 Power-On Self-Test ....................... 188 9.1.4 ITCM Validation Block (IVB) ................. 189 9.1.5 DHCP Node-Boot Image .................... 189 9.1.6 Future Work .......................... 190 9.2 Visualising Neural Networks on SpiNNaker .............. 192 9.2.1 Modularity ........................... 192 9.2.2 Modalities ........................... 192 9.2.3 Interaction ........................... 193 9.2.4 Aggregation ........................... 194 9.2.5 Visualiser Limitations and Future Work ............ 195 9.3 SpiNNaker Management Framework .................. 196 9.3.1 Protocol Translation ...................... 196 9.3.2 SpiNNmate Performance .................... 198 9.3.3 Future Work .......................... 199 9.4 General SpiNNaker Observations .................... 201 9.4.1 Interconnecting Larger Machines ............... 202 9.4.2 Improving Load and Save Times ................ 202 9.4.3 Further Works in Software ................... 203 9.5 Summary ................................ 203 A Expansion of Abbreviations 204 Bibliography 209 Word Count: 55371 6 List of Tables 4.1 System RAM register allocations following Node-Boot ........ 72 4.2 Power-on self-tests and actions ..................... 73 4.3 ITCM validation block structure. .................... 83 6.1 Spike-rates found in fig. 6.10 simulation ................ 138 8.1 Loading SDRAM: SpiNNmate’s performance ............. 178 7 List of Figures 1.1 SpiNNaker torus ............................ 20 1.2 The system management time-line of SpiNNaker. ........... 22 2.1 The human brain. ............................ 30 2.2 A six-layer human cortical column ................... 31 2.3 Simplified neuronal structure. ...................... 32 2.4 Threshold Logic Unit (TLU) ...................... 34 2.5 Second generation Artificial Neural Network ............. 35 2.6 A three-layer Multi-Layer Perceptron (MLP). ............. 36 2.7 Izhikevich spiking model behaviour .................. 37 2.8 A third generation ANN ........................ 38 3.1 Photographs of the SpiNNaker chip .................. 52 3.2 SpiNNaker chip schematic ....................... 53 3.3 SpiNNaker machine tiled array ..................... 54 3.4 SpiNNaker packet formats. ....................... 57 3.5 Multicast and Point to Point packet flow and distribution trees. .... 58 3.6 Nearest Neighbour and Fixed Route packet flow and distribution trees. 60 3.7 SDP Framing Format .......................... 61 3.8 Communication layering diagram ................... 63 3.9 Partition And Configuration MANager (PACMAN) .......... 64 4.1 Management time-line: boot ...................... 67 4.2 SpiNNaker boot and application loading sequence. .......... 68 4.3 Labelled SpiNNaker chip plot ..................... 69 4.4 Node-Boot flowchart