UNIVERSITY COLLEGE LONDON A Practical Hardware Implementation of Systemic Computation by Christos Sakellariou A thesis submitted in partial fulfilment for the degree of Doctor of Engineering in the Faculty of Engineering Department of Computer Science December 2013 2 Declaration of Authorship I, Christos Sakellariou, declare that this thesis A Practical Hardware Implementation of Systemic Computation and the work presented in it are my own. I confirm that: This work was done wholly or mainly while in candidature for a research degree at University College London. Where any part of this thesis has previously been submitted for a degree or any other qualification at University College London or any other institution, this has been clearly stated. Where I have consulted the published work of others, this is always clearly attributed. Where I have quoted from the work of others, the source is always given. With the exception of such quotations, this thesis is entirely my own work. I have acknowledged all main sources of help. Where the thesis is based on work done by myself jointly with others, I have made clear exactly what was done by others and what I have contributed myself. Signed: Date: 3 Abstract Faculty of Engineering, Department of Computer Science UNIVERSITY COLLEGE LONDON Doctor of Engineering by Christos Sakellariou It is widely accepted that natural computation, such as brain computation, is far superior to typical computational approaches addressing tasks such as learning and parallel processing. As conventional silicon-based technologies are about to reach their physical limits, researchers have drawn inspiration from nature to found new computational paradigms. Such a newly-conceived paradigm is Systemic Computation (SC). SC is a bio-inspired model of computation. It incorporates natural characteristics and defines a massively parallel non-von Neumann computer architecture that can model natural systems efficiently. This thesis investigates the viability and utility of a Systemic Computation hardware implementation, since prior software-based approaches have proved inadequate in terms of performance and flexibility. This is achieved by addressing three main research challenges regarding the level of support for the natural properties of SC, the design of its implied architecture and methods to make the implementation practical and efficient. Various hardware-based approaches to Natural Computation are reviewed and their compatibility and suitability, with respect to the SC paradigm, is investigated. FPGAs are identified as the most appropriate implementation platform through critical evaluation and the first prototype Hardware Architecture of Systemic computation (HAoS) is presented. HAoS is a novel custom digital design, which takes advantage of the inbuilt parallelism of an FPGA and the highly efficient matching capability of a Ternary Content Addressable Memory. It provides basic processing capabilities in order to minimize time-demanding data transfers, while the optional use of a CPU provides high-level processing support. It is optimized and extended to a practical hardware platform accompanied by a software framework to provide an efficient SC programming solution. The suggested platform is evaluated using three bio-inspired models and analysis shows that it satisfies the research challenges and provides an effective solution in terms of efficiency versus flexibility trade-off. 4 Acknowledgements I sincerely believe I could not have been luckier when I first emailed Peter Bentley about five years ago on the prospect of working under his supervision towards a doctoral degree. I am extremely thankful to him for his tremendous level of support during these five years. I hugely appreciate all his help along this journey and clearly acknowledge that this thesis would not have been possible without his guidance and encouragement. I enjoyed the fact the he is a person that I could talk to freely and count on being listened every single time. I am grateful to consider him as a friend from whom I learned a lot. I also wish to thank the other researchers of my research group. Special thanks go to Erwan Le Martelot for sharing with me early drafts of his thesis when I started my studies. Further thanks go to Hooman Shayani, Arturo Araujo and Marjan Rouhipour for sharing parts of their work on Systemic Computation that greatly helped me move forward. I would also like to thank other colleagues at UCL: my second supervisor Prof. Steve Hailes and Prof. Izzat Darwazeh for their guidance during the reviews of my progress reports. I also would like to express my thanks to my industrial sponsor Toumaz Technology and especially to my industrial supervisor Alison Burdett for her invaluable support on the initial application towards my doctorate degree and the financial support along its duration. I further wish to acknowledge the EPSRC, the EngD VEIV doctoral centre and the department of Computer Science in UCL for their financial support and for hosting me along my studies. I would like to thank all my friends that supported me during happy and rough times during these last five years (thank you so much guys for endless coffees and nights in and out!!): my good friends from Greece Psilos, Spirakos, Thomas and Giannakis, my good Toumaz friend Andrikos, my Physics friends Thiseas and Thanos, my soton friend Giannis, my UCL friends Melinos, Dimitris, Theo, Tim and Alexis, my weekend ciccilones friends Pippo, Adrian, Vero and Andreea, my Reading friends Stamo and Dia, my neighbours Bogo and Stella, my childhood friends Sakis, Nantia, Zetta, Stelios and Panagiotakis, my Imperial friend Dimos, all my dear cousins and so many more that need to be here and aren't due to my long-lasting sleep deprivation.. :) Of course I want to thank my family for their continuous support before, along and beyond my studies. Special thanks to my parents, Aspasia and Thomas, and my little sister Kleiw for their unconditional love. Further thanks to all my extended family (grandparents, uncles, aunts and cousins) for helping me become who I am. Last but not least, I'd like to express my gratitude to my loving partner Marilia for always being there for me along these years, following and supporting me on every step of the way. 5 Table of Contents Abstract ......................................................................................................................... 3 Acknowledgements ...................................................................................................... 4 List of Figures ............................................................................................................. 10 List of Tables .............................................................................................................. 14 List of Listings ............................................................................................................ 16 List of Algorithms ....................................................................................................... 16 Chapter 1 Introduction ........................................................................................... 17 1.1 Introduction to Natural Computation ............................................................. 17 1.2 Introduction to Systemic Computation .......................................................... 20 1.3 Systemic Computation in Practice ................................................................. 21 1.4 Hypothesis .................................................................................................... 23 1.5 Objectives ..................................................................................................... 25 1.6 Publications .................................................................................................. 26 1.7 Thesis Organization ...................................................................................... 27 Chapter 2 Background ........................................................................................... 29 2.1 Approaches to Natural Computation .............................................................. 30 2.1.1 Software Approaches ............................................................................. 31 2.1.2 Alternative Paradigms ........................................................................... 32 2.2 Hardware-based Approaches to Natural Computation .................................... 33 2.2.1 Conventional Hardware Approaches ...................................................... 33 2.2.2 Unconventional Hardware Approaches .................................................. 38 2.2.3 Other silicon-based designs ................................................................... 55 2.2.4 Hardware Approaches Summary ............................................................ 62 6 2.3 Systemic Computation .................................................................................. 63 2.4 Prior Systemic Computation Implementations ............................................... 68 2.4.1 Original SC Implementation .................................................................. 68 2.4.2 High-level SC Implementation ............................................................... 70 2.4.3 GPU SC Implementation ....................................................................... 71 2.5 Summary ...................................................................................................... 72 Chapter 3 Designing a Suitable