Neuroscience-Inspired Dynamic Architectures
Total Page:16
File Type:pdf, Size:1020Kb
University of Tennessee, Knoxville TRACE: Tennessee Research and Creative Exchange Doctoral Dissertations Graduate School 5-2015 Neuroscience-Inspired Dynamic Architectures Catherine Dorothy Schuman University of Tennessee - Knoxville, [email protected] Follow this and additional works at: https://trace.tennessee.edu/utk_graddiss Part of the Artificial Intelligence and Robotics Commons, and the Theory and Algorithms Commons Recommended Citation Schuman, Catherine Dorothy, "Neuroscience-Inspired Dynamic Architectures. " PhD diss., University of Tennessee, 2015. https://trace.tennessee.edu/utk_graddiss/3361 This Dissertation is brought to you for free and open access by the Graduate School at TRACE: Tennessee Research and Creative Exchange. It has been accepted for inclusion in Doctoral Dissertations by an authorized administrator of TRACE: Tennessee Research and Creative Exchange. For more information, please contact [email protected]. To the Graduate Council: I am submitting herewith a dissertation written by Catherine Dorothy Schuman entitled "Neuroscience-Inspired Dynamic Architectures." I have examined the final electronic copy of this dissertation for form and content and recommend that it be accepted in partial fulfillment of the requirements for the degree of Doctor of Philosophy, with a major in Computer Science. J. Douglas Birdwell, Major Professor We have read this dissertation and recommend its acceptance: Mark E. Dean, Tsewei Wang, Itamar Arel, Bruce MacLennan Accepted for the Council: Carolyn R. Hodges Vice Provost and Dean of the Graduate School (Original signatures are on file with official studentecor r ds.) University of Tennessee, Knoxville Trace: Tennessee Research and Creative Exchange Doctoral Dissertations Graduate School 5-2015 Neuroscience-Inspired Dynamic Architectures Catherine Dorothy Schuman University of Tennessee - Knoxville, [email protected] This Dissertation is brought to you for free and open access by the Graduate School at Trace: Tennessee Research and Creative Exchange. It has been accepted for inclusion in Doctoral Dissertations by an authorized administrator of Trace: Tennessee Research and Creative Exchange. For more information, please contact [email protected]. To the Graduate Council: I am submitting herewith a dissertation written by Catherine Dorothy Schuman entitled "Neuroscience- Inspired Dynamic Architectures." I have examined the final electronic copy of this dissertation for form and content and recommend that it be accepted in partial fulfillment of the requirements for the degree of Doctor of Philosophy, with a major in Computer Science. J. Douglas Birdwell, Major Professor We have read this dissertation and recommend its acceptance: Mark E. Dean, Tsewei Wang, Itamar Arel, Bruce MacLennan Accepted for the Council: Carolyn R. Hodges Vice Provost and Dean of the Graduate School (Original signatures are on file with official student records.) Neuroscience-Inspired Dynamic Architectures A Dissertation Presented for the Doctor of Philosophy Degree The University of Tennessee, Knoxville Catherine Dorothy Schuman May 2015 c by Catherine Dorothy Schuman, 2015 All Rights Reserved. ii For my nieces and nephews: Hannah, Austin, Addison, Aiden, and Anna-Beth. May this inspire you to do great things. In memory of my grandparents: Fred Wright (1905 { 1973) and Dorothy Babcock Wright (1909 { 1971) Richard Schuman, Jr. (1920 { 2007) and Catherine Dunn Schuman (1917 { 2014) iii Acknowledgements I would like to thank the National Science Foundation Graduate Research Fellowship Program and the Department of Electrical Engineering and Computer Science Bodenheimer Fellowship (especially Dr. Robert Bodenheimer and Mr. Mike Crabtree) for providing me funding to conduct research of my choosing. I would also like to thank the Department of Electrical Engineering and Computer Science for allowing me to serve as a teaching assistant throughout my time as a graduate student. I would like to thank my advisor, Dr. Doug Birdwell, for his constant support and feedback, and for being willing to go down this unknown path with me. I would like to thank Dr. Mark Dean for believing in my work and for developing the accompanying hardware implementation. I would also like to thank the other members of my committee, Dr. Tsewei Wang, Dr. Itamar Arel, and Dr. Bruce MacLennan for not only their insight and support, but also for the amazing courses they offered that helped to inspire aspects of this work. I would like to thank Dr. Jim Plank for teaching me how to do computer science research and for allowing me to be his teaching assistant for five years. I would like to thank Dr. Lynne Parker for her mentorship and Dr. Mike Langston for his support. I would like to thank the senior design team who built the first hardware implementation: Chris Daffron, Josh Willis, Jacob Tobin, Belinda Loi, David Bashai, Joonki Rhee, and Tommy Nguyen. I would also like to thank Chris, Josh, and Jason Chan for their ongoing work with the hardware implementation. I would like iv to thank Meg Drouhard for her amazing visualizations of NIDA networks, some of which are included in this work. I would also like to thank our undergraduate research assistants, Luke Bechtel and Ryan Wagner, for their dedication in working with NIDA and DANNA. I would like to thank UT for their Newton Cluster, on which I ran many of the simulations described in this work, and Gerald Raggianthi for his prompt replies to emails and his dedicated efforts to keep Newton up and running as often as possible. I would like to thank Randy Bond, Bryan Burke, Markus Iturriaga Woelfel, and all of the EECS IT staff for keeping all of our computing resources up and running. I would like to thank Dana Bryson, Julia Elkins, Kim Cox, Melanie Kelley, Linda Robinson, Sonja Hill, and all of the EECS staff who do amazing work and don't get nearly the recognition that they deserve. I would like to thank Lily Huxford and Maggie Bouldin for being some of the best friends anyone could ask for, and for understanding my crazy schedule. I would like to thank Denise Koessler Gosnell, Zahra Mahoor, Lonnie Yu, and Nick Lineback for being the best graduate study group ever, and Ashleigh Ellison, Jordan Deyton, Chad Braden, Chris Craig, Wade Jenkins, Blake Hitchcock, and Rob Fletcher for being the best undergraduate study group ever. I would like to thank Denise, Zahra, Meg, Nicole Pennington, Sadika Amreen, and Casey Miller (the Systers) for their passionate support of women in electrical engineering and computer science. I would like to thank my parents, John and Sherry Schuman, for their unwavering support and love. I would like to thank my sister, Tracey Lawson, for listening to every stressed phone call, and my brother, Alan Oliver, for getting me interested in computer science in the first place. I would like to thank my aunts and uncles, my many cousins, and my other family members, for their support, even when they didn't understand what I was talking about it. I would like to thank my nieces and nephews, Hannah Lawson, Austin Oliver, Addison Oliver, Aiden Oliver, and Anna-Beth Oliver, for being a welcome distraction from research and for being just generally awesome people. Without their support, this would not have been possible. v This material is based upon work supported by the National Science Foundation Graduate Research Fellowship Program under Grant No. DGE-0929298. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author and do not necessarily reflect the views of the National Science Foundation. vi Abstract Biological brains are some of the most powerful computational devices on Earth. Computer scientists have long drawn inspiration from neuroscience to produce computational tools. This work introduces neuroscience-inspired dynamic archi- tectures (NIDA), spiking neural networks embedded in a geometric space that exhibit dynamic behavior. A neuromorphic hardware implementation based on NIDA networks, Dynamic Adaptive Neural Network Array (DANNA), is discussed. Neuromorphic implementations are one alternative/complement to traditional von Neumann computation. A method for designing/training NIDA networks, based on evolutionary optimization, is introduced. We demonstrate the utility of NIDA networks on classification tasks, a control task, and an anomaly detection task. There are known neural structures (such as cortical columns) that are repeated many times in the brain, and there are other structures that are useful for a variety of different tasks. We speculated that \useful structures" will also emerge in NIDA networks. Three methods for identifying useful substructures within a NIDA network are presented: common structure, activity-based, and causality paths. We explored reusing activity-based useful substructures over the course of evolutionary optimization, but the results for those tests were inconclusive. One component of biological brains that is often ignored in biologically-inspired computation is the influence of affective, or emotion-related, systems. We define artificial affective systems and explore the effect of affective systems in NIDA networks in terms of vii behavior of the network and on the evolutionary optimization design method. We conclude with an outline of future research opportunities identified during this effort. viii Table of Contents 1 Introduction1 2 Related Work 11 2.1 Neuroevolution.............................. 12 2.2 Embedding in a Geometric Space.................... 26 2.3 Dynamic Networks...........................