DOCSLIB.ORG

Biologically Realistic Artificial Neural Networks

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Biologically Realistic Artificial Neural Networks Biologically Realistic Artificial Neural Networks Matthijs Kok University of Twente P.O. Box 217, 7500AE Enschede The Netherlands [email protected] ABSTRACT (computational models) for neural learning, to gain an un- Artificial Neural Networks (ANNs) play an important role derstanding of how biological systems, such as the brain, in machine learning nowadays. A typical ANN is tradi- work. Furthermore, neuroscience also aims to make con- tionally trained using stochastic gradient descent (SGD) tributions towards progress in artificial intelligence, such with backpropagation (BP). However, it is unlikely that as inspiration and validation for new computer algorithms a real biological neural network is trained similarly. Neu- and architectures [5]. To improve the understanding in roscience theories, such as Hebbian Theory could inspire both fields, models that better resemble biological neural adaption from the traditional training method SGD to networks are of utmost importance. make ANNs more biologically plausible. Two mathemati- ANNs are vaguely inspired by biological neural networks. cal descriptions of Hebbian theory will be suggested based However, the method SGD-BP that is used to train ANNs on the limitations of the mathematical framework of Heb- does not resemble the way in which the human brain learns. bian theory: A competitive Hebbian learning rule and an This research aims to contribute to closing the gap be- imply Hebbian learning rule. Finally, this research will tween artificial and biological neural networks, as it will propose a method, called the Hebbian Plasticity Term investigate a neuroscience theory called Hebbian theory (HPT) method, that incorporates the mathematical de- [16] as starting point to modify the traditional training scription of Hebbian theory to modify the traditional train- method SGD of ANNs aiming to make the learning pro- ing method SGD. Therefore two variants of the HPT- cess more biologically plausible. Within this research, two method are proposed: HPT-Competitive and HPT-Imply. mathematical formulations are proposed for different Heb- The influence of the hebbian plasticity term ϕ on SGD bian learning rules, which are incorporated in traditional shows more biologically realistic plasticity of a synaptic SGD by a newly developed method. The results of this connection between neurons in the ANN at the cost of research can be used as a basis for developing more bio- performance. logically realistic artificial neural networks. This paper will investigate the following research ques- Keywords tions: Artificial Neural Network, Stochastic Gradient Descent RQ 1 How could Hebbian theory be used to mathemati- with Backpropagation, Hebbian Theory, Hebbian Plastic- cally describe a biologically realistic evolution of a synapse? ity Term method RQ 1.1 What are the limitations to Hebbian the- ory when using it to formulate a mathematical de- 1. INTRODUCTION scription of a weight update? Deep learning is a very important area in the study of RQ 1.2 Which types of mathematical descriptions machine learning nowadays, with wide applications [21, of Hebbian theory could be used, as a result of these 18, 10] such as speech recognition [9, 15], natural language limitations? processing [7], and image recognition [11, 32]. It describes RQ 2 Could the training algorithm BP-SGD of a typi- a powerful set of techniques for learning in artificial neural cal standard ANN be modified using Hebbian theory to networks (ANNs) [4, 25, 31]. A traditional method to train become a more biologically plausible training method? ANNs is stochastic gradient descent (SGD), which makes use of backpropagation (BP) [13, 25]. RQ 2.1 How could a mathematical description of Hebbian theory be incorporated into the standard In the biological sciences, understanding the biological ba- training algorithm BP-SGD? sis of learning is considered to be one of the ultimate chal- lenges [17]. Theoretical or computational neuroscientists RQ 2.2 Does a connection weight between two strive to make a link between observed biological pro- neurons of an artificial neural network evolve more cesses and biologically plausible theoretical mechanisms biologically plausible as a result of incorporating the new method? Permission to make digital or hard copies of all or part of this work for RQ 3 To what extent does the performance of an ANN personal or classroom use is granted without fee provided that copies trained using the newly developed more biologically realis- are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy oth- tic training method differ from the performance of an ANN erwise, or republish, to post on servers or to redistribute to lists, requires trained using the traditional training method BP-SG? prior specific permission and/or a fee. 33th Twente Student Conference on IT July 3rd, 2020, Enschede, The Netherlands. Copyright 2020, University of Twente, Faculty of Electrical Engineer- ing, Mathematics and Computer Science. 1 2. ANALOGY BETWEEN THE ANN AND of the pre-synaptic neurons, according to equation (1). THE BRAIN l l−1 l l aj = σ (ai · wji)+ bj (1) 2.1 Neurons in the brain " i # X Here, σ denotes a sigmoid-activation function and the bias l is unique to a neuron and is denoted as bj . The weighted l−1 l l l input sum i(ai · wji)+ bj will be summarized as zj throughout the rest of this paper. Furthermore, the su- perscript isP written as l to denote any layer, as integer to denote a specific layer, or as L,L − 1,L − 2,.., where L denotes the last layer. Both, i and j denote the position of the neuron within a layer. The use of subscript letters will be alphabetically coherent with the order of layers l−2 l−1 l l+1 (e.g. ..., ah ,ai ,aj ,ak ,...). The dependence of the l post-synaptic neuron aj on multiple pre-synaptic neurons l−1 ai is also present in real biological post-synaptic neu- rons, which can be connected to multiple (pre-synaptic) Figure 1. Schematic image of two neurons in a neurons through their (multiple) dendrites, which deter- brain mine together whether or not an action potential is trig- gered. Figure 1 depicts two neurons in a brain that are separated An ANN shows connections as edges between nodes. The by a synapse. When the leftmost neuron, the pre-synaptic edge or connection is analogous to the synapse between l neuron, is excited by an action potential, the vesicles in its two neurons in a brain. In an ANN, a weight wji is asso- axon terminals fuse with its membrane and release their ciated to a connection and is considered to represent the contents, neurotransmitters, into the synaptic cleft. Once strength of the respective connection. In other words, a released, the neurotransmitters can bind to the receptors large weight in an ANN implies a large synaptic efficacy of on the dendrite of the rightmost neuron, the post-synaptic the analogous biological synapse. The subscript ji denotes neuron, acting as a chemical signal. That binding opens the position of the synapse within a layer. Here, j denotes ion channels on its dendrite that allow charged ions to the post- and i the pre-synaptic neuron, which together l flow in and out of the cell, converting the chemical signal form the synaptic connection wji. into an electrical signal. If the combined effect of multiple dendrites (which can be connected to other neurons) of 3. TRAINING AN ANN the post-synaptic neuron changes the overall charge of the A common ANN training method is stochastic gradient de- cell enough, then it triggers an action potential, also called scent (SGD), which requires backpropagation (BP). This a spike [24, 27]. section discusses how traditional SGD is performed on an 2.2 The relation between an ANN and neu- ANN, to serve as a basis for section 6 and 7. In an ANN, one feedforward is performed by computing rons in the brain l equation (1) sequentially for each layer. The weights wji A typical feedforward ANN could be described as a struc- l tured collection of nodes with edges between them, as de- and biases bj used to compute equation (1) are initial- picted in Figure 2. The ANN that is depicted in Fig- ized as random variables drawn from a standard normal distribution. Since the computation of each separate com- ponent is cumbersome, vector-wise multiplication is used. Below, matrix multiplication for a feed-forward is shown (in this example layer l has 2 units and layer l + 1 has 4 units): l+1 l+1 l+1 l+1 z1 w11 w12 b1 l+1 l+1 l+1 l l+1 z2 w21 w22 a1 b2 l = l l · l + l (2) z +1 w +1 w +1 a b +1 3 31 32 2 3 zl+1 wl+1 wl+1 bl+1 4 41 42 4 where l+1 l+1 a1 z1 l+1 l+1 a2 z2 l+1 = σ( l+1 ) (3) Figure 2. Example feedforward ANN with 2 neu- a3 z3 al+1 zl+1 rons in the input layer, 3 neurons in the hidden 4 4 layer, and 2 neurons in the output layer In short, this can be notated as: l 1 l l 1 z + = W l+1 · a + b + (4) ure 2 represents connected neurons in a brain, as con- nected nodes. The spike in a neuron is considered a bi- and l 1 l 1 nary phenomenon: it is there or not. Therefore, the acti- a + = σ(z + ) (5) vation in an ANN represents the firing rate of the neuron. l l l l The more often a pre-synaptic neuron fires, the higher the Here, W is a matrix and a , b , z are vectors.
Load more

Recommended publications
  • Song Exposure Affects HVC Ultrastructure in Juvenile Zebra Finches
    Song Exposure affects HVC Ultrastructure in Juvenile Zebra Finches Houda G Khaled Advisor: Dr. Sharon M.H. Gobes Department of Neuroscience Submitted in Partial Fulfillment of the Prerequisite for Honors in Biochemistry May 2016 © 2016 Houda Khaled Table of Contents ABSTRACT ........................................................................................................................................................ 2 ACKNOWLEDGEMENTS .............................................................................................................................. 3 INTRODUCTION ............................................................................................................................................. 4 I. SYNAPTIC PLASTICITY IN LEARNING AND MEMORY ................................................................................ 4 Basic principles of synapse structure ............................................................................................................. 4 Changes in synapse number, size and shape ................................................................................................ 6 II. SONG ACQUISITION AS A LEARNING PARADIGM .................................................................................. 7 The song system ................................................................................................................................................. 9 Structural synaptic plasticity in the song system .....................................................................................
    [Show full text]
  • Interplay of Multiple Plasticities and Activity Dependent Rules: Data, Models and Possible Impact on Learning
    Interplay of multiple plasticities and activity dependent rules: data, models and possible impact on learning Gaetan Vignoud, Alexandre Mendes, Sylvie Perez, Jonathan Touboul*, Laurent Venance* (* equal contributions, email: fi[email protected]) College` de France, Center for Interdisciplinary Research in Biology, 11 Place Marcelin Berthelot, 75005 Paris, France Abstract the millisecond timing of the paired activities on either side of Synaptic plasticity, the activity-dependent evolution neuronal the synapse (Feldman, 2012). As such, plasticity in the stria- connectivity, is admitted to underpin learning and memory. A tum has been widely studied and a variety of spike-timing de- paradigmatic plasticity depending on the spike timings of cells pendent plasticity were observed in response to a large num- on both sides of the synapse (STDP), was experimentally ev- ber of presentations of a fixed pre- and post-synaptic stim- idenced through multiple repetitions of fixed pre- and post- ulation pattern. Those plasticities were attributed to a di- synaptic spike patterns. Theoretical and experimental com- verse neurotransmitters and pathways including for instance munities often implicitly admit that plasticity is gradually estab- endocannabinoids (eCB), NMDA receptors, AMPA receptors lished as stimulus patterns are presented. Here, we evaluate or voltage-sensitive calcium channels. Moreover, the vast ma- this hypothesis experimentally and theoretically in the stria- jority of the studies focused on cortical inputs, and much
    [Show full text]
  • A Theory of Consumer Fraud in Market Economies
    A THEORY OF FRAUD IN MARKET ECONOMIES ADISSERTATIONIN Economics and Social Science Presented to the Faculty of the University of Missouri-Kansas City in partial fulfillment of the requirements for the degree DOCTOR OF PHILOSOPHY by Nicola R. Matthews B.F.A, School of Visual Arts, 2002 M.A., State University College Bu↵alo, 2009 Kansas City, Missouri 2018 ©2018 Nicola R. Matthews All Rights Reserved A THEORY OF FRAUD IN MARKET ECONOMIES Nicola R. Matthews, Candidate for the Doctor of Philosophy Degree University of Missouri-Kansas City, 2018 ABSTRACT Of the many forms of human behavior, perhaps none more than those of force and fraud have caused so much harm in both social and ecological environments. In spite of this, neither shows signs of weakening given their current level of toleration. Nonetheless, what can be said in respect to the use of force in the West is that it has lost most of its competitiveness. While competitive force ruled in the preceding epoch, this category of violence has now been reduced to a relatively negligible degree. On the other hand, the same cannot be said of fraud. In fact, it appears that it has moved in the other direction and become more prevalent. The causes for this movement will be the fundamental question directing the inquiry. In the process, this dissertation will trace historical events and methods of control ranging from the use of direct force, to the use of ceremonies and rituals and to the use of the methods of law. An additional analysis of transformation will also be undertaken with regards to risk-sharing-agrarian-based societies to individual-risk-factory-based ones.
    [Show full text]
  • Arxiv:1711.09601V4 [Cs.CV] 5 Oct 2018 Unlabeled Data Towards What the Network Needs (Not) to Forget, Which May Vary Depending on Test Conditions
    Memory Aware Synapses: Learning what (not) to forget Rahaf Aljundi1, Francesca Babiloni1, Mohamed Elhoseiny2, Marcus Rohrbach2, and Tinne Tuytelaars1 1 KU Leuven, ESAT-PSI, iMEC, Belgium 2 Facebook AI Research Abstract. Humans can learn in a continuous manner. Old rarely utilized knowl- edge can be overwritten by new incoming information while important, frequently used knowledge is prevented from being erased. In artificial learning systems, lifelong learning so far has focused mainly on accumulating knowledge over tasks and overcoming catastrophic forgetting. In this paper, we argue that, given the limited model capacity and the unlimited new information to be learned, knowl- edge has to be preserved or erased selectively. Inspired by neuroplasticity, we propose a novel approach for lifelong learning, coined Memory Aware Synapses (MAS). It computes the importance of the parameters of a neural network in an unsupervised and online manner. Given a new sample which is fed to the net- work, MAS accumulates an importance measure for each parameter of the net- work, based on how sensitive the predicted output function is to a change in this parameter. When learning a new task, changes to important parameters can then be penalized, effectively preventing important knowledge related to previous tasks from being overwritten. Further, we show an interesting connection between a local version of our method and Hebb’s rule, which is a model for the learning process in the brain. We test our method on a sequence of object recognition tasks and on the challenging problem of learning an embedding for predicting <subject, predicate, object> triplets. We show state-of-the-art performance and, for the first time, the ability to adapt the importance of the parameters based on arXiv:1711.09601v4 [cs.CV] 5 Oct 2018 unlabeled data towards what the network needs (not) to forget, which may vary depending on test conditions.
    [Show full text]
  • Building a “Cross-Roads Discipline” at Mcgill University: a History of Early Experimental Psychology in Postwar Canada
    BUILDING A “CROSS-ROADS DISCIPLINE” AT MCGILL UNIVERSITY: A HISTORY OF EARLY EXPERIMENTAL PSYCHOLOGY IN POSTWAR CANADA ERIC OOSENBRUG A dissertation submitted to the Faculty of Graduate Studies in partial fulfillment of the requirements for the degree of Doctor of Philosophy Graduate Program in Psychology. Graduate Program in Psychology York University Toronto, Ontario October 2020 © Eric Oosenbrug, 2020 Abstract This dissertation presents an account of the development of psychology at McGill University from the late nineteenth century through to the early 1960s. The department of psychology at McGill represents an alternative to the traditional American-centered narrative of the cognitive revolution and later emergence of the neurosciences. In the years following World War II, a series of psychological experiments established McGill as among the foremost departments of psychology in North America. This thesis is an institutional history that reconstructs the origins, evolution, and dramatic rise of McGill as a major center for psychological research. The experiments conducted in the early 1950s, in the areas of sensory restriction, motivation, and pain psychology, were transformative in their scope and reach. Central to this story is Donald O. Hebb, author of The Organization of Behavior (1949), who arrived at McGill in 1947 to find the charred remains of a department. I argue that the kind of psychology Hebb established at McGill was different from most departments in North America; this is developed through a number of interwoven storylines focused on the understanding of a particular character of McGill psychology - a distinctive “psychological style” - and its broader historical importance for Canadian psychology, for North American psychology, and for psychology across the globe.
    [Show full text]
  • Neuromodulated Dopamine Plastic Networks for Heterogeneous Transfer Learning with Hebbian Principle
    S S symmetry Article Neuromodulated Dopamine Plastic Networks for Heterogeneous Transfer Learning with Hebbian Principle Arjun Magotra and Juntae Kim * Department of Computer Science and Engineering, Dongguk University, 30, Pildong-ro 1-gil, Jung-gu, Seoul 04620, Korea; [email protected] * Correspondence: [email protected] Abstract: The plastic modifications in synaptic connectivity is primarily from changes triggered by neuromodulated dopamine signals. These activities are controlled by neuromodulation, which is itself under the control of the brain. The subjective brain’s self-modifying abilities play an essential role in learning and adaptation. The artificial neural networks with neuromodulated plasticity are used to implement transfer learning in the image classification domain. In particular, this has application in image detection, image segmentation, and transfer of learning parameters with significant results. This paper proposes a novel approach to enhance transfer learning accuracy in a heterogeneous source and target, using the neuromodulation of the Hebbian learning principle, called NDHTL (Neuromodulated Dopamine Hebbian Transfer Learning). Neuromodulation of plasticity offers a powerful new technique with applications in training neural networks implementing asymmetric backpropagation using Hebbian principles in transfer learning motivated CNNs (Convolutional neural networks). Biologically motivated concomitant learning, where connected brain cells activate positively, enhances the synaptic connection strength between the network neurons. Using the NDHTL algorithm, the percentage of change of the plasticity between the neurons of the CNN layer is directly managed by the dopamine signal’s value. The discriminative nature of transfer learning Citation: Magotra, A.; Kim, J. fits well with the technique. The learned model’s connection weights must adapt to unseen target Neuromodulated Dopamine Plastic datasets with the least cost and effort in transfer learning.
    [Show full text]
  • Machine Awareness
    Wright State University CORE Scholar International Symposium on Aviation International Symposium on Aviation Psychology - 2019 Psychology 5-7-2019 Machine Awareness Steven D. Harbour Jeffery D. Clark William D. Mitchell Krishnamurthy V. Vemuru Follow this and additional works at: https://corescholar.libraries.wright.edu/isap_2019 Part of the Other Psychiatry and Psychology Commons Repository Citation Harbour, S. D., Clark, J. D., Mitchell, W. D., & Vemuru, K. V. (2019). Machine Awareness. 20th International Symposium on Aviation Psychology, 480-485. https://corescholar.libraries.wright.edu/isap_2019/81 This Article is brought to you for free and open access by the International Symposium on Aviation Psychology at CORE Scholar. It has been accepted for inclusion in International Symposium on Aviation Psychology - 2019 by an authorized administrator of CORE Scholar. For more information, please contact [email protected]. MACHINE AWARENESS Dr. Steven D. Harbour Dr. Jeffery D. Clark Mr. William D. Mitchell Riverside Research 2640 Hibiscus Way, Beavercreek, Ohio 45431, USA Dr. Krishnamurthy V. Vemuru Riverside Research 2900 Crystal Dr., Arlington, Virginia 22202, USA Current and future research that embodies a pathway to achieving machine common sense (MCS), including Capsule Neural Networks, Hebbian Plasticity Theory, Dual Process Theory, and machine awareness (MA). The final frontier may well involve a framework that is capable of machine curiosity, exploration, automatic self- direction and adaptation. The artificial intelligence (AI) system of the future will possess an innate curiosity and explore its own environment to gain knowledge, exhibiting a basic element of human cognition and awareness. The resulting MA system will possess inherent self-driven curiosity and related entropy in the decision space as it explores the environment in much the same manner as humans do.
    [Show full text]
  • Evidence for a Causal Inverse Model in an Avian Cortico-Basal Ganglia Circuit
    Evidence for a causal inverse model in an avian cortico-basal ganglia circuit Nicolas Gireta,b, Joergen Kornfelda, Surya Gangulic, and Richard H. R. Hahnlosera,b,1 aInstitute of Neuroinformatics and bNeuroscience Center Zurich, University of Zurich and ETH Zurich, 8057 Zurich, Switzerland; and cDepartment of Applied Physics, Stanford University, Stanford, CA 94305-5447 Edited by Thomas C. Südhof, Stanford University School of Medicine, Stanford, CA, and approved March 7, 2014 (received for review September 11, 2013) Learning by imitation is fundamental to both communication and nucleus of the anterior nidopallium (LMAN) forms the output of social behavior and requires the conversion of complex, nonlinear a basal-ganglia pathway involved in generating subtle song vari- sensory codes for perception into similarly complex motor codes ability (14–17). HVC neurons produce highly stereotyped firing for generating action. To understand the neural substrates un- patterns during singing (18, 19), whereas LMAN neurons produce derlying this conversion, we study sensorimotor transformations highly variable patterns (15, 20, 21). in songbird cortical output neurons of a basal-ganglia pathway To gain insights into LMAN’s role in motor control, we consider involved in song learning. Despite the complexity of sensory and recent theoretical work that establishes a conceptual link between motor codes, we find a simple, temporally specific, causal corre- inverse models and vocal-auditory mirror neurons (22–24). We spondence between them. Sensory neural responses to song consider three (nonexhaustive) possibilities about the flow of playback mirror motor-related activity recorded during singing, sensory information into motor areas. First, auditory afferents with with a temporal offset of roughly 40 ms, in agreement with short some feature sensitivity could map onto motor neurons involved in feedback loop delays estimated using electrical and auditory generating those same features (Fig.
    [Show full text]
  • Term Memory by ©2017 Liying Li
    Drosophila CPEB, Orb2, a Putative Biochemical Engram of Long- term Memory By ©2017 Liying Li Submitted to the graduate degree program in Molecular and Integrative Physiology and the Graduate Faculty of the University of Kansas in partial fulfillment of the requirements for the degree of Doctor of Philosophy __________________________ Co-Chair: Dr. Kausik Si __________________________ Co-Chair: Dr. Paul Cheney __________________________ Dr. Tatjana Piotrowski __________________________ Dr. Hiroshi Nishimune __________________________ Dr. Matthew Gibson Date Defended: June 22nd, 2017 The Dissertation Committee for Liying Li certifies that this is the approved version of the following dissertation: Drosophila CPEB, Orb2, a Putative Biochemical Engram of Long- term Memory _______________________________ Co-Chair: Dr. Kausik Si _______________________________ Co-Chair: Dr. Paul Cheney Date Approved: July 18th, 2017 ii Abstract How a transient experience creates an enduring yet dynamic memory remains a fundamental unresolved issue in studies of memory. Experience-dependent aggregation of the RNA-binding protein CPEB/Orb2 is one of the candidate mechanisms of memory maintenance. Here, using tools that allow rapid and reversible inactivation of Orb2 protein I find that Orb2 activity is required for encoding and recall of memory. Blocking the Orb2 oligomerization process by interfering with the protein phosphorylation pathway or expressing an anti-amyloidogenic peptide impairs long-term memory. Facilitating Orb2 aggregation by a DNA-J family chaperone, JJJ2, enhances the animal’s capacity to form long-term memory. Finally, I have developed tools to visualize training-dependent aggregation of Orb2. I find that aggregated Orb2 in subset of mushroom body neurons can serve as a “molecular signature” of memory and predict memory strength.
    [Show full text]
  • Fodor and Aquinas: the Architecture of the Mind and the Nature of Concept Acquisition
    FODOR AND AQUINAS: THE ARCHITECTURE OF THE MIND AND THE NATURE OF CONCEPT ACQUISITION A Dissertation submitted to the Faculty of the Graduate School of Arts and Sciences of Georgetown University in partial fulfillment for the requirements for the degree of Doctor of Philosophy in Philosophy By Justyna M. Japola, M.A. Georgetown University, Washington DC December 2009 Copyright 2009 by Justyna M. Japola All Rights Reserved ii FODOR AND AQUINAS: THE ARCHITECTURE OF THE MIND AND THE NATURE OF CONCEPT ACQUISITION Justyna M. Japola, M.A. Thesis Advisor: Wayne A. Davis, Ph. D. ABSTRACT Fodor (1975 and 1981b) explains the paradigm empiricist method of concept acquisition as consisting in forming and testing hypotheses about objects that fall under a concept. This method, he notices, can only work for complex concepts, because we have to possess some concepts in order to form hypotheses. If so, then none of our simple (or primitive) concepts can be learned. If we still have them then they must be innate. Aquinas, on the other hand, is famous for his opposition to Platonic nativism, and is universally considered an empiricist with respect to cognition. In my dissertation I show that Fodor’s and Aquinas’s accounts of the architecture of the mind are quite similar. I argue that because one's position in the empiricism-nativism debate should be a function of one's account of the architecture of the mind, Fodor and Aquinas should be on the same side of the debate. My claim is that they should be on the side of nativism, but not the kind of radical concept nativism that Fodor is famous for.
    [Show full text]
  • Hopfield Networks
    Hopfield Networks Jessica Yarnall John Hopfield • Son of two physicists • Earned PhD in physics from Cornell University in 1958 • Currently a professor of molecular biology at Princeton University • Developed a model in 1982 to explain how memories are recalled by the brain Now known as the Hopfield model Hebbian Theory and Associative Memory Hebbian Theory was introduced in 1949 to explain associative learning “Neurons that fire together, wire together.” Associative memory is the ability to learn and remember the relationship between two unrelated things Hopfield Network • Network is trained to store a number of patterns or memories • Can recognize partial or corrupted information about a pattern and returns the closest pattern or best guess • Comparable to human brain, it has stability in pattern recognition Hopfield Network • Single-layered, recurrent network Neurons are fully connected • Symmetric weights Given neurons i and j, wij = wji • Neuron is either on (firing) or off (not firing) On corresponds to 1 Off corresponds to -1 Training • Hebbian learning rule: Incremental – does not require information about patterns previously learnt Local - uses information from both nodes whose connectivity weight is updated • Formula: • Where µ εx is the state of node x in pattern µ Updating Hopfield Networks • Generally update nodes asynchronously Node chosen randomly • Update Rule: • Example: • http://faculty.etsu.edu/knisleyj/neural/ Energy Function • The energy function either decreases or stays the same with asynchronous updating • Energy Function: • Converges to a local minima Problems with Hopfield Networks • The more complex the thing being recalled, the more pixels and weights you’ll need • Spurious states References • https://www.fi.edu/laureates/john-j-hopfield • https://humans101.wordpress.com/tag/associative-memory/ • https://www.doc.ic.ac.uk/~ae/papers/Hopfield-networks-15.pdf • https://www.doc.ic.ac.uk/~sd4215/hopfield.html • https://www.youtube.com/watch?v=gfPUWwBkXZY.
    [Show full text]
  • Researching the Lived Experience of Individuals Practicing Self-Directed Neuroplasticity
    St. Catherine University SOPHIA Master of Arts in Holistic Health Studies Research Papers Holistic Health Studies 5-2019 Changing Brains, Changing Lives: Researching the Lived Experience of Individuals Practicing Self-Directed Neuroplasticity Tim Klein St. Catherine University, [email protected] Beth Kendall St. Catherine University, [email protected] Theresa Tougas St. Catherine University, [email protected] Follow this and additional works at: https://sophia.stkate.edu/ma_hhs Part of the Alternative and Complementary Medicine Commons Recommended Citation Klein, Tim; Kendall, Beth; and Tougas, Theresa. (2019). Changing Brains, Changing Lives: Researching the Lived Experience of Individuals Practicing Self-Directed Neuroplasticity. Retrieved from Sophia, the St. Catherine University repository website: https://sophia.stkate.edu/ma_hhs/20 This Thesis is brought to you for free and open access by the Holistic Health Studies at SOPHIA. It has been accepted for inclusion in Master of Arts in Holistic Health Studies Research Papers by an authorized administrator of SOPHIA. For more information, please contact [email protected]. Running head: LIVED EXPERIENCE OF SELF-DIRECTED NEUROPLASTICITY Changing Brains, Changing Lives: Researching the Lived Experience of Individuals Practicing Self-Directed Neuroplasticity Tim Klein, Beth Kendall, & Theresa Tougas St. Catherine University May 22, 2019 LIVED EXPERIENCE OF SELF-DIRECTED NEUROPLASTICITY ii Acknowledgements We would like to collectively express heartfelt gratitude to the many people who made this project possible. We begin with our teacher and advisor, Dr. Carol Geisler, associate professor at St. Catherine University. Thank you, Carol, for your profound passion and wisdom, sharp insights, and spirited sense of humor. Because of you, we continue to see possibilities.
    [Show full text]