University of New Mexico UNM Digital Repository Electrical and Computer Engineering ETDs Engineering ETDs 8-28-2012 A new class of neural architectures to model episodic memory : computational studies of distal reward learning Shawn Taylor Follow this and additional works at: https://digitalrepository.unm.edu/ece_etds Recommended Citation Taylor, Shawn. "A new class of neural architectures to model episodic memory : computational studies of distal reward learning." (2012). https://digitalrepository.unm.edu/ece_etds/247 This Dissertation is brought to you for free and open access by the Engineering ETDs at UNM Digital Repository. It has been accepted for inclusion in Electrical and Computer Engineering ETDs by an authorized administrator of UNM Digital Repository. For more information, please contact [email protected]. i A New Class of Neural Architectures to Model Episodic Memory: Computational Studies of Distal Reward Learning by Shawn E. Taylor B.S.E.E., New Mexico Institute of Mining and Technology, 2001 M.S.E.E., University of New Mexico, 2006 DISSERTATION Submitted in Partial Fulfillment of the Requirements for the Degree of Doctor of Philosophy Engineering The University of New Mexico Albuquerque, New Mexico July, 2012 c 2012, Shawn E. Taylor iii Dedication To my children: may the ever expanding work of developing academics, of which this contribution is but a tiny part, make it possible to experience your world with greater understanding. iv Acknowledgments I would like to thank my advisor, Professor Thomas Caudell, for his support and guidance. I would also like to thank the sources of financial support through my graduate eduction; the National Science Foundation, the Defense Threat Reduction Agency, and Sandia National Labs. Numerous people have been instrumental in making the details of my relationships with those entities work. A broad stroke is unfair for all the people who enabled that support, without you nothing would get done. Moving into the realm of professional and academic colleagues, I would like to thank Michael Bernard for opportunities and insights, and Howard Eichenbaum, and Neal Cohen for insights that enriched this research greatly. My dissertation committee, including Michael Healy, deserves praise for the work that has gone into enabling this to be a successful dissertation. Finally, thank you to my wife for putting up with an apparently never ending stream of work that demanded my time. v A New Class of Neural Architectures to Model Episodic Memory: Computational Studies of Distal Reward Learning by Shawn E. Taylor B.S.E.E., New Mexico Institute of Mining and Technology, 2001 M.S.E.E., University of New Mexico, 2006 PhD, Engineering, University of New Mexico, 2012 Abstract A computational cognitive neuroscience model is proposed, which models episodic memory based on the mammalian brain. A computational neural architecture in- stantiates the proposed model and is tested on a particular task of distal reward learning. Categorical Neural Semantic Theory informs the architecture design. To experiment upon the computational brain model, embodiment and an environment in which the embodiment exists are simulated. This simulated environment realizes the Morris Water Maze task, a well established biological experimental test of distal reward learning. The embodied neural architecture is treated as a virtual rat and the environment it acts in as a virtual water tank. Performance levels of the neu- ral architectures are evaluated through analysis of embodied behavior in the distal reward learning task. Comparison is made to biological rat experimental data, as well as comparison to other published models. In addition, differences in perfor- mance are compared between the normal and categorically informed versions of the architecture. vi Contents List of Figures xiv List of Tables xxvii 1 Introduction 1 1.1 Motivation................................. 1 1.2 Overview.................................. 2 1.2.1 ProblemStatement ........................ 2 1.2.2 ProposedArchitecture . .. .. 3 1.2.3 DissertationResearchElements . 4 1.2.4 DocumentStructure ....................... 6 2 Background 8 2.1 DistalReward............................... 9 2.1.1 Characterization ......................... 9 2.1.2 OtherDistalRewardModels. 11 2.2 EpisodicMemory ............................. 28 2.3 Neuroscience and Cognitive Psychology Background on Hippocampus 29 2.3.1 HumanCases ........................... 30 2.3.2 Animal Evidence for Hippocampal Function . 31 2.3.3 Episodic Memory Experiments . 32 2.4 DopaminergicSubsystem. 35 vii Contents 2.5 Experimental Results of Computational Cortical Hippocampal Model 36 2.6 CategoryTheory ............................. 36 2.7 ART, fuzzy ART, and fuzzy LAPART . 39 2.7.1 ART................................ 39 2.7.2 FuzzyART ............................ 42 2.7.3 LAPART ............................. 44 2.7.4 FuzzyLAPART.......................... 45 2.8 TemporalEpisodes ............................ 46 2.8.1 TemporalIntegrator . 46 2.8.2 TemporalSequenceLearning. 46 2.8.3 Background of Artificial Neural Temporal Methods . 48 2.8.4 Top-DownRecall ......................... 57 3 Apparatus: Embodied Environment and Neural Architecture 60 3.1 NeuralSimulator ............................. 60 3.2 MorrisWaterMaze ............................ 61 3.3 Embodiment................................ 62 3.3.1 Embodied Agent and Flatworld . 62 3.3.2 NeuralArchitecture. 63 3.3.3 ArchitectureParameters . 69 3.3.4 Expected Reward Visualization . 71 3.3.5 ModelAssumptions........................ 74 3.4 WaterMazeEnvironment . .. .. 75 3.5 Architectural Addition of Categorical Limits . 77 4 Approach 79 4.1 DistalRewardLearning ......................... 79 4.2 PersistencetoGoal ............................ 83 4.3 RapidTransferLearning . .. .. 84 viii Contents 4.4 Comparison of Baseline to Limits Architectures . 86 5 Results 88 5.1 BaselineArchitecture........................... 90 5.1.1 DistalRewardLearning . .. .. 90 5.1.2 PersistencetoGoal ........................ 93 5.1.3 RapidTransferLearning . 95 5.2 CategoricalLimitsArchitecture . 99 5.2.1 DistalRewardLearning . .. .. 99 5.2.2 PersistencetoGoal . .. .. 100 5.2.3 RapidTransferLearning . 104 5.3 Comparison of Baseline to Limits Architectures . 106 5.3.1 DistalRewardLearning . 106 5.3.2 PersistencetoGoal . .. .. 107 5.3.3 RapidTransferLearning . 108 5.4 Summary ................................. 109 6 Discussion 110 6.1 Statistical Analysis of Baseline Architecture . 110 6.1.1 DistalRewardLearning . 110 6.1.2 PersistencetoGoal . .. .. 111 6.1.3 RapidTransferLearning . 112 6.2 Statistical Analysis of Categorical Limit Architecture . 113 6.2.1 DistalRewardLearning . 113 6.2.2 PersistencetoGoal . .. .. 114 6.2.3 RapidTransferLearning . 115 6.3 Comparison of Baseline Architecture to Categorical Limit Architecture 116 6.3.1 DistalRewardLearning . 116 6.3.2 PersistencetoGoal . .. .. 116 ix Contents 6.3.3 RapidTransferLearning . 117 6.4 Comparison to Biological Experiments . 118 6.4.1 PersistencetoGoal . .. .. 120 6.4.2 RapidTransferLearning . 120 6.5 ComparisontoOtherModels . 120 6.5.1 Izhikevich ............................. 121 6.5.2 SamsonovichandAscoli . 122 6.5.3 McKinstryetal. ......................... 123 6.5.4 Dolle et al. 124 6.5.5 WeberandTriesch ........................ 125 6.5.6 Martinetetal. .......................... 125 7 Summary and Conclusion 127 8 Future Work 131 8.1 Structure ................................. 131 8.2 Feedback in Reward Realization Motor Plan . 132 8.3 Morris Water Maze Rat Platform Placement After Retrieval . 133 8.4 Decouple Environment Learning from Distal Reward Learning . 134 8.5 Hippocampus ............................... 135 8.6 FurtherBrainModeling ......................... 136 8.7 DevelopmentalPerception . 136 8.8 HighPerformanceComputing . 141 8.9 Robotics.................................. 142 A Introduction to Category Theory 143 A.1 Category Theory from Thirty Thousand Feet . 143 A.2 ACommentonStructure. 144 A.3 CategoryTheoryfromFiveThousandFeet . 146 x Contents A.3.1 Objects .............................. 146 A.3.2 Morphisms............................. 147 A.3.3 CommutativeDiagrams . 147 A.3.4 Functors.............................. 148 A.3.5 NaturalTransformations . 150 A.3.6 Limits ............................... 150 A.4 ConceptualCapacity . .. .. 152 A.5 Theoretical Bounds on Categorical Concept Capacity . 161 A.6 Categorical Encoding of Conceptual Structure . 164 B Categorical Limits 168 B.1 LimitsART Implements Limits and Colimits . 168 B.1.1 The Colimit Representations . 169 B.1.2 TheLimitRepresentations . 171 B.1.3 Limit Representations Control Vigilance . 173 B.1.4 Apparatus............................. 174 B.1.5 Experiment ............................ 175 C Extended Background Material 176 C.1 Hippocampus ............................... 176 D Computational Cortical Hippocampal Model 181 D.1 EpisodicMemory ............................. 181 D.2 CorticalModel .............................. 181 D.3 HippocampalModel ........................... 182 D.3.1 EntorhinalCortex. 184 D.3.2 DentateGyrus .......................... 185 D.3.3 CA3 ................................ 186 D.3.4 CA1andSubiculum ....................... 188 xi Contents D.3.5 Computational Cortical Hippocampal Process . 189 D.4 ExperimentalResults. 200 D.4.1 AssociatingObject/ScenesPairs. 200 D.4.2 Co-occurrence of Shared Scenes with Novel Objects . 204 E LimitsART Experiment 214 E.0.3 ART and Colimits . 214 E.1 ExperimentalProcedure . 218 E.2 ExperimentalResults.