INTEGER SPARSE DISTRIBUTED MEMORY AND MODULAR COMPOSITE REPRESENTATION by Javier Snaider A Dissertation Submitted in Partial Fulfillment of the Requirements for the Degree of Doctor of Philosophy Major: Computer Science The University of Memphis August, 2012 Acknowledgements I would like to thank my dissertation chair, Dr. Stan Franklin, for his unconditional support and encouragement, and for having given me the opportunity to change my life. He guided my first steps in the academic world, allowing me to work with him and his team on elucidating the profound mystery of how the mind works. I would also like to thank the members of my committee, Dr. Vinhthuy Phan, Dr. King-Ip Lin, and Dr. Vasile Rus. Their comments and suggestions helped me to improve the content of this dissertation. I am thankful to Penti Kanerva, who introduced the seminal ideas of my research many years ago, and for his insights and suggestions in the early stages of this work. I am grateful to all my colleagues at the CCRG group at the University of Memphis, especially to Ryan McCall. Our meetings and discussions opened my mind to new ideas. I am greatly thankful to my friend and colleague Steve Strain for our discussions, and especially for his help editing this manuscript and patiently teaching me to write with clarity. Without his amazing job, this dissertation would hardly be intelligible. I will always be in debt to Dr. Andrew Olney for his generous support during my years in the University of Memphis, and for being my second advisor, guiding me academically and professionally in my career. My most sincere thanks to you. To my sons, Gaston and Adam, who have accepted having a cybernetic father for the last two years, and for being the source of my commitment to show them that they ii have to follow their dreams no matter the obstacles in the path, or the effort and energy that they demand. I love you guys. Finally, to my wife, Ester Bruden, who has been my greatest supporter. She believed in me long before I believed in myself. Without her unconditional love, support, and encouragement I would have never started this life-changing journey. I am so fortunate to share my life with her. Ester, you raise me up to more than I can be1. I love you. 1 From “You Raise Me Up” by Brendan Graham. iii Abstract Javier Snaider. Ph.D. The University of Memphis. Aug/2012. Integer Sparse Distributed Memory and Modular Composite Representation. Major Professor: Stan P. Franklin. Challenging AI applications, such as cognitive architectures, natural language understanding, and visual object recognition share some basic operations including pattern recognition, sequence learning, clustering, and association of related data. Both the representations used and the structure of a system significantly influence which tasks and problems are most readily supported. A memory model and a representation that facilitate these basic tasks would greatly improve the performance of these challenging AI applications. Sparse Distributed Memory (SDM), based on large binary vectors, has several desirable properties: auto-associativity, content addressability, distributed storage, robustness over noisy inputs that would facilitate the implementation of challenging AI applications. Here I introduce two variations on the original SDM, the Extended SDM and the Integer SDM, that significantly improve these desirable properties, as well as a new form of reduced description representation named MCR. Extended SDM, which uses word vectors of larger size than address vectors, enhances its hetero-associativity, improving the storage of sequences of vectors, as well as of other data structures. A novel sequence learning mechanism is introduced, and several experiments demonstrate the capacity and sequence learning capability of this memory. Integer SDM uses modular integer vectors rather than binary vectors, improving the representation capabilities of the memory and its noise robustness. Several iv experiments show its capacity and noise robustness. Theoretical analyses of its capacity and fidelity are also presented. A reduced description represents a whole hierarchy using a single high- dimensional vector, which can recover individual items and directly be used for complex calculations and procedures, such as making analogies. Furthermore, the hierarchy can be reconstructed from the single vector. Modular Composite Representation (MCR), a new reduced description model for the representation used in challenging AI applications, provides an attractive tradeoff between expressiveness and simplicity of operations. A theoretical analysis of its noise robustness, several experiments, and comparisons with similar models are presented. My implementations of these memories include an object oriented version using a RAM cache, a version for distributed and multi-threading execution, and a GPU version for fast vector processing. v Table of Contents ! 1! Introduction ...................................................................................................... 1! High dimensional vector spaces........................................................................ 11! Parallel computing becoming cheap ................................................................. 13! Contributions of this Work ............................................................................... 14! Structure of this Dissertation ............................................................................ 16! 2! Sparse Distributed Memory ............................................................................ 18! Mathematical Background ................................................................................ 21! Memory Description ......................................................................................... 28! SDM compared with other models ................................................................... 38! Extensions and improvements .......................................................................... 43! Applications ...................................................................................................... 47! 3! Vector Representation .................................................................................... 53! Reduced Descriptions ....................................................................................... 58! Basic Operations to Combine Vectors .............................................................. 61! Spatter Code ...................................................................................................... 66! Holographic Reduced Representation............................................................... 68! Hyperdimensional Computing .......................................................................... 71! Other Models .................................................................................................... 78! vi 4! Extended Sparse Distributed Memory ............................................................ 81! Sequence Learning ............................................................................................ 82! Extended SDM .................................................................................................. 88! Storing sequences and other data structures ..................................................... 91! Simulations and Experiments ........................................................................... 94! Conclusions ..................................................................................................... 104! 5! Integer Sparse Distributed Memory ............................................................. 105! Integer Sparse Distributed Memory ................................................................ 107! Radius of the Access Sphere ........................................................................... 112! Fidelity and Capacity ...................................................................................... 113! Experiments and Results ................................................................................. 123! Extensions ....................................................................................................... 129! Conclusions ..................................................................................................... 131! 6! Modular Composite Representation ............................................................. 133! Modular Integer Vectors ................................................................................. 134! Manhattan Distance in a Modular Space ........................................................ 137! Basic Operations ............................................................................................. 141! Hyperdimensional Computing with Composite Modular Representation ...... 149! Normalized Distance and Similarity ............................................................... 156! Expected Value and Variance of the Similarity of Selected Expressions ...... 158! Summary of Comparisons: MCR, HRR and Spatter Code ............................. 163! Conclusions ..................................................................................................... 164! vii 7! Implementations ........................................................................................... 166! Object Oriented Design................................................................................... 168! Cached Implementation .................................................................................