ISSN: 1402-1544 ISBN 978-91-7583-XXX-X Se i listan och fyll i siffror där kryssen är DOCTORAL T H E SIS BLERIM EMRULI Department of Computer Science, Electrical and Space Engineering Division of EISLAB ISSN 1402-1544 Ubiquitous Cognitive Computing: ISBN 978-91-7583-003-2 (print) ISBN 978-91-7583-004-9 (pdf) A Vector Symbolic Approach Ubiquitous Cognitive Computing: A Vector Symbolic Approach Symbolic Vector A Computing: Ubiquitous Cognitive Luleå University of Technology 2014 BLERIM EMRULI Ubiquitous Cognitive Computing: A Vector Symbolic Approach BLERIM EMRULI EISLAB Luleå University of Technology Luleå, Sweden Supervisors: Jerker Delsing, Fredrik Sandin and Lennart Gustafsson Printed by Luleå University of Technology, Graphic Production 2014 ISSN 1402-1544 ISBN 978-91-7583-003-2 (print) ISBN 978-91-7583-004-9 (pdf) Luleå 2014 www.ltu.se To my parents Abstract A wide range of physical things are currently being integrated with the infrastructure of cyberspace in a process that is creating the so-called Internet of Things. It is expected that Internet-connected devices will vastly outnumber people on the planet in the near future. Such devices need to be easily deployed and integrated, otherwise the resulting systems will be too costly to configure and maintain. This is challenging to accomplish using conventional technology, especially when dealing with complex or heterogeneous systems consisting of diverse components that implement functionality and standards in different ways. In addi- tion, artificial systems that interact with humans, the environment and one-another need to deal with complex and imprecise information, which is difficult to represent in a flexible and standardized manner using conventional methods. This thesis investigates the use of cognitive computing principles that offer new ways to represent information and design such devices and systems. The core idea underpinning the work presented herein is that functioning systems can potentially emerge autonomously by learning from user interactions and the environment provided that each component of the system conforms to a set of general information-coding and communication rules. The proposed learning approach uses vector-based representations of information, which are common in models of cognition and semantic spaces. Vector symbolic architectures (VSAs) are a class of biology-inspired models that represent and manipulate structured rep- resentations of information, which can be used to model high-level cognitive processes such as analogy-making. Analogy-making is a central element of cognition that enables animals to identify and manage new information by generalizing past experiences, possibly from a few learned examples. The work presented herein is based on a VSA and a binary associative memory model known as sparse distributed memory. The thesis outlines a learning architecture for the au- tomated configuration and interoperation of devices operating in heterogeneous and ubiq- uitous environments. To this end, the sparse distributed memory model is extended with a VSA-based analogy-making mechanism that enables generalization from a few learned examples, thereby facilitating rapid learning. The thesis also presents a generalization of random indexing, which is an incremental and lightweight feature extraction method for streaming data that is commonly used to generate vector representations of semantic spaces. The impact of this thesis is twofold. First, the appended papers extend previous theoret- ical and empirical work on vector-based cognitive models, in particular for analogy-making and learning. Second, a new approach for designing the next generation of ubiquitous cog- nitive systems is outlined, which in principle can enable heterogeneous devices and systems to autonomously learn how to interoperate. Contents Part I 1 Introduction 3 1.1 Problem formulation ................................. 5 1.2 Delimitations ..................................... 7 1.3 Methodology ..................................... 7 1.4 Thesis outline ..................................... 7 2 Cognitive computation 9 2.1 Highlights of the last 60 years ............................ 9 2.2 Desirable properties of cognitive computation .................. 13 2.3 The geometric approach to cognition ........................ 16 2.4 New challenges .................................... 17 3 Representation, memory and analogy 19 3.1 Representation ..................................... 19 3.2 Vector symbolic architectures ............................ 22 3.3 Encoding a distributed representation ....................... 26 3.4 Memory ........................................ 27 3.5 Analogical mapping with holistic vectors ..................... 30 4 Conclusions and future work 33 4.1 Conclusions ...................................... 33 4.2 Future work ...................................... 35 References 37 Part II Paper A 53 Paper B 81 Paper C 103 Paper D 131 Preface This thesis is the result of five years’ effort. In keeping with the tradition, this doctoral thesis is a compilation thesis that consists of two parts. Part I presents a general introduction and description of the problem that is addressed, and outlines some of the key ideas and concepts needed to understand the appended papers in Part II. The research presented in this thesis is interdisciplinary in nature. As such, a comprehensive introduction to each relevant disci- pline would be beyond the scope of Part I. Consequently, I focus only on those aspects and issues that I consider essential to understand the papers in Part II, and motivate the rationale and the approach behind the work presented in this thesis. I found the studies that resulted in the production of this thesis to be rewarding and satisfying, mainly for two reasons. First, because they enabled me to use a mathematical approach while exploring the intersection of several different disciplines including cogni- tive science, artificial intelligence, information theory and ubiquitous computing. Second, because I had the benefit of collaborating with knowledgeable and inspiring people at all stages while continuously being challenged to further my development as an independent and responsible researcher. The work has generated many useful experiences and has al- ready been recognized by some pioneers in the field as innovative and stimulating, which has resulted in co-authored publications. I joined EISLAB after finishing an M.Sc. in computer engineering with specialization in applied artificial intelligence and a B.Sc. in computer science. Immediately after joining the lab (and also during my recruitment process) I was introduced to the concept that Internet- connected sensors and actuators are being embedded everywhere and in everything, and the great challenges and opportunities that are being created as a result. At that time, most workers in the field and at EISLAB were focused on aspects such as communication, sensing and energy harvesting, which are very important topics even today. Moreover, EISLAB had done some very interesting work on using these Internet-connected sensors and actuators in human, infrastructure and environmental monitoring, home automation, and intelligent transportation systems. However, many of the existing applications are not adaptive and depend heavily on human labor. From this perspective, my supervisor Jerker Delsing has argued that we must start thinking about how we can enable the Internet of Things to per- form more brainlike computation in order to address the growing challenges it presents. Together with my assistant supervisor Fredrik Sandin, at the time a newly hired postdoc with a background in Physics, we set out to determine how this vision could be realized by reviewing the literature and critically thinking about how to approach the problem. After having spent several months and a great deal of effort exploring biologically plausible mod- els for information processing and psychologically inspired models of cognition, we decided to investigate a hybrid approach that integrates some key aspects of both these approaches in a single mathematical framework. X Preface There are many people who have helped me to reach this point and I owe them all a debt of gratitude. First, I would like to thank my supervisors Jerker Delsing, Fredrik Sandin and Lennart Gustafsson. I thank Jerker for his support and guidance; Fredrik for being my day-to-day supervisor, and for always challenging me to think deeper and reach further; and Lennart for believing from the very beginning that I am well prepared and strongly motivated to pursue research and education. I also thank my co-authors, Ross Gayler and Magnus Sahlgren, for their inputs, sugges- tions and enlightening discussions. In addition, I thank Chris Eliasmith for never hesitating to take the time to answer my questions, and Serge Thill for his helpful comments. The work for this thesis was carried out mostly at the Department of Computer Science, Electrical and Space Engineering of Luleå University of Technology in Luleå, Sweden. I ex- tend my sincere thanks to everyone in the department for providing a pleasant working at- mosphere. During the work leading up to this thesis I was fortunate to have the opportunity to visit other research groups. I am thankful to Asad Khan and Andrew Paplinski for hosting me during my visit at Monash University in Australia, and Pentti Kanerva and Bruno Olshausen for hosting me during my visit at the University of California, Berkeley. Furthermore, I thank all the