IENTIFIC SERIES ON Series A Vol. 43 EAR SCIENC Editor: Leon 0. Chua HI EMERGENT COMPUTATION IN CELLULAR NEURRL NETWORKS Radu Dogaru

World Scientific EMERGENT COMPUTATION IN CELLULAR NEURAL NETWORKS WORLD SCIENTIFIC SERIES ON NONLINEAR SCIENCE

Editor: Leon O. Chua University of California, Berkeley

Series A. MONOGRAPHS AND TREATISES Volume 27: The Thermomechanics of Nonlinear Irreversible Behaviors — An Introduction G. A. Maugin Volume 28: Applied Nonlinear Dynamics & Chaos of Mechanical Systems with Discontinuities Edited by M. Wiercigroch & B. de Kraker Volume 29: Nonlinear & Parametric Phenomena* V. Damgov Volume 30: Quasi-Conservative Systems: Cycles, Resonances and Chaos A. D. Morozov Volume 31: CNN: A Paradigm for Complexity L O. Chua Volume 32: From Order to Chaos II L. P. Kadanoff Volume 33: Lectures in Synergetics V. I. Sugakov Volume 34: Introduction to Nonlinear Dynamics* L Kocarev & M. P. Kennedy Volume 35: Introduction to Control of Oscillations and Chaos A. L. Fradkov & A. Yu. Pogromsky Volume 36: Chaotic Mechanics in Systems with Impacts & Friction B. Blazejczyk-Okolewska, K. Czolczynski, T. Kapitaniak & J. Wojewoda Volume 37: Invariant Sets for Windows — Resonance Structures, Attractors, Fractals and Patterns A. D. Morozov, T. N. Dragunov, S. A. Boykova & O. V. Malysheva Volume 38: Nonlinear Noninteger Order Circuits & Systems — An Introduction P. Arena, R. Caponetto, L Fortuna & D. Porto Volume 39: The Chaos Avant-Garde: Memories of the Early Days of Chaos Theory Edited by Ralph Abraham & Yoshisuke Ueda Volume 40: Advanced Topics in Nonlinear Control Systems Edited by T. P. Leung & H. S. Qin Volume 41: Synchronization in Coupled Chaotic Circuits and Systems C. W. Wu Volume 42: Chaotic Synchronization: Applications to Living Systems £. Mosekilde, Y. Maistrenko & D. Postnov

*Forthcoming |k I WORLD SCIENTIFIC SERIES ON r"%. _ . . ,, . ._ NONLINEAR SCIENCE % series A vol. 43 Series Editor: Leon O. Chua

EMERGENT COMPUTATION IN CELLU NEURAL NETWORKS

Radu Dogaru Polytechnic university of Bucharest, Romania

Vfe World Scientific «l New Jersey • London • SingaporeSi, • Hong Kong Published by World Scientific Publishing Co. Pte. Ltd. 5 Toh Tuck Link, Singapore 596224 USA office: Suite 202, 1060 Main Street, River Edge, NJ 07661 UK office: 57 Shelton Street, Covent Garden, London WC2H 9HE

British Library Cataloguing-in-Publication Data A catalogue record for this book is available from the British Library.

UNIVERSALITY AND EMERGENT COMPUTATION IN CELLULAR NEURAL NETWORKS Copyright © 2003 by World Scientific Publishing Co. Pte. Ltd. All rights reserved. This book, or parts thereof, may not be reproduced in any form or by any means, electronic or mechanical, including photocopying, recording or any information storage and retrieval system now known or to be invented, without written permission from the Publisher.

For photocopying of material in this volume, please pay a copying fee through the Copyright Clearance Center, Inc., 222 Rosewood Drive, Danvers, MA 01923, USA. In this case permission to photocopy is not required from the publisher.

ISBN 981-238-102-3

Printed by Fulsland Offset Printing (S) Pte Ltd, Singapore Preface

The Cellular Neural Network (CNN), introduced by Chua and Yang from University of California at Berkeley in the late 1980's is an attractive computational structure, particularly from the perspective of implementation in various micro and nanoelectronic technologies. The CNN paradigm includes cellular automata (CAs) as a particular case and in addition it borrows many ideas and techniques from the field of Neural Computation. Computation in CNNs is brain-like rather than "classic" in the sense of the widespread computing architectures based on microprocessors. Emergent computation, viewed as the class of both dynamic and static patterns of activity emerging in an array of interconnected cells which are meaningful for various information processing tasks, is the equivalent of programming classic computers. It is thus of a paramount importance to find the equivalent of the "programming rules" for such cellular devices. The following image can be suggestive to understand the difference between cellular and classic computation: Assume that we have an array of memory cells (i.e. a Random Access Memory). In a classic computer the cells are sequentially updated and located by the central processing unit via some external buses while in a cellular computer each memory cell exchanges information locally only with the neighboring cells. A "gene" associated with each cell controls the exchange of information with the cells in a neighborhood. The cells are updated in parallel and there is no central processing unit to control the cells. Like in a classical computer, the array of cells starts from an initial state, which contains the problem, and the solution will be found in the same array of cells after a period of time during which computation emerges. While the designer of a classic computer focuses on the central processing unit, on data coding, address buses and instruction sets, the designer of a cellular computer has to focus mostly on the cell. The "program" is now coded in what Leon Chua called "cells' gene" (i.e. the entire set of parameters defining the cell). Quite often all cells have identical structure and parameters and various tasks can be "programmed" on the same CNN chip by simply changing the genes. The following problems are raised to the cellular computer designer:

• To what degree is a cell capable to perform arbitrary local computations, i.e. the universality of a cell? • What is the choice of the cell parameters such that emergent computation will occur? More sharply, one would like to find the exact values of the parameters for a given information processing task.

This book provides original answers to the above questions. It introduces novel techniques to understand and control better universality and emergence in cellular neural networks. After an introductory chapter and a chapter providing the basics ideas and concepts necessary to understand the remainder of the book, the problem of universal local computation is extensively described in Chapter 3. Our solution, based on the theory of piecewise-linear function approximations, is compact and efficient for both binary and continuous cells. A systematic approach to the second problem, grounded by the very recent theory of local activity [Chua, 1998] is provided in Chapter 4. A set of analytic tools is developed to identify a specific sub-domain called an "edge of chaos"

V vi Preface domain in the cell parameter space such that the probability of emergent computation is maximized. Several examples of applying this method are provided. A measure for emergence in discrete time cellular systems is then introduced in Chapter 5 and then exemplified to identify several interesting behaviors in a cellular system, which is a mutation of the widely known "Game of Life". The importance of mutations and evolutionary approaches for designing cellular systems with emergent computation is then emphasized in the same chapter for a discrete time cellular system with continuous states. A potential application of emergent dynamic patterns for biometric authentication is presented in Chapter 6. Why emergent computation in cellular computers when the technology of programming and designing classic computers is so well established and prolific? Here are some possible answers: (i) The type of computation taking place in a cellular computer is the one used by living entities. Life itself is an emergent phenomenon and several examples in this book will show that simple living-like entities may emerge as a pattern of cellular activities. Brain-like computation could be better mimicked by a compact CNN rather than by a classic computer. Particularly when such computations are required in micro-robotics, or in any circumstance requiring compact yet intelligent sensor systems, the CNN could be a better choice; (ii) The cellular systems are highly parallel and consequently they perform several orders of magnitudes faster than classic (serial) computing ones. Tera-ops processing speed (1012 elementary operation per second) is common for the actual generation of CNNs; (iii) Recent developments in the area of nanotechnology indicate that cellular structures made of lattices of interconnected active devices (for example, resonant tunneling diodes, quantum dots or single electron transistors) could be easily developed. Characterized by a very high density of cells, they can fully exploit the benefits of emergent computation for various tasks in information processing.

Radu Dogaru, November 2002 Acknowledgements

Most of the results presented in this book are the outcome of research conducted by the author since 1996 when he joined the Nonlinear Electronics Laboratory at the University of California at Berkeley being sponsored by a Fulbright scholarship. Part of this research was sponsored by several scholarships offered by the ISIA (The Graduate School in Intelligent Systems) at the Technical University of Darmstadt - Germany and by several Office of Naval Research (U.S.A.) grants. I express my gratitude to the Volkswagen Stiftung foundation who partially supported some of the research included in this book through its program "Cooperation with Natural and Engineering Scientists in Central and Eastern Europe". This book is in a significant part the result of the support and encouragement from the exceptional professor and scientist Leon O. Chua, director of the Nonlinear Electronics Laboratory at the University of California at Berkeley. Many thanks to my Berkeley colleagues and friends, Dr. Kenneth Crounse, Dr. Martin Haenggi, Dr. Pedro Julian, Tao Yang and many others who contributed a great deal to shape and improve some of the ideas exposed in this book. I am also deeply grateful to Professor Manfred Glesner, Chair of the Microelectronics Institute at Technical University of Darmstadt (Germany) for the opportunity he offered me to join a highly competitive research team. I am thankful for the valuable support of Tudor Murgan, Mihail Petrov, and Octavian Mitrea in the last stages of preparation of the manuscript. My family played an important role during the various stages of writing this book. I am grateful to my wife Ioana, for her love, patience and valuable support in the last stages of writing this book. I dedicate this book to the memory of my father Tudor, who enthusiastically encouraged me in most of my professional endeavors, this book being unfortunately the last where his direct encouragement reached me. Last but not least, I wish to express my gratitude to Professor Felicia Ionescu and many other colleagues within the Department of Applied Electronics and Information Engineering and within the "Politehnica" University of Bucharest who expressed interest in my work.

vn This page is intentionally left blank Contents

1. Introduction 1 1.1. Emergent computation as a universal phenomena 1 1.2. Emergence 2 1.3. Cellular computing systems 2 1.4. Universality 4 1.5. Designing for emergence, the essence of this book 4 1.6. Detecting the potential for emergence: the local activity theory 6

2. Cellular Paradigms: Theory and Simulation 9 2.1. Cellular systems 9 2.2. Major cellular systems paradigms 11 The Cellular Neural Network (CNN) model 12 The Generalized Cellular Automata 13 Reaction-Diffusion Cellular Nonlinear Networks 14 2.3. Matlab simulation of generalized cellular automata 15 Uncoupled GCAs 15 Coupled GCAs 17 Simulation of standard cellular neural networks 18 2.4. Simulation of Reaction-Diffusion Cellular Neural Networks 18 2.5. Concluding remarks 20

3. Universal Cells 21 3.1. Universality and cellular computation, basic ideas 21 Boolean universal cells 22 The simplicial cell - universality expanded to continuous states 22 3.2. Binary cells 23 3.2.1. What would be an "ideal" binary CNN cell? 24 Universality 24 Compactness 24 Robustness 25 Capability of evolution 26 3.2.2. Orientations and Projection Tapes 26 Local binary computation 26 Projections 28 Orientations 29 Projection tapes 30 Default orientations 31 Valid and non-valid projection tapes 32 Transitions and robust transitions 35 Finding the optimal orientation 38 Optimal orientations for totalistic and semi-totalistic Boolean functions 42 3.2.3. Universal cells with canonical discriminants 46 3.2.4. Compact universal cells with multi nested discriminants 48 Bifurcation tree for multi-nested discriminant function 49 Uniform multi-nested cells and their bifurcation trees 52

IX x Contents

The uniform multi-nested discriminant as an analog-to-digital converter 52 Uniform orientations and projection tapes 53 Boolean realizations: an analytic approach 53 Finding the genes for arbitrary Boolean functions 58 Other random search methods 68 3.3. Continuous state cells 73 3.3.1. Overview 73 3.3.2. Some theoretical issues on simplicial neural cells 79 Relationships with fuzzy logic 79 Training and testing samples 79 Quantization of gene's coefficients 80 3.3.3. Circuit implementation issues 81 Considerations regarding the implementation of the local Boolean logic 82 Software implementations 83 3.3.4. A general procedure for training the simplicial cell 84 3.3.5. Functional capabilities and applications 84 Square scratch removal 84 Median Filters 86 Edge detection 88 Pattern classification 89 3.3.6. Nonlinear expansion of the input space 90 3.3.7. Comparison with multi-layer perceptrons 91 3.4. Concluding remarks 93

4. Emergence in Continuous-Time Systems: Reaction-Diffusion Cellular Neural Networks 95 4.1. The theory of local activity as a tool for locating emergent behaviors 95 4.2. Narrowing the search, "Edge of chaos" domains 96 4.3. The methodology of finding "edge of chaos" domains 98 4.3.1. Four steps precluding the local activity testing 98 4.3.2. The concept of local activity 102 4.3.3. Testing for stable and unstable local activity 103 Local activity test for one diffusion coefficient 104 Local activity test for two diffusion coefficients 108 4.3.4. Unrestricted versus restricted local activity, the edge of chaos 109 Unrestricted local activity and passivity 109 The Edge of Chaos 112 4.3.5. Bifurcation diagrams 118 One-diffusion coefficient case 118 The two-diffusion case 121 4.3.6. Emergent behaviors near and within the "edge of chaos" 121 Mapping the Edge of Chaos 122 Static and dynamic patterns on the Edge of Chaos 125 Homogeneous static patterns 128 Turing-like patterns 129 Spiral wave patterns 136 Information computation patterns 137 Periodic dynamic patterns 139 Contents xi

Chaotic dynamic patterns 139 4.4. Emergent behaviors within the "edge of chaos" domains of the Brusselator CNN 143 4.4.1. Local Activity and Edge of Chaos Domains 143 The cell parameter projection profile (CPPP) 146 Bifurcation diagrams 146 Local activity and edge of chaos domain, two diffusion coefficients 148 4.4.2. Emergence at the edge of chaos 150 The one-diffusion coefficient case 150 Dynamic patterns 153 Static patterns 157 The two-diffusion coefficients case 159 4.5. Emergent behaviors within the "edge of chaos" domains of the Gierer-Meinhardt CNN 160 4.5.1. Local Activity and Edge of Chaos Domains 160 The cell parameter projection profile (CPPP) 165 Bifurcation diagrams 165 Local activity and edge of chaos domain, two diffusion coefficients 167 4.5.2. Emergence at the edge of chaos 168 The one-diffusion coefficient case 168 Static patterns 173 Dynamic patterns 173 The two-diffusion coefficients case 176 4.6. Concluding remarks and engineering perspectives 177 Heuristics in determining cell parameter points leading to emergence 178 Uncertainties in locating emergent behaviors 179 Engineering perspectives and potential applications of the edge of chaos analysis 181

Emergence in Discrete-Time Systems: Generalized Cellular Automata 183 5.1. Emergent phenomena in binary cellular automata 184 5.1.1. Introduction 184 5.1.2. Generalized Cellular Automata 187 Uncoupled GCA 189 Generalized Cellular Automata for the "Game of Life" 190 5.1.3. Failure boundaries and paving stones in the cell's parameter space 191 5.1.4. Locating emergent behaviors and their dynamics 193 Cellular Disorder Measure 195 Three Qualitative Classes of Dynamic Behaviors 196 Examples of dynamic behaviors 198 5.2. Emergent phenomena in continuous state cellular automata 210 5.2.1. Introduction 210 5.2.1. The cellular model 211 5.2.2. Emergence of autopoietic patterns by mutations in the "game of life" 211 The influence of initial condition 212 The influence of cells parameters 215 Other cell models 216 5.3. Emergence in coupled generalized cellular automata 218 xii Contents

5.3.1. Introduction 218 5.3.2. The cellular model 218 5.3.3. An application in pattern detection from extremely noisy background 220 5.4. Concluding remarks 222

6. Unconventional Applications: Biometric Authentication 225 6.1. Introduction 225 6.2. Generating the stimuli 227 6.3. Experimental results 229 6.4. Concluding remarks 232

References 233

Index 245 Chapter 1 Introduction

1.1. Emergent computation as a universal phenomena

Information exchange is all around us. According to Varela [Varela et al., 1974] life itself is a process of perpetual cognition (or information exchange with the environment). Computer scientists often call information processing and exchange with the word computation, which essentially express the idea of manipulating and transforming information in such a way that is meaningful for a certain purpose or objective. A pragmatic attribute is attached to the meaning of the word "computation" in that we often associate the word not only with the process itself but also with the effective means of achieving it. Therefore, the derived word computer would usually define a medium (electronic, mechanical, bio-chemical, or of other physical nature) used to achieve computation as an emergent property. Starting with the simple abacus, passing trough mechanical computing systems such as Babbage's first computer, until the nowadays digital computers, humans are always in a search for improved and sophisticated computing methods and mediums to satisfy their needs. This process of developing computing machines is a universal and a natural one, which obeys certain life laws according to which life itself is sustained by a continuous information exchange with the external world. It appears that while life itself is the result of a continuous flow of information exchange (or computation in a wider sense), hierarchies of other (superior) "living" entities develop as a result of information exchange between similar entities. So, simple cells collaborate to produce and maintain functional organs, which then are linked in a network forming individuals. Such individuals then form families, villages, countries, federations, etc. Note that in this picture, at a given level of the hierarchy, the actors (let us call them cells) are quite similar and they do usually exchange information only within a very small fraction of the entire population. This particular fraction constitutes a neighborhood, which is essentially of informational nature although some topological constraints may have an influence on which members belong to a neighborhood (for example, a group of scientists usually collaborate through e-mail although they are located geographically at very distances). The same structure is reflected in our brains. Our brain operates as an emergent computer where a network of concepts and relationships are stored. Following the above model, one concept is usually inter-related with only a few other concepts in its neighborhood. Sequentially firing the neurons associated with a mental concept or with a sensorial stimulus will allow the recall of a related concept in several "iterations" corresponding to the same number of neighborhoods activated in the internal network of concepts. A similar hierarchical and cellular organization is revealed in language, which could be considered an "image of our brains". Basic cells (phonemes or characters in the written language) collaborate in an emergent manner to form words, which then are again combined in phrases and so on providing a structured reflection of the outer world. Similar cellular models could be provided for networks of similar electronic circuits, networks of computers and so on. Recapitulating, universal characteristics of our world can be captured in the emergent dynamics of a cellular model where a population of similar cells exchanges information locally with the cells in their neighborhood. Emergent computation depends on how information is exchanged and processed in a cell neighborhood.

1 2 Universality and Emergent Computation in Cellular Neural Networks

1.2. Emergence

The dictionary' definition of emergence is based on the verb to emerge, which is defined as "to come out from inside or from being hidden". Yet, from a scientific point of view is rather difficult to come up with a definition for emergence. In a 1990 posting over the newsgroup comp.ai.philosophy David Chalmers proposes the following definition: "Emergence is the phenomenon wherein complex, interesting high-level function is produced as a result of combining simple low-level mechanisms in simple ways" while suggesting that "emergence is a psychological property". Indeed, according to Chalmers "Properties are classed as emergent based at least in part on (1) the interestingness to a given observer of the high-level property at hand; and (2) the difficulty of an observer's deducing the high-level property from low-level properties". Finally Chalmers proposes the following definition: "Emergence is the phenomenon wherein a system is designed according to certain principles, but interesting properties arise that are not included in the goals of the designer. Following the above line, Ronald [Ronald et al. 1999] proposes an emergence test based on a surprise effect and somehow similar to the Turing test for detecting intelligence. According to them, if using a (macro) language L2 to describe the global behavior of a system, designed according to a (micro) language LI (usually providing information about the cell structure and its local interconnectivity) the behaviors observed in L2 are non- obvious to the observer - who therefore experiences surprise - one may conclude that the global behavior is an emergent one. Standish [Standish, 2001] develops the ideas of Ronald observing that the surprise effect in the above test disappears after the first observation of an emergent behavior. Then he proposes the following definition of emergence: "An emergent phenomenon is simply one that is described by atomic concepts available in the macro language, but cannot be so described in the micro language".

1.3. Cellular computing systems

The above observations are linked to the idea of developing a computing paradigm called cellular computing. The structure of such a computer is composed of a grid (often two-dimensional) of cells locally interconnected with their neighbors. Each cell can be in a number of states (ranging from 2 to infinity) and the state of a cell depends by itself and by the states of its neighbors through a nonlinear functional, which can be defined in different ways. This functional is associated with a practical implementation of the cell and it includes a set of tunable parameters. By similarity to biology, where the overall functionality of a living being is determined by a finite string of proteins, in [Chua, 1998] it was proposed that the set of tunable parameters will be called a gene. By tuning the gene parameters one can achieve programmability i.e. different behaviors within the same basic cellular architecture. The cell assumes an initial state and may have one or more external inputs. In a cellular computer, computation can be considered any form of meaningful emergent global phenomenon resulted from a proper design of the cell functional (gene). Usually the initial state and the inputs code the problem to be solved while the answer to this problem is coded in an equilibrium steady state but it can be also coded in the emergent dynamics of the cellular system. By meaningful I mean a subjective definition according to

Longman dictionary of contemporary English, special edition, 1992.