Biol. Cybern. 75, 229–238 (1996) Biological Cybernetics c Springer-Verlag 1996 A complex-valued associative memory for storing patterns as oscillatory states Srinivasa V. Chakravarthy, Joydeep Ghosh Department of Electrical and Computer Engineering, The University of Texas, Austin, TX 78712, USA Received: 4 December 1995 / Accepted in revised form: 18 June 1996 Abstract. A neuron model in which the neuron state is de- Real neural systems exhibit a range of phenomena such scribed by a complex number is proposed. A network of as oscillations, frequency entrainment, phase-locking, and these neurons, which can be used as an associative memory, even chaos, which are rarely touched by applied models. operates in two distinct modes: (i) fixed point mode and In particular, a great range of biological neural behavior (ii) oscillatory mode. Mode selection can be done by vary- cannot be modeled by networks with only fixed point be- ing a continuous mode parameter, ν, between 0 and 1. At havior. There is strong evidence that brains do not store one extreme value of ν (= 0), the network has conservative patterns as fixed points. For example, work done in the dynamics, and at the other (ν = 1), the dynamics are dissi- last decade by Freeman and his group with mammalian ol- pative and governed by a Lyapunov function. Patterns can factory cortex revealed that odors are stored as oscillating be stored and retrieved at any value of ν by, (i) a one-step states (Skarda and Freeman 1987). Deppisch et al. (1994) outer product rule or (ii) adaptive Hebbian learning. In the modeled synchronization, switching and bursting activity fixed point mode patterns are stored as fixed points, whereas in cat visual cortex using a network of spiking neurons. in the oscillatory mode they are encoded as phase relations It has also been suggested that oscillations in visual cor- among individual oscillations. By virtue of an instability in tex may provide an explanation for the binding problem the oscillatory mode, the retrieval pattern is stable over a (Gray and Singer 1989). Substantial experimental and the- finite interval, the stability interval, and the pattern gradu- oretical investigations led to Malsburg’s labeling hypothe- ally deteriorates with time beyond this interval. However, at sis (von der Malsburg 1988), which postulates that neural certain values of ν sparsely distributed over ν-space the in- information processing is intimately related to the tempo- stability disappears. The neurophysiological significance of ral relationships between the phase- and/or frequency-based the instability is briefly discussed. The possibility of physi- “labels” of oscillating cell assemblies. Ritz et al. (1994; cally interpreting dissipativity and conservativity is explored Gerstner and Ritz 1993) proposed a model of spiking neu- by noting that while conservativity leads to energy savings, rons applied to binding and pattern segmentation. Abbot dissipativity leads to stability and reliable retrieval. (1990) studied a network of oscillating neurons in which binary patterns can be stored as phase relationships between individual oscillators. In summary, one may observe that 1 Introduction neural networks with oscillating behavior are gaining atten- tion, not only as more sophisticated models of biology, but Hopfield’s stochastic network of two-state cells (Hopfield also as models with practical significance. 1982), as well as a subsequent dynamic network of cells But why are fixed points and oscillations so impor- with graded response (Hopfield 1984), can be used as As- tant in dynamic neural modeling? In general, the stable sociative or Content Addressable Memories. Several oth- states of a dynamic system with a vector field, X, can be ers have also proposed similar dynamic models that can be fixed points, limit cycles (oscillations) or strange attractors 0 used as autoassociating, heteroassociating or bidirectional (Zeeman 1976). According to the C -density theorem due to memories (Kosko 1987). In most of these models, perti- Smale (1973), by making an arbitrarily small perturbation to nent information is encoded as fixed points of the net- X, we may be assured that the only attractors of X are fixed work dynamics, and a Lyapunov or “energy” function is points or oscillations (limit cycles). associated with the dynamics to prove stability. Therefore, In this paper, we propose a dynamic neural network these neural systems turn out to be dissipative systems model that operates in a continuum between two distinct (Cohen and Grossberg 1983; Hopfield 1984; Kosko 1987), modes: (i) the fixed point mode and (ii) the oscillatory mode. and one may wonder whether there are fundamental physi- In the fixed point mode memories are stored as fixed points cal reasons why neural memories need to be dissipative. of the network dynamics, whereas in the oscillatory mode they are encoded as phase relations among individual oscil- Correspondence to: J. Ghosh 230 lators. Mode selection is done by varying a mode parameter, dynamics, and Ij is the sum of the external currents entering ν, between 0 and 1 ( for fixed point mode, ν = 1; oscilla- the neuron, j. Henceforth, we drop τa (τa = 1) without tory mode, ν = 0). Pattern storage is accomplished in one lack of generality. Tjk, a complex quantity, is the weight or step by a complex outer product rule, or adaptively, by a strength of the synapse connecting jth and kth neurons. We Hebbian-like learning mechanism. The stored patterns can have chosen the complex tanh( ) function as the activation be retrieved and adaptively learnt at any value of ν. function, g( ), because of its sigmoidal· nature 1. The paper is organized as follows. Our neuron model is The model· exists in two distinct modes: (i) the fixed presented in Sect. 2. The network operation in fixed point point mode and (ii) the oscillatory mode. Mode switching mode is discussed in Sect. 3, and in Sect. 4 the oscillatory can be done by varying a mode parameter, ν. In (1, 2), let behavior is described. An elaborate simulation study is given β = ν + i(1 ν), and α = λβ, where λ is a positive number, in Sect. 6. In Sect. 7, the neurophysiological and cognitive known as steepness− factor, and ν lies within the interval significance of the proposed model is discussed. [0; 1]. (1, 2) become: z˙ = T V (ν + i(1 ν))z + I ; (3) j jk k − − j j 2 The complex neuron model Xk Vj = g(λ(ν + i(1 ν))zj∗) (4) Most models of neural oscillators that are not driven by − periodic signals, can be placed in one of three categories: We now study the two extreme modes of the network models in which (i) neuron state is described by a variable more closely in Sects. 3 and 4. pair (Fitzhugh 1961; Abbot 1990), (ii) a pair of neurons or neuronal group (Chang et al. 1992) are coupled to produce 3 Fixed point mode oscillations, or (iii) equations of dynamics are second-order equations. These three categories are closely related mathe- The network exhibits fixed point dynamics when the mode matically. A single second-order equation can be expressed parameter, ν, equals 1. The network dynamics (for λ =1) as two first-order equations by introducing a new variable. can then be described by the following equations: Two neurons with first-order dynamics when coupled sim- dzj ulate second-order dynamics. Second-order systems are a = TjkVk zj + Ij (5) natural choice for modeling oscillatory activity since there dt − Xk is a well-known theorem that states that if y is a measure- V = g(z ) (6) ment of a closed orbit of an arbitrary dynamical system, then j j∗ there exists a second-order differential equation having y as If the conductance matrix, T , is a Hermitian, the dynamics its unique attractor (Zeeman 1976). are stable since a Lyapunov function, E, can be associated Following approach (i) described above, we propose the with the above system: use of complex numbers to create a powerful model of 1 associative memory that can exhibit second-order dynam- E = TjkVj∗Vk −2 j ics. A possible physiological interpretation of the complex X Xk state is explored in Sect. 7. There has been some interest Vj∗ 1 in complex-valued neuron models recently. A complex ver- +Re[ g− ( )dV + IjV ∗] (7) · j sion of Hopfield’s continuous model has been proposed by j 0 j X Z X Hirose (1992). In this model all input components as well Since a Lyapunov function exists, the above equations de- as the cell states are constrained to be of unit magnitude. scribe a dissipative system. V usually settles in a fixed point, Other studies in the area of complex-valued networks in- with each Re[Vj] approaching 1. clude a complex backpropagation algorithm to train mul- We chose the complex tanh(±) function as the sigmoidal tilayered feed-forward networks. Details of this work can nonlinearity since it has certain· properties required for sta- be found in Little et al. (1990), among others. Recently we bility of (5, 6). We now give the main result. have proposed discrete and continuous versions of a complex associative memory and have made capacity estimates for Theorem 1. If the weight matrix, T , is Hermitian i.e., Tjk = T∗ j; k, then the energy function, E in (7) is a Lyapunov the discrete model (Chakravarthy and Ghosh 1994). These kj ∀ models have only fixed point behavior. Here we present a function of the dynamic system (5, 6), defined over a region in which g( ) is analytic and @g =@x > 0. (z = x + iy; g = more general model that exhibits both oscillatory and fixed · x point dynamics. gx + igy:) In our complex neuron model the neuron state is repre- Proof.