Stability Conditions of Bicomplex-Valued Hopfield Neural Networks
Total Page:16
File Type:pdf, Size:1020Kb
LETTER Communicated by Marcos Eduardo Valle Stability Conditions of Bicomplex-Valued Hopfield Neural Networks Masaki Kobayashi Downloaded from http://direct.mit.edu/neco/article-pdf/33/2/552/1896796/neco_a_01350.pdf by guest on 30 September 2021 [email protected] Mathematical Science Center, University of Yamanashi, Kofu, Yamanashi 400-8511, Japan Hopfield neural networks have been extended using hypercomplex num- bers. The algebra of bicomplex numbers, also referred to as commutative quaternions, is a number system of dimension 4. Since the multiplica- tion is commutative, many notions and theories of linear algebra, such as determinant, are available, unlike quaternions. A bicomplex-valued Hopfield neural network (BHNN) has been proposed as a multistate neu- ral associative memory. However, the stability conditions have been in- sufficient for the projection rule. In this work, the stability conditions are extended and applied to improvement of the projection rule. The com- puter simulations suggest improved noise tolerance. 1 Introduction A Hopfield neural network (HNN) is an associative memory model using neural networks (Hopfield, 1984). It has been extended to a complex-valued HNN (CHNN), a multistate associative memory model that is useful for storing multilevel data (Aoki & Kosugi, 2000; Aoki, 2002; Jankowski, Lo- zowski, & Zurada, 1996; Muezzinoglu, Guzelis, & Zurada, 2003; Tanaka & Aihara, 2009; Zheng, 2014). Moreover, a CHNN has been extended using hypercomplex numbers (Kuroe, Tanigawa, & Iima, 2011; Kobayashi, 2018a, 2019). The algebra of hyperbolic numbers is a number system of dimen- sion 2, like that of complex numbers. Several models of hyperbolic HNNs (HHNNs) have been proposed. An HHNN using a directional activation function is an alternative of CHNN and improves noise tolerance. The al- gebra of quaternions is a number system of dimension 4 and the multiplica- tion is not commutative. Several activation functions have been proposed for quaternion-valued HNNs (QHNNs): 1. A split activation function (Isokawa, Nishimura, Kamiura, & Matsui, 2006, 2007, 2008). 2. A phasor-represented activation function (Isokawa, Nishimura, Saitoh, Kamiura, & Matsui, 2008; Isokawa, Nishimura, & Matsui, 2009). Neural Computation 33, 552–562 (2021) © 2021 Massachusetts Institute of Technology https://doi.org/10.1162/neco_a_01350 Stability Conditions of Bicomplex-Valued Hopfield Neural Networks 553 Table 1: Multiplication Table of Bicomplex Numbers. ijk i −1 kj jk+1 i kj i−1 Downloaded from http://direct.mit.edu/neco/article-pdf/33/2/552/1896796/neco_a_01350.pdf by guest on 30 September 2021 3. Acontinuous activation function (de Castro & Valle, 2017; Valle, 2014; Valle & de Castro, 2018). 4. A twin-multistate activation function (Kobayashi, 2017). A twin-multistate activation function was also applied to an HNN using split quaternions, referred to as split quaternion-valued HNNs (SQHNN) (Kobayashi, 2020b). The algebra of bicomplex numbers, also referred to as commutative quaternions, is another number system of dimension 4. The multiplication is commutative, unlike that of quaternions. Isokawa, Nishimura, and Matsui (2010) proposed a bicomplex-valued HNN (BHNN) employing a phasor-represented activation function. Since many notions and theories of linear algebra are available for bicomplex matrices, they are useful for analyzing BHNNs. The twin-multistate activation function was proposed to reduce the number of weight parameters and also introduced a BHNN (Kobayashi, 2018b). The BHNN employing the twin-multistate activation function is available as an alternative to CHNN. Although the stability conditions were given, they were insufficient for the projection rule; the conventional pro- jection rule does not theoretically ensure convergence. This problem ap- pears in the models using algebra with a unipotent imaginary unit, such as the HHNN. In the HHNN, stability conditions were extended such that an HHNN with projection rule converges (Kobayashi, 2020a). In this work, the stability conditions of BHNN are extended. The extended stability con- ditions provide a new projection rule. In addition, computer simulations show that the new projection rule improves noise tolerance. 2 Bicomplex Numbers In this section, we briefly describe bicomplex numbers (Catoni, Cannata, & Zampetti, 2006; Kosal & Tosun, 2014). A bicomplex number is repre- = + + + sented as q q0 q1i q2 j q3k. Bicomplex numbers form a commuta- α = + β = + tive number system based on Table 1. Putting q0 q1i and q2 q3i, the bicomplex number is represented as q = α + β j. Then the product of q = α + β j and q = α + β j is qq = αα + ββ + (αβ + βα ) j. (2.1) 554 M. Kobayashi The conjugate is defined as q = α + β j, (2.2) and qq = qq holds. The real part of q is that of α and denoted as Re(q). We note that the algebra of bicomplex numbers is not a field, that is, there exist zero divisors. In fact, for q = 1 + j and q = 1 − j, qq = 0holds. For bicomplex matrices, many notions and theories of linear algebra are Downloaded from http://direct.mit.edu/neco/article-pdf/33/2/552/1896796/neco_a_01350.pdf by guest on 30 September 2021 available. If division is not required for the definitions and proofs, they are available. Since division is not required for the definition of a determinant, the determinant is well defined for bicomplex matrices. Commutativity is not required for the definition of the determinant, though it is necessary for the proof of basic properties. Thus, the determinant is not available for quaternion matrices. Inverse matrices are necessary for projection rule. For a square bicomplex matrix A,letB be the cofactor matrix of A. Then AB = det(A)I holds. Division is not required in the proof for this equality. − − If det(A) is not a zero divisor, A 1 = (det(A)) 1 B is obtained. We can deter- mine the inverse matrix by gaussian elimination. Moreover, representation αβ α + β using complex matrices is available by corresponding βαto j. 3 Bicomplex-Valued Hopfield Neural Networks = + w = + v Let za xa ya j and ab uab ab j be the state of neuron a and weight from neuron b to neuron a, respectively. For the resolution factor K,wede- fine θK = π/K and the set = θ + θ K−1 . V cos(2k K ) sin(2k K ) i k=0 (3.1) The weighted sum input to neuron a is defined as Sa = Xa + Ya j (3.2) N = w , abzb (3.3) b=1 where N is the number of neurons. The update rule of neuron is given as f (za, Sa ) = arg max Re(xXa ) + arg max Re(yYa ) j. (3.4) x∈V y∈V When the first part has multiple maximal arguments, the value is defined as xa, that is, the original value maintains. The second part is also defined = , = in the same way. If za f (za Sa ) za, then we obtain = + , Re(zaSa ) Re(xaXa ) Re(yaYa ) > Re(zaSa ) = Re(xaXa ) + Re(ya Ya ). (3.5) Stability Conditions of Bicomplex-Valued Hopfield Neural Networks 555 A bicomplex-valued neuron can be regarded as two complex-valued neu- rons. A BHNN with N neurons stores complex vectors of dimension 2N. The following stability conditions were provided in Kobayashi (2018b): w = w , ab ba (3.6) waa = 0. (3.7) Downloaded from http://direct.mit.edu/neco/article-pdf/33/2/552/1896796/neco_a_01350.pdf by guest on 30 September 2021 The BHNNs converge under these stability conditions, though they are in- sufficient for the projection rule. Thus, we need to extend stability condition 3.7. From condition 3.6, uaa and vaa are real numbers. Stability condition 3.7 is modified to uaa ≥|vaa|. (3.8) Condition 3.8 is an extension of condition 3.7. We prove that a BHNN con- verges under new stability conditions 3.6 and 3.8. The energy of BHNN is defined as 1 E =− Re (zaw z ) . (3.9) 2 ab b a,b Since a BHNN has finite states, it is sufficient to prove that the energy de- creases when the state changes. Suppose that the state of neuron c changes = from zc to zc zc. We calculate the change of energy: ⎛ ⎞ 1 ⎝ ⎠ E =− Re z w z + Re zawacz + Re z wccz 2 c cb b c c c b=c a=c ⎛ ⎞ 1 ⎝ ⎠ + Re (zcw z ) + Re (zawaczc ) + Re (zcwcczc ) (3.10) 2 cb b b=c a=c 1 =− Re z w z − Re z wccz c cb b 2 c c b=c 1 + Re (zcw z ) + Re (zcwcczc ) (3.11) cb b 2 b=c N 1 =− Re z w z + Re z wcczc − Re z wccz c cb b c 2 c c b=1 N 1 + Re (zcw z ) − Re (zcwcczc ) (3.12) cb b 2 b=1 556 M. Kobayashi N N =− w + w Re zc cbzb Re zc cbzb b=1 b=1 1 − Re (z − zc )wcc z − zc . (3.13) 2 c c From condition 3.5, we obtain Downloaded from http://direct.mit.edu/neco/article-pdf/33/2/552/1896796/neco_a_01350.pdf by guest on 30 September 2021 1 E < − Re (z − zc )wcc z − zc . (3.14) 2 c c = − v We put zc zc zc. Note that ucc and cc arerealnumbers.Thenwehave w = + + v + Re zc cczc Re (xc yc j)(ucc cc j)(xc yc j) (3.15) = + v + + v Re xc uccxc xc ccyc yc uccyc yc ccxc (3.16) = | |2 + v + | |2 ucc xc 2 ccRe xc yc ucc yc (3.17) v 2 = −|v | | |2 +| |2 +|v | + cc . (ucc cc ) xc yc cc xc yc (3.18) |vcc| From stability condition 3.8, E < 0isproved. 4 Projection Rule Projection rule is a practical learning algorithm (Kobayashi, 2017, 2018a, 2018b, 2020a). The conventional projection rule for BHNNs has the follow- ing problems: 1. A BHNN does not converge theoretically.