Stability Conditions of Bicomplex-Valued Hopfield Neural Networks

LETTER Communicated by Marcos Eduardo Valle

Masaki Kobayashi Downloaded from http://direct.mit.edu/neco/article-pdf/33/2/552/1896796/neco_a_01350.pdf by guest on 30 September 2021 [email protected] Mathematical Science Center, University of Yamanashi, Kofu, Yamanashi 400-8511, Japan

Hopfield neural networks have been extended using hypercomplex numbers. The algebra of bicomplex numbers, also referred to as commutative quaternions, is a number system of dimension 4. Since the multiplication is commutative, many notions and theories of linear algebra, such as determinant, are available, unlike quaternions. A bicomplex-valued Hopfield neural network (BHNN) has been proposed as a multistate neural associative memory. However, the stability conditions have been insufficient for the projection rule. In this work, the stability conditions are extended and applied to improvement of the projection rule. The computer simulations suggest improved noise tolerance.

1 Introduction

A Hopfield neural network (HNN) is an associative memory model using neural networks (Hopfield, 1984). It has been extended to a complex-valued HNN (CHNN), a multistate associative memory model that is useful for storing multilevel data (Aoki & Kosugi, 2000; Aoki, 2002; Jankowski, Lo- zowski, & Zurada, 1996; Muezzinoglu, Guzelis, & Zurada, 2003; Tanaka & Aihara, 2009; Zheng, 2014). Moreover, a CHNN has been extended using hypercomplex numbers (Kuroe, Tanigawa, & Iima, 2011; Kobayashi, 2018a, 2019). The algebra of hyperbolic numbers is a number system of dimension 2, like that of complex numbers. Several models of hyperbolic HNNs (HHNNs) have been proposed. An HHNN using a directional activation function is an alternative of CHNN and improves noise tolerance. The algebra of quaternions is a number system of dimension 4 and the multiplication is not commutative. Several activation functions have been proposed for quaternion-valued HNNs (QHNNs):

1. A split activation function (Isokawa, Nishimura, Kamiura, & Matsui, 2006, 2007, 2008). 2. A phasor-represented activation function (Isokawa, Nishimura, Saitoh, Kamiura, & Matsui, 2008; Isokawa, Nishimura, & Matsui, 2009).

Table 1: Multiplication Table of Bicomplex Numbers.

ijk i −1 kj jk+1 i kj i−1 Downloaded from http://direct.mit.edu/neco/article-pdf/33/2/552/1896796/neco_a_01350.pdf by guest on 30 September 2021

3. Acontinuous activation function (de Castro & Valle, 2017; Valle, 2014; Valle & de Castro, 2018). 4. A twin-multistate activation function (Kobayashi, 2017).

A twin-multistate activation function was also applied to an HNN using split quaternions, referred to as split quaternion-valued HNNs (SQHNN) (Kobayashi, 2020b). The algebra of bicomplex numbers, also referred to as commutative quaternions, is another number system of dimension 4. The multiplication is commutative, unlike that of quaternions. Isokawa, Nishimura, and Matsui (2010) proposed a bicomplex-valued HNN (BHNN) employing a phasor-represented activation function. Since many notions and theories of linear algebra are available for bicomplex matrices, they are useful for analyzing BHNNs. The twin-multistate activation function was proposed to reduce the number of weight parameters and also introduced a BHNN (Kobayashi, 2018b). The BHNN employing the twin-multistate activation function is available as an alternative to CHNN. Although the stability conditions were given, they were insufficient for the projection rule; the conventional projection rule does not theoretically ensure convergence. This problem ap- pears in the models using algebra with a unipotent imaginary unit, such as the HHNN. In the HHNN, stability conditions were extended such that an HHNN with projection rule converges (Kobayashi, 2020a). In this work, the stability conditions of BHNN are extended. The extended stability conditions provide a new projection rule. In addition, computer simulations show that the new projection rule improves noise tolerance.

2 Bicomplex Numbers

In this section, we briefly describe bicomplex numbers (Catoni, Cannata, & Zampetti, 2006; Kosal & Tosun, 2014). A bicomplex number is repre- = + + + sented as q q0 q1i q2 j q3k. Bicomplex numbers form a commuta- α = + β = + tive number system based on Table 1. Putting q0 q1i and q2 q3i, the bicomplex number is represented as q = α + β j. Then the product of q = α + β j and q = α + β j is

qq = αα + ββ + (αβ + βα ) j. (2.1) 554 M. Kobayashi

The conjugate is defined as

q = α + β j, (2.2) and qq = qq holds. The real part of q is that of α and denoted as Re(q). We note that the algebra of bicomplex numbers is not a field, that is, there exist zero divisors. In fact, for q = 1 + j and q = 1 − j, qq = 0holds.

For bicomplex matrices, many notions and theories of linear algebra are Downloaded from http://direct.mit.edu/neco/article-pdf/33/2/552/1896796/neco_a_01350.pdf by guest on 30 September 2021 available. If division is not required for the definitions and proofs, they are available. Since division is not required for the definition of a determinant, the determinant is well defined for bicomplex matrices. Commutativity is not required for the definition of the determinant, though it is necessary for the proof of basic properties. Thus, the determinant is not available for quaternion matrices. Inverse matrices are necessary for projection rule. For a square bicomplex matrix A,letB be the cofactor matrix of A. Then AB = det(A)I holds. Division is not required in the proof for this equality. − − If det(A) is not a zero divisor, A 1 = (det(A)) 1 B is obtained. We can determine the inverse matrix by gaussian elimination. Moreover, representation αβ α + β using complex matrices is available by corresponding βαto j.

3 Bicomplex-Valued Hopfield Neural Networks

= + w = + v Let za xa ya j and ab uab ab j be the state of neuron a and weight from neuron b to neuron a, respectively. For the resolution factor K,wede- fine θK = π/K and the set = θ + θ K−1 . V cos(2k K ) sin(2k K ) i k=0 (3.1) The weighted sum input to neuron a is defined as

Sa = Xa + Ya j (3.2) N = w , abzb (3.3) b=1 where N is the number of neurons. The update rule of neuron is given as

f (za, Sa ) = arg max Re(xXa ) + arg max Re(yYa ) j. (3.4) x∈V y∈V When the first part has multiple maximal arguments, the value is defined as xa, that is, the original value maintains. The second part is also defined = , = in the same way. If za f (za Sa ) za, then we obtain = + , Re(zaSa ) Re(xaXa ) Re(yaYa )

> Re(zaSa ) = Re(xaXa ) + Re(ya Ya ). (3.5) Stability Conditions of Bicomplex-Valued Hopfield Neural Networks 555

A bicomplex-valued neuron can be regarded as two complex-valued neurons. A BHNN with N neurons stores complex vectors of dimension 2N. The following stability conditions were provided in Kobayashi (2018b):

w = w , ab ba (3.6)

waa = 0. (3.7) Downloaded from http://direct.mit.edu/neco/article-pdf/33/2/552/1896796/neco_a_01350.pdf by guest on 30 September 2021 The BHNNs converge under these stability conditions, though they are insufficient for the projection rule. Thus, we need to extend stability condition 3.7. From condition 3.6, uaa and vaa are real numbers. Stability condition 3.7 is modified to

uaa ≥|vaa|. (3.8)

Condition 3.8 is an extension of condition 3.7. We prove that a BHNN converges under new stability conditions 3.6 and 3.8. The energy of BHNN is defined as 1 E =− Re (zaw z ) . (3.9) 2 ab b a,b

Since a BHNN has finite states, it is sufficient to prove that the energy de- creases when the state changes. Suppose that the state of neuron c changes = from zc to zc zc. We calculate the change of energy: ⎛ ⎞ 1 ⎝ ⎠ E =− Re z w z + Re zawacz + Re z wccz 2 c cb b c c c b=c a=c ⎛ ⎞ 1 ⎝ ⎠ + Re (zcw z ) + Re (zawaczc ) + Re (zcwcczc ) (3.10) 2 cb b b=c a=c 1 =− Re z w z − Re z wccz c cb b 2 c c b=c 1 + Re (zcw z ) + Re (zcwcczc ) (3.11) cb b 2 b=c N 1 =− Re z w z + Re z wcczc − Re z wccz c cb b c 2 c c b=1 N 1 + Re (zcw z ) − Re (zcwcczc ) (3.12) cb b 2 b=1 556 M. Kobayashi

N N =− w + w Re zc cbzb Re zc cbzb b=1 b=1 1 − Re (z − zc )wcc z − zc . (3.13) 2 c c From condition 3.5, we obtain Downloaded from http://direct.mit.edu/neco/article-pdf/33/2/552/1896796/neco_a_01350.pdf by guest on 30 September 2021 1 E < − Re (z − zc )wcc z − zc . (3.14) 2 c c = − v We put zc zc zc. Note that ucc and cc arerealnumbers.Thenwehave w = + + v + Re zc cczc Re (xc yc j)(ucc cc j)(xc yc j) (3.15)

= + v + + v Re xc uccxc xc ccyc yc uccyc yc ccxc (3.16)

= | |2 + v + | |2 ucc xc 2 ccRe xc yc ucc yc (3.17) v 2 = −|v | | |2 +| |2 +|v | + cc . (ucc cc ) xc yc cc xc yc (3.18) |vcc|

From stability condition 3.8, E < 0isproved.

4 Projection Rule

Projection rule is a practical learning algorithm (Kobayashi, 2017, 2018a, 2018b, 2020a). The conventional projection rule for BHNNs has the following problems: 1. A BHNN does not converge theoretically. 2. As the number of training patterns increases, the noise tolerance rapidly deteriorates. These problems also appear in the HHNNs. The HHNNs solve them by reducing self-feedbacks. The same scheme is applied to the projection rule for BHNNs. The training matrix is defined as ⎛ ⎞ z1 z2 ··· zP ⎜ 1 1 1 ⎟ ⎜ z1 z2 ··· zP ⎟ = ⎜ 2 2 2 ⎟ , Z ⎜ . . . . ⎟ (4.1) ⎝ . . .. . ⎠ 1 2 ··· P zN zN zN

p = p, p,..., p T where the pth column vector z (z1 z2 zN ) is the pth training pattern = ∗ −1 ∗ + and P is the number of training patterns. For U Z(Z Z) Z ,letsab tab j Stability Conditions of Bicomplex-Valued Hopfield Neural Networks 557

, w = + the (a b) element. The conventional projection rule is given by ab sab ∗ = tab j. Then stability condition 3.6 is satisfied from U U, though stability condition 3.8 is not satisfied. We should also note that saa and taa are real numbers. From Uzp = zp,

N p = w p za abzb (4.2) = b 1 Downloaded from http://direct.mit.edu/neco/article-pdf/33/2/552/1896796/neco_a_01350.pdf by guest on 30 September 2021 is obtained, and the training patterns are fixed points. If waa is replaced with O, stability condition 3.7 is satisfied, though the training patterns are not fixed points. In fact, the weighted sum input is w p = − − p. abzb (1 saa taa j)za (4.3) b=a

p Therefore, if the phases of the weighted sum input coincide with that of za , taa = 0 is necessary. Based on stability condition 3.8, the projection rule is modified to s + t j (a = b) w = ab ab ab (4.4) |taa|+taa j (a = b).

Then, stability condition 3.8 is satisfied. The weighted sum input is

N w p = − +| | p, abzb (1 saa taa )za (4.5) b=1

p and the phases of za and the weighted sum input are same. ∗ − We have to determine (Z Z) 1 for the projection rule. Z is the N × P ma- ∗ trix. If bicomplex numbers are represented by 2 × 2 matrices, Z and Z Z are transformed to 2N × 2P and 2P × 2P matrices, respectively. P ≤ N is neces- ∗ − sary for the existence of (Z Z) 1, as proved in the appendix. Therefore, the storage capacity of the projection rule is N. A bicomplex number consists of two complex numbers; a bicomplex vector of dimension N can be regarded as a complex vector of dimension 2N. We note that the storage capacity is half-dimension of training vectors.

5 Computer Simulations and Discussion

To evaluate the noise tolerance of the proposed projection rule, we conduct the same computer simulations as those in Kobayashi (2018b). We compare the noise tolerance of the conventional and proposed projection rules. We also provide the simulation results of QHNNs and SQHNNs as references. To begin, we employ the random data and impulsive noise. We generate 100 558 M. Kobayashi Downloaded from http://direct.mit.edu/neco/article-pdf/33/2/552/1896796/neco_a_01350.pdf by guest on 30 September 2021

Figure 1: Computer simulations using random data. training sets randomly, and 100 trials are conducted for each for a total of 10,000 trials. The noise rate of impulsive noise is denoted as r. The simulation results are shown in Figure 1. For P = 10, the proposed projection rule a bit outperforms the conventional one. The noise tolerance of the conventional projection rule rapidly deteriorates with the increase in P, whereas that of the proposed projection rule gradually deteriorates. The results of SQHNNs and conventional BHNNs are similar, and those of QHNNs and proposed BHNNs are similar. Next, the gray-scale image data and gaussian noise are employed. The 256-level images with 32 × 32 pixels are obtained by transforming those provided by the CIFAR-10 data set (Krizhevsky, 2009). Therefore, N and K Stability Conditions of Bicomplex-Valued Hopfield Neural Networks 559

Figure 2: Examples of training data. Downloaded from http://direct.mit.edu/neco/article-pdf/33/2/552/1896796/neco_a_01350.pdf by guest on 30 September 2021

Figure 3: Computer Simulations Using Image Data.

are 512 and 256, respectively. The examples are shown in Figure 2. For each P, 1000 trials are conducted. Figure 3 shows the simulation results, where σ is the standard deviation of gaussian noise. We compare the conventional and proposed projection rule based on the simulation results. These results can be explained from the standpoint of the self-feedbacks (Kobayashi, 2018b). In general, the self-feedbacks stabi- lize the neuron’s state. As a result, pseudo-memories increase and noise tolerance deteriorates. The self-feedbacks tend to increase with an increase in P. The proposed projection rule improves the noise tolerance by reducing the self-feedbacks. These phenomena appear in the projection rule of HHNNs (Kobayashi, 2018a, 2020a). Since the projection rule for QHNNs can remove the self-feedbacks, the QHNNs provide better noise tolerance than the SQHNNs and conventional BHNNs. 560 M. Kobayashi

6Conclusion

Bicomplex numbers are hypercomplex numbers of dimension 4. Multi- plication of bicomplex numbers is commutative and useful. A BHNN is implemented using bicomplex numbers. In this work, we extended the stability conditions of BHNNs. The conventional stability conditions are insufficient for the projection rule. The conventional projection rule does not satisfy the stability conditions, and it has not been clarified whether Downloaded from http://direct.mit.edu/neco/article-pdf/33/2/552/1896796/neco_a_01350.pdf by guest on 30 September 2021 the BHNNs employing conventional projection rule converge. We modified the projection rule using extended stability conditions and proved that the BHNNs employing the proposed projection rule converge. The proposed rule also reduces self-feedbacks. Since self-feedbacks deteriorate noise tolerance, the proposed projection rule improves noise tolerance. As computer simulations show, the effect of reduction in self-feedbacks increases with the increase in P. Although the self-feedbacks are reduced, they are not removed, unlike the QHNNs. If the self-feedbacks are removed, noise tolerance would improve further. We plan to study the algorithms such that self-feedbacks are removed.

Appendix

∗ − ∗ We prove that P ≤ N is necessary for the existence of Z(Z Z) 1Z .Wehave ∗ − only to discuss the existence of (Z Z) 1. First, we consider the case that Z ∗ is a complex matrix. Since Z is the N × P matrix, Z Z is the P × P matrix. ∗ − ∗ ∗ There exists (Z Z) 1 if and only if the rank of Z Z is P.FromRank(Z Z) ≤ RankZ ≤ min{P, N}, P ≤ N is obtained. Next, we consider the bicomplex case. A bicomplex number can be represented by a 2 × 2 matrix. Then Z is the 2N × 2P complex matrix. By the same discussion the complex case 2P ≤ 2N is obtained. This implies P ≤ N.

References

Aoki, H. (2002). A complex-valued neuron to transform gray level images to phase information. In Proceedings of the International Conference on Neural Information Pro- cessing (pp. 1084–1088). Piscataway, NJ: IEEE. Aoki, H., & Kosugi, Y. (2000). An image storage system using complex-valued associative memories. In Proceedings of the International Conference on Pattern Recogni- tion (pp. 626–629). Piscataway, NJ: IEEE. de Castro, F. Z., & Valle, M. E. (2017). Continuous-valued quaternionic Hop- field neural network for image retrieval: A color space study.In Proceedings of the Brazilian Conference on Intelligent Systems (pp. 186–191). Piscataway, NJ: IEEE. Catoni, F., Cannata, R., & Zampetti, P. (2006). An introduction to commutative quaternions. Advances in Applied Clifford Algebras, 16, 1–28. Stability Conditions of Bicomplex-Valued Hopfield Neural Networks 561

Hopfield, J. J. (1984). Neurons with graded response have collective computational properties like those of two-state neurons. In Proceedings of the National Academy of Sciences, 81, 3088–3092. Isokawa, T., Nishimura, H., Kamiura, N., & Matsui, N. (2006). Fundamental properties of quaternionic Hopfield neural network. In Proceedings of the International Joint Conference on Neural Networks (pp. 1610–1615). Piscataway, NJ: IEEE. Isokawa, T., Nishimura, H., Kamiura, N., & Matsui, N. (2007). Dynamics of discrete-

time quaternionic Hopfield neural networks. In Proceedings of the International Downloaded from http://direct.mit.edu/neco/article-pdf/33/2/552/1896796/neco_a_01350.pdf by guest on 30 September 2021 Conference on Artificial Neural Networks (pp. 848–857). Piscataway, NJ: IEEE. Isokawa, T., Nishimura, H., Kamiura, N., & Matsui, N. (2008). Associative memory in quaternionic Hopfield neural network. International Journal of Neural Systems, 18, 135–145. Isokawa, T., Nishimura, H., & Matsui, N. (2009). An iterative learning scheme for multistate complex-valued and quaternionic Hopfield neural networks. In Pro- ceedings of the International Joint Conference on Neural Networks (pp. 1365–1371). Piscataway, NJ: IEEE. Isokawa, T., Nishimura, H., & Matsui, N. (2010). Commutative quaternion and multistate Hopfield neural networks. In Proceedings of the IEEE World Congress on Com- putational Intelligence (pp. 1281–1286). Piscataway, NJ: IEEE. Isokawa, T., Nishimura, H., Saitoh, A., Kamiura, N., & Matsui, N. (2008). On the scheme of quaternionic multistate Hopfield neural network. In Proceedings of the Joint 4th International Conference on Soft Computing and Intelligent Systems and 9th International Symposium on Advanced Intelligent Systems (pp. 809–813). J-Stage. Jankowski, S., Lozowski, A., & Zurada, J. M. (1996). Complex-valued multistate neural associative memory. IEEE Transactions on Neural Networks, 7, 1491–1496. Kobayashi, M. (2017). Quaternionic Hopfield neural networks with twin-multistate activation function. Neurocomputing, 267, 304–310. Kobayashi, M. (2018a). Hyperbolic Hopfield neural networks with directional multistate activation function. Neurocomputing, 275, 2217–2226. Kobayashi, M. (2018b). Twin-multistate commutative quaternion Hopfield neural networks. Neurocomputing, 320, 150–156. Kobayashi, M. (2019). Storage capacity of hyperbolic Hopfield neural networks. Neu- rocomputing, 369, 185–190. Kobayashi, M. (2020a). Noise robust projection rule for hyperbolic Hopfield neural networks. IEEE Transactions on Neural Networks and Learning Systems, 31, 352– 356. Kobayashi, M. (2020b). Split quaternion-valued twin-multistate Hopfield neural networks. Advances in Applied Clifford Algebras, 30, 30. Kosal, H. H., & Tosun, M. (2014). Commutative quaternion matrices. Advances in Applied Clifford Algebras, 24, 769–779. Krizhevsky, A. (2009). Learning multiple layers of features from tiny images (Technical Report). University of Toronto. Kuroe, Y., Tanigawa, S., & Iima, H. (2011). Models of Hopfield-type Clifford neural networks and their energy functions: Hyperbolic and dual valued networks. In Proceedings of the International Conference on Neural Information Processing (pp. 560– 569). Berlin: Springer. 562 M. Kobayashi

Muezzinoglu, M. K., Guzelis, C., & Zurada, J. M. (2003). A new design method for the complex-valued multistate Hopfield associative memory. IEEE Transactions on Neural Networks, 14, 891–899. Tanaka, G., & Aihara, K. (2009). Complex-valued multistate associative memory with nonlinear multilevel functions for gray-level image reconstruction. IEEE Transaction on Neural Networks, 20, 1463–1473. Valle, M. E. (2014). A novel continuous-valued quaternionic Hopfield neural net-

work. In Proceedings of the Brazilian Conference on Intelligent Systems, 97–102. Pis- Downloaded from http://direct.mit.edu/neco/article-pdf/33/2/552/1896796/neco_a_01350.pdf by guest on 30 September 2021 cataway, NJ: IEEE. Valle, M. E., & de Castro, F. Z. (2018). On the dynamics of Hopfield neural networks on unit quaternions. IEEE Transactions on Neural Networks and Learning Systems, 29, 2464–2471. Zheng, P. (2014). Threshold complex-valued neural associative memory. IEEE Trans- actions on Neural Networks and Learning Systems, 25, 1714–1718.

Received July 5, 2020; accepted September 3, 2020.