AREVIEW OF DESIGNS AND APPLICATIONS OF ECHO STATE NETWORKS Chenxi Sun1,2, Moxian Song1,2, Shenda Hong3,4, and Hongyan Li 1,2∗ 1Key Laboratory of Machine Perception (Ministry of Education), Peking University, Beijing, China. 2School of Electronics Engineering and Computer Science, Peking University, Beijing, China. 3National Institute of Health Data Science, Peking University, Beijing, China. 4Institute of Medical Technology, Health Science Center, Peking University, Beijing, China. December 8, 2020 ABSTRACT Recurrent Neural Networks (RNNs) have demonstrated their outstanding ability in sequence tasks and have achieved state-of-the-art in wide range of applications, such as industrial, medical, economic and linguistic. Echo State Network (ESN) is simple type of RNNs and has emerged in the last decade as an alternative to gradient descent training based RNNs. ESN, with a strong theoretical ground, is practical, conceptually simple, easy to implement. It avoids non-converging and computationally expensive in the gradient descent methods. Since ESN was put forward in 2002, abundant existing works have promoted the progress of ESN, and the recently introduced Deep ESN model opened the way to uniting the merits of deep learning and ESNs. Besides, the combinations of ESNs with other machine learning models have also overperformed baselines in some applications. However, the apparent simplicity of ESNs can sometimes be deceptive and successfully applying ESNs needs some experience. Thus, in this paper, we categorize the ESN-based methods to basic ESNs, DeepESNs and combinations, then analyze them from the perspective of theoretical studies, network designs and specific applications. Finally, we discuss the challenges and opportunities of ESNs by summarizing the open questions and proposing possible future works. 1 Introduction In the last decades, Recurrent Neural Networks (RNNs) based methods demonstrated their outstanding ability in time-series prediction tasks and have become very attractive for their potentially wide range of applications, such as arXiv:2012.02974v1 [cs.LG] 5 Dec 2020 biological, medical, linguistic, social and economic. RNNs represent a very powerful generic tool, integrating both large dynamical memory and highly adaptable computational capabilities. They are the Machine Learning (ML) model most closely resembling biological brains, the substrate of natural intelligence. Meanwhile, deep RNN are able to develop in their internal states a multiple time-scales representation of the temporal information, a much desired feature. In the context of deep learning, error backpropagation (BP) [1] is to this date one of the most important achievements in RNNs training, but only with a partial success. One of the limitations is that bifurcations can make training non-converging, computationally expensive and poor local minima [2]. Moreover, the exploding gradient problem often occurs in the training process, where the stable prediction performance cannot be easily ensured [3]. Some developments in BP for RNNs [4] perform better on problems which require long memory, but it’s hard for BP RNN training, unless networks are specifically designed to deal with them [5]. Long short-memory (LSTM) and gated recurrent unit (GRU) are advanced designs to mitigate shortcomings of RNN. But when the length of input sequence exceeds a certain limit, the gradient will still disappear. Meanwhile, each LSTM cell have four full connection layers, if the time span of LSTM is large and the network is very deep, the calculation will be very heavy and time-consuming. Further, too many parameters will lead to over fitting risk. ∗Corresponding author: [email protected] A PREPRINT -DECEMBER 8, 2020 Figure 1: Basic echo state networks Reservoir computing (RC) has emerged in the last decade as an alternative to gradient descent methods for training RNNs with a strong theoretical ground [6, 7, 8, 9, 10]. Echo State Network (ESN) [6] is one of the key RC. ESNs are practical, conceptually simple, and easy to implement. ESNs employ the multiple high-dimensional projection in the large number of states of the reservoir with strong nonlinear mapping capabilities, to capture the dynamics of the input. This basic idea was first clearly spelled out in a neuroscientific model of the corticostriatal processing loop [3]. ESNs enjoy, under mild conditions, the so-called echo state property [6], that ensures that the effect of the initial condition vanishes after a finite transient. The inputs with more similar short-term history will evoke closer echo states, which ensure the dynamical stability of the reservoir. Recently, the introduction of the Deep Echo State Network (DeepESN) model [11, 12] allowed to study the properties of layered RNN architectures separately from the learning aspects. Remarkably, such studies pointed out that the structured state space organization with multiple time-scales dynamics in deep RNNs is intrinsic to the nature of compositionality of recurrent neural modules. The interest in the study of the DeepESN model is hence twofold. On the one hand, it allows to shed light on the intrinsic properties of state dynamics of layered RNN architectures. On the other hand it enables the design of extremely efficiently trained deep neural networks for temporal data. Meanwhile, with the development of deep learning, a variety of neural network structures have been proposed, like auto-encoder (AE) [13], generative adversarial networks (GAN) [14], convolutional neural networks (CNNs) [15] and restricted Boltzmann machine (RBM) [16]. ESNs have combined these different structures and achieved state-of-the-art performance in specific tasks and different applications, such as industrial [17], medical [18], financial [19] and robotics with reinforcement learning [20, 21]. ESNs from their beginning proved to be a highly practical approach to RNN training. It is conceptually simple and computationally inexpensive. It reinvigorated interest in RNNs, by making them accessible to wider audiences. However, the apparent simplicity of ESNs can sometimes be deceptive. Successfully applying ESNs needs some experience. Thus, we summarize the advancements in the development, analysis and applications of ESNs to summarize the experience of existing work. The rest of the paper is organized as follows. Section 2 analyzes the basic ESNs from the basic definitions to specific designs. Section 3 introduces the DeepESNs with its different structures. Section 4 describes the existing combinations of ESN and other machine learning methods. Section 5 gives the benchmarks, the data-driven and the real-ward applications of ESNs. Section 6 and 7 raise the challenges, opportunities and make conclusions. 2 Basic echo state networks 2.1 Preliminaries Definition 1 (Echo State Networks (ESNs)) Echo state networks are a type of fast and efficient recurrent neural network. A typical ESN consists of an input layer, a recurrent layer, called reservoir, with a large number of sparsely connected neurons, and an output layer. The connection weights of the input layer and the reservoir layer are fixed after initialization, and the output weights are trainable and obtained by solving a linear regression problem. In an ESN, u(t) 2 RD×1 denotes the input value at time t of the time series, x(t) 2 RN×1 denotes the state of the M×1 N×D reservoir at time t. y(t) 2 R denotes the output value. Win 2 R represents the connection weights between N×N the input layer and the hidden layer, Wres 2 R notes the connection weights in hidden layer. The state transition equation is in Equation 1, where f is a nonlinear function such as tan and sigmoid. x(t) = f(Winu(t) + Wresx(t − 1)) (1) y(t) = Woutx(t) 2 A PREPRINT -DECEMBER 8, 2020 Only parameters of readout weights Wout are subject to training. Which can obtain a closed-form solution by extremely fast algorithms such as ridge regression. Assuming the internal states and desired outputs are stored in X and Y . By training to find a solution to the least squares problem in Equation 2, the calculation formula of the readout weights Wout is got in Equation 3. 2 min jjWoutX − Y jj2 (2) Wout −1 Wout = Y · X (3) in Before training phase, three are three main hyper-parameters of ESNs need to initialize - w , α and ρ(Wres). in • w is an input-scaling parameter, the elements in Win are commonly randomly initialized from a uniform distribution in [−win; win]. • α is the sparsity parameter of Wres, denoting the proportion of non-zero elements in matrix. • ρ(Wres) is the spectral radius parameter (the largest eigenvalue in absolute value) of Wres. Wres initialize from a matrix W , where the elements of W are generated randomly in [−1; 1] and λmax(W ) is the largest eigenvalue. W Wres = ρ(Wres) · (4) λmax(W ) The extreme training efficiency of the ESN approach derives from the fact that only the readout weights are trained, while the weights of input and reservoir are initialized under certain conditions and then are left untrained. After training phase, ESNs can give predictions Y^ of new data U^ with the trained Wout by Equation 5. X^t = f(WinU^t + WresX^t−1) (5) Y^t = WoutX^t Assuming that the size of the reservoir are fixed by N, The input time series data has T-length D-dimension. For the typical reservoir computing Equation 1, its complexity is computed in Equation 6. 2 Ctypical = O(αT N + TND) (6) A pivotal property of ESNs is Echo State Property (ESP), which characterizes valid ESN dynamics. ESP essentially describes that the states of reservoir should asymptotically depend only on the driving input signal, which means the state is an echo of the input, meanwhile, and the influence of initial conditions should progressively vanish with time. Briefly, according to Jeager’s paper [6], the ESN is that every echo state vector x is uniquely determined for every input sequence u. It implies that the nearby echo states have the similar input histories.