Portland State University PDXScholar Dissertations and Theses Dissertations and Theses Winter 3-27-2013 Reward-driven Training of Random Boolean Network Reservoirs for Model-Free Environments Padmashri Gargesa Portland State University Follow this and additional works at: https://pdxscholar.library.pdx.edu/open_access_etds Part of the Dynamics and Dynamical Systems Commons, and the Electrical and Computer Engineering Commons Let us know how access to this document benefits ou.y Recommended Citation Gargesa, Padmashri, "Reward-driven Training of Random Boolean Network Reservoirs for Model-Free Environments" (2013). Dissertations and Theses. Paper 669. https://doi.org/10.15760/etd.669 This Thesis is brought to you for free and open access. It has been accepted for inclusion in Dissertations and Theses by an authorized administrator of PDXScholar. Please contact us if we can make this document more accessible: [email protected]. Reward-driven Training of Random Boolean Network Reservoirs for Model-Free Environments by Padmashri Gargesa A thesis submitted in partial fulfillment of the requirements for the degree of Master of Science in Electrical and Computer Engineering Thesis Committee: Christof Teuscher, Chair Richard P. Tymerski Marek A. Perkowski Portland State University 2013 Abstract Reservoir Computing (RC) is an emerging machine learning paradigm where a fixed kernel, built from a randomly connected "reservoir" with sufficiently rich dynamics, is capable of expanding the problem space in a non-linear fashion to a higher dimensional feature space. These features can then be interpreted by a linear readout layer that is trained by a gradient descent method. In comparison to traditional neural networks, only the output layer needs to be trained, which leads to a significant computational advantage. In addition, the short term memory of the reservoir dynamics has the ability to transform a complex temporal input state space to a simple non-temporal representation. Adaptive real-time systems are multi-stage decision problems that can be used to train an agent to achieve a preset goal by performing an optimal action at each timestep. In such problems, the agent learns through continuous interactions with its environment. Conventional techniques to solving such problems become computationally expensive or may not converge if the state-space being considered is large, partially observable, or if short term memory is required in optimal decision making. The objective of this thesis is to use reservoir computers to solve such goal- driven tasks, where no error signal can be readily calculated to apply gradient descent methodologies. To address this challenge, we propose a novel reinforce- ment learning approach in combination with reservoir computers built from simple Boolean components. Such reservoirs are of interest because they have the poten- tial to be fabricated by self-assembly techniques. We evaluate the performance of our approach in both Markovian and non-Markovian environments. We compare the performance of an agent trained through traditional Q-Learning. We find that i the reservoir-based agent performs successfully in these problem contexts and even performs marginally better than Q-Learning agents in certain cases. Our proposed approach allows to retain the advantage of traditional param- eterized dynamic systems in successfully modeling embedded state-space repre- sentations while eliminating the complexity involved in training traditional neural networks. To the best of our knowledge, our method of training a reservoir read- out layer through an on-policy boot-strapping approach is unique in the field of random Boolean network reservoirs. ii Acknowledgements This work was partly sponsored by the National Science Foundation under grant number 1028378. iii Contents Abstract i Acknowledgements iii List of Tables vi List of Figures vii List of Algorithms xvi 1 Overview 1 1.1 Introduction . .1 1.2 Goal and Motivation . .1 1.3 Challenges . .3 1.4 Our Contributions . .6 2 Background 9 2.1 Artificial Neural Networks . .9 2.2 Recurrent Neural Networks . 12 2.3 Reservoir Computing . 15 2.3.1 Random Boolean Networks . 20 2.4 Introduction to Reinforcement Learning . 23 2.4.1 Basics . 23 2.4.2 Generalized Policy Iteration - Cyclic nature of convergence . 31 2.4.3 Markov Decision Processes . 35 iv 2.4.4 Temporal Difference Method . 37 2.5 Function Approximators . 40 3 Related Work 42 4 Methodology 47 4.1 Framework . 48 4.1.1 Flowchart . 56 4.1.2 Code Implementation . 61 4.2 Optimization of Initial Parameter Space . 68 4.3 Comparison with other approaches . 72 5 Experiments 73 5.1 Supervised Learning with Temporal Parity Task . 74 5.1.1 Problem Description - Temporal Parity Task . 75 5.1.2 Results and Discussion - Temporal Parity Task . 75 5.2 Non Temporal Reinforcement Learning with n-Arm Bandit Task . 77 5.2.1 Problem Description - n-Arm Bandit Task . 78 5.2.2 Results and Discussion - n-Arm Bandit Task . 88 5.3 Temporal Reinforcement Learning with Tracker Task . 94 5.3.1 Problem Description - Tracker Task . 96 5.3.2 Results and Discussion - Tracker Task . 111 6 Conclusion 130 References 134 v List of Tables 4.1 Initial parameters critical for the policy convergence to an optimal one.................................... 68 5.1 Truth table for a 3 input odd-parity task . 75 5.2 RBN setup for the temporal odd-parity task . 76 5.3 Initial parameters list for n-arm bandit task . 82 5.4 The initial parameter settings considered for the n-arm bandit task 87 5.5 The actual reward values associated to the state transition proba- bilities used in our experiments . 87 5.6 Initial parameters list for the RBN-based agent trained on the 2-D grid-world task . 102 5.7 Best individual for the 2-D John Muir grid-world task . 106 5.8 Initial parameters list for the Q-Learning agent trained on the 2-D grid-world task . 107 5.9 Best individual for the 2-D Santa Fe grid-world task . 111 vi List of Figures 1.1 Block diagram depicting a reservoir set in a goal-driven context. A 2-D grid-world is depicted as the environment with which the reservoir interacts to learn. The reservoir should learn to function as a decision making agent. .4 1.2 Block diagram depicting a reservoir set in a goal-driven context to function as a decision making agent. The read-out layer of the reser- voir is trained based on reinforcement learning, through rewards re- ceived with its interaction with the environment. In this case, the environment is depicted as a 2-D grid-world. .5 2.1 Adaline with a sigmoid activation function; Source: [1] . 10 2.2 Feed forward vs Recurrent neural networks; Source: [2]. 13 2.3 A. Traditional gradient descent based RNN training methods adapt all connection weights (bold arrows), including input-to-RNN, RNN internal, and RNN-to-output weights. B. In Reservoir Computing, only the RNN-to-output weights are adapted; Source: [3] . 16 2.4 Trajectories through state space of RBNs within different phases, N = 32. A square represents the state of a node. The initial states on top, time flows downwards. a) ordered, K = 1. b) critical, K = 2. c) chaotic, K = 5 Source: [4]. 22 2.5 Agent and environment interaction in reinforcement learning; Source: [5]. 24 2.6 Value and policy interaction until convergence; Source: [5] . 34 vii 2.7 Q-Learning on-policy Control; Source: [5] . 39 3.1 A T-MAZE problem. 43 4.1 This design block diagram indicates an RBN-based reservoir aug- mented with a read-out layer. It interacts with a model-free envi- ronment by observing the perceived environment state through the Sensor Signal, st. Based on the input signal values, estimates are generated at each read-out layer node. The read-out layer node that generates the maximum value estimate is picked up as the Action Signal, at. This Action Signal, at is evaluated on the Environment and the resulting Reinforcement Signal, rt, is observed. This Rein- forcement Signal, along with the RBN's state, xt, the value estimate of the Action Signal, at, and the value estimate of the Action Signal generated during the next timestep t+1 is used to generate the error signal by an Error Calculator. This error signal is used to update the read-out layer weights. The block labelled Z−1 representing a delay tab, introduces a delay of one timestep on the signal in its path. 48 4.2 This diagram shows an RBN being perturbed by input signal, st, of dimension m. The random Boolean nodes in the RBN-based reservoir are connected recurrently based on average connectivity K. The input signal through the reservoir dynamics is transformed to a high-dimensional reservoir state signal xt, of dimension N, such that N > m. Each Boolean node in the reservoir is activated by a random Boolean function. The table represents lookup table (LUT) for the random Boolean function that activates node 4 in the reservoir. 50 viii 4.3 Block diagram depicting the readout layer with p output nodes. Each output node receives the weighted reservoir state, xt, as input and generates value estimates for each possible action, based on the activation function associated to it. The node associated to the maximum value estimate emerges as the winner. The action signal, at is the signal associated to the winning output node. 52 4.4 Block diagram depicting the interaction between the environment and the agent. This block receives as input the signal Q(st; at), from the read-out layer. Q(st; at), is a vector of action value estimates of dimension p, where p is the number of output nodes in the read-out layer. The Action Selector block selects the action to be performed at each timestep.