Effective Reinforcement Learning Through Evolutionary Surrogate-Assisted Prescription

Effective Reinforcement Learning through Evolutionary Surrogate-Assisted Prescription

Olivier Francon1, Santiago Gonzalez1,2, Babak Hodjat1, Elliot Meyerson1, Risto Miikkulainen1,2, Xin Qiu1, and Hormoz Shahrzad1 1Cognizant Technology Solutions and 2The University of Texas at Austin

ABSTRACT Outcomes There is now significant historical data available on decision mak- O ing in organizations, consisting of the decision problem, what decisions were made, and how desirable the outcomes were. Using this data, it is possible to learn a surrogate model, and with that model, evolve a decision strategy that optimizes the outcomes. This paper introduces a general such approach, called Evolutionary Surrogate-Assisted Prescription, or ESP. The surrogate is, for ex- Predict ample, a random forest or a neural network trained with gradient descent, and the strategy is a neural network that is evolved to C A maximize the predictions of the surrogate model. ESP is further Context Prescribe Actions extended in this paper to sequential decision-making tasks, which makes it possible to evaluate the framework in reinforcement learn- Figure 1: Elements of ESP. A Predictor is trained with histori- ing (RL) benchmarks. Because the majority of evaluations are done cal data on how given actions in given contexts led to specific on the surrogate, ESP is more sample efficient, has lower variance, outcomes. The Predictor can be any machine learning model and lower regret than standard RL approaches. Surprisingly, its trained with supervised methods, such as a random forest solutions are also better because both the surrogate and the strat- or a neural network. The Predictor is then used as a surro- egy network regularize the decision making behavior. ESP thus gate in order to evolve a Prescriptor, i.e. a neural network forms a promising foundation to decision optimization in real-world implementing a decision policy that results in the best pos- problems. sible outcomes. The majority of evaluations are done on the KEYWORDS surrogate, making the process highly sample-efficient and robust, and leading to decision policies that are regularized Reinforcement Learning, Decision Making, Surrogate-Assisted Evo- and therefore generalize well. lution, Genetic Algorithms, Neural Networks ACM Reference Format: Olivier Francon1, Santiago Gonzalez1,2, Babak Hodjat1, Elliot Meyerson1, and, in principle, make better decisions, i.e. those that lead to more Risto Miikkulainen1,2, Xin Qiu1, and Hormoz Shahrzad1. 2020. Effective desirable outcomes. However, while prediction is necessary, it is Reinforcement Learning through Evolutionary Surrogate-Assisted Prescrip- only part of the process. Predictive models do not specify what tion. In Genetic and Evolutionary Computation Conference (GECCO ’20), the optimal decisions actually are. To find a good decision strategy, July 8–12, 2020, Cancun, Mexico. ACM, New York, NY, USA, 9 pages. https: different approaches are needed. //doi.org/10.1145/3377930.3389842 The main challenge is that optimal strategies are not known, so standard gradient-based machine learning approaches cannot 1 INTRODUCTION be used. The domains are only partially observable, and decision arXiv:2002.05368v2 [cs.NE] 22 Apr 2020 Many organizations in business, government, education, and health- variables and outcomes often interact nonlinearly: For instance, care now collect significant data about their operations. Such data allocating marketing resources to multiple channels may have a is transforming decision making in organizations: It is now possi- nonlinear cumulative effect, or nutrition and exercise may interact ble to use machine learning techniques to build predictive models to leverage or undermine the effect of medication in treating an of behaviors of customers, consumers, students, and competitors, illness [9, 32]. Such interactions make it difficult to utilize linear programming and other traditional optimization approaches from Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed operations research. for profit or commercial advantage and that copies bear this notice and the full citation Instead, good decision strategies need to be found using search, on the first page. Copyrights for components of this work owned by others than ACM i.e. by generating strategies, evaluating them, and generating new, must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a hopefully better strategies based on the outcomes. In many domains fee. Request permissions from [email protected]. such search cannot be done in the domain itself: For instance, test- GECCO ’20, July 8–12, 2020, Cancun, Mexico ing an ineffective marketing strategy or medical treatment could © 2020 Association for Computing Machinery. ACM ISBN 978-1-4503-7128-5/20/07...$15.00 be prohibitively costly; evaluating an engineering design through https://doi.org/10.1145/3377930.3389842 simulation, or a behavioral strategy in game playing could require GECCO ’20, July 8–12, 2020, Cancun, Mexico Francon, Gonzalez, Hodjat, Meyerson, Miikkulainen, Qiu, and Shahrzad a prohibitive amount of computation time. However, given that training. Surprisingly, optimizing against the surrogate also has a historical data about past decisions and their outcomes exist, it is regularization effect: the solutions are sometimes more general and possible to do the search using a predictive model as a surrogate to thus perform better than solutions discovered in the domain itself. evaluate them. Only once good decision strategies have been found Further, ESP brings the advantages of population-based search out- using the surrogate, they are tested in the real world. lined above to RL, i.e. enhanced exploration, multiobjectivity, and Even with the surrogate, the problem of finding effective de- scale-up to high-dimensional search spaces. cision strategies is still challenging. Nonlinear interactions may The ESP approach is evaluated in this paper in various RL bench- result in deceptive search landscapes, where progress towards good marks. First its behavior is visualized in a synthetic domain, illus- solutions cannot be made through incremental improvement: Dis- trating how the Predictor and Prescriptor learn together to discover covering them requires large, simultaneous changes to multiple optimal decisions. Direct evolution (DE) is compared to evolution variables. Decision strategies often require balancing multiple ob- with the surrogate, to demonstrate how the approach minimizes jectives, such as performance and cost, and in practice, generating the need for evaluations in the real world. ESP is then compared a number of different trade-offs between them is needed. Conse- with standard RL approaches, demonstrating better sample effi- quently, search methods such as reinforcement learning (RL), where ciency, reliability, and lower cost. The experiments also demon- a solution is gradually improved through local exploration, do not strate regularization through the surrogate, as well as ability to lend themselves well to searching solution strategies either. Fur- utilize different kinds of Predictor models (e.g. random forests and ther, the number of variables can be very large, e.g. thousands or neural networks) to best fit the data. ESP thus forms a promising even millions as in some manufacturing and logistics problems [6], evolutionary-optimization-based approach to sequential decision- making methods such as Kriging and Bayesian optimization [5, 46] making tasks. ineffective. Moreover, the solution is not a single point but astrat- egy, i.e. a function that maps input situations to optimal decisions, exacerbating the scale-up problem further. 2 RELATED WORK Keeping in mind the above challenges, an approach is developed Traditional model-based RL aims to build a transition model, em- Evolutionary Surrogate-Assisted Prescription in this paper for (ESP; bodying the system’s dynamics, that estimates the system’s next Figure 1), i.e. for discovering effective solution strategies using state in time, given current state and actions. The transition model, evolutionary optimization. With a population-based search method, which is learned as part of the RL process, allows for effective action it is possible to navigate deceptive, high-dimensional landscapes, selection to take place [13]. These models allow agents to leverage and discover trade-offs across multiple objectives [27]. The strategy predictions of the future state of their environment [54]. However, is expressed as a neural network, making it possible to use state- model-based RL usually requires a prohibitive amount of data for of-the-art neuroevolution techniques to optimize it. Evaluations of building a reliable transition model while also training an agent. the neural network candidates are done using a predictive model, Even simple systems can require tens- to hundreds-of-thousands trained with historical data on past decisions and their outcomes. of samples [41]. While techniques such as PILCO [7] have been Elements of the ESP approach were already found effective in developed to specifically address sample efficiency in model-based challenging real-world applications. In an autosegmentation ver- RL, they can be computationally intractable for all but the lowest sion of Ascend by Evolv, a commercial product for optimizing dimensional domains [51]. designs of web pages [28], a neural network was used to map user As RL has been applied to increasingly complex tasks with real- descriptions to most effective web-page designs. In CyberAg, ef- world costs, sample efficiency has become a crucial issue. Model- fective growth recipes for basil were found through search with free RL has emerged as an important alternative in such tasks a surrogate model trained with outcomes of past recipes [23]. In because they sample the domain without a transition model. Their both cases, evolutionary search found designs that were more ef- performance and efficiency thus depend on their sampling and fective than human designs, even surprising and unlikely to be reward estimation methods. As a representative model-free off- found by humans. In ESP, these elements of strategy search and policy method, Deep Q-Networks (DQN) [30] solves the sample surrogate modeling are combined into a general approach for de- efficiency issue by modeling future rewards using action values, cision strategy optimization. ESP is implemented as part of LEAF, also known as Q values. The Q-network is learned based on a i.e. Cognizant’s Evolutionary AI platform, and is currently being replay buffer that collects training data from real-world interactions. applied to a number of business decision optimization problems. Advanced techniques such as double Q-learning [14] and dueling The goal of this paper is to introduce the ESP approach in gen- network architectures [52] makes DQN competitive in challenging eral, and to extend it further into decision strategies that consist of problems. sequences of decisions. This extension makes it possible to evaluate In terms of on-policy model-free techniques, policy gradient ap- ESP against other methods in RL domains. Conversely, ESP is used proaches (sometimes referred to as deep RL) leverage developments to formalize RL as surrogate-assisted, population based search. This in deep neural networks to provide a general RL solution. Asynchro- approach is particularly compelling in domains where real-world nous Advantage Actor-Critic (A3C) [29] in particular builds policy evaluations are costly. ESP improves upon traditional RL in several critics through an advantage function, which considers both action ways: It converges faster given the same number of episodes, indi- and state values. Proximal Policy Optimization (PPO) [44] further cating better sample-efficiency; it has lower variance for best policy makes actor-critic methods more robust with a clipped surrogate performance, indicating better reliability of delivered solutions; and objective. Unfortunately, policy gradient techniques are typically it has lower regret, indicating lower costs and better safety during Effective Reinforcement Learning through Evolutionary Surrogate-Assisted Prescription GECCO ’20, July 8–12, 2020, Cancun, Mexico

quite sensitive to hyperparameter settings, and require overwhelm- Surrogate Model ing numbers of samples. One reason is the need to train a policy Train Evolve neural network. While expensive, policy gradient methods have Contexts Predictor Prescriptor had success in simulated 3D locomotion [15, 43], Atari video games Actions Outcomes P C A O P C A [30], and, famously, Go [45]. d( , ) = ′ s( ) = ESP will be compared to both DQN and PPO methods in this paper. Compared to ESP, these existing RL approaches have several limitations: First, their performance during real-world interactions Collect Apply cannot be guaranteed, leading to safety concerns [37]. In ESP, only elite agents selected via the surrogate model are evaluated on the Real-World Real World Prescribed Outcomes (Black Box) Actions real world, significantly improving safety. Second, the quality of the best-recognized policy is unreliable because it has not been sufficiently evaluated during learning. ESP solves this issueby Figure 2: The ESP Outer Loop. The Predictor can be trained evaluating all elite policies in the real world for multiple episodes. gradually at the same time as the Prescriptor is evolved, us- Third, existing RL methods rely heavily on deep neural networks. In ing the Prescriptor to drive exploration. That is, the user can contrast, ESP treats the Predictor as a black box, allowing high flex- decide to apply the Prescriptor’s outputs to the real world, ibility in model choices, including simpler models such as random observe the outcomes, and aggregate them into the Predic- forests that are sufficient in many cases. tor’s training set. Evolutionary approaches have been used in RL in several ways. For instance, in evolved policy gradients [17], the loss function given context. The Predictor is therefore defined as against which an agent’s policy is optimized is evolved. The policy ′ loss is not represented symbolically, but rather as a neural network Pd (C, A) = O , (2) that convolves over a temporal sequence of context vectors; the Í ′ such that j L(Oj ,Oj ) across all dimensions j of O is minimized. parameters to this neural network are optimized using evolutionary The function L can be any of the usual loss functions used in ma- strategies. In reward function search [33], the task is framed as a chine learning, such as cross-entropy or mean-squared-error, and genetic programming problem, leveraging PushGP [47]. the model Pd itself can be any supervised machine learning model More significantly, entire policies can be discovered directly such as a neural network or a random forest. with many evolutionary techniques including Genetic Algorithms, The Prescriptor takes a given context as input, and outputs a set Natural Evolutionary Strategies, and Neuroevolution [1, 11, 39, 48, of actions: 49, 55]. Such direct policy evolution can only leverage samples from Ps (C) = A , (3) the real-world, which is costly and risky since many evaluations such that Í O ′(C , A ) over all possible contexts i is maximized. have to take place in low-performing areas of the policy space. i, j j i i This issue in evolutionary optimization led to the development It thus approximates the optimal decision policy for the problem. of surrogate-assisted evolution techniques [12, 18, 22]. Surrogate Note that the optimal actions A are not known, and must therefore methods have been applied to a wide range of domains, ranging be found through search. from turbomachinery blade optimization [35] to protein design The ESP algorithm then operates as an outer loop that constructs [42]. ESP aims to build upon this idea, by relegating the majority the Predictor and Prescriptor models (Figure 2): of policy evaluations to a flexible surrogate model, allowing for a (1) Train a Predictor based on historical training data; wider variety of contexts to be used to evolve better policies. (2) Evolve Prescriptors with the Predictor as the surrogate; (3) Apply the best Prescriptor in the real world; 3 THE ESP APPROACH (4) Collect the new data and add to the training set; The goal of the ESP approach is to find a decision policy that opti- (5) Repeat until convergence. mizes a set of outcomes (Figure 1). Given a set of possible contexts As usual in evolutionary search, the process terminates when a (or states) C and possible actions A, a decision policy D returns a satisfactory level of outcomes has been reached, or no more progress set of actions A to be performed in each context C: can be made. Note that in Step 1, if no historical decision data exists initially, a random Predictor can be used. Also note that not all data D(C) = A , (1) needs to be accumulated for training each iteration. In domains where C ∈ C and A ∈ A. For each such (C, A) pair there is a set of where the underlying relationships between variables might change outcomes O(C, A), i.e. the consequences of carrying out decision over time, it might be advisable to selectively ignore samples from A in context C. For instance, the context might be a description of the older data as more data is added to the training set in Step 4. a patient, actions might be medications, and outcomes might be It is thus possible to bias the training set towards more recent health, side effects, and costs. In the following, higher values of O experiences. are assumed to be better for simplicity. Building the Predictor model is straightforward given a (C, A,O) In ESP, two models are employed: a Predictor Pd , and a Prescrip- dataset. The choice of algorithm depends on the domain, i.e. how tor Ps . The Predictor takes, as its input, context information, as well much data there is, whether it is continuous or discrete, struc- as actions performed in that context. The output of the Predictor is tured or unstructured. Random forests and neural networks will the resulting outcomes when the given actions are applied in the be demonstrated in this role in this paper. The Prescriptor model, GECCO ’20, July 8–12, 2020, Cancun, Mexico Francon, Gonzalez, Hodjat, Meyerson, Miikkulainen, Qiu, and Shahrzad

(b) (c)

(a)

(d)

Figure 3: Visualizing ESP Behavior. (a) True reward for every state-action pair; (b−c) After 1000 episodes, the top Prescriptors for direct evolution (DE) and PPO are still far from optimal; (d) ESP Prescriptor (orange) and Predictor (background) for several iterations. The translucent circles indicate the state-action pairs sampled so far, i.e., the samples on which the Predictor is trained. By 125 episodes, ESP has converged around the optimal Prescriptor, and the ESP Predictor has converged in the neighborhood of this optimum, showing how ESP can leverage Predictors over time to find good actions quickly. Note that the Prescriptor does not exactly match the actions the Predictor would suggest as optimal: the Prescriptor regularizes the Predictor’s overfitting by implicitly ensembling the Predictors evolved against over time. For a full video of the algorithms, see http://evolution.ml/demos/espvisualization. in contrast, is built using neuroevolution in ESP: Neural networks system keeps track of the best Prescriptor so far, as evaluated in because they can express complex nonlinear mappings naturally, the actual domain, and makes sure it stays in the parent population and evolution because it is an efficient way to discover such map- during evolution. This process discovers good Prescriptor agents pings [48], and naturally suited to optimize multiple objectives efficiently, as will be described in the experiments that follow. [4, 10]. Because it is evolved with the Predictor, the Prescriptor is not restricted by a finite training dataset, or limited opportunities 4 EXPERIMENTS to evaluate in the real world. Instead, the Predictor serves as a fitness function, and it can be queried frequently and efficiently. In a ESP was evaluated in three domains: Function approximation, where multiobjective setting, ESP produces multiple Prescriptors, selected its behavior could be visualized concretely; Cart-pole control [2] from the Pareto front of the multiobjective neuroevolution run. where its performance could be compared to standard RL meth- Applying the ESP framework to RL problems involves extending ods in a standard RL benchmark task; and Flappy Bird, where the the contexts and actions to sequences. The Prescriptor can be seen regularization effect of the surrogate could be demonstrated most as an RL agent, taking the current context as input, and deciding clearly. what actions to perform in each time step. The output of the Pre- The neuroevolution algorithm for discovering Prescriptors evolves dictor, O ′, can be seen as the reward vector for that step, i.e. as weights of neural networks with fixed topologies. Unless otherwise Q values [53] (with a given discount factor, such as 0.9, as in the specified, all experiments use the following default setup for evo- experiments below). Evolution thus aims to maximize the predicted lution: candidates have a single hidden layer with bias and tanh reward, or minimize the regret, throughout the sequence. activation; the initial population uses orthogonal initialization of The outer loop of ESP changes slightly because in RL there is no layer weights with a mean of 0 and a standard deviation of 1 [40]; dataset to train the Predictor; instead, the data needs to be generated the population size is 100; the top 10% of the population is car- by applying the current Prescriptors to the domain. An elite set ried over as elites; parents are selected by tournament selection of of several good Prescriptors are used in this role to create a more the top 20% of candidates; recombination is performed by uniform diverse training set. The initial training set is created randomly. crossover at the weight-level; there is a 10% probability of multi- The loop now is: plying each weight by a mutation factor, where mutation factors are drawn from N(1, 0.1). (1) Apply the elite Prescriptors in the actual domain; (2) Collect Q values for each time step for each Prescriptor; (3) Train a Predictor based on data collected in Step 2; 4.1 Visualizing ESP Behavior (4) Evolve Prescriptors with the Predictor as the surrogate; This section demonstrates the behavior of ESP in a synthetic func- (5) Repeat until convergence. tion approximation domain where its behavior can be visualized. The evolution of Prescriptors continues in each iteration of this The domain also allows comparing ESP to direct evolution in the loop from where it left off in previous iteration. In addition, the domain, as well as to PPO, and visualizing their differences. Effective Reinforcement Learning through Evolutionary Surrogate-Assisted Prescription GECCO ’20, July 8–12, 2020, Cancun, Mexico

Problem Description. The domain has a one-dimensional context C and a one-dimensional action A, with outcome O given by the C function O = −|A − 3 sin 2 |. The optimal action for each context lies on a periodic curve, which captures complexity that can arise from periodic variables such as time of day or time of year. The outcome of each action decreases linearly as the action moves away from the optimal action. Episodes in this domain consist of single action in [−10, 10], which is taken in a context drawn uniformly over [−10, 10]. The full domain is shown in Figure 3(a). Algorithm Setup. ESP begins by taking ten random actions. There- after, every iteration, ESP trains a Predictor neural network with two hidden layers of size 64 and tanh activation for 2000 epochs using the Adam optimizer [24] with default parameters to minimize (a) True Performance (b) Regret MSE, and evolves Prescriptor neural networks for 20 generations against this Predictor. Then, the top Prescriptor is run in the real Figure 4: Performance in the Function Approximation Do- domain for a single episode. Prescriptors have a single hidden layer main. The horizontal axis indicates the total number of real- of size 32 with tanh activation; default parameters are used for world episodes used by training and the vertical axis the evolution. performance at that point. Ten independent runs were per- Direct Evolution (DE) was run as a baseline comparison for ESP. formed for each method. Solid lines represent the mean over It consists of the exact same evolution process, except that it is run 10 runs, and colored areas show the corresponding standard directly against the real function instead of the Predictor. That is, deviation. (a) The true performance of the returned best in each generation, all 100 candidates are evaluated on one episode agent converges substantially faster with ESP. (b) ESP also from the real function. operates with much lower regret than DE or PPO, converg- PPO was run as an RL comparison, since it is a state-of-the-art RL ing to very low regret behavior orders-of-magnitude faster approach for continuous action spaces [44]. During each iteration than the other approaches (the plot shows the moving aver- it was run for ten episodes, since this setting was found to perform age regret over the last 100 training episodes). The standard best during hyperparameter search. PPO defaults1 were used for deviation of ESP is small in both metrics, attesting to the re- the remaining hyperparameters. liability of the method. Ten independent runs were performed for each algorithm. The returned policy at any time for DE and ESP is the candidate with the highest fitness in the population; for PPO it is the learned policy run without stochastic exploration. Qualitative Results. Snapshots of the convergence behavior for Quantitative Results. The numerical performance results in Fig- each method are shown in Figures 3(b-d). After 1000 episodes, ure 4 confirm the substantial advantage of ESP. ESP converges neither DE nor PPO converged near the optimal solution. On the rapidly to a high-performing returned solution (Figure 4(a)), while other hand, ESP discovered the periodic nature of the problem taking actions with significantly lower regret (defined as the reward within 50 episodes, and converged almost exactly to the optimal difference between optimal solution and current policy) than the within 125 episodes. other methods during learning (Figure 4(b)). On both metrics, ESP The Predictor’s predicted reward for each state-action pair is converges orders-of-magnitude faster than the other approaches. shown using the background colors in each snapshot of ESP. The In particular, after a few hundred episodes, ESP reaches a solution rapid convergence of the Predictor highlights the sample efficiency that is significantly better than any found by DE or PPO, even after of ESP, due to aggressive use of historical data (shown as translu- 3,000 episodes. This result shows that, beyond being more sample cent circles). Note, however, that the Predictor does not converge efficient, by systematically exploiting historical data, ESP is able to the ground truth over the entire domain; it does so just in the to find solutions that direct evolution or policy gradient search neighborhood of the optimal Prescriptor. Thus, ESP avoids exces- cannot. sive costly exploration of low-quality actions once the structure of The next sections show how these advantages of ESP can be optimal actions has become clear. harnessed in standard RL benchmarks, focusing on the advantage of Note also that the Prescriptor does not follow the optimal action surrogate modeling over direct evolution in the domain, comparing suggested by the Predictor at every iteration exactly. Since it maps to DQN and PPO, and demonstrating the regularization effect of states directly to actions, the Prescriptor provides a smoothing reg- the surrogate. ularization in action space that can overcome Predictor overfitting. Also, since the top ESP Prescriptors must survive across many dif- 4.2 Comparing with Standard RL ferent Predictors over time, ESP benefits from an implicit temporal The goal of the Cart-pole experiments was to demonstrate ESP’s ensembling, which further improves regularization. performance compared to direct evolution and standard RL meth- 1https://stable-baselines.readthedocs.io/en/master/modules/ppo2.html ods. GECCO ’20, July 8–12, 2020, Cancun, Mexico Francon, Gonzalez, Hodjat, Meyerson, Miikkulainen, Qiu, and Shahrzad

ESP 200 DE PPO 200 175 DQN

150 150 125

100 100

50 50 ESP

Mean Regret per Episode during Training DE 25 PPO Mean Reward of Returned Policy on 100 Episodes 0 DQN

0 100 200 300 400 500 600 700 800 0 200 400 600 800 Training Episodes Training Episodes

(a) True Performance (b) Regret

Figure 5: Performance in the Cart-Pole Domain. The same experimental and plotting conventions were used as in Figure 4. (a) True performance of the best policy returned by each method during the learning process. True performance is based on the average reward of 100 real-world episodes (this evaluation is not part of the training). ESP converges significantly faster and has a much lower variance, demonstrating better sample efficiency and reliability than the other methods. (b) Average regret over all past training episodes. ESP has significantly lower regret, suggesting that it has lower cost and is safer in real-world applications.

Problem Description. The Cart-pole control domain is one of the In order to be evaluated against the Predictor, a Prescriptor can- standard RL benchmarks. In the popular CartPole-v0 implemen- didate has to prescribe an action for each observation vector from tation on the OpenAI Gym platform [3] used in the experiments, the collected samples. The action is then concatenated with the there is a single pole on a cart that moves left and right depending observation vector and passed to the Predictor to get the predicted on the force applied to it. A reward is given for each time step that future reward. The fitness of the candidate is the average ofthe the pole stays near vertical and the cart stays near the center of the predicted future rewards. track; otherwise the episode ends. Every five generations, data is collected from the game fromthe five elites, for five episodes each. The new data is aggregated into the training set and a new Predictor is trained. The generation’s Algorithm Setup. DE was run with a population size of 50 candi- candidates are then evaluated on the new Predictor with the new dates. A candidate is a neural network with four inputs (observa- training data. The top elite candidate is also evaluated for 100 tions), one hidden layer of 32 units with tanh activation, and two episodes on the game for reporting purposes only. Evolution is outputs (actions) with argmax activation functions. The fitness of stopped after 160 generations, which corresponds to 800 episodes each candidate is the average reward over five episodes in the game, played from the game, or once an elite receives an average reward where the maximum episode length is 200 time steps. of 200 on five episodes. ESP runs similarly to DE, except that the fitness of each candidate In addition to DE and ESP, two state-of-the-art RL methods were is evaluated against the Predictor instead of the game. The Predictor implemented for comparison: double DQN with dueling network is a neural network with six inputs (four observations and two architectures [30, 52] and actor-critic style PPO [44]. The imple- actions), two hidden layers with 64 units each and tanh activation, mentation and parametric setup of DQN and PPO were based on and one output (the predicted discounted future reward) with tanh OpenAI Baselines [8]. For PPO, the policy’s update frequency was activation. It is trained for 1,000 epochs with the Adam optimizer set to 20, which was found to be optimal during hyperparameter [24] with MSE loss and batch size of 256. search. All other parametric setups of DQN and PPO utilized default The first Predictor is trained on samples collected from five setups as recommended in OpenAI Baselines. random agents playing five episodes each. Random agents choose a uniform random action at each time step. A sample corresponds Results. Figure 5(a) shows how the true performance of the best to a time step in the game and comprises four observations, two policy returned by ESP, DE, PPO, and DQN changes during the actions, and the discounted future reward. Reward is +1 on each learning process in CartPole-v0. For ESP and DE, the elite candidate time step, except for the last one where it is adjusted to +2, 000 in that has the best real-world fitness is selected as the best policy so − × case of success, 2, 000 in case of failure (i.e. 10 max time steps). far. For DQN and PPO, whenever the moving average reward of The discount factor is set to 0.9. The reward value is then scaled to the past 100 episodes of training is increased, the best policy will lie between -1 and 1. be updated using the most recent policy. One hundred additional Table 1

Effective Reinforcement Learning through Evolutionary Surrogate-Assisted Prescription GECCO ’20, July 8–12, 2020, Cancun, Mexico

Ground-Truth ESP Surrogate 3500 DE

3000

2500 Fitness

2000

1500

Domain Feature 1000

Figure 6: Surrogate Approximation of the Fitness Landscape. 500

The fitness in the actual domain may be deceptive and Mean Reward of Returned Policy on 10 Episodes nonlinear, for instance single actions can have large conse- 0 quences. The surrogate learns to approximate such a land- 0 100000 200000 300000 400000 500000 600000 700000 800000 Training Episodes scape, thereby creating a surrogate landscape that is easier to search and optimize. Figure 7: Performance in the Flappy Bird Domain. The same experimental and plotting conventions were used as in Fig- real-world episodes were used to evaluate the best policies (these ure 5(a), except true performance of the best policy returned evaluations are not part of the training). is based on the average reward of 10 real-world episodes dur- ESP converges significantly faster than the other methods, imply- ing the training. ESP discovers a policy that solves the task ing better sample-efficiency during learning. Moreover, the variance very quickly, whereas DE cannot discover it even though it of the true performance becomes significantly smaller for ESP after is run an order of magnitude longer. This result is likely due an early stage, while all other algorithms have high variances even to the regularization effect that the surrogate provides as during later stages of learning. This observation demonstrates that shown in Figure 6. the solutions delivered by ESP are highly reliable. Figure 5(b) shows the average regret for training processes of all algorithms in CartPole-v0. ESP has significantly lower regret each a neural network with eight inputs (observations), one hidden during the entire learning process, indicating not only lower costs layer of 128 nodes with tanh activation, and two outputs (actions) but also better safety in real-world interactions. with argmax activation. The ESP Predictor was a random forest with 100 estimators, approximating reward values for state-action 4.3 Regularization Through Surrogate pairs frame by frame. The state-action pairs were collected with Modeling the ten best candidates of each generation running ten episodes on the actual game, for the total of a hundred episodes per generation. Whenever a surrogate is used to approximate a fitness function, there is a risk that the surrogate introduces false optima and mis- Results. Figure 7 shows how the true performance of the best leads the search [21] (for an inspiring collection of similar empirical policy returned by ESP and DE improved during the learning pro- phenomena, reference Lehman et al. [25].) ESP mitigates that risk by cess. The elite candidate that has the best real-world fitness was alternating between actual domain evaluations and the surrogate. selected as the best policy so far. In about 80,000 episodes, ESP dis- However, the opposite effect is also possible: Figure 6 shows how covered a policy that solved the task, i.e. was able to guide the bird the surrogate may form a more regularized version of the fitness through the entire course of pipes without hitting any of them. It is than the real world, and thereby make it easier to learn policies interesting that DE converged to a suboptimal policy even though that generalize well [18, 34]. it was run an order of magnitude longer. This result is likely due to the regularization effect illustrated in Figure 6. Direct evolution Problem Description. Flappy Bird is a side-scroller game where overfits to the nonlinear effects in the game, whereas the surrogate the player controls a bird, attempting to fly it between columns helps smooth the search landscape, thereby leading evolution to of pipes without hitting them by performing flapping actions at policies that perform better. carefully chosen times. This experiment is based on a PyGame [50] implementation of this game, running at a speed of 30 frames per 5 DISCUSSION AND FUTURE WORK second. The goal of the game is to finish ten episodes of two minutes, or 3,600 frames each, through random courses of pipes. A reward The results in this paper show that the ESP approach performs well is given for each frame where the bird does not collide with the in sequential decision making tasks like those commonly used as boundaries or the pipes; otherwise the episode ends. The score of benchmarks for RL. Compared to direct evolution and state-of-the- each candidate is the average reward over the ten episodes. art RL methods, it is highly sample efficient, which is especially important2 in domains where exploring with the real world is costly. Algorithm Setup. Both DE and ESP were setup in a similar way Its solutions are also reliable and safe, and the complexity of its as in the preceding sections. DE had a population of 100 candidates, models can be adjusted depending on the complexity of the data and GECCO ’20, July 8–12, 2020, Cancun, Mexico Francon, Gonzalez, Hodjat, Meyerson, Miikkulainen, Qiu, and Shahrzad task. These advantages apply to ESP in general, including decision used to evaluate candidate policies with minimal exploration cost. strategies that are not sequential, which suggests that it is a good Extended into sequential decision making, ESP is highly sample effi- candidate for improving decision making in real-world applications, cient, has low variance, and low regret, making the policies reliable including those in business, government, education, and healthcare. and safe. Surprisingly, the surrogate also regularizes decision mak- When ESP is applied to such practical problems, the process ing, making it sometimes possible to discover good policies even outlined in Section 3 can be extended further in several ways. First, when direct evolution fails. ESP is therefore a promising approach ESP can be most naturally deployed to augment human decision to improving decision making in many real world applications making. The Presciptor’s output is thus taken as advice, and the hu- where historical data is available. man decision maker can modify the actions before applying them. These actions and their eventual outcomes are still captured and REFERENCES processed in Step 4 of the ESP process, and thus become part of [1] Ackley, D. and Littman, M. 1991. Interactions between learning and evolution. the learning (Figure 2). Second, to support human decision making, Artificial Life II 10 (1991), 487–509. an uncertainty estimation model such as RIO [36] can be applied [2] Barto, A. G., Sutton, R. S., and Anderson, C. W. 1983. Neuronlike Adaptive to the Predictor, providing confidence intervals around the out- Elements That Can Solve Difficult Learning Control Problems. IEEE Transactions ′ on Systems, Man, and Cybernetics 13 (1983), 834–846. come O . Third, the continual new data collection in the outer loop [3] Brockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J., and makes it possible to extend ESP to uncertain environments and to Zaremba, W. 2016. OpenAI Gym. CoRR abs/1606.01540 (2016). [4] Coello Coello, C. A. 1999. A Comprehensive Survey of Evolutionary-Based dynamic optimization, where the objective function changes over Multiobjective Optimization Techniques. International Journal of Knowledge and time [18, 19, 57]. By giving higher priority to new examples, the Information Systems 1 (1999), 269–308. Predictor can be trained to track such changing objectives. Fourth, [5] Cressie, N. 1990. The origins of kriging. Mathematical Geology 22, 3 (1990), 239–252. in some domains, such as those in financial services and healthcare [6] Deb, K. and Myburgh, C. 2017. A population-based fast algorithm for a billion- industries that are strongly regulated, it may be necessary to justify dimensional resource allocation problem with integer variables: Breaking the the actions explicitly. Rather than evolving a Prescriptor as a neural Billion-Variable Barrier in Real-World. European Journal of Operational Research 261 (2017), 460–474. network, it may be possible to evolve rule-set representations [16] [7] Deisenroth, M. and Rasmussen, C. E. 2011. PILCO: A model-based and data- for this role, thus making the decision policy explainable. Such ex- efficient approach to policy search. In Proceedings of the 28th International Con- ference on Machine Learning (ICML) (ICMLâĂŹ11). 465–472. tensions build upon the versatility of the ESP framework, and make [8] Dhariwal, P., Hesse, C., Klimov, O., Nichol, A., Plappert, M., Radford, A., Schulman, it possible to incorporate the demands of real-world applications. J., Sidor, S., Wu, Y., and Zhokhov, P. 2017. OpenAI Baselines. https://github.com/ Although the Predictor does not have to be perfect, and its ap- openai/baselines. (2017). [9] Dreer, L. E. and Linley, A. 2017. Behavioral Medicine: Nutrition, Medication proximate performance can even lead to regularization as was Management, and Exercise. In Practical Psychology in Medical Rehabilitation, discussed in Section 4.3, it is sometimes the bottleneck in building M. Budd, S. Hough, S. Wegener, and W. Stiers (Eds.). Springer. an application of ESP. In the non-sequential case, the training data [10] Emmerich, M. and Deutz, A. 2018. A tutorial on multiobjective optimization: Fundamentals and evolutionary methods. Natural Computation 17 (2018), 585âĂŞ– may not be sufficiently complete, and in the sequential case, itmay 609. be difficult to create episodes that run to successful conclusion early [11] Gomez, F., Schmidhuber, J., and Miikkulainen, R. 2006. Efficient non-linear control through neuroevolution. In Proceedings of the European Conference on Machine in the training. Note that in this paper, the Predictor was trained Learning. Springer, 654–662. with targets from discounted rewards over time. An alternative [12] Grefenstette, J. J. and Fitzpatrick, J. M. 1985. Genetic search with approximate approach would be to incrementally extend the time horizon of function evaluations. In Proceedings of the International Conference on Genetic Algorithms and their Applications. 112–120. Predictors by training them iteratively [38]. Such an approach could [13] Ha, D. and Schmidhuber, J. 2018. Recurrent World Models Facilitate Policy help resolve conflicts between Q targets, and thereby help in early Evolution. In Advances in Neural Information Processing Systems 32 (NIPSâĂŹ18). training. Another approach would be to make the rewards more in- Curran Associates Inc., Red Hook, NY, USA, 2455âĂŞ2467. [14] Hasselt, H. V. 2010. Double Q-learning. In Advances in Neural Information cremental, or evolve them using reward function search [17, 33]. It Processing Systems 23, J. D. Lafferty, C. K. I. Williams, J. Shawe-Taylor, R. S. Zemel, may also be possible to evaluate the quality of the Predictor directly, and A. Culotta (Eds.). Curran Associates, Inc., 2613–2621. and adjust sampling from the real world accordingly [20]. [15] Heess, N., TB, D., Sriram, S., Lemmon, J., Merel, J., Wayne, G., Tassa, Y., Erez, T., Wang, Z., Eslami, S., and others, . 2017. Emergence of locomotion behaviours in An interesting future application of ESP is in multiobjective rich environments. arXiv:1707.02286 (2017). domains. In many real-world decision-making domains, there are [16] Hodjat, B., Shahrzad, H., Miikkulainen, R., Murray, L., and Holmes, C. 2018. PRETSL: Distributed Probabilistic Rule Evolution for Time-Series Classification. at least two conflicting objectives: performance and cost. Asan In Genetic Programming Theory and Practice XIV. Springer, 139–148. evolutionary approach, ESP lends itself well to optimizing multiple [17] Houthooft, R., Chen, Y., Isola, P., Stadie, B., Wolski, F., Ho, O. J., and Abbeel, P. objectives [4, 10]. The population forms a Pareto front, and multi- 2018. Evolved policy gradients. In Advances in Neural Information Processing Systems 31. Curran Associates, Inc., 5400–5409. ple Prescriptors can be evolved to represent the different tradeoffs. [18] Jin, Y. 2011. Surrogate-assisted evolutionary computation: Recent advances and Extending the work in this paper, it would be illuminating to com- future challenges. Swarm and Evolutionary Computation 1, 2 (06 2011), 61–70. pare ESP in such domains to recent efforts in multiobjective RL https://doi.org/10.1016/j.swevo.2011.05.001 [19] Jin, Y. and Branke, J. 2005. Evolutionary optimization in uncertain environments- [26, 31, 56], evaluating whether there are complementary strengths a survey. IEEE Transactions on Evolutionary Computation 9, 3 (June 2005), 303–317. that could be exploited. https://doi.org/10.1109/TEVC.2005.846356 [20] Jin, Y., Husken, M., and Sendhoff, B. 2003. Quality measures for approximate models in evolutionary computation. In Proceedings of the Bird of a Feather Workshop, Genetic and Evolutionary Computation Conference (GECCO). 170–173. 6 CONCLUSION [21] Jin, Y., Olhofer, M., and Sendhoff, B. 2000. On Evolutionary Optimization with ESP is a surrogate-assisted evolutionary optimization method de- Approximate Fitness Functions. 786–793. [22] Jin, Y., Wang, H., Chugh, T., Guo, D., and Miettinen, K. 2019. Data-Driven signed specifically for discovering decision strategies in real-world Evolutionary Optimization: An Overview and Case Studies. IEEE Transactions applications. Based on historical data, a surrogate is learned and on Evolutionary Computation 23 (2019), 442–458. Effective Reinforcement Learning through Evolutionary Surrogate-Assisted Prescription GECCO ’20, July 8–12, 2020, Cancun, Mexico

[23] Johnson, A. J., Meyerson, E., de la Parra, J., Savas, T. L., Miikkulainen, R., and [40] Saxe, A. M., Mcclelland, J. L., and Ganguli, S. 2014. Exact solutions to the nonlinear Harper, C. B. 2019. Flavor-Cyber-Agriculture: Optimization of plant metabolites dynamics of learning in deep linear neural network. In Proceedings of the Second in an open-source control environment through surrogate modeling. PLOS ONE International Conference on Learning Representations (ICLR). Citeseer. (2019). https://doi.org/10.1371/journal.pone.0213918. [41] Schmidhuber, J. and Huber, R. 1991. Learning to generate artificial fovea trajecto- [24] Kingma, D. P. and Ba, J. 2014. Adam: A method for stochastic optimization. ries for target detection. International Journal of Neural Systems 2, 01n02 (1991), arXiv:1412.6980 (2014). 125–134. [25] Lehman, J., Clune, J., Misevic, D., Adami, C., Beaulieu, J., Bentley, P. J., Bernard, [42] Schneider, G., Schuchhardt, J., and Wrede, P. 1994. Artificial neural networks and S., Beslon, G., Bryson, D. M., Chrabaszcz, P., Cheney, N., Cully, A., Doncieux, S., simulated molecular evolution are potential tools for sequence-oriented protein Dyer, F. C., Ellefsen, K. O., Feldt, R., Fischer, S., Forrest, S., Frénoy, A., Gagné, design. Bioinformatics 10, 6 (1994), 635–645. C., Goff, L. K. L., Grabowski, L. M., Hodjat, B., Hutter, F., Keller, L., Knibbe, C., [43] Schulman, J., Moritz, P., Levine, S., Jordan, M., and Abbeel, P. 2016. High- Krcah, P., Lenski, R. E., Lipson, H., MacCurdy, R., Maestre, C., Miikkulainen, R., dimensional continuous control using generalized advantage estimation. In Pro- Mitri, S., Moriarty, D. E., Mouret, J., Nguyen, A., Ofria, C., Parizeau, M., Parsons, ceedings of the Fourth International Conference on Learning Representations (ICLR). D. P., Pennock, R. T., Punch, W. F., Ray, T. S., Schoenauer, M., Shulte, E., Sims, K., [44] Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. 2017. Proximal Stanley, K. O., Taddei, F., Tarapore, D., Thibault, S., Weimer, W., Watson, R., and Policy Optimization Algorithms. CoRR abs/1707.06347 (2017). Yosinksi, J. 2018. The Surprising Creativity of Digital Evolution: A Collection [45] Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., Van Den Driessche, G., of Anecdotes from the Evolutionary Computation and Artificial Life Research Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., and others, Communities. arXiv:1803.03453 (2018). . 2016. Mastering the game of Go with deep neural networks and tree search. [26] Liu, C., Xu, X., and Hu, D. 2015. Multiobjective Reinforcement Learning: A Nature 529, 7587 (2016), 484. Comprehensive Overview. IEEE Transactions on Systems, Man, and Cybernetics [46] Snoek, J., Larochelle, H., and Adams, R. P. 2012. Practical Bayesian Optimization 45 (2015), 385–398. of Machine Learning Algorithms. In Advances in Neural Information Processing [27] Miikkulainen, R. 2019. Creative AI Through Evolutionary Computation. In Systems 25, F. Pereira, C. J. C. Burges, L. Bottou, and K. Q. Weinberger (Eds.). Evolution in Action: Past, Present and Future, Banzhaf et al. (Ed.). Springer, New Curran Associates, Inc., 2951–2959. York. [47] Spector, L., Goodman, E., Wu, A., B. Langdon, W., Voigt, m. H., Gen, M., Sen, [28] Miikkulainen, R., Brundage, M., Epstein, J., Foster, T., Hodjat, B., Iscoe, N., Jiang, J., S., Dorigo, M., Pezeshk, S., Garzon, M., Burke, E., and Kaufmann Publishers, M. Legrand, D., Nazari, S., Qiu, X., Scharff, M., Schoolland, C., Severn, R., and Shagrin, 2001. Autoconstructive Evolution: Push, PushGP, and Pushpop. Proceedings of A. In Press. Ascend by Evolv: AI-Based Massively Multivariate Conversion Rate the Genetic and Evolutionary Computation Conference (GECCO) (05 2001). Optimization. AI Magazine (In Press). [48] Stanley, K. O., Clune, J., Lehman, J., and Miikkulainen, R. 2019. Designing Neural [29] Mnih, V., Badia, A. P., Mirza, M., Graves, A., Lillicrap, T., Harley, T., Silver, D., and Networks through Evolutionary Algorithms. Nature Machine Intelligence 1, 1 Kavukcuoglu, K. 2016. Asynchronous methods for deep reinforcement learning. (2019), 24âĂŞ35. In Proceedings of the 33rd International Conference on International Conference on [49] Such, F. P., Madhavan, V., Conti, E., Lehman, J., Stanley, K. O., and Clune, J. Machine Learning (ICML) (ICMLâĂŹ16). 1928–1937. 2017. Deep neuroevolution: Genetic algorithms are a competitive alternative [30] Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., for training deep neural networks for reinforcement learning. arXiv:1712.06567 Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., and others, . 2015. (2017). Human-level control through deep reinforcement learning. Nature 518, 7540 [50] Tasfi, N. 2016. PyGame Learning Environment. https://github.com/ntasfi/ (2015), 529–533. PyGame-Learning-Environment. (2016). [31] Mossalam, H., Assael, Y. M., Roijers, D. M., and Whiteson, S. 2016. Multi-Objective [51] Wahlström, N., Schön, T. B., and Deisenroth, M. P. 2015. From pixels to torques: Deep Reinforcement Learning. arXiv:1610.02707 (2016). Policy learning with deep dynamical models. arXiv:1502.02251 (2015). [32] Naik, P. A., Raman, K., and Winer, R. S. 2005. Planning Marketing-Mix Strategies [52] Wang, Z., Schaul, T., Hessel, M., Van Hasselt, H., Lanctot, M., and De Freitas, in the Presence of Interaction Effects. Marketing Science 24 (2005), 25–34. N. 2016. Dueling Network Architectures for Deep Reinforcement Learning. In [33] Niekum, S., Barto, A. G., and Spector, L. 2010. Genetic programming for reward Proceedings of the 33rd International Conference on International Conference on function search. IEEE Transactions on Autonomous Mental Development 2, 2 (2010), Machine Learning (ICML) (ICMLâĂŹ16), Vol. 48. JMLR.org, 1995âĂŞ2003. 83–90. [53] Watkins, C. J. C. H. 1989. Learning from delayed rewards. Ph.D. Dissertation. [34] Ong, Y., Nair, P., and Keane, A. 2003. Evolutionary Optimization of Computa- Cambridge University. tionally Expensive Problems via Surrogate Modeling. AIAA Journal (04 2003). [54] Werbos, P. J. 1987. Learning how the world works: Specifications for predictive https://doi.org/10.2514/2.1999 networks in robots and brains. In Proceedings of the IEEE International Conference [35] Pierret, S. and Braembussche, Van den R. 1999. Turbomachinery blade design on Systems, Man and Cybernetics, NY. using a Navier-Stokes solver and Artificial Neural Network. Journal of Turboma- [55] Whiteson, S. 2012. Evolutionary computation for reinforcement learning. In chinery 121, 2 (1999), 326–332. Reinforcement Learning. Springer, 325–355. [36] Qiu, X., Meyerson, E., and Miikkulainen, R. 2020. Quantifying Point-Prediction [56] Yang, R., Sun, X., and Narasimhan, K. 2019. A Generalized Algorithm for Multi- Uncertainty in Neural Networks via Residual Estimation with an I/O Kernel. In Objective Reinforcement Learning and Policy Adaptation. In Advances in Neural Proceedings of the Eighth International Conference on Learning Representations Information Processing Systems 32, H. Wallach, H. Larochelle, A. Beygelzimer, (ICLR). F. d'Alché-Buc, E. Fox, and R. Garnett (Eds.). Curran Associates, Inc., 14610– [37] Ray, A., Achiam, J., and Amodei, D. 2019. Benchmarking Safe Exploration in 14621. Deep Reinforcement Learning. (2019). https://cdn.openai.com/safexp-short.pdf [57] Yu, X., Jin, Y., Tang, K., and Yao, X. 2010. Robust optimization over time âĂŤ A new [38] Riedmiller, M. 2005. Neural fitted Q iteration–first experiences with a data perspective on dynamic optimization problems. In IEEE Congress on Evolutionary efficient neural reinforcement learning method. In Proceedings of the European Computation. 1–6. https://doi.org/10.1109/CEC.2010.5586024 Conference on Machine Learning. Springer, 317–328. [39] Salimans, T., Ho, J., Chen, X., Sidor, S., and Sutskever, I. 2017. Evolution Strategies as a Scalable Alternative to Reinforcement Learning. arXiv:1703.03864 (2017).