Expert-Assisted Transfer Reinforcement Learning

Dylan Slack

April 30, 2019 Version: Final Draft Advisor: Sorelle Friedler Abstract

Reinforcement Learning is concerned with developing machine learning approaches to answer the question: "What should I do?" Transfer Learning attempts to use previously trained Reinforcement Learning models to achieve better performance in new domains. There is relatively little work in using expert-advice to achieve better performance in the transfer of reinforcement learning models across domains. There is even less work that concerns the use of expert-advice in transfer deep reinforcement learning. This thesis presents a method that gives experts a route to incorporate their domain knowledge in the transfer learning process in deep reinforcement learning scenarios by presenting them with a decision set that they can edit. The decision set is then included in the learning process in the target domain. This work describes relevant background to both Reinforcement and Transfer Learning, provides the implementation of the proposed method, and suggests its usefulness and limitations by providing a series of applications.

iii Acknowledgements

I would like to thank my advisor, Sorelle Friedler, for her continued support and mentorship throughout my undergraduate course of study. Also, I also want to thank my second reader, Siddharth Bhaskar, for his thoughtful feedback and Chris Villalta for continued conversations about reinforcement learning throughout the year. Lastly, I would like to thank my parents, Phil and Jessica Slack for their support over the past four years

iv Contents

1 Introduction 2 1.1 What is Reinforcement Learning? ...... 2 1.1.1 Reinforcement Learning Setup ...... 3 1.1.2 Formal Deﬁnition of Reinforcement Learning ...... 3 1.2 Reinforcement Learning Policy Methods ...... 6 1.2.1 Value Iteration ...... 6 1.2.2 Q-Learning ...... 7 1.2.3 Deep Q-Learning ...... 8 1.2.4 Exploration vs. Exploitation Trade Off ...... 9 1.2.5 Transfer Reinforcement Learning ...... 10 1.2.6 Transfer Learning Procedure ...... 10 1.2.7 Skill Transfer in Reinforcement Learning with Advice ...... 11 1.2.8 Interpretability Concerns of Transfer Reinforcement Learning with Advice 12

2 Statement of The Problem 13 2.1 Explaining Black Box Models ...... 13 2.2 Problem Statement: Expert-Assisted (and Interpretable) Transfer Learning . . . . 14

3Methods 16 3.1 Model Road Map ...... 16 3.2 Deep Q-Learning With Experience Replay ...... 16 3.3 Decision Lists as an Interpretable Transfer Learning Source ...... 17 3.3.1 CN2 Rule Induction ...... 17 3.3.2 Expert modiﬁed Rules ...... 18 3.3.3 Adapting DQN to Incorporate Transferred Knowledge ...... 18

4 Incorporating Domain Knowledge in Transfer Learning Process 19 4.1 Circle to Circle Environment ...... 19 4.2 Circle to Ellipse Environment ...... 21 4.3 Mountain Car Environment ...... 22

5 Conclusion and Future Work 27

Bibliography 28

1 Introduction 1

1.1 What is Reinforcement Learning?

Reinforcement Learning refers to a class of problems and solutions that involve decision making and determining what actions to take to maximize a reward output. Like other machine learning models, reinforcement learners are not told how to behave. They discover what actions to take through trial and error. Actions taken in one situation may affect the state of the situation later on. This description deﬁnes three distinguishing features of reinforcement learners. [Sutton and Barto, 1998]

• Closed-loop: Actions taken at one point affect the inputs to the algorithm later on.

• No Instructions: Speciﬁc actions to take are not speciﬁed before hand. The model deter- mines what actions to take.

• Consequences: Actions taken at one point affect the rewards generated over time.

There are two other classes of machine learning problems commonly referenced in literature. Supervised Learning attempts to generate a function fˆ(x) that best estimates a preexisting set of labeled training data y. The goal is to infer characteristics of the underlying data sets so that new data can be estimated by fˆ. Supervised learning is successful in cases where there is already a lot of data or it is very easy to acquire new data. However when data is costly to obtain or very sparse, it can be difﬁcult to build a supervised learning model. In addition, Unsupervised Learning is concerned with discovering the underlying structure of unlabeled data [Le, 2013].

Both supervised learning and unsupervised learning are different from reinforcement learning. Supervised learning requires pre-labeled data. Generating a labeled data set for an interaction scenario is often infeasible because it is difﬁcult to determine all the different scenarios before hand [Sutton and Barto, 1998]. While there is no labeled data in unsupervised learning, the goal of unsupervised learning is to determine the structure of data whereas the goal of reinforcement learning is to generate a maximum reward signal.

To provide concreteness, consider the game of chess. Building a comprehensive data set of chess games would be time consuming and expensive. Further, if we were to train a supervised learner

2 Tab. 1.1: An illustration of the reinforcement learning cycle. Here, t is the current time step [Sutton and Barto, 1998].

off of such a data set, the approach would imitate human experts and fail to uncover strategies that exceed the ability of human experts. By using reinforcement learning, we can create a chess playing algorithm that learns how to play chess by continuously simulating games of chess. The algorithm incrementally improves over simulations as it tests different strategies and learns the best moves to make. In this way, we only have to provide a simulation environment for the algorithm and the algorithm is not bound by current human chess ability.

1.1.1 Reinforcement Learning Setup

The general setup for reinforcement learning is where the reinforcement learner is given a set of actions in can take in an environment. The environment could be a video game. The actions could be choosing to move left or right in the game. The reinforcement learner takes actions for some number of steps in the environment and receives reward based on how good the new environment is after taking an action. For instance, killing an enemy in a video game might generate a lot of reward while dying would be punished (negative reward). The reinforcement learner uses the actions, states of the environment, and rewards to learn a policy function (e.g. artiﬁcial neural network) that maps between states of the environment and actions.

1.1.2 Formal Deﬁnition of Reinforcement Learning

We now provide formality for the intuition introduced previously. We can model the reinforcement learning problem as a finite state Markov decision process (MDP). We can think of the finite state MDP as a tuple ( , , ,⁄, , ). Here, is a finite set of states, is a finite set of S A T D R S A actions (where (s) such that s S yields the set of possible actions at s), is a set of state A œ T transition probabilities (where is defined as the probability distribution of the next state given T an initial state and an action), ⁄ is a discount factor, and is a reward function. The reward R function yields the reward given a state and an action [Abbeel and Ng, 2004].

For all s , the probability some new state s and reward r occurs at time t is deﬁned œS Õ œS œR by the Markov Property. The Markov Property is a classic result in the theory of MDP’s [Markov, 1954].

1.1 What is Reinforcement Learning? 3 p(sÕ,r s, a)˙=Pr St = sÕ,Rt = r St 1 = s, At 1 = a (1.1) | { | ≠ ≠ }

The Markov Property says that the probability of a state occurring only depend on its previous state and action. Each choice s and a define a probability distribution for sÕ and r. The Markov Property is important for reinforcement learning because it allows us to define a function called a policy that only considers the previous state and action when defining the distribution of future actions.

The policy function is used to determine the probability of choosing a particular action at a given state. Formally, The policy function deﬁnes a mapping between (state, action) pairs and their probability [Sutton and Barto, 1998].

ﬁ : [0, 1] S◊Aæ (1.2) ﬁ(a s)=P (a = a s = s) | t | t

The goal of the agent is to find a policy that achieves the maximum expect value of rewards over all states s . Formally, the agent attempts to maximize the present value of returns G at step œS t t given a discount rate ⁄. Aside, ⁄ is known as a hyperparameter in machine learning literature and is programmer specified based on desired behavior. The present value of rewards can be defined as follows. Here, k is used to denote the additional steps after time step t.

Œ k Gt = ⁄ Rt+k+1 (1.3) kÿ=0

We use the discount rate ⁄k to denote the discount applied at a future time step. The choice of ⁄ is problem speciﬁc. Sometimes, it is useful to heavily discount future rewards to make the agent short-sited, and in other situations the opposite is more desirable.

The expected reward of following a policy fi at state s can be defined through the value œS function on fi. [Sutton and Barto, 1998]

v (s)=E [G S = s] (1.4) ﬁ ﬁ t| t

This definition allows us to compare polices. A policy fia is better or equal than a policy fi s ,v (s) v (s). Because we can rank policies, there must be a policy that is b ≈∆ ’ œS fia Ø fib

1.1 What is Reinforcement Learning? 4 better than or equal to all the other policies. This is an optimal policy ﬁ (there may be more ú than one) where all optimal policies share the same optimal value function. The optimal value function is denoted v and is as follows. ú

v (s)=max vﬁ(s) (1.5) ú ﬁ

We can also deﬁne the expected reward of following a policy given an action and state. This is called the Action Value Function and is denoted qﬁ. The policy that generates the maximum reward for all states and actions is the optimal Action Value Function denoted q . ú

qfi(s, a)=Efi[Gt St = s, At = a] | (1.6) q (s, a)=max qfi(s, a) for all s and a (s) ú fi œS œA

We can write q in terms of v by observing q follows the optimal policy by maximizing the ú ú ú expected return of possible actions given the state.

q (s, a)=E[Rt+1 + “v ( t+1) t = s, t = a] (1.7) ú ú S |S A

We can also write v in terms of q . It should be clear to see that the expected reward from the ú ú optimal policy applied across all states is equivalent to the reward given by the action choices by the optimal policy at every state.

v (s)=max qﬁ (s, a) (1.8) ú a ú œA

We can now derive a useful result in reinforcement learning called the Bellman optimality equation. This equation is useful because it expresses the value of the current state in terms of the value of future states. This structure lends itself nicely to algorithm design. This relationship is converted into a classic reinforcement learning solution known as value iteration. The equation states something very intuitive: the optimal value of the current state is the optimal value of the future state from following the best action. Sutton and Barto provide a derivation of the Bellman equation[Sutton and Barto, 1998]. Their derivation is repeated here and follows from the deﬁnitions introduced earlier.

1.1 What is Reinforcement Learning? 5 v (s, a)= max qfi (s, a) ú s A(s) ú œ = max Efi [Gt St = s, At = a] a ú | Œ k = max Efi [ “ Rt+k+1 St = s, At = a] a ú | k=0 ÿ (1.9) Œ k = max Efi [Rt+1 + “ “ Rt+k+2 St = s, At = a] a ú | kÿ=0 = max E[ t+1 + “v ( t+1) t = s, t = a] a R ú S |S A = max p(sÕ,r s, a)[r + “v (sÕ)] s A(s) | ú œ ÿsÕ,r

The last two equations are the most important. We will see that they form the basis to the value iteration approach to reinforcement learning. In addition, we will need the Bellman optimality equations for q . Fortunately, these follow more easily. Note, the ﬁrst equivalence is given by ú substituting (1.6) into (1.7).

q (s, a)=E[Rt+1 + “max q (St+1,aÕ) St = s, At = a] ú aÕ ú | (1.10) = p(sÕ,r s, a)[r + “max q (sÕ,aÕ)] | aÕ ú ÿsÕ,r

Ultimately, it is our goal to find the policy that achieves the optimal value function. There are analytic solutions to finding the optimal policy but these are computationally inefficient and will not work in complex environments. There are a variety of policy function the rely on neural networks that work well in practice and are discussed in section 1.2.

1.2 Reinforcement Learning Policy Methods

There are numerous ways to solve the reinforcement learning problem. The most state of the art methods use neural networks. We will introduce one classic method called value iteration that ﬁnds the optimal policy and then describe state of the art methods.

1.2.1 Value Iteration

The value iteration algorithm works by ﬁrst initializing an array with a location for every possible state of the environment to an arbitrary value. This array is the value function. Until the value

1.2 Reinforcement Learning Policy Methods 6 function changes only a small amount, we update each of the states by the maximum of the current expected reward of a state times the probability of moving to that state. We approximate the value function converging by stopping once changes to the expected rewards becomes sufﬁciently small.

Once we have solved the value function, the optimal policy follows easily. For a given state, we chose the action that generates the largest expected value of returns. This is given by taking the maximum value of the transition probabilities times the respective estimated reward. This time, we select the action that outputs the maximum value instead of using the value itself. It can be shown that the value iteration algorithm converges to an optimal solution [Sutton and Barto, 1998]. The exact value iteration algorithm is given in Algorithm 1.

Algorithm 1 Value Iteration 1: procedure VALUE ITERATION(V (s)=0for all s ,◊) œS 2: =0 3: repeat 4: for each s do œS 5: v = V (s) 6: V (s)=max p(sÕ,r s, a)[r + “V (sÕ)] a sÕ,r | 7: =max(,qv V (s) ) | ≠ | 8: end for 9: until >◊ 10: ﬁ (s)=argmax p(sÕ,r s, a)[r + “V(sÕ)] ú a sÕ,r | 11: Return ﬁ q ú 12: end procedure

The value iteration algorithm is useful approach to reinforcement learning when the reward from moving to a state can be calculated before hand and there is a low number of states. Value iteration in this situation is useful because we are guaranteed to ﬁnd an optimal solution. However, the method becomes infeasible when these criteria are not met due to computational constraints.

1.2.2 Q-Learning

Q-learning is another approach to reinforcement learning where the Q-function (also called action value function given in equation 1.6) is approximated. State of the art approaches use the Q-learning algorithm with neural networks to approximate the Q-function. Q-learning introduces the notion of creating a set of features to represent useful aspects of the possible states. Meaning, an environment consisting of many features is reduced to a few derived features by hand. The goal is to reduce the number of possible states and in this make approximation of the Q-function easier [Watkins and Dayan, 1992].

1.2 Reinforcement Learning Policy Methods 7 In Q-learning, the expected future discounted reward for the state action pair (s, a), where s and a , is called the -value of the pair, (s, a). There are many different ways to œS œA Q Q approximate the the -function. One such way is to approximate the -function as a linear Q Q function of some features [Bertsekas and Tsitsiklis, 1996]. If in a particular state, Â(s, a) are the state action pairs the agent could choose from out of the possible state action pairs s, a , then (s, a)= Â(s, a), where is a randomly initialized unknown vector. This equation encodes Q ◊ + , all the Q-values of the possible state actions pairs. The action selected would be the state action pair with the highest Q-value. Upon interacting with the environment and receiving reward r and progressing to state sÕ, the algorithm updates according to the following deﬁnition, where ⁄ is the learning rate, t is time, and “ is the discount rate [Claus and Boutilier, 1998].

difference =[r + “max (sÕ,aÕ)] (s, a) aÕ Q ≠Q (1.11) = + ⁄ difference Â (s, a) t+1 t ◊ ◊ t

Over time, this will force the weights to converge towards those of the optimal policy for the given feature set. This method works by incorporating the reward at the current time step with the current understanding of the future expected reward to push towards converging at the optimal vector.

While we can generate an optimal policy for a given feature set, we are bound by how well the feature set describes the data. If features are poorly chosen, the solution derived from Q-learning may not be useful. The use of neural networks eliminates the need to derive feature sets.

1.2.3 Deep Q-Learning

The previous description of Q-learning uses a linear function of some features. These features are hand coded in the implementation of the algorithm. It would be more desirable to ﬁnd solutions that incorporate the entire data set into the solution to reduce the error introduced by approximating the data set.

Neural networks have been shown to serve as optimal Q-functions because it is not necessary to hand craft a set a features, and they can approximate complex non-linear functions. Unlike linear-function approximation, neural network approaches do not actually store the -value. Q Instead, the -value is encoded for each state-action pair within the computation of the neural Q network. Every time the agent takes a step, the neural network updates its weights using the optimization methods discussed in the introduction. The disadvantage of neural networks is that training them is a costly computational process.

1.2 Reinforcement Learning Policy Methods 8 This approach is called Deep Q-Learning and used a Deep Q-Network (DQN) [Mnih, Kavukcuoglu, Silver, Rusu, et al., 2015]. DQN uses an artiﬁcial neural network combined with Q-learning to achieve expert human performance at Atari games where the environment of the game is fully observable. By fully observable, we mean that the neural network has access to every aspect of the environment at every step.

While previous -learning methods used hand crafted features or only could achieve intermediate Q level results, DQN represents a critical advancement in reinforcement learning because models can achieve expert-human level performance in complex domains.

The DQN algorithm is applied in this thesis. The full algorithm is given in section 3.2. Note that the neural network is used to approximate the action value function Q. In the original paper, the authors used a convolution neural network but the examples in section 4 use a fully connected neural network.

One notable drawback is that fully connected networks only work on domains where the environment is fully observable [Mnih, Kavukcuoglu, Silver, Rusu, et al., 2015]. In the Atari example, the agent has access to the state of the entire environment. If the environment were a race car game, the method would have access to the location of every car on the track, the velocities of all the cars, and the entire track itself. This is not a realistic approximation of many environments. Consider an agent navigating a maze. The agent may reach a "T" intersection that looks like other "T" intersections. If DQN was applied to this problem, it would interpret each "T" intersection the same. However, the ordering of the intersection is critical to solving the problem, so DQN would fail.

One solution to this problem is to use neural networks that store the history of their previous decisions. In the maze example, if it is known that entering the maze and turning right only leads to dead ends, once the agent navigates to the entry point again, the agent will know to turn left. Such logic can be encoded into a neural network design called a recurrent neural network. While this work does not implement a recurrent neural network approach, it is useful to understand where the fully connected approach may be unsuccessful.

1.2.4 Exploration vs. Exploitation Trade O

Another consideration in reinforcement learning is the trade off between exploration and exploitation Exploration. is the process of testing different states where the reward outcomes are unknown Exploitation. is using knowledge the learner has obtained to generate reward. The exploration-exploitation trade off is a well studied area, and there are numerous sophisticated approaches to the problem [Crepinˇ šek et al., 2013; Tokic, 2010]. The intuition for why properly balancing exploration and exploitation is that while the method may be able to generate more

1.2 Reinforcement Learning Policy Methods 9 reward in the short term, there could be unexplored areas in the state-space that lead to greater rewards later on down the road once found.

One somewhat simple method called ‘-greedy is used in numerous state of the art results [L. Li et al., 2010; Mnih, Kavukcuoglu, Silver, Rusu, et al., 2015; Mnih, Kavukcuoglu, Silver, Graves, Antonoglou, Wierstra, and M. A. Riedmiller, 2013]. The method states: choose the action speciﬁed by the reinforcement learner with a probability of 1 ‘. Otherwise, choose a random ≠ action. Usually, this method involves decaying epsilon by some constant over time so that the algorithm explores more in the beginning and less later.

1.2.5 Transfer Reinforcement Learning

Often there is some relationship between environments for reinforcement learning. This gives rise to transfer reinforcement learning. Instead of learning a policy for an environment completely from scratch, it is possible to transfer over skills learned from one source domain to another target domain. Transfer learning can be useful in the reinforcement learning setting in the following ways [Taylor and Stone, 2009].

• Jump start Training: The speed at which a reinforcement learning method converges to an optimal solution can be improved.

• Improving Asymptotic Performance: The ﬁnal reward threshold generated is improved.

1.2.6 Transfer Learning Procedure

The transfer learning process for reinforcement learning can be characterized in the following way.

1. A source method is learned from a source domain.

2. The source method is mapped to the target domain.

3. The target domain is trained using the source method.

This description is somewhat vague because there are many different approaches to the above methods in the transfer reinforcement learning model. For instance, in some situations, the source method may simply be a selection of potentially useful data points that are added to the target domain [Lazaric et al., 2008]. In some situations, the features learned for the source domain might perfectly match the features in the target domain, so a method learned on the source domain can be directly transferred to the target domain. However sometimes it is not

1.2 Reinforcement Learning Policy Methods 10 Initial Skill pass(Teammate) :- distBetween(a0, Teammate) 15, Ø distBetween(a0, Teammate) 27, Æ angleDefinedBy(Teammate, a0, minAngleDefender(Teammate)) 24, Ø distToRightEdge(Teammate) 10, Æ distBetween(a0, Opponent) 4 Ø Expert Modified Skill pass(Teammate) :- distBetween(a0, goal) - distBetween(Teammate, goal) 1 Ø distBetween(a0, Teammate) 15, Ø distBetween(a0, Teammate) 27, Æ angleDefinedBy(Teammate, a0, minAngleDefender(Teammate)) 24, Ø distToRightEdge(Teammate) 10, Æ distBetween(a0, Opponent) 4 Ø Fig. 1.1: The human advice skill transfer process described by Torrey. The Initial Skill is learned from the source domain. An expert modified the rules by adding a new rule to the top of the list.

immediately clear how the source method might be adapted to the target method. Thus, there exists both algorithmic and human expert approaches to mapping the source to target domain [Taylor and Stone, 2009; Madden and Howley, 2004].

1.2.7 Skill Transfer in Reinforcement Learning with Advice

In some cases, the expert adaption method is the easiest and most effective route to applying a source method to target domain. The expert has some knowledge of how the domains might interact, so they manually adapt the source domain to the target domain. In the process of adapting the source to the target domain, experts can alter the source method to make it better suite the target domain.

In her work, Lisa Torrey provides a method that allows for experts to update skills learned from a source domain in a linear function approximation approach to reinforcement learning [Torrey et al., 2010]. Skills are the criteria under which certain actions can be taken. The criteria are inductive logic programs that look much like a rule list. For instance, she highlights a simple soccer domain. The agent learns the skill “shoot” and deﬁnes the right time to take a shot when the agent is a certain distance from the goal and at an appropriate angle between the goal posts. An expert can update the skills by adding new rules or editing previous ones. An example of the “pass” skill learned for the soccer domain and corresponding update are given in ﬁgure 1.1.

Once the skills from the source domain are learned, they serve as constraints on the linear function approximation algorithm in the target domain. Torrey ﬁnds that the method signiﬁcantly improves convergence on the optimal policy in a variety of test environments.

1.2 Reinforcement Learning Policy Methods 11 1.2.8 Interpretability Concerns of Transfer Reinforcement Learning with Advice

Besides providing increased performance, transfer learning with advice also begins to address another recent concern in machine learning called interpretability. Particularly in the case where neural networks are used as policies as described in section 1.2.3 and also in the function approximation method, it is difﬁcult to determine the agent’s rationale behind its actions.

Understanding the rationale behind the agent’s actions is useful for a number of reasons. If the agent is a self-driving car, it is important to make sure the car is choosing decisions for good reasons. If the reason for the car taking a left turn because the sky is blue, there is likely something wrong with the model. A car that makes decisions for such superﬂuous reasons could likely be dangerous.

There is not one satisfactory answer to interpreting machine learning models. A variety of approaches have been developed to suite the varying needs of practitioners.

The initial consideration in interpretability is whether a simpler model can be used to solve the problem. The use of a simple model is called intrinsic interpretability. For example, it could be possible to determine how a neural network behaves if it only has one hidden layer with three nodes. Other machine learning models such as logistic regression and naive bayes can be considered intrinsically interpretable because of their simplicity.

As is the case with deep neural networks, the model is not intrinsically interpretable. Methods that interpret a trained machine learning model are called post-hoc interpretability. There is a subdivision within post-hoc interpretability between global post-hoc methods and local post-hoc methods.

Global post-hoc methods attempt to provide a simpler way to understand the model in its entirety. A class of global post-hoc solutions is global surrogate methods. Global surrogate methods attempt to train an intrinsically interpretable model to mimic the behavior of the un-interpretable model. The application of inductive logic programs to deﬁne skills is a form of global post-hoc interpretability because the methods are trained on the more opaque linear function approximation model.

Local post-hoc methods determine the rationale for a single decision. One often cited approach is LIME or local interpretable model agnostic explanations [Ribeiro et al., 2016]. LIME offers insight into the features that most influence a decision. This method makes that observation that features which do not affect the output when modified are less likely the reason for the output. LIME first perturbs an input, then collects a data set of predictions made by the model, trains an interpretable model on the collected data, and then interprets that model.

1.2 Reinforcement Learning Policy Methods 12 Statement of The Problem 2

2.1 Explaining Black Box Models

Although there is much work in improving machine learning methods, understanding the underlying reasons behind why machine learning models make certain classifications is less studied. One common method meant to understand obscure models globally is the proxy decision tree. [Craven and Shavlik, 1995] The proxy decision tree method trains a decision tree on the predictions of an artificial neural network. There is evidence that suggests decision trees are easier to understand than artificial neural networks. [Slack et al., 2019] By observing a decision tree, the user can better assess the hypothesis of the model.

Related to decision trees, decision sets can be used successfully as proxy models as well. Decision sets are models in the form of if statements that can be read in any order. Like decision trees, its easy to visualize their classifications. To illustrate how decision sets might be more interpretable than artificial neural networks, a decision set and neural network were trained on the small toy data set pictured in figure 2.1. The data set has two classes: 1 and 0. An 80/20 train test split was applied to the data. Each had a test accuracy of 100%.

The neural network used had 2 input nodes, 3 hidden nodes, and a single output node that predicted the binary class. The 3 hidden nodes used the ReLu activation function and the output

Fig. 2.1: The Toy Data Set

13 Input Layer

in1 = x, in2 =y

Hidden Layer

h = max(0,in 0.77 + in 0.25 + 0.34) 1 1 ◊≠ 2 ◊ h = max(0,in 0.85 + in 0.10 + 0.28) 2 1 ◊≠ 2 ◊≠ h = max(0,in 0.66 + in 0.70 + 0.14) 3 1 ◊ 2 ◊≠ Final Layer

1 Result = 1 if 1+e (h 0.34+h 1.02+h 1.2+0.14) 0.5 else 0 ≠ 1◊ 2◊ 3◊≠ Ø

Tab. 2.1: Exact computation of neural network for the toy data set

node used the Sigmoid activation function. The neural network was trained using the Keras software package.[Chollet et al., 2015] The neural network produced is given in table 2.1.

To produce a decision set, the CN2 unordered Rule Induction in the Orange data mining kit was used [Demšar et al., 2013]. The decision set the algorithm produced is given in ﬁgure 2.2.

While it is possible to tell after some computation that the neural network in table 2.1 correctly classifies the data given in figure 2.1, it is much easier to determine the the decision set in figure 2.3 classifies the data correctly.

2.2 Problem Statement: Expert-Assisted (and Interpretable) Transfer Learning

One problem in transfer reinforcement learning is how do we explain the knowledge transferred from the source to the target task? Also, can we use the advice of human experts to assist in the transfer process to make the source method better suited for the target domain? As described in section 1.2.6, the transferred policy is adapted to the target task. If this policy is modeled by

Fig. 2.2: The decision set for the toy data set

2.2 Problem Statement: Expert-Assisted (and Interpretable) Transfer Learning 14 a neural network, it is unclear what knowledge is transferred between domains. As shown in section 2.1, it is difﬁcult to understand the behavior of a neural network.

Further, Torry’s method from section 1.2.7 is not well suited to a deep learning setting. It could also be better for the understanding of the expert to provide a global explanation of model behavior instead of subdividing by action. By providing domain experts a way to understand and modify the policy transferred from source to target domain in a deep learning setting, we may be able to address a range of relevant problems.

For example, this method may be useful in computational chemistry. Deep reinforcement learning has shown promising results in the process of discovering active chemical compounds from a data set of compounds [Zhou et al., 2017]. In chemical reaction discovery systems like that of the Dark Reactions Project chemical compound data sets are changed somewhat frequently [Raccuglia et al., 2016]. In order to generate useful results of a reinforcement learning approach to a discovery in computational chemistry, it is important to generate well formed reactions as rapidly as possible. Transfer learning can help in this instance. Incorporating the expertise of chemists could both help exploration and understanding of the chemists.

2.2 Problem Statement: Expert-Assisted (and Interpretable) Transfer Learning 15 Methods 3

In this section, we propose a method that uses a decision set as an interpretable transfer model for deep reinforcement learning. Experts can audit and edit the decision set before applying it to the target domain. The method focuses on applications where the source and target domain use variants of deep reinforcement learning.

3.1 Model Road Map

First, we train a reinforcement learning method on the target domain. This is assumed to be a neural network. A record of the previous k epochs worth of (state,action) pairs is stored. Here, k is a parameter specified by the programmer. Next, we train a decision list off the k epochs worth of data. Then, an expert edits the decision set to better suite the target domain. This includes adding or removing rules to aide the decision set but also fitting the source model to a new feature space if required by the target domain. The updated decision set is added as a node in the output layer of the neural network in the target domain. An outline of the method is given in figure 3.1.

3.2 Deep Q-Learning With Experience Replay

We train a neural network on both the source and target domain using a variant of Q-Learning called Deep Q-Learning with Experience Replay. Going forward, a neural network that imple- ments deep Q-learing with experience replay is referred to as a DQN. We also use the e-greedy exploration strategy. In experience replay, instead of updating the Q-value of state-action pairs as they occur, the pairs are stored then randomly sampled later on in mini-batches. This variant of

Fig. 3.1: Outline of Process

16 Q-learning has shown to be successful in a variety of settings. The algorithm is adapted from the original paper in algorithm 2.2 [Mnih, Kavukcuoglu, Silver, Rusu, et al., 2015]. Their algorithm uses image data that is pre-processed using a function represented as „. This is omitted from the following algorithm because it is unnecessary for the current application. In addition, ◊ refers to the current weights on the neural network. Thus, the expression argmaxa Q(st,a; ◊) indicates: evaluate the state st on network Q with weights ◊ and choose the action a corresponding to the maximum value produced.

Algorithm 2 Deep Q-Learning with Experience Replay 1: procedure DQL-ER 2: D = replay memory 3: Q = action-value function with random weight 4: for episode = 1 to M do 5: Initialize sequence s1 = {starting state} 6: for t = 1 to T do 7: at = random action with probability ‘ else argmaxa Q(st,a; ◊) 8: Execute action at and retrieve reward rt and new state st+1 9: Store transition (st,at,rt,st+1) in D 10: Sample random minibatch of transitions (sj,aj,rj,sj+1) from D 11: if episode terminates at step j +1then 12: yj = rj 13: else 14: yj = rj + “maxaÕ Q(sj+1,aÕ; ◊) 15: end if 16: Perform a gradient descent step on (y Q( ,a ; ◊))2 w.r.t ◊ j ≠ j j 17: end for 18: end for 19: end procedure

3.3 Decision Lists as an Interpretable Transfer Learning Source

3.3.1 CN2 Rule Induction

To generate interpretable and editable decision sets, the CN2 rule induction algorithm is used [Clark and Boswell, 1991]. A software package called Orange provides an implementation of the CN2 unordered rule induction algorithm [Demšar et al., 2013]. Decision sets are convenient when an expert needs to edit them because each rule is isolated. So, if an expert wants to remove one rule, it is easy to see the effects it has on the other rules. Further, the Orange implementation also accepts arguments that specify the minimum number of examples a rule must describe to be speciﬁed in the rule list. Further, it also allows the user to specify the max number of parameters in a rule. Both of these arguments make it easier to generate rules that are interpretable.

3.3 Decision Lists as an Interpretable Transfer Learning Source 17 Fig. 3.2: Rule Selection Interface

3.3.2 Expert modiﬁed Rules

The CN2 rule algorithm produces rules of the style in figure 2.2. Once a decision set is produced from a source method, the expert can freely remove rules from the decision set or add new rules through a selection interface. For instance, the interface for editing or removing the first rule from the decision set in figure 2.2 is given in figure 3.2.

The problem is further complicated however when the source domain features do not match up perfectly with the target domain. For instance, there could be a related problem where the source domain has two features and the target domain has three features. The problem, known as domain adaption, is well studied in transfer learning [Csurka, 2017; Chu and Wang, 2018]. For the purposes of this thesis, the expert is assumed to adapt the source method to the target domain manually using their domain speciﬁc knowledge to best solve the problem. In this case, there is still further work needed to adapt the decision set than allowed by the interface in ﬁgure 3.2.

3.3.3 Adapting DQN to Incorporate Transferred Knowledge

Once a decision set is trained from the source method and modified by an expert, it can be included in the target DQN training process by including the transferred decision set as a node in the final layer of the target neural network. For instance, if the target domain has four possible actions, the transferred decision set would be the fifth action. Each node in the final layer represents an action. Thus, the addition of a node for the decision set would represent its addition as another possible action. The neural network training process proceeds as normal. When the decision set node is selected, the decision set computes an action based on the current state. The agent makes that action. The expected reward of choosing the decision set is then updated by the neural network based on how good the action determined by the decision set is.

3.3 Decision Lists as an Interpretable Transfer Learning Source 18 Incorporating Domain Knowledge in 4 Transfer Learning Process

This section provides applications of the proposed method. First, a domain is proposed where the target domain is the same as the source domain and no expert advice is applied to show the usefulness of the decision set for transfer learning. Second, the example is extended to a scenario where the target domain differs from the source domain. Finally, a third scenario is proposed where the state and action spaces do not line up exactly between the source and target domain.

4.1 Circle to Circle Environment

In the first toy environment, the agent must figure out how to navigate into the unit circle from a location on the x, y-plane. A visualization of the environment is given in figure table 4.1. The 1 agent receives reward +1 reward for being inside the circle and (1+d)2 reward (where d is the distance to the edge of the circle) for being outside the circle. The agents starts at a random location in the 4 by 4 square centered at the origin. The agent exists in the environment for a 200 step episode. Meaning, it takes 200 actions before it terminates. The agent can either move up, down, left, or right 0.25 on every step.

Hyperparameters Hidden Layers 2 Hidden Layers, 10 Nodes ‘ 0.9 – 0.001 “ 0.9 Memory Size (Experience Replay) 2,000 Tab. 4.1: DQN hyperparameter conﬁguration across all tasks.

The DQN conﬁguration given in table 4.1 was used. The DQN converged on an optimal solution at 75 epochs. So, 200 epochs of test data once the DQN reached 150 epochs (meaning k is set to 200) was used to train the decision set using the CN2 algorithm with the minimum number of examples a rule can cover set to 250 and the max rule length set to 4.

19 (a) The circle environment. The (b) The ellipse environment. The agent receives agent receives +1 reward reward in the same way as the circle, except 1 and (1+d)2 , reward where d to the edge of the ellipse. is the distance to the edge of the circle for being outside.

Fig. 4.1: The ellipse and circle source and target domains respectively. Note that in the ﬁrst task both the source and target domain are the circle.

For this initial example, the goal is to verify the usefulness of the decision set in transfer learning, so no expert-advice is applied. A DQN with the conﬁguration in table 4.1 with an additional node in the ﬁnal layer for the decision set was trained on the circle environment.

The decision set modified DQN solution started to converge on an optimal solution at around 25 epochs. The unmodified DQN starts to produce optimal solutions around 75 epochs but there is considerable noise in the data. The results are shown in figure 4.2. In addition, the decision set that was transferred to the target DQN is shown in figure 4.7.

Besides converging to an optimal solution quicker, the decision set modified DQN is much less noisy than the unmodified DQN. There are many times in the unmodified DQN that the rewards drastically fall off. To see if the noise was due to – being too large in the gradient descent step, – was reduced by 1, 2, and 3 orders of magnitude. However, the solution failed to converge in a timely manner for all reductions. Additionally, the simulations were let to run for 800 epochs because it could be the case that the source DQN needed more time to fully converge. While there were less sharp declines in the total reward over time, there was still the same level of noise in the reward signal. The decision set modified DQN exhibits much more consistent levels of reward irrespective of the – and epochs trained.

As seen in ﬁgure 4.3, the DQN relies heavily on the decision set for the initial part of training. This makes sense because the DQN has no knowledge of the appropriate actions to take. However, as training progresses, reliance on the transferred decision set drastically decreases. After episode 25, the decision set is used on average only 0.2% of the time, with many runs not using it at all.

Further, table 4.2 displays the average rewards for each model for 300 epochs once the model has been "fully trained." For the source DQN, this was taken as after 125 epochs (to be generous

4.1 Circle to Circle Environment 20 Unmodiﬁed DQN Decision set Decision set modiﬁed DQN 177.57 159.27 187.88 Tab. 4.2: Average Rewards For Fully Trained Model

Fig. 4.2: The total reward over each training epoch Fig. 4.3: The fraction of the time where the for both the unmodified DQN and the transferred decision set is used per epoch. transferred decision set DQN. regarding the drastic dip in reward around 100 epochs), 25 epochs for the decision set modified DQN, and immediately for the decision set on its own. The decision set modified DQN produces the greatest reward on average by 10 points.

Thus in this example, the DQN with decision set is able to produce more consistent reward signals that are on average better than both the original DQN and the learned decision set. This suggests that the transfer step of incorporating the decision set in the training process of the neural network permanently improves the ability of the DQN in this example by (1) making the results more consistent (2) improving the asymptotic reward threshold and (3) boosting pre-training.

4.2 Circle to Ellipse Environment

In this environment, the previous example is extended to transfer from the circle to an ellipse. The ellipse used is pictured in ﬁgure 4.1. The ellipse environment is very similar to the circle environment. The agent gets +1 reward for being inside the ellipse and 1/(1 + d)2 reward, where d is the distance from a point to the closest point on the ellipse, when the agent is outside the ellipse.

The DQN used to train the source and target domains is the same as given by table 4.1. For the expert-modiﬁed case, the x-conditions in ﬁgure 4.7 were incremented by +2 if they were greater

4.2 Circle to Ellipse Environment 21 Fig. 4.4: Circle to ellipse transfer reward over training epoch. The reward is given as the running average over the past ﬁve epochs. or equal to 0 and -2 if they were less than zero in order to reﬂect the expansion of the circle along the x-axis.

The results when comparing the reward signals across the unmodified DQN, the decision set modified DQN, and the expert-modified decision set DQN all on the target ellipse environment are shown in figure 4.4. Because the noise in each reward signal makes differentiating between the three signals confusing, the running average of the previous five results is taken. Both the expert-modified decision set DQN and decision set modified DQN outperform the baseline substantially. Also, the expert-modified decision set DQN noticeably outperforms the decision set modified DQN. Thus, the expert advice seems useful in this example.

4.3 Mountain Car Environment

In the next environment, the goal is to push a car up a mountain. However, the car cannot drive up the mountain immediately. It must learn to ﬁrst accelerate backwards to gain momentum before it can reach the top of the mountain. These scenarios are illustrated in ﬁgure 4.5. The difference between the two environments is that in the two-dimension version, the car can only change its velocity left or right on one axis. In the three-dimension version, it can change its velocity west, east, south, or north. Further, in the two-dimension version there are two state variables: the x-position and the x-velocity. In the three-dimension version, there are

4.3 Mountain Car Environment 22 (a) The two-dimension mountain car (b) The three-dimension mountain car problem. problem.

Fig. 4.5: The mountain car source and target domains. Both of these images originate from [Taylor, Jong, et al., 2008]. four state variables: x-position, x-velocity, y-position, and y-velocity. Transferring to a domain with a different state-space complicates the problem because the decision set learned from the two-dimension version does not perfectly align with the three-dimension version.

The implementation of the two-dimension mountain car environment is from a popular reinforcement learning library called OpenAI gym [Brockman et al., 2016]. The x-axis is restricted to [ 1.2, 0.6] and the velocity is restricted to [ 0.07, 0.07]. The agent starts from the x-location ≠ ≠ fi . The agent can choose actions from left, nothing, right which change the velocity by ≠ 3 { } 0.0015, 0, 0.0015 respectively. To simulate gravity on the hill, 0.025(cos(3x)) is added to {≠ } ≠ velocity on every time step. The goal is to reach x 0.5. Reward is 1 for every time step Ø ≠ the agent does not reach the goal. Unlike the base environment, if x 0.3, 10 x is added to Ø ◊ the reward and +100 reward is given on completion. This modification is added because DQN struggles to converge efficiently on the normal environment due to the sparsity in the reward signal.

The DQN used is the same as in 4.1 and was trained on the two-dimension version for 400 epochs. The states and actions in the last 100 epochs were recorded. A decision set with the minimum number of examples a rule can cover set to 250 and max rule length of 4 was trained on the data and is given in ﬁgure 4.8. Note that in some cases, the Orange implementation of the CN2 algorithm gives repetitive criteria. For instance, one rule indicates it should be followed if position -0.575 and if position -0.69. While there may be some issues with repetitiveness Æ Æ with the implementation, the rules seem logical for the most part.

Next, the two-dimension mountain car was extended by the addition of two more state variables y-position and y-velocity. Both the x and y ranges were set to [ 1.2, 0.6]. The agent starts from the ≠ location ( fi , fi ) Now, the agent selects from five actions: nothing, west, east, south, north . ≠ 3 ≠ 3 { } The south and north actions modify the y-velocity by 0.0015 and +0.0015 respectively and the ≠ west and east actions modify the x-velocity by 0.0012 and +0.0012 respectively. Gravity adds ≠ 0.025(cos(3x)) to the x-velocity and 0.025(cos(3y)) to the y-velocity on each time step. ≠ ≠

4.3 Mountain Car Environment 23 The agent accomplishes its task when both x, y 0.5 and recieved +100 reward for ﬁnishing. It Ø receives -1 reward for every time step. If both x, y 0.3, then it receives an additional x+y 10 Ø 2 ◊ reward.

Adapting the decision set to the target environment is difficult. The state variables do not line up directly and there are two more possible actions. The problem is further complicated by the fact that the agent can only accelerate in a direction along either the x-axis or the y-axis. Thus, it is not possible to simply apply the decision set along each axis. Further, because applying the decision set to work in the target domain requires clever human intervention, it is difficult to separate the expert modified case from the solely decision set transferred version. Thus, for this example, there is only an expert modified decision set version.

While multiple strategies were tried in adapting the decision set to the target domain, the one that worked best is given in algorithm 3. This approach states that if one of the predicted actions indicates “do nothing,” follow the decision set applied to other axis. Otherwise, ﬁrst dependably accelerate in the negative direction along the x-axis if the decision set dictates left along the x state variables. Else, accelerate in the negative direction along the y-axis. The idea here is that the decision set should provide the knowledge to accelerate backwards unless there is clearly another choice (e.g. no acceleration along one axis and acceleration along another).

Algorithm 3 Procedure for adapting the two-dimension mountain car decision set to three- dimension mountain car. Note, “negative” means to accelerate in the negative direction along the speciﬁed axis and “positive” means to accelerate in the positive direction along the speciﬁed axis. 1: procedure DECISION SET ADAPTION TO 3D ENVIRONMENT 2: xaction = DecisionSet.predict(xstate, xvelocity) 3: yaction = DecisionSet.predict(ystate,yvelocity) 4: if xaction = do nothing and yaction = do nothing then 5: action = Do Nothing 6: else if xaction = negative and yaction = do nothing then 7: action = negative along x-axis 8: else if xaction = positive and yaction = Do Nothing then 9: action = positive along x-axis 10: else if xaction = do nothing and yaction = negative then 11: action = negative along y-axis 12: else if xaction = do nothing and yaction = negative then 13: action = negative along y-axis 14: else if xaction = negative then 15: action = negative along x-axis 16: else 17: action = negative along y-axis 18: end if 19: end procedure

The results of the experiment are presented in figure 4.6. Upon evaluation however, there is not clearly a difference between the expert-modified decision set and a the unmodified DQN

4.3 Mountain Car Environment 24 Fig. 4.6: Reward over training epochs for transfer of 2D Mountain Car to 3D Mountain Car. The reward is expressed as the running average over the past 10 epochs. trained on the three-dimension mountain car environment. While the expert-modified approach starts to converge quicker than the unmodified DQN, it eventually falls off and the unmodified DQN reaches a the maximum reward quicker than the expert modified DQN. This result may be explained by inability to find the optimal adaption strategy or failures due to DQN in this environment. DQN is known to struggle with the mountain car environment and successful implementations of transfer learning in the two-dimension to three-dimension mountain car task typically use other variants of reinforcement learning [Taylor, Jong, et al., 2008]. While the circle to ellipse application proved successful, the mountain car application struggled to perform well. These difficulties in large part are caused by issues when adapting the decision set to the target environment.

4.3 Mountain Car Environment 25 Fig. 4.7: The decision set transferred to the target domain in the circle to circle and circle to ellipse examples. Action 0: +.25 in y, Action 1: -.25 in y, Action 2: -.25 in x, Action 3: +.25 in x

Fig. 4.8: The decision set learned on the two-dimension mountain car environment. Action 0: accelerate -0.0012 in the x-axis, Action 1: Do nothing, Action 2: accelerate +0.0012 is the x-axis.

4.3 Mountain Car Environment 26 Conclusion and Future Work 5

This work presents an novel method to apply expert-advice in transfer dep reinforcement learning. It finds that the method is successful is some situations, particularly those where the state-space between source and target domain aligns well, such as circle to ellipse. It also finds that the method struggled in situations were the state-space did not align, such as mountain car. Further, it is difficult to separate whether this was the case because of the core expert-assisted transfer method or because of issues with the use of DQN to solve the environment. Thus, it would be interesting in the future to apply the expert-assisted method to other variants of deep reinforcement learning such as policy gradient methods. Such work could help uncover if the transfer learning method or the core reinforcement learning algorithm was at fault in the mountain car environment. Further, it would be useful to test how successful the method is on a real world applications such as chemistry reaction exploration. Lastly, it would be interesting to examine other methods in transfer learning that perform adaption of the source policy to target domains algorithmically and apply them to the expert-modified decision set case.

27 Bibliography

Abadi, Martin et al. (2016). “TensorFlow: A system for large-scale machine learning”. In: 12th USENIX Symposium on Operating Systems Design and Implementation (OSDI 16), pp. 265–283.

Abbeel, Pieter and Andrew Y. Ng (2004). “Apprenticeship Learning via Inverse Reinforcement Learning”. In: Proceedings of the Twenty-ﬁrst International Conference on Machine Learning. ICML ’04. Banff, Alberta, Canada: ACM, pp. 1– (cit. on p. 3).

Adler, Philip et al. (2016). Auditing Black-box Models for Indirect Inﬂuence. eprint: arXiv:1602.07043. Agrawal, Shipra and Navin Goyal (2017). “Near-Optimal Regret Bounds for Thompson Sampling”. In: J. ACM 64.5, 30:1–30:24.

“Artiﬁcial Neural Network” (2018). In: Wikipedia. eprint: https://en.wikipedia.org/wiki/Artificial_ neural_network. Bakker, Bram (2001). “Reinforcement Learning with Long Short-term Memory”. In: Proceedings of the 14th International Conference on Neural Information Processing Systems: Natural and Synthetic. NIPS’01. Vancouver, British Columbia, Canada: MIT Press, pp. 1475–1482.

Bertsekas, Dimitri P. and John N. Tsitsiklis (1996). Neuro-Dynamic Programming. 1st. Athena Scientiﬁc (cit. on p. 8).

Breiman, Leo (2001a). “Random Forests”. In: Mach. Learn. 45.1, pp. 5–32.

– (2001b). “Random forests”. In: Machine learning 45.1, pp. 5–32.

Brockman, Greg et al. (2016). OpenAI Gym. eprint: arXiv:1606.01540 (cit. on p. 23). Brown, Alexander and Marek Petrik (2018). “Interpretable Reinforcement Learning with Ensemble Methods”. In: CoRR abs/1809.06995.

Choi, Sungwoon et al. (2018). Reinforcement Learning based Recommender System using Biclustering Technique. eprint: arXiv:1801.05532.

Chollet, François et al. (2015). Keras. https://keras.io (cit. on p. 14). Christopher J.C.H. Watkins, Peter Dayan (1992). Q-Learning.

Chu, Chenhui and Rui Wang (2018). “A Survey of Domain Adaptation for Neural Machine Translation”. In: CoRR abs/1806.00258. arXiv: 1806.00258 (cit. on p. 18). Clark, Peter and Robin Boswell (1991). “Rule induction with CN2: Some recent improvements”. In: Machine Learning — EWSL-91. Ed. by Yves Kodratoff. Berlin, Heidelberg: Springer Berlin Heidelberg, pp. 151–163 (cit. on p. 17).

28 Claus, Caroline and Craig Boutilier (1998). The Dynamics of Reinforcement Learning in Cooperative Multiagent Systems. AAAI (cit. on p. 8).

Craven, Mark W. and Jude W. Shavlik (1995). “Extracting Tree-structured Representations of Trained Networks”. In: Proceedings of the 8th International Conference on Neural Information Processing Systems. NIPS’95. Denver, Colorado: MIT Press, pp. 24–30 (cit. on p. 13).

Crepinˇ šek, Matej, Shih-Hsi Liu, and Marjan Mernik (2013). “Exploration and Exploitation in Evolutionary Algorithms: A Survey”. In: ACM Comput. Surv. 45.3, 35:1–35:33 (cit. on p. 9).

Csurka, Gabriela (2017). “A Comprehensive Survey on Domain Adaptation for Visual Applications”. In: Domain Adaptation in Computer Vision Applications. Ed. by Gabriela Csurka. Cham: Springer International Publishing, pp. 1–35 (cit. on p. 18).

Cun, Y. Le et al. (1990). “Advances in Neural Information Processing Systems 2”. In: ed. by David S. Touretzky. San Francisco, CA, USA: Morgan Kaufmann Publishers Inc. Chap. Handwritten Digit Recognition with a Back-propagation Network, pp. 396–404.

Demšar, Janez et al. (2013). “Orange: Data Mining Toolbox in Python”. In: Journal of Machine Learning Research 14, pp. 2349–2353 (cit. on pp. 14, 17).

Doya, Kenji (1995). “Temporal Difference Learning in Continuous Time and Space”. In: Proceedings of the 8th International Conference on Neural Information Processing Systems. NIPS’95. Denver, Colorado: MIT Press, pp. 1073–1079.

Duan, Yan et al. (2016). RL2: Fast Reinforcement Learning via Slow Reinforcement Learning. eprint: arXiv:1611.02779. Dulac-Arnold, Gabriel et al. (2015). Deep Reinforcement Learning in Large Discrete Action Spaces. eprint: arXiv:1512.07679.

Elzayn, Hadi et al. (2018). Fair Algorithms for Learning in Allocation Problems. eprint: arXiv:1808.10549. Fisher, A., C. Rudin, and F. Dominici (2018). “All Models are Wrong but many are Useful: Variable Importance for Black-Box, Proprietary, or Misspeciﬁed Prediction Models, using Model Class Reliance”. In: ArXiv e-prints. arXiv: 1801.01489 [stat.ME]. Guo, Xiaoxiao, Satinder Singh, Honglak Lee, Richard L Lewis, and Xiaoshi Wang (2014). “Deep Learning for Real-Time Atari Game Play Using Ofﬂine Monte-Carlo Tree Search Planning”. In: Advances in Neural Information Processing Systems 27. Ed. by Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q. Weinberger. Curran Associates, Inc., pp. 3338–3346.

Hausknecht, Matthew J. and Peter Stone (2015). “Deep Recurrent Q-Learning for Partially Observable MDPs”. In: CoRR abs/1507.06527. arXiv: 1507.06527. Hein, Daniel, Alexander Hentschel, Thomas Runkler, and Steffen Udluft (2016). “Particle Swarm Op- timization for Generating Interpretable Fuzzy Reinforcement Learning Policies”. In: eprint: arXiv: 1610.05984. Hein, Daniel, Steffen Udluft, and Thomas A. Runkler (2017). “Interpretable Policies for Reinforcement Learning by Genetic Programming”. In: CoRR abs/1712.04170. arXiv: 1712.04170. Henson, Alon B., Piotr S. Gromski, and Leroy Cronin (2018). “Designing Algorithms To Aid Discovery by Chemical Robots”. In: ACS Central Science 4.7. PMID: 30062108, pp. 793–804. eprint: https: //doi.org/10.1021/acscentsci.8b00176. Joseph, Matthew, Michael Kearns, Jamie Morgenstern, and Aaron Roth (2016). Fairness in Learning: Classic and Contextual Bandits. eprint: arXiv:1605.07139.

Bibliography 29 Koh, Pang Wei and Percy Liang (2017). Understanding Black-box Predictions via Inﬂuence Functions. eprint: arXiv:1703.04730. Lazaric, Alessandro, Marcello Restelli, and Andrea Bonarini (2008). “Transfer of Samples in Batch Reinforcement Learning”. In: Proceedings of the 25th International Conference on Machine Learning. ICML ’08. Helsinki, Finland: ACM, pp. 544–551 (cit. on p. 10).

Le, Q. V. (2013). “Building high-level features using large scale unsupervised learning”. In: 2013 IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 8595–8598 (cit. on p. 2).

LeCun, Yann and Corinna Cortes (2010). “MNIST handwritten digit database”. In:

Levine, Sergey, Zoran Popovic, and Vladlen Koltun (2010). Feature Construction for Inverse Reinforcement Learning.

Li, Lihong, Wei Chu, John Langford, and Robert E. Schapire (2010). “A Contextual-Bandit Approach to Personalized News Article Recommendation”. In: CoRR abs/1003.0146. arXiv: 1003.0146 (cit. on p. 10).

Li, Yuxi (2017). “Deep Reinforcement Learning: An Overview”. In: CoRR abs/1701.07274. arXiv: 1701. 07274.

Lillicrap, Timothy P. et al. (2015). Continuous control with deep reinforcement learning. eprint: arXiv: 1509.02971.

Lipton, Zachary C. (2016). The Mythos of Model Interpretability. eprint: arXiv:1606.03490. Liu, Guiliang, Oliver Schulte, Wang Zhu, and Qingcan Li (2018). “Toward Interpretable Deep Reinforce- ment Learning with Linear Model U-Trees”. In: CoRR abs/1807.05887.

Madden, Michael G. and Tom Howley (2004). “Transfer of Experience Between Reinforcement Learning Environments with Progressive Difﬁculty”. In: Artiﬁcial Intelligence Review 21.3, pp. 375–398 (cit. on p. 11).

Markov, A.A. (1954). Theory of Algorithms. TT 60-51085. Academy of Sciences of the USSR (cit. on p. 3).

Mnih, Volodymyr, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller (2013). Playing Atari with Deep Reinforcement Learning. eprint: arXiv:1312.5602. Mnih, Volodymyr, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wier- stra, and Martin A. Riedmiller (2013). “Playing Atari with Deep Reinforcement Learning”. In: CoRR abs/1312.5602. arXiv: 1312.5602 (cit. on p. 10). Mnih, Volodymyr, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, et al. (2015). “Human-level control through deep reinforcement learning”. In: Nature 518.7540, pp. 529–533 (cit. on pp. 9, 10, 17).

Molnar, Christoph (2018). Interpretable Machine Learning. eprint: https://github.com/christophM/ interpretable-ml-book. Ng, Andrew Y. and Stuart Russel (2000). Algorithms for Inverse Reinforcement Learning.

Noothigattu, Ritesh et al. (2018). “Interpretable Multi-Objective Reinforcement Learning through Policy Orchestration”. In: AAAI abs/1809.08343.

Phillips, Richard L., Kyu Hyun Chang, and Sorelle A. Friedler (2017). Interpretable Active Learning. eprint: arXiv:1708.00049. Popova, Mariya, Olexandr Isayev, and Alexander Tropsha (2018). “Deep reinforcement learning for de novo drug design”. In: Science Advances 4.7. eprint: http://advances.sciencemag.org/content/4/ 7/eaap7885.full.pdf.

Bibliography 30 Raccuglia, Paul et al. (2016). Machine-learning-assisted materials discovery using failed experiments. Nature (cit. on p. 15).

Reinforcement Learning as Classiﬁcation: Leveraging Modern Classiﬁers (2003).

Ribeiro, Marco Tulio, Sameer Singh, and Carlos Guestrin (2016). “"Why Should I Trust You?": Explaining the Predictions of Any Classiﬁer”. In: Proceedings of the 22Nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16. San Francisco, California, USA: ACM, pp. 1135–1144 (cit. on p. 12).

Riedmiller, Martin and Heinrich Braun (1993). “A Direct Adaptive Method for Faster Backpropagation Learning: The RPROP Algorithm”. In: IEEE INTERNATIONAL CONFERENCE ON NEURAL NETWORKS. Pp. 586–591.

Sak, Hasim, Andrew Senior, and Francoise Beaufays (2014). Long Short-Term Memory Recurrent Neural Network Architectures for Large Scale Acoustic Modeling. INTERSPEECH.

Selbst, Andrew D and Julia Powles (2017). “Meaningful information and the right to explanation”. In: International Data Privacy Law 7.4, pp. 233–242. eprint: /oup/backfile/content_public/journal/ idpl/7/4/10.1093_idpl_ipx022/1/ipx022.pdf. Shannon, C. E. (1988). “Computer Chess Compendium”. In: ed. by David Levy. Berlin, Heidelberg: Springer-Verlag. Chap. Programming a Computer for Playing Chess, pp. 2–13.

Slack, Dylan, Sorelle A. Friedler, Chitradeep Dutta Roy, and Carlos Scheidegger (2019). Assessing the Local Interpretability of Machine Learning Models. arXiv: 1902.03501 [cs.LG] (cit. on p. 13). Sutton, Richard S. and Andrew G. Barto (1998). Introduction to Reinforcement Learning. 1st. Cambridge, MA, USA: MIT Press (cit. on pp. 2–5, 7).

Taylor, Matthew E., Nicholas K. Jong, and Peter Stone (2008). “Transferring Instances for Model-Based Reinforcement Learning”. In: Machine Learning and Knowledge Discovery in Databases. Ed. by Walter Daelemans, Bart Goethals, and Katharina Morik. Berlin, Heidelberg: Springer Berlin Heidelberg, pp. 488–505 (cit. on pp. 23, 25).

Taylor, Matthew E. and Peter Stone (2009). “Transfer Learning for Reinforcement Learning Domains: A Survey”. In: J. Mach. Learn. Res. 10, pp. 1633–1685 (cit. on pp. 10, 11).

Tesauro, Gerald (1994). “TD-Gammon, a Self-teaching Backgammon Program, Achieves Master-level Play”. In: Neural Comput. 6.2, pp. 215–219.

Thorndike, E (1898). Animal intelligence: An experimental study of the associative processes in animals.

Tokic, Michel (2010). “Adaptive ‘-Greedy Exploration in Reinforcement Learning Based on Value Dif- ferences”. In: KI 2010: Advances in Artiﬁcial Intelligence. Ed. by Rüdiger Dillmann, Jürgen Beyerer, Uwe D. Hanebeck, and Tanja Schultz. Berlin, Heidelberg: Springer Berlin Heidelberg, pp. 203–210 (cit. on p. 9).

Torrey, Lisa, Jude Shavlik, Trevor Walker, and Richard Maclin (2010). “Transfer Learning via Advice Taking”. In: Advances in Machine Learning I: Dedicated to the Memory of Professor Ryszard S.Michalski. Ed. by Jacek Koronacki, Zbigniew W. Ra´s, Sławomir T. Wierzchon,´ and Janusz Kacprzyk. Berlin, Heidelberg: Springer Berlin Heidelberg, pp. 147–170 (cit. on p. 11).

Verma, Abhinav, Vijayaraghavan Murali, Rishabh Singh, Pushmeet Kohli, and Swarat Chaudhuri (2018). “Programmatically Interpretable Reinforcement Learning”. In: ICML.

Bibliography 31 Wachter, Sandra, Brent Mittelstadt, and Luciano Floridi (2017). “Why a right to explanation of automated decision-making does not exist in the general data protection regulation”. In: International Data Privacy Law 7.2, pp. 76–99.

Watkins, Christopher J. C. H. and Peter Dayan (1992). “Q-learning”. In: Machine Learning 8.3, pp. 279–292 (cit. on p. 7).

Werbos, P. J. (1990). “Backpropagation through time: what it does and how to do it”. In: Proceedings of the IEEE 78.10, pp. 1550–1560.

Widrow, B. and M. A. Lehr (1990). “30 years of adaptive neural networks: perceptron, Madaline, and backpropagation”. In: Proceedings of the IEEE 78.9, pp. 1415–1442.

Zhao, Xiangyu et al. (2017). Deep Reinforcement Learning for List-wise Recommendations. eprint: arXiv: 1801.00209. Zhou, Zhenpeng, Xiaocheng Li, and Richard N. Zare (2017). “Optimizing Chemical Reactions with Deep Reinforcement Learning”. In: ACS Central Science 3.12. PMID: 29296675, pp. 1337–1344. eprint: https://doi.org/10.1021/acscentsci.7b00492 (cit. on p. 15).

Bibliography 32