Explorations in E Cient Reinforcement Learning

Home , Action selection

Explorations in Ecient Reinforcement Learning

ACADEMISCH PROEFSCHRIFT

Ter verkrijging van de graad van do ctor

aan de Universiteit van Amsterdam

op gezag van Rector Magnicus

Prof dr JJM Franse

ten overstaan van een do or het

college voor promoties ingestelde commissie

in het op enbaar te verdedigen

in de Aula der Universiteit

op woensdag februari te uur

do or

Marco Wiering

geb oren te Schagen

Promotiecommissie

Promotor Prof dr ir FCA Gro en

CoPromotor Dr Hab JH Schmidhuber

Overige leden Prof dr P van Emde Boas Universiteit van Amsterdam

Dr ir BJA Krose Universiteit van Amsterdam

Prof dr PMB Vitanyi Universiteit van Amsterdam

Dr M Dorigo Universite Libre de Bruxelles

Prof Dr J van den Herik Universiteit van Maastricht

Faculteit der Wiskunde Informatica Natuurkunde en Sterrenkunde

Universiteit van Amsterdam

Cover design and photography by Marco Wiering

The work describ ed in this thesis was p erformed at IDSIA Istituto Dalle Molle di Studi

sullIntelligenza Articiale in Lugano Switzerland It was made p ossible due to funding of

IDSIA and was supp orted in part by the Swiss National Fund SNF grant

Long ShortTerm Memory

The more a man knows ab out himself

in relation to every kind of exp erience

the greater his chance of suddenly

one ne morning realizing who in fact he is

or rather Who in Fact he Is

Aldous Huxley in Island

For You

Contents

Introduction

Reinforcement Learning

Reinforcement Learning with Incomplete Information

Current Problems of RL

Goals of the Thesis

Outline and Principal Contributions

Markov Decision Pro cesses

Value Functions

Finite Horizon Problems

Innite Horizon Problems

Action Evaluation Functions

Contraction

Dynamic Programming

Policy Iteration

Value Iteration

Linear Programming

Exp eriments

Description of the Maze Task

Scaling up Dynamic Programming

More Dicult Problems

In this chapter Markov decision pro cesses and dynamic programming were describ ed In

this section we describ e some diculties for the practical utility of MDPs and give a short

description of some p ossible metho ds which allow to at least partially deal with these di

culties

Continuous State Spaces

When the state space is continuous we have to use a function approximator for representing

the value function since there is not a nite number of states which we can represent in a

lo okup table A function approximator maps an input vector which describ es a state to its

value Combining DP with function approximators was already discussed by Bellman

who prop osed using quantization and loworder p olynomial interpolation for approximately

representing a solution Gordon a There are two metho ds for combining function

approximators with DP algorithms

We can just approximate the value function by the function approximator Then we

can use a nite set of sampling states for which we p erform value iteration Gordon b

After computing B V i with the Bellman backup op erator for all sampling states i we

learn a new function approximation by tting these new sample p oints Gordon shows

conditions under which the approximate value iteration algorithm converges when combined

with function approximators

We can approximate the value function and the reward and transition functions by

using state quantization and resampling Given the probability density function of the state

space and the p ositions of the quantized cells we can use sampling techniques to estimate the

transition function and the reward function We may use Monte Carlo simulations to sample

states from within each cell select an action use the MDPmo del to nd the successor state

and map this back to the next active cell By counting how often transitions o ccur we can

estimate the transition probabilities and likewise we can estimate the reward function Then

we can use DP on the approximate mo del

The average reward criterion ARC has other problems however Eg consider an environment with

many p ositive rewards Using ARC an agent may quickly go to a terminal state neglecting larger reward

sums obtainable in a trial

CHAPTER MARKOV DECISION PROCESSES

Curse of Dimensionality

Completely evaluating a p olicy scales up with O n when n is the number of states Therefore

when the number of states is very large DP techniques b ecome computationally infeasible

The number of states scales with the number of dimensions d as O n where n is the

number of values a variable can have in each dimension When the number of dimensions

b ecomes large the state space will explo de and DP cannot b e used anymore with tabular

representations This is known as Bellmans curse of dimensionality One metho d to cop e

with this large number of states is to use function approximators as describ ed ab ove Other

ways to reduce the number of states are

Only store imp ortant states When the state space is large most states have probabil

ity zero of b eing visited and therefore they do not need to b e stored Since the p olicy moves

around on a lower dimensional manifold than the dimension of the state space would suggest

Landelius we can just compute a p olicy for the states which are likely to b e visited

Another way to deal with multiple variables is to reduce the exp onential number of

states which result from combining all variables This can b e done in the following two ways

a A state is usually describ ed by all comp onents but often some comp onents are not

very useful Instead we can decrease the number of variables by using principal comp onent

analysis see eg Jollife

b We can break states into parts by making indep endency assumptions b etween vari

ables and compute separate value functions for the dierent parts blo cks For evaluating a

complete state we add the evaluations of the parts This can b e done by mo dels based on

Bayesian networks or probabilistic indep endence networks Lauritzen and Wermuth

A similar approach based on CMACs Albus a will b e used in Chapter

Conclusion

In this chapter we have describ ed dynamic programming metho ds for computing optimal

p olicies for MDPs We also evaluated them on a set of maze tasks of dierent sizes The ex

p erimental results showed the to our knowledge novel result that value iteration outp erforms

p olicy iteration The reason for this is that p olicy iteration wastes a lot of time evaluating

cyclic p olicies Value iteration up dates the p olicy while evaluating it and in this way uses

the information earlier and more eciently We also saw that DP can quickly solve problems

containing thousands of states and that we have to b e careful to discount the future to o much

esp ecially in case of large goal rewards Finally we discussed some problems which limit the

applicability of the MDP framework for real world problems continuous state spaces and the

curse of dimensionality A nal severe limitation of DP is that it needs a mo del of the MDP

In the next chapter we describ e reinforcement learning metho ds which learn p olicies and do

not need a mo del

Chapter

Reinforcement Learning

In the previous chapter we showed how dynamic programming can b e used to compute optimal

p olicies for Markov decision problems Although dynamic programming techniques are useful

for solving a wide range of problems they need a world mo del consisting of the reward and

transition functions of the Markov decision problem as input Since for real world tasks world

mo dels are not a priori available they have to b e developed rst Engineering a mo del is not

an easy task however many interesting problems are very messy complex and unsp ecied

Vennix and mo deling them is a very complex and time consuming pro cess For most

complex problems a complete mo del is not even needed since it is unlikely that the agent

p olicy will traverse all states the agents sub jective world is only an approximation of

the ob jective world

Reinforcement Learning Reinforcement learning RL provides us with a framework

for training an agent by exploring an environment and learning from the outcomes of such

trials Samuel Sutton In reinforcement learning problems an agent receives

input from the environment selects and executes an action and receives reward which tells

how go o d its last action was The goal of the agent is to select in each state the action

which leads to the largest future discounted cumulative rewards To solve this RL metho ds

Watkins Bertsekas and Tsitsiklis Kaelbling et al try out dierent action

sequences and learn how much long term reinforcement the agent receives on average by

selecting a particular action in a particular state These estimated values are stored in the

Qfunction which is used by the p olicy to select an action

Direct vs Indirect RL This chapter describ es direct RL metho ds which learn the Q

function directly from exp erienced interactions with the world Direct metho ds are opp osed

to indirect RL approaches which rst estimate a world mo del from the exp eriences and then

use DPlike metho ds to compute the Qfunction using the imp erfectly estimated mo del This

type of mo delbased approaches will b e discussed in the following chapter

TD and Qlearning In the next sections we will describ e the most imp ortant RL

metho ds TD metho ds Sutton and Qlearning Watkins These metho ds are

driven by the error or dierence b etween temp orally successive predictions or Qvalues There

have b een some successful applications of these metho ds Tesauro showed impressive results

with TDmetho ds by designing TDGammon a program which learned to play backgammon

by selfplay Tesauro For the knowledge representation he used a feedforward neural

network which was able to generalize well given the facts that there are ab out dierent

backgammon p ositions and that the program reached human exp ert level after ab out

CHAPTER REINFORCEMENT LEARNING

training games Other successful applications of RL to complex problems were done by Zhang

who used TD with timedelay neural networks LeCun et al for iteratively repairing

jobshop schedules Zhang and Liu and by Crites who used a team of Qlearning agents

to train a neural network for controlling multiple elevators in a simulated environment Crites

and Barto

Outline Section describ es the principles of RL Section describ es TD metho ds

Sutton It describ es the distinctions b etween two existing algorithms for dealing with

eligibility traces replacing and accumulating traces Singh and Sutton Section

describ es QLearning Watkins and Section describ es Qlearning Peng and

Williams which combines Qlearning and TDlearning In Section we introduce a

more ecient online Q algorithm Wiering and Schmidhuber b which improves the

worst case up date complexity for online Qlearning from OjS jjAj to OjAj where jS j is

the number of states and jAj the number of actions In Section we present Team TD

a novel idea for combining tra jectories generated by multiple agents and we will also present

the Team Q algorithm which combines this idea with our Q algorithm In Section

we demonstrate the practical sp eed up of the novel online Q algorithm by comparing it to

the previous one discussed in Peng and Williams on exp eriments with mazes

Then we also show exp erimental results with the replace and accumulate traces algorithms

and with our Team Q algorithm In Section we nish with a short conclusion

Principles of Algorithms

In reinforcement learning we have an agent and an environment and we want to iteratively

improve the agents p olicy by letting it learn from its own exp eriences This is done by

testing the p olicy in the environment ie we let the agent execute its b ehavior and observe

which actions it executes in which states and where it obtains rewards or punishments Using

such exp eriences consisting of input action and received reward we adjust the agents p olicy

Testing the p olicy has to b e done with care If we would simply test p olicies by always

greedily selecting the action with the largest value the agent might rep eat the same b ehavior

trial after trial Although we would p erfectly learn the Qfunction for this b ehavior the agent

would not b e able to learn from completely distinct exp eriences which may change the

value function and the p olicy The reason why the agent might rep eatedly select sub optimal

actions is that alternative optimal actions may b e underestimated due to the small number of

exp eriences and unlucky results biased in a negative way when they were executed Thus

we need a certain amount of exploration

ExplorationExploitation For reinforcement learning we must use exploration which

exploits the p olicy but selects nonp olicy actions as well If we rep eatedly test all actions

in all states we get sucient exp eriences to estimate the real evaluation of each action

Exploration actions aim at trying out new exp eriences which can b e used to improve the

p olicy and is opp osed to exploiting actions which bring the agent more immediate reward

The explorationexploitation dilemma Thrun is to nd a strategy for choosing b etween

exploration and greedy actions in order to achieve most reward in the long run and will b e

This algorithm can also b e applied to implement an O algorithm for online TD learning with a

tabular representation of the value function

For sto chastic environments the agent may learn from novel exp eriences although for small noise values

the probability distribution over tra jectories is still very p eaked towards the greedy tra jectory

PRINCIPLES OF ALGORITHMS

the topic of Chapter In this chapter we assume that there is a given exploration rule X

which relies on the p olicy for choosing the action The simplest rule which we use in this

chapter is the Maxrandom exploration rule which selects the greedy action with probability

P and selects a random action all with equal probability otherwise

max

Online vs oine RL We distinguish online RL and oine RL Online RL up dates

mo diable parameters after each visit of a state Oine RL delays up dates until a trial is n

ished that is until a goal has b een reached or a time limit has expired Without explicit trial

b oundaries oine RL do es not make sense at all b ecause it would not b e able to learn from

the whole sequence But even where applicable oine RL tends to get outp erformed by on

line RL which uses exp erience earlier and therefore more eciently Rummery and Niranjan

Online RLs advantage can b e huge For instance online metho ds that punish actions

to prevent rep etitive selection of identical actions can discover certain environments goal

states in p olynomial time Ko enig and Simmons while oine RL requires exp onential

search time for them Whitehead

Now we will fo cus on dierent metho ds for learning from exp eriences A reinforcement

learning agent starts in a sp ecic starting state s and rep eats the following cycle throughout

a trial

a Cho ose an action with p olicy and exploration rule X given s

t t

r Execute the selected action and receive a scalar reward

s Observe the new state

Use the exp erience to up date the p olicy

The last step up dates the p olicy after a new exp erience a quadruple s a r s is

t t t t

known

Delayed Reward and Credit Assignment Problem

Reinforcement learning algorithms have to b e able to deal with delayed reward and the credit

assignment problem The problem of delayed reward is that the results of actions are often

not immediately known eg in some problems reward is only given after some goal has b een

reached From this the problem of credit assignment follows since we do not have a unique

measure of go o dness for each indep endent action we have to split the nal reward signal

obtained by a whole sequence of actions in the contributions of single actions and not all of

them are likely to b e equally resp onsible for the nal outcome For example if an agent follows

a particular path and nally reaches a goal it is not immediately clear which actions did and

which actions did not bring the agent closer to the goal see Figure Even if there are

many tra jectories connecting a state and the goal we cannot b e sure which action is optimal

in that state as long as the state is not connected to the goal by the optimal tra jectory which

p ossibly consists of parts of all generated tra jectories

Markov Prop erty

In this chapter we will assume that the environment is stationary which means that the transi

tion and reward functions do not change with time Furthermore we assume that the Markov

prop erty holds The Markov prop erty requires that the rewards and probabilities of making

transitions to next states after executing an action in a state are fully determined by the

CHAPTER REINFORCEMENT LEARNING

GOAL

START

Figure A trajectory going from the start to the goal state Which actions brought the

agent closer to the goal

current stateaction pair SAP More formally this means that for all p ossible stateaction

sequences the following equation holds

P s ijs a P s ijs a s a

t t t t t t

where P s ij is the conditional probability of b eing in state i at time t

TD Learning

Temporal dierence TD learning Sutton can b e applied for learning to predict the

outcomeresults of a sto chastic pro cess

Prediction problems Supp ose we observe a Markov chain see App endix A which

emits a reward on each transition and always reaches a terminal eg goal state The

goal of a prediction problem is to learn the exp ected total future cumulative reward from

each state More precisely in prediction problems we receive training data in the form of

histories tra jectories of M trials H H consisting of sequences of statereward pairs

g where s is the state at time t r is the emitted reward r H fs r s r s

t t T i N

i i

at time t and T is the length of the i trial The goal is to learn the exp ected cumulative

future rewards value V s for each state s S

V s E r js s

i t

where T is a random variable denoting the trial length and the exp ectation op erator E

accounts for the sto chasticity of the pro cess including dierent times of o ccurrence of the

state Note that a state can b e visited zero one or multiple times during a single trial

Monte Carlo sampling We can learn ab out the outcomes of a Markov chain by using

Monte Carlo MC sampling which uses multiple simulations of a random pro cess to collect

exp erimental data Then we can compute the exp ected outcome by averaging the outcomes

s will denote the indicator function of the dierent indep endent trials In what follows

which returns if s o ccurred at time t in the i trial and otherwise We simply write

TD LEARNING

s if the trial number do es not matter In oine or batchmode MC sampling with the

WidrowHo algorithm Widrow and Ho estimates the prediction V s of the future

emitted rewards given that the pro cess is currently in state s as follows

P P P

T T

t i

i i

s r

i j t

j t

V s

P P

i t

to take the trial number i into account We can also learn more incre where we wrote r

mentally online by up dating the predictions after each trial The online metho d learns to

minimize the dierence b etween the observed cumulative future reward in the current trial

since state s was visited and the current prediction V s We minimize this dierence by

p erforming gradient descent on the squared error function E We dene the error of the visit

of s at time step t E s as

E s r j V s

t i

To minimize this error we rst dierentiate E s with resp ect to V s

E s

V s r j

V s

From this we construct the up date rule which decreases this error using iterative steps with

learning rate The learning rate inuences the size of up date steps and can b e used to

guarantee convergence We will use k where k is the number of o ccurrences of state

V s V s r j

s i

The problem of learning on the MC WidrowHo estimates is that the variance in the

up dates can b e large if the pro cess is noisy and therefore many trials are needed b efore the

condence intervals of the estimates b ecome small the variance go es down with the square

ro ot of the number of state o ccurrences The variance can b e reduced by using information

contained in the structure of the generated state sequence Eg supp ose a visit of some state

is always followed by a visit of one particular state which can also follow other states When

the rst state is visited times and the second times we could compute the rst states

value by using the value of the second state since our exp eriences tell us with certainty that

we will make a transition to this state The second states estimate is computed over

trials and thus a more reliable estimate than the MC estimate of the rst state which is an

average of the results of trials

Temporal Dierence Learning

We can reduce the variance in the up dates by using temp oral dierence TD learning TD

metho ds do not only use future rewards but also the current predictions of successive states

For online learning the values V s are initialized b efore learning to some value such as

CHAPTER REINFORCEMENT LEARNING

for up dating state predictions Temporal dierence metho ds such as Qlearning and TD

metho ds enforce a time continuity assumption Pineda states which are visited in the

same time interval should predict ab out the same outcomes In essence TD metho ds try to

minimize the dierence b etween predictions of successive time steps

TD up dating Instead of adjusting a states prediction on the cumulative reward

received during the entire future we can also just learn to minimize the dierence b etween

the current states prediction and the reward which immediately follows the visit of the state

plus the next states prediction The TD algorithm which is by the way very similar to Q

learning is exactly doing this The learning algorithm uses temp oral dierences of successive

state values to adapt the predictions Lets dene V s s TDerror e as

t t

e r V s V s

t t t t

This can b e simply used to adjust V s

V s e

t s t

TD has the largest bias inside the TDfamily since the learning signal is very much cor

related with the current state Its variance is small however since in general we may exp ect

to have only small TD errors Sutton showed that by rep eatedly learning from

sequences with TD the estimates asymptotically converge to the true exp ected outcome

TD metho ds TD metho ds are parameterized by a value and although they use

the entire future tra jectory for up dating state predictions rewards and predictions further

away in the future are weighted exp onentially less according to their temp oral distance by

the parameter TD metho ds have b een proved to converge with probability to the

true prediction for all Dayan Dayan and Sejnowski Jaakkola et al

Tsitsiklis Pineda provided each state is visited by innitely many tra jectories

and the learning rates diminish to zero at a suitable rate see App endix B TD uses

is dened as to discount TDerrors of future time steps The TDerror e

T t

e e and V s e

ti t s

t t

where we use e as dened in equation It is imp ortant to set the parameter correctly

for ecient learning since it determines the tradeo b etween the bias and the variance If

is larger the variance of up date steps is larger since the cumulative rewards computed over

many steps will generally o ccupy a much larger range of outcomes than singlestep rewards

and the values of successive states

The TD errors ab ove cannot b e computed as long as TD errors of future time steps

are not known We can compute them incrementally however by using eligibility traces

Barto et al Sutton Sutton

Eligibility traces Given a new observation we can immediately up date all predictions

of states visited previously in the same trial In Figure we show an example of a simulated

Markov chain After each new visit of a state a new TDerror b ecomes known The gure

shows how these TDerrors are backpropagated to previous states To memorize which states

This time continuity assumption holds for Markov pro cesses and follows from the rules of causality and

the direction of time For nonMarkov pro cesses the assumption do es not hold and thus we should b e careful

using TDlearning for them

TD LEARNING

1 2 3 4

e e

e1 2 3

Figure A simulated Markov chain going through states The gure shows how

TDerrors are backpropagated to al l previous states once new states are encountered

o ccurred and how long ago they o ccurred eligibility traces are used Barto et al The

eligibility of a state tells us how much a state is eligible to learn from current TDerrors

This value dep ends on the recency of its o ccurrence and the frequency of its o ccurrence

Omitting the learning rate for simplicity V ss increment for the complete trial is

V s e s

T T

X X

it t

e s

t T

X X

ti i

e s

t T

X X

ti i

s e

e l s

t t

Where at the last step we simplied the expression by introducing the symbol l s the

accumulating eligibility trace for state s

ti i

l s s

Computing this trace can b e implemented using the following recursive form

l s l s if s s

t t t

l s l s if s s

t t t

Thus at each time step eligibility traces of all states decay whereas the currently visited state

has its trace increased This has the eect that only states which have b een visited recently

are made eligible to learn on future examples

CHAPTER REINFORCEMENT LEARNING

Now the online up date at time t b ecomes with learning rate

s a S A do V s V s e l s

s t t

The eligibility trace can b e thought of as an attentional trace As long as the eligibility trace

of some state is on larger than this state will have its value adapted on TD errors of

new exp eriences The ab ove metho d uses an accumulating eligibility trace after each visit of

a state its eligibility trace gets strengthened by adding to it

Replacing Traces

A dierent metho d is to replace traces when a state is revisited Replacing traces Singh and

Sutton resets the trace if a state is revisited

l s l s if s s

t t t

l s if s s

t t

The eect is that only recency determines the eligibility trace and that frequency do es not

matter To show the dierence b etween the accumulate and replace traces algorithms we

will lo ok at the case and will analyze the bias of the estimators using a simple example

following Singh and Sutton

RAA; P

RB; PB

Figure A Markov chain consisting of two states starting state A and terminal state B

Transition probabilities are denoted as P and P transition rewards are denoted as R and

A B A

Consider the Markov chain of Figure It consists of two states the starting state A

and the terminal state B In A the transition to the next state is sto chastic a step is made

to A and B with probability P and P resp ectively The chain starts at A we assume

A B

V B and we are interested in computing the value V A

Optimal value The optimal value V A can easily b e computed according to

V A P R V A P R V B

A A B B

which has the following solution

R R V A

A B

P B

TD LEARNING

Replacing tracesFirst visit MC Singh and Sutton used for the analysis

which is the worstcase for accumulating traces for b oth metho ds p erform the

same up dates For this value it can b e shown Singh and Sutton that replacing

traces implements a rstvisit Monte Carlo metho d given batch up dates oine learning

and prop er learning rate annealing This metho d estimates the value of a state by simply

summing the rewards received during a trial after the rst time the state was visited This

means that in case of multiple o ccurrences of the state during a trial only the rst one matters

For the Markov chain in Figure the rst visit MC algorithm note that for this Markov

chain the replace traces algorithm would make the same up dates for all values of would

compute V A as follows given a single trial

V A R A R A R A R B

for which the exp ectancy over all p ossible tra jectories is Singh and Sutton

P P

A A

R R E fV Ag R P R

B A B B A

P P

A B

When we compare equations and we can see that the rst visit MC algorithm computes

a correct unbiased estimate The estimate over multiple trials is simply computed as the

average of the estimates for the single trials and thus it is also an unbiased estimate

Accumulating tracesEvery visit MC The every visit MC metho d computes the

same up dates as the accumulating trace algorithms for and prop er learning rate an

nealing It computes the value V a for the Markov chain in Figure as follows given a

single i trial

w R A R A k R A k R B

V A

k k

Considering all p ossible trials and weighting them according to their probability of o ccurring

the estimate after one trial is

P A

E fV Ag R R

e B A

P B

P A

Thus the estimate for a single trial is biased with B I AS E fV Ag V A R

e A

P B

The estimate over multiple trials is computed as

V A

its bias after M trials is Singh and Sutton

M P

and thus the bias go es to when the number of trials M go es to At the rst glance it may

seem strange that the bias go es to However if we have multiple trials we will get many

dierent outcomes and due to reasons of combinatorics eg there are p ossibilities to have

in one trial and in another trial revisits of state A whereas there is only one p ossibility

to have two times zero revisits our estimate is less determined by a single outcome with an

overestimated probability immediately going to state B

The analysis of Singh and Sutton furthermore shows that initially the mean squared error

Bias Variance is larger for the rst visit MC algorithm but that it will catch up in the

long run This implies that for dicult problems replacing traces is the b etter candidate

CHAPTER REINFORCEMENT LEARNING

QLearning

One of the simplest reinforcement learning algorithms for learning to control is Qlearning

Watkins Watkins and Dayan Qlearning enables an agent to learn a p olicy

by rep eatedly executing actions given the current state as input At each time step the

algorithm uses step lo okahead to up date the currently selected stateaction pair SAP Q

learning up dates all SAPs along a solutionpath a single time thus moving the nal reward

for a trial one step back in the chain Therefore it takes a lot of time b efore the goal

reward will b e propagated back to the rst SAP Eg for learning a solutionpath costing

steps naive Qlearning needs to traverse the path for at least trials involving

SAPup dates Although slow Qlearning has b een proven to converge to the optimal p olicy

provided all stateaction pairs are tried out innitely many times and the learning rate is

prop erly annealed Watkins and Dayan The Qlearning up daterule is as follows

Conventional Qlearnings a r s

t t t t

e r V s Qs a

t t t t

Qs a Qs a s ae

t t t t

Here V i max Qi a is the discount factor which assigns preference to immediately

obtaining reward instead of p ostp oning it for a while s a is the learning rate for the k

up date of SAP s a and e is the temp oral dierence or TDerror which tends to decrease

over time

Learning rate adaption The learning rate is determined by s a which should b e

decreased online so that it fullls two sp ecic conditions for sto chastic iterative algorithms

Watkins and Dayan Bertsekas and Tsitsiklis The conditions on the learning

rate s a are

s a and

s a

Learning rate adaptions for which the conditions are satised may b e of the form

where k is a variable that counts the number of times a stateaction pair has b een up dated In

b oth conditions App endix B we present a to our knowledge novel pro of that for

hold

Qlearning

Qlearning Watkins Peng and Williams is an imp ortant reinforcement learn

ing RL metho d It combines Qlearning and TD Q is widely used it is generally

b elieved to outp erform simple onestep Qlearning since it uses single exp eriences to up date

evaluations of multiple stateaction pairs SAPs that have o ccurred in the past

Qs TD error First note that Qlearnings up date at time t may change V s

r V s Qs a Following Peng and Williams we in the denition of e

t t t t

dene the TDerror of V s as

e r V s V s

t t t t

The dierence with Watkins Q is that he prop osed e r V s Qs a

t+1 t+1 t+2 t+1 t+1

QLEARNING

Q uses a factor to discount TDerrors of future time steps

Qs a Qs a s a e

t t t t t t

is dened as where the TDerror e

e e e

t t

the dierence b etween the Qvalue of the current stateaction and immedi Here we add e

ate reward plus next Vvalue to the TD error of subsequent steps computed from the

dierences b etween a Vvalue and the immediate reward plus the Vvalue of the next state

Eligibility traces Again we make use of eligibility traces Barto et al Sutton

In what follows s a will denote the indicator function which returns if s a

o ccurred at time t and otherwise Omitting the learning rate for simplicity Qs as

increment for the complete trial is

Qs a lim e s a

k k

X X

t it t

lim e s a e s a

k t

X X

t ti i

lim e s a e s a

t k

X X

ti i t

s a lim e s a e

To simplify this we use the symbol l s a the eligibility trace for SAP s a

ti i

l s a s a

and the online up date at time t b ecomes

s a S A do Qs a Qs a e s a e l s a

t t

We can see that this up date is done for all SAPs This makes online up dating very slow

Fortunately there are metho ds to circumvent having to up date all SAPs One of them only

needs OjAj up dates and will b e presented in the next section

Online Q We will fo cus on Peng and Williams algorithm PW although

there are other p ossible variants eg Rummery and Niranjan PW uses a list H of

SAPs that have o ccurred at least once SAPs with eligibility traces b elow are removed

from H Bo olean variables v isiteds a are used to make sure no two SAPs in H are identical

CHAPTER REINFORCEMENT LEARNING

PWs Qup dates a r s

t t t t

e r V s Qs a

t t t t

e r V s V s

t t t t

For each SAP s a H Do

a l s a l s a

t t

b Qs a Qs a e l s a

t t

c If l s a

c H H n s a

c v isiteds a

Qs a Qs a e

t t t t

l s a l s a

t t t t t t

If v isiteds a

t t

a v isiteds a

t t

b H H s a

t t

Comments The sarsa algorithm Rummery and Niranjan replaces the right

hand side in lines and by r Qs a Qs a

t t t t t

For replacing eligibility traces Singh and Sutton step should b e

a l s a l s a

t t t t t

Representing H by a doubly linked list and using direct p ointers from each SAP to its

p osition in H the functions op erating on H deleting and adding elements see lines c

and b cost O

Complexity Deleting SAPs from H step c once their traces fall b elow a certain

threshold may signicantly sp eed up the algorithm If is suciently small then this will

keep the number of up dates p er time step manageable For large PW do es not work

as well it needs a sweep sequence of SAP up dates after each time step and the up date

cost for such sweeps grows with Let us consider worstcase b ehavior which means that

each SAP o ccurs just once Near trial b egin the number of up dates increases linearly until

log

at some time step t some SAPs get deleted from H This happ ens as so on as t

log

Since the number of up dates is b ounded from ab ove by the number of SAPs the total up date

complexity increases towards O jS jjAj p er up date for

Fast Qlearning

There have b een several ways to sp eed up Q Lins oine Q creates an action

replay set of exp eriences after each trial Computing the TDreturns can b e done linearly

in the length of the history However oine Qlearning is in general much less ecient

than online Q Cichosz semionline metho d combines Lins oine metho d and

online learning It needs fewer up dates than Peng and Williams online Q but p ostp ones

Qup dates until several subsequent exp eriences are available Hence actions executed b efore

the next Qup date are less informed than they could b e This may result in p erformance

loss For instance supp ose that the same state is visited twice in a row If some hazardous

actions Qvalue do es not reect negative exp erience collected after the rst visit then it may

get selected again with higher probability than wanted

Our prop osed metho d Previous metho ds are either not truly online and thus are

likely to require more exp eriences or their up dates are less ecient than they could b e and

FAST QLEARNING

thus require more computation time We will present a Q variant which is truly online

and ecient its up date complexity do es not dep end on the number of states Wiering and

Schmidhuber b The metho d can also b e used for sp eeding up tabular TD to achieve

O up date complexity It uses lazy learning introduced in memorybased learning eg

Atkeson Mo ore and Schaal to p ostp one up dates until they are needed The algorithm

is designed for otherwise we can use simple Qlearning

Main principle The algorithm is based on the observation that the only Qvalues needed

at any given time are those for the p ossible actions given the current state Hence using lazy

learning we can p ostp one up dating Qvalues until they are needed Supp ose some SAP s a

t t

o ccurs at not necessarily successive time steps t t t Let us abbreviate s a

First we unfold terms of expression

t k

X X

ti i t

e e

t t t

3 2 1

X X X X X X

ti i t ti i t ti i t

e e e e e e

t t t

tt tt t

i i i

2 1

Since is only for t t t t and otherwise we can rewrite this as

t t

2 3

X X

tt tt tt

1 1 2

e e e e e

t t

t t t

1 2 3

tt tt

1 2

t t

2 3

X X

t t

e e e e e

t t

t t t

3 1 2

t t t

1 1 2

tt tt

1 2

t t t t

1 2 2 3

X X X X

t t t t

e e e e e e e

t t t t

t t t

1 2 3

t t t

1 1 2

t t t t

e this b ecomes Dening

i t

e e e

t t t t

t t t

2 1 3 2

1 2 3

t t t

1 1 2

This will allow for constructing an ecient online Q algorithm We dene a lo cal trace

l s a and use to write the total up date of Qs a during a trial as

i t

e s a l s a Qs a lim

t t

To exploit this we introduce a global variable which keeps track of the cumulative TD

error since the start of the trial As long as SAP s a do es not o ccur we p ostp one up dating

Qs a In the up date b elow we need to subtract that part of which has already b een used

see equations and We use for each SAP s a a lo cal variable s a which records

the value of at the moment of the last up date and a lo cal trace variable l s a Then

once Qs a needs to b e known we up date Qs a by adding l s a s a Figure

illustrates that the algorithm substitutes the varying eligibility trace l s a by multiplying a

t t

global trace by the lo cal trace l s a The value of changes all the time but l s a

do es not in intervals during which s a do es not o ccur

CHAPTER REINFORCEMENT LEARNING

lt (s,a)

l'(s,a)t 1 φ t

t ->

t1 t2 t 3

Figure SAP s a occurs at times t t t The standard eligibility trace l s a equals

the product of and l s a

Algorithm overview The algorithm relies on two pro cedures the Local Update pro ce

dure calculates exact Qvalues once they are required the Global Update pro cedure up dates

the global variables and the current Qvalue Initially we set the global variables

and We also initialize the lo cal variables s a and l s a for all SAPs

Lo cal up dates Qvalues for all actions p ossible in a given state are up dated b efore an

action is selected and b efore a particular Vvalue is calculated For each SAP s a a variable

s a tracks changes since the last up date

Lo cal Up dates a

t t

Qs a Qs a s a s a l s a

t t t t t t t t t t

s a

t t

The global up date pro cedure After each executed action we invoke the pro cedure

Global Update which consists of three basic steps To calculate V s which may have

changed due to the most recent exp erience it calls Local Update for the p ossible next SAPs

It up dates the global variables and It up dates s a s Qvalue and trace

t t

variable and stores the current value in Local Update

FAST QLEARNING

Global Up dates a r s

t t t t

a A Do

a Local U pdates a

r V s Qs a e

t t t t

e r V s V s

t t t t

t t

Local U pdates a

t t

Qs a Qs a s a e

t t t t t t

l s a l s a

t t t t

For replacing eligibility traces Singh and Sutton step should b e changed as follows

a l s a l s a

t t t

Machine precision problem and solution Adding e to in line may create

a problem due to limited machine precision for large absolute values of and small

there may b e signicant rounding errors More imp ortantly line will quickly overow any

machine for The following addendum to the pro cedure Global Update detects when

falls b elow machine precision up dates all SAPs which have o ccurred again we make

use of a list H and removes SAPs with l s a from H Finally and are reset to

their initial values

Global Up date addendum

If v isiteds a

t t

a H H s a

t t

b v isiteds a

t t

a Do s a H

a Local U pdates a

a l s a l s a

a If l s a

a H H n s a

a v isiteds a

a s a

Comments Recall that Local Update sets s a and up date steps dep end on

s a Thus after having up dated all SAPs in H we can set and s a

t t

Furthermore we can simply set l s a l s a and without aecting the

expression l s a used in future up dates this just rescales the variables Note that if

no sweeps through the history list will b e necessary

Complexity The algorithms most exp ensive part are the calls of Local Update whose

total cost is O jAj This is not bad even simple Qlearnings action selection pro cedure

costs O jAj if say the Boltzmann rule Thrun Caironi and Dorigo is used

Concerning the o ccasional complete sweep through SAPs still in history list H during each

sweep the traces of SAPs in H are multiplied by l e SAPs are deleted from H once their

trace falls b elow e In the worst case one sweep p er n time steps up dates n SAPs and costs m

CHAPTER REINFORCEMENT LEARNING

O on average This means that there is an additional computational burden at certain

time steps but since this happ ens infrequently our metho ds total average up date complexity

stays O jAj

The space complexity of the algorithm remains O jS jjAj We need to store the following

variables for all SAPs Qvalues eligibility traces previous delta values the visited bit

and three p ointers to manage the history list one from each SAP to its place in the history

list and two for the doubly linked list Finally we need to store the two global variables

Comparison to PW Figure illustrates the dierence b etween theoretical worstcase

b ehaviors of b oth metho ds for jAj jS j and We plot up dates p er time step

for f g The accuracy parameter used in PW is set to in practice less

precise values may b e used but this will not change matters much The machine precision

parameter is set to The spikes in the plot for fast Q reect o ccasional full sweeps

through the history list due to limited machine precision the corresp onding average number

of up dates however is very close to the value indicated by the horizontal solid line as

explained ab ove the spikes hardly eect the average No sweep is necessary in fast Qs

plot during the shown interval Fast Q needs on average a total of up date steps in

chooseaction for calculating V s for up dating the chosen action and for taking

into account the full sweeps

400 80000

Fast Q lambda = .7 350 Fast Q lambda = .9 70000 PW lambda = .7 PW lambda = .7 PW lambda = .9 PW lambda = .99 300 PW lambda = .9 60000 PW lambda = .99 Fast Q lambda = .7 Fast Q lambda = .9 250 50000

200 40000 Nr Updates Nr Updates 150 30000

100 20000

50 10000

0 0 0 80 160 240 320 400 0 80 160 240 320 400

Time Time

Figure Number of updates plotted against time a worst case analysis for our method and

Peng and Wil liams PW for dierent values of A Number of updates per time step

B Cumulative number of updates

Multiple Trials We describ ed a singletrial version of our algorithm One might b e

tempted to think that in case of multiple trials all SAPs in the history list need to b e up dated

and all eligibility traces reset after each trial This is not necessary we may use crosstrial

learning as follows

M th

We introduce variables where index M stands for the M trial Let N denote the

TEAM Q

current trial number and let variable v isiteds a represent the trial number of the most

recent o ccurrence of SAP s a Now we slightly change Local Update

Lo cal Up dates a

t t

M v isiteds a

t t

Qs a Qs a s a s a l s a

t t t t t t t t t t

s a

t t

If M N

a l s a

t t

b v isiteds a N

t t

Thus we up date Qs a using the value of the most recent trial M during which

SAP s a o ccurred and the corresp onding values of s a and l s a computed during

t t t t

the same trial In case SAP s a has not o ccurred during the current trial we reset the

eligibility trace and set v isiteds a to the current trial number In Global Update we need

to change lines and b by adding trial subscripts to and we need to change line b in

which we have to set v isiteds a N At trial end we reset to increment the

t t

trial counter N and set This allows for p ostp oning certain up dates until after the

current trials end

Team Q

This section contains a novel idea for using multiple agents with eligibility traces Consider

that the value function is shared by a number of agents and that we generate multiple tra jec

tories in parallel to learn the value function Note that the shared value function do es not

make it p ossible for agents to sp ecialize since the team consists of homogeneous agents When

we immediately use TD for multiple agents we will let the agents only learn from their

own exp eriences However sometimes it would b e a go o d thing to let agents learn from each

other if they interact at some p oint For example in a so ccer simulation we could reward an

agent for passing the ball to another agent who makes a goal shortly afterwards In principle

we could design value functions where the value for passing the ball would b e determined by

what happ ens with the ball afterwards This makes a new kind of parallel learning p ossible in

which simultaneously generated tra jectories are combined to create new tra jectories In the

following we will not consider such general cases where agents can interact through passing

ob jects or through communication but will consider the case where two agents are said to in

teract if their tra jectories cross each other In the framework presented in this section called

Team TD we extend single agent TD by linking tra jectories generated by multiple

agents Then if an agents tra jectory traverses another agents tra jectory previous steps of

one agent will b e up dated on the future of the other agent

Extending the Eligibility Trace Framework

The accumulate traces algorithm computes large eligibility traces for trials in which a state

is visited multiple times Supp ose that some state is visited by two dierent tra jectories For

one tra jectory the state may b e visited a single time and for another the state may b e visited

many times The accumulating traces algorithm will cause much larger state value adaptions

CHAPTER REINFORCEMENT LEARNING

for the second tra jectory even though b oth tra jectories may b e equiprobable Therefore we

should sometimes b e careful in up dating a state value

Another view on eligibility traces In principle there are many ways for manipulating

the traces and we have already seen a couple of them In the following we want to extend

our reasoning ab out eligibility traces An eligibility trace is directly related to the temp oral

distance b etween the time step the state was visited and the time step another state is visited

A 1 C

Figure An example trajectory through state space The Markov chain consists of three

abstracted states A B and C At B the chain can return to B or go to C

Have a lo ok at Figure We are interested in the eligibility traces As we have seen

these inuence the amount with which a previous state learns on the current transition in

the tra jectory Thus lets just consider the sequence of states or temp orally directed paths

through state space Eg there can b e sequences

A B B C

A B B B C

A B C

For these dierent sequences replacing traces only eects the estimate of B and not of A

compared to the accumulating traces We could also reset the trace from A to a larger value

eg in this case to whenever B is revisited We could also split a sequence AB B B C into a

tree combining the sequences AB C BBB or AB C BB BB In these ways we would

still learn from the transitions numbered and shown in Figure We should note

that by transforming a sequence into a tree and then using TD the up dates will in general

not b e the same for The second tree however is just a pro jection of the sequence unto

to transition BB and to transition BC Such a Markov chain assigning a probability of

dierent ways of dealing with eligibility traces could help to strengthen the relationship to

full DP techniques or fulledged lo okahead searches The up dates b ecome more complex

however Therefore although some improvement of the learning sp eed may b e obtained this

advantage may b ecome a disadvantage due to the larger computational requirements

Combining Eligibility Traces

A way of extending the eligibility trace framework is to allow for multiple traces Consider

Figure There are two tra jectories which have b een generated by the same underlying

Optimally we use the probabilities of generating a tra jectory Storing probabilities explicitly is done in

world mo dels Similar eects can b e obtained by storing and learning from traces in dierent ways however

TEAM Q

A 1 B EF

Figure Multiple trajectories through state space Trajectory one visits states A B E F

in that order and trajectory two visits states G B C D E and H

Markov chain and we want to up date all state estimates based on b oth tra jectories The

simplest is to let states visited by one tra jectory only learn on future dynamics of that

tra jectory In this way we do not use all p ossible information however If two tra jectories

meet there are multiple p ossible future paths and these can b e used to adapt state values

Eg in gure there is a path b etween state A visited by tra jectory and transition C D

visited by tra jectory and thus we can adapt state As value on C D

The algorithm The algorithm uses dierent eligibility traces and history lists for dier

ent tra jectories and keeps track of the order in which states are visited ie the most recently

visited states are the rst in the list if a state is revisited its p osition in the list advances

We will rst introduce some notation

i i

g is the history list of tra jectory i and is an ordered list of all its s H fs

visited states

U s lists all history lists of tra jectories which have visited state s

i i

pr eds H lists all states which precede s in the history list H

i i i i

The general way of connecting traces is as follows Once two tra jectories have visited the

same state s ie they meet in s we traverse the whole list of states which were visited

m m

by the tra jectories b efore the meeting state to connect them Thus after tra jectory i has

made a step we p erform the following op erations

We insert the new state s at the end of tra jectory is history list

If s H then H inserts H

m i i m i

We check which tra jectories have also visited state s The history lists of these

tra jectories are inserted in the lists U s of all states s previously visited by

i i

tra jectory i

H H U s s pr eds H if H U s then

i m i m i i

U s U s fH g

i i

Finally we check which other tra jectories have visited state s Then we put

tra jectory i in the lists U s of all states s previously visited by one of the other

tra jectories

H H U s s pr eds H U s U s fH g

i m m i

CHAPTER REINFORCEMENT LEARNING

Computing eligibility traces Ab ove we only describ ed which states learn on which

tra jectories We will now describ e how their eligibility traces are computed Each state s has

s to learn on each tra jectory is future steps In the b eginning we initialize an eligibility l

all eligibility traces for all K tra jectories and for all states

s i s l

If a tra jectory visits a particular state this states eligibility for that tra jectory is up dated

as in the fast online Q algorithm For connecting traces we also compute new eligibility

traces if we insert a tra jectory in the list of visited tra jectories of some state Thus after we

p erform the op eration U s U s fH g where s pr eds H we also compute

j m i

s l

s s l l

j j

s l

Thus all states visited by one tra jectory H for which the eligibility trace is for the

other tra jectory H are assigned an eligibility trace to the tra jectory H which is calculated

j j

by multiplying the eligibility trace of p oint s on the tra jectory H by where d is the

m j

temp oral distance along tra jectory H b etween state s and state s In this way they will

i m

start learning on the future dynamics from the other tra jectory

To make things more ecient we traverse the lists backwards and stop once some state

already has non zero eligibility trace on the other tra jectory If such a state is only visited

once we can b e sure that all previous states also have a nonzero eligibility trace If a state

has b een visited multiple times however it can b e the case that it already has an eligibility

on the other tra jectory whereas the preceding states have not see Figure Therefore

we only stop traversing the list backwards if we nd a state with nonzero eligibility which

has not b een revisited see App endix C for the complete algorithm If states are revisited

often the pro cedure can still b e quite exp ensive since it is p ossible that we would traverse

the whole list The probability that one agent has revisited a large number of consecutive

states is fortunately very low

A 2 X

Figure An example of two trajectories crossing each other where we cannot stop tracing

trajectory back after noticing that state X already has a non zero eligibility trace on trajectory

At the moment that trajectory has hit trajectory in X for the rst time X and A got

a non zero eligibility trace on trajectory and therefore B would stay without eligibility if we

would stop in X

Connecting multiple tra jectories If there are multiple tra jectories things get a

little bit more complicated Now if tra jectory meets tra jectory and tra jectory has

EXPERIMENTS

hit tra jectory b efore we have to follow tra jectory and let tra jectory states also get

eligibility on tra jectory

Again we rst insert the new state in the history list of tra jectory i H inserts H

i m i

We will now describ e the recursive pro cedure C onnectH s H which connects all states

i m j

visited by tra jectory i b efore state s to tra jectory j Furthermore it also connects all states

visited by another tra jectory which traverses one of these states to tra jectory j

C onnectH s H

i m j

A s pr eds H U s U s fH g

i m i i i j

B s pr eds H H H H U s C onnectH s H

i m i i j i i j

k k

Then we call this pro cedure for all pairs which include the new tra jectory H

H H U s C onnectH s H and C onnectH s H

i m i m m i

Some limitations and their implications There are two problems with the approach

The Markov prop erty is essential for the p ossibility to do multiple trace learning since

otherwise it may not b e p ossible to jump from one tra jectory to another The eligibility

trace from a state visited by one tra jectory on another tra jectory is determined by the order

in which the tra jectories meet However the usual eligibility traces have the same problem

we cannot eciently make eligibility traces invariant to the order in which exp eriences

o ccur

Comment The approach resembles the replacing traces in the manner that it deals

with multiple crossings of tra jectories For computing eligibility traces for single tra jectories

we can use the replacing traces algorithm or the accumulating traces We have used this in

an algorithm based on Team Qlearning which enables us to train a p olicy with multiple

agents see App endix C

Complexity of the algorithm Note that the new complexity of the lo cal up date rule

dep ends on the number of agents tra jectories ie it b ecomes OK for a single up date step

where K is the number of agents

Stopping connecting the lists in the way describ ed ab ove may help to keep the average

computational requirements low When states are not revisited by the same tra jectory we

could at most connect each state visited by of the tra jectories to all other tra jectories Thus

we could have made TK links after T steps which makes an average up date complexity of

O K In the worst case states are often revisited so that we need to traverse the whole

list backwards Then the worst case complexity would b ecome O jH jK where jH j is the

length of the history list For most problems agents will not revisit long sequences of states

and therefore it may b e exp ected that the up date complexity scales up linearly with the

number of tra jectories Computationally we do therefore not exp ect a large dierence in the

up date complexity by learning from many indep endent trials or by learning from combining

the traces However if we plan to use multiple agents and generating tra jectories is exp ensive

there may b e a big gain in using the new algorithm since we learn more on each exp erience

Exp eriments

In this section we will describ e the following three exp erimental comparison studies We

compare our novel fast Q to PWs Q We compare replacing traces to accumulating

traces and We compare Team Q to Indep endent Q

CHAPTER REINFORCEMENT LEARNING

Evaluating Fast Online Q

To evaluate the practical p erformance improvement of the fast Q metho d over Peng and

Williams Q we use dierent mazes than in Chapter We created a set of dierent

randomly initialized mazes each with ab out blo cked elds All mazes share

a xed start state S and a xed goal state G we discarded mazes that could not b e

solved by Dijkstras shortest path algorithm See gure for an example maze

In each eld the agent can select one of the four actions go north go east go south go

west Actions that would lead into a blo cked eld are not executed Once the agent nds the

goal state it receives a reward of and is reset to the start state The action execution

is without noise thus the transition function is deterministic this makes it p ossible to use

larger values for All other steps are punished by a reward of The discount factor

is set to

Figure A maze used in the experiments Black states denote blocked elds

White squares are accessible states There are about accessible states The agents aim

is to nd the shortest path from starting state S to goal state G

Exp erimental setup A single run on one of the twenty mazes consisted of

steps Every steps the learners p erformance was monitored by computing its cumu

lative reward so far We must note that we measure p erformance as the cumulative reward

sum while the agent learns to maximize a discounted sum Since all emitted rewards are

equal however the evaluation criteria measure the same increasedecrease in p erformance

The optimal p erformance is ab out K reward p oints this corresp onds to

step paths To select actions we cannot use the Qfunction in a deterministic way and by

selecting the action with the largest Qvalue since this quickly leads to rep etitive sub optimal

b ehavior Instead we used the Maxrandom exploration rule which selects an action with

maximal Qvalue with probability P and a random action with probability P Note

max max

that this probability P has nothing to do with the noise in the execution of actions which max

EXPERIMENTS

stems from the environment During each run we linearly increased P from start

max

until end

To show how learning p erformance dep ends on we set up multiple exp eriments with

dierent values for and used for annealing the learning rate As shown in App endix

B the learning rate annealing parameter must theoretically lie b etween and but

may b e lower in practice to increase the learning p erformance If is large and to o small

however then the Qfunction may diverge

Parameters We tried to choose the lowest and hence the largest learning rate such

that the Qfunction did not diverge Final parameter choices were Q and Q with

Q with Q with Q with Q with

PWs trace was set to Larger values scored considerably less well lower

ones costed more time Machine precision was set to Time costs were computed

by measuring CPU time including action selection and simulator time on a MHz Sun

SPARC station

Results Learning p erformance for fast Q is plotted in Figure A results with PWs

Q are very similar We observe that larger values of increase p erformance much faster

than smaller values although the nal p erformances are b est for standard Qlearning and

Q Figure B however shows that fast Q is not much more timeconsuming than

standard Qlearning whereas PWs Q consumes a lot of CPU time for large

3500 Q-learning 40000 Q-learning Fast Q(0.7) Fast Q(0.5) 3000 Fast Q(0.9) Fast Q(0.7) Fast Q(0.99) Fast Q(0.95) 30000 2500 PW Q(0.7) PW Q(0.9) PW Q(0.99) 20000 2000

1500 10000 Time in seconds 1000 Average Reinforcement 0 500 -10000 0 0 1e+06 2e+06 3e+06 4e+06 5e+06 0 1e+06 2e+06 3e+06 4e+06 5e+06

Nr steps Nr steps

Figure A Learning performance of Qlearning with dierent values for B CPU

time plotted against the number of learning steps

Table shows more detailed results It shows that Q led to the largest cumulative

reward which indicates that its learning sp eed is fastest Note that for this b est value

fast Q was more than four times faster than PWs Q

In the table we can also see that Qlearning achieved lowest cumulative reward although it

achieved almost optimal nal p erformance Most Q metho ds also achieved near optimal

p erformance An exception is fast Q which p erformed slightly worse After an analysis

we found that the learning rate decay was not sucient to keep the Qfunction always from

diverging With a larger learning rate decay Q p erformed as well as most other

Q metho ds

Note that we use a constant learning rate if we set We can use such a learning rate if the

environment is deterministic

CHAPTER REINFORCEMENT LEARNING

Although fast Q was to times faster than PWs Q for the tested values of the

dierence could b e much larger PWs cuto metho d works only when traces decay which

they did due to the chosen and parameters For worst case however PW would

consume ab out hours whereas our metho d needs around minutes

Finally we should remark that for sto chastic environments the b est value for

For this value our algorithm was ab out times faster than PWs

Exp eriments with Accumulating and Replacing Traces

The accumulate traces algorithm is still used most often although the replace traces algorithm

Singh and Sutton may work b etter for dicult problems Since Q is not exact

when combined with exploration Watkins Lin Peng and Williams we

also tested resetting to if an exploration action was selected which did not lead to

an increase of the Qvalue after a single step ie if Qlearning would increase the Qvalue of

the SAP we do not reset the trace Thus we have four metho ds for using eligibility traces

accumulatereplace withwithout resetting traces for exploration actions

Task setup We use the same mazes as in the previous chapter Again we use

dierent generated mazes We also use noise in the execution of actions we replace actions

by random actions with probability without informing the agent We can view this

noise as a change of the transition function instead of having deterministic transitions the

transition function b ecomes sto chastic The new transition function consists of probabilities

f g of making a step towards the intended direction and each of the

other directions We have used p olicy iteration PI using the a priori mo del the exact

statetransition probabilities and reward function of the maze to compute a solution We

stopp ed the p olicy evaluation once the maximum dierence b etween state values b etween

two subsequent evaluation steps was smaller than The p olicy iteration ended

once the p olicy remained unchanged after a p olicy improvement step Computed over the

dierent mazes the optimal p olicy receives a cumulative reinforcement of K K K

System Final p erformance Total p erformance Time

Qlearning K K M M

Fast Q K K M M

PW Q K K M M

Table Average results for dierent online Q methods on twenty mazes

standard deviation Final performance is the cumulative reward during the nal

steps Total performance is the cumulative reward during a simulation Time is measured in

CPU seconds

EXPERIMENTS

means within steps The standard deviation is caused by the sto chasticity of

the environment

We tested the metho ds each steps for steps Then we recorded the time step

of the rst testing trial in which the metho ds were able to collect of what the PIoptimal

p olicy collected in that particular maze A total of steps were allowed during a

run Parameters were as follows We compare values f g learning

rate the learning rate decay For accumulate traces with we used

the learning rate decay which gave the b est results we also tested other learning

rates and decay rates but these did not work b etter We use Maxrandom exploration for

all metho ds where the probability of selecting the greedy action was set to P during

max

the entire run

Accumulate Traces Replace Traces

800000 800000

600000 600000

400000 400000 Nr steps Nr steps 200000 200000 20 20 20 17 1 20 18 20 19 10 0 0 0.1 0.3 0.5 0.7 0.9 0.1 0.3 0.5 0.7 0.9

lambda lambda Accumulate Traces Reset Explore Replace Traces Reset Explore

800000 800000

600000 600000

400000 400000 Nr steps Nr steps 200000 200000 19 19 19 19 17 19 19 19 19 17 0 0 0.1 0.3 0.5 0.7 0.9 0.1 0.3 0.5 0.7 0.9

lambda lambda

Figure A comparison between dierent Qlearning methods Plots show the number

of steps before the policy collected of what the PIoptimal policy collected Values in the

boxes denote the number of successful simulations out of

Results Figure shows the results We observe that resetting traces when selecting

exploration actions is useful for making large values p erform b etter Replace traces with

resetting exploration traces achieves the most robust p erformance for the dierent values

gives the b est overall p erformance Due to the p enalty states and noise we cannot

use a large constant value for

Conclusion Managing the traces in dierent ways eects learning p erformance By

CHAPTER REINFORCEMENT LEARNING

resetting traces once exploration actions have b een executed we can use larger values which

helps to make more eective use of the eligibility traces The sp eed up gained by replac

ingresetting traces are not so large however Eligibility traces are most useful for large

environments in which the p olicy makes the same steps or receives more or less the same

rewards over time so that can b e large For complex environments some more ecient

eligibility trace management seems necessary We want to keep large but this is only

p ossible if exp eriences are not noisy since otherwise the variance of up dates gets very large

which causes large changes of the p olicy and that makes p olicy evaluation hard More e

cient managing would therefore try to reduce the noise in the TDerrors This can b e done

by using sampling techniques from a single state eg partial roll outs In this way we can

return more valuable TDup dates which allows us to send information back to many more

statesaction pairs Another p ossibility is to use variable and to detect sp ecial landmark

states which can b e used to return reliable estimates of future rewards

Multiagent Exp eriments

The last exp eriments in this chapter validate the usefulness of the novel Team Q algorithm

We use one of the mazes from the previous exp eriment and put multiple agents in this

environment which share the same Qfunction and p olicy The agents select actions one after

the other and after each action the global Qfunction is up dated If an agent hits the goal

it is reset to the starting p osition whereas the other agents keep their p osition We use

and agents and compare the Team Q algorithm to using multiple agents which only

learn from their own exp eriences Indep endent Q The Q metho d we use is replace

traces with combined with resetting traces for exploration actions

We p erform exp eriments for which we always use the same maze to keep the variance

in our results small Again we use Maxrandom exploration and keep P Our

max

ob jective is again to attain of what the PIp olicy attains

Results Table shows the results of the exp eriments We also p erformed exp eriments

with a single agent

SystemAgents

Indep endent M M M M M

Team Q M M M M

Table A A comparison between Team Q and Independent multiagent Q Results

show for dierent numbers of agents and how often a policy was learned which

obtained of the optimal policy and the total number of actions after which it was found

M stands for Mil lion A total of simulations were performed on the same maze

System

Indep endent M M M M M

Team Q M M M M

Table A A comparison between Team Q and Independent multiagent Q Results

show time requirements in CPU seconds the cumulative rewards for an entire simulation

We note that there are no signicant dierences b etween the algorithms The Team

CONCLUSION

Q algorithm attains the level slightly more often but needs on average more steps

Although Team Q learns more from agent exp eriences for the two agent case it makes

K additional connections with an average trace of and for the player case it makes

K connections which equals of all exp eriences with average trace this do es

not mean that these exp eriences are always used to improve the p olicy A reason why the

dierences are so small is that we reset traces which means that only short parts of the

tra jectories will b e connected

Table shows the time requirements of the multiagent metho ds in CPU seconds on a

Ultrasparc MHz workstation Both algorithms require additional time for multiple agents

The reason why Indep endent Q needs more time for more agents is due to the more time

demanding lo cal up date rule The computational cost of the Team Q algorithm scales up

worse The reason for this is that connecting traces consumes some time due to the lo op in

the pro cedure connect

We can see however that using multiple agents with or without connecting traces works

well if the number of agents is not to o large since the cumulative rewards obtained by the

group of agents stays ab ove M rewardpoints The sp eed up of having multiple agents

op erating in parallel is linear in the number of exp eriences for small number of agents but

the gain b ecomes smaller if the number of agents is larger than or which is caused by the

fact that goaldirected information is used later

We also used the Team Q algorithm in the deterministic maze with set

to and without resetting traces Here it turned out that Team Q did p erform slightly

worse than the indep endent version A reason could b e that initial exploration was to o large

Another problem with connecting traces is that later tra jectories are connected to

tra jectories which were generated early in the simulation This causes up dates of the value

function using old information bad tra jectories and may therefore harm the p olicy

Discussion For the maze environment Team Q do es not lead to improvements One

of its problems is that new tra jectories can b e connected to tra jectories which were generated

long ago by a completely dierent and p ossibly bad p olicy Therefore some up dates may b e

harmful To up date a sp ecic state many dierent future tra jectories are used Some of which

are created by b etter p olicies than others It is hard to select on which tra jectories we want to

learn since the inuence of noise makes it imp ossible to learn only on the b est p ossible future

tra jectories This is dierent in mo delbased RL which will b e discussed in the next chapter

since in mo delbased RL we compute a new function based on single transitions which are

always sampled from the underlying MDP In the case of connecting traces we use a complete

tra jectory which is not just sampled from the MDP but uses the p olicy as well Although

there are ways to improve the algorithm such as decaying the imp ortance of learning from

tra jectories generated a longer time ago the algorithm may in principle b e b etter suited for

tasks where the agents co op eratively have to solve a task and where one agent is help ed by

other agents in its way to reach a sp ecic goal For such tasks Team Q may backpropagate

reward to other agents which interacted with the agent in its way reaching the goal

Conclusion

We have describ ed the well known reinforcement learning metho ds temp oral dierence learn

ing and Qlearning These RL metho ds learn from generating tra jectories through stateaction

space For complex tasks in which many steps should b e p erformed TD metho ds are ef

CHAPTER REINFORCEMENT LEARNING

ciently combined with Qlearning by our novel fast online Q algorithm This metho d

implements exact up dates and in contrast to previous online metho ds its up date complex

ity is indep endent of the number of states Exp erimental results indicate that single step

lo okahead metho ds like Qlearning learn slower than Q metho ds This is esp ecially the

case in deterministic large environments for which often the same large subtra jectories are

generated Therefore the fast online Q algorithm is very useful for such problems and the

exp erimental results showed that it can signicantly sp eed up learning

The success of TD approaches relies heavily on the utility of the eligibility traces

For deterministic environments we can use large values of so that eligibility traces remain

activated for a long time so that state values are up dated using the whole future path Thus

the estimates may more quickly converge to the true exp ected results of the Markov chain

For noisy environments or environments where a lot of exploration is needed the traces are

less eective however We have seen that in particular noisy maze environments it is useful

to reset traces whenever exploration steps are made This makes it p ossible to use larger

values of We have also compared replacing traces to accumulating traces and found a

small advantage in using replacing traces This is mainly the case for large values of which

can cause very large undesirable up dates in the case of accumulating traces

We have also developed a new framework for combining eligibility traces in the case of

multiagent Qlearning The metho d makes it p ossible to adjust Qvalues of stateaction

pairs which o ccurred in the tra jectory of one agent on the future tra jectory of another agent

if there exists an interaction p oint b etween the tra jectories Exp erimental results on mazes

did not show advantages of the Team Q algorithm compared to using multiple agents

which all learn from their own tra jectories Although Team Q can use exp eriences more

eciently it has the diculty that it should nd out which previously generated tra jectories

are worth to learn from Some may b e generated by bad p olicies and can only damage the

current Qfunction

Another metho d of using eligibility traces was prop osed by Sutton He introduced

TDmetho ds for dealing with traces on dierent time scales which is a step to approaches

which learn by dealing with the future in dierent ways The decaying trace is also not

necessary we could use a variable as well Thus we could try to nd landmark states

which have small variance and let tra jectories backpropagate the values of such landmark

states However it is considered to b e quite dicult to measure variance of changing functions

without a statistical mo del Therefore we will forget ab out eligibility traces for the moment

and turn our heads in the next chapter to indirect reinforcement learning metho ds which use

exp eriences very eciently by estimating a world mo del

Chapter

Learning World Mo dels

Although world mo dels are usually not known a priori it is p ossible to learn the world mo del

from exp eriences Learning world mo dels by RL has wide applicability If we rst learn a

mo del and then compute a Qfunction we can signicantly improve the learning sp eed for

particular environments Furthermore mo dels are useful to improve exploration b ehavior to

explain causal relationships to explain why the agent prefers some choice over another and

to rep ort what the agent has b een doing most of its time

This chapter describ es how causal mo dels can b e learned by monitoring the agent in the

environment and how they can b e used to sp eed up learning These causal mo dels can b e

represented in dierent ways We will concentrate on statistical maximum likelihoo d mo dels

which return a probability distribution over successor states given the current state and

selected action These mo dels are approximations of the underlying transition and reward

functions of the MDP and can b e used by DPlike metho ds to compute Qfunctions The

mo delbased RL agent has two goals to minimize the prediction error of the mo del so

that optimal or near optimal p olicies may b e computed and to sp end most computational

resources on the most promising parts of the stateaction space to obtain high cumulative

rewards with little computational eort

Outline We will assume nite MDPs First we will explain how gathered exp eriences

are integrated into a mo del Dynamic programming DP techniques could immediately b e

applied to the estimated mo del but online DP tends to b e computationally very exp ensive

The estimated world mo del is up dated after each exp erience and fully recomputing the p olicy

is not practical for large stateaction spaces Hence we will use metho ds which can change the

p olicy much faster These metho ds direct state value up dates towards the most interesting

parts of the space The rst metho d we will prop ose uses step lo okahead Qlearning a

member of the family of Real Time Dynamic Programming Barto et al algorithms

By using the mo del Qlearning is signicantly improved but the metho d do es not eectively

use the complete mo del A second very ecient algorithm is prioritized sweeping PS

Mo ore and Atkeson which only up dates SAPs for which there will b e large up date

errors Finally we present an alternative prioritized sweeping algorithm which is quite similar

to Mo ore and Atkesons PS but manages the up dates in a dierent more exact way

Other metho ds which we will shortly describ e in the discussion are Dyna Sutton

If an agent is acting in the real world and uses a world mo del it can simulate p ossible scenarios resulting

from a particular action and use desired or undesired happ enings inside these scenarios to explain why it

prefers some action

CHAPTER LEARNING WORLD MODELS

and QueueDyna Peng and Williams These metho ds learn action mo dels by recording

frequently observed exp eriences and replay these exp eriences to sp eed up directed reinforce

ment learning These metho ds work well for deterministic environments but for sto chastic

environments they have no principled way to deal with multiple successor states Eg if only

one successor state is replayed the Qfunction will b ecome biased towards it

Extracting a Mo del

In the previous chapter we up dated the Qfunction using generated tra jectories and made

use of traces to track state dep endencies for weighing up date steps Now we want to use con

nections in the stateaction mo del for up dating the Qfunction When we store the complete

mo del and we are given a new exp erience we can propagate a change of a state value to all

other state values using the directed probabilistic graph

Given a set of examples we have to choose a mo del and compute its parameters To

dene which mo del and parameters reect the exp erimental data d b est we can use a

likelihoo d function This likelihoo d function gives us the probability P jd that our mo del

and parameters are the right one given the data We can rewrite this with Bayes rule as

P dj P jP

P jd

P d

Where P d is a normalizing constant and denotes the probability of generating the data

the complete set of generated exp eriences One reason why the likelihoo d function is of

imp ortance is given by the likelihoo d principle This tells us that assuming the mo del is

correct all the data has to tell us ab out the go o dness of parameters is contained in the

likelihoo d function Box et al

In our case we know which mo del is the right one namely a connected stateaction

transition graph consisting of transition probabilities and rewards One simple way of induc

ing these parameters from a set of exp eriences is to count the frequency of the o ccurrence

of exp erimental data which are quadruples of the form s a r s received during the

t t t t

interaction with the environment For this the agent uses the following variables

number of transitions from state i to state j after executing action a C

number of times the agent has executed action a in state i C

sum of the rewards received by the agent by executing action a in state i R

after which a transition was made to state j

The maximum likelihoo d mo del MLM consists of maximum likelihoo d estimates which max

imize the likelihoo d function We represent the MLM by matrices of transition probabilities

and matrices of rewards and estimate the parameters of these matrices by computing the

average probabilities over p ossible transitions and the average reward

P a

R i a j

C ij

MODELBASED QLEARNING

After each exp erience the variables are adjusted and then the MLM mo del is up dated It

is easy to see that these parameters maximize the probability of generating the data single

exp eriences If observations are without noise the agent always p erceives the real world

state we have a deterministic reward function and the estimated reward for a particular

transition Ri a j is known and equal to R i a j after a single exp erience For estimating

the transition probabilities we need to have multiple o ccurrences of the transition in our

exp erimental data since there are multiple outcomes in this case next states given the

o ccurrence of each stateaction pair and we want to have a go o d estimate of the probability

of each p ossible outcome To see how many exp eriences we need to compute fairly precise

mo dels we show the bias and variance of the estimates as a function of the number of times

a stateaction pair has o ccurred

Bias There is no bias This is clear since exp eriences are directly sampled from the

underlying probability distribution and we use the correct mo del consisting of transition

probability and reward matrices and so the estimator is unbiased Note also that it is a

maximum likelihoo d mo del We do not use a prior on the mo del but initialize all counters

to this has the eect that initial p olicy changes can b e very large though

Variance While up dating each transition probability changes in a sto chastic manner

As a measure of the size of up date steps we can use the variance The variance of P a

after n o ccurrences of the SAP i a is

n P a P a k

ij ij

nk k

P a P a P a V ar P ajn

ij ij ij ij

n n k

The variance go es to as n and so in the limit we have the exact estimate To give

an indication of the number of needed o ccurrences assume that P a Then we have

to try out the SAP i a times b efore the standard deviation S tdev P a For

general problems however we do not need to know all transition probabilities as accurately

but use exploration to fo cus on some parts of the state space Note also that since the p olicy

is computed from the mo del learning from new exp eriences may decrease the p erformance of

the p olicy as long as the variance is signicant Therefore mo delbased learning is in principle

a sto chastic approximation algorithm

Mo delBased Qlearning

The MLM can b e combined with the DPalgorithms from Section to compute a p olicy

Although the resulting p olicy may make optimal use of the exp eriences generated so far it

usually costs a lot of computational time to use DPalgorithms in an online setting esp ecially

when the statespace is quite large A more ecient way to recompute the p olicy introduces

some bias to choose which computations really need to b e p erformed One simple but quite

eective metho d is using onestep Qlearning on the mo del which is the simplest version in

the family of adaptive Real Time Dynamic Programming metho ds Barto et al This

metho d up dates the Qvalue for the current stateaction pair after it has b een executed and

the mo del has b een up dated by

Note that they do not maximize the probability of generating the complete statetra jectories since many

new spurious tra jectories b ecome p ossible

CHAPTER LEARNING WORLD MODELS

P aR i a j V j Qi a

This up date rule makes more ecient use of the exp eriences than standard Qlearning We

can see this as follows For Qlearning a particular up date lets say of Qi a which in our

case equals V i will only b e used to adapt other Qvalues if afterwards there o ccurs another

stateaction pair j b leading to state i For mo delbased learning up dates are already used

if the mo del contains a transition from a stateaction pair j b to state i This transition

do es not have to reo ccur A new o ccurrence of j b will automatically take V is up date

into account Therefore it may need signicant fewer exp eriences although single up date

steps are computationally more exp ensive the up date complexity dep ends on the number

of p ossible outgoing transitions Note however that for deterministic environments b oth

metho ds are identical

Variance of Mo delbased Q vs Q We will now verify whether mo delbased Qlearning

also reduces the variance in the up dates The exp ected size of the up date of Qi a after its

n o ccurrence with mo delbased Qlearning is

X X

C a C a

R i a j V j R i a k V k Qi a P a

n n

k j

If all the neighbors have not b een up dated in the meanwhile this can b e simplied according

X X

C a C a

R i a j V j R i a k V k P a

n n

k j

C a C a

R i a j V j R i a k V k

n n

k j

X X

n C a C a

P a R i a j V j R i a k V k

nn nn

k j

X X

P a P a

R i a j V j R i a k V k P a

n n

k j

R i a j V j Qi a P a

the variance is For Qlearning where the learning rate anneals as

R i a j V j Qi a P a

Thus in case of no up dates of any neighbors state value the variances are equal If some

neighbors have had their values up dated it dep ends whether their values moved closer to the

previous value of Qi a or not Usually in the initial phase all values in the same region

of state space are increasing or decreasing together For such environments mo delbased

learning is exp ected to make larger up date steps and thus will have larger variance Note

however that Qlearning usually do es not anneal the learning rate with and thus may have

a much larger variance

PRIORITIZED SWEEPING

Prioritized Sweeping

We want to use the directed probabilistic graph to eciently propagate current statevalue

up dates to other statevalues Although mo delbased Qlearning is a fast way for using

the mo del it only p erforms a onestep lo okahead which makes information passing b etween

distant states slow When distant states are connected by high probability tra jectories in

a state space a change in the value of the state at the end of the tra jectory will cause a

corresp onding change in the other state Since mo delbased Qlearning is not able to do such

deep information passing quickly it may need much more exp eriences than DP algorithms

which will execute sucient up dates to propagate the change of a states value to all other

states Although this always keeps the value function uptodate DP p erforms so many

up dates for each exp erience that it b ecomes very slow To sp eed up dynamic programming

algorithms some management of up datesteps should b e p erformed so that only the most

useful up dates are made To nd out which up dates are imp ortant we consider an exp erience

as a newsash it brings information to a sp ecic region which is in some way connected to

it but it do es not need to b e known everywhere unless it is really big news

An ecient management metho d which determines which up dates have to b e p erformed is

prioritized sweeping Mo ore and Atkeson This metho d assigns priorities to up dating

the Qvalues of dierent states according to a heuristic estimate of the size of the Qvalues

up dates The algorithm keeps track of a backward mo del which relates states to predecessor

stateaction pairs After some up date of a state value the predecessors of this state are

inserted in a priority queue Then the priority queue is used for up dating the Qvalues for

the actions of those states which have the highest priority For the exp eriments we will use

a priority queue for which a promoteremove op eration takes Olog n with n the number

of states in the priority queue Van Emde Boas Kaas and Zijlstra describ e a more

ecient priority queue which only needs Olog log n for a promoteremove op eration but

which only handles integer valued priorities in the interval n

Mo ore and Atkesons PS uses a set of lists P r edsj where the list P r edsi contains all

predecessor stateaction pairs j a of a state i The priority of state i is stored in i

When state i has its value up dated the transition from j a to i contributes to the up date

of Qj a The priority of a state j is the maximal value of such contributions The algorithm

is shown on the next page The parameter U denotes the maximal number of up dates

max

which is allowed to b e p erformed p er up datesweep The parameter controls the up date

accuracy The algorithm pushes the current stateaction pair on the top of the queue

then it rep eats for all queue items Remove the top state step up date it steps

and store the size of the up date in the variable D step reset its priority step

assign new priorities to all predecessors of the state steps and substeps and insert them

if their new priority is larger than the threshold step

The op erations are also called insert delete usually the top element and heapify which means reinsert

an element once it has b een assigned a new priority

CHAPTER LEARNING WORLD MODELS

Mo ore and Atkesons Prioritized Sweeping

Promote the most recent state to the top of the priority queue

While n U AND the priority queue is not empty

max

Remove the top state i from the priority queue

a do

P aR i a j V j Qi a

V i max Qi a

D jV i V ij

V i V i

j a P r edsi do

P P aD

j i

If P j

j P

If P

Promote j to new priority j

n n

Our Prioritized Sweeping Our implementation of the algorithm uses a set of prede

cessor lists containing all predecessor states of a state jij denotes the priority of state

i This priority is at all times equal to the true size of the up date of value V i since the

last time it was pro cessed by the priority queue To calculate this we constantly up date all

Qvalues of predecessor states of currently pro cessed states and track changes of V i Our

PS lo oks as follows

Our Prioritized Sweeping

Update the most recent state s a do

P aR s a j V j Qs a

Promote the most recent state to the top of the priority queue

While n U AND the priority queue is not empty

max

Remove the top state s from the priority queue

Predecessor states i of s do

V i V i

a do

P aR i a j V j Qi a

V i max Qi a

i i V i V i

If jij

Promote i to priority jij

n n

Make priority queue empty but keep the i values

In Mo ore and Atkesons PS MAs PS states are inserted in the priority queue b efore

their Qvalues are up dated In our metho d Qvalues of states are up dated b efore the states

are inserted We do this to have an exact estimate of the up date of the state value since it

EXPERIMENTS

has b een used for up dating the priority of its predecessors This makes the priorities exact

whereas MAs PS calculates priorities based up on the largest single up date of a successor

state This means that states are only inserted if at some sp ecic time step their successor

state makes a large up date step larger than If the up datesteps of successor states are

smaller but more frequent it may happ en that the states are never inserted in the priority

queue although the up date could b e quite large

Our version uses priority values of states which are calculated by summing the new change

of the state value with the old change i in step This has the advantage of b eing

exact but the disadvantage is that in our metho d Qvalues of all actions for a particular

predecessor state have to b e up dated which is more time consuming

Clearing the priority queue Another choice is whether the priority queue should b e

emptied or not after the up date sweep PScall In the algorithm shown ab ove our metho d

empties the queue after the PScall step while keeping the values of all states Mo ore

and Atkesons PS do es not empty the queue The decision whether to empty the queue or not

is indep endent of the selected metho d Keeping the queue results in a quite dierent up date

sweep if for example at some time step the size of the up date of the current state value

is small The states still on the queue are then pro cessed These states have probably b een

inserted few time steps ago but it is also p ossible that they were inserted while the agent

visited dierent parts of the state space Up dating them costs time and since the up dates

did not b elong to the most imp ortant U up dates at the moment they were inserted in the

max

queue we can argue that we do not want to up date them at all Note that our metho d keeps

the priority values even when items are not in the priority queue Then if their priority is

large enough they will b e inserted in the future anyway b eing predecessor of some other

visited state

Exp eriments

We have evaluated the mo delbased RL metho ds describ ed in this chapter by p erforming a

series of maze exp eriments The rst exp eriments involve again the same mazes

used in Chapters and In all exp eriments selected actions are replaced by random actions

with probability We have used p olicy iteration using the a priori mo del to compute a

solution We stopp ed the p olicy evaluation once the maximum dierence b etween state values

b etween two subsequent evaluation steps was smaller than Policy iteration needed

a total of sweeps for computing the p olicies This means that ab out million

value up date steps were p erformed We will call this p olicy the optimal p olicy although it is

not guaranteed to b e the optimal p olicy due to the use of the cuto parameter see Chapter

Stopping the evaluation prematurely sp eeds up the pro cess signicantly however

The optimal solutionpaths cost on average a bit more than steps The optimal p olicy

receives a cumulative reinforcement of K K K means within steps

Within steps the cumulative reward is K K

This could b e overcome by making small enough although this would result in inserting many more

unimp ortant states as well

CHAPTER LEARNING WORLD MODELS

Comparison b etween Mo delbased and Mo delfree RL

We compare Qlearning Qlearning mo delbased Qlearning and our prioritized sweep

ing For all Q exp eriments we have used the fast online Q algorithm presented in

Section

Exp erimental setup To compare the dierent RL metho ds we executed them on

dierent mazes For all metho ds we used the Maxrandom exploration rule where we linearly

increased P from to P Thus in the b eginning the agent selects

max max

the action with maximum Qvalue with probability and at the end of the simulation it

always greedily selects the action with largest Qvalue We ran actions during a

trial and recorded the cumulative rewards during intervals of training actions After

each trials we also p erformed test trials to compute the number of steps to reach the

goal Finally we recorded the approximation of the learned value function by calculating the

average distance of state values to the optimal V function computed with p olicy iteration

Parameters For Qlearning we used and For Qlearning we use

the same learning rate parameters and set For mo delbased Qlearning there are no

additional parameters to set We have used our version of the prioritized sweeping metho d

with clearing the queue For PS the maximum number of up dates U and

max

Results Figure A plots the learning p erformance of the dierent metho ds It clearly

shows the advantage of using mo delbased RL Both mo delbased approaches converge much

faster the linear increase of cumulative reward is only due to the decreasing amount of ex

ploration sto chasticity of the tested p olicy and reach much b etter nal p erformance The

prioritized sweeping metho d needs the fewest exp eriences and reaches nearoptimal p erfor

mance levels record that the optimal p olicy collects K K p oints Q improves

its p erformance faster than Qlearning although the nal p erformances are equal

System Qlearning Q mo delbased Q PS

Performance K K K K K K K K

Time

Time for Solution

Table Final performance levels average time needed by the dierent methods for a sin

gle simulation steps and the time needed to reach the interval of the highest

performance

Table shows the nal p erformances and time costs of the metho ds Time costs were

computed by measuring CPU time for the entire simulation including almost neglectable

simulator time on a MHz Sun SPARC station Note that the mo delbased approaches

sp end more time their up daterules lo op over all outgoing transitions They could already

b e stopp ed much earlier however and the last row of the table indicates how much time it

costs until the p erformance reaches the interval of the linear increase in the graphs which is

due to decaying exploration Thus we can see that our prioritized sweeping nds the b est

solution and nds this fastest Prioritized sweeping only consumes more time during the rst

trials When the goal state has b een found for the rst time many up dates are made After

the goal has b een found a couple of times and most states have b een visited only few up dates

are made after each exp erience most up dates would b e smaller than

Approximation error Figure B shows the approximation errors of the dierent

learning algorithms It is clear that there is not sucient exploration to learn go o d value

EXPERIMENTS

100000 800 Q-learning Q-learning Q(0.5)-learning Q(0.5)-learning Model-based Q 700 Model-based Q 80000 Prioritized Sweeping Prioritized Sweeping

600 60000

500 40000

400

20000

Approximation error 300

0 Cumulative reward per 10000 steps 200

-20000 100

-40000 0 0 250000 500000 750000 1e+06 0 250000 500000 750000 1e+06

#steps #steps 1e+06 Q-learning Q(0.5)-learning Model-based Q Prioritized Sweeping

100000

10000

1000 number of steps in trial

100

10 0 2000 4000 6000 8000

trial number

Figure A A comparison between the learning performance of the RL methods The

plot shows averages over simulations B The approximation error of the RL methods

The gure plots the average distance of state values to their optimal values against number of

agent steps C Average number of steps per testtrial Note the logarithmic scaling of the

yaxis The peaks in the modelfree approaches indicate unsuccessful and long trials

function approximations Only PS learns a reasonable value function approximation The

direct mo delfree RL metho ds initially increase the approximation error since they learn

negative state values After this they hardly improve the value function approximation It

may seem surprising that the metho ds were able to learn reasonable p olicies at all with

such bad value function approximations This is exactly why real time RL metho ds work

p olicies only learn to approximate values of states which they frequently visit Thus we

conclude that approximating the p erformance of the optimal p olicy tends to b e much easier

than approximating the optimal value function

Test trial results We have p erformed test trials after each learning trials We used

the greedy p olicy and measured the number of steps needed to reach the goal The trials

CHAPTER LEARNING WORLD MODELS

were stopp ed if the agent could not nd the goal within steps In Figure C the

average number of steps p er test trial is displayed The p eaks in the mo delfree approaches

indicate unsuccessful or long trials The Q agent did not nd the goal in mazes In

of these unsuccessful simulations the goal had b een found a couple of times but the agent

made some up date step which ruined its p olicy Hence it could not reach the goal anymore

from trial onwards the p olicy must have b een caught in cycles through a number

of states from which it could not escap e even though actions were selectedexecuted non

deterministically The Qlearning agent failed for only maze although its average solutions

for the other mazes were worse than for Q which we can verify by considering that the

average results are the same

Comment We also used value iteration to compute the value function while estimating

a maximum likelihoo d mo del As exp ected online VI is computationally very demanding for

simulations consisting of steps it required CPU seconds using Its

p erformance was also not b etter than that of PS

More noise We also tested the eect of more noise in the execution of actions in the

same mazes For this we p erformed simulations using dierent mazes and noise in

the execution of actions For Q the b est exploration rate for the Maxrandom exploration

rule is as follows P Figure A shows the results For Q accumulate

max

traces with the value for worked b est Whereas the p erformance on the same mazes

with noise was o from maximum p erformance with noise the b est metho d is

o Thus the p erformance of Qlearning is quite sensitive to noise

Figure B shows the results of mo delbased Qlearning with Maxrandom exploration

with dierent exploration rates indicated as P where P equals P and with

exp exp max

an exploration metho d which initializes the Qvalues to Mo delbased Qlearning do es

not seem to have any problems with the increased amount of noise It reached stable p erfor

mance after K steps and the nal p erformance is close to optimal The b est p erformance

of K is only o optimal

To study how well prioritized sweeping deals with more noise we set up exp eriments with

our PS metho d We use simulations during which K steps can b e made We used

and maxup dates Figure C show the results exploration rates are indicated as

P and are annealed to It indicates that PS works very well although the initial

max

value of P P should not b e to o small The b est result is attained with

exp max

P which achieves a nal result of K which is o the optimal p erformance

max

K We exp ect slightly worse but in overall similar results for MAs PS

When we compare the three plots we clearly see that mo delbased approaches work much

b etter This is due to the fact that they are much less sensitive to noise than direct RL

approaches Finally we note that PS reaches its nearoptimal p erformance levels much faster

ab out times than mo delbased Qlearning

Discussion Mo delfree approaches work worse for sto chastic environments since tra jec

tories generated in dierent trials are not combined in an ecient way Have a lo ok at Figure

where we have designed two paths from the start S to goal G The solid line is the

b est known path and the dashed line is the current path We want to combine the two paths

to learn to go from S to A using the dashed line and from A to G using the solid line Given

b oth lines mo delbased RL would easily learn this Qmetho ds however cannot learn

this in a single pass The problem is that for the states just b efore state A Qlearning

Although learning from many trials solves the problem single trials should make much larger improve

EXPERIMENTS

Replace traces/ Accumulate Traces 25% noise Model based Q-learning 25% Noise 100000 100000 lambda = 0.1 Max-random 0.1 - 0.0 lambda = 0.4 Max-random 0.3 - 0.0 lambda = 0.7 Max-random 0.5 - 0.0 80000 Replace lambda = 0.1 80000 Max-random 0.7 - 0.0 Replace lambda = 0.4 Start 10000 Replace lambda = 0.7

60000 60000

40000 40000

20000 20000

reinforcement per 10000 steps 0 0 Cumulative reward per 10000 steps

-20000 -20000

-40000 -40000 0 250000 500000 750000 1e+06 0 250000 500000 750000 1e+06

#steps #steps Prioritized sweeping 25% noise 20000 Max-random p: 0.1 Max-random p: 0.3 Max-random p: 0.5 Max-random p: 0.7 15000 Max-random p: 1.0

10000

5000

0 Cumulative reward per 2000 steps

-5000

-10000 0 25000 50000 75000 100000

#steps

Figure A A comparison between dierent values for and the accumulate and replace

traces algorithms for Qlearning on twenty mazes with noise in the execution

of actions B A comparison between dierent amounts of exploration for modelbased Q

learning C Dierent amounts of exploration with prioritized sweeping note the dierent

scalings of the X and Yaxis

partially learns from As state value and partially from the tra jectory shown by the dashed

line after A When is large the TDreturn from A will reach p oint S with a lower value

than the Qvalue of the action leading to the solid path since the complete dashed path is

worse than the solid path Eg for we would compare at each state the entire future

solid path to the dashed path and no changes would b e made at all If is small the value

from A would not b e returned to S Prioritized sweeping do es not suer from this problem

ment steps

Resetting at A would solve the problem but this is dicult for online approaches since we cannot b e

certain b efore the goal is reached whether we should learn from the current trial or not If we choose to learn

the up date steps cannot simply b e undone

CHAPTER LEARNING WORLD MODELS

Figure The diculty for Qlearning to improve The solid line is the current best

known path and the dashed line is the current path consisting of a good part from start S to

point A and a bad part from A to goal G

The mo delbased approaches nd near optimal p erformance levels and are able to learn from

less exp eriences Prioritized sweeping achieves excellent p erformance and is able to quickly

solve large mazes

Prioritized Sweeping Sensitivity Analysis

In this subsection we shortly describ e parameter setting for prioritized sweeping PS Two

parameters have to b e set and we have analyzed the sensitivity of the learning b ehavior for

these parameters

Setting the accuracy parameter

The parameter controls the level of up date accuracy By setting to larger values less up

dates have to b e p erformed each time step so that the up date sp eed is higher Of course when

is set to very large values the value function will b e badly tted to the exp eriences and this will

result in p erformance loss We tried out values for out of the set f g

We found that the time requirements dep ends a lot on the values of where values of

or larger result in very fast p erformance b elow seconds p er simulation of actions

with Maxrandom exploration Surprisingly it is not true that a higher accuracy lower

always results in b etter p erformance this may result in overtting the value function

This overtting happ ens when the mo del is not very accurate but is used to compute an

accurate value function If we only have partial data it can happ en that some distant states

are connected through highprobability paths which do not reect the real MDP very well

Therefore if we have partial data and make to o many up date steps the value function can

b ecome very biased Therefore we should use more exploration The values and for

EXPERIMENTS

worked b est in b oth p erformance and time requirements

Tuning the maximal number of up dates

Each up date sweep makes a number of up dates In order to increase the up datesp eed we

have limited the maximal number U of up dates which are allowed to b e made p er up date

max

sweep Computational costs are less sensitive to the maximal number of up dates than to

the accuracy parameter however We may again overt the p olicy to a wrong mo del

when we increase the number of up dates the p erformance do es not always improve but may

b ecome worse If we up date less we will up date relatively more on nearby lo cated states

and thus we will less likely up date on a sp ecic erroneously estimated tra jectory which may

introduce a large bias in the Qestimates

We found that using up dates resulted in the b est p erformance for For larger

problems having more up dates may b e necessary to keep the states connected we may

want to assure that each up date could result in an up date of any other state lying on the

current shortest path

Comparison b etween PS Metho ds

Our PS metho d is more selective in calculating the priorities since it p erforms a full parallel

backup for determining priorities Mo ore and Atkesons PS considers only one single output

transition at a sp ecic timestep for computing the priority We have set up exp eriments

to compare our PS metho d to Mo ore and Atkesons PS We test the p olicies each

steps for steps and record when they collect and of what the optimal PI

p olicy collects One simulation ends when actions have b een executed We combine

all metho ds with Maxrandom exploration where P stays during the run We also

max

measure the required CPU time in seconds on a Ultrasparc MHz Sun station For these

exp eriments and U

max

PS Clear Q steps to steps to Final Reward time

Our Yes K K K K K K

Our No K K K K K K

MA Yes K K K K K K

MA No K K K K K K

Table Required number of actions for PS methods to nd policies for the mazes

which are and o the maximal obtainable cumulative reward Between brackets we

show the number of simulations in which they succeeded

Results The results are shown in Table For nding optimal p olicies the

dierences are not very large However unlike MAs PS our PS with clearing the queue

found optimal p olicies in all mazes Furthermore this metho d also reaches the b est

nal p erformance We do not show here that the cumulative rewards for all metho ds are

more or less the same although there is a small advantage for not clearing the queue This

contradicts the nal results the cumulative reward over the nal K steps which shows

b etter results for clearing the queue The dierence can b e explained by the fact that not

clearing the queue may sp eed up initial learning since more up dates are p erformed but as

seen b efore many early up dates may b e harmful since it may lead to more initial bias in the

value function

CHAPTER LEARNING WORLD MODELS

Conclusion Our PS metho d prots from exactness and therefore it will always work

well We exp ect that our PS metho d is currently the b est mo delbased RL metho d for small

to mediumsized discrete MDPs

Discussion

Problems of mo delbased RL One problem of the metho ds is that computational memory

requirements can b e huge in the worst case we have to store a full mo del using OjS j jAj

space On current machines this b ecomes infeasible for jS j larger than However for

large problems not all transitions are exp erienced in the limited lifetime of the agent so that

only partial mo dels need to b e stored However sometimes it is a big problem Take for

example the game of backgammon For some states there may b e many successor

states b ecause there are many p ossible actions For such problems we should only store state

transitions of actions which are frequently chosen by the p olicy or group transitions together

if they lead to states with similar values Of course this results in some loss of accuracy

Statistical world mo dels In this chapter we used statistical world mo dels The use

of statistical metho ds makes it p ossible to estimate mo dels which can b e used with dynamic

programming like algorithms Using statistical learning theory Kearns and Singh

proved that a variant of a mo delbased reinforcement learning metho d converges in the

probably approximately correct PAC framework That means that we probably succeed in

learning an approximately correct value function after learning on a number of exp eriences

which is b ounded by the logarithm of the probability of success a p olynomial of the size of

the stateaction space and the inverse of the approximation error

Other world mo dels include functional mo dels consisting of a set of dierential equations

which are used in many sciences eg in system dynamics Vennix Such mo dels make

cost assessment much harder however since we need to employ simulation techniques for

predicting the results of actions The advantage of such mo dels is that they can b e quite

compact and based on physical prop erties of the problem However there is no systematic

way of learning such mo dels although evolutionary metho ds Holland Rechenberg

could b e used there is no guarantee that these metho ds will nd a correct mo del

Other world mo dels Suttons Dyna systems and relatives Sutton Peng and

Williams do not estimate a probabilistic transition function but store tra jectories

consisting of sequences of exp eriences During the course of learning the system may choose to

learn from previously stored exp eriences instead of making real world steps This is useful for

learning in real worlds for which generating new exp eriences is often much more exp ensive than

replaying old exp eriences For deterministic environments these systems work very well

They may b e less suited for sto chastic environments however since up dates are less informed

and biased to the exp eriences which are replayed Instead the PS metho ds manage the up date

sequences up dates are unbiased and up dates are more informed An advantage of the Dyna

approach is that it is easy to replay exp eriences for all kinds of function approximators

Jordan and Jacobs and similarly Nguyen and Widrow describ e an approach

based on two mo dels one neural network the world mo del tries to predict the successor

state and another the action mo del is used for selecting actions Given some discrepancy

b etween the desired state and a predicted state errors are backpropagated from the world

mo del to the action mo del so that an action is selected which minimizes that discrepancy

Since it is a global mo del however only single successor states can b e learned but it can

CONCLUSION

generalize well Schmidhuber do es the same kind of forward mo deling with recurrent

mo dels and controllers

Lin describ es exp erience replay and action mo dels for TDlearning with feedforward

networks For exp erience replay he records exp eriences and replays them in backward order

He only records p olicy actions and not exploration actions since exp eriences generated by

exploration actions do not reect the true dynamics generated by the p olicy Exp erience

replay was compared to learning action mo dels For learning action mo dels he learned to

predict the most likely reward and successor state with a feedforward neural network He

found that using exp erience replay led to larger sp eedups than learning an action mo del He

did not estimate a probability distribution over successor states in his action mo del however

This would also b e very dicult b ecause he used a partially observable changing environment

for which there were hundreds of p ossible successor states for each state Feedforward neural

networks are not well suited for learning such mo dels due to their static architecture which

makes it dicult to store transition probabilities of a variable number of transitions

Planning Planning metho ds build explicit search trees starting at the goal state as ro ot

note backward planning or at the start state forward planning in order to connect the

goal and start states Most planning mechanisms use heuristic functions and build search

trees of a limited length after which they select the step leading to the leave no de with the

largest value Davies Ng and Mo ore discuss online search metho ds for improving

a sub optimal value function and showed some improvements in p erformance at the cost of

consuming more time After the planning phase they did not adapt the value function

however It is also p ossible to use the planning phase for up dating the values of states which

were traversed during the planning pro cess This can b e simply done by adapting state values

to match the results of the planning pro cess a similar thing is done in Baxter et al

which combine planning with TD The problem of planning is that it is computationally

exp ensive esp ecially for nondeterministic environments We also tried forward planning

together with adjusting the value function for the planning steps but found that PS makes

it easier to backpropagate goalrelated information Once the goal is found PS makes many

up dates so that the agent immediately stores tra jectories to the goal state from many other

states For planning up dates are only made if the planning pro cess was successful and thus

if the goal is found by accident the planning metho d will not learn how it got there We

may use backward planning instead but then we already need to know where the goal is and

since the environment is sto chastic it is not clear how we should compute tra jectories linking

the goal state to the current state

Conclusion

This chapter describ ed RL metho ds which estimate a maximum likelihoo d mo del and compute

the Qfunction using the mo del We used exp eriments to compare dierent RL metho ds

describ ed so far We have seen that mo delbased approaches outp erform the mo delfree RL

approaches in nite sto chastic mazes They make more ecient use of exp eriences Mo del

based approaches can prot from management techniques to keep computational requirements

for making p olicy changes small Prioritized sweepings management rule Focus on the largest

errors problems is very eective

We also showed that the mo delbased approaches were very resistant against noise Using

a lot of noise in executing actions hardly inuenced the nal results whereas this was not true

CHAPTER LEARNING WORLD MODELS

at all for direct RL metho ds which suered a lot from more noise We consider our mazes

to b e a prototype of goaldirected sto chastic nite MDPs where the number of paths leading

to the goal state the number of p ossible choices at each step the amount of noise in the

transition function and the length of the path to the goal are the most imp ortant parameters

which determine the problems complexity We think that for such problems mo delbased

RL metho ds will always outp erform direct approaches The dierence will b e larger if the

noise in the transition function is larger and the length of the path to the goal is larger

Mo delbased techniques rely more on the Markov prop erty than Q metho ds however

which can learn p olicies using Monte Carlo estimates for which the Markov prop erty is not required

Chapter

Exploration

A reinforcement learning agent only learns from what it exp eriences and therefore if it always

sees the same world states since it always makes the same decisions it will not increase its

knowledge or p erformance Only when an agent would b e following an optimal p olicy the

agent do es not need to explore Otherwise there is always some need to select actions which

lo ok sub optimal to the agent Such actions which deviate from what the agent b elieves is

b est the agents greedy actions are called exploration actions Even though it is true that

most exploring actions are indeed not optimal esp ecially not if the agent already has acquired

a lot of knowledge exploring is always helpful if the agent has still a long future changing

something now may b e costly but will b e b enecial for a much longer time and thus it will

slowly pay back its costs

Optimal exp erimental design Fedorov Do dge et al and active learning

Cohn try to gather those exp eriences data which are most useful for computing

go o d approximative solutions In reinforcement learning the problem of selecting exploration

actions is called exploration or dual control Dayan and Hinton Deviating from the

current greedy p olicy which always selects the action with the highest Qvalue however

usually causes some loss of immediate reinforcement intake This is usually referred to as the

explorationexploitation dilemma Therefore in limited life scenarios the agent usually tries

to maximize its cumulative reward over time and it faces the problem of trying to sp end as

little time as p ossible on exploration while still b eing able to nd a highly rewarding p olicy

Schmidhuber a Schmidhuber Another goal however could b e to nd the b est

p ossible p olicy in a xed time or to nd a near optimal p olicy in least time For such problems

we only care ab out exploration and thus exploitation can b e discarded completely

Previous work Thrun presents comparisons b etween dierent directed and

undirected exploration metho ds Directed exploration metho ds use sp ecial exploration sp ecic

knowledge to guide the search through alternative p olicies Undirected exploration metho ds

use randomized action selection metho ds to try their luck in generating useful novel exp eri

ences Previous research has shown signicant b enets for using directed exploration see

eg Schmidhuber Thrun and Storck Ho chreiter and Schmidhuber Ko enig

and Simmons show how undirected exploration techniques can b e improved by using

the so called actionp enalty rule This rule p enalizes actions which have b een selected in

particular states so that the unexplored actions for some state lo ok more promising this

decreases the advantage of directed exploration

When we want to solve the explorationexploitation dilemma it is also necessary to

CHAPTER EXPLORATION

switch b etween exploration and exploitation Thrun and Moller use a comp etence

map which predicts the controllers accuracy and their bistable system switches attention

b etween exploration and exploitation dep ending on exp ected cost and knowledge gain

Exploration is also very imp ortant for dealing with nonstationary dynamic environ

ments Eg supp ose that we have learned a p olicy for going from one ro om to another Then

if a do or b etween these ro oms had always b een closed the p olicy could not have learned to

pass through this do or If the do or is op ened afterwards the agent should nd this out in

order to relearn a b etter p olicy for reaching the other ro om Dayan and Sejnowskis dual

control algorithm is designed for such environments They slowly increase Qvalues

so that actions which have not b een tried out for some time will b e selected again

A completely dierent exploration approach is describ ed in Schmidhuber where

one agent gets reward if she is able to bring the other agent to regions which he do es not

know Thus a coevolutive exploration b ehavior takes places where the agent is presented

with a continuous stream of unknown situations until it nally learned a p olicy for all of

them

The Interval Estimation IE algorithm Kaelbling uses second order statistics to

detect whether certain actions have a p otential of b elonging to the optimal p olicy IE com

putes condence intervals of Qvalues and always selects the action with largest upp er interval

b oundary Previous results Kaelbling show that IE works well for action selection in

bandit problems Berry and Fristedt N armed bandit problems are problems in which

we can select b etween n arms each one with dierent payo probabilities and payos Thus

we have to collect statistics in order to infer which arm to pull Since pulling arms costs

money as well and we only have limited time we want to gain information ab out the payo

ratios of the arms and infer when we can b est fo cus on the most promising one Since we are

dealing with sto chastic problems IE is interesting for us and we will describ e how we can

combine IE with mo delbased RL

Outline of this chapter We rst describ e undirected exploration metho ds in Section

In Section we describ e directed exploration metho ds which are based on the design of

an exploration reward function In Section we describ e mo delbased exploration and a new

exploration metho d which extends interval estimation IE to mo delbased RL In Section

we present exp erimental comparisons b etween indirectdirect exploration metho ds Section

concludes this chapter with a discussion

Undirected Exploration

In this section we will summarize undirected exploration techniques These metho ds usually

rely on pseudorandom generators

Maxrandom Exploration Rule

The Maxrandom also known as pseudosto chastic Caironi and Dorigo and greedy

Sutton exploration rule is the simplest exploration rule The rule uses a single

parameter P which denotes the probability of selecting the action with highest Qvalue

max

generate a number r from the uniform distribution

If r P max

UNDIRECTED EXPLORATION

select the action with highest Qvalue

Else select a random action a fA A A g

For step it can happ en that there are multiple actions which have the highest Q

value In this case we select one of them sto chastically The Maxrandom rule has the nice

prop erty that the time overhead for selecting an action can b e small in the optimal case

where the maxaction stays the same it is O A go o d metho d for learning p olicies is

to linearly increase P after each step during the trial in the b eginning of a simulation a

max

lot of exploration makes it p ossible to compare many alternative p olicies and at the end the

probability of selecting greedy actions b ecomes Thus exploration b ecomes more and

more fo cused around the b est found p olicy

Boltzmann Exploration Rule

The BoltzmannGibbs rule is one of the most widely used exploration rules The Boltzmann

rule assigns probabilities to actions according to their Qvalues It uses a temp erature variable

T which is used for annealing the amount of exploration The Boltzmann exploration rule

computes the probability P ajs for selecting an action a given state s and Qvalues Qs i

for all i A as follows

QsaT

P ajs

QsiT

The Boltzmann rule causes a lot of exploration in states where Qvalues for dierent

actions are almost equal and little exploration in states where Qvalues are very dierent

This is helpful for risk minimization purp oses Heger for which we may prefer not

to explore actions which lo ok signicantly worse than others However initially learned Q

estimates can b e incorrect due to noise so some exploration is still needed in case of large

dierences in Qvalues Using an annealing schedule for the temp erature should get around

this problem but nding a go o d annealing schedule can b e quite dicult and is reward

function dep endent Furthermore since the temp erature is a global variable it may happ en

that some state is not visited for a long time after which the amount of exploration in

this state is very small Although this last problem could b e solved using lo cal temp erature

variables it is still a problem that the agent may concentrate on dierent tra jectories which

it already knows well

MaxBoltzmann Exploration Rule

Maxrandom exploration may lead to bad results when particular actions lead to large negative

rewards It assigns equal probabilities to all nonoptimal actions and therefore assigns to o

large probabilities to explore really bad actions Boltzmann exploration has large problems

fo cusing on the b est actions while still b eing able to sometimes deviate from them We may

also combine the Maxrandom and Boltzmann exploration rules The maxBoltzmann rule

combines taking the Max action with the Boltzmann distribution

The action with maximal Qvalue can b e stored when learning the Qvalues

If we change the reward function by eg scaling all rewards we would interfere with the exploration

b ehavior This would not b e the case for Maxrandom exploration

CHAPTER EXPLORATION

generate a number r from the uniform distribution

If r P

max

select the action with highest Qvalue

Else select an action according to the probabilities computed by Eq

Although the rule requires more parameters to set the temp erature T and P there will

max

in general b e more go o d combinations We will use this rule for exploration with ambiguous

inputs in Chapter since there it is imp ortant to b e almost deterministic for inputs which

are not ambiguous and to fo cus on a particular action for an ambiguous input while still

sometimes exploring go o d alternative actions

Initialize High Exploration

Another p ossible exploration rule is to start with high initial Qvalues the Qfunction is

initialized to the upp erb ound of the optimal Qfunction and to use the normal RL up date

rule Given the fact that action values will drop when they have b een explored taking the

action with maximal Qvalue will initially return that action which has b een selected least

times The metho d is quite similar to counting the number of times particular stateaction

pairs have o ccurred although it may happ en that some actions which have o ccurred more

often are still preferred ab ove other actions since they are more rewarding

Dierent kinds of exploration techniques based on this idea have b een used used in eg

Ko enig and Simmons Prescott Ko enig and Simmons Ko enig

presents a pro of that nding a goal at all in particular environments by initializing high and

p enalizing actions will only take p olynomial time whereas Whitehead had shown

that zerorewarding actions may take exp onential time for such worstcase environments An

example of such an environment consists of a simple sequence of states and two actions go

left and go right and where the action go right brings the agent to the successor state and

the action go left brings the agent all the way back to the initial leftmost state To nd the

goal the agent has to select only go right actions However if we do not p enalize actions

the probability of selecting a long sequence with only go right actions decreases exp onentially

with the length of the path Thus we have to record which actions have o ccurred to nd the

goal in p olynomial time in the number of states

Directed Exploration

For directed exploration all we need to do is to create an exploration reward function which

assigns rewards to trying out particular exp eriences In this way it determines which ex

p erience is interesting for gaining information Storck et al Directed exploration

metho ds learn an exploration value function in the same way standard RL metho ds learn

a problemoriented value function Therefore we may simply dene an exploration reward

function determining immediate exploration rewards and let the selected RL metho d learn ex

ploration Qvalues We can use the same RL metho d for learning the exploration Qfunction

and the exploitation Qfunction

If we want to solve the explorationexploitation problem we have to nd a metho d to

choose which of the two Qfunctions to use for action selection There are dierent heuristics

for switching from exploration to exploitation We note that an optimal switching strategy

DIRECTED EXPLORATION

is in general hard to compute except for the most simple problems such as armed bandit

problems Gittins Kaelbling et al

Reward Function Frequency Based

Explore actions which have b een executed least frequently The lo cal reward function is

simply

C a

R s a

Here R s a is the exploration reward assigned to selecting action a in state s It is

determined by dividing the lo cal counter C a by a scaling constant K The asterisk stands

s C

for the dontcare symbol In general the resulting p olicy will try to explore all stateaction

pairs uniformly although the resulting p olicy still dep ends on the discount factor If the

discount factor is the p olicy will try to follow paths leading to transitions with the minimal

number of o ccurrences Still the resulting exploration b ehavior can b e quite complex Lets

have a lo ok at Figure After a while the exploration b ehavior will much more often

execute the action goright than goup since the goup action leads immediately to transitions

which are well explored Note that although this environment is deterministic the same holds

for sto chastic environments Thus the exploration b ehavior dep ends strongly on the top ology

of the state space Furthermore the exploration b ehavior dep ends also on the discount factor

Figure A deterministic environment where the frequency based rule cannot explore al l

stateaction pairs uniformly The environment consists of a number of states and actions

Actions lead up or right for most states with the exception of the top state for which both

actions lead to the same successor After trying out al l upactions a single time the explorer

becomes aware of the bottleneck transitions and which are crossed so often that up

actions immediately leading to them wil l get low exploration values and not be considered for

a long time

Reward Function Recency Based

Select the actions which have b een selected least recently The reward for exploring SAP

s a is

R s a

K T

CHAPTER EXPLORATION

where K is a scaling constant and t the current time step Note that the exploration reward

is not entirely lo cal since it dep ends on the global time counter This exploration reward

rule makes a lot of sense for dealing with changing environments

Reward Function Error Based

The third reward rule is as follows

R s a s Q s a Q s a K

t t t t t t t t t P

Here Q s a is the Qvalue of the stateaction pair SAP b efore the up date and Q s a

t t t t t t

is the Qvalue of the SAP after the last up date which has b een made b efore computing the

exploration reward The constant K ensures that all rewards are negative This reward

rule prefers to keep on selecting stateaction pairs which have strongly increasing Qvalues

Initially it makes unexplored SAPs prime exp eriences due to the negative rewards assigned

to all explored steps

Comments Schmidhuber used the absolute dierence jV V j instead of

t t

the directional change Thrun combined the error based learning rule with the

frequency based learning rule He did not nd any improvements over just using the frequency

based rule however

False Exploration Reward Rules

We can of course construct all kinds of lo cal exploration reward rules However we have to

b e careful some rules may lo ok quite go o d but may lead to unwanted exploration b ehavior

To examine whether an exploration reward rule will work we should analyze the following

condition if we continuously select exploration actions we should never b e absorb ed in a

subspace of the state space unless we have gathered so much information that we can b e sure

that exploring dierent subspaces will not change our learned p olicy

An example of a wrong exploration reward rule is the following prefer stateaction pairs

for which the successor states are least predictable ie the transition probabilities have high

entropy

P a log P a c R s a

sk sk

Here c is a p ositive constant ensuring that exploration rewards are negative That no pref

erence should b e given to exploring high entropy SAPs can b e shown by considering the

following counter example

We have an environment which consists of a number of states connected to each other by

a line The starting state is in the middle of the line The agent can select the action goleft

or goright In of the cases actions are executed prop erly and in of the cases they are

replaced by the opp osite action There is a sp ecial state lo cated one step b efore the extreme

right of the line In this state the action goleft results in a completely random transition

to the state left or right of it If exploring with the highentropy exploration rule the agent

may initially try all actions a couple of times After a while it would favor the state at the

right since the exploration reward is highest there Therefore it will not sample correctly

We use negative rewards so that we can zeroinitialize the exploration Qfunction and guarantee that

untried SAPs lo ok more promising

LEARNING EXPLORATION MODELS

but continuously go to this state and learn its transition probabilities p erfectly whereas other

transition probabilities are approximated much worse The exploration reward rule considers

the complete randomness as b eing information carrying but in this example it is not

Learning Exploration Mo dels

We can use all RL metho ds to learn exploration Qvalues Previous metho ds used Qlearning

for learning where to explore Schmidhuber a Thrun Storck et al We

prop ose to use prioritized sweeping instead PS allows for quickly learning exploration mo dels

which may b e useful for learning Qvalues estimating global information gain taking into

account yet unexplored regions of the statespace

Replacing reward A nice option in using separate transition reward values for learning

exploration functions is that all explorative transition rewards can b e based entirely on the

current reward Supp ose that an agent selects an action which has already b een executed

several times Computing the exploration reward by averaging over all previous transition

rewards would not result in the desired reward measure For instance with frequency based

exploration we would receive rewards C a for the rst second etc o ccurrence of

the stateaction pair Our estimated transition reward R i a would b e the average over

these rewards instead of just equal to C a the current reward value and therefore some

undesired rescaling takes place Using MBRL we just replace the estimated reward Ri a j

by R i a j for all j with P a that is we up date all rewards for outgoing transitions

from the current stateaction pair to take the latest available information into account

Neverending exploration The exploration utilities continually change there is

not a stable optimal exploration function This is not a problem at all since the goal of

exploration is to search for alternative paths in order to nd b etter and b etter p olicies and

therefore the exploration p olicy should never converge Continuous exploration is useful when

one wants to obtain a go o d nal p olicy without caring for intermediate rewards For particular

problems we may want to increase or switch to exploitation after some time however Eg in

limited lifetime scenarios Schmidhuber et al or bandit problems Berry and Fristedt

we want to optimize the accumulative reward over the entire life time There are a

couple of simple ways to get around purely explorative b ehavior

Switch to the greedy p olicy once the value function of the real task p olicy is hardly

changed We could implement this as follows during TS steps compute the change of V If

this average change p er step is smaller than this means that we hardly learn anymore so

that we can switch to the greedy p olicy

Anneal the exploration rate P from in the b eginning always select according

exp

to the exploration mo del to at the end If the lifetime is unknown we can try to predict

it or rep eatedly extend the horizon The latter is sometimes done in gameplaying programs

which have xed thinking time for playing an entire game for which it is never sure how

many moves will b e played in total

Start with exploration and switch after a xed amount of time to exploitation This

is useful for huge state spaces for which we can never visit all states

CHAPTER EXPLORATION

Mo delBased Interval Estimation

To explore eciently an agent should not rep eatedly try out actions that certainly do not

b elong to the optimal p olicy To reduce the set of optimal action candidates we extend the

interval estimation IE algorithm Kaelbling to combine it with mo delbased RL

Standard IE selects the action with the largest upp er b ound for its Qvalue To compute

upp er b ounds it keeps track of the means and standard deviations of all Qvalues The

standard deviation is caused by the sto chastic transition function although it could as well

b e caused by a sto chastic reward function Although IE seems promising it do es not always

outp erform Qlearning with Boltzmann exploration due to problems of estimating the variance

of a changing Qfunction in the b eginning of the learning phase Kaelbling

Mo delBased Interval Estimation MBIE uses the mo del to compute the upp er

b ound of Qvalues Given a set of outgoing transitions from SAP i a MBIE increases the

probability of the b est transition the one which maximizes V j R i a j dep ending

on its standard deviation Then MBIE renormalizes the transition probabilities and uses the

result for computing the Qvalues See Figure The leftmost transition is the b est given

the current state For this transition we have computed the condence interval and set its

probability to its upp erb ound after which we renormalize the other probabilities

0.45 0.33 0.1

[0.3,0.6] 0.6 0.07

Figure ModelBased Interval Estimation changes the transition probability of the best

transition from a stateaction pair to its upper probability bound After this the other tran

sition probabilities are renormalized

The following algorithm can b e substituted for lines and in our PS algorithm

Section

Mo delBased Interval Estimationi a

a m Ar g max fR i a j V j g

j P a

b n C a

c P P a

2 2

z z z

d P a P P P

n n n

e P a P a

P im

f j m

C a

g P a P a

C aC a

i im

P aR i a j V j h Qi a

Here z is a variable which determines the size of the condence b ounds Step d elab orates

on the commonly used z P P n and is designed to give b etter results for small values

of n see Kaelbling for details

EXPERIMENTS

MBIE hybrids Since MBIE also relies on initial statistics we prop ose to circumvent

problems in estimating the variance by starting out with some other exploration metho d and

switching to IE once some appropriate condition holds

This is done as follows we start with frequency based exploration and keep tracking the

cumulative change of the problemoriented value function Once the average up date of the

V function computed over the most recent TS timesteps falls b elow IR we I copy

the rewards and Qvalues from the problemoriented mo del to the exploration mo del and

I I switch to IE we apply asynchronous value iteration to the mo del the iteration pro cedure

calls MBIE for computing Qvalues and ends once the maximal change of some state value is

less than IR

Simultaneous p olicy learning The mo delbased learner simultaneously learns b oth

exploration p olicy and problemoriented p olicy After each exp erience we up date the mo del

and use PS to recompute the value functions If actions are only selected according to one

p olicy we can p ostp one recomputing the other value function and p olicy and use PS or value

iteration once it is required Note that we can use value iteration here since we may exp ect

many changes of the mo del and only need to recompute the value function once in a long

while

Comments Note that we approximate the upp er b ound of the probability of making

the b est transition by assuming the normal distribution We basically make the distinction

b etween the b est transition and any other transition We could also compute the optimistic

value function in a dierent way rst set all transition probabilities to their lower b ounds

and then iteratively set the b est transitions to their upp er b ound as long as the probabilities

do not sum to See also Givan et al for a description of condence b ounded Markov

decision problems or see Camp os et al for mathematical metho ds for computing

probability intervals

Exp eriments

In the previous chapter we have used prioritized sweeping with Maxrandom exploration and

saw that it reaches almost optimal p erformance levels Still the attained value function

approximations were quite bad which indicates that the state space was badly explored In

this section we p erform exp eriments to nd out whether learning exploration mo dels can

sp eed up nding optimal or almost optimal p olicies We will rst describ e exp eriments in

which we compare using exploration mo dels to undirected exploration Here the goal is to

quickly nd go o d p olicies Then we will examine how well the dierent exploration metho ds

handle the explorationexploitation dilemma Finally in subsection we will use the

exploration mo dels and MBIE for nding very close to optimal p olicies for a maze consisting

of multiple goal states

Exploration with Prioritized Sweeping

In order to study p erformance of the directed exploration metho ds we combined them with

PS and tested them on dierent mazes of size and The mazes

are the same as in previous chapters and contain noise in the actionexecution The

randomly generated mazes consist of ab out blo cked states and p enalty states

Parameters and exp erimental setup The accuracy parameter and U

max

The discount factor was set to for learning the exploration reward function We

CHAPTER EXPLORATION

compare the following metho ds

Maxrandom with P

max

Frequency based The constant K is set to

Recency based K is set to

Error based K is set to To get b etter results with the error based rule we

chose in of the cases the Maxaction

We record how fast the metho ds are able to learn p olicies which collect and of

what the optimal PIp olicy collects For this we have tested the optimal p olicy for times

steps and computed the average over steps The learning metho ds are allowed

steps for the mazes and steps for the other mazes The agent is tested

after each steps for times steps

Exploration Rule

Max random K K K K K K

Frequency based K K K K K K

Recency based K K K K K K

Error based K K K K K K

Table Learning exploration models with PS for the dierent maze sizes The table shows

the number of steps which were needed by the exploration rules to nd policies which collect

of what the PIpolicy collects Explanation simulation did not meet the require

ments

Exploration Rule

Max random K K K K K K

Frequency based K K K K K K

Recency based K K K K K K

Error based K K K K K K

Table Learning exploration models for the dierent maze sizes The table shows the

number of steps which were needed by the exploration rules to nd policies which collects

of what the PIpolicy collects Explanation and in and simulations

respectively the requirements were not met

Results Tables and show the results We can see that learning an exploration

mo del with the frequency based reward rule works b est When state spaces b ecome larger

the costs of undirected exploration metho ds such as Maxrandom exploration increases sig

nicantly The costs of frequency based exploration seems to scale up almost linearly with

the size of the state space however

Comments We used for learning the exploration mo del This results in

very global exploration b ehavior We also tried out using lower values for Those worked

signicantly worse The entirely lo cal p olicy with was only able to nd optimal

p olicies in out of simulations and to ok steps for the maze Higher values

worked b etter although in the interval gave the b est results

Qlearning needed K steps to learn optimal p olicies with Maxrandom explo

ration for the maze We also tried Q on mazes We found that

p erformed b est although we have not tried replacing and resetting traces Qlearning needed

EXPERIMENTS

M steps b efore nding reasonable p olicies compared to K for mazes There

fore as exp ected we can say that Qlearning scales up much more p o orly than mo delbased

approaches

In our exp erimental results Qapproaches did not b enet from learning exploration

value functions The results were more or less the same as using Maxrandom It seems very

dicult for Qlearning to learn useful exploration values Since we p enalize actions the ma jor

gain in using exploration reward rules to select alternative actions for a state is lost

Conclusion Using exploration mo dels can signicantly sp eed up nding go o d p oli

cies Using the frequency based reward rule we can compute p olicies by sampling all

stateaction pairs ab out times Although the variance in the estimated transition proba

bilities of the maximum likelihoo d mo del is still quite large this do es not seem to matter for

nding go o d p olicies Due to the randomly generated environment there seem to b e many

p olicies which are quite go o d and it is not very dicult to learn one of them

The other exploration techniques do not distribute the number of exp eriences fairly over

the stateaction space and they have more diculties nding go o d p olicies The recency

based reward rule might outp erform frequency based exploration in changing environments

Finally undirected exploration works well for the smallest problems but consumes a lot of

time for the more dicult problems Thus we need to make use of exploration mo dels for

solving large tasks

ExploitationExploration

The previous subsection showed that we are able to quickly learn go o d p olicies by using ex

ploration mo dels For limited lifetime problems involving exp ensive rob ots or exp eriments

we want to maximize the cumulative p erformance over time This creates the exploita

tionexploration problem In this section we show how well the exploration metho ds maximize

cumulative rewards

We set up exp eriments with mazes and use prioritized sweeping and

U and let the agent learn for steps during which we computed the cumu

max

lative and nal rewards over intervals of steps

For the metho ds which learn exploration mo dels we increase P from to in

max

order to exploit more and more We also included a metho d which uses the frequency based

rule to learn exploration mo dels but this system shifts to exploitation if the value function

for the task is hardly changed Each time the goal is found the system checks whether a

minimal of steps have b een executed for which the average up date of the Vfunction is

lower than We will call this system frequency based stop explore This system also

uses P keeping the system fully explorative b efore switching to exploitation

max

do es not change things much For Boltzmann exploration we linearly anneal T from to

Results Table shows that although the value functions of the directed exploration

metho ds are much b etter the nal p erformance for most exploration metho ds except Boltz

mann exploration is more or less equal The cumulative reward of using most exploration

mo dels is worse than using Maxrandom Remember however that go o d p olicies are found

earlier using directed exploration and that the dierence in cumulative reinforcement is mainly

b ecause exploration p olicies lead the agent away from the greedy p olicies thereby causing

some loss in reward The table shows that frequency based exploration and stop explore is

able to accumulate a lot of reward Figure A shows the learning curves for the dierent

CHAPTER EXPLORATION

System End Rewards Total Rewards Vapprox Time

Max random K K K K

Boltzmann K K K K

Error based K K K K

Recency based K K K K

Frequency based K K K K

Frequency based Stop explore K K K K

Table Results for dierent exploration rules with prioritized sweeping

exploration metho ds Positive Qup dates refers to the error based exploration rule Figure

B shows that the exploration mo dels are very useful for learning go o d value function

approximations The recency and frequency based reward rules work b est They are able to

learn quite go o d value function approximations The fact that such go o d approximations are

attained by sampling all stateactions only times exp eriences

shows that the value function is not very sensitive to noise The reason is that noisy actions

take the agent to a eld closeby so that even in case of strong bias due to noise in the

exp eriences for some state the value function will not dier to o much

700 Max random (0.3-0) Max random (0.3-0.0) 20000 Positive Q-updates Positive Q-updates Time recency Time recency Counter based 600 Counter based Counter based + Stop explore Counter based + Stop explore 15000 500

10000 400

5000 300 Average Euclidean Distance reinforcement per 2000 steps 0 200

-5000 100

-10000 0 0 25000 50000 75000 100000 0 25000 50000 75000 100000

#steps #steps

Figure A A comparison between dierent learning exploration rules with prioritized

sweeping The gure plots the cumulative rewards achieved over intervals of steps B

The approximation error of the value function using dierent exploration methods

The frequency based stop explore system scores very high on all ob jective criteria the

nal p erformance is very go o d the cumulative p erformance is the b est and the obtained value

function is much b etter than that of indirect exploration metho ds

Comments Initialize high exploration p erformed very p o orly with PS The reason

for this is that exploration values can keep their large initial values for a long time

EXPERIMENTS

Error and recency based exploration cost more computational time than frequency based

exploration The reason is that reward values are bigger in size

Conclusion The explorationexploitation dilemma is not handled very well by the fully

explorative mo delbased exploration techniques However we can augment them simply by

a heuristic switching rule which makes the agent switch from exploration to full exploitation

Our rule which switched once the value function did not change very much anymore made

it p ossible to rst quickly learn a quite go o d value function approximation which could af

terwards b e exploited for a long time Although for the current problem simple undirected

exploration metho ds also worked quite well they were unable to learn a go o d value function

We exp ect that for more general MDPs Maxrandom exploration will p erform worse

since for the current problems they did not have any problems nding the global terminal

goal state If there are multiple rewarding states they may have larger problems and the

advantage of using the directed exploration metho ds could b e larger

Exp eriments with Sub optimal Goals

The nal exp erimental comparison shows results with directed exploration metho ds and MBIE

on a maze with sub optimal goalstates The maze is shown in Figure The starting state

S is lo cated eld northeast of the southwest corner There are three absorbing goal

states two of them are sub optimal The optimal goal state G is lo cated eld southwest

of the northeast corner the sub optimal goal states F are lo cated in the northwest and

southeast corners Selected actions are replaced by random actions with probability

Reward function Rewards and the discount factor are the same as b efore For nding

a sub optimal goal state the agent gets a reward of

F G

S F

Figure The maze used in the experiments Black squares denote blocked elds

grey squares penalty elds The agent starts at S and searches for a minimal ly punishing path

to the optimal goal G Good exploration is required to avoid focusing on suboptimal goal states F

CHAPTER EXPLORATION

Comparison We compare the following exploration metho ds Maxrandom directed

mo delbased exploration techniques using frequency based and recency based reward rules

and MBIE The latter starts out with mo delbased exploration using the frequency based

reward rule and switches to IE once the value function hardly changes any more

The goal is to learn go o d p olicies as quickly as p ossible We computed an optimal p olicy

using value iteration Bellman and tested this optimal p olicy by executing it for

steps We computed its average reinforcement intake by testing it times which resulted

in rewardpoints For each metho d we conduct runs of learning steps

During each run we measure how quickly and how often the agents p olicy collects

and of what the optimal p olicy collects This is done by averaging the results of

test runs conducted every learning steps each test run consists of executing the greedy

p olicies always selecting actions with maximal Qvalue for steps

Parameters We set the accuracy parameter and U for b oth learning

max

the problemoriented and exploration value functions The exploration rewards discount

factor is set to for frequency based and to for recency based exploration The

constant K used by the frequency based reward rules is set to K used by the recency

C T

based reward rule is set to We used two values for P for Maxrandom exploration

max

and The value of z for MBIE is set to which corresp onds to a condence

interval of The combination of MBIE and mo delbased exploration switches to MBIE

once the value function has not changed by more than on average within the

TS most recent steps

Exploration Rule freq freq freq

MBIE K K K

Frequency based K K K

Recency based K K K

Maxrandom K K K

Maxrandom

Table The number of runs of several exploration methods which found poptimal policies

and how many steps were required for obtaining it

Exploration Rule Best run result Training Performance

MBIE K K K K

Frequency based K K K K

Recency based K K K K

Maxrandom K K K K

Table Average and standard deviation of the best test result during a run and the total

cumulative reward during training

Results Table shows signicant improvements achieved by learning an exploration

mo del The undirected exploration metho ds fo cus to o much on sub optimal goals which are

closer and therefore easier to nd This often prevents them from discovering nearoptimal

p olicies On the other hand mo delbased learning with directed exploration do es favor paths

DISCUSSION

8000

6000

4000

2000 MBIE Frequency-based Recency-based Max-random 0.2 0 Max-random 0.4 reinforcement per 1000 steps

-2000

0 20000 40000 60000 80000 100000

#steps

Figure Cumulative reward during test runs averaged over simulations

leading to the optimal goal Using the frequency based reward rule by itself the agent always

nds the optimal goal although it fails to always nd optimal p olicies

Figure shows a large dierence b etween learning p erformances of exploration mo dels

and Maxrandom exploration It do es not clearly show the distinction b etween frequency

based or recency based exploration and the extension with MBIE Switching to MBIE after

some time which happ ens b etween and steps however signicantly improves

matters First of all Table shows that this strategy nds optimal or nearoptimal p olicies

in of the cases whereas the others fail in at least of the cases The second improve

ment with MBIE is shown in Table MBIE collects much more reward during training than

all other exploration metho ds thereby eectively addressing the explorationexploitation

dilemma In fact it is the only exploration rule leading to a p ositive cumulative reward score

Conclusion The results with multiple goals showed a large advantage of using directed

exploration metho ds Undirected metho ds are attracted to those rewarding states which

come rst on the path and therefore we cannot trust them for nding go o d p olicies The b est

metho d MBIE combined directed exploration with condence levels in order to prune away

actions which almost certainly do not b elong to the optimal p olicy This results in the b est

p erformance since it takes care that only actions which can b elong to the optimal p olicy are

selected Finally this also results in the highest cumulative p erformance We exp ect that this

metho d will p erform b est for most sto chastic discrete MDPs although a disadvantage of the

metho d is that we need to set the right switching time

Discussion

If we use undirected exploration metho ds we rely on our random generator for selecting ac

tions However if we learn where to explore we can direct our exploration b ehavior so that

more information is gained We formulated a number of exploration reward rules which de

termine which information is interesting and which can b e used for learning exploration value

functions Our exp erimental results with sto chastic mazes showed that learning exploration

p olicies can sp eed up nding go o d p olicies Esp ecially when there are particular complex

paths leading to much more rewards then other easier paths go o d exploration is essential

CHAPTER EXPLORATION

The frequency based reward rule p erformed b est for the environments we have considered

We exp ect this to hold as well for other MDPs For nonstationary problems an exploration

p olicy based up on the recency based reward rule may work b est since it is able to discover

much earlier which fragments of the state space have changed

Optimal exploration When it comes to nding optimal p olicies metho ds using exter

nal MDP rewards for fo cusing on actions with large probability of b eing optimal will often

save some SAP visits over metho ds based solely on exploration reward rules Furthermore

cumulative external MDP rewards obtained during training should b e taken into account in

attempts at approaching the exploitexplore dilemma That is why we introduced MBIE

a metho d combining Kaelblings interval estimation IE algorithm Kaelbling and

mo delbased reinforcement learning Since MBIE heavily relies on initial statistics we switch

it on only after an initial phase during which an exploration mo del is learned according to

say the frequency based exploration reward rule In our exp eriments this approach almost

always led to optimal p olicies We exp ect MBIE to b e advantageous for all kinds of

MDPs esp ecially when the reward function is far from uniform Other exploration metho ds

do not work well for such problems since their exploration p olicies do not make distinctions

b etween dierent actions leading to dierent payos

Dierent top ologies For particular environments it may b e a go o d idea to explore

entire subregions b efore exploring other subregions Eg when there is a b ottleneck con

necting two subregions b oth subregions may alternatingly b ecome interesting to explore but

the b ottleneck will b ecome uninteresting after some time Our exploration rules could detect

such b ottlenecks and would sp end a lot of time exploring one subspace b efore going to the

other since going through the b ottleneck brings a lot of negative exploration reward

Function approximators The ideas are presented for tabular representations but can

easily b e extended to function approximators For this we can use the same representation

for the exploration value function as for the original value function Exploration metho ds

for large or continuous spaces are more complex however since it is infeasible to explore

all states Therefore we rely on the generalization abilities of the function approximator to

determine the interestingness of exp eriences Furthermore the reward function gets a little

bit more complicated since states may never b e visited twice Therefore the reward function

should b e based up on the distribution of previous exp eriences around a p oint for which we

compute an exploration value Eg a Gaussian function can b e used to weigh neighboring

exp eriences according to their distance

Chapter

Partially Observable MDPs

We have seen how we can eciently compute p olicies for Markov decision pro cesses MDPs

consisting of a nite number of states and actions MDPs require that all states are fully

observable which means that the agent is able to use p erfect sensors to always know the exact

state of the world For real world problems this requirement is usually violated Agents receive

their information ab out the state of the world through imp erfect sensors which sometimes

cause world states to b e mapp ed to the same observations This problem is called perceptual

aliasing Whitehead and is the reason for the uncertainty of the agent ab out the state

of the world

As example have a lo ok at the environment depicted in Figure There are a total of

states but the agent can only see dierent observations a circle or a rectangle Thus

solely based on the current observation the agent can never b e certain in which state it is

1 2 3 4 5 6

Figure An environment consisting of states Whenever the agent is in the rst two

states or in states or it always sees a circle as observation In the third state it observes

a circle or rectangle both with probability and in the sixth state it always observes a

rectangle

Markov decision problems for which the agent is uncertain ab out the true state of the

world are called partially observable Markov decision processes POMDPs eg Littman

The simplest algorithms for solving them would directly map observations to actions

but since observations are received in dierent states which require dierent actions such

algorithms will not b e able to nd solutions Better algorithms need to use previous observa

tions and actions which were encountered on the current tra jectory through the state space

to disambiguate p ossible current states This is not an easy task even deterministic nite

horizon POMDPs are NPcomplete Littman Thus general and exact algorithms are

feasible only for small problems which explains the interest in heuristic metho ds for nding

go o d but not necessarily optimal solutions

Hidden state Since an observation conceals some part of the true underlying state

we say that some part of the state is hidden The solution to the hidden state problem

CHAPTER PARTIALLY OBSERVABLE MDPS

HSP is to up date an agents internal state IS based on its previous observations and

actions This internal state summarizes the trace of previous events and is used to augment

the information of the current observation in the hop e that the state ambiguities are resolved

Lets have another lo ok at Figure Supp ose that the agent sees a rectangle steps left sees

a circle steps right and sees a circle Supp ose also that there is no noise in the execution

of actions then in which state is the agent The answer is in state since only state can

emit a rectangle and a circle symbol Thus the augmented state is uniquely dened and

can b e used by the p olicy for selecting the optimal action Figure shows how previous

observationsactions of the environment are stored and used to create the IS Then the IS is

combined with the current observation to generate a description of the state which is used by

the p olicy to select an action

Environment

Current Action Observation Right

state 3 I S t-2 t-1

Left Right

Recollection

Figure The decision making loop of an agent in a partially observable environment The

agent receives an observation from the environment dened in Figure The recollected

observations and actions are used to create an internal state IS This internal state is then

augmented with the current observation and the resulting state is presented to the policy

The policy then selects an action which together with the current observation is memorized

Final ly the action is executed and changes the environment

Outline of this chapter In Section we describ e the POMDP framework and

also optimal algorithms from op erations research based on dynamic programming algorithms

which allow computing optimal p olicies by explicitly representing the uncertainty of the state

in a probabilistic description In Section we will describ e our novel algorithm HQlearning

and show its capability in solving a number of partially observable mazes In Section we

summarize related work In Section we conclude with some nal statements ab out the

problems and algorithms

OPTIMAL ALGORITHMS

Optimal Algorithms

Op erations research has investigated optimal algorithms for solving POMDPs for some time

already see Lovejoy for an overview They treat POMDPs as planning problems

which means that the state transition function reward function and observation function are

given and that the goal is to compute an optimal plan Note that this plan is not represented

by a single sequence of actions since such a plan would probably not b e carried out due to

the sto chasticity of the problem in eg the transition function Instead the plan is again

represented by a p olicy mapping states to actions This is similar to the MDP framework

of Chapter The big dierence with MDPs is that even when we know the mo del of the

POMDP computing an optimal p olicy is extremely hard An agent may confuse dierent

states due to the noise in the observation function and may b e totally mistaken ab out the

true state of the world So how can it then compute an optimal p olicy Well it can by taking

its uncertainty into account in its decision making We will see that this allows for selecting

information gathering actions and goaldirected actions based on maximizing its exp ected

future discounted cumulative reward

POMDP Sp ecication

A POMDP is a MDP where the agent do es not receive the state of the world as input but

an observation of this state Thus the agent should use multiple observations in its decision

making p olicy As for MDPs the systems goal is to obtain maximal discounted cumulative

reward A POMDP is formally sp ecied by

t is a discrete time counter

S is a nite set of states

A is a nite set of actions

P s js a denotes the probability of transition to state s given state s and action

t t t t t

R maps stateaction pairs to scalar reinforcement signals

O is a nite set of observations

P is the probability distribution over initial states

P ojs denotes the probability of observing a particular ambiguous input o in state s

We write o B s to denote the observation emitted at time t while visiting state s

t t t

is a discount factor which trades o immediate rewards against future

rewards

Belief States

To compute an optimal p olicy we need to use an optimal description of the world state

Due to the uncertainty in action outcomes noise in the execution of actions and the partial

observability of world states we are usually not certain that the agent is in a particular single

CHAPTER PARTIALLY OBSERVABLE MDPS

environmental state Taking the most probable state is p ossible but this cannot lead to

optimal p olicies for particular POMDPs since it do es not take into account that the agent

can b e in many dierent states For computing optimal p olicies we compute o ccupancy

probabilities over all states Such a probability distribution over p ossible world states is

called a belief state We will denote the b elief state at time t as b where b s is the b elief

t t

that we are in state s at time t The b elief state gives us all necessary information to decide

up on the optimal action Sondik additional information ab out previous states and

actions is sup eruous

Computing b elief states At trial start up the a priori probability of b eing in each

state s is P s Since the agent is thrown in the world at time t it also receives an

observation o Therefore we can compute the b elief state at t using Bayes rule

P sP o js

b s P sjo P

P iP o ji

At time t we know b the selected action a and after making a new observation o

t t t

we can compute b as follows

b iP sji a P o js

t t t

P P

b s P sjo b a

t t t t

b iP j ji a P o jj

t t t

i j

In the nominator we sum the probabilities of entering state s over the whole ensemble of states

We can do this since the state space is nite The denominator equals P o jb a and

t t t

can b e seen as a normalizing factor to keep the constraint b j true

Example of b elief state dynamics Consider again the environment of Figure

Supp ose that the actions goleft and goright are executed with probability If the

initial probability distribution over states is f g and the agent observes

a circle then b f g Now supp ose the agent executes action go

right and observes a circle Then the b elief state b f g Again the

agent go es right and observes a circle then the b elief state b f g and

again b f g Finally after actions and circles the agent knows with

certainty that it is in state thus it could also infer that it started in state In this case

the actions were deterministic and the uncertainty decreased after each action In general

the uncertainty can increase or decrease dep endent on the b elief state and the transition

and observation functions In such environments the agent can reduce state uncertainty by

visiting particular states with exceptional observations Such states are generally referred to

as landmark states which play an imp ortant role in designing algorithms for POMDPs

Computing an Optimal Policy

The p olicy maps a b elief state to an action

a b

t t

Note that the b elief state b has the dimension of the state space to b e precise its dimension

is jS j and consists of continuous variables Computing optimal p olicies for such spaces

is extremely dicult esp ecially if there are many states There are some sp ecial prop erties

of the space setup by the b elief states however which makes it p ossible to compute optimal

Even deterministic actions do not imply that the uncertainty or entropy always decreases

OPTIMAL ALGORITHMS

p olicies The imp ortant p oint is that there are nitely many segments in the b elief state

space for which the same optimal action is required when we consider a nite horizon of the

POMDPs This makes it p ossible for optimal algorithms to store for all crucial b elief states

the corners of line segments in b elief space the optimal action and the Qvalues

There are two kinds of exact algorithms one works with vectors in the b elief state space

the other works with pregenerated p olicy trees which are then evaluated on dierent segments

of b elief space

DP on vectors in b elief state space We can derive a DP algorithm which at each

iteration computes all p ossible b elief states given the previous set of b elief states by using

Equation and iterating over all p ossible actions and observations Then the algorithm

computes their values for some b elief state b we can compute its step value as follows

V bsR s a b max

Note that this value is a linear pro duct of the b elief state and the reward function Then the

tstep value can b e computed using value iteration as follows

X X X X

bsP ijs aP ojiV bo a V b max bsR s a

t t

s o s

where bo a is dened as the b elief state resulting from observing observation o after selecting

action a in b elief state b Thus we basically use transition probabilities b etween b elief states

and compute the value of a b elief state in a very similar way as for MDPs

One problem is that although the number of b elief states is nite it grows exp onentially

with the horizon of the planning pro cess Therefore some pruning in b elief state space is

needed to make the computations feasible We will not discuss those topics further Interested

readers are referred to Cassandras thesis

Another problem is that we have assumed that we start with an initial b elief state which is

known a priori Now assume that we do not yet know the initial b elief state when we want to

compute the value function although this b elief state is given later on once we get a problem

instance It is still p ossible to store the value function since it is piecewise linear which means

that there are nitely many facets see Cassandra for a formal pro of We know that

one p olicy is always b est for a sp ecic region of the b elief state Each p olicies value function

is linear in the b elief state see Equation Thus since we always take the maximum of

the linear functions the optimal value function is a piecewise linear convex function Figure

We cannot compute the facets using the simple algorithm describ ed ab ove however

since we would need to compute b elief state intervals which are recursively rened Then for

each b elief state interval we have to compute the optimal action Therefore we need to use

more dicult algorithms for solving this more general problem

Policy trees For computing the optimal p olicy given an unknown a priori state we

can make use of policy trees Littman Cassandra et al Kaelbling et al

Policy trees fully describ e how the agent should act in the next t steps and use the sequence

of observations in determining correct action selection see Figure

Given each particular b elief state one tstep p olicy tree is optimal We want to know

however for which interval in b elief state space some particular p olicy tree is optimal To

b e able to compute this we can generate all p ossible tstep p olicy trees and evaluate them

on each state Then we are able to compute the value function of a p olicytree as a linear

function of the b elief state

CHAPTER PARTIALLY OBSERVABLE MDPS

Q(s,p ) 1 Expected t-step policy Q(s,p ) 2 V(s) value

Q(s,p 3) 0 1

b(s)

Figure A two state problem dened by the probability we are in state s We show the

optimal value function for this belief state space together with Qfunctions of dierent

policies Dependent on the belief state one policy is the best one Since the Qfunctions of

policies are a linear function of the belief state the optimal value function is a piecewise

linear convex function

A t steps to go O 1 O O2 m AAA t-1 steps to go

A 2 steps to go O 1 O O2 m

AAA 1 step to go

Figure A tstep policy tree After each action an observation is made and this observation

determines which branch in the tree is fol lowed after which the rest of the policy tree is

executed

Thus again we start simple If the agent can only select a single action we say that the

agent executes a step p olicy tree We denote the p olicy tree as p and ps ro ot action by

ap We compute the Qvalue of executing the step p olicy tree p in state s as

Qs p R s ap

Where as usual R s ap is the immediate reinforcement for selecting action ap in state s

jO j

p g Then we construct a tstep p olicy tree p from a set of t step p olicy trees fp

OPTIMAL ALGORITHMS

by dening which observations has lead to them we denote p o as the p olicytree p

t t

which follows observation o and by merging these trees by a ro otno de with action ap

We can compute the value function of a tstep p olicy tree by combining the value functions

of its tstep p olicy subtrees The value of executing the tstep p olicy tree p in state s can

b e computed as

X X

Qs p R s ap P s js a P ojs Qs p o

t t t

Thus we just add the immediate reward to the exp ected value of executing a particular

selected p olicy subtree which one dep ends on the observation in the next state We store

all values Qs p of executing the p olicy tree p in state s

t t

Then when given a b elief state we can simply weigh the dierent values according to

their o ccupancy probabilities to obtain the evaluation of executing a p olicy tree on a b elief

state b

bsQs p Qb p

Now the tstep value of a particular b elief state b is the maximal value over all p olicy trees

Qb p V b max

t t

Complexity The algorithm is intractable for all but small problems Esp ecially for long

or innite planning horizons the costs are infeasible There are jO j dierent tstep observation

t t

sequences and jO j jO j jO j jO j jO j no des in each tstep p olicy

tree Since at each no de we can choose b etween jAj dierent actions the total number of

jO j jO j

p olicy trees is jAj which grows sup erexp onentially Finally we note that the

number of facets of the optimal value function may b e innite for innite horizon problems

Faster optimal algorithms In principle it is p ossible that each one of these p olicy

trees can b e optimal for some p oint in b elief space Fortunately in practice many p olicy trees

are completely dominated by other p olicy trees a p olicy tree is dominated if for each p ossible

b elief state there exists another p olicy tree with larger or equal Qvalue so that they can

b e pruned away This allows for signicant reductions in computational time and storage

space Further reductions are made p ossible by using particular online generation algorithms

eg the Witness algorithm Cassandra et al Littman or algorithms prop osed

by Cassandra Zhang and Liu which only generate p olicy trees which have the

p ossibility of b eing optimal for some p oints in b elief space These metho ds can signicantly

reduce computational requirements for practical problems Still computing optimal p olicies

for general problems remains extremely exp ensive

Littman et al and Cassandra compare dierent POMDP algorithms using

b elief states They rep ort that small POMDPs with less than states and few actions do

not p ose a very big problem for most metho ds Larger POMDPs to states however

can cause ma jor problems Thus although the theory allows us to compute optimal solutions

the gap b etween theory and practice do es not allow us to use these algorithms for real world

problems Therefore we will fo cus on heuristic metho ds which can b e used to nd sub optimal

solutions to much larger problems

CHAPTER PARTIALLY OBSERVABLE MDPS

HQlearning

We have seen that general and exact algorithms are feasible only for small problems Fur

thermore they require a mo del of the POMDP That explains the interest in reinforcement

learning metho ds for nding go o d but not necessarily optimal solutions Examples are the

use of recurrent networks Schmidhuber d Lin hidden Markov mo dels McCal

lum higher order neural networks Ring memory bits Cli and Ross

and sto chastic p olicies Jaakkola Singh and Jordan Unfortunately however most

heuristic metho ds do not scale up very well Littman Cassandra and Kaelbling and

are only useful for particular POMDPs

We will now describ e HQlearning Wiering and Schmidhuber a novel approach

based on nite state memory implemented in a sequence of agents HQ do es not need a mo del

and can solve large deterministic POMDPs

Memory in HQ

To select the optimal next action it is often not necessary to memorize the entire past in

general this would b e infeasible since the necessary memory grows exp onentially with the

order of the Markov chain see App endix A A few memories corresp onding to imp ortant

previously achieved subgoals can b e sucient For instance supp ose your instructions for the

way to the station are Follow this road to the trac light turn left follow that road to the

next trac light turn right there you are While you are on your way only a few memories

are relevant such as I already passed the rst trac light Between two such subgoals a

memoryindep endent reactive policy RP will carry you safely

Overview HQlearning uses a divideandconquer strategy to decomp ose a given POMDP

into a sequence of reactive policy problems RPPs RPPs can b e solved by RPs all states

causing identical inputs require the same optimal action The only critical p oints are those

corresp onding to transitions from one RP to the next

To deal with such transitions HQ uses multiple RPPsolving subagents Each agents RP

is an adaptive mapping from observations to actions At a given time only one agent can b e

active and the systems only type of shortterm memory is embo died by a p ointer indicating

which one Thus the internal state IS of the system is just the p ointer to the active agent

RPs of dierent agents are combined in a way learned by the agents themselves The rst

active agent uses a subgoal table its HQtable to generate a subgoal for itself subgoals are

represented by desired inputs if these inputs are unique in a sp ecic region we call them

landmark states Then it follows the p olicy embo died by its Qfunction until it achieves its

subgoal Then control is passed to the next agent and the pro cedure rep eats itself After the

overall goal is achieved or a time limit is exceeded each agent adjusts b oth its RP and its

subgoal This is done by two learning rules that interact without explicit communication

Qtable adaptation is based on slight mo dications of Qlearning HQtable adaptation

is based on tracing successful subgoal sequences by Qlearning on the higher subgoal

level Eectively subgoalRP combinations leading to higher rewards b ecome more likely to

b e chosen

POMDPs as RPP sequences The optimal p olicy of any deterministic nite POMDP

with xed starting state and nal goal state is decomp osable into a nite sequence of optimal

reactive memoryless p olicies for appropriate RPPs along with subgoals determining transi

tions from one RPP to the next The trivial decomp osition consists of singlestate RPPs and

HQLEARNING

the corresp onding subgoals In general POMDPs whose only decomp osition is trivial are

hard there is no ecient algorithm for solving them This is simple to see if we consider

an environment where there is only observation the agent is thus blind To solve such

problems all T step p olicies have to b e tried out from the initial state where T denotes the

length of the optimal solution path which results in jAj p ossibilities HQ however is aimed

at situations that require few transitions b etween RPPs which means that each RPP can b e

used for many states

Architecture System life is separable into trials A trial consists of at most T

max

discrete time steps t T where T T if the agent solves the problem in fewer

max

than T time steps

max

There is an ordered sequence of M agents C C C each equipp ed with a Q

table an HQtable and a control transfer unit except for C which only has a Qtable see

Figure Each agent is resp onsible for learning part of the systems p olicy Its Qtable

represents its lo cal p olicy for executing an action given an input It is given by a matrix of

size jO j jAj where jO j is the number of dierent p ossible observations and jAj the number

of p ossible actions Q o a denotes C s Qvalue utility of action a given observation

i t j i j

o The agents current subgoal is generated with the help of its HQtable a vector with jO j

elements For each p ossible observation there is an HQtable entry representing its estimated

value as a subgoal HQ o denotes C s HQvalue utility of selecting o as its subgoal

i j i j

The systems current p olicy is the p olicy of the currently active agent If C is active at

time step t then we will denote this by Activ et i The variable Activ et represents the

only kind of shortterm memory in the system

Selecting a subgoal In the b eginning C is made active Once C is active its HQ

table is used to select a subgoal for C To explore dierent subgoal sequences we use the

Maxrandom rule the subgoal with maximal HQ value is selected with probability P a

i max

random subgoal is selected with probability P Conicts b etween multiple subgoals

max

with maximal HQ values are solved by randomly selecting one o denotes the subgoal

i i

selected by agent C This subgoal is only used in transfer of control as dened b elow and

should not b e confused with an observation

Selecting an action C s action choice dep ends only on the current observation o Dur

i t

ing learning at time t the active agent C will select actions according to the MaxBoltzmann

distribution see Chapter The temp erature T adjusts the degree of randomness involved

in agent C s action selection in case the Boltzmann rule is used

Transfer of control Control is transferred from one active agent to the next as follows

Each time C has executed an action its control transfer unit checks whether C has reached

i i

the goal If not it checks whether C has solved its subgoal to decide whether control should

b e passed on to C We let t denote the time at which agent C is made active at system

i i i

startup we set t

IF no goal state reached AND current subgoal o

AND Activ et M AND B S o

t i

THEN Activ et Activ et AND t t

i+1

Learning Rules

We will use oline learning for up dating the tables this means storing exp eriences and

p ostp oning learning until after trial end no intratrial parameter adaptation In principle

CHAPTER PARTIALLY OBSERVABLE MDPS

Q-TABLE 1 HQ-TABLE 1 AGENT 1

TRANSFER CONTROL

Q-TABLE 2 HQ-TABLE 2 AGENT 2

TRANSFER CONTROL

Q-TABLE 3

AGENT 3

Figure Basic HQarchitecture Three agents are connected in a sequential way Each

agent has a Qtable an HQtable and a control transfer unit except for the last agent which

only has a Qtable The Qtable stores estimates of actual observationaction values and is

used to select the next action The HQtable stores estimated subgoal values and is used to

generate a subgoal once the agent is made active The solid box indicates that the second agent

is the currently active agent Once the agent has achieved its subgoal the control transfer unit

passes control to its successor

however online learning is applicable as well see b elow We will describ e two HQ variants

one based on Qlearning the other on Qlearning Q overcomes Qs inability to solve

certain RPPs The learning rules app ear very similar to those of conventional Q and Q

One ma jor dierence though is that each agents prosp ects of achieving its subgoal tend to

vary as various agents try various subgoals

Learning the Qvalues We want Q o a to approximate the systems exp ected

i t t

discounted future reward for executing action a given o In the onestep lo okahead case we

t t

have

X X

P s js a V B s P s jo iR s a Q o a

j t j t j t i t t

k k

Activ et

s S s S

where P s jo i denotes the probability that the system is in state s at time t given

j t j

observation o all architecture parameters denoted and the information that i Activ et

HQlearning do es not dep end on estimating this probability although b elief states or a world

mo del might help to sp eed up learning V o is the utility of observation o according to

i t t

agent C which is equal to the Qvalue for taking the b est action

V o max fQ o a g

i t a A i t j

Qvalue up dates are generated in two dierent situations T T denotes the total

max

number of executed actions during the current trial and is the learning rate

Q Let C and C denote the agents active at times t and t p ossibly i j If t T

i j

HQLEARNING

then

Q o a Q o a R s a V o

i t t Q i t t Q t t j t

Q If agent C is active at time T and the nal action a has b een executed then

i T

Q o a Q o a R s a

i T T Q i T T Q T T

Note that R s a is the nal reward for reaching a goal state if T T A main dierence

T T max

with standard onestep Qlearning is that agents can b e trained on Qvalues which are not

their own see Q

Learning the HQvalues intuition Recall the introductions trac light task The

rst trac light is a go o d subgoal We want our system to discover this by exploring initially

random subgoals and learning their HQvalues The trac lights HQvalue for instance

should converge to the exp ected discounted future cumulative reinforcement to b e obtained

after it has b een chosen as a subgoal How Once the trac light has b een reached and the

rst agent passes control to the next the latters own exp ectation of future reward is used to

up date the rsts HQvalues Where do the latters exp ectations originate They reect its

own exp erience with nal reward to b e obtained at the station

More formally In the optimal case we have

t t

i+1 i

HQ o E R HV

i j i i

i+1

where E denotes the average over all p ossible tra jectories R R s a C s

i t t i

discounted cumulative reinforcement during the time it will b e active note that this time

interval and the states encountered by C dep end on C s subtask HV max fHQ o g

i i i o O i

is the estimated discounted cumulative reinforcement to b e received by C and following

agents

We adjust only HQvalues of agents active b efore trial end N denotes the number of

agents active during the last trial denotes the learning rate and o the chosen subgoal

HQ i

for agent C

HQ If C is invoked b efore agent C then we up date according to

i N

t t

i+1 i

HQ o HQ o R HV

i i HQ i i HQ i i

t t

HQ If C C then HQ o HQ o R R

i N i i HQ i i HQ i N

HQ If C C and i M then HQ o HQ o R

i N i i HQ i i HQ i

The rst and third rules resemble traditional Qlearning rules The second rule is an

additional improvement for cases in which agent C has learned a p ossibly high value for a

subgoal that is unachievable due to subgoals selected by previous agents

HQlearning motivation Qlearnings lo okahead capability is restricted to one

step It cannot solve all RPPs b ecause it cannot prop erly assign credit to dierent actions

leading to identical next states Whitehead For instance supp ose you walk along a

wall that lo oks the same everywhere except in the middle where there is a picture The goal

is to reach the left corner where there is reward This RPP is solvable by a RP Given the

picture input however Qlearning with onestep lo okahead would assign equal values to

actions go left and go right b ecause they b oth yield identical wall observations

Consequently HQlearning may suer from Qlearnings inability to solve certain RPPs

To overcome this problem we augment HQ by TDmetho ds for evaluating and improving

CHAPTER PARTIALLY OBSERVABLE MDPS

p olicies in a manner analogous to Lins oine Qmetho d TDmetho ds can

learn from longterm eects of actions and thus disambiguate identical shortterm eects of

dierent actions Our exp eriments indicate that RPPs are solvable by Qlearning with

suciently high

Q For the Qtables we rst compute desired Qvalues Q o a for t T

t j

Q o a R s a

T T T T

Q o a R s a V o Q o a

t t t t t t t

Activ et

Q Then we up date Qvalues b eginning with Q o a and ending with Q o a

N T T

according to

Q o a Q o a Q o a

i t t Q i t t Q t t

o for i N HQ For the HQtables we also compute desired HQvalues HQ

HQ o R

N N

t t

HQ o R R

N N N

t t

i+1 i

o o R HV H Q HQ

i i i i

i i

HQ Then we up date the HQvalues for agents C C according to

Min N M

o HQ o HQ o HQ

i i i HQ i i HQ

In principle online Q may b e used as well Online Q should not use actionp enalty

Ko enig and Simmons however b ecause punishing varying actions in resp onse to

ambiguous inputs can trap the agent in cyclic b ehavior

Combined dynamics Qtable p olicies are reactive and learn to solve RPPs HQtable

p olicies are metastrategies for comp osing RPP sequences Although Qtables and HQtables

do not explicitly communicate they inuence each other through simultaneous learning Their

co op eration results in complex dynamics quite dierent from those of conventional Qlearning

Utilities of subgoals and RPs are estimated by tracking how often they are part of suc

cessful subgoalRP combinations Subgoals that never or rarely o ccur in solutions b ecome

less likely to b e chosen others b ecome more likely In a certain sense subtasks comp ete for

b eing assigned to subagents and the subgoal choices coevolve with the RPs Maximizing

its own exp ected utility each agent implicitly takes into account frequent decisions made by

other agents Each agent eventually settles down on a particular RPP solvable by its RP and

ceases to adapt This will b e illustrated by Exp eriment in Section

Estimation of average reward for choosing a particular subgoal ignores dep endencies on

previous subgoals This makes lo cal minima p ossible If several rewarding sub optimal subgoal

sequences are close in subgoal space then the optimal one may b e less probable than

sub optimal ones We will show in the exp eriments that this actually happ ens

Exploration issues Initial choices of subgoals and RPs may inuence the nal result

there may b e lo cal minimum traps Exploration is a partial remedy it encourages alternative

comp etitive strategies similar to the current one Too little exploration may prevent the

system from discovering the goal at all Too much exploration however prevents reliable

estimates of the current p olicys quality and reuse of previous successful RPs To avoid

overexploration we use the MaxBoltzmann Maxrandom distribution for Qvalues HQ

values These distributions also make it easy to reduce the relative weight of exploration as

opp osed to exploitation to obtain a deterministic p olicy at the end of the learning pro cess

we increase P during learning until it nally achieves a maximum value max

HQLEARNING

Selecting actions according to the traditional Boltzmann distribution causes the following

problems It is hard to nd go o d values for the temp erature parameter The degree of

exploration dep ends on the Qvalues actions with almost identical Qvalues given a certain

input will b e executed equally often For instance supp ose a sequence of dierent states

in a maze leads to observation sequence O O O O O where O represents a single

observation Now supp ose there are almost equal Qvalues for going west or east in resp onse

to O Then the Qup dates will hardly change the dierences b etween these Qvalues The

resulting random walk b ehavior will cost a lot of simulation time

For RP training we prefer the MaxBoltzmann rule instead It fo cuses on the greedy

p olicy and only explores actions comp etitive with the optimal actions Subgoal exploration

is less critical though The Maxrandom subgoal exploration rule may b e replaced by the

MaxBoltzmann rule or others

Using gatevalues The shown learning rules are almost the same as for conventional

Qlearning but since we use a multiple agent system the task of one particular agent is

usually not xed during the learning pro cess As long as dierent subgoals of previous agents

are tried out the subtask of an agent changes Since during the learning pro cess subgoals of

agents are generated sto chastically we must take care that an agent fo cuses on the subtask

which it has to carry out most of the times We can do this by using little exploration but

also by constructing a learning rule which uses the probabilities of the generated subgoals

of previous agents to determine the magnitude size of parameter changes for each learning

step We can do this by computing gatevalues g which stand for the probability that the

current sequence of i subgoals is selected

b ecause b efore agent is made active no subgoals are selected

g g P o

i i j

which means that the gatevalue for agent i is equal to the gatevalue for agent i mul

tiplied with the probability that agent i has chosen its particular subgoal For Maxrandom

max

if subgoal o has maximal HQvalue we add the prob exploration P o P

j j max

jO j

ability of selecting the maxsubgoal to the probability of randomly selecting the subgoal and

max

P o otherwise These gatevalues are then multiplied with the learning rate

jO j

to determine the size of each learning step Note that we make a indep endency assumption

among subgoals to keep the system simpler It might b e true that dierent sequences of

subgoals lead to the same systemstate just like in normal Qlearning dierent sequences of

actions might lead to the same environmental state but in the general case it is imp ossi

ble to calculate all the dierent combinations of the subgoals which result in the obtained

systemstate We will call HQlearning with the gatevalues HQGlearning

Exp eriments

We tested our system on three tasks in partially observable environments The rst task is

comparatively simple it will serve to exemplify how HQ discovers and stabilizes appropriate

subgoal combinations It requires nding a path from start to goal in a partially observable

maze and can b e collectively solved by three or more agents We study system

p erformance as more agents are added The second quite complex task involves nding a

CHAPTER PARTIALLY OBSERVABLE MDPS

Figure A partially observable maze The task is to nd a path leading from start S to

goal G The optimal solution requires steps and at least three reactive agents The gure

shows a possible suboptimal solution that costs steps Asterisks mark appropriate subgoals

key which op ens a do or blo cking the path to the goal The optimal solution which requires

at least agents costs steps The third task shows that HQGlearning can b e used for

inductive transfer problem solving knowledge acquired in one task can b e used to sp eedup

solving another similar but more complex task

Learning to Solve a Partially Observable Maze

Task The rst exp eriment involves the partially observable maze shown in Figure

The system has to discover a path leading from start p osition S to goal G There are four

actions with obvious semantics go west go north go east go south p ossible observations

are computed by adding the eld values of blo cked elds next to the agents p osition

where the eld value of the west north east and south eld is and resp ectively

the agent can only see which of the adjacent elds are blo cked Although there are

p ossible agent p ositions there are only highly ambiguous inputs Not all of the p ossible

observations can o ccur in this maze This means that the system may o ccasionally generate

unsolvable subgoals such that control will never b e transferred to another agent There is

no deterministic memoryfree p olicy for solving this task Sto chastic memoryfree p olicies

such as the one describ ed in Jaakkola Singh and Jordan will also p erform p o orly For

instance input stands for elds to the left and to the right of the agent are blo cked The

optimal action in resp onse to input dep ends on the subtask at the b eginning of a trial it

is go north later go south near the end go north again Hence at least three reactive

agents are necessary to solve this POMDP

Reward function Once the system hits the goal it receives a reward of Otherwise

the reward is zero The discount factor

Parameters and exp erimental setup We compare systems with and

agents and noisefree actions We also compare systems with and agents whose actions

selected during learningtesting are replaced by random actions with probability One

exp eriment consists of simulations of a given system Each simulation consists of

HQLEARNING

1000 1000 3 agents 4 agents 4 agents 8 agents 6 agents 12 agents 800 8 agents 800 optimal 12 agents optimal 600 600

400 400

200 200

0 0

0 5000 10000 15000 20000 0 5000 10000 15000 20000

Figure left HQlearning results for the partially observable maze for and

agents We plot average test run length against trial numbers means of simulations

The system almost always converges to nearoptimal solutions Using more than the required

agents tends to improve performance Right results for and agents whose actions

are corrupted by noise In most cases they nd the goal although noisy actions decrease

performance

trials T is After every th trial there is a test run during which actions and

max

subgoals with maximal table entries are selected P is set to If the system do es not

max

nd the goal during a test run then the trials outcome is counted as steps

After a coarse search through parameter space we use the following parameters for all

exp eriments i T for b oth HQtables and Qtables

Q HQ i

P is set to and linearly increased to All table entries are initialized with

max

Results Figure A plots average test run length against trial numbers Within

trials most systems almost always nd nearoptimal deterministic p olicies Consider Table

The largest systems are always able to decomp ose the POMDP into a sequence of RPPs

The average number of steps is close to optimal In approximately out of cases the

optimal step path is found In most cases one of the step solutions is found Since the

number of step solutions is much larger than the number of step solutions there are

many more appropriate subgoal combinations this result is not surprising

Systems with more than agents are p erforming b etter here the system prots from

having more free parameters More than agents do not help though All systems p erform

signicantly b etter than the random system which nds the goal in only of all step

trials

In case of noisy actions the probability of replacing a selected action by a random action

is the systems still reach the goal in most of the simulations see Figure B In the

nal trial of each simulation systems with and agents nd the goal with probabilities

of and p ercent resp ectively There is no signicant dierence b etween smaller and

larger systems

CHAPTER PARTIALLY OBSERVABLE MDPS

System Av steps Goal Av sol Optimal

agents

agents noise

Random

Table HQlearning results for random actions replacing the selected actions with proba

bility and Al l table entries refer to the nal test trial The nd column lists average

trial lengths The rd column lists goal hit percentages The th column lists average path

lengths of solutions The th column lists percentages of simulations during which the optimal

path is found

We also studied how the system adds agents during the learning pro cess The agent

system found solutions using agents in simulations

Using more agents tends to make things easier During the rst few trials agents were used

on average during the nal trials Less agents tend to give b etter results however Why

Systems that fail to solve the task with few subgoals start using more subgoals until they

b ecome successful But the more subgoals there are the more p ossibilities to comp ose paths

and the lower the probability of nding a shortest path in this maze

Exp erimental analysis How do es the agent system discover and stabilize subgoal

combinations SCs The only optimal step solution uses observation as the rst subgoal

th top eld and observation as the second southwest inner corner There are several

step solutions however eg SCs

Figure shows how SCs evolve by plotting them every trials observation stands

for an unsolved subgoal The rst SCs are quite random and the second agent often

is not able to achieve its subgoal at all Later however the system gradually fo cuses on

successful SCs Although useful SCs are o ccasionally lost due to exploration of alternative

SCs near simulation end the system converges to SC

The goal is hardly ever found prior to trial the Figure do es not show this Then

there is a sudden jump in p erformance most later trials cost just steps From this

moment on observation is used as second subgoal in more than of all cases and the

goal is found in ab out The rst subgoal tends to vary among observations and

Finally around trials the rst subgoal settles down on observation although

observation would work as well

The Key and the Do or

Task The second exp eriment involves the maze shown in Figure Starting

at S the system has to fetch a key at p osition K move towards the do or the

shaded area which normally b ehaves like a wall and will op en disapp ear only if the agent

is in p ossession of the key and pro ceed to goal G There are only dierent highly

HQLEARNING

3-agent system

#Trials

15000

10000

5000

16 12 0 8 4 8 4 12 0 Subgoal 2

Subgoal 1 16

Figure Subgoal combinations SCs generated by a agent system sampled at intervals

of trials Initial ly many dierent SCs are tried out After trials HQ explores less

and less SCs until it nal ly converges to SC

ambiguous inputs the key do or is observed as a free eld wall The optimal path takes

steps

Reward function Once the system hits the goal it receives a reward of For all

other actions there is a reward of There is no additional intermediate reward for taking

the key or going through the do or The discount factor

Parameters The exp erimental setup is analogous to the one in section We use

systems with and agents and systems with agents whose actions are corrupted by

dierent amounts of noise and i T P is

Q HQ i max

linearly increased from to Again for b oth HQtables and Qtables and all table

entries are zeroinitialized One simulation consists of trials

Results We rst ran thousandstep trials of a system executing random actions

It never found the goal Then we ran the random system for step trials The

shortest path ever found to ok steps We observe goal discovery within steps and

without action p enalty through negative reinforcement signals for each executed action is

very unlikely to happ en

Figure A and Table show HQlearning results for noisefree actions Within

CHAPTER PARTIALLY OBSERVABLE MDPS

Figure A partially observable maze containing a key K and a door grey area Starting

at S the system rst has to nd the key to open the door then proceed to the goal G The

shortest path costs steps This optimal solution requires at least three reactive agents The

number of possible world states is

1000 1000 3 agents 5% noise 4 agents 10% noise 6 agents 25% noise 800 8 agents 800 optimal optimal

600 600

400 400

200 200

0 0

0 5000 10000 15000 20000 0 5000 10000 15000 20000

Figure left HQlearning results for the key and door problem We plot average test

run length against trial number means of simulations Within trials systems

with and agents nd good deterministic policies in and of the

simulations Right HQlearning results with an agent system whose actions are replaced by

random actions with probability and

trials go o d deterministic p olicies are found in almost all simulations Optimal step paths

are found with agents in of all simulations During the few runs

that did not lead to go o d solutions the goal was rarely found at all This reects a general

HQLEARNING

System Av steps Goal Av sol Optimal

agents

agents noise

Random

Table Results of HQlearning simulations for the key and door task Al l table

entries refer to the nal test trial The second column lists average trial lengths The third

lists goal hit percentages The fourth lists average path lengths of solutions The fth lists

percentages of simulations during which the optimal step path was found HQlearning

could solve the task with a limit of steps per trial Random search needed a step

limit

problem in the POMDP case exploration issues are trickier than in the MDP case much

remains to b e done to b etter understand them

If random actions are taken in of all cases the agent system still nds

the goal in of the nal trials see table In many cases long paths

steps are found The b est solutions use only steps though Interestingly

a little noise eg decreases p erformance a lot but much more noise do es not lead to

much worse results

Inductive transfer

Partial observability is not always a big problem For example it may facilitate generalization

and knowledge transfer if dierent world states leading to similar inputs require similar

treatment In the next exp eriment we exploit this p otential for incremental learning the

system learns to transfer knowledge from a relatively simple mazes solution to a more complex

ones

Task First the system is trained to solve the maze from the rst exp eriment see Figure

After trials the system is placed in the more complex maze shown in Figure

A without resetting its Q and HQtables Then we sp end an additional trials on

solving the second maze We compare the results to those obtained by learning from scratch

using only the more dicult maze for training

Reward function Once the system hits the goal it receives a reward of Once it

bumps against a wall the reward is All other actions are p enalized by negative reward

The discount factor is

Parameters and exp erimental setup We test systems with and agents for b oth

transfer learning and learning from scratch We use HQG the gatebased HQ system which

leads to more stable learning p erformance which is an advantage for incremental learning

although the learning rate is slightly lower and it nds optimal p olicies less often

One exp eriment consists of simulations of a given system The maximal number of

steps p er trial is Every trials there is a test run

After a coarse search through parameter space we use the following parameters for all

CHAPTER PARTIALLY OBSERVABLE MDPS

Q HQ

comp etitors i T for b oth HQtables and Qtables

P is linearly increased from in the b eginning to at the end of the rst maze and

max

then from to at the end of the simulation now consisting of trials instead

of For systems which learn from scratch P is linearly increased from in the

max

b eginning to at the end of a simulation also consisting of trials

RESULTS WITH KNOWLEDGE TRANSFER 1000 Transfer 6-agents Transfer 8-agents optimal Scratch 6-agents 800 Scratch 8-agents

600

400 average nr steps

200

0 S 0 8000 16000 24000 32000 40000

nr epochs

Figure A The system is rst trained on the maze from Figure then moved to the

similar but more complex partially observable maze depicted above There are possible

ambiguous observations The optimal solution costs steps B HQlearning results for

the incremental learning task for and agents with and without know ledge transfer The

transferbased systems are trained for trials on the maze from Figure Then they are

moved to the maze from Figure A for which they have to learn additional observation

action pairs We plot average test run length against trial epoch numbers After being

primed by the rst task the system quickly learns good policies for the second maze

Results Figure B plots average test run length against trial numbers for systems

with and without knowledge transfer The plot shows p erformance on the more complex maze

except for the rst trials of transferbased systems which corresp ond to p erformance

on the simpler maze All systems almost always nd nearoptimal deterministic p olicies

Transferbased systems with and agents nd deterministic p olicies leading to the goal

in and of the simulations resp ectively Both systems that learn from scratch

nd go o d deterministic p olicies in of the simulations Obviously knowledge transfer is

b enecial the system quickly renes solutions to the rst partially observable maze to obtain

go o d solutions for the second maze For instance after b eing moved to the second maze the

agent system has found times a go o d solution after trials

Average trial length after trials in the second maze is steps whereas the optimal

solution requires steps the worst solution ever found requires steps Note that the

system often do es not immediately nd the solution to the new maze This is largely due to

the new observation in the left corridor for which a new action gonorth has to b e learned

Why do es the system not always nd optimal solutions but sometimes gets stuck in

lo cal maxima One reason may b e that lo cal maxima are close to global maxima the

HQLEARNING

cumulative discounted reward for a step solution is at the starting state whereas

the cumulative reward for a step solution is only slightly less namely

Discussion

HQs advantages

Most POMDP algorithms need a priori information ab out the POMDP such as the

total number of environmental states the observation function or the action mo del

HQ do es not HQ only has to know which actions can b e selected

Unlike history windows HQlearning can in principle handle arbitrary time lags b e

tween events worth memorizing To fo cus this p ower on where it is really needed short

history windows may b e included in the agent inputs to take care of the shorter time

lags This however is orthogonal to HQs basic ideas

To reduce memory requirements HQ do es not explicitly store all exp eriences with dier

ent subgoal combinations Instead it estimates the average reward for choosing particu

lar subgoalRP combinations and stores its exp eriences in a single sequence of Q and

HQtables These are used to make successful subgoalRP combinations more likely

HQs approach is advantageous in case the POMDP exhibits certain regular structure

if one and the same agent tends to receive RPPs achievable by similar RPs then it can

reuse previous RPP solutions

HQlearning can immediately generalize from solved POMDPs to similar POMDPs

containing more states but requiring identical actions in resp onse to inputs observed

during subtasks see the third exp eriment

HQs current limitations

Like Qlearning HQlearning allows for representing RPs and subgoal evaluations by

function approximators other than lo okup tables We have not implemented this com

bination however

An agents current subgoal do es not uniquely represent previous subgoal histories This

means that HQlearning do es not really get rid of the hidden state problem HSP

HQs HSP is not as bad as Qs though Qs is that it is imp ossible to build a Q

p olicy that reacts dierently to identical observations which may o ccur frequently

Appropriate HQp olicies however do exist

To deal with this HSP one might think of using subgoal trees instead of sequences All

p ossible subgoal sequences are representable by a p olicy tree whose branches are lab eled

with subgoals and whose no des contain RPs for solving RPPs Each no de stands for

a particular history of subgoals and previously solved subtasks there is no HSP any

more Since the tree grows exp onentially with the number of p ossible subgoals however

it is practically infeasible in case of large scale POMDPs

In case of noisy observations transfer of control may happ en at inappropriate times A

remedy may b e to use more reliable inputs combining successive observations

CHAPTER PARTIALLY OBSERVABLE MDPS

In case of noisy actions an inappropriate action may b e executed right b efore or after

passing control The resulting new subtask may not b e solvable by the next agents RP

A remedy similar to the one mentioned ab ove may b e to represent subgoals as pairs of

successive observations Another p ossibility is to create an architecture which allows

for returning control

If there are many p ossible observations then subgoals will b e tested infrequently This

may delay convergence To overcome this problem one might either try function approx

imators instead of lo okup tables or let each agent generate a set of multiple alternative

subgoals In the latter instance once a subgoal in the set is reached control is trans

ferred to the next agent

HQ has a severe exploration problem It relies on random walk b ehavior for nding

the goal the rst time which can b e quite exp ensive for some problems in the worst

case the agents needs exp onential search time Whitehead Using actionp enalty

or counting metho ds for POMDPs may result in even worse cyclic b ehavior Thus

b etter ways should b e found for POMDP exploration Possibilities include selecting a

consistent action for the same observation so that eg counting can b e used or using

p olicy trees as mentioned ab ove so that the HSP completely disapp ears It remains

a challenging issue to nd exploration algorithms which work well for a large variety of

POMDPs and algorithms solving them however

Related Work

Other authors prop osed hierarchical reinforcement learning techniques eg Schmidhuber

b Singh Dayan and Hinton Tham Sutton b and Thrun

and Schwartz Their metho ds however have b een designed for MDPs Since the

current fo cus is on POMDPs this section is limited to a summary of previous POMDP

approaches and metho ds related to HQ due to their use of abstract b ehaviors instead of only

single actions

Recurrent neural networks Schmidhuber c uses two interacting gradientbased

recurrent networks for solving POMDPs The mo del network serves to mo del predict the

environment the other one uses the mo del net to compute gradients maximizing reinforcement

predicted by the mo del this extends ideas by Nguyen and B Widrow and Jordan and

Rumelhart To our knowledge this work presents the rst successful reinforcement

learning application to simple nonMarkovian tasks eg learning to b e a ipop Lin

also uses combinations of controllers and recurrent nets He compares timedelay

neural networks TDNNs and recurrent neural networks

Despite their theoretical p ower standard recurrent nets run into practical problems in

case of long time lags b etween relevant input events Although there are recent attempts at

overcoming this problem for time series prediction eg Schmidhuber Hihi and Bengio

Ho chreiter and Schmidhuber Lin Horne and Giles In general learning to

control is also more complex since the system do es not only have to predict but also make

the right choices Another imp ortant dierence is that the system itself is resp onsible for

the generated tra jectories Therefore if some tra jectories need little memory whereas other

b etter tra jectories need more memory the latter may have much smaller probability of b eing

RELATED WORK

found Furthermore exploration and system adaptions can cause previously acquired memory

to b ecome useless since tra jectories may change signicantly

Map learning An altogether dierent approach for solving POMDPs is to learn a map of

the environment This approach is currently considered very attractive and esp ecially suitable

for navigating mobile rob ots where the agent has a priori information ab out the result of its

actions the o dometry is known Thrun successfully applies a maplearning algorithm

to his rob ot Xavier which rst learns a gridbased map of an oce building environment and

then compiles this gridbased map into a top ological map to sp eed up goal directed planning

Map learning approaches may suer when state uncertainty and o dometry errors are large

however Therefore some mobile rob ots are equipp ed with multiple sensors and inputs from

these sensors are combined to decrease the uncertainty ab out the state Van Dam

describ es techniques for fusing the data from dierent sources in order to learn environmental

mo dels

Bayse Dean and Vitter discuss the complexity of map learning in uncertain en

vironments If there is no uncertainty in observations or actions in the environment then

map learning is simple we can just try all unexplored edges In case of uncertainty in

b oth actions and observations they prove the existence of PAC algorithms but restrict the

environment in two ways the agent needs to know the direction in which it went after

executing an action although it do es not have to know to which unintended state it went

this is needed for b eing able to nd the way back and there should exist some set of

landmarks with unique observations

Hidden Markov Mo dels McCallums utile distinction memory is an extension of

Chrismans p erceptual distinctions approach which combines Hidden Markov Mo dels

HMMs and Qlearning The system is able to solve particular small noisy POMDPs by

splitting inconsistent HMM states whenever the agent fails to predict their utilities but

instead exp eriences quite dierent returns from these states The approach cannot solve

problems in which conjunctions of successive p erceptions are useful for predicting reward

while indep endent p erceptions are irrelevant HQlearning do es not have this problem it

deals with p erceptive conjunctions by using multiple agents if necessary

Belief state Some authors have prop osed sp eeding up computing solutions to POMDPs

using b elief states by using function approximators Boutilier and Poole use Bayesian

networks to represent POMDPs and use these more compact mo dels to accelerate p olicy

computation Parr and Russell use gradient descent metho ds on a continuous rep

resentation of the value function although the use of a function approximator makes their

metho d an heuristic algorithm Their exp eriments show signicant sp eedups on certain small

problems DAmbrosio uses K abstraction to discretize b elief state space and then uses

Qlearning to learn value functions He shows how his algorithm can nd solutions to diag

nosis problems containing states A b elief network is used for mo deling the problem

The results show that the metho d can quickly improve its p olicy although the success of the

algorithm may heavily dep end on the amount of uncertainty the agent has ab out the true

state

Memory bits Littman uses branchandbound heuristics to nd sub optimal

memoryless p olicies extremely quickly To handle mazes for which there is no safe determin

istic memoryless p olicy he replaces each conventional action by two actions each having the

additional eect of switching a memory bit on or o Go o d results are obtained with a toy

problem Thus the metho d directly searches in p olicy space It may have problems if only

few deterministic p olicies will lead to the goal Therefore to b etter evaluate p olicies Littman

CHAPTER PARTIALLY OBSERVABLE MDPS

started the agent from each p ossible initial state

Cli and Ross use Wilsons classier system ZCS for POMDPs There

are memory bits which can b e set and reset by actions ZCS is trained by bucketbrigade and

genetic algorithms The system is rep orted to work well on small problems but to b ecome

unstable in case of more than one memory bit Also it is usually not able to nd optimal

deterministic p olicies Wilson recently describ ed a more sophisticated classier system

which uses prediction accuracy for calculating tness and a genetic algorithm working in

environmental niches His study shows that this makes the classiers more general and more

accurate

Martin also used memory bits in his system Candide which could b e put on or o

indep endently by dierent actions He used Q and Monte Carlo exp eriments to evaluate

actions and was quite successful in solving large deterministic POMDPs He used a clever

exploration metho d which did not allow the agent to choose a dierent action for a sp ecic

observation if that observation is seen multiple times in an exp eriment

One p ossible problem with memory bits is that tasks such as those in Section require

switching ono memory bits at precisely the right moment and keeping them switched

ono for long times During learning and exploration however each memory bit will b e very

unstable and change all the time algorithms based on incremental solution renement will

usually have great diculties in nding out when to set or reset it Even if the probability of

changing a memory bit in resp onse to a particular observation is low it will eventually change

if the observation is made frequently Like HQlearning Candide did not suer a lot from

such problems These systems take care that the internal memory is not changed to o often

which makes them much more stable

Program evolution with memory cells Certain techniques for automatic program

synthesis based on evolutionary principles can b e used to evolve shortterm memorizing pro

grams that read and write memory cells during runtime eg Teller A recent such

metho d is Probabilistic Incremental Program Evolution PIPE Salustowicz and Schmid

huber PIPE iteratively generates successive p opulations of functional programs ac

cording to an adaptive probability distribution over all p ossible programs On each iteration

it uses the b est program to rene the distribution Thus it sto chastically generates b etter and

b etter programs A memory cellbased PIPE variant has b een successfully used to nd well

p erforming sto chastic p olicies for partially observable mazes On serial machines however

their evaluation tends to b e computationally much more exp ensive than HQ

Learning p olicy trees Rings system constructs a p olicy tree implemented in

higher order neural networks To disambiguate inconsistent states new higherlevel no des are

added to incorp orate information hidden deep er in the past The system is able to quickly

solve certain nonMarkovian maze problems but often is not able to generalize from previous

exp erience without additional learning even if the optimal p olicies for old and new task are

identical HQlearning however can reuse the same p olicy and generalize well from previous

to similar problems

McCallums Utree is quite similar to Rings system It uses prediction sux trees

see Ron Singer and Tishby in which the branches reect decisions based on current or

previous inputsactions Qvalues are stored in the leaves Statistical tests are used to decide

whether groups of instances in a leave corresp ond to signicantly dierent utility estimates

If so the cluster is split McCallums recent exp eriments demonstrate the algorithms ability

to control a simulated car by learning to switch lanes on a highwaytask involving a huge

number of dierent inputs while still containing hidden state

RELATED WORK

Another problem with Rings and McCallums approaches is that they have to store the

whole sequence of observations and actions Hence for large time windows used for sampling

observations the metho ds will suer from space limitations

Consistent Representations Whitehead uses the Consistent Representation

CR Metho d to deal with inconsistent internal states which result from p erceptual alias

ing CR uses an identication stage to execute p erceptual actions which collect the infor

mation needed to dene a consistent internal state Once a consistent internal state has b een

identied actions are generated to maximize future discounted reward Both identier and

controller are adaptive One limitation of his metho d is that the system has no means of re

membering and using any information other than that immediately p erceivable HQlearning

however can prot from remembering previous events for very long time p erio ds

Levin Search Wiering and Schmidhuber use Levin search LS through program

space Levin to discover programs computing solutions for large POMDPs LS is of

interest b ecause of its amazing theoretical prop erties for a broad class of search problems

it has the optimal order of computational complexity For instance supp ose there is an

algorithm that solves a certain type of maze task in O n steps where n is a p ositive

integer representing the problem size Then LS will solve the same task in at most O n

steps Wiering and Schmidhuber show that LS supplied with a set of conditional plans

abstract b ehaviors can solve very large POMDPs for which the algorithmic complexity Li

and Vitanyi of the solutions is low

SuccessStory Algorithm Wiering and Schmidhuber also extend LS to obtain

an incremental metho d for generalizing from previous exp erience adaptive LS To guar

antee that the lifelong history of p olicy changes corresp onds to a lifelong history of reinforce

ment accelerations they use the successstory algorithm SSA eg Schmidhuber Zhao and

Schraudolph a Zhao and Schmidhuber Schmidhuber Zhao and Wiering b

This can lead to further signicant sp eedups SSA is actually not LSsp ecic but a general

approach that allows for plugging in a great variety of learning algorithms For instance

in additional exp eriments with a selfreferential system that embeds its p olicymo difying

metho d within the p olicy itself SSA is able to solve huge POMDPs using the prop er initial

bias Schmidhuber et al a

Using multiple abstract b ehaviors

Using abstract b ehaviors instead of single actions makes it p ossible to plan on a higher level

and compress solutions thereby facilitating p olicy construction considerably Some early

work in this direction has b een p erformed by Singh who developed comp ositional Q

learning Other early work was done by Thrun who developed Skills a system which

was able to learn and reuse particular reactive p olicies HQlearning went a step b eyond these

approaches however since it showed how abstract b ehaviors could directly learn from the

Qvalues of subsequent b ehaviors without taking single action values into account

Like HQlearning Humphrys Wlearning uses multiple agents which use Qlearning

to learn their own b ehaviors A ma jor dierence is that his agents skills are prewired dif

ferent agents fo cus on dierent input features and receive dierent rewards Go o d reward

functions are found by genetic algorithms An imp ortant goal is to learn which agent to select

for which part of the input space Eight dierent learning metho ds implementing co op erative

and comp etitive strategies are tested in a rather complex dynamic environment and seem to

lead to reasonable results

CHAPTER PARTIALLY OBSERVABLE MDPS

Digney describ es a nested Qlearning technique based on multiple agents learning

indep endent reusable skills To generate quite arbitrary control hierarchies simple actions

and skills can b e comp osed to form more complex skills Learning rules for selecting skills

and for selecting actions are the same however This may make it hard to deal with long

reinforcement delays In exp eriments the system reliably learns to solve a simple maze task

Sutton Precup and Singh and also Parr and Russel discuss approaches based

on using abstract b ehaviors for sp eeding up nding solutions to Markov decision problems

Using Suttons terminology here these systems consist of options abstractmacro actions

instead of single step actions which are asso ciated with a termination condition Executing

an option can take a variable amount of time and that is why the problems are mo deled

as semiMarkov decision pro cesses Precup Sutton and Singh prove that RL with

options still converges as long as we can guarantee that options stop Results with Qlearning

and abstract b ehaviors show that their metho ds signicantly sp eed up p olicy construction

Although these approaches have not yet b een used for solving POMDPs the use of options

could just like HQs subgoals get rid of a lot of hidden state

Martin developed Candide a system which rst uses Qlearning to learn a set

of b ehaviors after which the system learns to hierarchically comp ose them to learn to solve

more dicult problems By rst teaching the system a set of basic tasks it was later able to

learn to combine them in general p olicies for solving quite dicult blo ck world problems

Dietterich developed the MAXQ value function decomp osition metho d for hier

archical reinforcement learning It also consists of multiple abstract b ehaviors which are

combined through learning Qno des in a p olicy tree Each Qno de determines which branch

of the tree is selected and is followed by a primitive action p ossibly an abstract b ehavior

and a descendant if the no de is not a leave no de In this way highly complex b ehaviors can

b e learned by rst engineering by hand useful abstract p olicy trees and then learning the

Qvalues

The dierence of these approaches compared to HQlearning is that HQlearning learns to

decomp ose the task into subtasks and at the same time learns b ehaviors solving the subtasks

without a priori knowledge ab out the tasks If solutions do not work HQlearning will

change subtasks and try to learn new b ehaviors solving them Most other systems use either

a prewired hierarchical decomp ositions or prewired skills which makes learning faster but

requires human engineering

HQlearning can also b e seen as a system which combines learning a structure and the

parameters of the structure The structure is adapted to make learning the parameters easier

Learned parameters which work well ensure that the structure will remain more stable In this

way the coupled system can have advantages compared to other onesided learning systems

Eg we also tried using a prewired subtask decomp osition so that only p olicies needed to b e

learned For this we used the rst maze Figure and used the subgoals b elonging to the

decomp osition of the optimal path SCs and Surprisingly this resulted in worse training

p erformance than learning the decomp osition at the same time This might b e explained

by the fact that the number of goalnding solutions had b ecome much smaller than that of

the coupled system Furthermore the agent was restricted to learning the optimal solution

Therefore its learning dynamics suered from the additional constraints

CONCLUSION

Conclusion

POMDPs are a dicult class of problems Exact algorithms for solving them are infeasible

for large problems and most heuristic metho ds do not scale up very well We developed a

new algorithm HQlearning which is based on learning to decomp ose a problem into a set

of smaller problems solvable by dierent learning agents This metho d has b een shown to

solve a number of semilarge deterministic POMDPs Since HQ scales up with the number of

required agents and not with the number of states it can in theory solve very large POMDPs

for which there exist low complexity solutions HQ do es not solve all problems in particular

it has problems with uncertainty in the actions and observations although it may still nd

comp etitive solutions for such problems HQlearning also supp oses a xed starting state

which is another limitation

POMDP algorithms are based on the construction of an internal state IS based on the

sequence of observations Ecient POMDP algorithms should therefore fo cus themselves

on controlling the dynamics of their internal state Based on recollected exp eriences an

algorithm could try to transform its IS using mental actions If an algorithm could learn

to control these mental actions an agent might b e able to learn the skill of reection The

imp ortance of reection for intelligent b ehavior may b e clear but many problems remain to

b e solved until it b ecomes realistic One imp ortant question is how mental states can b e

constructed as long as they are not automatically asso ciated with obtaining reward

Finally there is a need for algorithms which can explore POMDPs more eciently Even

map learning algorithms usually rely on random walk b ehavior for nding some landmark

state Bayse et al As long as exploration issues are not resolved we cannot exp ect

to obtain ecient heuristic metho ds for a large number of POMDPs

Reinforcement learning approaches using the construction of a priori p olicy trees or ab

stract b ehaviors can play an imp ortant role for solving more dicult RL problems Current

investigations in this direction are already b eing made on a variety of problems and show

promising results eg Sutton et al Dietterich HQ is related to such metho ds

as well as to exact p olicy tree construction metho ds Kaelbling et al since it allows

for using p olicy trees where each no de is represented as a b ehavior reactive p olicy and tran

sitions b etween no des are stored by subgoals observations Thus HQlearning can store the

optimal p olicy for POMDPs given a unique initial b elief state Furthermore HQ can allow

for compression of p olicy trees since often the same action for a particular observation may

b e optimal in a large region of the state space Using p olicy no des with M subgoals p er

p olicy transitions from one no de to the next we might b e able to decrease the complexity

M T

jO j jO j

where K is the length of the to OjAj of naive exact algorithms from OjAj

longest subgoal sequence We can see the large gain in case the horizon T is large and the

number of subgoals M and the length of subgoal sequences K is small More investigation

is needed to combine p olicy no des with such exact algorithms however

CHAPTER PARTIALLY OBSERVABLE MDPS

Chapter

Function Approximation for RL

In the rst part of this thesis we have describ ed RL metho ds for nite state spaces for which

it is p ossible to exactly store the optimal value function with lo okup table representations

For highdimensional or continuous state spaces however we need to employ function ap

proximators for compactly representing an approximation of the value functions

Function approximators FAs such as neural networks can b e used to represent the value

or Qfunction by mapping an input vector to a Qvalue FAs consist of many adjustable

parameters which are trained on learning examples to minimize the FAs approximation error

of the target function A very useful prop erty of FAs is that they are able to generalize

if we train a FA on a set of training examples the FA can interpolate b etween or even

extrap olate from them to create a mapping from inputs to outputs for the complete input

space Learning a go o d approximation is very dicult Judd however It can take a

huge amount of time and even b e unsuccessful We call this the learning problem However

sometimes it is p ossible to learn very useful value functions for huge state spaces after training

the FAs on a relatively small number of training examples This was demonstrated by the

success of learning to play backgammon with neural networks Tesauro

Outline of this chapter In Section we will describ e three dierent function ap

proximators linear networks neural gas Martinetz and Schulten Fritzke and

CMACs Albus b Albus a Then in Section we describ e how they can b e

combined with direct RL metho ds In Chapter we have seen that the use of world mo dels

sp eeds up computing go o d value functions Therefore we would like to use world mo dels in

combination with function approximators as well In Section we will describ e how we can

combine mo dels with the function approximators describ ed in Section Then in Section

we will describ e an exp erimental study on using function approximators in a challenging

domain learning to play so ccer with multiple agents Here we compare dierent metho ds by

their ability to learn to b eat a prewired so ccer team In Section we conclude this chapter

with some closing remarks

Function Approximation

Principles of FAs To use function approximators we construct a udimensional input

vector x based on selecting characteristic features for describing the state of the system or

1 20

There are on the order of p ositions states on the backgammon b oard whereas TDGammon had

already learned a go o d p olicy after learning on ab out examples

CHAPTER FUNCTION APPROXIMATION FOR RL

agent Constructing an input vector is quite imp ortant since it provides the FA with all

information to make its estimations from and therefore particular input designs can ease the

burden of the learning problem When we are constructing an input vector it is imp ortant

not to have ambiguities see Chapter More generally we should try to circumvent having

to deal with small distances b etween input vectors which must b e mapp ed to very dierent

outputs Furthermore learning sp eed is largest if all input dimensions contribute something

to the output the output should b e sensitive to all inputs otherwise it is b etter to remove

the noncontributing inputs Usually we do not have a priori knowledge which enables us to

decide which inputs are imp ortant and thus we should examine the outputs sensitivity to an

inputs value after some amount of learning time

FAs receive the input vector x and compute a scalar output y it is straightforward to

extend the following presentation to using an output vector using the mapping F

y F x

The FA F consists of a number of adjustable parameters eg weights and pro cessing

elements no des neurons structured in a sp ecic topology architecture The top ology can

b e static or dynamic Static topologies are dened a priori by the engineer and during the

training phase only the parameters in the FA are changed Dynamic topologies develop and

adjust themselves during the learning pro cess Although dynamic structures could make

learning functions much easier with particular FAs Baum nding correct structures

is generally considered to b e a hard problem

Lo cal vs Global FAs There are many dierent function approximators and each kind

of FA has its own advantages and disadvantages One imp ortant characteristic for FAs is their

degree of lo cality We will call a FA a local model if the FA only uses a fraction eg less than

of all pro cessing elements for computing the output given an input Completely lo cal

mo dels such as lo okup tables only use a single pro cessing element eg a table entry for

computing the output Examples of lo cal mo dels are Kohonen networks Kohonen

neural gas Martinetz and Schulten Fritzke bumptrees Omohundro

Landelius decision trees and CMACs Albus b Albus a The counterpart

of lo cal mo dels are global models which use almost the entire representation for deriving the

output They usually interpolate more smo othly b etween training examples than lo cal mo dels

but also tend to overgeneralize more Examples of global mo dels are linear networks sigmoidal

feedforward neural networks trained with backpropagation Werbos Rumelhart et al

Hopeld Networks Hopeld and Boltzmann machines Hinton and Sejnowski

Linear Networks

Probably the simplest function approximator is a network which is linear in the comp onents

of x The linear network can b e written as

y w x b

If inputs which are lo cated near to each other require very dierent outputs we need to train the FA to

represent steep slop es which is dicult and hinders successful generalization

FUNCTION APPROXIMATION

where w is the udimensional weight vector and the parameter b is the bias Figure A

shows how a linear networks is used for regression tasks where we want to t a straight u

dimensional hyperplane through a number of examples Linear networks can only b e used

for learning linear functions however and that is why their expressive p ower is quite limited

eg for the regression problem demonstrated in Figure B linear networks cannot b e used fruitfully

x x

Figure A A linear networks approximation to a set of training examples data points

B Training examples which cannot be represented wel l by a single linear hyperplane

Training the network Given a set of training examples inputoutput pairs x d

i i

where i f M g M is the number of training examples we want to minimize the

mean squared error b etween the desired outputs d and the networks outputs y For online

i i

learning we derive the learning rule from the squared error function E

d y d w x E

i i i T i

Given a set of example transitions training patterns x d we can compute w in closed

i i

form First we collect all vectors x in the matrix X by putting all vectors as row vectors

in the matrix Thus the vector X denes the k row of the mbyu m is the number of

training pairs which should b e larger or equal to u matrix X Then we compute the weight

vector w as

T T

w X X X d

T T

where d is the mdimensional vector storing all desired outputs d d and X X X

is the pseudoinverse Mo orePenrose inverse of the matrix X we assume that X X

exists otherwise we have to use lim X X aI which always has an inverse for a

See Rao and Mitra for details

We can also iteratively nd a solution after dierentiating E with resp ect to w we get

the deltarule Widrow and Ho which decreases the error function making up dates

of the weight vector with learning rate

To simplify the following discussion we put the additional parameter b in the weight vector and add an

input to the inputvector which is always set to

We assume that the bias is part of the weight vector

CHAPTER FUNCTION APPROXIMATION FOR RL

w d y x

i i i

Instead of linear networks usually the much more p owerful multilayer neural networks

are used Such networks consist of at least one hidden layer with nonlinear activation

functions and are able to represent any Borel measurable function Cyb enko They

can b e trained by using the backpropagation algorithm Werbos Rumelhart et al

which implements gradient descent on the error function We refer to Krose and vd

Smagt for a full description of these more p owerful function approximators

Lo cal Function Approximators

Global function approximators such as linear or multilayer neural networks compute the

output using all adjustable parameters Furthermore they adapt all parameters on each

example This creates interference or forgetting problems after we have trained the FA to

learn a go o d approximation in some part of the input space and then train it to approximate

dierent parts of the input space the learning algorithm tries to use the same parameters to

decrease the error over a new input distribution so that the FA forgets what it had learned

b efore

Lo cal function approximators LFAs do not have this problem They consist of many

indep endent pro cessing elements which we will call pro cessing cells or simply cells Each

cell has a centre in the input space a vector and an output value Given some input we

compute cell activations based on the distance b etween cell centres and the input For this

some weighting rule and a distance function is used which ensures that closeby lo cated cells

are activated most Then cell outputs are combined by rst weighing them by the activities

of the cells and then summing them Finally cell centres and outputs are adapted on the

learning example where learning steps dep end on cell activities If we want to approximate

the function in some unknown parts of the input space most previously trained cells are not

activated since the distance of their centres to the new inputs is to o large Therefore they are

not adapted for these parts and the previously learned function approximation stays more or

less the same

Winner take all We can lo ok at such learning architectures as a comp etition b etween

the cells for b eing active in dierent parts of the input space Each cell tries to approximate the

target function in some region b est so that it will b ecome the most activated cell there LFAs

can use hard decision surfaces by implementing a winner take all WTA algorithm Using

WTA only the closest cell is activated and this cell will determine the output of the FA

Examples of LFAs which employ the WTA strategy are nearest neighbor learning vector

quantization LVQ Kohonen and rasters grids with xed or variable resolution

Mo ore and Atkeson When we use a nearest neighbor scheme with WTA cells to

divide the input space into subregions we say that we make a Voronoi tessellation of the

input space see Figure

Once we have partitioned the input space it b ecomes easy to learn output values The

output of each cell equals the average desired output of training examples falling inside a

cells region However learning a go o d partitioning is a hard problem and if we change

a partitioning eg by adding or removing a cell multiple output values of cells are not

anymore uptodate and need to b e reestimated

In case of multiple closest neurons we break ties arbitrarily

FUNCTION APPROXIMATION

Figure A A Voronoi tesselation of the input space derived from using WTA and the

locations of a set of cells shown as circles In between each two cells there is a hard decision

boundary The constructed tesselation determines those regions of the input space for which

a particular cell is used is activated for returning its output

Soft cell comp etition Instead of WTA cells we can also use a weighted combination of

the outputs of particular closeby lo cated cells as the nal output which results in a smo other

function approximation This is also referred to as soft comp etitive learning Nowlan

Examples of smo oth LFAs are k nearest neighbor with k Delauney triangulizations with

linear interpolations Omohundro Munos lo cally weighted regression Gordon

Atkeson Mo ore and Schaal Kohonen networks Kohonen and neural gas

Fritzke

Neural gas

We will now present a novel implementation of a smo oth growing neural gas architecture

We use a set of Z neurons cells fn n g initially Z Z They are placed in the

Z init

input space by assigning to each a centre w IR a udimensional vector Apart from the

centre each neuron n contains an output y

k k

We calculate the overall output of the FA by lo cally weighting the outputs of all neurons

although most will have a neglectable inuence on the nal output We calculate the

weighting factor g gate for each neuron n based on the cell centres and the environmental

k k

input x using

distw x

where IR is a userdened constant which determines the smo othness of the FA and

distw x is an arbitrary distance function of the family of L norms

L w x w i xi

Weighing inputs dierently for computing the distance by eg an additional parameter vector allows for

more useful decomp ositions of the input space although it also requires more parameters to b e learned

CHAPTER FUNCTION APPROXIMATION FOR RL

where w i and xi refer to the value of the i feature of the centervector and input resp ec

tively The overall output of the FA given input x is

g y y

j j

Learning Rules We want to maximize the probability of returning the correct output

value Since the output value dep ends on the lo cations of the neurons and on the output

values of the neurons we have to learn the network structure move the neurons to

lo cations where they help to minimize the overall error and learn correct output values

make individual neurons correctly evaluate the inputs for which they are used

In order to let the FA converge we will anneal the learning rate of neurons so that they

will stabilize over time For this we keep track of the overall usage of a particular neuron in

a resp onsibility variable C which is initially set to After each example each neuron n s

k k

resp onsibility variable C is adapted

C C g

k k k

Then C is used to determine the size of n s learning rate and also to determine conditions

k k

for adding a neuron

Learning Structure The structure of the neurons is of great imp ortance since

it determines which states are aggregated together and thus which family of functions can

b e represented We should keep the following principles in mind a neurons should b e

placed where they minimize the overall error and b we want to have most neurons a ner

resolution in regions with the largest fractions of the overall error

There are a number of dierent ways for learning the structure of neural gas architectures

all based on using particular op erations on the neurons The most common op erations are

moving adding and deleting or merging neurons

Fritzke presents a particular metho d for growing cell structures His metho d learns

a top ology by creating edges which link two neurons which together have b een closest to a

particular example His learning rule moves the closest neuron and its neighborho o d to the

input Furthermore his metho d incrementally adds neurons for which it keeps track of the

total error of neurons when they have won the WTA comp etition After a xed number of

example presentations a neuron is added near the neuron with the largest accumulated error

This neuron is initialized by interpolating from the nearest neurons and is assumed to b e able

to learn to decrease the error in that sp ecic region of the input space

Instead of using Fritzkes metho d we have created a dierent metho d which do es not

accumulate error over time but adds neurons when the error on a single particular example

is to o large The details are as follows If the error jd y j of the system is larger than

an errorthreshold T the number of neurons is less than Z and the closest neurons

E max

resp onsibility C exceeds the resp onsibility threshold T C grows with the density of the

k k

input distribution around neuron n then we add a new neuron n We set its lo cation

w to x and set the output value y d Finally we set Z Z Thus if the error

Z Z

on an example is large we immediately copy the example to a novel neuron The metho d

is a go o d metho d to immediately decrease the error of dicult regions in the input space

but it can also suer from noisy examples which causes newly added neurons to approximate

regions by a wrong output value although they could still adjust themselves later on

FUNCTION APPROXIMATION

If no neuron is added we calculate for each neuron n k f Z g a gatevalue h

k k

which reects the p osterior b elief that neuron n evaluates the input b est

g e

We then move each neuron n towards the example x according to

w w h x w

k k k k

where C is the system learning rate and is the learning rate decay factor

g g

k k

The eect of using the square of h instead of h is that neurons which are not a clear winner

k k

will not adapt themselves very fast Thus clear separations are made early on in the pro cess

whereas ner divisions of the input space emerge only gradually

Learning Qvalues We up date the output y of neuron n k f Z g by

k k

y y h d y

k k k k k

Note the similarities of our neural gas metho d with the hierarchical mixtures of exp erts

HME architectures Jacobs et al Jordan and Jacobs

Nearest neighbor search An imp ortant issue in lo cal FAs is to search for the set of

neurons which are activated for returning the output Even LFAs with smo oth comp etition

usually have only a small number of neurons which have non neglectable activations for

returning the output

The same problem called the dataretrieval problem is known in database systems Given

the input nding the closest neuronsdataitem is usually implemented by computing the

distance to all neurons and then selecting the closest ones The complexity of this is OZ u

where Z is the number of neurons and u the dimensionality of the input space If we have

thousands of neurons the input dimensionality is large and we have to return outputs for a

large number of queries inputs then the system gets very slow A solution to sp eed up the

search is to use prepro cessing and to store the neurons in a datastructure which allows for

faster online search

We can use datastructures and metho ds from computational geometry such as Voronoi

diagrams Okabe et al These metho ds signicantly sp eed up searching for the closest

neuron but the problem is that the size of such structures is exp onential in the dimensionality

of the input space

Kd trees Friedman et al Preparate and Shamos Mo ore are another

datastructure for sp eeding up nearestneighbor search To create a Kd tree we recursively

partition the p opulation of neurons by dividing the individuals up along a single input di

mension which is chosen to keep the tree more or less balanced The nearest neighbor is then

found by descending the tree to nd an initially b est candidate and then backtracking along

the tree to examine whether all alternative siblings are not closer while discarding siblings

which are surely not closer to the input than the current b est one Kd trees can b e used to

p erform searches in exp ected time close to mina l og Z uZ where a is a suitable constant

Mo ore p ersonal communication This means that searchtime is logarithmic in

Z but exp onential in the input dimensionality In practice the time requirements dep ends a

lot on the distribution of the data Omohundro Mo ore Constructing Kd trees

takes uZ l og Z prepro cessing time Preparate and Shamos

CHAPTER FUNCTION APPROXIMATION FOR RL

A problem of the ab ove metho ds is that they cannot b e easily combined with neural gas

neurons are moved around and added to the system so that the data structures used for

searching should change as well in an incremental way This makes using these structures

more exp ensive Omohundro describ es and exp erimentally evaluates some incremental

metho ds for constructing balltrees data structures very similar to Kd trees

A quite similar idea esp ecially suitable for radial basis functions is to use a hierarchy of

neurons where the higher level neurons are used for clustering lower level neurons Then

we can descend from the top layer to lower layers and use leaf neurons for the nal output

Although the primitive metho d is not guaranteed to return the closest neuron we could

decide to use this neuron anyway or to use backtracking eg used in bumptrees Omohundro

Landelius afterwards Such metho ds can sometimes save a lot of time see

exp erimental results with bumptrees in Omohundro Landelius

Another p ossibility which is an O metho d is to restrict the search among a xed set

of neighbors which are allowed to b e activated Eg we can construct a limited number of

edges to link neurons which are activated after each other or we can constrain the search by

using an a priori temp oral structure Although this metho d may work well it may b e the

case that many more neurons may b e needed since dierent neurons may b e implementing

the same function for the same region only the sequence of examples arriving at the region

was dierent

In our exp eriments we have used the simplest line search metho d although it resulted

sometimes in computationally very demanding simulations

CMACs

The third kind of FAs are also lo cal but are dierent from the ones describ ed previously

since they do not construct a tesselation based on the p ositions of a number of cells They

use a xed a priori decomp osition of the input space into subregions Usually the subregions

are structured in an ordered top ology such as hypercub es which makes searching for active

cells much faster compared to the metho ds describ ed earlier since random access instant

lo okup b ecomes p ossible This increased time eciency is as usual at the exp ense of a

huge increase of the storage space however A well known FA which uses for example

hypercub es to divide the input space into subregions is the CMAC Albus a A CMAC

Cereb ellar Mo del Articulation Controller uses lters mapping inputs to a set of activated

cells Each lter partitions the input space into subsections in a prewired way such that

each p ossibly multidimensional subsection is represented by exactly one discrete cell of the

lter For example a lter might consist of a nite number of cells representing an innite

set of colors represented by cub es with dimensions red blue and yellow and activate the

cell which encloses the current color input

CMACs uses multiple lters consisting of a number of cells with asso ciated output values

Applying the lters yields a set of activated cells a discrete distributed representation of the

input and their output values are averaged to compute the overall output

The dierence with vector quantization metho ds is that the partitioning is designed a

priori and that multiple cells are activated by using dierent views on the world

General remarks on lter design In principle the lters may yield arbitrary divisions

of the statespace such as hypercub es but also more complex partitionings which make use of

Higher level neurons are centered at the distribution of examples for which they are activated but nothing

is said ab out the p ositions of their lower level neurons

FUNCTION APPROXIMATION

prepro cessing may b e used To avoid the curse of dimensionality which o ccurs when one uses

all inputdimensions to construct hypercub es one may use hashing to group a random set of

inputs into an equivalence class or use hyperslices omitting certain dimensions in particular

lters Sutton Although hashing techniques may help to overcome storage problems

it collapses states together which can b e widely apart We exp ect that this will often not lead

to go o d generalization p erformance Therefore we prefer hyperslices which group inputs by

using subsets of all input dimensions Eg if we say that we see a green ob ject which is larger

than meters we could already infer that the ob ject might b e a tree although there may b e

other options as well such as a green house or a mountain If we add multiple descriptions

lters of the ob ject together and all descriptions agree with the ob ject we may b e certain

which ob ject we are examining

Computing the output The rst step in computing the output is applying all lters

to the input vector Each lter uses particular input dimensions to create its universe and

partitions that universe using hyperb oxes Given an input it pro jects the input vector in its

universe and examines which of its cells encloses the input

More formally a lter is represented as a multidimensional array Given an input x

the lter uses the relevant features to search for the cell out of fc c g where n is

n c

the number of cells in which the input can lie Applying all lters returns the active cells

ff f g where z is the number of lters

The output value y given input x is calculated by averaging the outputs of all activated

cells y f

k k

y y f z

k k

where y f is the output of the activated cell f of lter k

k k k

Learning rule Training CMACs is an extremely simple pro cedure we just adapt the

output values of all activated cells Thus for all activated cells we compute

d y f y f y f

k k k k k k

where is the learning rate

Filter design The design of the lters is very imp ortant since the construction of the

lters determines what can b e learned and how fast that can b e learned more cells need

more examples There are two imp ortant matters in designing lters Which features to

combine inside each lter This introduces bias on which relationships can b e learned

How coarsely to divide each lter into cells This determines the generalization b ehavior and

the attainable accuracy of a lter The p ossible choice of and is restricted by space

limitations since combining D input features each divided into n parts would mean that our

multidimensional cub e consists of n cells Note that there is no necessity to have a xed

uniform decomp osition We could also use adaptive variableresolution lters to optimize the

decomp osition of a lter into cells or activate a variable number of lters for dierent inputs

which makes richer descriptions p ossible for ob jects which require more sp ecication

Example of eective lter design Sometimes a particular lter design for CMACs

can b e very eective Eg consider Figure a which shows an empty maze with a starting

state and a goal state We can construct two lters for this maze one slicing the maze up

This is the worst case for which all cells can b e made active otherwise if some cells are never activated

an implementation based on hashing tables could signicantly decrease space requirements while keeping the

search for active cells ecient

CHAPTER FUNCTION APPROXIMATION FOR RL

in vertical slices Figure b and one for slicing the maze up into horizontal slices Figure c

G G

S S

Figure A A maze with starting state S and goal state G B One lter slices the maze

up into vertical slices the vertical position does not matter C The second lter slices

the maze up into horizontal slices Together the lters can be used for eciently learning a

solution

These lters do not allow for accurately representing the optimal value function but the

approximated value function can b e used for computing the optimal p olicy horizontal

lters except for the last will learn that going north is the b est action and vertical lters

except for the last learn that going east is the b est action Go o d lter design also means

that we have to learn less for the maze in Figure a and the usual actions we would

only have to learn instead of values

Overgeneralization If the maze would have blo cked states things get more compli

cated since the lters would overgeneralize and lter away the exceptions If there are few

blo cked states we can still hop e but not guarantee that the dynamics of the learning pro cess

will converge to a path which circumvents the blo cked states A more sophisticated approach

to deal with few exceptions is to construct additional lters which are made active in the

neighborho o d of these exceptions and which add something to or override the values of the

default lters With many blo cked states we would store almost all exceptions which would

mean that we would in principle combine b oth features in a lter and this would of course b e

equivalent to the lo okuptable representation

Generalization by sharing features CMACs also allow for another kind of general

ization by using multiple lters sharing the same feature space In Figure we show how

multiple lters on the same input creates a smo other generalization Using more than the

shown lters would make this even more pronounced Using many coarse lters leads to

a dynamical renement of the function each lter adds something while still b eing very

general

Function Approximation for Direct RL

We have introduced a number of dierent FAs which can b e used for approximating value

functions In this section we describ e how they can b e combined with direct RL metho ds and

we analyse p ossible problems arising from such combinations Even if we are well acquainted

with a particular FA we have to keep in mind that training a FA with RL instead of with

sup ervised learning puts dierent demands on the FA For instance in RL examples are gen

erated according to the p olicy and value function which themselves are changing Therefore

FUNCTION APPROXIMATION FOR DIRECT RL

A B

+ Filters

C D y y Training Examples x x

Function

Approximation

Figure A A design of two lters which splice a single input dimension in parts When

added they al low for a partitioning into dierent regions B A single lter which partitions

the input into regions C We examine what happens if we receive training examples

the circles the vertical distance to the line is the desired output We can see that the

combination of the two lters from A leads to some kind of generalization behavior D

With a single lter there is no generalization across cells The rst lter design results in a

smoother function due to its hierarchical approach some examples output is dynamically

rened if we add multiple lters

the function we try to learn is constantly changing which makes the task very hard On the

other hand in sup ervised learning we are usually interested in the most accurate approxi

mation whereas in RL we are much more concerned with the p erformance of the resulting

p olicy As we have seen in Chapter we can have go o d p olicies with quite p o or value function

approximations What is imp ortant however is that the preferential order b etween dierent

actions is reected in the value function

Representational issues We are interested in storing the Qfunction with FAs For

this we need to compute outputs Qx a for all actions a A We will do this by using

dierent functions for dierent actions F x

Qx a F x

A dierent way would b e to put the actions in the input but this makes learning dierent

mappings for dierent actions more dicult For continuous actions however we cannot use

one FA for one action and thus we would need to use dierent metho ds dep endent on the

used FA Eg for linear networks we could use the actions as part of the input and for

CMACs we could discretize the action space We assume in the following that we have a

nite set of actions

Thus with linear networks we will use jAj networks one for each action For a multilayer

CHAPTER FUNCTION APPROXIMATION FOR RL

feedforward neural network we can have dierent networks for each action or we can share

the hidden units of one network to approximate all Qvalues Lin compared b oth p os

sibilities and found that one network for one action worked b etter For other tasks sharing

the hidden layer may b e worthwhile however since the hidden units need to settle down

to extracting more characteristic features when they are used for describing the complete

problem domain shared by all actions and that may improve generalization p erformance

Eg Caruana added outputs for a prediction problem which puts an additional burden

on the learning task but lead to an improvement of the learned approximation for the re

quested output Baxter presents a general theory of bias learning based up on the idea

of learning multiple related tasks with a shared representation Generally sp eaking sharing

parameters for learning multiple actions is fruitful if the tasks share mutual information

The neural gas and CMAC architectures use for each neuron and ltercell Qfunctions

of the form Qn a and Q c a resp ectively

i j i j

Extending Q to function approximators

We can combine TD Qlearning and all Q metho ds with function approximators

One should b e careful choosing a combination and learning parameters however since the

resulting learning system may not always b e stable Eg if we allow a FA to adapt itself

quickly to maximize learning sp eed it can happ en that a sp ecic lucky tra jectory leads to a

large increase of values of states visited by the current tra jectory This may change the overall

value function dramatically and can lead to undesirable large changes of the p olicy Although

a large change of the approximation of a FA for sup ervised learning can also happ en we can

still relearn a previous go o d approximation since the training set is stored Since in RL the

p olicy generates the examples we may have much larger problems relearning go o d value

functions Furthermore since in RL the Qfunction changes and the learning examples are

drawn from a noisy source with often a sequence of highly correlated examples there can

b e many unreliable up dates or drift of the value function Thus our FAs should b e robust

enough to handle such problems

Oine Q To learn Qvalues we monitor agent exp eriences during a trial in a history

list with maximum size H At trial end the history list H is

max

H ffx 1 a 1 r 1 V x 1 g fx a r V x gg

t t t t

Here x is the input seen at time t a is the action selected at time t r is the reward received

t t t

at time t V x M ax fQx ag t denotes the end of the trial and t denotes the start

t a t

of the history list t if t H and t t H otherwise

max max

After each trial we calculate examples using oine Qlearning For the history list

new

H we compute desired Qvalues Q t for selecting action a given x t t t as

t t

follows

new

Q t r

new new

Q t r Q t V x

t t

Once the agent has created a set of oine Q training examples we train the FAs to

minimize the Qerrors

Oine multiagent Q In Chapter we have already discussed using multiple

autonomous agents simultaneously and presented two algorithms for using online Q to

FUNCTION APPROXIMATION FOR DIRECT RL

learn a shared value function We can also combine oine Q with multiple agents by

using a dierent historylist for each dierent agent In the exp eriments later in this chapter

we will use these historylists indep endently After having created multiple lists of training

examples we pro cess the historylists by dovetailing as follows we train the FAs starting

with the rst history list entry of agent then we take the rst entry of agent etc Once

all st entries have b een pro cessed we start pro cessing the second entries etc

Online Q for function approximators To combine online Q with FAs we use

eligibility traces The eligibility trace l w a is used for weight adjustable parameter w

k k

and action a as follows

ti i

l w a w a

k k

where w a is the gradient of Qx a with resp ect to the weight w

k k

Qx a

w a

and the online up date at time t b ecomes

w do w w e w a e l w a

t t

k k k k k

where as usual e r V x Qx a and e r V x V x

t t t t t t t t

Fast Q can also improve matters in case of LFAs Supp ose a LFA consists of jS j p ossible

state space elements eg neurons or total lter cells The Qvalues of z jS j elements

are combined to evaluate an input query Here the up date complexity of fast Q equals

jS j

in comparison to conventional online Q For O z jAj This results in a sp eedup of

global approximators z jS j so the metho d is not helpful For quantized state spaces with

ne resolution the gain can b e quite large however

Online Qlearning for CMACs We will show in detail how we combine fast

Q with a LFA in this case CMACs Using a xed partitioning a continuous input x is

t t t

g where z is the number of active f f transformed into a set of active features ff

features tilings

t t t

so f Before using Q f a we call the Local Update pro cedure for all features f

that their Qvalues are uptodate Local Update stays almost the same except that we use

dierent variable names

The Global Update pro cedure now lo oks as follows

CMAC Global Up datex a r x

t t t t

For i is to z a A Do

a Local U pdatef a

t z

az Q f V x max

i t a

e r V x Qx a

t t t t

e r V x V x

t t t t

t t

For i is to z Do

a a Local U pdatef

t t t

a e a f a Q f b Q f

t t t i i

i i i

t t

a a l f c l f

t t

i i

CHAPTER FUNCTION APPROXIMATION FOR RL

Remarks ab out online multiagent Q To use online Q with multiple agents

we could use the connecting traces approach from Chapter We have not done so in the

exp eriments in this chapter however We have used dierent eligibility traces for dierent

agents however since that is a necessity for multiagent learning

Notes on combining RL with FAs

In the following we will shortly describ e the representational and convergence issues for using

our dierent FAs for RL

Linear networks We know that if the number of input features is small compared to

the total number of distinct states we strongly limit the kind of value functions which can

b e represented accurately by linear networks However we can use many inputs and higher

order inputs as well which makes linear networks more p owerful Eg in the limit we use a

weight for each state and we have a tabular representation with known convergence results

The networks try to learn a linear approximation of the Qfunctions over the entire input

space minimizing the squared error over all training examples For many problems there may

b e small dierences b etween the Qvalues of dierent actions Eg p ostp oning the optimal

action for one time step may only cause the exp ected discounted future reward Qvalue to

decay by the discount factor if nothing changes we may still select the optimal action at

the next time step Thats why the learning dynamics may cause large p olicy changes which

makes p olicy evaluation hard Thus we should b e extremely careful setting the learning rate

for linear nets

Neural gas The approximation of a neural gas representation may b e very accurate

esp ecially with a large number of neurons Since RL b ecomes much slower when we use many

neurons retrieval time increases linearly and learning time even worse with the number of

neurons there exists a tradeo b etween accuracy and learning time Therefore we have to

b e careful choosing the number of neurons or alternatively the growthrate of a neural gas

structure Note that just a few neurons can already b e sucient for learning a go o d p olicy

and make the learning problem a lot easier

For the neural gas we have to b e careful changing the structure If neurons move fast

around this may cause large p olicy changes which makes learning a more or less xed

value function very hard If we add a neuron we initialize the Qvalue of the selected action

according to the desired Qvalue computed by eg oine Q shown ab ove The Qvalues

of the other actions of the new neuron are computed by weighing the Qvalues of existing

neurons according to their p osterior probabilities h One diculty is that the decision to

add neurons is based on TDerrors which themselves may b e very noisy Since it is dicult

to decide whether the example is noisy or the FA is unable to learn a correct mapping we

just add neurons and hop e they will shap e themselves inside the structure by learning from

future examples

CMACs Using CMACs with direct RL may work quite robustly since it always maps

the same state to the same set of activated cells Furthermore CMACs may approximate

the value function arbitrarily well given a large number of lters and cells and is shown to

converge to the least squared error approximation on a set of training data given prop er

learning rate annealing Lin and Chiang In practice a lot of CMACs functionality

dep ends on the grouping of states If a set of states with highly dierent values is group ed

together by a cell then the learned mean value of these states may not b e very useful and

the cells up dates will have large variance If each cell groups states together with similar

WORLD MODELING WITH FUNCTION APPROXIMATORS

Qvalues then learning may b e quite fast

For RL this means that future tra jectories of states group ed by a cell should b e as similar

as p ossible In general this will not b e the case however Furthermore cells may b e biased

or rep elled from particular actions This can cause problems for learning the p olicy Eg

some actions which are only go o d in few states may never get selected Such problems can

b e overcome by making the lters suciently sp ecialized however

Convergence issues Tsitsiklis and van Roy pro of convergence of TD metho ds

for function approximators using linear combinations of xed basis functions note that the

linear networks and CMACs b elong to this class neural gas do not since the neurons are

not xed In their pro of they stress the imp ortance of sampling from the steadystate

distribution of the Markov chains This can b e done by generating tra jectories using the

simulatorreal environment Learning by making single simulation steps starting in a xed

set of preselected input p oints may cause divergence Boyan and Mo ore however

Finally Tsitsiklis and van Roy suggest that higher values of are likely to pro duce

more accurate approximations of the optimal value function This may imply as well that

TD metho ds travel along lower mean squared error tra jectories in parameter space for

larger values of Unfortunately little understanding of the convergent b ehavior of changing

p olicies as that of using Q with function approximators is available however

World Mo deling with Function Approximators

Combining dynamic programming with function approximators was already discussed by

Bellman who prop osed using quantization and loworder p olynomial interpolation for

approximately representing a solution Gordon a

Transition mo deling We can use a function approximator to represent not only the

value function but also the world mo del For this it is b est to use lo cal function approxima

tors Global function approximators may b e very useful for mo deling deterministic pro cesses

but have problems to mo del a variable number of outgoing transitions due to their rigid struc

ture Examples of using global FAs for world mo deling are Lins world mo deling approaches

for mo deling a complex survival environment and explanation based neural network

learning for learning a mo del of chess Thrun Although using these mo dels help ed

sp eeding up learning a go o d p olicy they cannot b e easily combined with DP algorithms

On the other hand if we partition the space into a nite set of regions with lo cal function

approximators we allow for a much higher variety of transitions and estimating transition

probabilities b ecomes more precise and easier Furthermore when we use discrete states we

have the advantage of b eing able to apply DPlike algorithms to compute the value function

Estimating transitions The largest problem which has to solved b efore DP can b e

used fruitfully in combination with LFAs is that we need a go o d partitioning of the input

space so that estimated transition and reward functions reect the underlying mo del well

Given the p ositions of the cells or neurons we can use sampling techniques to estimate

the transition function We may use Monte Carlo simulations to sample a state map this

to an active cell select an action use the simulator or environment to make a transition to

the successor state and map this back to the next active cell By counting how often each

transition o ccurs we can estimate the transition probabilities b etween cells and similarly we

can compute the reward function see Chapter

When we have a coarse partitioning there may b e many transitions from a cell which stay

CHAPTER FUNCTION APPROXIMATION FOR RL

inside that cell Those recurrent transitions make computing a correct value function harder

Only rewarding transitions or transitions going to neighboring cells are useful for computing

the value function and p olicy since they show dierences b etween dierent actions The ner

the representation the b etter we may mo del the true underlying dynamics but the larger the

problem of noise Furthermore with a ne representation we need much more exp eriences

b efore we have learned reliable transition probabilities and rewards and DP b ecomes much

slower Finally if the partitioning is changing we have to recompute transition probabilities

and rewards which may not always b e easy

Note that a mo del gives us the advantage of dierent splitting criteria If a hypothet

ical split will change the value function signicantly we keep it and continue splitting

If a current transition is unexp ected according to the mo del we can split the cell to b e b etter

able to predict the dynamics Mo ore and Atkesons partigame algorithm do es exactly

the latter the algorithm greedily executes a goal directed b ehavior but whenever executed

actions in a cell do not make a transition to the next cell as exp ected the algorithm splits

the cell so that more accurate predictions b ecome p ossible A diculty of this approach is

that sto chasticity in the transition function makes it dicult to know whether splitting the

cell will make the ability to predict the successor more accurate or just increase the mo dels

complexity A solution to this may b e to use hypothetical splits and collect more exp erimental

data Another problem is that single splits may not always b e helpful to see any dierences

so that multiple splits eg in multiple dimensions have to b e tried out at the same time

which is computationally quite exp ensive

Sometimes learning a mo del is not very dicult if we have a deterministic MDP

discrete actions and discrete time we could learn a p erfect deterministic transition function

b etween cells by making the coarseness of the representation suciently small although for

highdimensional spaces the required space complexity may make this approach infeasible

Learning imp erfect mo dels Although learning an accurate mo del is hard we can

learn useful but incomplete mo dels Such mo dels can b e used for more reliable and faster

learning of go o d p olicies Eg consider a so ccer environment in which we do not try to

predict the p ositions of all players and the ball which would b e infeasible due to the huge

amount of p ossible game constellations but only try to estimate the p osition of the ball at

each following time step Such a mo del is much easier to learn and even although it may not

b e very accurately it could b e quite useful for planning action sequences

First we will discuss learning linear neural network mo dels and we will see that this leads

to a very simple equation for estimating the value function Then we will discuss estimating

mo dels for neural gas and CMAC architectures after which DP or PS is used to compute

value functions

Linear Mo dels

Linear mo dels are the simplest mo dels but they can only b e used to accurately mo del a

deterministic evolution of the pro cess The goal is to estimate the linear mo del L a square

matrix which maps the input vector at time t x to its successor x

t t

T T

x x L

t t

To simplify the following discussion we leave action parameters out of the nomenclature

WORLD MODELING WITH FUNCTION APPROXIMATORS

where x denotes the transp ose of x and is a vector which accounts for white noise We

also want to mo del the reward function For this we use a linear mo del w to compute the

exp ected reward r given that we see x

t t

r x w

t r

Note that we use a linear reward function which makes the following short introduction to

linear mo dels somewhat simpler but for practical use it has some disadvantages We should

realize that for innite state spaces the agent can continuously increase its received rewards

by making actions which follow the direction of the linear reward plane Therefore optimal

p erformance can never b e reached We can only use this linear reward function if the state

space is b ounded by the transition function Many other researchers have used a quadratic

reward function written as r x w x which ensures that all p ositive rewards are b ounded

t t r t

Computing the mo dels Given a set of example transitions training patterns x x

t t

and emitted rewards x r we can compute L and w First we collect all vectors x in the

t t r t

matrix X and the vectors x in the matrix Y by putting the vectors as row vectors in the

matrices Then we can compute L as follows

T T

L X X X Y

Note that if the vectors X X are orthonormal then X X is identical to the

identity matrix I so that we can simplify Equation to

L X Y

We compute the weight vector w for returning the immediate reward as

T T

w X X X r

where r is the mdimensional vector which stores all immediately emitted rewards r r

Computing the value function We approximate the value of a state by using the

linear network see ab ove

V x x w

where the vector w is the weight vector and for simplicity includes the bias parameter an

additional onbit is included in the input vector

Now we can rewrite the original DP Equation from Chapter

V P V D iag PR

w Lw w

for which we require that all absolute eigenvalues of L are smaller or equal to otherwise

the value function is nonexistent This we can rewrite as

w I L w

In practice this is not very likely to happ en although it could happ en if for example we use input vectors

with a single bit on to mo del dierent discrete states

CHAPTER FUNCTION APPROXIMATION FOR RL

Which gives us the solution for a linear mo del

Bradtke and Barto present an extensive analysis of linear least squared metho ds

using a quadratic reward function combined with TD learning They pro of that if the input

vectors are linearly indep endent and each input vector is of dimension equal to the number

of states the weights converge with probability to the true value function weights after

observing innitely many transitions examples

These linear mo dels are also called optimal linear asso ciative memory OLAM lters

Kohonen There also exist more complicated linear mo dels such as linear quadratic

regulation LQR mo dels which extends OLAM by allowing the mo deling of interactions

b etween inputs Landelius

Neural Gas Mo dels

In contrast to global FAs lo cal mo dels can b e simply combined with world mo dels and can

in principle mo del the underlying MDP arbitrary accurately we can make a negrained

partitioning of the input space into cells and compute transition probabilities by counting

how often one cell has b een activated after another

DistributedFactorial representation Counting is easiest if we use WTA to create a

set of discrete regions since after each transition we only have to change three variables the

counters of the transitions and the variable which sums the transition rewards If we would

use smo oth LFAs we need to up date all transitions b etween cells which were activated This

takes much longer in the worst case of a full distributed probabilistic representation this

takes OZ where Z is the number of neurons In this case we could just connect cells which

are more activated than some threshold to sp eed things up

Since WTA causes discontinuities at the b orders of two subspaces there may b e advan

tages in smo othly combining the values of a set of states Mo ore et al Munos

Schneider uses multiple states to mo del uncertainty in mo delbased learning and shows

that dealing with uncertainty can b e improved by interpolating over a larger neighborho o d

Gordon shows conditions under which the approximate value iteration algorithm

converges when combined with function approximators which compute weighted averages of

stored target values An interesting insight which he oers is the following if a sample p oint

uses a cell with some normalized weight then this weight can b e seen as the probability

of making a step to this cell from the sample p oint This may b e a go o d way of initializing

transition probabilities after inserting a new cell

Again we have to learn a neural gas structure and the Qfunction The latter is done by

estimating the mo del and using our prioritized sweeping algorithm to manage the up dates

Learning structure We used the same approach as b efore to learn the structure

Again after each game we computed new Qvalues according to Q but now we used these

values only to split states and not for computing a new Qfunction We also used the same

conditions as b efore for adding neurons Furthermore we initialize the transition variables to

and from the new neuron to Note that we may improve this initialization pro cedure by

weighing the transitions from neighboring neurons according to their distances Accurately

initializing the transition probabilities and rewards is dicult however since we never know

the exact dynamics at a new unmo deled region Another way to learn the structure is to use

hypothetical splits and mo del the dynamics at the new hypothetical cells without immediately

using them for evaluating actions Then we can estimate whether using them for real will

signicantly change the transition probabilities and the value function If they would we

WORLD MODELING WITH FUNCTION APPROXIMATORS

keep them otherwise we try new splits We have not tried this algorithm however although

it has the advantage that newly introduced cells may b e guaranteed to improve the mo del

and we have go o d initial transition probabilities for them

Estimating the mo delLearning Q To estimate the transition mo del for the i neu

t t

ron we count the transitions from the active neuron n i to the active neuron n j

at the next timestep given the selected action a Thus after each i a j transition we

compute

C a C a and C a C a

ij ij i i

We can also take all activated neurons in the smo oth comp etition case into account by

up dating counters dep endent on their gates We do this by computing i j

t t

C a C a g g and C a C a g

ij ij i i

i i

These counters are used see Chapter to estimate the transition probabilities P j ji a

t t

P n j jn i a For each transition we also compute in similar ways the average reward

R i a j by summing the immediate reinforcements over the transition from the active neuron

i to the next active neuron j by selecting action a

Our prioritized sweeping We apply our prioritized sweeping to up date the Qvalues

of the neuronaction pairs After each time step up dates are made via the usual Bellman

backup Bellman

Qi a P j ji a V j R i a j

After each action we up date the mo del and use PS to compute the new Qfunction Details

of the algorithm are given in App endix D

CMAC Mo dels

We now describ e a world mo deling approach for RL which uses CMACs We know that

learning accurate mo dels estimating transitions b etween complete world states is very hard

for complex tasks Eg the same world state may only app ears once in an agents life time

As an example consider a world state consisting of a number of input features for which the

rst feature would oscillate b etween two values x A second feature could

oscillate b etween values a third b etween values and so on If we would use one lter

for each feature we could easily learn the correct predictive mo del in eg to steps

Estimating the transition function of the complete state vector with a single lter would take

a lot of time eg steps however since a complete p erio d

in input space would cost so many steps

Therefore instead of estimating full transitions mo dels we can sometimes prot sig

nicantly from using a set of indep endent mo dels and estimating their uncoupled internal

dynamics In this way we can study the dynamics of an ob ject by separating it from the rest

of the world If the separation holds we can learn p erfect mo dels

Our goals are to create a set of lters which can b e used for mo deling the world and

estimate parameters given the set of designed lters

Creating lters We have seen that with a completely observable discrete world and

lo okup tables we can estimate a mo del and p erfectly store the transition probabilities to

successor states This allows to compute answers to questions such as what is the exp ected

CHAPTER FUNCTION APPROXIMATION FOR RL

probability of entering the goal state within N steps Furthermore it is for static goal

directed environments always p ossible to select an action which is exp ected to increase the

state value

For CMAC mo dels we can in general not answer the question ab ove and it may not always

b e p ossible to select actions which lead to a larger value for each lter If lters contradict

each other eg if one lter advises a particular action which is exp ected to decrease the value

of another lter learning problems may arise Usually this happ ens in the initial phase but

such problems may b e resolved due to learning however

Supp ose actions are executed which change the world state but keep the activated cell

of a particular lter the same Such actions will not b e considered by that lter to improve

the state value After some time an action is executed which changes the activated cell of

the lter which is now activated in a much b etter world state Therefore the lter learns a

large preference to select that action immediately This may lead to a p olicy change in a

new trial the new action may b e tried out a number of times but leads to failure The result

is that other lters will start p enalizing that action in their currently activated cells and this

will suppress that action from b eing selected immediately Finally the system may converge

to cellaction values which p ostp one the action for a while after which it will b e selected

Dierence to CMACQ The CMAC Qlearning system uses the overall evaluation of

the complete set of lters to up date previous ltercellaction triples whereas CMAC mo dels

keep lters separated from each other Thus for the example ab ove Qlearning may learn Q

values of the lter which approve of the rst action even although the action do es not change

the lters cell However once the other action needs to b e selected the lter itself may not

have learned a large preference for that action Thus for Qlearning the nal Qfunctions

of lters may contain Qvalues of dierent actions which are much closer to each other than

those learned by the CMAC mo del which may use large preferences and disapprovements

Estimating the mo del To estimate the transition mo del for the k lter we count

at the next timestep given the the transitions from activated cell f to activated cell f

selected action a Thus for all lters k Z we compute after each transition

C a C a

t+1 t+1

t t

f f f f

k k k k

These counters are used to estimate the transition probabilities for the k lter P c jc a

j i

c jf c a where c and c are cells and a is an action For each transition P f

j i j i

we also compute the average reward R c a c by summing the immediate reinforcements

i j

given that we make a step from active cell c to cell c by selecting action a

i j

Prioritized sweeping PS Again we apply prioritized sweeping PS to compute the

Qfunction This time we up date the Qvalue of the ltercellaction triple with the largest

size of the Qvalue up date b efore up dating others Each up date is made via the usual Bellman

backup Bellman

P c jc a V c R c a c Q c a

j i j i j i

f f f f

After each agent action we up date all lter mo dels and use PS to compute the new Q

functions Note that PS may use dierent numbers of up dates for dierent lters since some

lters tend to make larger up dates than others and the total number of up dates p er time step

is limited The complete PS algorithm is given in App endix D

A SOCCER CASE STUDY

Incrementally constructing lters If there would b e rare interactions b etween fea

tures we could mo del them using lters based on their combination This could mean that

we may start out with a lot of parameters however A b etter way may b e to incrementally

construct and add lters We can construct lters by making new combinations of inputs

Then we can add them in several ways we always use them lters can b e only invoked

if some condition holds eg if some other particular ltercell is activated lters may

replace other lters if some condition holds

The go o d thing of CMACs is that we can easily add lters without making large changes

to the p olicy Thus adding lters may only improve the p olicy on the long term although

the improvement is at the exp ense of larger computational costs evaluation costs are linear

in the number of lters

CMAC mo dels are related to dierent kinds of probabilistic graphical mo dels Lauritzen

and Wermuth Both can b e used to learn to estimate the dynamics of some state

variables CMACs uses a committee of exp erts for this whereas Bayesian networks fuse

probabilistic dep endencies according to Bayes rule This makes inference in CMAC mo dels

faster although they will in general need more space to store an accurate mo del

A So ccer Case Study

We use simulated so ccer to study the p erformances of the function approximation metho ds

So ccer provides us with an interesting environment since it features highdimensional input

spaces and also allows for studying multiagent learning

So ccer

So ccer has received attention by various researchers Sahota Asada et al

Littman a Stone and Veloso Matsubara et al Most early research fo

cused on physical co ordination of so ccer playing rob ots Sahota Asada et al

There also have b een attempts at learning lowlevel co op eration tasks such as pass play Stone

and Veloso Matsubara et al Littmans used a grid world with two

single opp onent players to examine the gain of using minimax strategies instead of using b est

average strategies Learning complete so ccer team strategies in more complex environments

is describ ed in Luke et al Stone and Veloso

Our case study will involve simulations with continuousvalued inputs and actions and

up to players agents on each team We let team players agents share action set and

p olicy making them b ehave dierently due to p ositiondep endent inputs All agents making

up a team are rewarded or punished collectively in case of goals We conduct simulations

with varying team sizes and compare several learning algorithms oine Qlearning with

linear neural networks Qlin oine Qlearning with neural gas Qgas online Q

with CMACs and mo delbased CMACs We want to nd out how well these metho ds work

what are their strengths and weaknesses This gives us the p ossibility to increase our

understanding of why some function approximators work well in combination with RL for

some tasks and why some do not

Evolutionary computation vs RL Finally we will compare the p erformances of the

RL metho ds to an evolutionary computation EC approach Holland Rechenberg

called Probabilistic Incremental Program Evolution PIPE introduced in Salustowicz

and Schmidhuber A PIPE alternative for searching program space would b e genetic

CHAPTER FUNCTION APPROXIMATION FOR RL

programming GP Cramer Koza but PIPE compared favorably with Kozas

GP variant in previous exp eriments Salustowicz and Schmidhuber With the compar

ison to an EC metho d we want to nd out in what RL is particularly go o d or bad When

we lo ok at backgammon for which b oth approaches have b een used b efore the EC metho d

describ ed in Pollack and Blair was able to learn much faster in the initial phase but

did not lead to such a go o d nal p erformance as that of TDGammon Tesauro Thus

it seems that EC can p erform a faster coarse search whereas getting high p erformance results

in the long term is easier done with RL

PIPE PIPE searches for a program which achieves the p erformance Each program is

a tree comp osed of a set of functions such as fsin log cos expg and terminal symbols

input variables and a generic random constant The system generates programs according

to probabilistic prototype trees PPTs The PPTs contain adaptive probability distributions

over all programs that can b e constructed from the predened instruction set The PPTs

are initialized with probabilities for selecting a terminal symbol at a no de and probabilities

for selecting a function If the former probabilities are set to high values the generated

programs are usually quite small although learning can cause generated programs to grow

in size After generating a p opulation of individuals each individual is evaluated on the

problem eg by playing it one game against the opp onent Then the b est p erforming

individual the one which maximizes the score dierence is used for adapting the PPTs all

probabilities of functions used by the programs of the b est individual are increased so that

generating the winning individual gets more probable eg after adapting the probabilities

the probability that a newly generated program would b e the b est program from the previous

generation is Thus the search always fo cuses around the b est p erforming program To

limit the problem of ending up in lo cal minima a mutation rate is added to the system which

randomly p erturbates probabilities in the PPTs A large dierence with GP is that instead

of individuals a probability distribution over individuals is stored and that no crossover is

used Therefore PIPE is closer related to Evolutionary Strategies ES Rechenberg

and PBIL Baluja

The So ccer Simulator

We wrote our own so ccer simulator although there are also other more sophisticated sim

ulators available Our discretetime simulations feature two teams with either or

players p er team We use a twodimensional continuous Cartesian co ordinate system for the

eld The elds southwest and northeast corners are at p ositions and resp ectively

Goal width is As in indo or so ccer the eld is surrounded by impassable walls except for

the two goals centered in the east and west walls Only the ball or a player with ball can

enter the goals There are xed initial p ositions for all players and the ball see Figure

PlayersBall Players and ball are represented by solid circles consisting of a centre

co ordinate and a realvalued orientation direction Initial orientations are directed to the

east west for the west east team A player whose circle intersects the ball picks it up and

owns it The ball can b e moved or shot by the player who owns it When shot the sp eed of

the ball v gains an initial sp eed unitstime step and then decreases over time due

to friction v t v t until v t This makes shots p ossible of units

b b b

of the length of the eld

See eg Rob oCup JavaSoccer from Tucker Balch which follows Rob o cup rules retrievable from httpwwwccgatechedugradsbTuckerBalchJavaBotsEDUga techccisdo csi ndexhtml

A SOCCER CASE STUDY

Figure players and ball in the centre in initial positions Players of a or player

team are those farthest in the back defenders andor goalkeepers

Players collide when their circles intersect This causes the player who made the resp on

sible action to b ounce back to his previous p ositions at the previous time step If one of the

collision partners owned the ball prior to collision the ball will change owners There are

four actions for each player

forward move player units in its current direction if without ball and go

units if he owns the ball

turn to ball change the players orientation so that the player faces the ball

to goal change the players orientation so that the player faces the opp onents turn

goal

shoot If the player do es not own the ball then do nothing Otherwise to allow for

imp erfect noisy shots turn the agent with an angle picked uniformly random from the

interval and then sho ot the ball according to the players new orientation

init

The initial ball sp eed is v

Action framework A game lasts from time t to time t The temp oral

end

order in which players execute their moves during each time step is chosen randomly Once

all players have selected a move the ball moves according to its sp eed and direction and we

set t t If a team scores or t t then all players and ball will b e reset to their

end

initial p ositions

Input At a given time t player ps input vector consists of basic features

Three b o olean inputs that tell whether the player has the ball a team member

has the ball the opp onent team has the ball

Polar co ordinates distance angle of b oth goals and the ball with resp ect to the players

orientation and p osition

Polar co ordinates of b oth goals relative to the balls orientation and p osition

CHAPTER FUNCTION APPROXIMATION FOR RL

Ball sp eed

These basic features do not provide information ab out the p osition of the other players

including the opp onents Therefore we designed also a more complex input consisting of

player or players features by adding the following features

Polar co ordinates distance and angle of all other players wrt the player ordered by

a teams and b distances to the player

For Qlin Qgas and PIPE we change the input of distance d and angle in by

applying the following functions to them b efore giving the input to the learning algorithms

This was mainly done to simplify Qlins function representation and e d

so that it is able to fo cus its action selection on small distances and angles For the other

algorithms it should not matter much

Comparison Qlin Qgas and PIPE

We rst compare the following metho ds linear networks and neural gas trained with oine

Q and the evolutionary metho d Rechenberg Holland Koza PIPE

Salustowicz and Schmidhuber Results have b een previously published in Salustowicz

et al Salustowicz et al a

We have used an oine Q variant using accumulating traces for training the FAs

The reason for using oine Qlearning is that reinforcement is only given once a goal is

scored so that the largest reason of using online learning improving initial exploration do es

not apply whereas oine learning is computationally cheaper

Exp erimental setup

Opp onent We train and test all learners against a biased random opp onent BRO BRO

randomly executes actions but due to the initial bias in the action set it p erforms a goal

directed b ehavior If we let BRO play against a nonacting opp onent NO all NO can do is

blo ck for twenty time step games then BRO always wins against NO with on average

to goals for team size to goals for team size to goals for team size

We also designed a simple but go o d team GO by hand GO consists of players which move

towards the ball as long as they do not own it and sho ot it straight at the opp onents goal

otherwise If we let GO play against BRO for twenty time step games then GO always

wins with on average to goals for team size to goals for team size and to

goals for team size For this simulator GO executes a very go o d single agent strategy

Note however that GO implements a nonco op erative strategy this makes it sub optimal for

larger team sizes

Game duration and inputs We play games of length t for team sizes

end

and we will not show results with players here since they are very comparable to

those of and players Every games we test current p erformance by playing test

games no learning against BRO and sum the score results

For all metho ds we used the basic input features so without the additional information

ab out other players

Linear network setup After a coarse search through parameter space we used the fol

lowing parameters for all Qlin runs H a small size of the history

max l

A SOCCER CASE STUDY

list worked b est Although in this case the FA is not trained on all p ossible examples the most

imp ortant ones leading to goals are learned All network weights are randomly initialized

in We use Boltzmann exploration where during each run the BoltzmannGibbs

rules greediness parameter the inverse of the temp erature is linearly increased from to

The discount factor to encourage quick goals or a lasting defense against opp onent

goals r the reinforcement at trial end is if opp onent team scores if own team scores

and otherwise

Neural gas setup For Qgas we used H note

g max

that this large value makes the FAs very lo cal the initial number of neurons Z and we

init

constrain the maximum of neurons Z We use Maxrandom exploration with P

max max

Parameters for adding neurons are the required error T and T We

E C

used the Manhattan distance or L norm although some trial exp eriments using Euclidean

distance resulted in similar p erformances The comp onents of the neuron centers w are

randomly initialized in Qvalues are zeroinitialized The discount factor

Again r is if opp onent team scores if own team scores and otherwise

PIPE setup We compare our RL metho ds with the evolutionary search metho d PIPE

Salustowicz and Schmidhuber PIPE searches for the individual consisting of ve

programs one for each action and one for determining the temp erature for the used Boltz

mann exploration rule achieving the b est score dierence when tested against the opp o

nent The most interesting parameters for PIPE runs are set to learning rate mutation

rate and p opulation size During p erformance evaluations we test the current b est

ofgeneration program except for the rst evaluation where we test a random program Note

that this is an advantage for PIPE since the program which is tested is known to outp erform

all other programs of the same generation

Exp erimental results

We compare average score dierences achieved during all test phases against BRO Figure

shows results for PIPE Qlin and Qgas It plots goals scored by learner and opp onent

against number of games used for learning averaged over simulations Larger teams score

more frequently b ecause some of their players start out closer to the ball and the opp onents

goal

PIPE learns fastest and always quickly nds an appropriate p olicy regardless of team

size Its score dierences continually increase Qlin and Qgas also improve but in a less

sp ectacular way They are able to win on average and tend to increase score dierences until

they score roughly twice as many goals as in the b eginning when action selection is still

random

For the single agent case Qgas is able to learn go o d defensive p olicies what can b e seen

from the strongly reduced opp onent scores For the multiagent case Qgas slowly increases

its score dierences although its learning p erformance shows many uctuations It seems

that the system is changing its p olicy a lot and do es not seem able to x a go o d p olicy Still

it is almost always able to win

For Qlin the score dierences start declining after a while the linear neural networks

cannot keep and improve useful value functions but tend to unlearn them instead We will

now describ e a deep er investigation of the catastrophic p erformance breakdown in the

player Qlin run followed by an explanation of neural gas instability problems

Qlins instability problems Some runs of Qlin led to go o d p erformance although

CHAPTER FUNCTION APPROXIMATION FOR RL

PIPE 1-player Q-lin 1-player Q-gas 1-player

300 learner 300 learner 300 learner opponent opponent opponent 250 250 250 200 200 200

goals 150 goals 150 goals 150 100 100 100 50 50 50 0 0 0 0 500 1000 1500 2000 2500 3000 0 500 1000 1500 2000 2500 3000 0 1000 2000 3000 #games #games #games PIPE 11-players Q-lin 11-players Q-gas 11-players 500 learner 500 learner 500 learner opponent opponent opponent 400 400 400

300 300 300 goals goals goals 200 200 200

100 100 100

0 0 0 0 500 1000 1500 2000 2500 3000 0 500 1000 1500 2000 2500 3000 0 1000 2000 3000

#games #games #games

Figure Average number of goals scored during al l test phases for team sizes and

Averages were computed over simulations

sudden p erformance breakdowns hint at a lack of stability of go o d solutions To understand

Qlins problems in the player case we saved a go o d network just b efore breakdown after

games We p erformed additional simulations for which we continued training the

saved network for games testing it after every training game by playing test games To

achieve more pronounced score dierences we set the temp erature parameter in the Boltzmann

this leads to more deterministic b ehavior than the value round exploration rule to

used by the saved network

Figure left plots the average number of goals scored by Qlin and BRO during all test

games against the number of games Although initial p erformance is very go o d the score

dierence is goals the network fails to keep it up To analyze a particular breakdown we

fo cus on a single run Figure middle shows the number of goals scored during the test

phases of this run and Figure right shows the relative frequencies of selected actions

TD-Q 11-players TD-Q 11-players TD-Q 11-players

500 learner 500 learner 1 P(go_forward) opponent opponent P(shoot) 400 400 P(turn_to_ball) P(turn_to_goal) 300 300

goals goals 0.5 200 200

100 100 relative action frequency 0 0 0 0 5 10 15 20 25 0 5 10 15 20 25 0 5 10 15 20 25

#games #games #games

Figure Performance breakdown study Left average numbers means of runs of goals

scored against BRO by a team with players starting out with a wel ltrained linear network

Midd le plot for a single run Right relative frequencies of actions selected during the single

run

Figure middle shows a p erformance breakdown o ccurring within just a few games

It is accompanied by dramatic p olicy changes displayed in Figure right Analyzing the

A SOCCER CASE STUDY

learning dynamics we found the following reasons for the instability

Since linear networks learn a global value function approximation they compute an

average exp ected reward for all game constellations This makes the Qvalues of many actions

quite similar their weight vectors dier only slightly in size and direction Hence small

up dates can create large p olicy changes

Qlin adapts some weight vectors more than others the weight vector of the most

frequent action of some rewarding unrewarding trial will grow shrink most For example

forward is selected most fre according to Figures middle and right the action go

quently by the initially go o d p olicy Two games later however the b ehavior of the team is

dominated by this action This results in worse p erformance and provokes a strong correc

tion in the third game This suddenly makes the action unlikely even in game constellations

where it should b e selected and leads to even worse p erformance

For player teams the eect of up date steps on the p olicy is fold and therefore

instabilities due to outliers can easily b ecome much more pronounced

The question can b e asked whether the RL metho d is the cause for the learning diculties

or whether it is the linear networks Although the linear networks are not very p owerful

they can still b e used to reach high p erformance levels even although the learned value

function approximation may b e very bad Linear mo dels could get rid of at least some of

the instability problems We have not tried them however since they are computationally

much more exp ensive

Neural gass instability Just like the linear network neural gas is not able to constantly

improve its p olicy For the single agent case it quickly diminishes the opp onents score to few

goals and increases its own score but do es not keep go o d p olicies pro ducing many goals

We also combined the neural gas with online Qlearning but this did not improve the

results Analysing the system we found one particularly imp ortant reason for neural gas

learning problems adding neurons usually helps to improve the p erformance but sometimes

adding a neuron can ruin go o d p olicies Noise can b e resp onsible for placing a neuron which

starts dominating some region adds errors to its Qfunction approximation and selects a

wrong action Ways to solve this problem is to design more clever algorithms for adding

neurons If a neuron is added the algorithm makes an unrealistically large up date step of

the value function A way to get around this is to use lo cal greediness values in Equation

for computing the neuron gatevalues Then we can initialize a greediness value of a newly

added neuron to a very low value and gradually increase it so that it has time to adapt to its

environment b efore b eing used fully by the system

The lesson we have learned here is that direct RL metho ds require evaluating p olicies

which do not change very fast Each p olicy needs many exp eriences to b e completely eval

uated and although we can up date the value function b efore we have collected a very large

number of exp eriences which would make learning very slow we should b e careful not to

make any large changes based on little exp erience since that may cause havoc previous

plans or strategies suddenly may not work anymore Thus using state quantization meth

o ds for RL seems a p owerful technique but applying it may b e quite dicult since they need

lots of parameter twiddling and lack robust structures We should try to gradually improve

the structure and make small Qvalue up dates

Neural gas mo dels We also tried using neural gas mo dels but these did not work

successfully either One of the main problems of this approach is that transition probabilities

are estimated according to a particular p ositioning of the neurons which constantly and

probably to o fast changes A second problem was that neural gas mo dels result in many

CHAPTER FUNCTION APPROXIMATION FOR RL

outgoing transitions from each neuron so that there is a lot of uncertainty ab out predicting

the next state and Bellman backups consume a lot of time Starting with a xed structure and

connective neighborho o d may circumvent this problem but choosing an a priori structure

is a dicult design issue on its own Finally instability problems are caused by the used

insertion pro cedure and initialization of new neurons

Conclusion

We found that linear networks despite of their lack of expressive p ower can b e used for rep

resenting and nding go o d p olicies Since their function approximation of the true evaluation

function is unstable however they quickly unlearn go o d p olicies once they are found

The neural gas metho d showed faster initial learning b ehavior than the linear networks

but in our exp eriments the neural gas systems did not continuously learn to improve p olicies

A problem of the neural gas metho d is that although adding neurons is usually useful some

times it leads to bad interference with the learned approximation Therefore more careful

algorithms for growing cell structures should b e used

Both algorithms p erformed worse than the evolutionary metho d PIPE which tests mul

tiple dierent individuals and has a large probability of generating winning programs of the

previous round again thereby safeguarding the b est found p olicies Furthermore PIPE was

able to quickly nd go o d p olicies since it was biased to start searching for low complexity

programs and quickly discovered which features were imp ortant to use Surprisingly some

low complexity programs implemented very go o d p olicies against the xed so ccer opp onent

Learning so ccer strategies with a single player or with multiple players do es not seem to

b e very dierent in our so ccer environment The only real diculty with multiple players is

that particular outliers get heavier weight when we use multiple players thus leading to more

learning instability and requiring more robustness from the FA

From the current results we conclude that value function based RL metho ds should b e

extended to make them also prot from a feature selection facilities b existence of low

complexity solutions c incremental search for more complex solutions where simple ones do

not work and d keeping and improving the b est solution found sofar

Comparison CMACs vs PIPE

The previous function approximators were to o unstable for reliably learning go o d so ccer

p olicies CMACs p ossess dierent prop erties while still b eing lo cal all cells have xed

lo cations This could p ossibly make them more stable and therefore more suitable for RL

We will now compare CMACs trained with online Q mo delbased CMACs trained with

PS and PIPE Results have b een previously published in Wiering et al

Exp erimental setup

Task We train and test the learners against handmade programs of dierent strengths

We use a dierent setup from the previous exp eriments to get some results against b etter

opp onents since we exp ect that evolutionary computation may b e faster when it comes to

nding solutions to simpler problems The opp onent programs are mixtures of the program

BRO which randomly executes actions and the program GO which moves players towards the

ball as long as they do not own it and sho ots it straight at the opp onents goal otherwise

A SOCCER CASE STUDY

Our ve mixture programs called OpponentP use BRO for selecting an action with prob

g and otherwise they use GO Thus O pponent equals BRO and ability P f

O pponent equals GO

We test team sizes and team sizes and do not show large dierences in learning

curves although team size consumes much more simulation time

For all metho ds we used the extended input vectors with the additional player dep endent

input features We have observed that for the neural gas and linear networks there were

minor advantages using extended inputs although the overall results lo ok more or less the

same Thus we use inputs for the player case and inputs for the player case

CMACQ setup We play a total of games Every games we test current

p erformance by playing test games against the opp onent and sum the score results The

reward is if the team scores and if the opp onent scores The discount factor is set

to We used fast online Q with replacing traces for the player case and

for the player case Initial learning rate lr lr decay rate We

used Maxrandom exploration with P We use lters p er input total of

max

or lters and set the number of cells n Qvalues are initially zero

PIPE setup For PIPE we play a total of games Every games we test p er

formance of the b est program found during the most recent generation Parameters for all

PIPE runs are the same as in previous exp eriments Salustowicz et al

CMAC mo del setup

For the CMAC mo dels we have some additional sp ecial features which improved the learning

p erformance

Multiple restarts CMAC mo dels turn out to b e very fast learners often they converge

to a particular p olicy after a small number of games The metho d sometimes gets stuck

with continually losing p olicies also observed with our previous simulations based on linear

networks and neural gas however We could not overcome this problem by adding standard

exploration techniques Instead we reset the Qfunction and WM once the team has not scored

for successive games whereas the opp onent scored during the most recent game we check

these conditions every games This multiple restarts combined with mo delbased CMACs

which b ehaves as a sto chastic hillclimber is a new way for searching p olicies in RL but is

also used in op erations research OR for searching for solutions to combinatorial optimization

problems Note that it is no problem whatso ever to use multiple restarts with any learning

algorithm for any problem eg it could also b e used for lifelong learning approaches

Nonp essimistic value functions Since we let multiple players share their p olicies

we fuse exp eriences of multiple dierent players inside a single representation These ex

p eriences are however generated by dierent player histories scenarios and therefore some

exp eriences of one player would most probably never o ccur to another player Thus there is

no straightforward way of combining exp eriences of dierent players For instance a value

function may assign a low value to certain actions for all players due to previous unlucky exp e

riences of one player To overcome this problem we compute nonp essimistic value functions

we decrease the probability of the worst transition from each cellaction and renormalize the

other probabilities Then we use PS with the new probabilities Thus basically we do a

similar thing as in Chapter for computing optimistic value functions The dierence is that

in Chapter we made go o d exp eriences more imp ortant Here we make bad exp eriences less

imp ortant Details are given in App endix D

CHAPTER FUNCTION APPROXIMATION FOR RL

Parameters We play a total of games Every games we test current p erformance

by playing test games against the opp onent and summing the score results The reward is

if the team scores and if the opp onent scores The discount factor is set to After

a coarse search through parameter space we chose the following parameters We use lters

p er input total of or lters and set the number of cells n Qvalues are initially

zero PS uses and a maximum of up dates p er time step

Exp erimental results

CMAC Model 1-Player CMAC-Q 1-Player

2 2 Opponent (1.00) Opponent (0.75) Opponent (0.50) 1.5 1.5 Opponent (0.25) Opponent (0.00) 1 1 Opponent (1.00) game points Opponent (0.75) game points 0.5 Opponent (0.50) 0.5 Opponent (0.25) Opponent (0.00) 0 0 0 50 100 150 200 0 50 100 150 200

#games #games PIPE 1-Player

1.5 Opponent (1.00) 1 Opponent (0.75) Opponent (0.50) game points Opponent (0.25) 0.5 Opponent (0.00)

0 0 200 400 600 800 1000

#games

Figure Number of points means of simulations during test phases for teams consisting

of player Note the varying xaxis scalings

Results Player case We plot number of p oints for scoring more goals than the

opp onent during the test games for ties for losses against number of training games

in Figure

We observe that on average our CMAC mo del wins against almost all training programs

Only against the b est player team P it wins as much as it loses Against the worst

two teams CMAC mo del always nds winning strategies

CMACQ nds programs that on average win against the random team although they

do not always win It learns to play ab out as well as the random and random teams

CMACQ p erforms p o orly against the b est opp onent and although it seems that the

p erformance jumps up at the end of the trial longer trials do not lead to b etter p erformances

PIPE is able to nd programs b eating the random team and quite often discovers programs

that win against random teams It encounters great diculties in learning go o d strategies

against the b etter teams though Although PIPE may execute more games vs

A SOCCER CASE STUDY

the probability of generating programs that p erform well against the go o d opp onents is very

small For this reason it tends to learn from the b est of the losing programs This in turn

do es not greatly facilitate the discovery of winning programs

Results Player case We plot number of p oints for scoring more goals than the

opp onent during the testgames against number of training games in Figure

Again CMAC mo del always learns winning strategies against the worst opp onents It

loses on average against the b est player team with P though Note that this

strategy mixture works b etter than always using the deterministic program P against

which CMAC mo del plays ties or even wins In fact the deterministic program tends to

clutter agents such that they obstruct each other The deterministic opp onents b ehavior

also is easier to mo del All of this makes the sto chastic version a more dicult opp onent

CMACQ is clearly worse than CMAC mo del it learns to win only against the worst

opp onent

PIPE p erforms well only against random and random opp onents Against the b etter

opp onents it runs into the same problems as mentioned ab ove

CMAC Model 3-Players CMAC-Q 3-Players

2 2

1.5 1.5 Opponent (1.00) Opponent (0.75) Opponent (0.50) 1 Opponent (1.00) 1 Opponent (0.25) Opponent (0.75) Opponent (0.00) game points Opponent (0.50) game points 0.5 Opponent (0.25) 0.5 Opponent (0.00) 0 0 0 50 100 150 200 0 50 100 150 200 #games #games PIPE 3-Players

1.5

Opponent (1.00) 1 Opponent (0.75)

game points Opponent (0.50) 0.5 Opponent (0.25) Opponent (0.00)

0 0 200 400 600 800 1000

#games

Figure Number of points means of simulations during test phases for teams consisting

of players Note the varying xaxis scalings

Score dierences We show the largest obtained score dierences in Table player

and Table players We should keep in mind that these score dierences can have a

large variance and may thus not convey to o much information Eg in the case of CMAC

mo dels with player Opp onent may sometimes win against CMAC mo dels since

the CMAC mo del had not already found a go o d p olicy and its p olicy had just b een resetted

b efore testing Although this only happ ens or times out of simulations these large

CHAPTER FUNCTION APPROXIMATION FOR RL

scores have a tremendous impact on the overall averaged score results PIPE has a small

advantage there since PIPE always uses the b est found program during the last generation

for testing and thus eectively diminishes the probability to almost of testing a really

bad p olicy Still the tables show that although PIPE is able to score more against the

bad opp onents than CMACmodels or CMACQ PIPE often cannot score against the go o d

opp onents CMACmodels do score against the go o d opp onents and were able to nd at

least time winning p olicies against all opp onents It is dicult to say whether winning

is b etter or worse than winning though We remark that PIPE p erforms b etter

against the weakest opp onent and CMAC mo dels p erform b est against the b est opp onents

Learning Algorithm

CMACmodels

CMACQ

PIPE

Table Best average score dierences for the dierent learning methods against dierent

strengths of the player opponent Although CMACmodels were sometimes able to score

goals they also sometimes lost

Learning Algorithm

CMACmodels

CMACQ

PIPE

Table Best average score dierences for the dierent learning methods against dierent

strengths of the player opponent

Instead of showing score dierences we may also plot relative score dierences We

compute relative scores as

g oal s P l ay er

r el ativ e scor e

P l ay er g oal s O pponent g oal s

This measure has as a drawback that the scores and are treated equally although

this makes comparing results against the strong opp onents with results against the weak

opp onents easier p ossible We plot relative scores against number of games in Figure

Here we can clearly see that relative scores against the player team are larger than those

against player teams

Comments

Filter design For CMACmodels we used cells for each singleinput lter whereas we

used cells for CMACQ Changing the number of cells do es not aect the results very much

although learning can b ecome slower Using instead of lters results in go o d p erformance

as well although the computational time is doubled A single lter results in less stable

learning and worse p erformance however Finally dierent lter designs combining dierent

inputs works go o d as well Thus it seems that the metho d is quite robust to the actual design

of the lters

A SOCCER CASE STUDY

CMAC Model 1-Player CMAC-Q 1-Player PIPE 1-Player

1 1 Opponent (1.00) 1 Opponent (0.75) 0.8 0.8 Opponent (0.50) 0.8 Opponent (1.00) Opponent (0.25) Opponent (0.75) Opponent (0.00) 0.6 0.6 0.6 Opponent (0.50) Opponent (0.25) Opponent (0.00) 0.4 Opponent (1.00) 0.4 0.4

Relative score Opponent (0.75) Relative score Relative score 0.2 Opponent (0.50) 0.2 0.2 Opponent (0.25) Opponent (0.00) 0 0 0 0 50 100 150 200 0 50 100 150 200 0 200 400 600 800 1000 #games #games #games CMAC Model 3-Players CMAC-Q 3-Players PIPE 3-Players

Relative score Opponent (0.75) Relative score Relative score 0.2 Opponent (0.50) 0.2 0.2 Opponent (0.25) Opponent (0.00) 0 0 0 0 50 100 150 200 0 50 100 150 200 0 200 400 600 800 1000

#games #games #games

Figure Relative scores own goals total goals for the metho ds for team sizes and

Non playersharing p olicies We also tried using dierent representations for the Q

functions of the player teams For this we used CMACmodels and used an indep endent

mo del for all players We did not use the nonp essimistic value function approach since the Q

functions were not shared Against Opp onent the results were comparable with sharing

p olicies except that the CMACmodels now tended to score slightly more the b est result

was but did not always win average p oints was Against Opp onent the

results were much worse CMAC mo dels lost on average and gained average p oints

Thus it seems that p olicy sharing improves learning when tasks get more complicated

Exploration in time critical problems In certain problems an agent has to reach

a sp ecic goal b efore some time criteria or timehorizon has elapsed For such time critical

problems p olicies need to b e go o d and almost deterministic exploiting or greedy to b e

successful since each badnonp olicy action will waste precious time This makes exploration

very dicult So ccer is an example of such a time critical problem If we play with a p olicy

that contains to o many bad actions against a very go o d opp onent then the opp onent will

score even b efore one of our own players has approached the ball Since in each state all actions

are tried ab out the same number of times it is very hard to learn which actions are really

bad This problem is shown by the results of all learners against the player O pponent

team There are no go o d ways to circumvent this problem with RL If the agent only receives

reward at a sp ecic goal state and that goal state is only reachable in a small p ercentage

of all trials other ways for initially exploring the p olicy space are needed such as exhaustive

search meansend analysis or a priori knowledge to b e able to test p olicies which p erform

well enough so that the goal is found

Discussion

Online vs Oine Q We used oine Q for the rst two FAs and online Q for

CMACs We switched to online learning b ecause we thought that it might improve learning

p erformance Although online learning improves things a little bit the dierent function

CHAPTER FUNCTION APPROXIMATION FOR RL

approximators are the main reason for the dierent learning p erformances Although all FAs

were often able to learn to b eat the simplest xed opp onent CMACs obtained the fastest

learning p erformance

RL vs EC When we compare RL to evolutionary computation for the so ccer task then

we observe that EC may b e b etter in learning very go o d p erformances for the easier problems

whereas RL is b etter in nding go o d p olicies for more dicult problems This conrms our

exp ectations RL may need more time eg CPU or mo deling but may nally come up with

b etter solutions for dicult problems

CMAC mo dels no CMAC mo dels are able to learn go o d reactive so ccer strategies

against all opp onents despite treating all features indep endently They prefer actions that

activate those cells of a lter which promise highest average reward The use of a mo del

stabilizes go o d strategies given sucient exp eriences the p olicy will hardly change anymore

The curse of lter dimensionality First of all a lot dep ends on the lter design For

particular problems we would need to construct lters combining many contextdependent

features Such problems would need a lot of cells which is esp ecially a problem if we use

mo delbased RL Still there may b e ways cutting down the number of parameters Eg we

could prepro cess all lter spaces and transform them into lowerdimensional spaces using

Indep endent Comp onent Analysis eg Oja and Karhunen Bell and Sejnowski

Limitations of CMAC mo dels Note that if there are a huge number of p ossible

actions we cannot store world mo del transitions conditioned on the selected action Instead

we mo del only the eect of the b est selected action Then we select an action by simulating

all p ossible actions with a mo del and evaluating the resulting states using the CMAC For

this we need to have a simulator or mo del mapping stateactions to subsequent states Then

we up date the CMAC mo del according to the selected action

Indep endent lter mo dels In chess there is so much contextual dep endent informa

tion that CMACs would probably not b e the b est representation for it However we can

generalize our CMAC mo dels to a set of indep endent lter mo dels Eg we could use de

cision trees DT to save a lot of memory whereas they are only slightly less timeecient

DTmo dels could b e used in the same way as CMAC mo dels we learn Qvalues for leave

no des and we learn transition probabilities and rewards b etween leave no des Furthermore to

allow for the p ower of the committee of exp erts we can use multiple DTlters where each

DTmo del is based on dierent subsets of all inputs Of course instead of DTlters we could

also use unsup ervised learning in a prepro cessing phase and use the resulting quantizations

for constructing dierent lters The thing to remember is that we are interested in using

multiple lters based on dierent views on the universe so that each p olicy can b e quickly

learned and the combined p olicy works well for the many dierent partial universes

Nonstationary lters Finally there may b e problems if lters are only worthwhile

during a particular stage of a pro cess After this stage invoking them would just cost time

However nothing forbids us to use temp oral sequences of ltersets or to use a higherlevel

decomp osition of the pro cess into subpro cesses all with their own lter design

Learning lters If we have constructed an architecture with a go o d set of lters the

metho d may b e exp ected to work well Still sometimes we would like to learn the lters

This can b e done by dierent techniques Since Indep endentlter mo dels are quite robust it

oers many p ossibilities for changing the architecture without causing large p olicy changes

This may make them suitable for incremental p olicy renement

PREVIOUS WORK

Previous Work

There have b een quite a few combinations of function approximators and RL We will give a

sample of what has b een done Chapman and Kaelbling implemented the Galgorithm

a metho d based on incrementally computing decision trees by splitting no des receiving in

consistent reinforcement signals The algorithm used Qlearning for learning the Qvalues

which were stored in the leave no des The system was successfully tested on an amazone

environment where the goal was to learn to sho ot arrows at ghosts One of the drawbacks

of their and similar systems is that it could not handle multivariate splitting so that it is

restricted to environments where singular inputs can show a dierence Another problem is

that it may generate huge trees if all inputs are relevant

Boyan used hierarchical neural networks to learn the games of Tic Tac Toe and

Backgammon and showed p erformance gains in using hierarchical nets compared to monolithic

networks Wiering extended this work and was able to reach optimal p erformance levels

on Tic Tac Toe using Qlearning Furthermore he showed which precision levels can b e

obtained by neural networks learning the evaluation function of the endgame of backgammon

Santamaria Sutton and Ram compare CMACs to memorybased function approx

imation with Sarsa Rummery and Niranjan Sutton on the double integrator

and p endulum swing up problems They found that b oth metho ds prot from a nonuniform

variable resolution allo cation of the cellsneurons and that CMACs protted most from this

They found an elegant way to change the input space by using a skewing function which in

creases the size of imp ortant regions such as those around start and goal states The obtained

results of b oth FAs were comparable

Tham incorp orated a set of CMACs in the hierarchical mixtures of exp erts HME

architecture Jacobs et al Jordan and Jacobs He used Singhs Comp ositional

Qlearning CQL architecture which uses relevant task inputs to gate the dierent

controllers His system quickly learned go o d p erformance levels for dierent sequences of

rob ot manipulator tasks

Krose and van Dam use RL with a neural gas architecture for collision free nav

igation The system gets distance measures as input and has to learn to steer a vehicle

left or right in a simulated environment Whenever there is a collision neurons are added

so that critical regions get high resolution The system also removes neurons which have

similar values as their neighbors The exp erimental results a virtual environment show that

the system can quickly nd collision free paths

Schraudolph Dayan and Sejnowski used TDlearning to learn to play Go on a

b oard and showed that the system achieved signicant levels of improvement The program

learned go o d op ening strategies but often made tactical mistakes later in the game In later

phases of the game many complex interactions b etween stones dominate p osition evaluation

This was to our knowledge the rst time anybo dy used a whole grid of lo cal reinforcement

signals to evaluate each p ossible b oard p osition as opp osed to just a single scalar evaluation

This made a big dierence in p erformance

Thrun implemented neuro chess a program which uses TDlearning and explanation

based neural network learning for learning to play chess He learned a mo del to predict likely

sequences of gameplay and used this mo del for training the p osition evaluator The resulting

p erformance of the neural network was of limited success a b etter level of play needs a

more accurate value function which takes the precise b oard constellation into account

Recently Baxter Tridgell and Weaver came up with a promising metho d for learn

CHAPTER FUNCTION APPROXIMATION FOR RL

ing to play chess which is very similar to Samuels approach in the sense that it combines

lo okahead search with reinforcement learning Their system used a simple initially hand

crafted p osition evaluator which was improved by TDlearning by up dating the values of

those leave no des which result from a lo okahead search for the b est move Using lo okahead

makes the accuracy of the p osition evaluator much less imp ortant for go o d play and that

makes learning a useful value function much easier The system played on the internet chess

server and was able to signicantly improve its level of play after learning from only

games

Tadepalli and Ok describ e Hlearning a mo delbased RL metho d maximizing aver

age reward p er time step They show that Hlearning outp erforms discounted mo delbased RL

for particular tasks Furthermore they extend Hlearning by introducing dynamic Bayesian

networks to compactly store the mo del so that the system can cop e with large statespaces

The DBN for storing the mo del is then combined with lo cal linear regression LLR for ap

proximating the value function The results of this combination on an automated guided

vehicle domain show promising results

Sampling approximation

There also exists a dierent way of using mo dels for computing a value function We still

represent the value function in a function approximator but now we generate a set of state

vectors use the mo del or simulator to p erform a onestep lo okahead for all states to obtain

more accurate values p ossibly in combination with Monte Carlo simulations for sto chastic

problems and train the function approximator to learn the new state values The metho d is

in principle a hybrid we can use one representation for the mo del eg Bayesian networks

and use another for the value function eg neural networks Once we have computed a new

generation of training examples we use a training metho d to adapt the function approximator

The simplest metho ds use gradient descent to minimize the Bellman error of the generated

examples This is called the direct metho d but is has b een shown that it may lead to learning

instability Boyan and Mo ore Baird

To overcome the problem of diverging value functions Boyan and Mo ore used a

metho d based on learning the value function backwards during which complete rollouts are

used for computing new value estimates while keeping a large batch set of training examples

This metho d showed promising results for deterministic problems but is computationally

exp ensive for sto chastic problems

Baird discussed residual gradient algorithms where steep est descent on the mean

squared error Bellman residual is p erformed These metho ds converge to a lo cal minimum

on the Bellman error landscap e but tend to b e slow Therefore Baird introduced residual

algorithms which are a mixture b etween the direct metho d and the residual gradient metho d

combining the prop erty of minimizing the Bellman residual at each step with fast learning

The metho d can b e applied for sto chastic problems by combining two indep endent successor

states in a single learning example This is called the two sample gradient metho d Bertsekas

and Tsitsiklis

We have not used these metho ds here b ecause it may b e dicult to select a set of target

p oints Furthermore the function can change dramatically if the environment is very noisy

since new target values may b e quite dierent dep endent on the successor state we have

sampled Finally for real world learning we cannot sample from arbitrary p oints so that the

metho d requires engineering or learning a mo del rst

CONCLUSION

Conclusion

In this chapter we discussed one of the most interesting topics in RL the application of

function approximators to learn the value function We describ ed three dierent types of

function approximators linear networks neural gas and CMACs as well as ways to combine

them with direct RL and mo delbased RL Then we compared them on a so ccer learning task

We found that linear networks had severe learning problems for the so ccer task Although

the linear networks were sometimes able to learn go o d p olicies small up date steps of the

weight vectors sometimes cause large p olicy changes after which go o d p olicies are unlearned

A diculty of reinforcement learning compared to sup ervised learning is that the output

value of examples change and are taken from a nonstationary distribution due to p olicy

changes and exploration b ehavior Therefore in order to receive sucient data from the

same inputoutput mapping the p olicy should not change to o fast and the used function

approximator should b e robust enough so that noise will not aect its approximation very

much We should b e careful in assessing whether we really make an improvement step of a

particular p olicy since it is easy to make changes which decrease the p erformance level of the

p olicy Esp ecially if we want to obtain high p erformance levels we should circumvent this

from happ ening

Approximators which use adaptive state quantization suer from noise which may cause

quick p olicy changes and therefore they are not always very stable One function approxima

tor which discretizes the input space using an a priori decomp osition CMACs turned to b e

quite stable esp ecially when combined with world mo dels CMAC mo dels are able to capture

lots of statistics and makes fast and robust learning p ossible One problem of CMACs is that

lters combining dierent input features need to b e designed by hand Therefore we would

like to use metho ds which can learn the p ossible lters

In this chapter we have not given any sp ecial attention to multiagent RL Future work

on rob otic so ccer may prot by studying team strategies and the integration of p erceptions

of dierent agents By mapping group states to team strategies a useful abstraction can b e

made Furthermore it may b e easier to assign rewards to team strategies than to single player

actions Therefore it may b e a go o d idea to dene a set of team strategies involving reactive

p olicies for each individual agent

CHAPTER FUNCTION APPROXIMATION FOR RL

Chapter

Conclusion

In this nal chapter we describ e our contributions to dierent topics of interest in the rein

forcement learning eld We also describ e the limitations of the prop osed metho ds and what

can b e done to overcome these limitations Finally we close with some nal remarks

Contributions

Exact Algorithms for Markov Decision Problems

Discussion

First of all we compared two dynamic programming algorithms and found that value iteration

is faster than p olicy iteration for computing p olicies for goal directed problems The reason

is that p olicy iteration completely evaluates p olicies b efore making p olicy up dates and that

p olicies in the initial iterations generate many cycles through the state space Such cycles

are certainly not optimal but do cost a lot of computation time On the other hand value

iteration uses its value function directly for greedily changing the p olicy whenever it needs to

b e changed In this way it quickly computes goaldirected p olicies which generate few cycles

and therefore it can stop evaluating p olicies much earlier We conclude that metho ds prot

from immediately using information so that p olicy changes are made earlier

Limitations

Solving Markov decision problems with dynamic programming requires a mo del of transition

probabilities and a reward function for computing p olicies which is often a priori not available

Furthermore if there are many state variables the computational costs b ecome overwhelming

The rst problem can b e solved by using reinforcement learning For solving the second

problem we need to use function approximators which aggregate states

Reinforcement Learning

Discussion

We developed a novel online Q algorithm which computes the same up dates as previous

exact online Q metho ds but is able to make up dates in time prop ortional to the number

of actions and the number of active elements of the function approximator In the case of

CHAPTER CONCLUSION

lo cal function approximators eg lo okup tables which activate a single element at each time

step the gain of the new metho d can b e very large The new algorithm is based on lazy

learning which p ostp ones computations until they are really needed The novel algorithm

was used in exp eriments with lo okup tables for maze problems and with CMACs for learning

to play so ccer Both applications resulted in a large sp eed up

We developed an algorithm for online multiagent Q called Team Q which allows

up dating the Qvalue of some state visited by some agent on a change of the Qvalue of

another state visited by another agent The only thing that is required is that there exists

a directed path b etween the states which now may go through some crossing p oint b etween

the tra jectories of the agents With the new metho d information is propagated along many

more tra jectories than would b e the case if we would only up date along single agent histories

Furthermore the Team Q algorithm can b e used for assigning credit in case of general

interactions b etween agents such as passing the ball where we would like to evaluate the

action on the basis of what happ ens to the ball in the future Although we exp ect the

metho d to b e b enecial for solving real co op erative tasks for which agents work as a group

exp erimental results in a nonco op erative maze environment did not show that Team Q

outp erformed letting agents learn from their individual traces only

Limitations

Direct RL metho ds such as TD and Qlearning have problems to backpropagate infor

mation over many time steps Qlearning can only backpropagate information one step in

the past whereas TD can in principle b e used to backpropagate information over many

time steps but this causes a large variance in the up dates esp ecially for sto chastic problems

which delays the convergence of value functions

The Team Q is able to combine traces of multiple agents but do es not take into account

when these traces have b een generated Therefore very old traces generated by old p olicies

are used to change the current value function which may decrease the p olicys p erformance

The algorithm is also not yet made amenable for dierent kinds of agent interactions the

only kind of interaction we have implemented so far is when dierent agents visit the same

state For more general applications the algorithm would need a denition of interaction

p oints b etween agents An example interaction p oint is a player which decides to pass the

ball to another player This action can b e evaluated on the trace of the other agent since the

decision of passing the ball has b een made and may b e b etter than to use the trace of the

player itself which may just b e waiting around in the middle eld after having passed the

ball

Op en problems

We would like to have metho ds which diminish the variance of TDup dates for large values

of One way of doing this is to use variable Eg we can detect stable landmark states

which have b een visited so often that their path to the goal has b een optimized already and

their value is quite stable Then we can use large values b efore making a transition to them

We could also let the value dep end on the standard deviation of some state value It is a

question whether and how much such metho ds improve TDs abilities

It is still unclear how we can extend the Team Q metho d to allow learning mostly from

traces which are useful for improving the p olicy For this we would need to discard some

CONTRIBUTIONS

traces while keeping others One option to do this is to only combine traces which have not

b een generated to o long ago Another p ossibility is to weigh traces according to their recency

An op en question is whether using such more sophisticated metho ds can really signicantly

sp eed up multiagent Q which uses separate traces

A p ossible large problem of multiagent RL is when traces connect in a tomany way

Eg if a mother is co oking for her children and afterwards the children get a b ellyache

then we should b e able to trace that back to an ingredient the mother used and punish using

that ingredient or even buying it The problem is that in this way the action of co oking has

multiple future paths since we have to take all her children into account Therefore the future

branches and probabilities do not sum to which makes credit assignment and computing

value functions really hard Thus another op en problem is to nd neat metho ds which can

combine such multiple futures into b ounded value functions

Mo delbased RL

Discussion

In Chapter we describ ed mo delbased RL approaches and introduced a novel implemen

tation of the prioritized sweeping PS algorithm Mo ore and Atkeson which is more

precise in calculating priorities for making dierent up dates than the original PS Exp erimen

tal results on some maze problems showed that mo delbased RL signicantly outp erformed

direct RL Furthermore the new PS metho d is more reliable and leads to b etter nal p erfor

mances than the previous PS

Op en questions

The exp erimental result may not necessarily hold for other problems containing many p ossible

outgoing transitions from each state The price our PS has to pay for its exactness is that

its computational eorts dep end on the number of successor states whereas the previous PS

metho d is not Therefore it is computationally more exp ensive Still for problems with

many successors we exp ect that the old PS will have much more problems in really making

the most useful up dates since priorities are only based on a single outgoing transition

Another question is whether even faster management metho ds can b e constructed Al

though this seems dicult dierent metho ds could use other criteria or b e more careful in

up dating states which leave the relative ordering of state values and therefore the p olicy

unchanged

Limitations

One problem of mo delbased RL is that it heavily relies on the Markov assumption TD

approaches can much b etter evaluate a p olicy in case the Markov assumption is violated

since in case we just test the p olicy according to Monte Carlo sampling One way

for extending world mo dels is to use TD world mo dels which combine TD with mo del

based learning Such a mo del may b e useful for learning from learn term consequences of

actions although there may b e many successor states so that it is uncertain whether we can

eciently use it for problems involving many states

CHAPTER CONCLUSION

Exploration

Discussion

We describ ed novel directed information based exploration metho ds which use a predened

exploration reward function for learning where to explore Exploration reward functions are

dened by the human engineer and determine how interesting a particular novel exp erience

for an agent is Examples of such reward functions assign p enalties to stateactions which

have o ccurred frequently or recently Using such reward functions the agent learns an explo

ration value function which allows the agent to select actions based on long term information

gain We used mo delbased RL for learning the exploration value functions and found large

advantages compared to undirected randomized exploration metho ds Metho ds which only

select actions according to information gain have as disadvantage that they do not take im

mediate problem sp ecic rewards into account for selecting actions To prop erly address the

explorationexploitation dilemma we should only select actions which have some signicant

probability of b elonging to the optimal p olicy Therefore we constructed mo delbased inter

val estimation MBIE which takes second order statistics into account for selecting actions

Since MBIE relies on initial statistics we combined it with the directed exploration tech

niques discussed ab ove Exp eriments showed that MBIE was able to improve the directed

mo delbased exploration metho ds in nding nearoptimal p olicies while collecting a much

higher rewardsum during the training phase

Op en problems

There are still some unresolved problems in exploration One problem is how to handle large

or continuous state spaces for which we are never able to visit all states For such problems

we have to use function approximators and learn exploration functions MBIE could then

again b e implemented by keeping track of the variance of all adjustable parameters of the

chosen function approximator

Another problem is how to explore eciently in nonstationary environments We could

use the recency reward rule for such environments but how could we use the mo del eec

tively Clearly we should discount early exp eriences but it is in general dicult to set such

discounting parameters Furthermore if the environment is partly changing we should b e

able to nd out how the environment is changing and thus our exploration p olicy should b e

tuned towards discovering sp ecic changes of the environment

Another problem in exploration is that in some problems it may b e very dicult to achieve

any reward at all Eg if only very few action sequences lead to the goal state and all others

lead to some other terminal state then how could we nd the goal at all We can use

exhaustive search trying out all action sequences but there could b e exp onentially many of

them It may b e that the only p ossibility is to use a priori goaldirected knowledge in order

to b e able to eciently nd the goal One p ossibility is to implement a b ehavior which is

guaranteed to nd the goal although it will most probably not nd a very ecient solution

Such a b ehavior could b e used to generate interesting initial exp eriences which can then b e

used to improve the p olicy This pro cess of designing and rening is also referred to as shaping

Dorigo and Colombetti

CONTRIBUTIONS

POMDPs

Discussion

We describ ed POMDPs in Chapter and introduced a new algorithm HQlearning for

quickly solving some large deterministic POMDPs HQlearning is based on decomp osing a

problem into a sequence of reactive p olicy problems which are solvable by p olicies mapping

observations to actions Dierent p olicies are represented by dierent agents Each agent

learns HQvalues which determine for which observations control will b e transferred to the

next agent HQs memory is solely embo died in a p ointer which determines which agent is

active at the current time step By using such a minimal representation for the shortterm

memory HQ can quickly learn how to use it

Limitations

HQ learning is designed for deterministic POMDPs featuring xed start states However HQ

could also b e used in combination with map learning algorithms Eg we can learn maps

for dierent regions where transitions b etween regions are determined by agent transitions

We could even go b eyond this and hierarchically learn a map where at the lowest level we

have detailed observation or state transitions and at higher levels we have transitions b etween

regions The advantage of such an approach is that hierarchical planning b ecomes p ossible

for solving large problems

HQlearning is a kind of hierarchical decomp osition metho d but do es not learn complete

p olicytrees It could b e extended to allow for storing a whole tree containing p olicy no des

instead of singular actions and observation sequences as transitions Then a p olicy is

followed for some time after which a transition is made to another p olicy etc In this way

ordinary p olicy trees of the kind describ ed in Kaelbling et al may b e compressed

by allowing for collapsing multiple subsequent branches or entire subtrees inside a single

reactive p olicy

HQ suered a lot from the limited exploration capabilities In many trials no solution

was found at all since the p olicy was generating cycles in state space If however we allowed

our undirected exploration metho ds to use more randomized actions we continually execute

a random walk b ehavior and do not really test the current p olicy Such a random walk is

not a very ecient exploration technique and neither do es it make learning the p olicy easier

Therefore we should construct directed exploration metho ds for POMDPs which is dicult

however since the input used by the p olicy is usually highly ambiguous A way to overcome

this is to use more sophisticated representations for the internal state such as b elief states

but these make p olicy construction hard in their own ways

Op en questions

A question remains whether we should not use dierent metho ds used for combinatorial

optimization for solving POMDPs since POMDPs are hard problems Eg we could use

genetic algorithms Holland tabu search Glover and Laguna or the ant colony

system Dorigo Maniezzo and Colorni Dorigo and Gambardella Gambardella

Taillard and Dorigo Although currently multiple researchers start applying RL for

combinatorial combination problems eg Boyan a real comparison b etween them

would b e very interesting The ant colony system is in eect a RL metho d and is very

CHAPTER CONCLUSION

successful solving a wide range of tasks Dorigo and Gambardella Since general RL

metho ds are very go o d in dealing with sto chasticity they may nally outp erform metho ds

such as tabu search for dealing with complex sto chastic problems RL makes small changes

which on average improve the p olicy Lo cal greedy search metho ds try out all changes

and keep the b est but for sto chastic problems they need many evaluations for testing which

swapmove will result in the largest improvement

Function Approximation

Discussion

We summarized the basics of a number of function approximators and introduced a new

variant of the neural gas algorithm We also describ ed a novel combination of CMACs and

world mo dels and showed that CMACmodels were able to outp erform a number of other

metho ds including the combination of Q with linear networks neural gas CMACs and

an evolutionary metho d called PIPE Salustowicz and Schmidhuber at learning to play

so ccer We noted that for ecient RL the function approximator should b e stable thereby

allowing a consistent p olicy evaluation CMAC mo dels are very stable and easily trained

which makes them a p owerful candidate for continuous state or action RL

Limitations

CMAC mo dels rely on an a priori dened structure which determines which relations can

b e learned b etween input features It would b e nice to learn the structure by reinforcement

learning Eg HQlearning could b e combined with CMACs mo dels where HQlearning tries

structural changes and keeps track of the average p erformance of the function approximator

given that particular input combinations are used Thus HQ can learn the structure which

leads to the largest reinforcement intake

We can also incrementally sp ecialize lters if their predictive ability is small or if the

variance of the Qvalues of its cells is large The advantage of learning a structure of a mo del

such as CMACs is that it is easy to add a lter without changing the overall p olicy This

is in contrast with lo cal mo dels such as neural gas where adding neurons can result in large

p olicy changes

Other algorithms to learn structures are oine metho ds such as GAs and tabu search

Their problem however is that they need to test multiple dierent structures from which

only few trained structures are used This results in throwing away a lot of information

HQlearning or an incremental approach can learn online and improve a single structure

Final Remarks

Reinforcement learning can b e used as a to ol for designing many dierent intelligent computer

systems it can b e used for improving simulation mo dels for designing autonomous vehicles

for controlling space missions and for controlling forest res Esp ecially for multiagent

environments a lot of research needs to b e done to make robust and well organized systems

Such environments often consist of multiple interacting elements which are adapted together

to optimize a sp ecic group criterion determining the desired groups b ehavior RL can b e

used very well for solving such problems and in fact we want to use RL for controlling a team

FINAL REMARKS

of forest re ghting agents which have the goal to minimize the costs of forest re ghting

op erations Wiering and Dorigo

The largest problem of applying reinforcement learning may ultimately b e that construct

ing a reward function can b e very hard First of all it may b e dicult to optimize multiple

evaluation criteria together since it is dicult to weigh them Eg consider a driver of a

racingcar who has the goal to minimize the risk of endangering his life while still maximizing

the probability of winning the race Which risk should he take or how much worth is his life

The second problem of reward functions is that in the real world dierent agents may try

to solve their own task but doing this they may interfere with each other If we consider

a large articial world with many tasks p erformed by dierent RL agents it is dicult to a

priori construct reward functions which take the interactions into account If we would just

give each agent their own reward function they would not care if they would b e an obstacle

for other agents p erforming their tasks and that may ultimately lead to a comp etition b etween

agents Eg one agent which is hindered by another one may learn to make it imp ossible

for the other agent to b ecome an obstacle by closing him in rst This is clearly not what we

want One solution is to weigh all tasks and design a group reward criterion but this makes

learning tasks for each agent much harder due to the agent credit assignment problem how

can we assign credit to each individual agent based on the groups outcome

Thus for complex multiagent environments there remain two options organize the

agents with clearly dened priorities and taskdep endencies or let the agent evaluate each

other and pass their evaluations as reward signals to other agents For the rst option

priorities could b e swapped and the b est organization maximizing the overall colonys reward

kept This could result in a set of complete team strategies or a set of interaction proto cols

which inuence each agents p erception of the world state Agents would not b e able to

change that organization

For the second metho d agent take so cial reward signals into account next to their task

oriented reward function If we are unable to sp ecify such so cial reward functions agents

should b e able to learn ethical reward functions which improve the groups b ehavior Learning

such reward functions may b e very dicult but it op ens up a huge variety of p ossibilities

Furthermore if the task reward functions are well sp ecied slowly adapting the so cial reward

functions based on the groups evaluation may only help

Since the real world consists of many agents which adapt themselves according to the

rewards they get RL may b e an imp ortant to ol for studying what kind of intelligent envi

ronments work b est There are so many ways so that we conclude by stating that there is

no way of knowing which role RL will play in the future Only time can tell

CHAPTER CONCLUSION

App endix A

Markov Pro cesses

A Markov pro cess MP is a mo del for capturing the probabilistic evolution of a system A

Markov pro cess is used to mo del time sequences of discrete or continuous random variables

states which satisfy the Markov prop erty The Markov prop erty states that the transition

to a new state only dep ends on the current state Markov chains are a kind of discrete time

Markov pro cesses which have a nite set of states The whole dynamics of the system is

governed by the initial state probability distribution and the probabilistic transition function

For simplicity we consider Markov chains which consist of

A discrete time counter t T where T denotes the length of the pro cess and

may b e innite

A set of states S fS S S g where the integer N denotes the number of states

The active state at time t is denoted as s j with j S

A probabilistic transition matrix P which determines the probability of the active state

s at the next time step given s The conditional probabilistic transition function

t t

P s j js i denotes the probability of making a transition to state j from state

t t t

i at time t We consider stationary probabilistic transition functions which have the

prop erty that they are time indep endent Therefore we will drop the time index from

P and simply write P

A probability distribution over initial states o where o j denotes the probability

that the sequence starts in state j

We will use matrix notation for denoting the probabilistic transition function The sto chas

tic transition matrix P consisting of entries P denotes the probability of making a transition

from state i to state j

P P s j js i

ij t t

The sto chastic matrices we are interested in have the following prop erties

All P

P i S

Note that most authors require the last sum to b e equal to However with our denition

we allow the pro cess to stop at some p oint with some probability This will b e useful in our

study of Markov chains with terminal states

APPENDIX A MARKOV PROCESSES

A Markov Prop erty

For a Markov chain the transition matrix is indep endent of previous states and dep ends only

on the current state that is for each t

ps j js i ps j js i s s

t t t t t

When this is true we say that the Markov property holds

A Generating a State Sequence

The Markov chain is now the nite state pro cess which is describ ed by the tuple S P o

Simulating Markov chains is fairly simple rst we select an initial state s according to the

probability distribution over initial states o and then we just select a state s given s

t t

according to the probabilities P s js The resulting generated sequence of states is often

t t

called a statetra jectory Each statetra jectory H s s s has a sp ecic probability

which can b e computed by

P H o s P s js

i i

If we could observe a huge amount of suciently long statetra jectories like the one ab ove

we would b e able to compute an approximation to the initial state probabilities and the state

transition function by counting how often initial states or state transitions have o ccurred and

then averaging them

A State Occupancy Probabilities

We are interested in predicting the future given a current active state and our mo del of the

Markov pro cess However we cannot just predict a single statetra jectory since each one of

them may have a tiny probability of really o ccurring Instead of statetra jectories we predict

the probability distribution that each state is active at a sp ecic future time step These

probabilities are also called state occupancy probabilities We will denote the state o ccupancy

T T

probabilities at time t by the vector o ps S ps S Where o denotes the

t t t n

transp ose of o

The dynamics of the state o ccupancy probabilities over the chain are resulting from re

cursively applying the following equation

T T

o P o

t t

When we want to predict the future we cannot b e sure in which state s the pro cess is at any

time t and therefore the uncertainty may b e quite high However when it is p ossible to use

intermediate results eg when we can observe the state at a particular timestep t this is

useful for calculating o The knowledge of b eing in a particular state s helps to diminish

t t

the prediction error over the next state o ccupancy probabilities o but do es not always

decrease the entropy of o Eg lo ok at Figure A we may have a case in which a state j

We consider regular Markov chains which means that there exists a number L so that all states are

connected by a path of length at most L

A STATE OCCUPANCY PROBABILITIES

go es to two successors l and m with probability each If at time t we know s i

we may have a transition to s j or s k b oth probability where k go es to state m

t t

with probability Then the entropy of the predicted state o ccupancy probabilities at time

t given that we are in state i at time t is l og l og bits

After we know that we have stepp ed to state j at time t and we predict again the entropy

is l og l og bit Thus additional information may make our prediction

ab out the future state less certain

l m St+1 1.0 0.5 0.5

j k St

0.5 0.5 i

St-1

Figure A A Markov chain where entropy is not monotonically increasing

When we do not know ps j but we know s we can compute the state o ccupancy

probability vector o which stores the probability that the system is in state i for i

N at timestep t k as follows

ps j js i P

Where P is the k step transition matrix We may also calculate this by using

k l

ps j js i P P

with l k which is the ChapmanKolmogorov equation The logic b ehind this equation

is that the probability of going from one state i to another state j in k steps is equal to the

probability of going from i to each p ossible inb etween state m in l steps probability P

k l

and then going from m to j in k l steps with probability P Note that the path which

is taken from i to j do es not matter due to the Markov prop erty

In particular when k is a p ower of we may use the following equation which uses

recursion to eciently calculate the probability

ps j js i P

Given o we can calculate o by

T T k

o o P

tk t

APPENDIX A MARKOV PROCESSES

A Stationary Distribution

T T

The stationary steadystate distribution x has the following prop erty x x P This

means that the probabilities will not change anymore by lo oking one step further in time ie

the dynamics of the state o ccupancy probabilities b ecomes a xed p oint has b een reached

T T

We can nd x by solving x x P directly by iteration or as follows

T T

x I ONE P

where ONE is a matrix containing a on each place and is a vector containing only s

I is the identity matrix Resnick

A Prop erties of Markov Chains

There are some prop erties of Markov chains which are useful for classifying states see also

Resnick Bertsekas and Tsitsiklis

we say that j When it is p ossible to go from state i to state j ie k so that P

is accessible from i

If for two states i and j it holds that i is accessible from j and j is accessible from i

that means that there is a path from one to the other and vv then we say that i and

j communicate

A Markov chain is irreducible if there is a path b etween each pair of states that is all

states in S communicate with each other

When P for some i we say i is an absorbing state

When P for all j S we say that i is a terminal state Note that this denition

is dierent from that of most other authors who use the same denition for terminal as

for absorbing states

When there exists a set of states C S with the prop erty that for each i C and

for all k and all states in C communicate we say that C is each j S C P

a recurrent class This means that the pro cess will not leave C once it has entered a

state i C

States that do not b elong to any recurrent class are called transient For transient

as k That means that the probability of returning to a states we have P

transient state go es to as the number of steps after the visit go es to innity which

means that o i for t

A chain is called ergodic if it is irreducible the complete set of states S is recurrent

and a stationary distribution exists

A COUNTING VISITS

A Counting Visits

Sometimes it is useful to know the average number of times the pro cess will b e in a particular

state j starting in the current state i until T steps in the future The exp ected number of

visits K of a state j starting in i until T steps in the future can b e calculated by

K P

For T we can calculate the complete counting matrix K with elements K by

K P I P

where I is the identity matrix and Q denotes the inverse of Q Note that in this case the

pro cess should terminate since otherwise K go es to for at least one i j pair Termination

may b e assured by including a terminal state which is accessible from all other states

A Examples

Example Lo ok at the following example The set of states is S f g and the

probabilistic transition matrix P is given by

0.4 0.6

1 2 0.2

0.3 0.6 0.5

0.4

Figure A The Markov chain of Example

This Markov chain in shown in Figure A The chain is irreducible since all states

communicate There are no transient states The stationary distribution x can b e calculated

as follows

x x x

x x x x

x x x

APPENDIX A MARKOV PROCESSES

From this follows x x and x x Since x x x we get x and it

x and x The chain is ergo dic Note that I P is singular follows that x

therefore we cannot compute the counting matrix for the innite case in this case all states

will b e visited innitely many times

Example Now have a lo ok at the following example Again S f g Now P is given

State is a transient state since once the pro cess has left state it will not go back to

it anymore States and form a recurrent class The stationary distribution is given by

Note that a transient state has probability in the stationary distribution x

The chain is also not ergo dic

Example Now have a lo ok at the following example S f g and P is given by

Here state is a terminal state Since the terminal state is accessible from all states this

means that the pro cess will always stop This also implies that all nonterminal states are

transient states The chain is not ergo dic We can calculate the future visits matrix K

I P This gives

We can see that the terminal state is reached one time from all states and that the exp ected

number of transitions b etween state and is b ounded

A Markov Order

When we consider Markov chains we make the requirement that the previous states are not

allowed to have any inuence on the current transition which is therefore solely based on the

current state When a transition matrix has this prop erty we call it memoryless However

sometimes states are not uniquely p erceived Instead we may b e in the p osession of a partial

description of a state This partial state or observation may not always contain all information

needed to nd a p erfect prediction of the subsequent state In such cases previous states may

b e used to improve the prediction abilities Therefore we will shortly describ e Markov chains

where previous visits of states inuence the current dynamics

The Markov order mo of a pro cess is dened as the minimal number of previous steps

which inuence the transition function When i S

ps i ps ijs s

t t t

A HIGHER ORDER TRANSITION FUNCTIONS

the Markov order is mo and we see that the current state and the previous states do

not inuence the next transition This means that there is a xed probability of visiting each

state

We will use a recursive denition for the Markov order The Markov order is mo for

mo when it is not mo and when the following holds i S

ps ijs s ps ijs s A

t t tmo t t

The equation says that no more than mo previous states can b e imp ortant for determining

the transition probabilities There are two sp ecial cases the rst is when mo which we

have already seen since it is a requirement for Markov chains the Markov prop erty holds

When mo t and go es to as t we say it is a pro cess of indenite Markov order

Dealing with pro cesses with denite Markov order higher than one which we will call

higher order Markov pro cesses can b e done as follows we construct new states which are

Cartesian pro ducts of the last mo steps The new space of higher order states contains N

elements although usually only a small fraction of all elements do o ccur In this new space

however we can consider Markov chains of order

A Higher Order Transition Functions

When we consider Markov pro cesses of higher order mo and we want to predict the

mo th

dynamics of the states we have to use higher order states i and mo order probabilistic

transition functions For this we introduce higherorder states i which contain all informa

tion ab out the last mo steps the history of the state tra jectory The state i is dened as

follows Given s j s j s j s j we compute i as follows

t t t tmo mo

mo mok

i s N A

That means that i assigns a number to uniquely describ e each dierent sequence of states

It is useful to number the rst order states as follows S f N g which gives

mo mo mo

the following numbering for i i HS f N g HS is the resulting set

of higher order states Note that when mo i s and HS S

For dening higher order transitions of order mo we use

mo mo

p j ji ps j s j s j js j s j

mo t t tmo mo t tmo mo

mo mo

with i and j dened by equation A

mo mo mo mo

p j ji denotes the probability of going from i to j in step based on the last

mo states which are covered in i We can again use matrix notations We construct the

mo mo

higher order matrix HP with size N N where we dene

mo mo

HP p j ji

i j mo

Note that each row of HP contains only n nonzero elements since only the rst state can

b e changed and thus we could represent the matrix more eciently The Markov order is the

longest sequence of previous states needed for determining the probability of all transitions

APPENDIX A MARKOV PROCESSES

from each p ossible history For most states i the Markov order will b e lower which means

mo mo l l

that we do not have to know p j ji but can rely on p j ji with l mo and therefore

l mo

i i In such cases we can represent the transition function more compactly by a tree

mo mo mo mo n

j ji that is the probability of going from i to j in n steps We can calculate p

using the previous mo steps as follows

n mo mo n

p j ji HP

mo mo

mo i j

A Examples

0.4 0.4

(0,0) = 0 (0,1) =1 0.6 0.4 0.8 | St-1 0= 0.5 0.6 0.8 0.6 0 1

0.5 | St-1 = 1 (1,1) = 3 (1,0) = 2

0.5 | St-1 = 1 0.2 | S 0= t-1 0.2

0.5

Figure A Second order Markov chain be

Figure A The resulting Markov chain of

fore introducing higher order states

gure after converting the chain to higher

order states and transitions

Example Given the order mo Markov chain in Figure A The states are

f g and in state the next transition dep ends on the previous state We construct the

higher order states HS f g or equivalently HS f g HP

is given by

F r omT o

Figure A shows the resulting Markov chain The resulting higher order transition matrix

is irreducible do es not contain absorbing states or transient states The stationary o ccupancy

probabilities are given by x The chain is ergo dic Finally for the stationary

that we observe a distribution over the states in Figure A we have probability p

and p

Example Given the order Markov chain in Figure A we can of course again dene

HS and HP in the same way but now HP would b e a matrix A b etter way in this

case is to dene HS with four states as follows not all states are needed HS f g

state is the set of order states f g state f g state

f g and nally state f g Note that we split the original state which

needs the previous state for calculating its transition probabilities into a set of two states

f g The resulting Markov chains is shown in Figure A The transition probability matrix

now lo oks as follows

A EXAMPLES

0.4 0.3 0.6 0.6 0.2 0.2 | St-1 <> 0

0 1 0 1

0.4 0.6 0.5 0.3 | St-1 <> 0 1.0 | S = 0 0.6 t-1

3 2 1.0 0.5 | S 0<> 0.4 2 t-1

0.4

Figure A Markov chain with splitted

Figure A Example Higher order Markov

higher order states

chain before introducing higher order states

F r omT o

Note that this matrix is invertible just check P v which means that all v

When we would have used the full higher order matrix then it would not have b een

invertible This is b ecause some rows would b e the same Therefore an invertible matrix

contains all necessary information and we should not try to make matrices unnecessarily

large by introducing redundant information as was the case in Example

Now we can calculate the stationary o ccupancy probabilities x

From these we can calculate the probabilities that we will see each of the states and

when the pro cess has settled down in the steadystate distribution by just summing up the

stationary o ccupancy probabilities of the splitted state

APPENDIX A MARKOV PROCESSES

App endix B

Learning rate Annealing

Consider the following conditions on the learning rate which dep ends on t Here we use t

to denote the number of visits of a sp ecic stateaction pair

The reason for the rst condition is that by up dating with learning rate each nite

distance b etween the optimal parametersetting and the initial parameters can b e overcome

The reason for the second condition is that the variance go es to zero so that convergence

in the limit will take place otherwise our solution will b e inside a ball with the size of the

variance containing the optimal parametersetting but the true singlep oint solution will b e

unknown We want to choose a function f t and set f t In Bertsekas and Tsitsiklis

is prop osed Here we prop ose to use more general functions of type f t f t

the two conditions on the learning rate are with and will pro of that for

satised

The function is a step function since a sum is used In the following we use a continuous

function in order to make it easy to compute Since there is a mismatch b etween the

continuous function and the step function we use two functions one which returns a smaller

value than the sum the lower b ound function and another which returns a larger value the

higher b ound function

For the rst condition we use a lower b ound on the sum

t t

From this and from dealing with the sp ecial case separately follows

For the second condition we use a higher b ound on the sum

t t

APPENDIX B LEARNING RATE ANNEALING

From this follows

Hence we can use the following family of metho ds for decreasing the learning rate f t

with Since in general we want to choose the learning rate as high as p ossible

should b e chosen close to In practical simulations often constant learning rates are used

ie but this may prohibit convergence

App endix C

Team Q Algorithm

We have extended the Fast Q algorithm to the multiagent case The complete Team Q

algorithm will b e describ ed here We will rst introduce additional indices for the dierent

agents histories and use the following variables

K Number of agents

H History list of agent i

s a Eligibility trace of agent i l

Global delta trace of agent i

Global eligibility trace of agent i

s a Lo cal delta trace for SAP s a of agent i

sel f v isited s a Bo olean which is true if SAP s a o ccurred in the tra jectory

of agent i

v isited s a Last trial number in which SAP s a o ccurred in the tra jectory

of agent j where j may or may not b e equal to i If it is not there should b e a

connected path b etween SAP s a to the tra jectory of agent i

r ev isited s a Bo olean which indicates whether SAP s a has b een visited

multiple times by agent i

First we adapt the previous pro cedures Global Update and Local Update to take into

account that we have to sp ecify the current agent These pro cedures should always b e used

for the multiagent Q where the value function is shared by the agents indep endent of the

choice whether we connect traces or not The new Local Update simply adds the contributions

from all agent tra jectories to up date a Qvalue

APPENDIX C TEAM Q ALGORITHM

Lo cal Up dates a

t t

For i to K Do

a If v isited s a

i t t

a M v isited s a

i t t

s a s a l a D D

t t i t t

i i

a s a

i t t

a If M N

s a a l

t t

a v isited s a N

i t t

Qs a Qs a s a D

t t t t t t

For Global Update we rst have to change the variableindices to take the active agent the

agent which made the last step into account this is straightforward to do so we do not show

the new pro cedure here In the pro cedure Global Updates addendum we change the rst

lines line a and b to make sure that we insert revisited states in the b eginning of the

history list Here the union op erator is implemented as an insertion at the head of a list

Global Up dates a i

t t

If v isiteds a

t t

a H H s a

i i t t

b v isited s a N

i t t

v isited s a c sel f

i t t

d r ev isited s a

i t t

Else

e H H s a

i i t t

f H H s a

i i t t

v isited s a g sel f

i t t

h r ev isited s a

i t t

Finally Global addendum should b e up dated so that we reset the variables r ev isited s a

and sel f v isited s a in lines a and a

Connecting traces algorithm Now we have adapted the existing pro cedures for the

value function sharing multiagent case We will call the multiagent Q metho d without

connecting traces Independent Q If we want to connect traces all we have to do is to

check if an agent executes a SAP which has b een executed already by another agent and call

the pro cedure Connect as shown b elow Note that in our algorithm we compare whether the

same SAP has o ccurred in two agent tra jectories another metho d would b e to check whether

an agent visits the same state as another agent Although this would cause more links b etween

tra jectories the second agent may have chosen a dierent action since the action selected by

the rst agent turned out not to b e very go o d and therefore learning on that future tra jectory

may b e more harmful than if we only learn if the same SAP has o ccurred

In the pro cedure Connect we have the meeting p oint s a and want to let SAPs from the

tra jectory of agent i learn on the future dynamics of tra jectory Goal The variables D el ta

and tr ace store lo cal values of s a for tra jectory Goal T r ace is the eligibility trace of

the current SAP P r ev which we insert

We traverse the tra jectory of agent i by going backwards in the chain The variables S

P r ev

and A denote the state and action of the history element P r ev of agent i Whenever SAPs

pr ev

P r ev from i have not b een visited by the tra jectory called Goal we insert them in the list

and up date the trial eligibility traces and values After inserting SAP we take its prede

cessor and check in which agent tra jectories the preceding SAP o ccurred so that they may

also start learning on the tra jectory of Goal This is implemented by the lines starting from

f b elow Essentially it implements a recursive call which stores SAPs from all tra jectories

in the history list of Goal if there is a directed path to SAP s a to the list of Goal If we

A A

hop over to another tra jectory we recompute the trace variables Finally line implements

the case in which the SAP P r ev already has a nonzero eligibility trace on the tra jectory of

Goal but since it has b een revisited its predecessor may still not have eligibility

Connects a P r ev i Goal del ta tr ace tr ace tr ial

If P r ev NULL return

s a tr ace l tr acel

P r ev P r ev new

If v isited s a AND sel f v isited s a AND

P r ev P r ev i P r ev P r ev

Goal

l AND v isited s a tr ial

new m i P r ev P r ev

a v isited s a v isited s a

P r ev P r ev

Goal Goal

A A

s a l b l

P r ev P r ev new

Goal

c s a del ta

P r ev P r ev

Goal

at end of l istH s a d H inser t

P r ev P r ev

Goal Goal

A A

e v isited s a v isited s a

P r ev P r ev

Goal Goal

A A

f For j to K

v isited s a f If j Goal AND sel f

j pr ev pr ev A

fa If j i

s a fa new tr ace l

pr ev pr ev

fa new tr ace l

new

fa tr ial v isited s a

j pr ev pr ev

fb else

tr ace tr ace fb new

fb tr ial v isited s a

i pr ev pr ev

fc P r ev P r ev histor y el tH s a

j P r ev P r ev

fd If tr ace AND tr ace

m m

tr ace new tr ace tr ial fd C onnects a P r ev j Goal del ta new

v isited s a Else If r ev isited s a AND sel f

i pr ev pr ev i pr ev pr ev

a P r ev P r ev histor y el tH s a

i P r ev P r ev

b C onnects a P r ev i Goal del ta tr ace tr ace tr ial

Finally we have to call the pro cedure connect This we do if we observe that the current

SAP s a has also o ccurred in the tra jectory of another agent There are two ways of con

necting tra jectories SAPs are inserted in the history list of the active agent so that they

will start learning on what the active agent do es in the future This is done in the rst part

of the pro cedure b elow a SAPs are inserted in the history list of the other agent so

that SAPs which o ccurred in the list of the active agent will start learning on the tra jectory

of the other agent a Below we present the complete Team Global Update pro cedure It

APPENDIX C TEAM Q ALGORITHM

rst calls Global Update and then tries to connect tra jectories We call the active agent Activ e

Team Global Up dates a Activ e

t t

Global Updates a Activ e

t t

For i To K

v isited s a a If i Activ e AND sel f

a P r ev P r ev histor y el tH s a

a del ta s a

Activ e

s a a tr ace l

a tr ace l s a

Activ e

a tr ial v isited s a

a if tr ace AND tr ace

m m

a C onnects a P r ev i Activ e del ta tr ace tr ace tr ial

For i To K

v isited s a a If i Activ e AND sel f

a P r ev P r ev histor y el tH s a

Activ e

a del ta s a

a tr ace l s a

Activ e

s a a tr ace l

a tr ial v isited s a

Activ e

a if tr ace AND tr ace

m m

a C onnects a P r ev Activ e i del ta tr ace tr ace tr ial

App endix D

PS for Function Approximators

D PS for Neural Gas

The mo delbased up date of the Qvalue Qn a Qup daten a lo oks as follows

i i

P aR n a j V j Qn a

n j i i

where P a P j jn a The details of our PS lo ok as follows

n j i

OurPrioritizedSweepingNGx

Compute active neurons c c

For k to z do

a Update c a do

a Qup datec a

b Set jc j to

c Promote c to top of queue

While n U queue nil

max

a Remove top c from the queue

b c

c Predecessor neurons i of c do

c V i V i

c a do

c Qup datei a

c V i max Qi a

c i i V i V i

c If jij

c Insert i at priority jij

d n n

Empty queue but keep i values

Here U is the maximal number of up dates to b e p erformed p er up datesweep The

max

parameter IR controls up date accuracy

APPENDIX D PS FOR FUNCTION APPROXIMATORS

D PS for CMACs

The mo delbased up date of the Qvalue Q c a Qup datek c a lo oks as follows

Q c a P aR c a j V j

k k k

a P j jc a The details of our PS lo ok as follows where P

OurPrioritizedSweepingCMACx

Compute active cells f f

For k to z do

a Update f a do

a Qup datek f a

b Set j f j to

k k

c Promote k f to top of queue

While n U queue nil

max

a Remove top k c from the queue

b c

c Predecessor cells k i of k c do

c V i V i

c a do

c Qup datek i a

c V i max Q i a

k k

c i i V i V i

k k k

c If j ij

c Insert i at priority j ij

d n n

Empty queue but keep i values

D NonPessimistic Value Functions

To compute nonp essimistic value functions for multiagent CMACs we decrease the prob

ability of the worst transition from each ltercellaction and then renormalize the other

probabilities Then we use the adjusted probabilities to compute the Qfunctions In the

following C a counts the number of transitions of cell i to j in lter k after selecting action

a counts the number of times action a was selected and cell i of lter k was ac a and C

a the estimated transition probability by dividing them Then we tivated We obtain P

substitute the following for Qup datek c a

D NONPESSIMISTIC VALUE FUNCTIONS

Compute NonPessimistic Qvaluek i a

m arg min fR i a j V j g

k k

n C a

P P a

P P P

2n 4n

P a

k k

P a P a

im im

j m

a C

k k

a a P a P

k k

ij ij

a aC C

im i

Qup datek i a

The variable z which determines the step size for decreasing worst transition proba

bilities To select the worst transition in step we only compare existing transitions we

check whether P a holds See Figure D for a plot of the function Note that if

there is only one transition for a given ltercellaction triplet then there will not b e any

renormalization Hence the probabilities may not sum up to Consequentially if some

ltercellaction with deterministic dynamics has not o ccurred frequently then it will con

tribute just a comparatively small Qvalue and thus have less impact on the computation of

the overall Qvalue

Non-Pessimistic

New P

1 0.8 0.6 0.4 0.2 0

100 90 80 70 60 0 50 0.2 40 n 0.4 30 0.6 20 P 0.8 10

Figure D The nonpessimistic function new P which decreases the probability of the worst

transition decreased as a function of the maximum likelihood probability P and the number

of occurrences n

APPENDIX D PS FOR FUNCTION APPROXIMATORS

Bibliography

Albus J S a A new approach to manipulator control The cereb ellar mo del articula

tion controller CMAC Dynamic Systems Measurement and Control pages

Albus J S b A theory of cereb ellar function Mathematical Biosciences

Asada M Uchibe E No da S Tawaratsumida S and Hoso da K A visionbased

reinforcement learning for co ordination of so ccer playing b ehaviors In Proceedings of

AAAI Workshop on AI and Alife and Entertainment pages

Atkeson C G Schaal S A and Mo ore A W Lo cally weighted learning Articial

Intel ligence Review

Axelro d R The evolution of cooperation Basic Bo oks New York NY

Baird L Residual algorithms Reinforcement learning with function approximation

In Prieditis A and Russell S editors Machine Learning Proceedings of the Twelfth

International Conference pages Morgan Kaufmann Publishers San Francisco

Baluja S Populationbased incremental learning A metho d for integrating genetic

search based function optimization and comp etitive learning Technical Rep ort CMU

CS Carnegie Mellon University

Barto A G Bradtke S J and Singh S P Learning to act using realtime dynamic

programming Articial Intel ligence

Barto A G Sutton R S and Anderson C W Neuronlike adaptive elements that

can solve dicult learning control problems IEEE Transactions on Systems Man and

Cybernetics SMC

Baum E A prop osal for more p owerful learning algortihms Neural Computation

Baxter J Tridgell A and Weaver L Knightcap A chess program that learns by

combining TD with minimax search Technical rep ort Australian National University

Canberra

Bayse K Dean T and Vitter J Coping with uncertainty in map learning Machine

Learning

Bell A and Sejnowski T The indep endent comp onents of natural scenes are edge

lters Vision Research To app ear

BIBLIOGRAPHY

Bellman R Adaptive Control Processes Princeton University Press

Berliner H Exp eriences in evaluation with BKG a program that plays backgammon

In Proceedings of IJCAI pages

Berry D and Fristedt B Bandit Problems sequential al location of experiments

Chapman and Hall LondonNew York

Bertsekas D P and Tsitsiklis J N Neurodynamic Programming Athena Scientic

Belmont MA

Boutilier C and Poole D Computing optimal p olicies for partially observable deci

sion pro cesses using compact representations In AAAI Proceedings of the Thir

teenth National Conference on Articial Intel ligence pages Portland OR

Box G Jenkins G M and Reinsel H C Time series analysis forecasting and

control Prentice Hall

Boyan J A Mo dular neural networks for learning contextdependent game strategies

Masters thesis University of Chicago

Boyan J A and Mo ore A W Generalization in reinforcement learning Safely

approximating the value function In Tesauro G Touretzky D S and Leen T K ed

itors Advances in Neural Information Processing Systems pages MIT Press

Cambridge MA

Boyan J A and Mo ore A W Using prediction to improve combinatorial optimiza

tion search In Proceedings of the Sixth International Workshop on Articial Intel ligence

and Statistics AISTATS page

Bradtke S J and Barto A G Linear leastsquares algorithms for temp oral dierence

learning Machine Learning

Caironi P V C and Dorigo M Training Qagents Technical Rep ort IRIDIA

Universite Libre de Bruxelles

Camp os L M D Huete J P and Moral S Probability intervals A to ol for

uncertain reasoning International Journal of Uncertainty Fuzziness and Know ledge

Based Systems

Cassandra A R Exact and Approximate Algorithms for Partial ly Observable Markov

Decision Processes PhD thesis Brown University Providence RI

Cassandra A R Kaelbling L P and Littman M L April Acting optimally in

partially observable sto chastic domains Technical Rep ort CS Brown University

Providence RI

Chapman D and Kaelbling L P Input generalization in delayed reinforcement learn

ing In Proceedings of the th International Joint Conference on Articial Intel ligence

IJCAI volume pages Morgan Kaufman

BIBLIOGRAPHY

Chrisman L Reinforcement learning with p erceptual aliasing The p erceptual dis

tinctions approach In Proceedings of the Tenth International Conference on Articial

Intel ligence pages AAAI Press San Jose California

Cichosz P Truncating temp oral dierences On the ecient implementation of TD

for reinforcement learning Journal on Articial Intel ligence

Cli D and Ross S Adding temp orary memory to ZCS Adaptive Behavior

Cohn D A Neural network exploration using optimal exp eriment design In Cowan

J Tesauro G and Alsp ector J editors Advances in Neural Information Processing

Systems pages San Mateo CA Morgan Kaufmann

Cramer N L A representation for the adaptive generation of simple sequential pro

grams In Grefenstette J editor Proceedings of an International Conference on Genetic

Algorithms and Their Applications pages Hillsdale NJ Lawrence Erlbaum As

so ciates

Crites R and Barto A Improving elevator p erformance using reinforcement learning

In Touretzky D Mozer M and Hasselmo M editors Advances in Neural Information

Processing Systems pages Cambridge MA MIT Press

Cyb enko G Approximation by sup erp ositions of a sigmoidal function Math Control

Signals Systems

DAmbrosio B POMDP learning using qualitative b elief spaces Technical rep ort

Oregon State University Corvallis

Davies S Ng A Y and Mo ore A W Applying online search techniques to

continuousstate reinforcement learning In Proceedings of the AAAI

Dayan P The convergence of TD for general lambda Machine Learning

Dayan P and Hinton G Feudal reinforcement learning In Lippman D S Mo o dy

J E and Touretzky D S editors Advances in Neural Information Processing Systems

pages San Mateo CA Morgan Kaufmann

Dayan P and Sejnowski T TD Convergence with probability Machine

Learning

Dayan P and Sejnowski T J Exploration b onuses and dual control Machine

Learning

DEp enoux F A probabilisitc pro duction and inventory problem Management

Science

Di Caro G and Dorigo M An adaptive multiagent routing algorithm inspired by

ants b ehavior In Proceedings of PART Fifth Annual Australasian Conference on

Parallel and RealTime Systems

BIBLIOGRAPHY

Dietterich T Hierarchical reinforcement learning with the MAXQ value function

decomp osition Technical rep ort Oregon State University

Digney B Emergent hierarchical control structures Learning reactivehierarchical

relationships in reinforcement environments In Maes P Mataric M Meyer JA

Pollack J and Wilson S W editors From Animals to Animats Proceedings of the

Fourth International Conference on Simulation of Adaptive Behavior Cambridge MA

pages MIT Press Bradford Bo oks

Dijkstra E A note on two problems in connexion with graphs Numerische Mathe

matik

Do dge Y Fedorov V V and Wynn H P editors Optimal Design and Analysis

of Experiments Proceedings of First International Conference on Optimal Design and

Analysis of Experiments Elsevier Publishers

Dorigo M and Colombetti M Robot Shaping An Experiment in Behavior Engi

neering MIT PressBradford Bo oks in press

Dorigo M and Gambardella L M Ant colony system A co op erative learning

approach to the traveling salesman problem Evolutionary Computation

Dorigo M Maniezzo V and Colorni A The ant system Optimization by a colony

of co op erating agents IEEE Transactions on Systems Man and CyberneticsPart B

Fedorov V V Theory of optimal experiments Academic Press

Friedman J Bentley J and Finkel R An algorithm for nding b est matches in

logarithmic exp ected time AMC Transactions on Mathematical Software

Fritzke B Sup ervised learning with growing cell structures In Cowan J Tesauro

G and Alsp ector J editors Advances in Neural Information Processing Systems

pages San Mateo CA Morgan Kaufmann

Gambardella L M Taillard E and Dorigo M Ant colonies for the QAP Tech

nical Rep ort IDSIA IDSIA Lugano Switzerland Submitted to Journal of the

Op erational Research So ciety

Gittins J C Multiarmed Bandit Al location Indices Wiley Chichester NJ

Givan R Leach S and Dean T Bounded parameter Markov decision pro cesses

Technical rep ort Retrievable from httpwwwcsbrownedupeopletldhomehtml

Glover F and Laguna M Tabu Search Kluwer Academic Publishers

Gordon G a Stable function approximation in dynamic programming Technical Re

p ort CMUCS School of Computer Science Carnegie Mellon University Pitts

burgh

Gordon G b Stable function approximation in dynamic programming In Prieditis

A and Russell S editors Machine Learning Proceedings of the Twelfth International

Conference pages Morgan Kaufmann Publishers San Francisco CA

BIBLIOGRAPHY

Heger M Consideration of risk in reinforcement learning In Machine Learning

Proceedings of the th International Conference pages Morgan Kaufmann

Publishers San Francisco CA

Hihi S E and Bengio Y Hierarchical recurrent neural networks for longterm

dep endencies In Touretzky D S Mozer M C and Hasselmo M E editors Advances

in Neural Information Processing Systems pages MIT Press Cambridge MA

Hinton G and Sejnowski T Optimal p erceptual inference In Proceedings of the

IEEE Conference on Computer Vision and Pattern Recognition pages

New york IEEE

Ho chreiter S and Schmidhuber J H Long shortterm memory Neural Computation

Holland J H Adaptation in Natural and Articial Systems University of Michigan

Press Ann Arb or

Hopeld J J Neural networks and physical systems with emergent collective com

putational abilities Proceedings of the National Academy of Sciences

Humphrys M Action selection metho ds using reinforcement learning In Maes

P Mataric M Meyer JA Pollack J and Wilson S W editors From Animals to

Animats Proceedings of the Fourth International Conference on Simulation of Adaptive

Behavior Cambridge MA pages MIT Press Bradford Bo oks

Jaakkola T Jordan M I and Singh S P On the convergence of sto chastic iterative

dynamic programming algorithms Neural Computation

Jaakkola T Singh S P and Jordan M I Reinforcement learning algorithm for

partially observable Markov decision problems In Tesauro G Touretzky D S and

Leen T K editors Advances in Neural Information Processing Systems pages

MIT Press Cambridge MA

Jacobs R A Jordan M I Nowlan S J and Hinton G E Adaptive mixtures of

lo cal exp erts Neural Computation

Jollife I T Principal Component Analysis New York Springer Verlag

Jordan M I and Jacobs R A Hierarchies of adaptive exp erts In Mo o dy J E

Hanson S J and Lippmann R P editors Advances in Neural Information Processing

Systems pages Morgan Kaumann

Jordan M I and Rumelhart D E Sup ervised learning with a distal teacher Techni

cal Rep ort Occasional Paper Center for Cognitive Science Massachusetts Institute

of Technology

Judd J Neural Network Design and the Complexity of Learning The MIT press

Cambridge

Kaelbling L P Learning in Embedded Systems MIT Press

BIBLIOGRAPHY

Kaelbling L P Littman M L and Cassandra A R Planning and acting in

partially observable sto chastic domains Unpublished rep ort

Kaelbling L P Littman M L and Mo ore A W Reinforcement learning A survey

Journal of Articial Intel ligence Research

Kearns M and Singh S P Nearoptimal p erformance for reinforcement learning in

p olynomial time Retrievable from httpwwwresearchattcommkearns

Kitano H Asada M Kuniyoshi Y No da I and Osawa E Rob o cup The rob ot

world cup initiative In Proceedings of the First International Conference on Autonomous

Agents Agents The ACM Press

Ko enig S and Simmons R G Complexity analysis of realtime exploration learning

applied to nding shortest paths in deterministic domains Technical Rep ort CMUCS

School of Computer Science Carnegie Mellon University

Ko enig S and Simmons R G The eect of representation and knowledge on goal

directed exploration with reinforcement learning algorithms Machine Learning

Kohonen T SelfOrganization and Associative Memory Springer second edition

Koza J R Genetic evolution and coevolution of computer programs In Langton C

Taylor C Farmer J D and Rasmussen S editors Articial Life II pages

Addison Wesley Publishing Company

Krose B J A and van Dam J W M Adaptive state space quantisation Adding and

removing neurons In Aleksander I and Taylor J editors Articial Neural Networks

pages NorthHollandElsevier Science Publishers Amsterdam

Krose B J A and Van de Smagt P An introduction to neural networks Autonomous

Systems University of Amsterdam

Landelius T Reinforcement Learning and distributed Local Model Synthesis PhD

thesis Linkoping Universtity Sweden

Lauritzen S and Wermuth N Graphical mo dels for asso ciations b etween variables

some of which are qualitative and some quantitative Annals of Statistics

LeCun Y Boser B Denker J S Henderson D Howard R E Hubbard W and Jackel

L D Backpropagation applied to handwritten zip co de recognition Neural

Computation

Levin L A Universal sequential search problems Problems of Information Trans

mission

Li M and Vitanyi P M B An Introduction to Kolmogorov Complexity and its

Applications Springer

Lin CS and Chiang CT Learning convergence of CMAC technique IEEE

Transactions on Neural Networks

BIBLIOGRAPHY

Lin LJ Reinforcement Learning for Robots Using Neural Networks PhD thesis

Carnegie Mellon University Pittsburgh

Lin T Horne B and Giles C How embedded memory in recurrent neural network

architectures helps learning longterm temp oral dep endencies Technical Rep ort CSTR

and UMIACSTR University of Maryland College Park MD

Lindgren K and Nordahl M G Co op eration and community structure in articial

ecosystems Articial Life

Littman M L a Markov games as a framework for multiagent reinforcement learning

In Prieditis A and Russell S editors Machine Learning Proceedings of the Eleventh

International Conference pages Morgan Kaufmann Publishers San Francisco

Littman M L b Memoryless p olicies Theoretical limitations and practical results

In Cli D Husbands P Meyer J A and Wilson S W editors Proceedings of the

International Conference on Simulation of Adaptive Behavior From Animals to Animats

pages MIT PressBradford Bo oks

Littman M L Algorithms for Sequential Decision Making PhD thesis Brown

University

Littman M L Cassandra A R and Kaelbling L P a Learning p olicies for partially

observable environments Scaling up In Prieditis A and Russell S editors Machine

Learning Proceedings of the Twelfth International Conference pages Morgan

Kaufmann Publishers San Francisco CA

Littman M L Dean T L and Kaelbling L P b On the complexity of solving

Markov decision problems In Proceedings of the Eleventh Annual Conference on Uncer

tainty in Articial Intel ligence UAI

Lovejoy W S A survey of algorithms metho ds for partially observable Markov

decision pro cesses Annals of Operations Research

Luke S Hohn C Farris J Jackson G and Hendler J Coevolving so ccer softb ot

team co ordination with genetic programming In Proceedings of the First International

Workshop on RoboCup at the International Joint Conference on Articial Intel ligence

IJCAI

Mahadevan S Sensitive discount optimality Unifying discounted and average re

ward reinforcement learning In Saitta L editor Machine Learning Proceedings of the

Thirteenth International Conference pages Morgan Kaufmann Publishers San

Francisco CA

Martin M Reinforcement Learning for Embedded Agents facing Complex tasks PhD

thesis Universitat Politecnica de Catalunya Barcelona

Martinetz T and Schulten K A neuralgas network learns top ologies In Kohonen

T Makisara K Simula O and Kangas J editors Articial Neural Networks pages

Elsevier Science Publishers BV NorthHolland

BIBLIOGRAPHY

Mataric M J Interaction and Intel ligent Behavior PhD thesis Massacusetts insti

tute of Technology

Matsubara H No da I and Hiraki K Learning of co op erative actions in multiagent

systems a case study of pass play in so ccer In Sen S editor Working Notes for the

AAAI Spring Symposium on Adaptation Coevolution and Learning in Multiagent

Systems pages

McCallum R A Overcoming incomplete p erception with utile distinction memory

In Machine Learning Proceedings of the Tenth International Conference pages

Morgan Kaufmann Amherst MA

McCallum R A Learning to use selective attention and shortterm memory in se

quential tasks In Maes P Mataric M Meyer JA Pollack J and Wilson S W

editors From Animals to Animats Proceedings of the Fourth International Confer

ence on Simulation of Adaptive Behavior Cambridge MA pages MIT Press

Bradford Bo oks

McDonald M A F and Hingston P Approximate discounted dynamic programming

is unreliable Technical Rep ort Department of Computer Science The University

of Western Australia Crawley WA

Mo ore A W Ecient Memorybased Learning for Robot Control PhD thesis

University of Cambridge

Mo ore A W Personal communication

Mo ore A W and Atkeson C G Prioritized sweeping Reinforcement learning with

less data and less time Machine Learning

Mo ore A W and Atkeson C G The partigame algorithm for variable resolution

reinforcement learning in multidimensional statespaces Machine Learning

Mo ore A W Atkeson C G and Schaal S A Lo cally weighted learning for control

Articial Intel ligence Review

Munos R A convergent reinforcement learning scheme in the continuous case the

nite element reinforcement learning In Saitta L editor Machine Learning Pro

ceedings of the Thirteenth International Conference pages Morgan Kaufmann

Publishers San Francisco CA

Myerson R Game Theory Harvard University Press

Nguyen and Widrow B The truck backerupper An example of self learning in

neural networks In IEEEINNS International Joint Conference on Neural Networks

Washington DC volume pages

Nilsson N J ProblemSolving Methods in Articial Intel ligence McGrawHill

Nowlan S Soft Competitive Adaption Neural Network Learning Algorithms based

on Fitting Statistical Mixtures PhD thesis Carnegie Mellon University Pittsburgh

BIBLIOGRAPHY

Oja E and Karhunen J Signal separation by nonlinear hebbian learning In

Palaniswami M Attikiouzel Y Marks I I R Fogel D and Fukuda T editors

Computational Intel ligence a Dynamic System Perspective pages IEEE Press

New York

Okabe A Bo ots B and Sugihara K Spatial Tesselations Concepts and applica

tions of Voronoi diagrams Wiley and Sons New York

Omohundro S M Foundations of geometric learning Technical Rep ort UIUCDCS

R University of Illinois Department of Computer Science

Omohundro S M Five balltree construction algorithms Technical Rep ort TR

International Computer Science Institute Berkeley CA

Omohundro S M Bumptrees for ecient function constraint and classication

learning In Lippman D S Mo o dy J E and Touretzky D S editors Advances

in Neural Information Processing Systems pages San Mateo CA Morgan

Kaufmann

Parr R and Russell S Approximating optimal p olicies for partially observable

sto chastic domains In Proceedings of the International Joint Conference on Articial

Intel ligence IJCAI pages Morgan Kaufmann

Parr R and Russell S Reinforcement learning with hierarchies of machines In

Advances in Neural Information Processing Systems

Peek N B Predictive probabilistic mo dels for treatment planning in paediatric cardi

ology In Proceedings of CESA IMACS Multiconference Computational Engineering

in Systems Applications Symposium on Signal Processing and Cybernetics

Peng J and Williams R Incremental multistep Qlearning Machine Learning

Peng J and Williams R J Ecient learning and planning with the DYNA frame

work Adaptive Behavior

Pineda F Meaneld theory for batched TD Neural Computation

Pollack J and Blair A Why did TDGammon work In Touretzky D Mozer M

and Hasselmo M editors Advances in Neural Information Processing Systems pages

Cambridge MA MIT Press

Precup D and Sutton R Theoretical results on reinforcement learning with tem

p orally abstract options In Proceedings of the Tenth European Conference on Machine

Learning ECML

Preparate F P and Shamos M I Computational Geometry an Introduction

Springer Verlag New York

Prescott T Exploration in Reinforcement and Modelbased Learning PhD thesis

University of Sheeld

BIBLIOGRAPHY

Press W Teukolsky S Vettering W and Flannery B Numerical recipes in C

Cambridge University Press

Rao C and Mitra S Generalized Inverse of Matrices and Its Applications Wiley

New York

Rechenberg I Evolutionsstrategie Optimierung technischer Systeme nach Prinzipien

der biologischen Evolution Dissertation Published by FrommanHolzbo og

Rechenberg I Evolution strategy Natures way of optimization In Bergmann

editor Methods and Applications Possibilities and Limitations pages Lecture

notes in Engineering

Resnick S Adventures in stochastic processes Birkhaeuser Verlag

Ring M B Continual Learning in Reinforcement Environments PhD thesis Uni

versity of Texas Austin Texas

Ron D Singer Y and Tishby N Learning probabilistic automata with variable

memory length In Aleksander I and Taylor J editors Proceedings Computational

Learning Theory ACM Press

Roth A and Erev I Learning in extensiveform games Exp erimental data and

simple dynamic mo dels in the intermediate term Games and Economic Behavior

Sp ecial issue Nob el Symp osium

Rumelhart D E Hinton G E and Williams R J Learning internal representations

by error propagation In Parallel Distributed Processing volume pages MIT

Press

Rummery G and Niranjan M Online Qlearning using connectionist sytems Tech

nical Rep ort CUEDFINFENGTR Cambridge University UK

Sahota M Realtime intelligent b ehaviour in dynamic environments So ccerplaying

rob ots Masters thesis University of British Columbia

Salustowicz R P and Schmidhuber J H Probabilistic incremental program evolu

tion Evolutionary Computation

Salustowicz R P Wiering M A and Schmidhuber J H a Evolving so ccer strate

gies In Proceedings of the Fourth International Conference on Neural Information Pro

cessing ICONIP pages SpringerVerlag Singap ore

Salustowicz R P Wiering M A and Schmidhuber J H b On learning so ccer strate

gies In Gerstner W Germond A Hasler M and Nicoud JD editors Proceedings

of the Seventh International Conference on Articial Neural Networks ICANN vol

ume of Lecture Notes in Computer Science pages SpringerVerlag Berlin

Heidelb erg

Salustowicz R P Wiering M A and Schmidhuber J H Learning team strategies

So ccer case studies Machine Learning

BIBLIOGRAPHY

Samuel A L Some studies in machine learning using the game of checkers IBM

Journal on Research and Development

Sandholm T W and Crites R H On multiagent Qlearning in a semicomp etitive

domain In Weiss G and Sen S editors IJCAI Workshop Adaption and Learning

in MultiAgent Systems pages SpringerVerlag

Santamaria J C Sutton R S and Ram A Exp eriments with reinforcement

learning in problems with continuous state and action spaces Technical Rep ort COINS

Georgia Institute of Technology Atlanta

Schmidhuber J H a Curious mo delbuilding control systems In Proceedings of the

International Joint Conference on Neural Networks Singapore volume pages

IEEE

Schmidhuber J H b Learning to generate subgoals for action sequences In Kohonen

T Makisara K Simula O and Kangas J editors Articial Neural Networks pages

Elsevier Science Publishers BV NorthHolland

Schmidhuber J H c A p ossibility for implementing curiosity and b oredom in mo del

building neural controllers In Meyer J A and Wilson S W editors Proceedings

of the International Conference on Simulation of Adaptive Behavior From Animals to

Animats pages MIT PressBradford Bo oks

Schmidhuber J H d Reinforcement learning in Markovian and nonMarkovian en

vironments In Lippman D S Mo o dy J E and Touretzky D S editors Advances

in Neural Information Processing Systems pages San Mateo CA Morgan

Kaufmann

Schmidhuber J H Learning complex extended sequences using the principle of

history compression Neural Computation

Schmidhuber J H A general metho d for incremental selfimprovement and multi

agent learning in unrestricted environments In Yao X editor Evolutionary Computa

tion Theory and Applications Scientic Publ Co Singap ore

Schmidhuber J H Whats interesting Technical Rep ort IDSIA IDSIA

Schmidhuber J H Zhao J and Schraudolph N N a Reinforcement learning with

selfmo difying p olicies In Thrun S and Pratt L editors Learning to learn Kluwer

Schmidhuber J H Zhao J and Wiering M A Simple principles of metalearning

Technical Rep ort IDSIA IDSIA

Schmidhuber J H Zhao J and Wiering M A b Shifting inductive bias with

successstory algorithm adaptive Levin search and incremental selfimprovement Ma

chine Learning

Schneider J G Exploiting mo del uncertainty estimates for safe dynamic control

learning In Mozer M C Jordan M I and Petsche T editors Advances in Neural

Information Processing Systems pages MIT PressBradford Bo oks Cam bridge

BIBLIOGRAPHY

Schraudolph N N Dayan P and Sejnowski T J Temporal dierence learning of

p osition evaluation in the game of go In Cowan J D Tesauro G and Alsp ector J

editors Advances in Neural Information Processing Systems volume pages

Morgan Kaufmann San Francisco

Schwartz A A reinforcement learning metho d for maximizing undiscounted rewards

In Machine Learning Proceedings of the Tenth International Conference pages

Morgan Kaufmann Amherst MA

Singh S P The ecient learning of multiple task sequences In Mo o dy J Hanson

S and Lippman R editors Advances in Neural Information Processing Systems

pages San Mateo CA Morgan Kaufmann

Singh S P and Sutton R S Reinforcement learning with replacing elibibility traces

Machine Learning

Singh S P and Yee R C An upp er b ound on the loss from approximate optimal

value functions Machine Learning

Sondik E J The Optimal Control of Partial ly Observable Markov Decision Processes

PhD thesis Standford California

Steels L Constructing and sharing p erceptual distinctions In van Someren M and

Widmer G editors Machine Learning Proceedings of the ninth European Conference

pages SpringerVerlag Berlin Heidelb erg

Stone P and Veloso M Beating a defender in rob otic so ccer Memorybased learn

ing of a continuous function In Tesauro G Touretzky D S and Leen T K editors

Advances in Neural Information Processing Systems pages MIT Press Cam

bridge MA

Stone P and Veloso M Teampartitioned opaquetransition reinforcement learning

In Proceedings of the Conference on automated learning and discovery CONALD

Robot Exploration and Learning Carnegie Mellon University Pittsburgh

Storck J Ho chreiter S and Schmidhuber J H Reinforcement driven informa

tion acquisition in nondeterministic environments In Proceedings of the International

Conference on Articial Neural Networks volume pages EC Cie Paris

Sutton R S Temporal Credit Assignment in Reinforcement Learning PhD thesis

University of Massachusetts Dept of Comp and Inf Sci

Sutton R S Learning to predict by the metho ds of temp oral dierences Machine

Learning

Sutton R S Integrated architectures for learning planning and reacting based on

dynamic programming In Machine Learning Proceedings of the Seventh International

Workshop

Sutton R S TD mo dels Mo deling the world at a mixture of time scales In Prieditis

A and Russell S editors Machine Learning Proceedings of the Twelfth International

Conference pages Morgan Kaufmann Publishers San Francisco CA

BIBLIOGRAPHY

Sutton R S Generalization in reinforcement learning Successful examples using

sparse coarse co ding In Touretzky D S Mozer M C and Hasselmo M E editors

Advances in Neural Information Processing Systems pages MIT Press

Cambridge MA

Sutton R S Precup D and Singh S P Between MDPs and semiMDPs Learn

ing planning learning and sequential decision making Technical Rep ort COINS

University of Massachusetts Amherst

Teller A The evolution of mental mo dels In Kinnear Jr K E editor Advances

in Genetic Programming pages MIT Press

Tesauro G Practical issues in temp oral dierence learning In Lippman D S

Mo o dy J E and Touretzky D S editors Advances in Neural Information Processing

Systems pages San Mateo CA Morgan Kaufmann

Tesauro G Temporal dierence learning and TDGammon Communications of the

ACM

Tham C Reinforcement learning of multiple tasks using a hierarchical CMAC archi

tecture Robotics and Autonomous Systems

Thrun S Ecient exploration in reinforcement learning Technical Rep ort CMU

CS CarnegieMellon University

Thrun S Learning to play the game of chess In Tesauro G Touretzky D and Leen

T editors Advances in Neural Information Processing Systems pages San

Fransisco CA Morgan Kaufmann

Thrun S Learning metrictop ological maps for indo or mobile rob ot navigation

Articial Intel ligence Journal

Thrun S and Moller K Active exploration in dynamic environments In Lippman

D S Mo o dy J E and Touretzky D S editors Advances in Neural Information

Processing Systems pages San Mateo CA Morgan Kaufmann

Thrun S and Schwartz A Finding structure in reinforcement learning In Tesauro

G Touretzky D S and Leen T K editors Advances in Neural Information Processing

Systems pages MIT Press Cambridge MA

Trovato K A Planning in Discrete Conguration Spaces of Autonomous Systems

PhD thesis University of Amsterdam

Tsitsiklis J N Asynchronous sto chastic approximation and Qlearning Machine

Learning

Tsitsiklis J N and Van Roy B An analysis of temp oraldierence learning with

function approximation Technical Rep ort LIDSP CambrdgeMA MIT Lab ora

tory for Information and Decision Systems

Van Dam J W M Environmental Modelling for Mobile Robots Neural Learning for

Sensor Fusion PhD thesis University of Amsterdam The Netherlands

BIBLIOGRAPHY

Van de Smagt P Visual robot arm guidance using neural networks PhD thesis

University of Amsterdam The Netherlands

Van der Wal J Stochastic Dynamic Programming Number in Mathematical

Centre tracts Mathematisch Centrum Amsterdam

Van Emde Boas P Kaas R and Zijlstra E Design and implementation of an

ecient priority queue Mathematical Systems Theory

Vennix J A M Systeemdynamica metho de for strategieontwikkeling Technical

rep ort Faculteit der Beleidswetenschappen Katholieke Universiteit Nijmegen

Watkins C J C H Learning from Delayed Rewards PhD thesis Kings College

Cambridge England

Watkins C J C H and Dayan P Qlearning Machine Learning

Werbos P J Beyond Regression New Tools for Prediction and Analysis in the

Behavioral Sciences PhD thesis Harvard University

Whitehead S Reinforcement Learning for the adaptive control of perception and

action PhD thesis University of Ro chester

Widrow B and Ho M E Adaptive switching circuits IRE WESCON

Convention Record New York IRE Reprinted in Anderson and Rosenfeld

Wiering M A TD Learning of Game Evaluation Functions with Hierarchical Neural

Architectures Masters thesis Department of Computer Systems University of Amster

dam

Wiering M A and Dorigo M Learning to control forest res In Haasis HD and

Ranze K C editors Proceedings of the th international Symposium on Computer

Science for Environmental Protection volume of Umweltinformatik Aktuel l pages

Marburg Metrop olis Verlag

Wiering M A Salustowicz R P and Schmidhuber J H CMAC mo dels learn to

play so ccer In Niklasson L Bo den M and Ziemke T editors Proceedings of the th

International Conference on Articial Neural Networks ICANN volume pages

SpringerVerlag London

Wiering M A and Schmidhuber J H Solving POMDPs with Levin search and

EIRA In Saitta L editor Machine Learning Proceedings of the Thirteenth Interna

tional Conference pages Morgan Kaufmann Publishers San Francisco CA

Wiering M A and Schmidhuber J H HQlearning Adaptive Behavior

Wiering M A and Schmidhuber J H a Ecient mo delbased exploration In Meyer

J A and Wilson S W editors Proceedings of the Sixth International Conference on

Simulation of Adaptive Behavior From Animals to Animats pages MIT

PressBradford Bo oks

BIBLIOGRAPHY

Wiering M A and Schmidhuber J H b Fast online Q Machine Learning Journal

Williams R J and Baird L C Tight p erformance b ounds on greedy p olicies based on

imp erfect value function Technical Rep ort NUCCS College of Computer Science

Northeastern University Boston MA

Wilson S ZCS A zeroth level classier system Evolutionary Computation

Wilson S Classier tness based on accuracy Evolutionary Computation

Zhang N L and Liu W Planning in sto chastic domains Problem characteristics

and approximation Technical Rep ort HKUSTCS Hong Kong University of Science

and Technology

Zhao J and Schmidhuber J H Incremental selfimprovement for lifetime multi

agent reinforcement learning In Maes P Mataric M Meyer JA Pollack J and

Wilson S W editors From Animals to Animats Proceedings of the Fourth Interna

tional Conference on Simulation of Adaptive Behavior Cambridge MA pages

MIT Press Bradford Bo oks

Index

BRO ChapmanKolmogorov equation

DP classier system

DT CMAC

EC mo del

FA complete state information

GO consistent representation

GP contraction

HMM control

IE credit assignment problem

IS dataretrieval problem

LFA decision tree

LP LS delayed reward

MBIE discount factor

MC dual control

MDP Dyna

MLM dynamic programming

MP and function approximation

NO real time

PAC eligibility traces

PI accumulating

PIPE another view

POMDP connecting

PPT multiple

PS replacing

PW environment

RL changing

RP nonstationary

RPP evolutionary computation

RTDP exploitation

SAP exploration

SC Boltzmann

SSA directed

TD initialize high

WM MaxBoltzmann

WTA Maxrandom

action evaluation function reward function

active learning error based

agent frequency based

b elief state recency based

INDEX

undirected rst visit

explorationexploitation dilemma multiagent RL

xed p oint nearest neighbor search

function approximator neural gas

and Q mo del

and world mo delling o ccupancy probabilities

dynamic top ology optimal exp eriment design

lo cal optimal linear asso ciative memory

static top ology partially observable MDP

genetic programming sp ecication

global mo del p erceptual aliasing

hidden Markov mo del p olicy

hidden state optimal

problem reactive

sp ecication shared

HQlearning p olicy iteration

incomplete state information p olicy tree

indep endent lter mo del prediction problems

internal state prioritized sweeping

interval estimation probabilistic incremental program evolution

mo delbased

landmark state probabilistic program tree

learn probably approximately correct framework

Levin search

linear mo dels program evolution

linear network Qfunction

linear programming Qlearning

lo cal mo del mo delbased

Kd tree Qlearning

map learning Fast Q

Markov chain oine

higher order Semionline

prop erties Team

Markov decision pro cess with function approximators

nite horizon reactive p olicy problem

innite horizon reactive strategy

Markov order recurrent neural network

Markov pro cess reinforcement learning

Markov prop erty direct

maximum likelihoo d mo del indirect

MAXQ mo delbased

memory bits multiagent

mo delbased oine

interval estimation and function approximation

Qlearning online

Monte Carlo sampling vs evolutionary computation

every visit reward function

INDEX

for exploration

rob otic so ccer

simulator

sampling and approximation

Sarsa

sequential decision making

soft cell comp etition

subgoal combination

successstory algorithm

TDlearning

team Qlearning

temp oral dierences

transition function

value function

optimal

shared

value iteration

Voronoi diagram

winner take all

world state

Summary

This thesis describ es reinforcement learning RL metho ds which can solve sequential decision

making problems by learning from trial and error Sequential decision making problems are

problems in which an articial agent interacts with a sp ecic environment through its sensors

to get inputs and eectors to make actions To measure the go o dness of some agents

b ehavior a reward function is used which determines how much an agent is rewarded or

p enalized for p erforming particular actions in particular environmental states The goal is to

nd an action selection p olicy for the agent which maximizes the cumulative reward collected

in the future

In RL an agents p olicy maps sensorbased inputs to actions To evaluate a p olicy a value

function is learned which returns for each p ossible state the future cumulative reward collected

by following the current p olicy Given a value function we can simply select the action with

the largest value In order to learn a value function for a sp ecic problem reinforcement

learning metho ds simulate a p olicy and use the resulting agents exp eriences consisting of

stateactionrewardnextstate quadruples

There are dierent RL problems and dierent RL metho ds for solving them We describ e

dierent categories of problems and introduce new metho ds for solving them

Markov Decision Problems

If the agents inputs allow for determining the state of the environment with certainty and

the environment is stationary we can mo del the decision making problem as a Markov De

cision Problem MDP In a MDP the probabilities and exp ected rewards of transitions to

p ossible next states only dep end on the current state and action and not on any previous

events Given an exact mo del of a MDP we can use Dynamic Programming DP metho ds

for computing optimal p olicies Such metho ds iterate over each state and constantly recom

pute the value function by lo oking ahead one step at each time There are two principal DP

metho ds value iteration and p olicy iteration Value iteration reup dates the p olicy after each

improvement of the value function by immediately mapping a state to the action with the

largest value whereas p olicy iteration rst completely evaluates a p olicy by using sucient

iterations after which a p olicy up date step is p erformed Although in theory p olicy iteration

always converges in nitely many iterations whereas value iteration do es not our exp eriments

on a set of mazetasks indicate that value iteration converges much faster to a solution The

main reason for this is that p olicy iteration p ostp oning p olicy up dates wastes a lot of time

evaluating p olicies which generate cycles in the state space

Reinforcement Learning

Dynamic programming metho ds can only b e used if we have an a priori mo del of the MDP

Therefore their practical utility is severely limited In case we do not p ossess such a mo del

we can use reinforcement learning metho ds which learn p olicies by interacting with a real

or simulated environment For this we use trials during which agents generate tra jectories

through the state space and let agents learn from the resulting exp eriences

RL metho ds such as Qlearning learn a value function by minimizing temp oral dierences

b etween the current states value and the reward which immediately follows plus the next

states value Qlearning up dates values of stateaction pairs along a solutionpath by lo oking

ahead only one single step in a trial and therefore it takes many trials b efore a goalreward

will b e backpropagated to the start TD metho ds are parameterized by a variable en

use the entire future for up dating the value of a state although the degree of inuence of

state values in the distant future is less than values of immediately successive states This

makes it p ossible to learn from eects of actions which show up after a long time Q

learning combines Qlearning and TD and uses single exp eriences to up date values of

multiple stateaction pairs that have o ccurred in the past Although Q metho ds can

sometimes learn p olicies from much less exp eriences than Qlearning naive implementations

of the online Q metho d suer from a large computational overhead We have introduced

a novel implementation of the algorithm which makes it p ossible to eciently p erform the

up dates

If multiple agents are used simultaneously in an environment they can share the p olicy

to collect more exp eriences at each time step which may sp eedup learning a p olicy Using

p olicysharing agents select actions according to the same p olicy but their actions may dif

fer since they receive dierent situation sp ecic inputs In such cases we can use each agents

generated tra jectory individually to up date the shared value function However sometimes

there exists an interaction p oint IP b etween agents tra jectories such as a state visited by

b oth of them In such cases we can connect the tra jectories and up date values of states

visited by one agent b efore the IP using values of states visited by the other agent after the

IP In this way we collect much more exp eriences to learn from which may increase learning

sp eed We describ e an implementation of this Team Q algorithm and evaluate it in a non

co op erative mazetask using dierent numbers of agents The results do not show that the

metho d improves learning sp eed One reason for this is that our implementation allows for

connecting a statetra jectory of one agent with another statetra jectory which may b e gen

erated a long time ago by a p ossibly much worse p olicy and learning from such a tra jectory

may degenerate the p olicy Therefore we need to study the tradeo b etween the quantity

and quality of learning examples

Mo delbased Reinforcement Learning

Qlearning and Q metho ds are direct memoryless RL metho ds which directly estimate the

value function from generated statetra jectories Indirect or mo delbased RL metho ds rst

learn a mo del of the MDP and then apply DP algorithms to compute the value function

When used in an online manner DP algorithms are very slow since they recompute the value

function after each exp erience by iterating over all states To sp eed up dynamic program

ming algorithms some algorithms manage which up datesteps should b e p erformed after each

exp erience so that only the most useful up dates are made One of the most ecient manage

ment metho ds is Prioritized Sweeping PS which assigns priorities to up dating the values

of dierent states according to a heuristic estimate of the size of the values up dates The

original PS algorithm by Mo ore and Atkeson calculates statepriorities based up on the

largest single up date of one of the successor states We describ e an alternative PS metho d

which uses the exact up date sizes of state values to compute priorities This may overcome

PS problems in case there are many tiny state value up dates resulting in a large up date

which remains undetected by PS

We compare mo delbased RL metho ds to direct RL metho ds on a set of mazetasks and

observe that mo delbased RL results in much faster learning p erformances and b etter nal

p olicies Finally we observe that our PS algorithm achieves b etter results than the original

PS algorithm

Exploration

If we test an agents p olicy by always selecting the action with the largest value we may

end up with a sub optimal p olicy which has almost never visited a large part of the state

space Therefore it is imp ortant that the agent also selects exploration actions and tries

to increase its knowledge of the environment Usually however exploration actions cause

some loss of immediate reward intake Therefore if an agent wants to maximize its cumu

lative reward during its limited lifetime it faces the problem of trying to sp end as little

time as p ossible on exploration while still b eing able to nd a highly rewarding p olicy the

explorationexploitation dilemma However if we are interested in learning a p olicy achiev

ing some minimal p erformance level as so on as p ossible we do not care ab out intermediate

rewards and thus only exploration is imp ortant

Exploration metho ds can b e split into undirected exploration metho ds which use pseudo

random generators to try their luck on generating novel interesting exp eriences and directed

exploration metho ds which use additional information such as the knowledge of what the

agent has seen or learned in order to send the agent to interesting unknown regions

Directed exploration metho ds can b e constructed by constructing an exploration reward

rule which makes it p ossible to learn an exploration value function for estimating global infor

mation gain for selecting particular actionsequences Although we can use any RL metho d

for learning an exploration value function we prop ose to use mo delbased RL for this Finally

in order to fo cus only on stateaction pairs which can b elong to the optimal p olicy we intro

duce MBIE an algorithm which uses the standard deviance for computing optimistic value

functions The results on exp eriments with mazetasks featuring multiple goals show that

directed exploration using mo delbased RL can signicantly outp erform indirect exploration

Furthermore the results show that MBIE can b e used to almost always obtain nearoptimal

p erformance levels and that MBIE eciently deals with the explorationexploitation dilemma

Partially Observable Problems

In the real world the agents sensory inputs may not always convey all information needed

to infer the current environmental state Therefore the agent will sometimes b e uncertain

ab out the real state and that makes the decision problem much harder Such problems are

usually called Partially Observable Markov Decision Problems POMDPs and even solving

deterministic POMDPs in NPhard To solve such problems the agent needs to use previous

events or some kind of shortterm memory to disambiguate inputs Exact metho ds make

use of a b elief state a vector representing the probabilities that the real environmental state

is each of the p ossible states Then they use DP algorithms to compute optimal solutions

in the b elief state space Exact metho ds are computationally infeasible if there are many

states since they are to o slow Therefore we are interested in heuristic algorithms which

can b e used for larger problems but do not necessarily nd optimal solutions We describ e

a novel algorithm HQlearning which is able to learn go o d p olicies for large deterministic

POMDPs HQ automatically decomp oses POMDPs into sequences of simpler subtasks that

can b e solved by memoryless p olicies learnable by reactive subagents Decomp osing the task

into subtasks and learning p olicies for each subtask is done by two co op erating Qlearning

rules Exp eriments on among other a large mazetask consisting of states and only

dierent highly ambiguous inputs show that HQlearning can quickly nd go o d or even

optimal solutions for dicult problems

Function Approximation in RL

An imp ortant topic in RL is the application of function approximation to learn value functions

for continuous or very large state spaces We describ e three dierent function approximators

linear networks neural gas and CMACs and describ e how they can b e combined with direct

and mo delbased RL metho ds Then we describ e a simulated multiagent so ccer environment

which we use to test the learning capabilities and problems of the function approximators

and RL metho ds The goal is to learn go o d so ccer strategies against a xed programmed

opp onent The results show that the linear networks and neural gas architecture trained with

Q had problems to steadily improve their learned p olicies Sometimes the p erformance

broke down due to catastrophic learning interference CMACs was more stable although

CMACmodels the combination of CMACs and mo delbased RL was the only metho d which

found really go o d p erformances and was able to b eat some go o d opp onents

Conclusions

We have shown in this thesis that RL can b e used to quickly nd go o d solutions for dierent

kinds of problems The metho ds describ ed in this thesis do not need a mo del of the world

but learn a p olicy for an agent by interacting with the real environment Esp ecially mo del

based RL can b e very ecient since it is able to use all information Therefore the practical

utility of mo delbased reinforcement learning is very large and we exp ect the research eld

to grow a lot in imp ortance in the forthcoming years

Samenvatting

Dit pro efschrift b eschrijft reinforcement leermetho den RLmetho den welke sequentiele b e

sluitvormings problemen op kunnen oplossen do or het leren van prob eren en fouten maken

Sequentiele b esluitsvormings problemen zijn problemen waarin een kunstmatige agent inter

acteert met een b epaalde omgeving do or middel van haarzijn sensoren om input te verkrij

gen en eectoren om acties te verrichten Voor het meten van de go edheid van het gedrag

van een sp ecieke agent maken we gebruik van een b eloningsfunctie welke b epaalt in ho ev

erre een agent b elo ond of gestraft mo et worden voor het verrichten van b epaalde handelingen

in b epaalde situaties Het do el is om een actieselecteer handelspro cedure voor de agent te

vinden welke de totale som der b eloningen verkregen in de to ekomst maximaliseert

In een RLsetting worden inputs gepro jecteerd op acties do or de handelspro cedure van

de agent Voor het evalueren van een handelspro cedure leren we een waardefunctie welke

voor elke mogelijke to estand van de wereld de som der b eloningen teruggeeft welke in de

to ekomst wordt verkregen do or het volgen van de huidige handelspro cedure Als we eenmaal

een waardefunctie hebb en verkregen kunnen we gewoon de actie selecteren met de ho ogste

waarde Om een waardefunctie te leren voor een b epaald probleem simuleren reinforce

ment leermetho den een handelspro cedure en gebruiken ze de resulterende ervaringen van de

agent welke b estaan uit to estand actie b eloning volgendetoestand quadruples om de

to ekomstige b eloningssom te schatten b eginnende in elke mogelijke to estand

Er zijn verschillende RLproblemen en verschillende RLmetho den om ze op te lossen We

b eschrijven verschillende probleemcategorieen en introduceren nieuwe metho den om ze op te

lossen

Markov Besluits Problemen

Als de input van een agent to estaat om de to estand van de omgeving met zekerheid te

b epalen en de causale wetten die in de omgeving opgaan zijn onveranderlijk dan kunnen

we het b esluitvorming probleem mo delleren als een Markov Besluits Probleem MBP In

een MBP hangen de kansen en b eloningen van overgangen naar mogelijke opvolgende to es

tanden enkel af van de huidige to estand en gekozen actie en niet van vorige geb eurtenissen

Gegeven een exact mo del van een MBP kunnen we Dynamisch Programmeer DP metho den

gebruiken om optimale handelspro cedures te b erekenen Zulke metho den itereren over elke

to estand en verbeteren de waardefunctie voortdurend do or steeds een stap je verder vooruit

te kijken Er zijn twee voorname DP metho den waarde iteratie en handelspro cedure iteratie

Waarde iteratie verbetert de handelspro cedure na elke verbetering van de waardefunctie

do or onmiddellijk elke to estand te pro jecteren op de actie met de ho ogste waarde terwijl

b esluitspro cedure iteratie eerst de handelspro cedure volledig evalueert om zo een exact evalu

atie van de handelspro cedure te verkrijgen waarna een verbetering van de handelspro cedure

wordt gemaakt Ho ewel handelspro cedure iteratie in theorie altijd in eindig veel iteraties

convergeert en waarde iteratie niet wijzen onze exp erimenten erop dat waarde iteratie veel

sneller een oplossing kan b erekenen De ho ofdreden hiervoor is dat handelspro cedure iteratie

verbeteringen in de b esluitspro cedure uitstelt en dusdanig veel tijd verliest do or het volledig

evalueren van handelspro cedures welke wederkerende paden in de to estandruimte genereren

Reinforcement Leren

Dynamisch programmeer metho den kunnen enkel gebruikt worden als we een a priori mo del

van de MBP tot onze b eschikking hebb en Daarom is hun praktische bruikbaarheid erg

b ep erkt In het geval dat we niet over zon mo del b eschikken kunnen we reinforcement

leermetho den gebruiken welke handelspro cedures leren do or te interacteren met een werkelijke

of gesimuleerde omgeving RLmetho den gebruiken exp erimenten waarin de agent to estands

paden do or de to estandsruimte genereert waarvan geleerd kan worden In het optimale geval

leert de agent dusdanig optimale paden te genereren

RLmetho den zoals Qleren leren een waardefunctie do or tijdelijke verschillen tussen de

waarde van de huidige to estand en de b eloning welke onmiddellijk volgt plus de waarde van

de volgende to estand te minimaliseren Qleren past waarden van to estandactie paren welke

op een oplossingspad liggen aan do or in elk opvolgend exp eriment slechts een stap vooruit te

kijken en daarom zijn er veel exp erimenten no dig voordat een b eloning welke verkregen wordt

bij de do elto estand teruggepropageerd wordt naar het b egin TD metho den gebruiken de

hele to ekomst voor het aanpassen van de waarde van een to estand ho ewel waarden van to e

standen welke ver in de to ekomst liggen minder zwaar wegen dan waarden van onmiddellijk

opvolgende to estanden Dit maakt het mogelijk om van de gevolgen van acties te leren welke

pas na een lange tijd zichtbaar worden Q combineert Qleren met TD en gebruikt

ervaringen om waarden van meerdere to estandactie paren welke in het verleden zijn opge

treden aan te passen Ho ewel Q metho den soms handelspro cedures kunnen leren van veel

minder ervaringen dan Qleren lijden nave implementaties van online Q metho den onder

een grote computationele overhead Wij hebb en een nieuwe implementatie van het algoritme

gentroduceerd welke het mogelijk maakt om de aanpassingen ecient te verrichten

Als meerdere agenten gelijktijdig gebruikt worden in een omgeving dan kunnen ze de han

delspro cedure delen om zo meer ervaringen in elke tijdstap te verzamelen welke het leren van

een handelspro cedure kan versnellen Als we gebruik maken van dit een handelspro cedure

voor allen dan selecteren agenten acties volgens dezelfde handelspro cedure maar hun acties

kunnen verschillen omdat ze verschillende situatiesp ecieke inputs verkrijgen Voor zulke

gevallen kunnen we paden gegenereerd do or verschillende agenten individueel gebruiken om

de gedeelde waardefunctie aan te passen Soms echter b estaat er een interactiepunt IP

tussen paden van agenten zoals een to estand welke do or b eiden b ezo cht is In zulke gevallen

kunnen we paden verbinden en waarden van to estanden welke do or een agent voor het IP

zijn b ezo cht aanpassen do or gebruik te maken van waarden van to estanden welke do or de

andere agent b ezo cht zijn na het IP Op deze manier verzamelen we veel meer ervaringen om

van te leren hetgeen de leersnelheid wellicht groter maakt We b eschrijven een implementatie

van dit Team Q algoritme en evalueren het in een nietcooperatieve do olhoftaak waarin

we gebruik maken van verschillende aantallen agenten De resultaten konden niet aantonen

dat de metho de de leersnelheid verbetert Een reden hiervoor is dat onze implementatie

verbindingen to estaat tussen een to estandpad van een agent met een ander to estandpad

welke al een hele lange tijd geleden gegenereerd is do or een mogelijkerwijze veel slechtere

handelspro cedure waardoor aanpassingen in de waardefunctie onterecht kunnen zijn Dus is

er een tradeo tussen de kwantiteit en kwaliteit van leervoorb eelden

Mo delgebaseerd Reinforcement Leren

Qleren en Q metho den zijn directe RL metho den welke de waardefunctie direct schatten

aan de hand van gegenereerde to estandpaden Indirecte of mo delgebaseerde RL metho den

leren eerst een mo del van het MBP en gebruiken dan DP algoritmes om de waardefunctie

te b erekenen Wanneer DP algoritmes op een online manier gebruikt worden zijn ze erg

langzaam omdat ze de waardefunctie na elke ervaring herb erekenen en dat do en do or over

alle to estanden te itereren Om DP algoritmes te versnellen zijn er algoritmes die regelen

welke aanpassingsstapp en verricht mo eten worden na elke ervaring zo dat enkel de meest

bruikbare aanpassingen gemaakt worden Een van de meest eciente metho den is Prioritized

Sweeping PS welke prioriteiten aanwijst voor het aanpassen van waarden van verschillende

to estanden Deze prioriteiten worden b epaald aan de hand van een schatting van de gro otte

van de aanpassingen Het o orspronkelijke PS algoritme van Mo ore en Atkeson b erekent

to estandprioriteiten gebaseerd op de gro otste enkelvoudige aanpassing van een van de opvol

gende to estanden Wij b eschrijven een alternatieve PS metho de welke de exacte gro ottes van

aanpassingen gebruikt om prioriteiten te b ereken Dit kan het probleem van PS verhelpen

als er vele kleine aanpassingen verricht worden welke tot een grote aanpassing van de waarde

van een to estand leiden maar welke niet do or PS gedetecteerd wordt

We vergelijken mo delgebaseerde metho den met directe RL metho den op een verzameling

do olhoftaken en observeren dat mo delgebaseerde RL in veel sneller leergedrag resulteert en

veel b etere handelspro cedures oplevert Tenslotte observeren we dat ons PS algoritme b etere

resultaten b ehaalt dan het originele PS algoritme

Exploratie

Als we de handelspro cedure van een agent testen do or altijd de actie met de gro otste waarde

te selecteren dan kan dat uiteindelijk leiden tot een sub optimale handelspro cedure welke

een gro ot deel van de to estandruimte nog no oit of nauwelijks b ezo cht heeft Daarom is het

b elangrijk dat de agent o ok exploratie acties selecteert en haarzijn kennis van de omgeving

prob eert te verbeteren Gewoonlijk kosten exploratie acties een b epaalde som aan onmid

dellijke b eloningen Daarom als een agent haarzijn som der b eloningen welke in haarzijn

b ep erkte levensduur verkregen wordt wil maximaliseren staat het voor het probleem om zo

weinig mogelijk tijd aan exploratie te verspillen welke echter wel voldoende is om een go ed

b elonende handelspro cedure te vinden het exploratieexploitatie dilemma Indien we echter

meer genteresseerd zijn in het zo snel mogelijk leren van een handelspro cedure welke een

b epaald prestatienivo levert dan geven we niks om de b eloningen welke in de tussentijd

verkregen worden en dus is enkel exploratie b elangrijk

Exploratie metho den kunnen ingedeeld worden in nietgerichte exploratie metho den welke

pseudorandom generatoren gebruiken om zo te prob eren nieuwe interessante ervaringen te

genereren en gerichte exploratie metho den welke extra informatie gebruiken zoals de kennis

wat de agent al gezien of geleerd heeft om zo do ende de agent te sturen naar interessante

onbekende gebieden

Gerichte exploratie metho den kunnen geconstrueerd worden do or een exploratie b elonings

functie te construeren waarmee we een exploratie waardefunctie kunnen leren voor het schat

ten van globale informatie verdiensten voor het verrichten van b epaalde acties Ho ewel

we alle RLmetho den kunnen gebruiken voor het leren van een exploratie waardefunctie

stellen we voor om mo delgebaseerd RL hiervoor te gebruiken Tenslotte om enkel op to e

standactie paren te concentreren welke deel uit kunnen maken van de optimale handelspro

cedure introduceren we MBIE een algoritme welke gebruik maakt van de standaard devi

atie voor het b erekenen van optimistische waardefuncties De resultaten van exp erimenten

met do olhoftaken b estaande uit meerdere do elto estanden tonen dat gerichte exploratie met

mo delgebaseerd RL ongerichte exploratie duidelijk overtreft Voorts demonstreren de re

sultaten dat MBIE gebruikt kan worden om bijna altijd haast optimale prestatie nivos te

b ehalen en verder ecient omgaat met het exploratieexploitatie dilemma

Gedeeltelijk Waarneembare Problemen

In de wereld dragen de inputs van de sensoren van de agent niet altijd geno eg informatie

over om de exacte huidige to estand van de omgeving af te leiden Daarom zal de agent

soms onzeker zijn over de daadwerkelijke to estand en dat maakt het b esluitsprobleem veel

mo eilijker Zulke problemen worden gewoonlijk Gedeeltelijk Waarneembare Markov Besluits

Problemen GWMBPen geno emd en zelfs het oplossen van deterministische GWMBPen is

NPmo eilijk Voor het oplossen van zulke problemen mo et de agent vorige geb eurtenissen en

een b epaalde vorm van kortetermijn geheugen gebruiken om inputs van de ambiguteiten te

ontdoen Exacte metho den gebruiken een gelo ofs to estand een vector welke de kansen repre

senteert dat de echte to estand van de omgeving gelijk aan elk van de mogelijke to estanden is

Daarna gebruiken ze DP algoritmes om optimale oplossingen in de gelo ofs to estandruimte te

b erekenen Exacte metho den zijn computationeel onbruikbaar als er teveel to estanden zijn

omdat ze te langzaam zijn Daarom zijn we genteresseerd in heuristieke algoritmes welke

gebruikt kunnen worden om grotere problemen op te lossen maar welke niet zeker optimale

oplossingen vinden We b eschrijven een nieuw algoritme HQleren welke go ede handelspro

cedures kan leren voor grote deterministische GWMBPen HQ deelt GWMBPen op in een

op eenvolging van makkelijkere deelproblemen welke opgelost kunnen worden do or geheugen

loze handelspro cedures welke leerbaar zijn do or reactieve subagenten De decomp ositie van

de taak in subtaken en het leren van handelspro cedures voor elke subtaak wordt gedaan do or

twee samenwerkende Qleerregels Exp erimenten op onder andere een grote do olhoftaak

b estaande uit to estanden en slechts verschillende ambigue inputs tonen dat HQleren

snel go ede of zelfs optimale oplossingen kan vinden voor b epaalde mo eilijke problemen

Functie Approximatie in RL

Een b elangrijk thema in RL is de to epassing van functie approximatie voor het leren van

waardefuncties voor continue of zeer grote to estandruimtes We b eschrijven drie verschil

lende functie approximatoren lineaire netwerken neuraal gas en CMACs en b eschrijven

ho e ze gecombineerd kunnen worden met directe en mo delgebaseerde RLmetho den Dan

b eschrijven we een gesimuleerde voetbal omgeving welke we gebruiken om de leercapaciteiten

te testen van de verschillende functie approximatoren en RL metho den Het do el is om go ede

voetbal strategieen te leren tegen een voorgeprogrammeerde tegenstander De resultaten to

nen dat de lineaire netwerken en neurale gas architecturen getraind met Q problemen

hadden om voortdurend hun handelspro cedures te verbeteren Soms stortte de prestatie in

do or een catastrofale leerinterferentie CMACs was meer stabiel ho ewel CMACmodellen de

combinatie van CMACs en mo delgebaseerd RL de enige metho de was welke daadwerkelijk

go ede prestaties leverde en in staat was enkele go ede tegenstanders te klopp en

Conclusies

In dit pro efschrift hebb en we aangeto ond dat RL gebruikt kan worden om snel go ede oplossin

gen voor verschillende so orten problemen te vinden De b eschreven metho den hebb en geen

mo del van de wereld no dig maar leren een handelspro cedure do or te interacteren met de

echte omgeving Mo delgebaseerde is de meest eciente leermetho de omdat het de do or de

exp erimenten verkregen data geheel gebruikt We b evestigen dus dat het praktisch nut van

reinforcement leren erg gro ot is en verwachten dat het onderzo eksveld aanzienlijk aan b elang

zal to enemen in de komende jaren

Acknowledgments

This thesis would never have seen the light if my promotor Frans Gro en would not have gone

through so many eorts to put things in order I am very thankful for this and have greatly

appreciated watching and listening to his sharp hawkeye views ying over the scientic elds

My copromotor sup ervisor and b oss J urgen Schmidhuber has always given me the op

p ortunity to explore interesting ideas which autonomously created themselves and have b een

astonishing my mind and computer Without J urgens careful editing continuous stimulation

and pragmatic reasoning these ideas would never have crystallized themselves and nd their

way to our resp ected scientic audience Im therefore very grateful to J urgen Maybe dier

ent from many other PhD students Ive never felt like a slave Instead I am very fortunate

that I could participate in J urgens very interesting and enjoyable research group

I have had a great time in Lugano and in IDSIA I very much enjoyed the parties presented

by my go o d friend Rafal Salustowicz There are few friends willing to give away their last

fo o d esp ecially if you are very hungry and the next day all shops are closed I was fortunate

to know Rafal and his lovely wife Malgorza who are like that Furthermore my discussions

with Rafal during coee breaks were often so stimulating that my research has highly proted

from them since they created new challenging views which I then carefully transmitted to

computer land resulting in a p ositive research feedback lo op

I also want to thank Marcuso Homan without whom life in IDSIA would never have

b een so pleasant Marcuso is one of the few p eople who are always in a go o d mo o d He

even stays reasonable when the cappuccino machine blows up I will certainly miss Marcusos

presence Italian pasta and number one cappuccio

Nic Schraudolph has always b een a very interesting and nice guy to talk to He can

tell you everything you always wanted to know ab out bugs viruses internet worms neural

networks and so much more He reminds me of a story of a scientist who had a theory which

told that large brains imply greater intelligence Once this scientist received ve brains of

German professors after they had died and found out that their brains were of mo derate

size His conclusion was that the professors were not so intelligent after all I have to say

that this will not happ en to Nic since he sure has a large brain

Felix Gers and his girlfriend Mara also help ed creating a blissful time in Lugano I really

liked Felix curiosity way of talking and great laugh Hitting the road with Felix and Mara

have always lead to a lot of joy

Thanks to Marco Dorigo for his comments on my thesis and for helping me develop new

scientic interests I also want to thank the other Idsiani thousand times thanks for the Cure

to Monaldo grazie mille for all his kindness to Marco Z thanks to Gianluca and Cristina for

interesting discussions thanks to Giusepp e for putting rob ots in space thanks a lot to Jieyu

for his remarkable presence and sharing many research interests thanks to Eric for cursing

his computer and not mine thanks to Andreas for his interesting stories ab out kangaroos

playing cricket although I could not always follow them exactly since they were sp oken in

fast Italian thanks to Fred for the great p oker nights and thanks to Nicky Sandro and

Giovanni for everything Finally thanks to Luca for his humor and fruitful eorts to make a

great lab from IDSIA thanks to Carlo for his patience supp ort and eorts to keep things

going Last but not least thanks to Monica for her endless help and devotion to settle all

imp ortant issues outside the eld of research

I want to thank Ben Krose for his everlasting help starting many years back while I was

his student I am indebted to him for putting me on the road of science and happily lo ok back

to the pleasant times enjoyed in his company Im also grateful to Stephen and Nikos for our

friendly discussions in the university of Adam Thanks also to Maria who help ed me to see

life in a very p ositive way and the happy times we shared together She help ed me b ecoming

less articial and much more artistic Thanks to Flower for our everlasting friendship and

all go o d times we have shared together Many thanks go to my parents for their everlasting

supp ort and love

Finally I want to express my warm feelings to Rene Wiering for his great friendliness and

hospitality I have always enjoyed and proted from our mind breaking discussions pursuing

the goal to b etter understand ourselves and our life I really hop e these discussions will never

stop but continuously evolve and b ecome truth