Maze Navigation Using Neural Networks Evolved with Novelty Search and Differential Evolution

Maze Navigation using Neural Networks Evolved with Novelty Search and Differential Evolution Karl Mason∗ Jim Duggan Enda Howley National University of Ireland Galway National University of Ireland Galway National University of Ireland Galway Galway, Ireland Galway, Ireland Galway, Ireland [email protected] [email protected] [email protected] ABSTRACT If the solution with the highest novelty score is considered the Novelty search is a recent approach to evolving neural networks best solution in novelty search, it will likely be used to create that focuses on searching for networks with new and different be- new solutions. These solutions are then likely to produce similar haviour rather than solely focusing on finding the network with the behaviour to the original highly novel solution. However if these best objective fitness. In reality the concept of novelty is short lived new solutions have similar behaviour to the original highly novel in the sense that nothing stays new indefinitely. Algorithms that solution, by definition the original solution that is passed through archive the best solutions to inform the search are therefore faced to subsequent generations is not novel anymore. with the problem that the novelty scores of these archived solutions It should be noted here that this is not an issue for algorithms will change from generation to generation. This research aims to that are memory-less, as these algorithms do not consider previous address this issue by proposing two methods of adjusting novelty solutions at future generations. Therefore there is no concern as scores of archived solutions: 1) Novelty Decay. 2) Recalculating to whether or not the behaviour of these solutions is still novel. Archived Novelty. Novelty decay enables novelty scores to decay There are however many examples of algorithms that do have a overtime thus enabling the search algorithm to progress while re- memory of best solutions: elitism used in genetic algorithms and calculating novelty scores of the archived solutions updates the genetic programming [1], the global best and personal best solu- novelty of these solutions at each generation. When tested on the tions are recorded and used to update particles in particle swarm problem of maze navigation, it is observed that novelty decay and optimisation [2] and the agents in differential evolution only update recalculating archived novelty converge faster than both objective their solution if the new solution produced is better than their best search and novelty search alone. Recalculating archived novelty previous one [26]. and novelty decay perform statistically equal to one another. This research will explore two methods of addressing this issue: 1) To enable archived novelty scores to decay at each generation. KEYWORDS 2) To recalculate the novelty of archived novelty scores at each generation. By simply allowing the novelty scores of recorded be- Neuroevolution; Novelty Search; Neural Networks; Evolutionary haviours to decay over time, the novelty scores of previously novel Algorithms; Differential Evolution behaviours will degrade and therefore be replaced. This novelty decay accounts for the fact that the concept of novelty is inherently 1 INTRODUCTION temporary and allows algorithms that utilize recorded best solu- The notion of evolving a neural network by selecting for behavioural tions to find novel behaviors at subsequent generations. The second novelty rather than fitness has received a lot of attention since it method of recalculating archived novelty scores is a more compu- was first proposed [13]. The reason for the success of novelty search tationally expensive approach, but will have the same effect of is that by ignoring the objective during the search process, a more accurately updating the novelty of previously observed behaviors. diverse range of behaviours is explored. The process of searching In order to validate the proposed novelty decay and recalculat- for different behaviours gives the evolutionary algorithm a better ing archived novelty approaches, this research will apply novelty ability to explore. This enhanced exploration give novelty search search to differential evolution (DE) [26]. This is a state of the art an advantage when problems are highly deceptive. evolutionary optimisation algorithm for real valued optimisation This research focuses on tailoring novelty search for evolution- problems. This makes it well suited to optimising the weights of a ary algorithms that keep a record of the best found solutions and neural network. DE also keeps a record of the best found solutions, use these previous solutions to inform the search process. A typi- which makes it ideal to test out the proposed novelty decay and cal example of this would be evolutionary algorithms that utilize recalculating archived novelty techniques. Finally novelty search elitism, the process of passing solutions with the highest fitness has not yet been applied to DE, this paper aims to utilize novelty directly onto the next generation unchanged. This practise has been search to significantly improve the performance of DE like ithas shown to be highly effective for the standard objective based search for other algorithms [8, 28]. In order to test the performance of [1]. novelty search with differential evolution, this research will con- Applying novelty search to algorithms that keep a record of the sider the maze navigation problem domain which is routinely used best found solutions to inform the search, e.g. by using elitism, to evaluate the performance of evolutionary neural networks and raises the question: is a behaviour that was novel at the generation novelty search. it was first observed still considered novel in future generations? The contributions of this paper are: ∗Corresponding Author (1) To apply novelty search to differential evolution. (2) The evaluation of two techniques, novelty decay and recal- Neuroevolution methods generally consist of a population of culating archived novelty, to better facilitate the application network configuration. The population of networks iteratively im- of novelty search to algorithms with a memory of the best proves over time as a result of the application of evolutionary found solutions. operators, i.e. selection, crossover, mutation, etc. In neuroevolution, aspects of the neural network design are encoded into geno- 2 BACKGROUND type. These typically contain network information such as synaptic This section will give brief overview of the areas of: novelty search, weight values, number of neurons, connectivity, etc. These network neuroevolution and differential evolution. traits form the genotype. These genotypes are evolved over a series of generations. The phenotype is the expression of the genotype. 2.1 Novelty Search In neuroevolution, the phenotype is the actual neural network. Traditionally in machine learning and optimisation, a solution is 2.3 Differential Evolution judged based on its ability to satisfy an objective. This measure of The Differential Evolution (DE) algorithm is an evolutionary optimi- quality, or fitness, is also what is used to inform and guide thealgo- sation algorithm [26]. Differential evolution has been successfully rithm in its search for better solutions. Novelty search on the other applied to evolve neural networks for many different problems, in- hand, uses the novelty or uniqueness of a solution or behaviour cluding: control [17], tire contact prediction [7], power generation as a means to guide the search process. Novelty search was first [16, 19], air conditioning load forecasting [14], CPU forecasting [20] proposed by Lehman and Stanley in 2011 [13]. It has since had and wind power forecasting [18]. The algorithm uses the principles many successful applications including: controlling underwater of evolutionary computing to solve global real valued optimisation soft robots [4], airfoil design [23], classification [21] and computer problems. The algorithm creates N agents with random solutions games [15]. to the optimisation problem. At each generation, each agent selects Novelty can be defined as how different a behaviour iswith three other distinct agents A, B and C. For each dimension i of respect to other behaviours. Equation 1 shows how novelty is cal- the problem, if rand»0; 1¼ < CR or i = indexR the new solution culated in novelty search. Where nov is the novelty score given to variable y = a + F × ¹b − c º otherwise y = x . Where CR is the current behaviour x, k is the number of nearest neighbours and i i i i i i the crossover rate, D is the number of dimensions, R is a random µ is the behaviour observed by the ith nearest neighbour. i index 2 D, F is the differential weight, x® and y® are the current and new solutions of the current agent and a®, b® and c® are the current k 1 Õ ® nov¹xº = dist¹x; µ º (1) solutions of agents A, B and C. If the fitness of the new solution y k i ® ® ® i=0 is better than the previous solution x, y replaces x. This is repeated until a predetermined number of evaluations has been completed. For the maze problem domain considered in this research, the One of the contributions of this research is the application of distance measure between two behaviours is the Euclidean distance novelty search to differential evolution. Novelty search has already between the final positions of the robot for each behaviour. There been successfully applied to many other algorithms such as parti- have been other proposed measures of behavioural distance, e.g. cle swarm optimisation [8] and grammatical evolution [28]. It is hamming distance, normalized compression distance, entropy, etc hypothesized that differential evolution can also be enhanced by [6, 10]. This paper will utilize the Euclidean distance, as it is the combining it with novelty search. There are currently no examples natural choice for the maze problem domain. of this in the literature. This research will implement the normalized novelty score proposed by Cuccu and Gomez as it is a more general measure of 3 NOVELTY DECAY AND RECALCULATING novelty at each generation [5].

Load more