Evolutionary Computing in Cooperative Multiagent Environments

EVOLUTIONARY COMPUTING IN COOPERATIVE MULTI-AGENT ENVIRONMENTS

Lawrence Bull and Terence C Fogarty

Faculty of Computer Studies and Mathematics University of the West of England, Bristol, BS161QY, England E-mail: {l_bull, tcf} @btc.uwe.ac.uk

Abstract Weuse a two-agent trail following task to demonstrate the new operators. Thefields of Artificial Intelligenceand Artificial Life have The paper is arranged as follows: both focused on complexsystems in which agents must section two describes the simulated task and the rule- cooperateto achievecertain goals. In our workwe examine based frameworkused with the GA. Sections three to the performanceof the genetic algorithm whenapplied to five describe the use of the new operators and finally systems of this type. That is, we examinethe use of we discuss our results. population-basedevolutionary computing techniques within cooperative multi-agent environments.In extending the genetic algorithmto such environmentswe introduce three 2 A Multi-Agent Trail Following Task macro-leveloperators to reduce the amountof knowledge requireda priori; the joining of agents(symbiogenesis), the 2.1 The Tracker Task transfer of genetic material between agents and the speciation of initially homogeneousagents. Theseoperators Jefferson et al. [1989] presented a modelbased on the are used in conjunction with a generic rule-based trail following behaviour of ants, the Tracker task. framework,a simplified version of Pittsburgh-styleclassifier Their artificial ants (neural networksand finite state systems, which we alter to allow for direct systemic machines)must evolve the ability to follow a winding communicationto evolve betweenthe thus represented broken trail across a toroidal rectilinear grid agents. In this paperwe use a simulatedtrail followingtask environment,the John Muir trail. The trail is designed to demonstratethese techniques,finding that they can give improvedperformance. such that it becomesincreasingly difficult toward the end. The ants’ fitness is determined by howmuch of the trail (max89) they cover in a given lifetime of 200 1 Introduction time steps. Oncea cell of the trail is covered it is Nature is full of examples of both inter and deleted to encourage movement.We use these aspects intraspecies cooperation; from the workings of ant of Jefferson et al.’s modelbut alter the ant analogyto that of two single tracked vehicles learning to follow colonies to the cleaning symbiosis seen between the Pederson shrimp and the fish of the Bahamas. The the John Muir trail. Weemphasize that this task is used simply to introduce the new operators. Each vehicle fields of Artificial Intelligence and Artificial Life have consequently focused on these phenomenaas a means has a classifier system [Smith 1980]controller able to detect whetherthe trail is in the two locations in front of dealing with complexsystems in which agents must cooperate to achieve certain goals. In this paper we of it and return one of five actions - turn on forward/ backwardone or two revolutions, or do nothing - at introduce three macro-level operators to enhance the each discrete time step. Onerevolution will movethe use of population-based evolutionary computing vehicle one square in the appropriate direction. This techniques within cooperative multi-agent means that individually the vehicle can only move environments: symbiogenesis, gene transfer and forwards or backwards, intimating that they can speciation. That is, we extend the genetic algorithm potentially only follow any part of the trail whichlays (GA) [Holland 1975] to include processes found in their original path. In this modela pair of vehicles nature which can only be implementedby population- initially stand next to one another on the start position based search techniques within multi-agent systems. indicated in figure 1, facing the trail. Wemaintain that The operators are used in conjunction with a generic whenever both vehicles occupy the same location in rule-based frameworkto represent each agent, which their world, they are next to each other (on the same we alter to allow for direct communicationto evolve.

22 side of each other as at the start), whichthey are also parallel, messagepassing, rule-based systems running able to sense. in discrete time steps. The rules are in the form of The cooperative advantage comes condition/action sets, usually specified over the fromthe fact that by coupling together the vehicles are alphabet {0,1,#} (#represents a match all wildcard), able to turn on the spot, giving them the capacity to with an associated strength. All external input and completethe trail. Weallow that if both vehicles are internal messagepassing is done via a blackboard-like ever at the samelocation, they are coupled, and one’s messagelist. The classifiers use the GAfor their rule motor is on forward whilst the other’s is on backward, discovery; each system is represented by its rules then they turn on the spot in the appropriate direction, strung together to form a traditional GAgenome. To For exampleif the vehicle on the left turns on forward use this framework within multi-agent systems we two revolutions and the vehicle on the right turns on have altered the rule structure so that each action backwardtwo revolutions, and they are coupled, then contains an extra address "tag" and give each classifier they turn right 180° on the spot. Wesay that coupled system agent an address. Thenon a given time step all vehicles cannot drag one another, therefore if one or active rules of all agents also have their actions posted both are off neither vehicle moves.Similarly if in the onto the message list of the classifier system(s) above examplethe left vehicle had turned on forward specified by their tag; direct communicationis able to evolve through an external version of the rule-bases’ I0 message passing mechanism.A full description of the use and results from implementing our framework on this task can be foundin [Bull & Fogarty 1994]; due to limited space the evolution of communicationwill not be discussed here. Wegive each agent twelve bi- conditional rules for this task (unless otherwise stated), resulting in a genomeof 228 genes. We will now introduce our first macro-level operator which effects the identifiable nodes/agents of a given multi-agent system.

3 Symbiogenesis

3.1 A Macro-level Operator Figure 1: The John Muir trail in a 32x32 toroidal grid. Scores for reaching various Symbiogenesis is the name given to evolutionary landmarksare indicated - grey cells are not part of the trail; they markthe fastest innovation through the establishment of symbiotic route of traversal to the reader (they appear associations. Symbiosis is the phenomenonin which whiteto the trail followers). organismsof different species live together, resulting in a raised level of fitness for one or more of the organisms, i.e. heterogeneous cooperation. one° revolution, they would have only turned right 90 on the spot as the right cannot movethe left during its Left Right Left Right second revolution. Each single tracked vehicle scores 0 0 I every time it movesonto an element of the trail and both score if they moveonto an dement at the same time, whether or not they are coupled; trail is only deleted to encourage movement. We use a generational GA, with Figure 2: Over time the population space roulette wheel selection, allele mutation (rate 0.01), of the separate agents can be invaded by multi-point crossover (rate 0.01), with populations the joined configuration (Left&Right). 5000 individuals (unless otherwise stated). Endosymbiosis is the name given to symbiotic 2.2 Evolving Communicating Classifier Systems relationships in whichpartners are contained within a host partner. A large number of endosymbioses are Pittsburgh-style classifier systems [Smith 1980] are hereditary, wherein the host’s endosymbiontsare

23 passed directly to its offspring, e.g. through evaluatedon its ownside of the trail. Agentsare paired transmission in the egg cytoplasm, as seen in insects. with their corresponding opposite for evaluation This joining together of heterogeneous organismscan (left[x] is paired with fight[x], where 0organism [Allee and each receive their ownfitness measure. At the end et al. 1949] with the combinedabilities of the partners of the first and each successive generation a macro- involved. This phenomenonis nowwidely accepted as level operator probability Psy is tested to see if a the mechanism by which eukaryotic cells evolved symbiogenesis event will take place (the reverse [Margulis 1970]. process can also occur to stop drift). A symbiogenesis Within multi-agent systems it can event leads to two randomly chosen agents being be the case that someagents are highly interdependent. joined and forming a competingconfiguration (figure Under these conditions the configuration/behaviour of 2), where each super-agent receives the combined one agent has a large, often derogatory [Kauffman fitness of its two parts. Selection under the GAis run 1993], effect on the performanceof one or moreother over the total population"space" as usual except that if agents in the system. Wesuggest that multi-agent a separate agent is chosen to be created a systems containing these kinds of agents will benefit complementarypartner offspring is also created to froma joining process since they will no longer suffer allow for the fact that joined agents have a combined from the oscillatory dynamics of selfish interest; fitness rating and that an equal numberof left and fight joined agents will act as one super-agent and will agents must exist. Duringthe application of the usual therefore maximise their performance for both micro-level GAoperators of recombination and allele mutation joined genomes of the chosen parents are md0~l~lc og~t~ 90.0, i , I ’ i ¯ i ’ treated as if they were separate; each child of the G---Obolt 80.0F- o--io meon joined species is a mix of the corresponding genomes 70.01- carded by its parents. In this waythe most efficient 60.0 configuration (joined or separate) for the agents will ~, ~.0 vo 4O.O emergerather than being prescribed a priori.

30.0 a _ 20,0 3.2 Results IO.O L Wetried various rates for the operator (e.g. Psy = l/P, 0’00.0 I0.0 20.0 30.0 40.0 50.0 9me,naLion such that the operator is applied once per generation) but were unable to get any population configuration containing an hereditary endosymbiosis to establish itself. Figure 3 shows the typical progress of a fixed population configuration in which all members contain genomesfor both agents (symbiogenesis). can be seen that on average (10 runs) the configuration has been able to evolve controllers which only turn right continually on a break in the trail. Whenthis is comparedto the separate, completely heterogeneous, configuration it can be seen that the separate agents perform muchbetter, on average almost finishing the trail. This explains why we were unable to Figure 3" Showingthe performance of successfully evolve any joined agent configurations; it the (fixed) joined and (fixed) separate is moreefficient in this systemfor the agents to stay configurations. separated. Wenow introduce a second operator aspects. To implementsymbiogenesis as a macro-level whicheffects the genetic configuration of agents. mutation operator we alter the standard GAslightly. Wesay that at anytime two heterogeneous agents can 4 Horizontal GeneTransfer join to form a new population configuration in which an agent is responsible for both aspects of the system, 4.1 A Macro-level Operator i.e. they form a super-agent carrying the genomesof both agents. We start with the system in the Horizontal gene transfer is defined as "the transfer of heterogeneous configuration where an agent is genetic information from one genometo another,

24 specifically betweentwo species ... [and] requires (1) genome,in order. a vehicle to transport the genetic information between organisms and cells and (2) the molecular machinery 4.2 Results for inserting the foreign piece of DNAinto the host genome... Retroviruses can accomplish both tasks" Figure 5 shows the average (10 runs) results from [Li & Graur 1991]. Such transfers are particularly implementinghorizontal gene transfer (initially only) significant within endosymbioses"with the transfer of fromthe left agents to the right agents at a rate of I/(P/ genes, a symbiosis becomesmore closely integrated .. 2), whereone rule (chosen randomly)is transferred. can be seen that around generation 40 two agents, Left Right Left Right which have experienced the transfer of genes, evolved the ability to makea right turn; a pair of vehicles able

HaTdyr~mlet 140’0ca ,.- ’ u ’ J ’ t ’ I ’ n ’ O-----OInit °l° hi 40,0 ~ HaT P HGT ~).0 70.0 = = Figure 4: Over time the population space 40.0 of one configuration can be invaded by 40.0 40.0 the configuration containing agents which ~ ~ have transferred genes (rules). 30.0 ~-" 20.0 10.0 ¯ L. ¢ , t , a , i . the new genome may underlie metabolic pathways 0"00.0tOO20~ 30.0 40.0 40.0 40.0 70.0 40.0 40.0 IO0.O leading to an advantageous product that neither partner was capable of producing alone" [Margulis 1992]. This is also considered to have been significant in the evolution of eukaryoticcells [ibid.]. We suggest that multi-agent systems can also benefit from such a process whenthe necessary computational complexity of agents is difficult to establish a priori. Toimplement horizontal gene transfer as a macro-level mutation operator we alter the standard GAin a similar wayto that described above in section 3.1. We say that at anytime two heterogeneous agents can transfer genes and form a new population configuration in which the agent Figure 5: Showingthe performance and species carry different numbersof genes/rules from dynamicsof the gene transfer operator. the initial configuration (figure 4). Westart with the (m = mean scores) system in the heterogeneous configuration where the left vehicles’ genomescontain enough genes for two rules and the right vehicles with enoughgenes for one rule. Wealso disable the left vehicles’ ability to turn on to score morethan 10 on the trail evolved. That is, the backwards or fast backwards. These (slightly horizontal transfer of genes led to the evolution of a convoluted) restrictions are significant in that the more complexstrategy from initially restricted agent agents are unable to makeany turns since the right complexities, as expected. The configuration of vehicle requires two or morerules for such a strategy; vehicles with this moreeffective genetic arrangement the right will evolve its rule for whenthe agent faces then began to establish itself within the overall the trail, turning on forward, enabling a maximum population space, eventually dominating at around score of 10 for both agents. Agents are again paired 70%. correspondingly and each receive their ownfitness Finally we introduce a third measure. As before, a macro-level operator probability operator which looks at the assignment of populations Pgt is tested at the end of each evaluation-generationto to nodes/agents. see if a transfer event will take place (again the reverse process also occurs to stop drift). Transferred genes are concatenated onto the end of the receiving

25 5 Speciation 5.2 Results

5.1 A Macro-level Operator For this implementation we used an initial homogeneous population containing 10000 members The two suggested mechanisms for speciation are (P--10000)and set the speciation rate Psp at 1/(5P), termed allopatric and sympatric. Allopatric speciation on average a speciation event occurs once every five [Mayr 1942] - speciation through the geographic generations. From figure 7 it can be seen that on separation of a parent species - is the most widely average (10 runs) the speciation macro-level operator accepted explanation for the natural phenomenon. has allowed the most efficient configuration of That is, such separation stops or severely restricts the (heterogeneous) populations to agents/nodes flow of genetic material between individuals in the different populations which, combined with their Sp~k]tlondynornlc$ obviously different environments, leads to differentiation. Sympatricspeciation is the divergence of individuals in the same environment. A numberof causes are suggested for this form of speciation such "°f i/ ...... as the emergenceof phenotypic characteristics which 40,0 affect breeding (shape of genitalia, breeding times), i " the infertility of particular gene combinations, reproductive events such as polyploidy, etc. We suggest that evolutionary computing in multi-agent systems will also benefit from such a process. To implement speciation as a macro-level mutation operator we alter the standard 90.0 GAin a similar wayto that described above in section 80.0 Horn(m) Hem. 70,0 Her(,,,) 60.0 Left Right Left Right 50.0 i 411.0 30.0 20.0 10.0 ii° I Het Figure 6: Over time the population space Figure 7: Showingthe performance and of homogeneousagents can be invaded dynamicsof the speciation operator. by the heterogeneousconfiguration.

3.1. Wesay that at anytime two homogeneousagents emerge from an initial (parent) homogeneous can separate to form a newpopulation configuration in configuration around generation 25. which an agent is only evaluated on one aspect of the system, i.e. they can form the heterogeneous 6 Conclusion configuration of one population for each agent (figure 6). We start with the system in the homogeneous We suggest that population-based evolutionary configuration where any agent can be evaluated on computing techniques can be expanded to enhance either the left or the right of the trail. Agentsare paired their performancein multi-agent environments.In this consecutively (agent[0] is paired with agent[l], paper we have briefly introduced three macro-level agent[2] with agent[3], etc.) and each receive their mutation operators: symbiogenesis, horizontal gene own fitness measure. As before, a macro-level transfer and speciation. It was suggested that the operator probability Psp is tested at the end of each process of symbiogenesis will compensatefor whena evaluation-generationto see if a speciation event will system contains highly interdependent agents/nodes. take place (again the reverse process also occurs to Wefound that for the system chosen the division of stop drift). In this waythe most appropriate mix of one agent to each system aspect was the optimal hetero/homogeneous agents will be able to emerge configuration and that there was not enough rather than being prescribed. interdependencefor joining to provide any benefit;

26 search using nodes at the task’s finest granularity KauffmanS A. ed. 1993. The Origins of Order: Self- proved optimal here. The conditions under which Organisation and Selection in Evolution: Oxford symbiogenesis represents an advanced multi-agent University Press. process was recently examinedin [Bull et al. 1995] and was shownto be a small sub-set of the total space Li W-H & Graur D. eds. 1991. Fundamentals of of possibilities. Howeverwe maintain that the operator Molecular Evolution: Sinauer Associates. should be included in the evolution of multi-agent systems since it has proved so significant in natural Margulis L. ed. 1970. Origin of Eukaryotic Cells: Yale evolution [Margulis 1970]. Wehave also introduced a University Press. macro-level operator which allows agents to transfer genetic material to each other and it was shownthat Margulis L. ed. 1992. Symbiosis in Cell Evolution: W such a process can improve system performance when H Freeman and Company. unused genes in one agent are transferred to, and subsequently used by, another agent. Finally it was Mayr E. ed. 1942. Systematics and the Origin of shownthat speciation (population differentiation) can Species: ColumbiaPress, NewYork. be used to allow the most appropriate population configurations to emergealong with the solution to a Smith S F. ed. 1980. A Learning System Based on distributed problem, again reducing the amount of Genetic Adaptive Algorithms", PhD Dissertation, knowledgerequired a priori. These operators will be University of Pittsburgh. used on larger multi-agent systems in the future.

7 Acknowledgments

This work was supported by a Hewlett Packard Laboratories External Research Program.

8 References

Alice WC, EmersonA E, Schmidt K P, Park T & Park O. eds. 1949. Principles of Animal Ecology: Saunders Company.

Bull L & Fogarty T C. 1994.Evolving Cooperative CommunicatingClassifier Systems. In Proceedings of the Third Conference on Evolutionary Programming, 308-315: WorldScientific.

Bull L, Fogarty T C, & Pipe A G. 1995. Artificial Endosymbiosis. In Proceedings of the Third European Conference on Artificial Life, 273-289: Springer- Verlag.

Holland J H. ed. 1975. Adaption in Natural and Artificial Systems: Univ. of Michigan Press, Ann Arbor.

Jefferson D, Collins R, CooperC, Dyer M, Flowers M, Korf R, Taylor C & Wang A. 1989. Evolution as a Theme in Artificial Life: The Genesys/Tracker System. In Artificial Life II, 549-578: Addison- Wesley.