Solution Concepts in Coevolutionary Algorithms ADissertation Presented to The Faculty of the Graduate School of Arts and Sciences Brandeis University Department of Computer Science Jordan B. Pollack, Advisor In Partial Fulfillment of the Requirements for the Degree Doctor of Philosophy by Sevan Gregory Ficici May, 2004 This dissertation, directed and approved by Sevan Gregory Ficici’s committee, has been accepted and approved by the Faculty of Brandeis University in partial fulfillment of the requirements for the degree of: DOCTOR OF PHILOSOPHY Adam B. Jaffe, Dean of Arts and Sciences Dissertation Committee: Jordan B. Pollack, DepartmentofComputer Science, Chair. James Pustejovsky, Department of Computer Science Michael J.Kearns,Department of Computer and Information Science, University of Pennsylvania Peter M. Todd, Center for Adaptive Behavior and Cognition, Max PlanckInstitute for Human Development c Copyright by Sevan Gregory Ficici 2004 Dedication To my dearest wife Hasmik and mother Silva In loving memory of my father Krikor and grandmother Hermine and my grandparents Siranoush, Himayak, and Sahag In memory of the 1.5 million Armenian martyrs of the 1915 Genocide, perpetrated by the Ottoman Turks and denied to this day by the Turkish government. iv Acknowledgments Ihavethepleasure to acknowledge some of the many people who have inspired, supported, and educated me over the years. Thanks to: Early influences:BruceM.Boghosian, Frank Blondale, Gordon Booth, Aleck Brinkman, J. Chapman Flack, Gregory Gargarian, Rick Inatome, Jacqueline Karaasla- nian, Tom Roeper, Pete Vanden Bosch. Colleagues and associates:Noubar Afeyan, Seth Bullock, Damon Centola, Sune Danø, Ezequiel A. Di Paolo, David B. Fogel, Joshua Knowles, Sandeep Krishna, Kamal Malek, Jason Noble, Ludo Pagie, R. Paul Wiegand, Darrell Whitley. Current and former members of the Brandeis community: Richard Alterman, Edward Balkovich, Eric Berkson, R. Scott Buchanan, Paul Buite- laar, Martin Cohn, Alan Danziger, Jeanne DeBaie,MyrnaFox, Andy Garland, Ira M. Gessel, Dan Griffin, Tamar Hajian, Timothy J. Hickey, Harry Mairson, Maja Mataric, Z. George Mou, James A. Storer, Patrick Tufts, David Waltz, Scott Waterman, David Wittenberg. Good friends:Clint Bidlack, David Davidian, Rachel E. Feldman, Rose Gulati, A.J. LoCicero, R. Steven Longmuir, Ohannes Salibian, Zarouhi Sarkisian, Taline Voskeritchian. Iammostfortunate to have had Jordan Pollack as my advisor. His enthusiasm, vision, encyclopedic knowledge, and intellectual rigor were all invaluable (especially considering my transition into a new field). I thank him for his kindness, patience, and guidance. I also thank him for assembling a first-rate group of researchers in the DEMO Lab; Jordan and the members (and alumni) of the DEMO Lab are the single most important source of intellectual inspiration for this dissertation. Grateful thanks to my friends: v Ari Bader-Natal, Alan D. Blair, Anthony Bucci, Keki Burjorjee, Paul Darwen, Edwin D. de Jong, Hugues Juill´e, Pablo Funes, GregoryHornby, Kristian Kime, Simon Levy, Hod Lipson, Ofer Melnik, Prem Melville, John Rieffel, Miguel Schneider- Font´an, Elizabeth Sklar, Ayal Spitz, Christian Swinehart, Shivakumar Viswanathan, and Richard A. Watson. Iwanttothank especially Shivakumar Viswanathan and Anthony Bucci, for the many discussions and feedback on my work. I owe particular thanks to Ofer Melnik, who was my collaborator on the research presented in Chapter 4. Foremost, I must thank Richard Watson for the many, many hours of challenging, imaginative, and insightful conversation that continually energized my efforts. Special thanks go to my dissertation committee, Jordan B. Pollack, James Puste- jovsky, Michael J. Kearns, and Peter M. Todd, for their insights and helpful feedback on this work. Finally, my deepest thanks, love, and affection go to my parents, Krikor and Silva Ficici, my grandmother, Hermine Panosyan, and my wife, Hasmik Vardanyan. For their love, patience, and support in all my endeavors, I am forever grateful. 8Jana[yl zimasdov;ivn yv zqrad5imanal zpans hanjaro319 Mysrob Ma,dox vi ABSTRACT Solution Concepts in Coevolutionary Algorithms Adissertation presented to the Faculty of the Graduate School of Arts and Sciences of Brandeis University, Waltham, Massachusetts by Sevan Gregory Ficici Inspired by the principle of natural selection, coevolutionary algorithms are search methods in which processes of mutual adaptation occur amongst agents that inter- act strategically. The outcomes of interaction reveal a reward structure that guides evolution towards the discovery of increasingly adaptive behaviors. Thus, coevolu- tionary algorithms are often used to search for optimal agent behaviors in domains of strategic interaction. Coevolutionary algorithms require little aprioriknowledge about the domain. We assume the learning task necessitates the algorithm to 1) discover agent behaviors, 2) learn the domain’s reward structure, and 3) approximate an optimal solution. Despite the many successes of coevolutionary optimization, the practitioner frequently observes a gap between the properties that actually confer agent adaptivity and those expected (or desired) to yield adaptivity, or optimality. This gap is manifested by a variety of well-known pathologies, such as cyclic dynamics, loss of fitness gradient, and evolutionary forgetting. This dissertation examines the divergence between expectation and actuality in co- evolutionary algorithms—why selection pressuresfailtoconform to our beliefs about adaptiveness, or why our beliefs are evidently erroneous. When we confront the pathologies of coevolutionary algorithms as a collection,wefind that they are essen- tially epiphenomena of a single fundamental problem, namely a lack of rigor in our solution concepts. Asolution concept is a formalism with which to describe and understand the in- centive structures of agents that interact strategically. All coevolutionary algorithms implement some solution concept, whether by design or by accident, and optimize ac- cording to it. Failures to obtain the desiderata of “complexity” or “optimality” often indicate a dissonance between the implemented solution concept and that required by our envisaged goal. We make the following contributions: 1) We show that solution concepts are the critical link between our expectations of coevolution and the outcomes actually deliv- ered by algorithm operation, and are therefore crucial to explicating the divergence between the two, 2) Weprovideanalytic results that show how solution concepts bring our expectations in line with algorithmic reality, and 3) We show how solution concepts empowerustoconstruct algorithms that operate more in line with our goals. viii Preface The research presented in Chapter 4 (and the authorship of Section 4.9) was done in collaboration with Ofer Melnik. The observation that coevolution can be regarded as a form of multi-objectiveoptimization, which we use in Chapter 7, was made in collaboration with Richard A. Watson. The research and portions of text presented in this dissertation appear in the following publications: • Chapter 3 S.G. Ficici and J.B. Pollack. A Game-Theoretic Approach to the Simple Coevolutionary Algorithm. In M. Schoenauer, et al, editors, Parallel Problem Solving from Nature VI, 467–476. Springer, 2000. • Chapter 4 S.G. Ficici, O. Melnik, and J.B. Pollack. A Game-Theoretic In- vestigation of Selection Methods Used in Evolutionary Algorithms. In Zalzala, et al, editors, Proc. of 2000 Congress on Evolutionary Computation, 880–887. IEEE Press, 2000. • Chapter 5 S.G. Ficici and J.B. Pollack. Game Theory and the Simple Co- evolutionary Algorithm: Some Results on Fitness Sharing. In R. Heckendorn, editor, 2001 Genetic and Evolutionary Computation Conference Workshop Pro- gram, 2–7. ISGEC, 2001. • Chapter 6 S.G. Ficici and J.B. Pollack. Effects of Finite Populations on Evolutionary Stable Strategies.InL.D.Whitley,etal,editors, Proceedings of ix the 2000 Genetic and Evolutionary Computation Conference, 927–934. Morgan- Kaufmann, 2000. • Chapter 7 S.G. Ficici and J.B. Pollack. Pareto Optimality in Coevolutionary Learning. In J. Kelemen and P. Sos´ık, editors, Sixth European Conference on Artificial Life (ECAL 2001), 316–325. Springer, 2001. • Chapter 8 S.G. Ficici and J.B. Pollack. A Game-Theoretic Memory Mech- anism for Coevolution. In Cant´u-Paz, et al, editors, 2003 Genetic and Evolu- tionary Computation Conference, 286–297. Springer, 2003. In addition, there exists a collection of relevant publications that are nevertheless either omitted from this dissertation or discussed only briefly: • S.G. Ficici and J.B. Pollack. Coevolving Communicative Behavior in a Linear Pursuer-Evader Game. In R. Pfeifer,B.Blumberg,J.-A.Meyer, and S.W. Wilson, editors, Proceedings of the Fifth International Conference of the Society for Adaptive Behavior, 505–514. MIT Press,1998. • S.G. Ficici and J.B. Pollack. Challenges in Coevolutionary Learning: Arms- Race Dynamics, Open-Endedness, and Mediocre Stable States. In C. Adami, et al, editors, Proc. of the Sixth Conf. on Artificial Life, 238–247. MIT Press, 1998. • S.G. Ficici and J.B. Pollack. Statistical Reasoning Strategies in the Pursuit and Evasion Domain. In D. Floreano, J.-D. Nicoud, F. Mondada, editors, Fifth European Conference on Artificial Life, 79–88. Springer, 1999. This document is Brandeis University Computer Science Department Technical Report CS-03-243. x Contents Dedication iv Acknowledgments v Abstract vii Preface ix Table of Contents xi List of Tables xix List of Figures