Fundamentals of Multiagent Systems with NetLogo Examples Jos´eM Vidal March 1, 2010 Fundamentals of Multiagent Systems by Jos´eM. Vidal Copyright c 2007 Jos´eM. Vidal. All rights reserved. Contents Preface 7 0.1 Usage . .7 0.2 Acknowledgments . .8 1 Multiagent Problem Formulation 9 1.1 Utility . .9 1.1.1 Utility is Not Money . 10 1.1.2 Expected Utility . 11 1.2 Markov Decision Processes . 12 1.2.1 Multiagent Markov Decision Processes . 14 1.2.2 Partially Observable MDPs . 15 1.3 Planning . 15 1.3.1 Hierarchical Planning . 16 1.4 Summary . 17 Exercises . 17 2 Distributed Constraints 19 2.1 Distributed Constraint Satisfaction . 19 2.1.1 Filtering Algorithm . 21 2.1.2 Hyper-Resolution Based Consistency Algorithm . 22 2.1.3 Asynchronous Backtracking . 24 2.1.4 Asynchronous Weak-Commitment Search . 27 2.1.5 Distributed Breakout . 29 2.2 Distributed Constraint Optimization . 31 2.2.1 Adopt . 32 2.2.2 OptAPO . 37 Exercises . 38 3 Standard and Extended Form Games 39 3.1 Games in Normal Form . 39 3.1.1 Solution Concepts . 40 3.1.2 Famous Games . 42 3.1.3 Repeated Games . 44 3.2 Games in Extended Form . 45 3.2.1 Solution Concepts . 46 3.3 Finding a Solution . 47 Exercises . 47 4 Characteristic Form Games and Coalition Formation 49 4.1 Characteristic Form Games . 49 4.1.1 Solution Concepts . 50 4.1.2 Finding the Optimal Coalition Structure . 55 4.2 Coalition Formation . 57 Exercises . 57 3 4 Contents 5 Learning in Multiagent Systems 59 5.1 The Machine Learning Problem . 59 5.2 Cooperative Learning . 61 5.3 Repeated Games . 61 5.3.1 Fictitious Play . 61 5.3.2 Replicator Dynamics . 63 5.3.3 The AWESOME Algorithm . 66 5.4 Stochastic Games . 67 5.4.1 Reinforcement Learning . 68 5.5 General Theories for Learning Agents . 71 5.5.1 CLRI Model . 71 5.5.2 N-Level Agents . 73 5.6 Collective Intelligence . 74 5.7 Summary . 76 5.8 Recent Advances . 76 Exercises . 76 6 Negotiation 77 6.1 The Bargaining Problem . 77 6.1.1 Axiomatic Solution Concepts . 78 6.1.2 Strategic Solution Concepts . 81 6.2 Monotonic Concession Protocol . 83 6.2.1 The Zeuthen Strategy . 84 6.2.2 One-Step Protocol . 86 6.3 Negotiation as Distributed Search . 86 6.4 Ad-hoc Negotiation Strategies . 87 6.5 The Task Allocation Problem . 88 6.5.1 Payments . 89 6.5.2 Lying About Tasks . 91 6.5.3 Contracts . 92 6.6 Complex Deals . 94 6.6.1 Annealing Over Complex Deals . 95 6.7 Argumentation-Based Negotiation . 96 6.8 Negotiation Networks . 98 6.8.1 Network Exchange Theory . 99 Exercises . 101 7 Auctions 103 7.1 Valuations . 103 7.2 Simple Auctions . 104 7.2.1 Analysis . 105 7.2.2 Auction Design . 107 7.3 Combinatorial Auctions . 107 7.3.1 Centralized Winner Determination . 108 7.3.2 Distributed Winner Determination . 113 7.3.3 Bidding Languages . 115 7.3.4 Preference Elicitation . 116 7.3.5 VCG Payments . 119 Exercises . 119 8 Voting and Mechanism Design 121 8.1 The Voting Problem . 121 8.1.1 Possible Solutions . 122 8.1.2 Voting Summary . 124 8.2 Mechanism Design . 124 8.2.1 Problem Description . 124 8.2.2 Distributed Mechanism Design . 131 Contents 5 8.2.3 Mechanism Design Summary . 134 9 Coordination Using Goal and Plan Hierarchies 137 9.1 tæms ................................... 137 9.2 GPGP . 139 9.2.1 Agent Architecture . 139 9.2.2 Coordination . 140 9.2.3 Design-to-Criteria Scheduler . 141 9.2.4 GPGP/tæms Summary . 141 10 Nature-Inspired Approaches 143 10.1 Ants and Termites . 143 10.2 Immune System . 143 10.3 Physics . 143 Bibliography 145 Index 153 Preface The goal of this book is to cover all the material that a competent multiagent practitioner or researcher should be familiar with. Of course, since this is a relatively new field the list of required material is still growing and there is some uncertainty as to what are the most important ideas. I have chosen to concentrate on the theoretical aspects of multiagent systems since these ideas have been around for a long time and are important for a wide variety of applications. I have stayed away from technological issues because these are evolving very fast. Also, it would require another textbook to describe all the distributed programming tools available. A reader interested in the latest multiagent technologies should visit the website www.multiagent.com. The book is linked to a large number of sample NetLogo programs (Wilensky, 1999). These programs are meant to be used by the reader as an aid in understanding emergent decentralized behaviors. It is hard for people, especially those new to Resnick points to examples distributed systems, to fully grasp the order that can arise from seemingly chaotic such as surveys of 8{15 year simple interactions. As Resnick notices: old kids, half of which believe that the government sets all \But even as the influence of decentralized ideas grows, there is a prices and salaries. deep-seated resistance to such ideas. At some deep level, people seem to have strong attachments to centralized ways of thinking. When peo- ple see patterns in the world (like a flock of birds), they often assume that there is some type of centralized control (a leader of the flock). Ac- cording to this way of thinking, a pattern can exist only if someone (or something) creates and orchestrates the pattern. Everything must have a single cause, and ultimate controlling factor. The continuing resis- tance to evolutionary theories is an example: many people still insist that someone or something must have explicitly designed the complex, Resnick created StarLogo in orderly structures that we call Life." (Resnick, 1994) order to teach the decentralized mindset. Wilensky, one of his The reader is assumed to be familiar with basic Artificial Intelligence techniques students, later extended (Russell and Norvig, 2003). The reader should also be comfortable with mathe- StarLogo and created NetLogo. matical notation and basic computer science algorithms. The book is written for a graduate or advanced undergraduate audience. I also recommend (Mas-Colell et al., 1995; Osborne and Rubinstein, 1999) as reference books. 0.1 Usage If you are using the pdf version of this document you can click on any of the citations and your pdf reader will take you to the appropriate place in the bibliography. From there you can click on the title of the paper and your web browser will take you to a page with a full description of the paper. You can also get the full paper if you have the user-name and password I provide in class. The password is only available to my students due to licensing restrictions on some of the papers. Whenever you see an icon such as the one on this margin it means that we have NetLogo implementation of a relevant problem. If you are using the pdf version ABTgc of this document you can just click on the name and your browser will take you to the appropriate applet. Otherwise, the url is formed by pre-pending http:// jmvidal.cse.sc.edu/netlogomas/ to the name and appending .html at the end. So the url for this icon is http://jmvidal.cse.sc.edu/netlogomas/ABTgc.html. 7 8 Contents 0.2 Acknowledgments I would like to thank the students at the University of South Carolina who have provided much needed feedback on all revisions of this book. Specifically, I thank Jimmy Cleveland, Jiangbo Dang, Huang Jingshan, and Alicia Ruvinsky. I am also especially grateful to faculty members from other Universities who have used this book in their classes and provided me with invaluable feedback. Specifically I thank Ram´onF. Brena, Muaz Niazi, and Iyad Rahwan. Chapter 1 Multiagent Problem Formulation The goal of multiagent systems' research is to find methods that allow us to build complex systems composed of autonomous agents who, while operating on local knowledge and possessing only limited abilities, are nonetheless capable of enacting the desired global behaviors. We want to know how to take a description of what a system of agents should do and break it down into individual agent behaviors. At its most ambitious, multiagent systems aims at reverse-engineering emergent phenom- ena as typified by ant colonies, the economy, and the immune system. Multiagent systems approaches the problem using the well proven.