Deploying A↵ect-Inspired Mechanisms to Enhance Agent Decision-Making and Communication Adissertationpresented by Dimitrios Antos to The School of Engineering and Applied Sciences in partial fulfillment of the requirements for the degree of Doctor of Philosophy in the subject of Computer Science Harvard University Cambridge, Massachusetts May 2012 c 2012 - Dimitrios Antos All rights reserved. Thesis advisor Author Barbara J. Grosz Dimitrios Antos Deploying A↵ect-Inspired Mechanisms to Enhance Agent Decision-Making and Communication Abstract Computer agents are required to make appropriate decisions quickly and effi- ciently. As the environments in which they act become increasingly complex, effi- cient decision-making becomes significantly more challenging. This thesis examines the positive ways in which human emotions influence people’s ability to make good decisions in complex, uncertain contexts, and develops computational analogues of these beneficial functions, demonstrating their usefulness in agent decision-making and communication. For decision-making by a single agent in large-scale environments with stochas- ticity and high uncertainty, the thesis presents GRUE (Goal Re-prioritization Using Emotion), a decision-making technique that deploys emotion-inspired computational operators to dynamically re-prioritize the agent’s goals. In two complex domains, GRUE is shown to result in improved agent performance over many existing tech- niques. Agents working in groups benefit from communicating and sharing information that would otherwise be unobservable. The thesis defines an a↵ective signaling mech- anism, inspired by the beneficial communicative functions of human emotion, that increases coordination. In two studies, agents using the mechanism are shown to iii Abstract iv make faster and more accurate inferences than agents that do not signal, resulting in improved performance. Moreover, a↵ective signals confer performance increases equivalent to those achieved by broadcasting agents’ entire private state information. Emotions are also useful signals in agents’ interactions with people, influencing people’s perceptions of them. A computer-human negotiation study is presented, in which virtual agents expressed emotion. Agents whose emotion expressions matched their negotiation strategy were perceived as more trustworthy, and they were more likely to be selected for future interactions. In addition, to address similar limitations in strategic environments, this thesis uses the theory of reasoning patters in complex game-theoretic settings. An algorithm is presented that speeds up equilibrium computation in certain classes of games. For Bayesian games, with and without a common prior, the thesis also discusses a novel graphical formalism that allows agents’ possibly inconsistent beliefs to be succinctly represented, and for reasoning patterns to be defined in such games. Finally, the thesis presents a technique for generating advice from a game’s reasoning patterns for human decision-makers, and demonstrates empirically that such advice helps people make better decisions in a complex game. Contents TitlePage.................................... i Abstract..................................... iii TableofContents................................ v Citations to Previously Published Work . vii Acknowledgments................................ viii Dedication . x 1 Introduction 1 1.1 Emotions . 7 1.2 Background: Basicdecision-makingmodels . 14 1.2.1 Markov Decision Processes . 14 1.2.2 Game theory . 16 1.3 Structure of the thesis . 21 2 Re-prioritizing Goals Using A↵ect-Like Computational Operators 22 2.1 Complexity in Decision-Making . 23 2.1.1 Scale,HardnessandUncertainty. 24 2.1.2 DeliberativeandReactiveMethods . 29 2.2 GRUE: An Emotion-Inspired Method for Decision-Making . 31 2.2.1 EvaluationofGRUE ....................... 43 2.3 Related Work . 53 2.4 Discussion . 59 3 Decisions in Groups: A↵ective signaling 64 3.1 Inference and Communication in Multi-Agent Systems . 68 3.1.1 InferenceandImplicitSignaling . 70 3.1.2 Explicit Signaling . 74 3.2 A↵ectiveSignaling ............................ 76 3.2.1 A↵ectiveGenericSignals . 78 3.3 Evaluation . 84 3.3.1 Simple iterated social dilemma . 84 v Contents vi 3.3.2 Task Domain . 89 3.4 Signal Truthfulness and Manipulability . 96 3.5 Discussion and Extensions . 98 4 Interacting with Computers: Emotion Expressions 101 4.1 ComputersasSocialActors . 103 4.2 Perceptions of Trustworthiness . 108 4.2.1 Experiment Design . 110 4.2.2 Hypotheses ............................ 119 4.2.3 Results . 121 4.3 Discussion and Extensions . 127 5 The Reasoning Patterns: Simplifying and Representing Games 131 5.1 Reasoning Patterns in Games . 134 5.1.1 Multi-AgentInfluenceDiagrams . 135 5.2 Extending the Reasoning Patterns to Bayesian Games . 156 5.2.1 TheCommonPriorAssumption(CPA) . 157 5.2.2 Graphically representing Bayesian games . 164 5.2.3 Anaugmentedtheoryofreasoningpatterns . 174 5.3 SimplifyingGamesWithReasoningPatterns . 185 5.3.1 Proof of correctness . 192 5.3.2 AlgorithmComplexity . 196 5.3.3 Time Savings in Equilibrium Computation . 199 6 The Reasoning Patterns: Helping Humans 202 6.1 Using Reasoning Patterns to Assist Human Decision-Makers . 203 6.1.1 The principal-agent game . 205 6.1.2 Experiment implementation . 209 6.1.3 Using reasoning patterns for advice generation . 212 6.1.4 Reasoning patterns in the p-a game . 214 6.1.5 Strategic assumptions and approximations . 218 6.1.6 Results . 220 6.2 Discussion . 222 7 Conclusion & Extensions 225 7.1 ReflectionsontheUseofEmotions . 227 7.2 Future Possibilities . 228 Bibliography 232 Citations to Previously Published Work Large portions of Chapters 2 – 6 have appeared in the following papers: “The influence of emotion expression on perceptions of trustworthiness in negotiation”, Dimitrios Antos, Celso De Melo, Jonathan Gratch and Barbara Grosz, In the Proceedings of the Twenty-Fifth Conference on Ar- tificial Intelligence (AAAI), San Francisco, CA, August 2011; “Using Emotions to Enhance Decision-Making”, Dimitrios Antos and Avi Pfe↵er, In the Proceedings of the Twenty-Second Joint Conference on Ar- tificial Intelligence (IJCAI), Barcelona, Catalonia, Spain, July 2011; “Reasoning Patterns in Bayesian Games”, extended abstract,Dimitrios Antos and Avi Pfe↵er, In the Proceedings of the Conference on Au- tonomous Agents and Multi-Agent Systems (AAMAS), Taipei, Taiwan, May 2011; “Representing Bayesian Games with Non-Common Priors”, extended ab- stract, Dimitrios Antos and Avi Pfe↵er, In the Proceedings of the Confer- ence on Autonomous Agents and Multi-Agent Systems (AAMAS), Toronto, Ontario, Canada, May 2010; “Using Reasoning Patterns to Help Humans Solve Complex Games”, Dim- itrios Antos and Avi Pfe↵er, In the Proceedings of the Twenty-First Joint Conference on Artificial Intelligence (IJCAI), Pasadena, CA, July 2009; “Simplifying Games Using Reasoning Patterns”, student abstract,Dim- itrios Antos and Avi Pfe↵er, In the Proceedings of the Twenty-Second Conference on Artificial Intelligence (AAAI), Chicago, IL, July 2008; “Identifying Reasoning Patterns in Games”, Dimitrios Antos and Avi Pf- e↵er, In the Proceedings of the Conference on Uncertainty in Artificial Intelligence (UAI), Helsinki, Finland, June 2008. vii Acknowledgments This thesis would not have existed, were it not for the support, encouragement, and guidance of a large number of people. A great many thanks are owed to my ad- visor, Professor Barbara Grosz, and also to my former advisor, Professor Avi Pfe↵er, for their continual mentorship and assistance, academically, financially and person- ally. My gratitude is also extended to Professor David Parkes for giving me invaluable advice throughout my graduate studies, and for granting me the opportunity to prac- tice and sharpen my teaching skills and interact with a large number of incredibly talented students in the process. Professor Krzysztof Gajos, also, has granted me the privilege to be part of his research group, and has consistently been an approachable and most useful mentor, for which I am most grateful. Special thanks goes to Professor Jonathan Gratch, who twice admitted to the summer internship program of the Institute for Creative Technologies (ICT) of the University of Southern California. These two summers were not only immensely enjoyable, but his guidance helped give form and shape to my ideas, making this thesis possible. Along with him, I must also thank two more professors of ICT, Louis-Philippe (LP) Morency and Stacy Marsella, as well as a large number of other colleagues and interns there, above all Celso De Melo for being a consistent and most passionate collaborator and coauthor. Several courses at Harvard shaped my thoughts and equipped me with the knowl- edge and skills to tackle interesting research problems. I would like to especially thank Harvard Professors Stuart Shieber and David Laibson for teaching me about Computational Linguistics and Behavioral Economics, respectively, as well as MIT Professor Rosalind Picard for allowing me to attend her amazing A↵ective Computing viii Acknowledgments ix course. Adoctoralthesisisnotpossiblewithoutthesupportofindividualsthatdilute graduate school’s stressful times and enrich the good ones. My colleagues at the Harvard Artificial Intelligence Research Group (AIRG), the Harvard Mind, Brain and Behavior (MBB) society, and especially my long-term office mates and friends, Kobi Gal, Ece Kamar, Philip Hendrix, Katharina Reinecke, Lahiru Jayatilaka, Anna Huang, and Elena Agapie, deserve special thanks