Machine Learning and Multiagent Preferences
Total Page:16
File Type:pdf, Size:1020Kb
Machine Learning and Multiagent Preferences Ritesh Noothigattu August 2020 CMU-ML-20-109 Machine Learning Department School of Computer Science Carnegie Mellon University Pittsburgh, PA 15213 Thesis Committee Ariel D. Procaccia (Chair), Harvard University Maria-Florina Balcan, Carnegie Mellon University Nihar B. Shah, Carnegie Mellon University Milind Tambe, Harvard University Submitted in partial fulfillment of the requirements for the degree of Doctor of Philosophy. Copyright © 2020 Ritesh Noothigattu This research was sponsored by the National Science Foundation award numbers CCF1525932 and IIS1714140, the Office of the Naval Research award number N000141712428, and the United States Army Research Office award number W911NF1320045. Keywords: machine learning, multiagent preferences, fairness, social choice, axioms, pooling, reinforcement learning. To everyone who helps me keep my smile :) Abstract One of the most well known settings dealing with multiagent preferences is voting and social choice. In classical social choice, each of the n agents presents a ranking over the m candidates, and the goal is to find a winning candidate (or a consensus ranking) that is the most \fair" outcome. In this thesis, we con- sider several variants of this standard setting. For instance, the domain may have an uncountably infinite number of alternatives, and we need to learn each voter's preferences from a few pairwise comparisons among them. Or, we have a markov decision process, and each voter's preferences are represented by its reward function. Can we find a consensus policy that everyone would be happy with? Another example is the setting of a conference peer review system, where the agents are the reviewers, and their preferences are given by the defining characteristics they use to accept a paper. Our goal is then to use these prefer- ences to make consensus decisions for the entire conference. We also consider the setting where agents have utility functions over a given set of outcomes, and our goal is to learn a classifier that is fair with respect to these preferences. Broadly speaking, this thesis tackles problems in three areas: (i) fairness in machine learning, (ii) voting and social choice, and (iii) reinforcement learning, each of them handling multiagent preferences with machine learning. Acknowledgments Without a doubt, I have been extremely fortunate to be advised by Ariel Procaccia. Even though it sounds obligatory, this has been absolutely true in my case. Ariel has been extremely supportive and very encouraging throughout my PhD studies. I literally could not ask for a better adviser. My first encounter with Ariel was during my MLD orientation, where Ariel presented his research, and I was really captivated by his talk, which was a perfect blend of discrete math and probability/statistics, and I ultimately decided to choose him as my adviser. Right from the beginning, Ariel has always used positive encourage- ment to motivate me, and I have never seen him be negative or forceful. His extreme niceness, friendliness and positivity definitely motivated me to push further, and I was also able to openly share my thoughts with him. His support has helped me achieve much more than I would have otherwise. Ariel is also an amazing perfectionist, which helped me improve in almost every aspect of research. For instance, I clearly remember when I was writing some of the sec- tions of my first papers with him, he spent substantial time in proofreading and updating them, and then gave me a list of points on how I could improve my writing. Looking at the extremely well-written and polished \after" versions of these sections, and his feedback, has been the main reason my writing improved several fold since then. I feel extremely fortunate to be advised by Ariel, and if I were given the option to go back in time and pick an adviser again, I would definitely choose him again without a doubt. And, if I ever become an adviser myself, I would aspire to be at least half as good as him :) I would also like to thank Maria-Florina Balcan, Nihar B. Shah, and Milind Tambe for being part of my thesis committee, and also all my other collabora- tors who helped make this thesis possible: Edmond Awad, Djallel Bouneffouf, Murray Campbell, Allissa Chan, Rachita Chandra, Travis Dick, Sohan Dsouza, Snehalkumar S. Gaikwad, Nika Haghtalab, Anson Kahng, Ji Tae Kim, Daniel Kusbit, Min Kyung Lee, Siheon Lee, Piyush Madan, Leandro S. Marcolino, Nicholas Mattei, Dominik Peters, Christos-Alexandros Psomas, Iyad Rahwan, Pradeep Ravikumar, Eric Rice, Francesca Rossi, Daniel See, Moninder Singh, Kush Varshney, Laura Onasch-Vera, Amulya Yadav, Tom Yan, and Xinran Yuan. I would also like to thank the department staff, especially Diane Stidle, for their very prompt help whenever anything was needed with respect to or around the department. A special thanks to all my friends who helped make my time at CMU a wonderful and unforgettable period of my life: especially Ian Char, Renato Negrinho, Biswajit Paria, and Tom Yan, but also Young Chung, Jeremy Co- hen, Mandy Coston, Ben Cowley, Nic Dalmasso, Chris Dann, Deepak Dilip- kumar, Avi Dubey, Karan Goel, Audrey Huang, Conor Igoe, Anson Kahng, Lisa Lee, Jeff Li, Terrance Liu, Octavio Mesner, Vaishnavh Nagarajan, Willie Neiswanger, Tolani Olarinre, Ben Parr, Darshan Patil, Barun Patra, Anthony Platanios, Adarsh Prasad, Ariel Procaccia (yes, again :D), Paul Schydlo, Otilia Stretcu, Mariya Toneva, Jake Tyo, Ezra Winston, Han Zhao, Xun Zheng, and many more I may have missed. I will never forget the fun times we had in the MLD PhD Lounge, especially the countless hours of playing smash, the board game nights, random chats, rock climbing, MLD retreats, working out, hanging out outside work and so on. I am especially extremely grateful to all of those who have been extremely supportive of me when I came out to them for the first time, and their support has definitely been priceless. These friends helped me pursue several non-academic trajectories of my life, and I cannot thank them enough for that. Lastly, it goes without saying, I would not have even been in a place to be able to start my PhD if my parents had not given me the opportunities I had growing up. In closing, I would again like to thank everyone who has been very supportive of me, and helped keep my positive energy burning :) Contents 1 Introduction1 1.1 Overview of Thesis Contributions.......................4 1.1.1 Fairness in Machine Learning.....................4 1.1.2 Voting and Social Choice........................4 1.1.3 Reinforcement Learning........................6 1.2 Bibliographic Remarks.............................7 1.2.1 Non-thesis research...........................7 I Fairness in Machine Learning9 2 Envy-Free Classification 11 2.1 Introduction................................... 11 2.1.1 Our Results............................... 13 2.1.2 Related Work.............................. 14 2.2 The Model.................................... 14 2.2.1 Envy-Freeness.............................. 15 2.2.2 Optimization and Learning....................... 15 2.3 Arbitrary Classifiers.............................. 16 2.4 Low-Complexity Families of Classifiers.................... 17 2.5 Implementation and Empirical Validation................... 18 2.5.1 Algorithm................................ 19 2.5.2 Methodology.............................. 20 2.5.3 Results.................................. 21 2.6 Conclusion.................................... 22 II Voting and Social Choice 25 3 Weighted Voting Via No-Regret Learning 27 3.1 Introduction................................... 27 3.2 Preliminaries.................................. 29 3.2.1 Social Choice.............................. 29 3.2.2 Online Learning............................. 30 3.3 Problem Formulation.............................. 31 ix 3.4 Randomized Weights.............................. 32 3.5 Deterministic Weights............................. 37 3.6 Discussion.................................... 40 4 Virtual Democracy: A Voting-Based Framework for Automating Deci- sions 43 4.1 Introduction................................... 43 4.2 Preliminaries.................................. 46 4.3 Aggregation of Permutation Processes..................... 47 4.3.1 Efficient Aggregation.......................... 48 4.3.2 Stability................................. 50 4.4 Instantiation of Our Approach......................... 50 4.5 Implementation and Evaluation........................ 52 4.5.1 Synthetic Data............................. 52 4.5.2 Moral Machine Data.......................... 54 4.6 Discussion.................................... 56 5 Loss Functions, Axioms, and Peer Review 57 5.1 Introduction................................... 57 5.2 Our Framework................................. 59 5.2.1 Loss Functions............................. 60 5.2.2 Axiomatic Properties.......................... 61 5.3 Main Result................................... 62 5.3.1 p = q = 1 Satisfies All Three Axioms................. 63 5.3.2 Violation of the Axioms When (p; q) 6= (1; 1)............. 65 5.4 Implementation and Experimental Results.................. 66 5.4.1 Varying Number of Reviewers..................... 67 5.4.2 Loss Per Reviewer........................... 68 5.4.3 Overlap of Accepted Papers...................... 69 5.5 Discussion.................................... 69 6 Axioms for Learning from Pairwise Comparisons 71 6.1 Introduction................................... 71 6.1.1 Why Is the Research Question Important?.............. 72 6.1.2