Social Inductive Biases for Reinforcement Learning by Dhaval D.K. Adjodah B.S., Physics (2011), Massachusetts Institute of Technology M.S., Technology and Policy (2011), Massachusetts Institute of Technology Submitted to the Program in Media Arts and Sciences, School of Architecture and Planning, in partial fulfillment of the requirements for the degree of Doctor of Philosophy in Media Arts and Sciences at the MASSACHUSETTS INSTITUTE OF TECHNOLOGY September 2019 @ Massachusetts Institute of Technology 2019. All rights reserved. Signature redacted A uthor .................. Q .gi'gratiFMediaArts and Sciences, Sch of Architecture and Planning, ignature redacted 2, 2019 Certified by.......... KlV 'Sandy P. Pentland Toshiba Pr fessor of Media Arts and Sciences Thesis Supe risor Accepted by......... Signature redacted MASSACUET LNSTITUTE Tod Machover OF TECHNOLOGY (2 Academic Head 0 OCT 0 4 2019 Program in Media Arts and Sciences LIBRARIES ci 77 Massachusetts Avenue Cambridge, MA 02139 MITLibraries http://Iibraries.mit.edu/ask DISCLAIMER NOTICE The pagination in this thesis reflects how it was delivered to the Institute Archives and Special Collections. The Table of Contents does not accurately represent the page numbering. Social Inductive Biases for Reinforcement Learning by Dhaval D.K. Adjodah Submitted to the Program in Media Arts and Sciences, School of Architecture and Planning, on July 2, 2019, in partial fulfillment of the requirements for the degree of Doctor of Philosophy in Media Arts and Sciences Abstract How can we build machines that collaborate and learn more seamlessly with humans, and with each other? How do we create fairer societies? How do we minimize the impact of information manipulation campaigns, and fight back? How do we build machine learning algorithms that are more sample efficient when learning from each other's sparse data, and under time constraints? At the root of these questions is a simple one: how do agents, human or machines, learn from each other, and can we improve it and apply it to new domains? The cognitive and social sciences have provided innumerable insights into how people learn from data using both passive observation and experimental intervention. Similarly, the statistics and machine learning communities have formalized learning as a rigorous and testable computational process. There is a growing movement to apply insights from the cognitive and social sci- ences to improving machine learning, as well as opportunities to use machine learning as a sandbox to test, simulate and expand ideas from the cognitive and social sciences. A less researched and fertile part of this intersection is the modeling of social learning: past work has been more focused on how agents can learn from the 'environment', and there is less work that borrows from both communities to look into how agents learn from each other. This thesis presents novel contributions into the nature and usefulness of social learning as an inductive bias for reinforced learning. I start by presenting the results from two large-scale online human experiments: first, I observe Dunbar cognitive limits that shape and limit social learning in two different social trading platforms, with the additional contribution that synthetic financial bots that transcend human limitations can obtain higher profits even when using naive trading strategies. Sec- ond, I devise a novel online experiment to observe how people, at the individual level, update their belief of future financial asset prices (e.g. S&P 500 and Oil prices) from social information. I model such social learning using Bayesian models of cognition, and observe that people make strong distributional assumptions on the social data 3 they observe (e.g. assuming that the likelihood data is unimodal). I were fortunate to collect one round of predictions during the Brexit market instability, and find that social learning leads to higher performance than when learning from the underlying price history (the environment) during such volatile times. Having observed the cog- nitive limits and biases people exhibit when learning from other agents, I present an motivational example of the strength of inductive biases in reinforcement learning: I implement a learning model with a relational inductive bias that pre-processes the environment state into a set of relationships between entities in the world. I observe strong improvements in performance and sample efficiency, and even observe the learned relationships to be strongly interpretable. Finally, given that most modern deep reinforcement learning algorithms are distributed (in that they have separate learning agents), I investigate the hypothesis that viewing deep reinforcement learn- ing as a social learning distributed search problem could lead to strong improvements. I do so by creating a fully decentralized, sparsely-communicating and scalable learn- ing algorithm, and observe strong learning improvements with lower communication bandwidth usage (between learning agents) when using communication topologies that naturally evolved due to social learning in humans. Additionally, I provide a theoretical upper bound (that agrees with our empirical results) regarding which communication topologies lead to the largest learning performance improvement. Given a future increasingly filled with decentralized autonomous machine learning systems that interact with humans, there is an increasing need to understand social learning to build resilient, scalable and effective learning systems, and this thesis provides insights into how to build such systems. Thesis Supervisor: Alex "Sandy" P. Pentland Title: Toshiba Professor of Media Arts and Sciences 4 Social Inductive Biases for Reinforcement Learning by Dhaval D.K. Adjodah Signature redacted Thesis Advisor.............. ... Alex "Sandy" P. Pentland Toshiba Professor of Media Arts and Sciences MIT Program in Media Arts and Sciences T h esis read er ........................................................ Neil Lawrence Professor of Machine Learning Chair in Neuroscience and Computer Science Department of Computer Science The University of Sheffield / .Signature redacted Thesis reader ............ 6/Tim Klinger Research Staff Member IBM Research Al IBM Thomas J. Watson Research Center T h esis reader ........................................................ Esteban Moro Full Professor of the University Department of Mathematics Universidad Carlos III de Madrid Social Inductive Biases for Reinforcement Learning by Dhaval D.K. Adjodah T hesis A dvisor....................................................... Alex "Sandy" P. Pentland Toshiba Professor of Media Arts and Sciences MIT Program in Media Arts and Sciences Signature redacted Thesis reader ............... Neil Lawrence Professor of Machine Learning Chair in Neuroscience and Computer Science Department of Computer Science The University of Sheffield T h esis read er ........................................................ Tim Klinger Research Staff Member IBM Research Al IBM Thomas J. Watson Research Center T hesis reader ........................................................ Esteban Moro Full Professor of the University Department of Mathematics Universidad Carlos III de Madrid Social Inductive Biases for Reinforcement Learning by Dhaval D.K. Adjodah Signature redacted Thesis Advisor.................. Alex "Sandy" P. Pentland Toshiba Professor of Media Arts and Sciences MIT Program in Media Arts and Sciences T h esis read er ........................................................ Neil Lawrence Professor of Machine Learning Chair in Neuroscience and Computer Science Department of Computer Science The University of Sheffield T h esis reader ........................................................ Tim Klinger Research Staff Member IBM Research Al IBM Thomas J. Watson Research Center Signature redacted Thesis reader ...................... steban Moro Full essor of the University Department of Mathematics Universidad Carlos III de Madrid Acknowledgments I often ponder how I ended up being able to start a career focused on thinking about, and unravelling the mysteries of the universe. I think I owe it to a lot of luck, some laziness, some existential dread, and a lot to the people that have been there to support me throughout this journey. Some of these people are unfortunately no longer here for me to excitedly share my journey with them. Maman who, through herculean efforts, carried our family through unbelievable odds: I wish you were still in this world for me to tell you what I am working on. But as you told me, if we have been together at some point in the history of the universe, we are always together. Thank you. Sam Bowring, who, during my first semester of my freshman year at MIT, gave me a literal and figurative microphone to speak my mind, and call on other people's bullshit: I wish I could thank you for believing in me, and choosing to accompany me in my various mischief to make sure I was okay, instead of getting me in trouble for it. Mr. and Mrs. Tracol, thank you for showing me what it is like to live a life with little social compromise, and showing me that even people from faraway lands and different ages can become great friends. To them, who are no longer with me, I keep you in my heart. I still remember fondly the many times my dad taught me to break the rules - going fishing at the river, playing with electronics to make them do things they were not supposed to. Thank you for showing me that sometimes rules were meant to