Combined Decision Making with Multiple Agents Thesis submitted for the degree Doctor of Philosophy Edwin Simpson Hertford College Supervisor: Prof. Stephen J. Roberts Pattern Analysis and Machine Learning Research Group Department of Engineering Science University of Oxford Michaelmas Term – April 17, 2014 Abstract In a wide range of applications, decisions must be made by combining information from multiple agents with varying levels of trust and expertise. For example, citizen science in- volves large numbers of human volunteers with differing skills, while disaster management requires aggregating information from multiple people and devices to make timely deci- sions. This thesis introduces efficient and scalable Bayesian inference for decision combi- nation, allowing us to fuse the responses of multiple agents in large, real-world problems and account for the agents’ unreliability in a principled manner. As the behaviour of individual agents can change significantly, for example if agents move in a physical space or learn to perform an analysis task, this work proposes a novel combination method that accounts for these time variations in a fully Bayesian manner us- ing a dynamic generalised linear model. This approach can also be used to augment agents’ responses with continuous feature data, thus permitting decision-making when agents’ re- sponses are in limited supply. Working with information inferred using the proposed Bayesian techniques, an information-theoretic approach is developed for choosing optimal pairs of tasks and agents. This approach is demonstrated by an algorithm that maintains a trustworthy pool of work- ers and enables efficient learning by selecting informative tasks. The novel methods developed here are compared theoretically and empirically to a range of existing decision combination methods, using both simulated and real data. The results show that the methodology proposed in this thesis improves accuracy and computa- tional efficiency over alternative approaches, and allows for insights to be determined into the behavioural groupings of agents. Acknowledgements Firstly, I would like to thank Professor Stephen Roberts, my supervisor, for all his ad- vice, help of various kinds and the freedom to pursue my own ideas. I must acknowledge the Engineering and Physical Sciences Research Council (EPSRC) and the Department of Engineering Science for funding my DPhil, and am also very grateful to the ORCHID project for their contributions to travel and providing an excellent forum for sharing and developing ideas. Huge thanks go to Arfon Smith and Chris Lintott, of the Zooniverse project, who con- tributed datasets for testing the ideas in this thesis. Many thanks also go to Steve Reece for great discussions, collaboration and the occasional beer, and also to Gopal Ramchurn and Antonio Penta, with whom we built the TREC crowdsourcing apparatus. Also to Ioannis Psorakis for developing the community analysis method applied in this work, and to Abby Levenberg for taking these ideas to a new domain. Overall, I would like to thank all the members of the Machine Learning group (MLRG) for all your thoughts and talks on topics of mutual interest, as well as making MLRG a great place to work. Similarly, thanks to the members of the ORCHID project for many stimulating conversations, to Hertford MCR and my other friends in Oxford for introducing me to many subjects outside Machine Learning. Finally, thanks are due to my family and friends for putting up with the writing process, especially Aew, whose ever-wonderful cooking kept me alive through it all. Related Publications Some of this work has previously appeared in the following publications: E. Simpson and S. Roberts and I. Psorakis and A. Smith (2013), Dynamic Bayesian Combination of Multiple Imperfect Classifiers. In Intelligent Systems Reference Library series: Decision Making with Imperfect Decision Makers, Intelligent Systems Reference Library series, Springer. E. Simpson and S. Reece and A. Penta and G. Ramchurn and S. Roberts (2013), Using a Bayesian Model to Combine LDA Features with Crowdsourced Responses. The Twenty- First Text REtrieval Conference (TREC 2012), Crowdsourcing Track. E. Simpson and S. Reece and G. Ramchurn and S. Roberts (2012), An Information Theoretic Approach to Managing Multiple Decision Makers. Human Computation for Sci- ence and Computational Sustainability Workshop, Neural Information Processing Systems (NIPS 2012). E. Simpson and S. Reece and G. Ramchurn and S. Roberts (2012), Dynamic Bayesian Combination of Multiple Imperfect Classifiers. Human Computation for Science and Com- putational Sustainability Workshop, Neural Information Processing Systems (NIPS 2012). E. Simpson and S. J. Roberts and A. Smith and C. Lintott (2011), Bayesian Combi- nation of Multiple, Imperfect Classifiers. 25th Annual Conference on Neural Information Processing Systems (NIPS), Workshop on Decision Making with Multiple Imperfect Deci- sion Makers. This thesis is entirely my own work, and the code that was used to run the experiments was also produced solely by the author, except for the overlapping community detection method, which was produced by Ioannis Psorakis. The dataset for the TREC Crowdsourc- ing challenge described in Chapter 5 was obtained using a system implemented collabora- tively with Sarvapali Ramchurn, Steven Reece and Antonio Penta. Contents 1 Introduction 1 1.1 Motivations . .4 1.1.1 Distributed Human Computation . .5 1.1.2 Ubiquitous, Mobile and Pervasive Computing . .7 1.1.3 Automation and Communication in Specialist Teams . .8 1.1.4 Information Overload . .9 1.2 Summary of Technical Challenges . 10 1.3 Bayesian Inference . 11 1.4 Contributions . 15 1.5 Overview of Thesis . 16 2 Decision Combination Methods 17 2.1 Fixed Combination Functions . 21 2.2 Supervised Methods . 24 2.2.1 Weighted Sums and LinOPs . 24 2.2.2 Weighted Products and LogOPs . 28 2.2.3 Supra-Bayesian Methods . 32 2.2.4 Sample Space Partitioning . 33 2.3 Unsupervised Methods . 34 2.3.1 Clustering Informative Agents . 35 2.4 Bayesian Classifier Combination . 41 2.4.1 IBCC Model . 42 i 2.4.2 Inference using Gibbs’ Sampling . 45 2.4.3 Relationships to other Combination Methods . 52 2.5 Empirical Comparison of Methods . 53 2.5.1 Evaluation Method . 56 2.5.2 Experiment 1: Weak Agents . 58 2.5.3 Experiment 2: Ability Varies by Target Value . 61 2.5.4 Experiment 3: Noise . 64 2.5.5 Experiment 4: Reversed Agents . 65 2.5.6 Experiment 5: Correlated Agents . 67 2.5.7 Experiment 6: Training the Combiner . 69 2.5.8 Discussion of Experimental Results . 71 2.6 Conclusions . 72 3 Efficient Application of Bayesian Classifier Combination 74 3.1 Application: Galaxy Zoo Supernovae . 75 3.2 Variational Bayesian IBCC . 77 3.2.1 Variational Bayes . 78 3.2.2 Variational Equations for IBCC . 80 3.2.3 The IBCC-VB Algorithm . 83 3.2.4 Variational Lower Bound . 86 3.3 Synthetic Data Experiments . 87 3.4 Galaxy Zoo Supernovae Experiments . 92 3.4.1 Balanced Data Results . 94 3.4.2 Imbalanced Data Results . 98 3.5 Galaxy Zoo Mergers Experiment . 100 3.6 HTTP Web Attack Classification . 103 3.7 Analysing Communities of Agents . 107 3.7.1 Π Communities . 108 3.7.2 Common Task Communities . 110 ii 3.8 Conclusions . 113 4 Modelling the Dynamics of Agents 115 4.1 Dynamic Independent Bayesian Classifier Combination . 116 4.2 Choosing a Dynamic Model for Confusion Matrices . 118 4.3 Dynamic Generalised Linear Model for DynIBCC . 120 4.3.1 Linear Models . 120 4.3.2 Generalised Linear Models . 121 4.3.3 Generalised Linear Model of Agent Responses . 122 4.3.4 Introducing Dynamics to the Generalised Linear Model . 123 4.3.5 Filtering . 124 4.3.6 Smoothing . 130 4.4 Variational Inference for DynIBCC . 132 4.4.1 Variational Lower Bound . 134 4.4.2 Duplicate and Missing Responses . 135 4.5 Synthetic Data Experiments . 136 4.6 Labelling Performance of DynIBCC with GZSN . 143 4.7 Dynamics of Galaxy Zoo Supernovae Contributors . 144 4.8 Dynamics of Π Communities . 149 4.9 Dynamics of Common Task Communities . 151 4.10 Discussion . 154 5 Intelligent Tasking 156 5.1 Related Work . 157 5.2 Case Study: TREC Crowdsourcing Challenge . 160 5.3 DynIBCC for Combining Probabilities . 161 5.3.1 TREC Results . 163 5.4 A Utility Function for Intelligent Tasking . 167 5.4.1 Exploitation and Exploration . 169 iii 5.5 Hiring and Firing for Crowdsourcing . 170 5.5.1 Online Screening Method . 174 5.6 Hiring and Firing Experiments . 174 5.6.1 Simulated Agents . 174 5.6.2 Alternative Methods . 175 5.6.3 Results . 176 5.6.4 Discussion . 182 6 Future Work and Conclusions 184 6.1 Sharing Agent Information . 185 6.2 Decentralised IBCC . 188 6.3 Collapsed VB . 190 6.4 Improved Decision-Making in Intelligent Tasking . 191 6.5 Optimising Future Rewards . 193 6.6 Preference Combination . 202 6.7 Summary of Future Work . 205 6.8 Limits to Decision Combination . 205 A Notation and Glossaries 207 B Algorithms 210 Bibliography 215 iv List of Figures 1.1 A diagram of a multi-agent system (MAS). .2 1.2 The web-based user interface for Galaxy Zoo. .6 (k) 2.1 K-means clustering on bi ......................... 37 (k) 2.2 K-means clustering on bi ......................... 38 (k) 2.3 Mean values of bi for each cluster at the current data point, i....... 40 2.4 Graphical Model for IBCC. 43 2.5 Experiment 1, varying sensor error rate, mean AUCs. 59 2.6 Experiment 1, varying sensor error rate, Brier score. 60 2.7 Experiment 1, ROCs at selected sensor error rates. 61 2.8 Experiment 2, varying class 1 error rate, ROC curves for agents. 62 2.9 Experiment 2, varying class 1 error rate, mean AUCs . 62 2.10 Experiment 2, ROC curves with class 1 error rate = 0:3.........