The Thirty-Fourth AAAI Conference on Artificial Intelligence (AAAI-20) Collective Information Ulle Endriss Institute for Logic, Language and Computation University of Amsterdam The Netherlands Abstract So what is this general pattern? We always start out with a set of agents, such as the students taking part in a MOOC. In Many challenging problems of scientific, technological, and fact, these agents need not be people. Rather, we may think societal significance require us to aggregate information sup- plied by multiple agents into a single piece of information of them simply as distinct sources of information. These of the same type—the collective information representing the agents are autonomous entities, so we have an obligation stance of the group as a whole. Examples include expressive to treat them fairly. Every agent provides us with a piece forms of voting and democratic decision making (where cit- of information—expressed in some formal language—and izens supply information regarding their preferences), peer we need to aggregate these individual pieces of information evaluation (where participants supply information in the form into a single piece of collective information, expressed in the of assessments of their peers), and crowdsourcing (where vol- same language. The formal language in question is deter- unteers supply information by annotating data). In this posi- mined by the domain of application. It could be a language tion paper, I outline the challenge of modelling, handling, and to encode expert judgments, restaurant appraisals, student analysing all of these diverse instances of collective informa- evaluations, budget allocations, or ontologies. tion using a common methodology. Addressing this challenge will facilitate a transfer of knowledge between different appli- This abstract perspective raises a number of questions: cation domains, thereby enabling progress in all of them. What are the best methods to implement such a process of aggregation? What general principles should guide our choice of method? But also: How should the particular fea- Introduction tures of the type of information at hand (i.e., of the formal When attempting to summarise the views of a committee of language used) affect our choice of method? Is it possible to experts, when trying to recommend a restaurant based on on- formulate general principles that are parametric in the type line reviews, or when hoping to compute a meaningful grade of information at hand? Answering the latter question, in for a student in a massively open online course (MOOC) particular, would allow us to generalise beyond specific so- that uses peer assessment, we always face the same daunt- lutions for specific applications and transfer insights about ing task: to aggregate multiple pieces of information of a designing good aggregation methods across applications. specific type, each contributed by a different agent, into a A specific domain where principles of aggregation have single piece of information of the same type that accurately been investigated in depth is that of preference aggregation, reflects the position taken by the group as a whole. Indeed, a which is the main object of study in social choice theory (Ar- wide range of challenging practical problems, originating in row, Sen, and Suzumura 2002). The methodology of clas- the most diverse corners of science, technology, and society, sical social choice theory encompasses conceptual analysis all fit this general pattern. Other examples include partic- informed by Philosophy, Political Science, and Economics ipatory budgeting (Cabannes 2004), collective argumenta- (to determine what makes for a good method of aggrega- tion (Bodanza, Tohme,´ and Auday 2017), and even ontology tion), a variety of mathematical tools (to formalise desider- merging (Flouris et al. 2008). While all of these application ata and explore their logical consequences), and empirical scenarios have been (and continue to be) addressed by dedi- studies (to better understand how different methods fare in cated research communities, to date there has been only rel- practice). Modern computational social choice, which has atively little effort directed towards understanding the gen- advanced the idea of thinking of methods of aggregation as eral pattern they all have in common. In this position paper, algorithms, extends this toolbox with a variety of techniques I argue that closing this gap has the potential to facilitate a from Computer Science and AI (Brandt et al. 2016). This much-needed transfer of knowledge between research areas, makes computational social choice a natural starting point thereby enabling progress on all of these applications. for investigating the phenomenon of collective information. Copyright c 2020, Association for the Advancement of Artificial The remainder of this position paper is organised as fol- Intelligence (www.aaai.org). All rights reserved. lows. I first discuss the phenomenon of collective informa- 13520 tion and how it may be formalised in a little more detail and for the output of F we may want to impose certain out- (what?). I then review a number of application domains that put constraints (Endriss 2018). would directly benefit from a deeper understanding of this For example, in the context of voting in our hiring com- phenomenon (why?) as well as a number of disciplines that mittee, we may think of L as the set of all weak orders on are likely to be relevant in developing such an understand- {Alice, Bob, Carol }, thereby allowing for ties in a ranking. ing (how?). Finally, I outline what I consider to be the main We could then impose antisymmetry as an input constraint challenge in this area: to understand how specific features of (forcing each agent to provide a strict ranking) and dichoto- the type of information to be aggregated should impact on mousness as an output constraint (forcing at most two levels, our choice of aggregation mechanism (whither?). thereby distinguishing only candidates accepted and candi- dates rejected for the job). As a second example, consider What? — Collective Information again the case of crowdsourcing annotations of a linguis- Collective information is what we obtain when we aggregate tic corpus. Maybe we want to allow each individual worker several individual pieces of information. Let L be a (finite) to annotate any number of sentence-pairs (so no input con- formal language for describing pieces of information rele- straint is required), but in the output we might require all vant to a specific application. For example, in the context pairs to be annotated (so the output constraint should ex- of the work of a hiring committee that has to rank three job clude annotations with question marks). candidates—Alice, Bob, and Carol—the language L might be the set of the 3! = 6 possible rankings of the candidates: Why? — Application Scenarios ⎧ ⎫ Being able to model, handle, and reason about collective in- ⎨ Alice Alice Bob Bob Carol Carol ⎬ formation is important, because collective information is a L = , , , , , ⎩ Bob Carol Alice Carol Alice Bob ⎭ core component of a wide range of application domains of Carol Bob Carol Alice Bob Alice scientific, technological, and societal relevance. Let us re- For other applications, we may need a very different lan- view some of them here. guage. For example, suppose we want to annotate a large The first three examples concern applications with a clear corpus of linguistic data with semantic information that can societal component. They belong, broadly speaking, to the be used by researchers in natural language processing (NLP) domain of politics and the design of democratic institutions. as a gold standard when testing machine learning techniques They particularly relate to collective decision making at the to recognise such semantic features automatically. We could local level, thereby promoting citizen involvement. try to do this by crowdsourcing many (likely low-quality) • Expressive voting. Running an election amounts to ag- annotations using a tool such as AMAZON’S MECHANICAL gregating information on the preferences of individual TURK and then aggregate this information into a (hopefully) voters into collective information regarding the prefer- high-quality gold standard annotation (Snow et al. 2008). ences of the electorate as a whole. Specifically, if L is In this context, L might be the set of all conceivable (par- the set of weak orders on a set of candidates, then F is tial) annotations. Imagine, for instance, we want to annotate a so-called social welfare function. By adding suitable the 800 sentence-pairs collected by Dagan, Glickman, and input and output constraints, we can model voting rules Magnini (2006) for the purpose of training NLP systems to for returning single election winners or sets of winners automatically recognise textual entailment. Then a partial of a given size. However, many real-world settings call annotation might be modelled by the list of the 800 iden- for more expressive models that allow, for instance, for tifiers used in this dataset, with each of them being labelled the possibility to not rank all candidates (Terzopoulou and by (if the first sentence of this pair entails the second), Endriss 2019) or to delegate your vote (Brill 2018). (if this is not the case), or a question mark (if this pair has not been annotated at all). Thus, L now has 3800 elements: • Community-driven policy design. Several innovative ⎧ ⎫ ideas fit under this heading. One that already is widely ⎪ ⎪ ⎪ 001 : 001 : 001 : 001 : ⎪ used in practice is participatory budgeting (Cabannes ⎨ . ⎬ 2004), which is about allowing citizens to directly par- L = . , . , . , . ,... ⎪ 799 : 799 : 799 : 799 : ⎪ ticipate in decisions about the allocation of budgets to ⎩⎪ ⎭⎪ 800 : 800 : 800 : ? 800 : different public projects. A few selected contributions al- ready employ the terminology of social choice theory to L Every agent reports an expression belonging to . The ele- discuss suitable mechanisms (Goel et al. 2016; Benade` L ments of are also the pieces of information an aggregation et al. 2017; de Haan 2018).