Participation is not a Design Fix for Machine Learning Mona Sloane * 1 Emanuel Moss * 2 Olaitan Awomolo * 3 Laura Forlano * 4 Abstract consent and is based on (post-)colonial structures of global This paper critically examines existing modes power (Peet & Hartwick); in the corporate sector where of participation in design practice and machine ”users” are invited into ”co-creation” sessions in order to learning. Cautioning against ”participation- create new product ideas; in the philanthropic sector where washing”, it suggests that the ML community ”the public” is challenged to join in defining new problems must become attuned to possibly exploitative and/or solutions to ”wicked problems”; or in the urban de- and extractive forms of community involvement sign or architecture sector where stakeholder engagement and shift away from the prerogatives of context- protocols often legitimize injustices in the (material) plan- independent scalability. ning of space and systematically devalue user needs as part of profit- and scale-oriented design practices, or design in- equality (Sloane, b;c). 1. Introduction Over the past years, we have seen mounting evidence 2. Participatory Design of the disparate impact of ML systems on already Participatory design methods can be traced to the 1970s oppressed and disadvantaged groups (Bolukbasi et al.; when workers in Scandinavia worked together collabora- Buolamwini & Gebru; Eubanks; Noble; O’Neal). The ex- tively to design the technologies that they would use in cor- periences of oppression and privilege are structural chal- porate settings (Schuler & Namioka; Sanders; Spinuzzi). lenges that are incredibly complex, and they are not Over the past several decades, participatory design and new particularly not to the communities that suffer from related concepts such as codesign and co-creation have them. But they have heightened alongside the exponen- been introduced as a way of engaging with ethics, values tial growth of wealth inequality alongside planetary de- in design (Nissenbaum), value-sensitive design (Friedman struction (Piketty; Hickel). It is therefore both unsurpris- 1996), and values levers in design (Shilton). Participatory ing and promising that the ML community wishes to build design with its rich history in socially democratic countries “more democratic, cooperative, and participatory ML sys- in Europe, has sought to engage multiple stakeholders in tems” (see workshop call). deliberative processes in order to achieve consensus. At Whilst this is an honorable goal, we want to caution the same time, other approaches have emphasized agonism against a familiar-sounding impulse towards “participation- and the importance of dissensus, friction and disagreement washing” that we have seen in other areas of design and (Keshavarz & Maze; DiSalvo, a; Mouffe; Hansson et al.). technology. For example, in the international development In this tradition of participatory design, the focus has been sector where “participation” of local communities at the re- on designing publics (DiSalvo, b) to engage in matters of ceiving end of powerful agencies is based on manufactured concern around complex socio-technical systems. In order to facilitate the engagement of multiple stakeholders in par- * 1 Equal contribution Institute for Public Knowledge, New ticipatory design processes, designers often use prototypes, York University, New York, USA; Tubingen¨ AI Center, BMBF games (Flanagan & Nissenbaum) and other structured ac- Competence Centre for Machine Learning (TUE.AI). Eberhard Karls University of Tubingen,¨ Tubingen,¨ Germany. Mona tivities. Sloane’s work was supported by the German Federal Ministry More recently, scholars have argued that nonhuman actors of Education and Research (BMBF): Tubingen¨ AI Center, FKZ: 01IS18039A. 2Data Society Research Institute, New York, USA; such as algorithms and machines (Choi et al.) as well as Department of Anthropology, CUNY Graduate Center, New York, the multispecies (microbes, plants, animals and the natu- USA 3Temple University, Philadelphia, Pennsylvania 4Illinois ral environment) be considered as stakeholders in partic- Institute of Technology, Chicago, Illinois. Correspondence to: ipatory design processes (Forlano & Halpern; Forlano, b; < > Mona Sloane [email protected] . Heitlinger et al.). Finally, with the introduction of criti- Proceedings of the 37 th International Conference on Machine cal and speculative design and experiential futures in the Learning, Vienna, Austria, PMLR 119, 2020. Copyright 2020 by early 2000s, design researchers have become interested in the author(s). Participation is not a Design Fix for Machine Learning the ways in which participatory design and design futures scraped from the open web and labeled by mTurk work- might come together to create new modes of experiential fu- ers (Krizhevsky et al.). Image classification tools are of- tures (Candy; Candy & Dunagan), design fiction (Bleecker; ten built on top of models trained on the ImageNet dataset. Forlano & Mathew), speculative design (Dunne & Raby), Photographers, web designers, and mTurk workers all par- speculative civics (DiSalvo et al.) and critical futures ticipate in every such application. A similar case presents (Forlano & Halpern; Forlano, c) in order to think through itself for Natural Language Processing applications which, the social consequences of emerging technologies. for over a decade, have sourced from Wikipedia for training language corpora (Gabrilovich & Markovitch). Participatory design methods have often been seen as a way of overcoming supposed difficulties that users have in un- Billions of ordinary web users also continually partici- derstanding ostensibly complex technologies, particularly pate in the production and refinement of ML, as their in healthcare settings (Neuhauser & Kreps). Participatory online (and offline) activities produce neatly labeled methods have also been employed where designers antic- rows of data on how they click their way around the ipate public resistance or skepticism to a product or ser- web, navigate their streets, and engage in any num- vice (Asaro). The use of participatory methods in tech- ber of other commercial, leisure, or romantic activities nology settings follows the development of participatory (Mayer-Schonberger¨ & Cukier). Users also improve the methods in other domains, particularly international devel- performance of ML models as they interact with them, a opment (Peet & Hartwick) where participation was seen as single unanticipated click can update a model’s parameters a means for overcoming local resistance to international de- and future accuracy. This work sometimes is so deeply velopment schemes (Goldman). integrated into the ways in which users navigate the In- ternet that it is performed unconsciously, e.g. when us- ML already incorporates certain forms of participation ing Google Maps and producing data movement patterns throughout the design of models and their integration into that enable traffic predictions. But other times it becomes society, however participatory design practices from other more conscious, e.g. when classifying photos when com- domains hold important lessons for ML. We will expand pleting a reCAPTCHA (O’Malley), or ranking Uber drivers the notion of ”participation” beyond the forms of involve- (Rosenblat). Where ML technology does not live up to it’s ment that are commonly understood as participatory design. mythos, people work behind the veil to complete tasks as Following the review of key literature on participatory de- if by the magic of AI. Behind some mobile apps claim- sign and ML, we will introduce three different forms of ing to use AI are real people transcribing images of pa- participation: participation as work, participation as con- per receipts and populating a purchase history database sultation, participation as justice, each illustrated with a list (Gray & Suri), moderating content (Roberts). The labor of examples. Through this framing, it becomes possible to of integrating new technologies, such as AI applications, understand how participatory design, a necessarily situated into everyday life and existing work processes and even out and context-dependent endeavor, articulates with industrial their rough edges, e.g. in healthcare (Sendak et al.), is the prerogatives of context-independent scalability. It also be- “human infrastructure” without which the socio-technical comes possible to recognize where the discourse of partici- system cannot function (Elish & Mateescu). Labor, here, pation fails to account for existing power dynamics and ob- is multi-layered and includes affective and emotional labor, scures the extractive nature of collaboration, openness, and e.g. coping with stress and sleep-deprivation when inte- sharing, particularly in corporate contexts. We conclude grating medical devices into everyday life (Forlano, a), or the paper with a set of recommendations drawn from con- social labor, e.g. when explaining ML outcomes to users or sidering a more expansive definition of participation in the even out their glitches such as when chatbots fail. All this context of ML. work often happens without consent or acknowledgement, and remains uncompensated. Such ML design processes 3. Different Forms of Participation are cases of “designing for”, i.e. processes that are void of a genuine integration of design users, relying on them to 3.1. Participation as Work make the design product work ex post. Much of ML plays out upon what is an intensely par- ticipatory