Participation is not a Design Fix for Machine Learning

Mona Sloane * 1 Emanuel Moss * 2 Olaitan Awomolo * 3 Laura Forlano * 4

Abstract consent and is based on (post-)colonial structures of global This paper critically examines existing modes power (Peet & Hartwick); in the corporate sector where of participation in design practice and machine ”users” are invited into ”co-creation” sessions in order to learning. Cautioning against ”participation- create new product ideas; in the philanthropic sector where washing”, it suggests that the ML community ”the public” is challenged to join in defining new problems must become attuned to possibly exploitative and/or solutions to ”wicked problems”; or in the urban de- and extractive forms of community involvement sign or architecture sector where stakeholder engagement and shift away from the prerogatives of context- protocols often legitimize injustices in the (material) plan- independent scalability. ning of space and systematically devalue user needs as part of profit- and scale-oriented design practices, or design in- equality (Sloane, b;c). 1. Introduction Over the past years, we have seen mounting evidence 2. Participatory Design of the disparate impact of ML systems on already Participatory design methods can be traced to the 1970s oppressed and disadvantaged groups (Bolukbasi et al.; when workers in Scandinavia worked together collabora- Buolamwini & Gebru; Eubanks; Noble; O’Neal). The ex- tively to design the technologies that they would use in cor- periences of oppression and privilege are structural chal- porate settings (Schuler & Namioka; Sanders; Spinuzzi). lenges that are incredibly complex, and they are not Over the past several decades, participatory design and new particularly not to the communities that suffer from related concepts such as codesign and co-creation have them. But they have heightened alongside the exponen- been introduced as a way of engaging with ethics, values tial growth of wealth inequality alongside planetary de- in design (Nissenbaum), value-sensitive design (Friedman struction (Piketty; Hickel). It is therefore both unsurpris- 1996), and values levers in design (Shilton). Participatory ing and promising that the ML community wishes to build design with its rich history in socially democratic countries “more democratic, cooperative, and participatory ML sys- in Europe, has sought to engage multiple stakeholders in tems” (see workshop call). deliberative processes in order to achieve consensus. At Whilst this is an honorable goal, we want to caution the same time, other approaches have emphasized agonism against a familiar-sounding impulse towards “participation- and the importance of dissensus, friction and disagreement washing” that we have seen in other areas of design and (Keshavarz & Maze; DiSalvo, a; Mouffe; Hansson et al.). technology. For example, in the international development In this tradition of participatory design, the focus has been sector where “participation” of local communities at the re- on designing publics (DiSalvo, b) to engage in matters of ceiving end of powerful agencies is based on manufactured concern around complex socio-technical systems. In order to facilitate the engagement of multiple stakeholders in par- * 1 Equal contribution Institute for Public Knowledge, New ticipatory design processes, designers often use prototypes, York University, New York, USA; Tubingen¨ AI Center, BMBF games (Flanagan & Nissenbaum) and other structured ac- Competence Centre for Machine Learning (TUE.AI). Eberhard Karls University of Tubingen,¨ Tubingen,¨ Germany. Mona tivities. Sloane’s work was supported by the German Federal Ministry More recently, scholars have argued that nonhuman actors of Education and Research (BMBF): Tubingen¨ AI Center, FKZ: 01IS18039A. 2Data Society Research Institute, New York, USA; such as algorithms and machines (Choi et al.) as well as Department of Anthropology, CUNY Graduate Center, New York, the multispecies (microbes, plants, animals and the natu- USA 3Temple University, Philadelphia, Pennsylvania 4Illinois ral environment) be considered as stakeholders in partic- Institute of Technology, Chicago, Illinois. Correspondence to: ipatory design processes (Forlano & Halpern; Forlano, b; < > Mona Sloane [email protected] . Heitlinger et al.). Finally, with the introduction of criti- Proceedings of the 37 th International Conference on Machine cal and speculative design and experiential futures in the Learning, Vienna, Austria, PMLR 119, 2020. Copyright 2020 by early 2000s, design researchers have become interested in the author(s). Participation is not a Design Fix for Machine Learning the ways in which participatory design and design futures scraped from the open web and labeled by mTurk work- might come together to create new modes of experiential fu- ers (Krizhevsky et al.). Image classification tools are of- tures (Candy; Candy & Dunagan), design fiction (Bleecker; ten built on top of models trained on the ImageNet dataset. Forlano & Mathew), speculative design (Dunne & Raby), Photographers, web designers, and mTurk workers all par- speculative civics (DiSalvo et al.) and critical futures ticipate in every such application. A similar case presents (Forlano & Halpern; Forlano, c) in order to think through itself for Natural Language Processing applications which, the social consequences of emerging technologies. for over a decade, have sourced from Wikipedia for training language corpora (Gabrilovich & Markovitch). Participatory design methods have often been seen as a way of overcoming supposed difficulties that users have in un- Billions of ordinary web users also continually partici- derstanding ostensibly complex technologies, particularly pate in the production and refinement of ML, as their in healthcare settings (Neuhauser & Kreps). Participatory online (and offline) activities produce neatly labeled methods have also been employed where designers antic- rows of data on how they click their way around the ipate public resistance or skepticism to a product or ser- web, navigate their streets, and engage in any num- vice (Asaro). The use of participatory methods in tech- ber of other commercial, leisure, or romantic activities nology settings follows the development of participatory (Mayer-Schonberger¨ & Cukier). Users also improve the methods in other domains, particularly international devel- performance of ML models as they interact with them, a opment (Peet & Hartwick) where participation was seen as single unanticipated click can update a model’s parameters a means for overcoming local resistance to international de- and future accuracy. This work sometimes is so deeply velopment schemes (Goldman). integrated into the ways in which users navigate the In- ternet that it is performed unconsciously, e.g. when us- ML already incorporates certain forms of participation ing Google Maps and producing data movement patterns throughout the design of models and their integration into that enable traffic predictions. But other times it becomes society, however participatory design practices from other more conscious, e.g. when classifying photos when com- domains hold important lessons for ML. We will expand pleting a reCAPTCHA (O’Malley), or ranking Uber drivers the notion of ”participation” beyond the forms of involve- (Rosenblat). Where ML technology does not live up to it’s ment that are commonly understood as participatory design. mythos, people work behind the veil to complete tasks as Following the review of key literature on participatory de- if by the magic of AI. Behind some mobile apps claim- sign and ML, we will introduce three different forms of ing to use AI are real people transcribing images of pa- participation: participation as work, participation as con- per receipts and populating a purchase history database sultation, participation as justice, each illustrated with a list (Gray & Suri), moderating content (Roberts). The labor of examples. Through this framing, it becomes possible to of integrating new technologies, such as AI applications, understand how participatory design, a necessarily situated into everyday life and existing work processes and even out and context-dependent endeavor, articulates with industrial their rough edges, e.g. in healthcare (Sendak et al.), is the prerogatives of context-independent scalability. It also be- “human infrastructure” without which the socio-technical comes possible to recognize where the discourse of partici- system cannot function (Elish & Mateescu). Labor, here, pation fails to account for existing power dynamics and ob- is multi-layered and includes affective and emotional labor, scures the extractive nature of collaboration, openness, and e.g. coping with stress and sleep-deprivation when inte- sharing, particularly in corporate contexts. We conclude grating medical devices into everyday life (Forlano, a), or the paper with a set of recommendations drawn from con- social labor, e.g. when explaining ML outcomes to users or sidering a more expansive definition of participation in the even out their glitches such as when chatbots fail. All this context of ML. work often happens without consent or acknowledgement, and remains uncompensated. Such ML design processes 3. Different Forms of Participation are cases of “designing for”, i.e. processes that are void of a genuine integration of design users, relying on them to 3.1. Participation as Work make the design product work ex post. Much of ML plays out upon what is an intensely par- ticipatory field. Whether acknowledged or not, a broad 3.2. Participation as Consultation range of participants play an important role in produc- ing the data that is used to train and evaluate ML mod- In the case of participation as consultation, cf. els. For example, ImageNet, which laid the foundations (Martin Jr. et al., 2020), designers and technologists for deep learning and most image recognition applications engage in episodic, short-term projects in which diverse and is still used for ML benchmarking, is an dataset of mil- stakeholders might be consulted at various stages of the lions of images, taken by hundreds of thousands of people, process. This model is most common in architecture and urban planning as well as among major philanthropic Participation is not a Design Fix for Machine Learning foundations and private corporations. Architecture ism, and settler colonialism (Costanza-Chock); crip techno- and urban planning practices use citizen participation science, which refuses demands to eliminate disability, un- approaches to engage different stakeholder groups in derscores that disabled people are expert designers of every- project development. As these projects are complex and day life, and centers technoscientific activism, critical de- have significant socio-economic impacts on communi- sign practices, and disability justice (Hamraie & Fritsch); ties, participatory workshops can provide an integrated data feminism, which focuses on ideas of intersectional framework where experts work with stakeholder groups feminism (D’Ignazio & Klein); and tech activism and re- to identify context-specific needs (Bratteteig & Verne; sistance, both from people affected by potentially harmful Saad-Sulonen & Horelli). Here, participation might be technology, such as the Atlantic Towers Resident Associa- facilitated through small, face-to-face workshops or larger tion in Brooklyn, NY (Gagne) and those designing it, see design sprints or hackathons as well as through the use of for example the Tech Worker Movement (Tarnoff), or a mix online platforms for crowdsourcing ideas. of both, such as Data for Black Lives, Black in AI, or Lat- inX in AI. What ties these approaches together is favoring There are several challenges that can limit the effectiveness using language around ”designing with” in order to ensure of participation as consultation. For a variety of reasons in- that outcomes are valuable to people from diverse back- cluding intellectual property concerns, in this model, long- grounds and communities, including the disability commu- term partnerships are either impossible, undesirable, unnec- nity. Participation as justice has social and political impor- essary or cost prohibitive. As this type of top-down de- tance, but it may be difficult to do it well, especially in a sign process also takes the form of ”designing for” a par- corporate context. Here, design justice can almost be seen ticular group without an ongoing commitment to their in- as an oxymoron: given the extractive and oppressive capi- clusion in the process, systemic inequalities that can be talist logics and contexts of ML systems, it appears impos- hard-coded into consultation and representation protocols sible to design ML products that are genuinely “just” and (Sloane, c). Experts do not often have a good understand- “equitable”. ing of how to design effective participatory processes or engage the right stakeholders to achieve the desired out- comes. A third challenge occurs as cities begin to require 4. Critiques of Participation participation workshops as part of the permitting and ap- The dominant mode of extraction within the ML industry provals process. Participation workshops can become per- is deeply entangled with the capitalist paradigm of scale, formative, where experts do not actually take the needs or referring to the ability to gain revenue at a greater pro- recommendations of the different stakeholder groups into portion per unit cost of inputs (Chandler & Hikino). But consideration (Crosby et al.). as a tech industry buzzword, the verb ”to scale” refers to the ability of products to spread far beyond the context of 3.3. Participation as Justice development to new applications in new markets. Part of In the case of participation as justice, designers and tech- the promise of ML is that statistical generalizations learned nologists engage in more-long term partnerships with di- from finite datasets will allow for inferences to be made verse stakeholders. In order to build trust, it is impor- across broader contexts, and that capabilities engendered tant to create ongoing relationships based on mutual ben- by ML can be applied to additional settings without adding efit, reciprocity, equity and justice. Here, all members proportional costs. However, datasets are deeply context- of the design process engage in more tightly coupled re- bound, and that context, as well as the appropriateness of lationships with more frequent communication (which of- the use of those datasets, is lost in the scaling of ML appli- ten happens through a blended communication and in- cations (boyd & Crawford). teraction approach, e.g. online/offline). The canon of Acknowledging the modes of participation that are already participation as collaboration notably comprises participa- components of ML challenges understandings of how these tory action research, which is focused on researchers and tools are able to scale. As such technologies scale across participants undertake action-oriented and self-reflexive contexts, the generalizations that are learned inevitably re- practices that leads to them having more control over quire updating, by providing additional training data or cor- their lives (Baum et al.); infrastructuring, which centers recting errors (Selbst et al.). This often requires the partici- designers’ locations, the materials and systems intrinsic pation of users interacting with the system who experience to designing, as well as (community) capacity building the friction of providing additional information to the sys- (Agid; Hillgren et al.; Le Dantec & DiSalvo); design jus- tem (as with CAPTCHAs) or bearing the burden of system tice, which goes beyond value-focused design and centers errors. As discussed above, representation/consultation is typically marginalized groups in collaborative and creative often prohibitively costly. Where a cost-benefit analysis design processes that challenge and dismantle the matrix of may encourage such forms of participation in the earlier domination, i.e. white supremacy, heteropatriarchy, capital- Participation is not a Design Fix for Machine Learning stages of product development, in later stages that product must be revisited and reexamined to ensure they are gath- is expected to scale without incurring additional costs. The ering the right information from the right people. As ML initial utility of representative and consultative forms of systems affect a wide range of groups, marginalized stake- participation are thus diluted as products scale beyond the holders should be given the space and voice to co-design context in which that mode of participation contributed to and co-produce these systems (Crosby et al.). Document- the overall design of the product in earlier stages. For ML ing these processes and their contexts can form a knowl- products to simultaneously scale and engage in meaningful edge base for long term, effective participation. partnerships oriented toward justice, they also require addi- 3. Participation as justice must be genuine and long tional inputs of participation, and budgets must be set aside term. This means to engage in creating processes that pro- for that. vide transparency and genuine knowledge sharing. This This can be thought of as levelling the playing field of futur- can be difficult particularly for proprietary design cases. ing: product futures are often made very concrete for ven- Further, using the language of design justice without ac- ture capitalists. But what kind of imaginative work do en- tually engaging in actual design justice processes and prac- trepreneurs do when it comes to the communities that they tices can only lead to corporate co-optation. For example, seek out as users (or targets) of their products? There is the ML field has seen a hype of “ethical AI” serving as an existing imbalance between market-fit and community- a smokescreen for continuing with non-participatory and fit. To address that and pave the way for design justice non-justice oriented ML design approaches (Sloane, a), de- processes to become integral to ML, it is key to expand spite good intentions. To avoid that, it may be helpful to the notion of value beyond monetary value and the extrac- make the tensions that characterize the goal of long term tive logics underpinning the invasive data collection that is participation in ML visible, acknowledging that partner- necessitated by most ML system designs. Promising devel- ships and justice do not scale in frictionless ways, but re- opments have recently been made in the context of Indige- quire constant maintenance and articulation with existing nous data sovereignty which includes access, control and social formation in new contexts (Tsing). governance of Indigenous data (Anderson & Hudson). We argue that it is crucial to enhance the ability for lat- Against that backdrop we suggest three cues for consider- eral thinking across applications and academic disciplines ing participation in ML in a more equitable way: (“holistic futuring”), because harms can be produced by the same ways of thinking that produce the technology 1. Recognize participation as work. Users already labor that causes the harms. This maps onto Vaughn’s (Vaughan in, for, and through ML systems across a number of dimen- 1996) normalization of deviance and could benefit from sions (affective, social, emotional). This labor upholds and cross-checking or lateral thinking between disciplines and improves ML systems and therefore is valuable for the own- forms of expertise. Such an approach could facilitate the ers of the ML systems. To acknowledge that, users should development of an ontology of (design) harms or “design be asked for consent, be provided with opt-out options or inequalities” (Sloane, a). To facilitate these efforts, we pro- alternatives, and, if they chose to participate through labor, pose to develop a searchable database of design precedents be offered compensation. This could mean to clarify when across applications and disciplines that highlights design and how data generated by user behaviour is used for the failures, especially failures of design participation, cross- training and improvement of ML systems (e.g. via a ban- referenced with socio-structural dimension (e.g. issues per- ner on the Wikipedia page, or in Google Maps); to give taining to racial inequality, or class-based inequity). This an alternative security option for reCAPTCHA; to not pun- database should cover design projects across all sectors ish users for refusing to leave reviews; to provide appropri- and domains, not just ML, and explicitly acknowledge de- ate support for content moderators; to compensate “ghost liberate absences and outliers which often are the most workers” fairly (Gray & Suri); to develop reward systems interesting and relevant social phenomena we can learn for users that labor to integrate technologies into their lives from (e.g. transgender identities). It may also acknowl- and thereby provide rich data for profit-oriented ML com- edge and educate on the deliberate refusal to “get counted” panies. (D’Ignazio & Klein). 2. Participation as consultation must be designed for specific contexts. If short-term participation is the most 5. Conclusion feasible and desired version for ML participation, then there needs to be a commitment to context-specificity, es- In this paper, we have cautioned against “participation- pecially in terms of how the participation is facilitated. Ev- washing” of ML by critically examining the existing kinds ery context is different, so participation has to be designed of participation in design practice and ML. Existing forms to address these different contexts. Rather than a one-size- of participation can be classified as work, as consultation, fits-all approach, consultation and representation processes and as justice, but we have argued that the notion of ”par- Participation is not a Design Fix for Machine Learning ticipation” should be expanded to acknowledge more sub- Bratteteig, T. and Verne, G. Does AI make PD ob- tle, and possibly exploitative, forms of community involve- solete?: exploring challenges from artificial intel- ment in participatory ML design. This framing allows for ligence to participatory design. In Proceedings of understanding participatory design as a necessarily situated the 15th Participatory Design Conference on Short and context-dependent endeavor which is at odds with in- Papers, Situated Actions, Workshops and Tutorial dustrial prerogatives of extraction and context-independent - PDC ’18, pp. 1–5. ACM Press. ISBN 978-1- scalability. Against that backdrop, it is imperative to rec- 4503-5574-2. doi: 10.1145/3210604.3210646. URL ognize design participation as work; to ensure that partic- http://dl.acm.org/citation.cfm?doid=3210604.3210646 ipation as consultation is context-specific; and that partic- ipation as justice must be genuine and long term. There- Buolamwini, J. and Gebru, T. Gender shades: Intersec- fore, we argue that developing a cross-sectoral database of tional accuracy disparities in commercial gender classi- design participation failures that is cross-referenced with fication. In Proceedings of Machine Learning Research, socio-structural dimensions and highlights “edge cases” volume 18. that can and must be learned from. Candy, S. The futures of everyday life: Politics and the design of experiential scenarios. References Candy, S. and Dunagan, J. Designing an experiential sce- Agid, S. ”...it’s your project, but it’s not necessarily nario: the people who vanished. 86:136–153. ISSN your work...”: infrastructuring, situatedness, and de- 0016-3287. Type: Journal Article. signing relational practice. In Proceedings of the 14th Participatory Design Conference on Full papers Chandler, A. D. and Hikino, T. Scale and scope: the dy- - PDC ’16, pp. 81–90. ACM Press. ISBN 978-1- namics of industrial capitalism. Belknap Press. ISBN 4503-4046-5. doi: 10.1145/2940299.2940317. URL 978-0-674-78994-4. http://dl.acm.org/citation.cfm?doid=2940299.2940317. Choi, J. H.-j., Forlano, L., and Kera, D. Situated automa- Anderson, J. and Hudson, M. IP and indigenous data tion: Algorithmic creatures in participatory design. In sovereignty: The traditional knowledge and biocultural Proceedings of the Participatory Design Conference. labels and notice system. In NYU Engelberg Innovation Costanza-Chock, S. Design Justice. MIT Press. Type: Colloquium Paper. Book.

Asaro, P. M. Transforming society by transforming tech- Crosby, Bryson, Quick, and Slotterback. Designing public nology: the science and politics of participatory design. participation processes. In Leadership for the common 10:34. good: tackling public problems in a shared-power world, Public Administration Review. Baum, F., MacDougall, C., and Smith, D. Partic- ipatory action research. 60(10):854–857. ISSN D’Ignazio, C. and Klein, L. F. Data Feminism. Strong ideas 0143-005X. doi: 10.1136/jech.2004.028662. URL series. The MIT Press. ISBN 978-0-262-04400-4. http://jech.bmj.com/cgi/doi/10.1136/jech.2004.028662. DiSalvo, C. Adversarial design. MIT Press, a. ISBN 0- Bleecker, J. Design fiction: A short essay 262-01738-5. Type: Book. on design, science, fact and fiction. URL https://bit.ly/2B9rqjg. Type: Journal DiSalvo, C. Design and the construction of publics. 25(1): Article. 48–63, b. Type: Journal Article. DiSalvo, C., Jenkins, T., and Lodato, T. Designing spec- Bolukbasi, T., Chang, K.-W., Zou, J. Y., Saligrama, V., and ulative civics. In Proceedings of the 2016 CHI Con- Kalai, A. T. Man is to computer programmer as woman ference on Human Factors in Computing Systems, pp. is to homemaker? debiasing word embeddings. In Pro- 4979–4990. ACM. ISBN 1-4503-3362-1. Type: Con- ceedings of the 30th Conference on Neural Information ference Proceedings. Processing Systems, pp. 9. Dunne, A. and Raby, F. Speculative Everything: Design, boyd, d. and Crawford, K. Critical questions for big data: Fiction, and Social Dreaming. MIT Press. ISBN 0-262- Provocations for a cultural, technological, and scholarly 01984-1. Type: Book. phenomenon. 15(5):662–679. ISSN 1369-118X, 1468-4462. doi: 10.1080/1369118X.2012.678878. URL Elish, M. C. and Mateescu, A. AI in context: The labor of http://www.tandfonline.com/doi/abs/10.1080/1369118X.2012.678878integrating new technologies. . Participation is not a Design Fix for Machine Learning

Eubanks, V. Automating Inequality: How High-Tech Tools Short Papers, Situated Actions, Workshops and Tutorial- Profile, Police, and Punish the Poor. St. Martin’s Press. Volume 2, pp. 51. ACM. ISBN 1-4503-5574-9. Type: Conference Proceedings. Flanagan, M. and Nissenbaum, H. Values at Play in Digital Games. MIT Press. Type: Book. Hickel, J. The Divide Global Inequality from Conquest to Free Markets. W. W. Norton & Company. Forlano, L. The danger of intimate algorithms. a. Type: Magazine Article. Hillgren, P.-A., Seravalli, A., and Emilson, A. Prototyping and infrastructuring in design for social innovation. 7(3): Forlano, L. Posthumanism and design. 3(1):16–29, b. Type: 169–183. ISSN 1571-0882. Type: Journal Article. Journal Article. Forlano, L. Stabilizing/destabilizing the driverless city: Keshavarz, M. and Maze, R. Design and dissensus: fram- Speculative futures and autonomous vehicles. 13, c. ing and staging participation in design research. 11(1): Type: Journal Article. 7–29. ISSN 1448-7136. Type: Journal Article. Forlano, L. and Halpern, M. Reimagining work: Entangle- Krizhevsky, A., Sutskever, I., and Hinton, G. E. Im- ments and frictions around future of work narratives. ageNet classification with deep convolutional (26):32–59. Type: Journal Article. neural networks. 60(6):84–90. ISSN 0001- 0782, 1557-7317. doi: 10.1145/3065386. URL Forlano, L. and Mathew, A. From design fiction to design https://dl.acm.org/doi/10.1145/3065386. friction: Speculative and participatory design of values- embedded urban technology. 21(4):7–24. Type: Journal Langley, P. Crafting papers on machine learning. In Lang- Article. ley, P. (ed.), Proceedings of the 17th International Con- ference on Machine Learning (ICML 2000), pp. 1207– Gabrilovich, E. and Markovitch, S. Wikipedia- 1216, Stanford, CA, 2000. Morgan Kaufmann. based semantic interpretation for natural lan- guage processing. 34:443–498. ISSN Le Dantec, C. A. and DiSalvo, C. Infrastructuring and the 1076-9757. doi: 10.1613/jair.2669. URL formation of publics in participatory design. 43(2):241– https://jair.org/index.php/jair/article/view/10595264. ISSN 0306-3127.. Type: Journal Article. Gagne, Y. How we fought our landlord’s secre- Martin Jr., D., Prabhakaran, V., Kuhlberg, J., Smart, A., tive plan for facial recognition—and won. URL and Isaac, W. S. Participatory Problem Formulation https://bit.ly/2Ce9ODt. Library Catalog: for Fairer Machine Learning Through Community Based www.fastcompany.com. System Dynamics. arXiv:2005.07572 [cs, stat], May 2020. Goldman, M. Imperial Nature: The World Bank and Strug- gles for Social Justice in the Age of Globalization. Yale Mayer-Schonberger,¨ V. and Cukier, K. Big Data: A Revolu- Agrarian Studies Series. Yale University Press. tion that Will Transform how We Live, Work, and Think. Houghton Mifflin Harcourt. ISBN 978-0-544-00269-2. Gray, M. L. and Suri, S. Ghost Work: How to Stop Silicon Valley from Building a New Global Underclass. Mouffe, C. Pluralism, dissensus and democratic citizen- Houghton Mifflin Harcourt. ISBN 978-1-328-56624-9. ship. pp. 1–10. Type: Journal Article.

Hamraie, A. and Fritsch, K. Crip technoscience Neuhauser, L. and Kreps, G. L. Participatory design and manifesto. 5(1):1–33. ISSN 2380-3312, 2380- artificial intelligence: Strategies to improve health com- 3312. doi: 10.28968/cftt.v5i1.29607. URL munication for diverse audiences. In AI and Health Com- https://catalystjournal.org/index.php/catalyst/article/view/29607munication — Papers from the AAAI. 2011 Spring Sympo- sium (SS-11-01), pp. 4. AAAI. Hansson, K., Forlano, L., Choi, J. H.-j., DiSalvo, C., Pargman, T. C., Bardzell, S., Lindtner, S., and Joshi, S. Nissenbaum, H. How computer systems embody values. Provocation, conflict, and appropriation: the role of the 34(3):117–119. Type: Journal Article. designer in making publics. 34(4):3–7. ISSN 0747-9360. Type: Journal Article. Noble, S. U. Algorithms of Oppression: How Search En- gines Reinforce Racism. NYU Press. Heitlinger, S., Foth, M., Clarke, R., DiSalvo, C., Light, A., and Forlano, L. Avoiding ecocidal smart cities: par- O’Malley, J. Captcha if you can: how you’ve been train- ticipatory design for more-than-human futures. In Pro- ing AI for years without realising it | TechRadar. URL ceedings of the 15th Participatory Design Conference: https://bit.ly/37H1esA. Participation is not a Design Fix for Machine Learning

O’Neal, C. Weapons of Math Destruction: How Big Data Sloane, M. On the need for mapping design in- Increases Inequality and Threatens Democracy. Crown. equalities. 35(4):3–11, b. ISSN 0747-9360, 1531-4790. doi: 10.1162/desi a 00559. URL Peet, R. and Hartwick, E. Theories of Development: Con- https://www.mitpressjournals.org/doi/abs/10.1162/desi_a_00559 tentions, Arguments, Alternatives. The Guildford Press. Sloane, M. The shelves that won’t hold: Material Piketty, T. Capital in the Twenty-first Century. The Belk- politics and social inequality in spatial design nap Press of Harvard University Press. ISBN 978-0-674- practice. c. doi: 10.1162/desi a 00565. URL 43000-6. https://www.mitpressjournals.org/doix/abs/10.1162/desi_a_00565 Roberts, S. T. Behind the sSreen: Content Moderation Publisher: MIT Press One Rogers Street, Cambridge, in the Shadows of Social Media. Yale University Press. MA 02142-1209 USA [email protected]. ISBN 978-0-300-23588-3. OCLC: on1055263168. Spinuzzi, C. The methodology of participatory design. 52 Rosenblat, A. Uberland: How Algorithms are Rewriting (2):163–174. ISSN 0049-3155. Type: Journal Article. the Rules of Work. Univ of California Press. Tarnoff, B. The Making of the Tech Saad-Sulonen, J. and Horelli, L. The value of community Worker Movement. Logic. URL informatics to participatory urban planning and design a https://logicmag.io/the-making-of-the-tech-worker- case study in helsinki. 6(2). Library Catalog: logicmag.io.

Sanders, E. B.-N. From user-centered to participatory de- Tsing, A. Friction: An Ethnography of Global Connection. sign approaches. pp. 1–8. Type: Journal Article. Princeton University Press.

Schuler, D. and Namioka, A. Participatory design : principles and practices. L. Erlbaum Associates. ISBN 0805809511 (cloth) 080580952X (paper). URL http://www.loc.gov/catdir/enhancements/fy0745/92027297-d.html. Type: Book.

Selbst, A. D., Boyd, D., Friedler, S. A., Venkatasubra- manian, S., and Vertesi, J. Fairness and abstraction in sociotechnical systems. In Proceedings of the Confer- ence on Fairness, Accountability, and Transparency - FAT* ’19, pp. 59–68. ACM Press. ISBN 978-1- 4503-6125-5. doi: 10.1145/3287560.3287598. URL http://dl.acm.org/citation.cfm?doid=3287560.3287598.

Sendak, M., Elish, M. C., Gao, M., Futoma, J., Ratliff, W., Nichols, M., Bedoya, A., Balu, S., and O’Brien, C. ”the human body is a black box”: supporting clinical decision-making with deep learning. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency, pp. 99–109. ACM. ISBN 978-1- 4503-6936-7. doi: 10.1145/3351095.3372827. URL http://dl.acm.org/doi/10.1145/3351095.3372827.

Shilton, K. Values levers: Building ethics into design. 38 (3):374–397. doi: 10.1177/0162243912436985. URL http://sth.sagepub.com/content/38/3/374.abstract. Type: Journal Article.

Sloane, M. Inequality is the name of the game: Thoughts on the emerging field of technology, ethics and so- cial justice. a. doi: 10.34669/WI.CP/2.9. URL https://www.ssoar.info/ssoar/handle/document/62583. Publisher: WI - Weizenbaum Institute for the Networked Society Version Number: 1.